ContentJune 16th, 2005 by DeWitt Clinton
Whoo. Unto.net is back in the Google index. It’s funny — you can’t miss the difference in the logs. 3 out of the last 10 referrers were all via Google. Whereas even though unto.net ranks high on Yahoo, MSN, and Ask, barely one out of 50 hits come from any of those search engines. That goes to show just how substantial the difference is between the number of users of each site…
Now we just need to get the search for DeWitt Clinton back up there… Do that many people really care about the $1000 dollar bill?
June 16th, 2005 at 10:28 am
I’ve always wondered–are you named after that other DeWitt Clinton?
June 16th, 2005 at 10:38 am
Er. I’m named after my father. (Who was named for his father, who was named for his father, and so on, and so on back a few more generations) So in a transitive sense, yes. : )
June 16th, 2005 at 12:39 pm
Actually, I think it says more about what those sites users typically search for.
June 16th, 2005 at 1:24 pm
I was wondering that myself… But my guess would be that Yahoo Search’s users are the most technically sophisticated, MSN’s the least, and Google’s all over the map…
My last 5 searches coming from Yahoo were for:
And the last from Google:
One scary thing is that I’ve actually written at length about all of those things in the past year. Jack of all trades, master of….
June 16th, 2005 at 1:24 pm
Hm. Maybe it was a re-indexing issue after your WordPress migration? I’ll keep an eye on what happens to my site in the next few days… But I’m still showing up #1 for “linkstew” at the moment.
June 16th, 2005 at 9:00 pm
Benjy — it might have had to do with the migration, but the timing just didn’t seem right. And looking at your site in their index, it seems like they’ve already picked up your new content, so I wouldn’t expect a drop.
Then again, the talk is about how Google is getting serious about spam — so maybe I did something wrong and they shot first and asked questions later. Not that I would argue against that (search engine spam drives me nuts, too) but I would really like a way of getting more information about why something like that happens. I mean, imagine if you were paying rent off of ad revenues…
June 17th, 2005 at 7:33 pm
Unfortunately, I realized there were some additional variables regarding my migration: 1. before I figured out my mod_rewrite bug, I set up a simple redirector that was returning 301s, which googlebot was handling strangely. 2. Shortly after my migration, I also added adsense to my comment pages, which will supposedly induce a reindex if the pages the ads are on are unknown. Since google hadn’t yet indexed my new link format, I got a flood of googlebot requests not too long after adding adsense.
I think I’m going to change the old link format back to 301ing later to hopefully flush those old links out of the system eventually.
June 18th, 2005 at 8:24 am
I had Googlebot freak out on me for a while, too. It was crawling every random permutation of paths like “://2004/10/2004/05/” over and over. I have absolutely no idea where it was getting them. Most were 404s, but some were actually valid because they had a search query in them.
I ended up just blocking the root of it with robots.txt. Googlebot picked that up a few hours later and then all was good. I also noticed that the bot is now doing a complete crawl every other day, with a steady scattering of hits all day long. It is also pulling the feeds (including comment feeds) on a very regular basis. It seems to treat RSS feed URLs differently than HTML URLs. The main index seems to be updating a bit less than once a month, but their real-time updates are coming in well under a day. Impressive stuff.