DeWitt Clinton

I’m not sure I even know if I want to write about this issue. For one, it is probably nothing. And two, this is my career — it is a small community, and I bear no ill will toward anyone in it. I’m not suggesting anyone in particular is at fault. But I briefly alluded to this earlier, but it is still weighing on my mind, so I can’t help but muse further.

I noticed a few days ago that my incoming traffic from Google searches all but disappeared.

There was an odd coincidence in the timing of this. It happened just a few days after I wrote a few articles about a new feature on their site. And of course, I work for another major company in the same industry…

As it turns out, those posts were getting a lot of traffic and they were high up in the search results. Google index weights recent blog posts higher for a few days, so I am not surprised that those articles ultimately dropped back down. But I am surprised that suddenly all searches that used to return results at unto.net are now buried deep in the index.

Still curious, I went through my logs and found a half dozen searches that, only a week ago, returned a link to unto.net on the first page of results. Here are a half dozen, taken in reverse chronological order, that once appeared in the top 10 results on Google (try and find them now):

• t9 synonyms
• turn off google history
• perl as an enterprise language
• aim over ssh
• aws opensearch
• dewitt clinton

Today, not one unto.net result is found in the top hundred or so results on Google.

Contrast this with the same queries on Yahoo! Search (current position of an unto.net result in parentheses):

• t9 synonyms (1)
• turn off google history (5)
• perl as an enterprise language (10)
• aim over ssh (5)
• aws opensearch (2)
• dewitt clinton (1)

Each of those queries is ranked in the top 10 or so results on MSN and Ask, too.

It gets weirder. Phrases like “t9 synonyms” and “aws opensearch” are so uncommon that Google’s search results even include my own writing — but only from other sites that quote and link to unto.net. In other words, the results were clearly relevant, but the index completely skips over unto.net itself.

So the question is… why? Why did unto.net get bumped off? I don’t think I violated anyone’s terms of service. I.e., I’ve never tried to push unto.net higher up in the search index. I’ve never used any sort of meta tag or attempted to collect incoming links. I’ve never even spoken with someone in the SEO business, much less used one. In fact, I never particularly cared about search result rankings. I thought it was cool that people were hitting unto.net from search engines, but since they tended to get here via reasonable queries, I just figured things worked out the way they should and never gave it another minute’s consideration.

But here’s a scary thought — what if a major search company, and it doesn’t matter which one — decided they didn’t like the content on someone’s site and censored them. This isn’t political censorship, this isn’t governed by free speech laws or anything serious like that. It might just an arbitrary decision on one individual’s part. Someone might hold a grudge or something. But whatever the cause, does the public have any right to know what is going on behind the scenes? Is there supposed to be any transparency to the process? By all logic, no — there is no accountability, and there probably shouldn’t be. I definitely wouldn’t want the government interfering with something like this.

Let’s be honest here — I don’t actually think anyone decided to censor unto.net. Sure, it bugs me that it happened, but it bothers me more that it can happen, and that we won’t know why or how or by whom. This is just a little tech blog, so no one, not even me, really cares that much about what I say here. But it could matter someday on an issue of far greater importance, and someone needs to ask the question.

Update 2005-05-23:

Thank you for the emails. I just want to make sure this is clear — no, I do not think that unto.net was intentionally removed from anyone’s search index. I am sure that there is perfectly normal explanation for what happened, and that it will be fixed in the next reindex.

The reason I mentioned it at all was because the coincidental timing of the ranking drop made me start thinking about the trust issues associated with search engines. As we know, these issues are relevant to me on a daily basis.

So the question is, what if a search engine, any search engine, can arbitrarily decide what content it does or does not expose. Google has done a very good job — better than anyone else — distinguishing sponsored search results from “normal” algorithmic search results. In fact, I believe a large part of their popularity is due entirely to this. We feel we can trust their results because we know that they are not trying to fool us into clicking on a result that someone paid for.

But the question remains — in an environment when ranking and relevancy algorithms are secret and proprietary, what is to stop any search engine from executing editorial control over content? And since we have grown to trust those algorithms to return the “best” results, what happens when they don’t? Are we the public educated enough by the search engine companies to know that the search engines — similar to the media — can be biased? We trust the computers because they supposedly don’t have bias. But those computers were programmed by humans with bias, and a simple flip of a few bits by a human can subtly alter what we see. Without transparency into those hidden processes, we will never know the truth.

Case in point: nearly all interview candidates I meet with mention that they took a look at unto.net beforehand. And nearly all of them arrived here by doing a search for “DeWitt Clinton” or “DeWitt Clinton A9″. Up until a few days ago, that would clearly return this site. Now that unto.net is dropped from the index, many of those candidates will never see these pages. This will have a subtle, but direct, impact on how their interview goes. Since I talk about the types of things that are important to us and can help them prepare, this may even change the outcome of the interview. So yes, it is subtle, but the act of questioning objectivity and truth is always a subtle concern.

This entry was posted on Sunday, May 22nd, 2005 by DeWitt Clinton and is filed under meta. You can bookmark this post at del.icio.us. You can digg this post. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.