Why Search Engines Annoy Me

Don't get me wrong - I use search engines a lot, and find them invaluable for navigating the Web. But for finding new sites whose owners don't have the resources to launch an advertising or public relations campaign? They're not so good any more.

An oversimplified history: It wasn't so long ago that if you put up a website and worked to develop it, it could become an authority site. People relied heavily on directories to locate sites, listings were free, and search engines focused to a significant degree on page content when determining relevance. Then came the rise of "pay for inclusion" coupled with Google. Directories became less and less important to traffic, while search engines became dominant. Initially this didn't make much difference, as Google's big advance over other search engines was its analysis of linking between sites and pages. That type of weighting was emulated by other sites. But problems developed for the smaller webmaster.

First, as search engines became dominant, the proliferation of "links pages" and small directories that used to be found all over the Internet started to diminish, and their owners became less interested in maintaining pages of links, making it harder to develop "natural" links. Second, many of what were once common mechanisms for building links (e.g., trading links with another webmaster) are automatically suspect and likely to be devalued by search engines, as they are so easily abused. From a webmaster's perspective, the best links are one-way links from topically relevant pages of popular sites.

For a new site, no matter how good, the owner usually faces the conundrum that the site lacks sufficient links to rise to the top of search engine results pages, and as nobody is finding the site through search engines it isn't developing "natural" links... and thus languishes in obscurity. The response you often hear from representatives of the search engines is that quality sites will develop links over time - but from what I have seen, that's usually not the case. When it is the case it is often because the site has invested in a link development service, which charges an hourly or "per link obtained" fee to obtain one-way links to the new website.

Oh, how painful it can be watching a lousy website float at the top of the SERPs, like a turd in a septic tank, just because "they got there first".

Blogging to some extent revived natural linking - bloggers link to other weblogs that they like, usually whether or not they get a link back. Many early weblogs gained lots of links, thus being classified as "important" by search engines. Newer bloggers face the problem that many established bloggers neglect or simply choose not to update or expand their "blogrolls", and from the fact that no matter how good their content there are now usually other blogs which address the same topic reasonably well, making it harder for them to stand out. Also, blog searches have to a significant degree been shifted from regular search engine results to dedicated blog search interfaces, making it less likely that a new blogger will be found through a standard search engine.

Until the next big thing comes along, and you can again get in on the ground floor, the status quo makes it difficult to develop traffic to a new site. The work-around, perhaps, is to find an on-topic site with a reasonable URL and to try to acquire it from its current owner. (Anticipate, though, that the current owner will believe that a site that hasn't been updated in five years, and which looks like it was designed by a color-blind eight-year-old, will insist that the site is worth a small fortune.) But given the cost of generating similar traffic through advertising, on may commercial subjects they may be right.