Thursday, January 04, 2007

Why Search Engines Annoy Me

Don't get me wrong - I use search engines a lot, and find them invaluable for navigating the Web. But for finding new sites whose owners don't have the resources to launch an advertising or public relations campaign? They're not so good any more.

An oversimplified history: It wasn't so long ago that if you put up a website and worked to develop it, it could become an authority site. People relied heavily on directories to locate sites, listings were free, and search engines focused to a significant degree on page content when determining relevance. Then came the rise of "pay for inclusion" coupled with Google. Directories became less and less important to traffic, while search engines became dominant. Initially this didn't make much difference, as Google's big advance over other search engines was its analysis of linking between sites and pages. That type of weighting was emulated by other sites. But problems developed for the smaller webmaster.

First, as search engines became dominant, the proliferation of "links pages" and small directories that used to be found all over the Internet started to diminish, and their owners became less interested in maintaining pages of links, making it harder to develop "natural" links. Second, many of what were once common mechanisms for building links (e.g., trading links with another webmaster) are automatically suspect and likely to be devalued by search engines, as they are so easily abused. From a webmaster's perspective, the best links are one-way links from topically relevant pages of popular sites.

For a new site, no matter how good, the owner usually faces the conundrum that the site lacks sufficient links to rise to the top of search engine results pages, and as nobody is finding the site through search engines it isn't developing "natural" links... and thus languishes in obscurity. The response you often hear from representatives of the search engines is that quality sites will develop links over time - but from what I have seen, that's usually not the case. When it is the case it is often because the site has invested in a link development service, which charges an hourly or "per link obtained" fee to obtain one-way links to the new website.

Oh, how painful it can be watching a lousy website float at the top of the SERPs, like a turd in a septic tank, just because "they got there first".

Blogging to some extent revived natural linking - bloggers link to other weblogs that they like, usually whether or not they get a link back. Many early weblogs gained lots of links, thus being classified as "important" by search engines. Newer bloggers face the problem that many established bloggers neglect or simply choose not to update or expand their "blogrolls", and from the fact that no matter how good their content there are now usually other blogs which address the same topic reasonably well, making it harder for them to stand out. Also, blog searches have to a significant degree been shifted from regular search engine results to dedicated blog search interfaces, making it less likely that a new blogger will be found through a standard search engine.

Until the next big thing comes along, and you can again get in on the ground floor, the status quo makes it difficult to develop traffic to a new site. The work-around, perhaps, is to find an on-topic site with a reasonable URL and to try to acquire it from its current owner. (Anticipate, though, that the current owner will believe that a site that hasn't been updated in five years, and which looks like it was designed by a color-blind eight-year-old, will insist that the site is worth a small fortune.) But given the cost of generating similar traffic through advertising, on may commercial subjects they may be right.


  1. I recall about 8 years ago I had a site for a video game that floated to the top, or near the top, of many search engines and in doing so gained traffic I never thought it would. It was decent, it offered various mods etc for the game as well as the ability to have new ones uploaded by users.

    I've definately noticed that sites like this, useful ones, but not sites out to make money or spend any, seem to be very hard to find now whereas they used to be the norm . Now the the standard stuff found by search engines seem to be industry sites that a basically online magazines that may only offer a paragraph or two on what you're looking for and no interaction.

    It's nice to know what caused this shift at least and I have to agree that it is a very bad shift.

  2. I had no idea, when I started my web site in 1996, that I was getting in on the ground floor. But at the time, there was very little American history on the Web, and my early start led to abundant link popularity.

    I once estimated that the effort to build my site would have cost at least $250,000 if someone had hired appropriately skilled and credentialed people to do all the work. Probably the site would be in better shape today if it had full-time people to work on it, but I doubt the financial returns would have justified any of it. If I was just in it for the money, it would have been a huge loss.

    A lot of now-popular web sites would never have been created or survived if they were subject to a strict rate of return calculus all along.

    I'm guessing that a new site that was rich with with unique, interesting, useful content could find a sizeable niche market. But with billions of web sites out there, it would be difficult today to come up with a uniquely good idea.

    Very much agreed that online writing and commentary, regardless how sparkling, is not easy to market these days. Specialized commentary, however, may offer some opportunities.


Note: Only a member of this blog may post a comment.