Google.com Site Search Not Achieving Goal

Google.com Site Search Not Achieving Goal
We use a Google.com script inside the page that allow users of HoboTraveler.com to search the site.

The goal is to search the whole site of HoboTraveler.com, however this is not working.

---------------------------------
Bangkok, Thailand
Wednesday, October 22, 2008
Blog of Andy HoboTraveler.com --- Add a Hotel --- Travel Bag Design Survey --- Professional Traveler Bag
----------------------------------

Confusingly complicated and troublesome to understand, however I am annoyed by Google.com one of my best benefactors.

This is the problem, Google.com only will provide search for pages that it deems worthy of being included in its search index. It uses a complicated algorithm to make this decision. Because my sites are huge, I often I use Google.com to search for pages in my own site and I am finding too many pages missing.

Well, what has happened is that Google.com has at an ever-increasing rate indexed less and less pages. They age a page and when it become of older they drop it out of the search or put it into the supplemental pages, the omitted group at the bottom of many searches and often they just refuse to index pages.

BLOG SEARCH

Blogs are a special problem for Google.com, the majority of Blog are full of irrelevant and valueless information. I do understand when they do not wish to include my Blog post in the general index. They do have a point there, some of my post are not for prime time reading and are rather irrelevant.

However, when they Google.com search team does not allow me to search within my own site there is a problem. What is relevant to me is my site, and I should be able to search my whole site.

Sadly, we are in the process of making our own proprietary search system that will allow users to search the complete sites.

I would assume we are not the first person to think of this, and maybe there are some off the shelf answers that can work. We truly need all the pages indexed and a way for readers or me to search the whole site with no pages omitted.

Google.com Site Search Not Achieving Goal

« Previous | Next »

Newer Posts

4 comments

Join the conversation!

Have you tried creating a sitemap? I have tried looking for one on your site, but can't find one. Typically, they're referenced from robots.txt.

I would typically refrain from creating your own search engine. They are *very* difficult to do well, and even if you invest hundreds of man hours and many thousands of dollars you're unlikely to do it as well as a google search of your website.

Even off the shelf solutions are probably not as good as google will be (though they will usually be better than anything you create yourself).

Google's indexing algorithm is usually very good, but it works much better if you provide it with hints. This can be done by creating a denser internal link structure (e.g. bringing together links to related pages, creating mini-landing pages) and creating a sitemap (http://en.wikipedia.org/wiki/Site_map).

In general, the more links away from the main page something is the less likely is to index it. If you rearranged your main page so that most things were available from the main page in 2 or 3 clicks I bet all of your site would be indexed again.

This is partly why I advised you to move your information to mediawiki. I believe that would make it easier to add context and create a taxonomy that would enable humans and search engine spiders alike dig deeper into your content and find useful info. However, just re-arranging your link structure will probably do just as well.

We do all that, and much much more...

Hi Andy,

My company, SLI Systems, has a solution you might be interested. It may well be more cost effective than you building something yourself - it will almost definitey be a better solution and it will be easier for you. You can see our search working on a blog at Search Engine Land. contact us at sales@sli-systems.com if you're interested.

Thanks for this article...

Google.com Site Search Not Achieving Goal

Join the conversation!

Travel Photos

Find us here:

Google.com Site Search Not Achieving Goal

Blog > Thailand

Join the conversation!

Travel Photos

Find us here: