Confusingly complicated and troublesome to understand, however I am annoyed by Google.com one of my best benefactors.
This is the problem, Google.com only will provide search for pages that it deems worthy of being included in its search index. It uses a complicated algorithm to make this decision. Because my sites are huge, I often I use Google.com to search for pages in my own site and I am finding too many pages missing.
Well, what has happened is that Google.com has at an ever-increasing rate indexed less and less pages. They age a page and when it become of older they drop it out of the search or put it into the supplemental pages, the omitted group at the bottom of many searches and often they just refuse to index pages.
Blogs are a special problem for Google.com, the majority of Blog are full of irrelevant and valueless information. I do understand when they do not wish to include my Blog post in the general index. They do have a point there, some of my post are not for prime time reading and are rather irrelevant.
However, when they Google.com search team does not allow me to search within my own site there is a problem. What is relevant to me is my site, and I should be able to search my whole site.
Sadly, we are in the process of making our own proprietary search system that will allow users to search the complete sites.
I would assume we are not the first person to think of this, and maybe there are some off the shelf answers that can work. We truly need all the pages indexed and a way for readers or me to search the whole site with no pages omitted.