Why Is It So Hard To Search Inside A Single Website Compared To The Entire WWW?

Those of you familiar with the world of search before Google would know that Google’s search engine attained massive following because of its blazing speed and exhaustive coverage of the Internet. Awed by the speed of Google Search across billions of websites spread in the far corners of the world wide web, have you ever wondered

1. Why does it take far longer to do a Windows Search of your PC’s hard disk?
2. Why do search results on one website leave a lot to be desired even when you’re using Google CSE (Custom Search Engine) to conduct your search?
3. Why is it so hard to find a particular transaction within your Internet Banking transaction history?
4. Why is it even more difficult to search for a particular post or comment on a blog? As TechCrunch says, “TechCrunch has published thousands of blog posts over its nearly 5 and a half years. Many are good one-day stories, some we’d like to forget, but others are gems. These classics are just as interesting today as when they were first written … But try discovering them. It’s nearly impossible.” By the way, TC uses Google CSE for site search.

I’ve obsessed over this subject for a long time. Only recently I learned that this is not a problem with any specific website. Since Google and many other search engines rank a web page on the basis of backlinks to it from other websites, they fail to do a good job within a website where the concept of backlinks doesn’t apply.

Is there a solution to this problem, then?

Looks like I stumbled on to one.

I was recently trying to recall the name of a PPC search engine about which I’d read a post on TechCrunch a few weeks ago. When I used Google CSE to find this post on TC, I got long list of results but none of them took me to the right one.

I was about to give up in exasperation when it suddenly struck me that, like most posts on TechCrunch, this one also might’ve been retweeted a few times. I did a Twitter Search for “TechCrunch PPC” and struck paydirt. There were only two results to go through and both of them took me to the TC post that I was looking for.

Given the realtime nature of Twitter, I’m not sure if its search would be as effective a few days or weeks later, but for now, it works!

UPDATE DATED 15 APRIL 2020:

I know it’s damn hard to search inside a single website compared to WWW but WhatsApp takes the cake: I can manually scroll through my chats to find a message faster than getting the results from WhatsApp Search.