Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I had the same "fun" idea 6 years ago. In 3 months Yahoo Site Explorer reported 150 million pages indexed. As of today Google still shows 55 million pages (in comparison, https://www.google.com/search?q=site:http://en.wikipedia.org... reports 34 million pages).

I had to kill the experiment (no more new "pages" crawled) because of the CPU load and bandwidth costs, even throttling robots



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: