Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You need to update those crawlers to enforce some hard limits on crawl depth, methinks, especially for low pagerank domains (if there's some easy way to estimate pagerank, i.e.).


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: