Live data from Hacker News

Ask HN: Scaling a targeted web crawler beyond 500M pages/day

news.ycombinator.com

Post reply on HN

Print view View original

11–13 of 13 posts

Re: Ask HN: Scaling a targeted web crawler beyond 500M pages/day

#11

Post by ccgreg » Sun, Apr 26, 2026, 6:18 AM UTC

fragmede wrote: post #8

have you already incorporated common crawl into your index?

Common Crawl is a sample of the web, so it's not that directly helpful for someone wanting to make a product price dataset.

Re: Ask HN: Scaling a targeted web crawler beyond 500M pages/day

#12

Post by nicbou » Sun, Apr 26, 2026, 10:14 AM UTC

jeong_jeong wrote: post #6

nicbou wrote: post #5

Earlier quoted context omitted.

Our industry's understanding of consent is terrifying

It’s called hacker news, bro

The hacker ethos is to think for yourself and find ways around rigid social structures. It is not to plunder without thought.

Re: Ask HN: Scaling a targeted web crawler beyond 500M pages/day

#13

Post by ddactic » Tue, Apr 28, 2026, 10:59 AM UTC

[dead]

Post reply on HN