Live data from Hacker News

How did OpenAI crawl the web so effectively without getting blocked?

news.ycombinator.com

Post reply on HN

Print view View original

1–5 of 5 posts

How did OpenAI crawl the web so effectively without getting blocked?

#1

Post by BrownSol » Fri, Jul 28, 2023, 4:27 AM UTC

No post body was provided.

Re: How did OpenAI crawl the web so effectively without getting blocked?

#2

Post by cratermoon » Fri, Jul 28, 2023, 4:28 AM UTC

It didn't. Others did it, OpenAI used it. https://commoncrawl.org/

Re: How did OpenAI crawl the web so effectively without getting blocked?

#3

Post by arthurcolle » Fri, Jul 28, 2023, 7:51 AM UTC

Very carefully

Re: How did OpenAI crawl the web so effectively without getting blocked?

#4

Post by throwawayadvsec » Fri, Jul 28, 2023, 3:05 PM UTC

I'm pretty sure scraping is easier than building a groundbreaking LLM used by 100M people

Re: How did OpenAI crawl the web so effectively without getting blocked?

#5

Post by JoeyBananas » Fri, Jul 28, 2023, 4:05 PM UTC

It's not hard to scrape the internet

Post reply on HN