Do you need to scrape data from 100s to 1000s of websites?

#1

Post by asim-shrestha » Fri, Sep 22, 2023, 7:34 PM UTC

We've been working on AI agents and in the process found what we built could be good at dynamically scraping data across a variety of websites (10s to 100s of different sites at a time)

Existing web scraping tools are bad at this because you need to write custom configurations per site. Not only that, but when a site changes styling, it might completely break your automation

We’re curious how useful this might be for people. If you’ve experienced issues that this might solve or have already explored the space, we’d love to talk to you!

Re: Do you need to scrape data from 100s to 1000s of websites?

#2

Post by airstrike » Fri, Sep 22, 2023, 7:40 PM UTC

you should post this either as a Show HN or Launch HN post, explaining why it's relevant. This current post feels spammy and requires readers to copy-and-paste the link*

https://news.ycombinator.com/shownew

https://hn.algolia.com/?q=Launch+HN

having said that, it seems like you may have already posted? https://hn.algolia.com/?q=reworkd.ai

Re: Do you need to scrape data from 100s to 1000s of websites?

#3

Post by asim-shrestha » Fri, Sep 22, 2023, 7:43 PM UTC

airstrike wrote: post #2

you should post this either as a Show HN or Launch HN post, explaining why it's relevant. This current post feels spammy and requires readers to copy-and-paste the link* https://news.ycombinator.com/shownew https://hn.algolia.com/?q=Launch+HN having said that, it seems like you may have already posted? https://hn.algolia.com/?q=reworkd.ai

Ah, none of those existing posts are relevant to this. Updated the messaging a bit here to sound less spammy - thank you

Additionally, we don't actually have something to show/launch. Was just interested in hearing perspectives of people that have had this problem or have tried to tackle this problem

Re: Do you need to scrape data from 100s to 1000s of websites?

#4

Post by airstrike » Fri, Sep 22, 2023, 7:49 PM UTC

asim-shrestha wrote: post #3

airstrike wrote: post #2

you should post this either as a Show HN or Launch HN post, explaining why it's relevant. This current post feels spammy and requires readers to copy-and-paste the link* https://news.ycombinator.com/shownew https://hn.algolia.com/?q=Launch+HN having said that, it seems like you may have already posted? https://hn.algolia.com/?q=reworkd.ai

Ah, none of those existing posts are relevant to this. Updated the messaging a bit here to sound less spammy - thank you Additionally, we don't actually have something to show/launch. Was just interested in hearing perspectives of people that have had this problem or have tried to tackle this problem

Got it. Maybe an Ask HN then? Otherwise the question may not be noticed in the hundreds of daily submissions. It may be worth resubmitting as an Ask HN. The question mark makes it sound like the start of an infomercial, but it may just be me ;-)

Re: Do you need to scrape data from 100s to 1000s of websites?

#5

Post by verdverm » Fri, Sep 22, 2023, 7:49 PM UTC

asim-shrestha wrote: post #3

airstrike wrote: post #2

you should post this either as a Show HN or Launch HN post, explaining why it's relevant. This current post feels spammy and requires readers to copy-and-paste the link* https://news.ycombinator.com/shownew https://hn.algolia.com/?q=Launch+HN having said that, it seems like you may have already posted? https://hn.algolia.com/?q=reworkd.ai

Ah, none of those existing posts are relevant to this. Updated the messaging a bit here to sound less spammy - thank you Additionally, we don't actually have something to show/launch. Was just interested in hearing perspectives of people that have had this problem or have tried to tackle this problem

use AskHN then

Re: Do you need to scrape data from 100s to 1000s of websites?

#6

Post by asim-shrestha » Fri, Sep 22, 2023, 7:52 PM UTC

airstrike wrote: post #4

asim-shrestha wrote: post #3

Earlier quoted context omitted.

Ah, none of those existing posts are relevant to this. Updated the messaging a bit here to sound less spammy - thank you Additionally, we don't actually have something to show/launch. Was just interested in hearing perspectives of people that have had this problem or have tried to tackle this problem

Got it. Maybe an Ask HN then? Otherwise the question may not be noticed in the hundreds of daily submissions. It may be worth resubmitting as an Ask HN. The question mark makes it sound like the start of an infomercial, but it may just be me ;-)

Gotcha, will do

Appreciate the feedback, very new to posting here