I think that you're building an unethical business, and should not get advice or publicity from this community.
Doubly so given you already had a very high visibility Show HN for this just a few months ago.
Show HN: MrScraper AI – Dead simple web scraper (powered by AI)
11–20 of 30 posts
Re: Show HN: MrScraper AI – Dead simple web scraper (powered by AI)
#12Maybe the tag line could be “extract anything from the web without the selectors” or something. Then put your ai stuff in the subtitle. Either way, currently the grammar doesn’t quite work right now. Definitely a cool idea though! Web scraping needs to be 100% deterministic. That would be my only question is if/how you’ve achieved that.
You’re not only sacrificing simplicity for determinism but also stability. What good is being deterministic when the underlying web page keeps changing all the time, breaking your selectors? This seems like a more stable approach.
However if the selectors break you can notice that quite easily.
Re: Show HN: MrScraper AI – Dead simple web scraper (powered by AI)
#13Maybe the tag line could be “extract anything from the web without the selectors” or something. Then put your ai stuff in the subtitle. Either way, currently the grammar doesn’t quite work right now. Definitely a cool idea though! Web scraping needs to be 100% deterministic. That would be my only question is if/how you’ve achieved that.
> Web scraping needs to be 100% deterministic
Says who?
Re: Show HN: MrScraper AI – Dead simple web scraper (powered by AI)
#14Cool project but isn’t the first rule of proxy rotation not to talk about proxy rotation?
Shh! Don't talk about it
Re: Show HN: MrScraper AI – Dead simple web scraper (powered by AI)
#15This will definitely take away the burden of clients (mostly non-technical people) having to choose the selectors. I've had a scraping service business recently for this specific reason. I hope AI can be a great middle player here. Let's see how it turns out.
Good luck Kai.
Re: Show HN: MrScraper AI – Dead simple web scraper (powered by AI)
#16I think I understand pagination — but can you elaborate on proxy rotation?
> combines the practicality of language models with the powerful features of a traditional scraper such as pagination and proxy rotation
Re: Show HN: MrScraper AI – Dead simple web scraper (powered by AI)
#17I think I understand pagination — but can you elaborate on proxy rotation? > combines the practicality of language models with the powerful features of a traditional scraper such as pagination and proxy rotation
You basically switch out the proxy you use to scrape. Services by Google or others can identify scrapers cause they'll use the same proxy to request paged
Re: Show HN: MrScraper AI – Dead simple web scraper (powered by AI)
#18It's just a landing page with a video and an email signup form. No usable product. Is this kind of thing allowed on Show HN threads?
Re: Show HN: MrScraper AI – Dead simple web scraper (powered by AI)
#19I think that you're building an unethical business, and should not get advice or publicity from this community. Doubly so given you already had a very high visibility Show HN for this just a few months ago.
I don't think this community agrees with you that web scraping is inherently unethical.
In fact I think many (most?) in this community would argue that web scraping is an almost fundamental feature of the web itself, and that attempts at preventing it are more unethical than scraping.
Re: Show HN: MrScraper AI – Dead simple web scraper (powered by AI)
#20Maybe the tag line could be “extract anything from the web without the selectors” or something. Then put your ai stuff in the subtitle. Either way, currently the grammar doesn’t quite work right now. Definitely a cool idea though! Web scraping needs to be 100% deterministic. That would be my only question is if/how you’ve achieved that.
> Web scraping needs to be 100% deterministic Says who?
well evidently the potential customer above, deterministic though can mean multiple things - for example if you are scraping the major headline on a page based on a selector and the class is .headline, when .headline goes away and the content becomes
My cool fashions!
well someone can say that the crawler should get the new version of the title, but others can say it should warn you that the selector no longer works.
If the crawler determines via AI what the headline is and you have crawled 10000 pages and it turns out the crawler has made a mistake regarding the headline then you might be soured on the idea of AI making this kind of decision for you and be more amenable to being warned, but then you have to do a lot more work with your crawler than you might otherwise want to do.