I know many people that follow limited/exclusive releases for things like Yeezy/Air Jordan sneakers as well as PS5's and graphics cards. They pay $500/mo for access to a bot that will allow them to make these purchases. Most of the community lives on discord.
It would be relatively easy to solve this problem if the original supplier wanted the problem to be solved. Instead of releasing a batch of inventory at a certain time, run a raffle over a week or two and then randomly select folks to allow to purchase the item.
I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…
There are pro-social and anti-social uses of web scraping. If you have ever used Kayak or any other price discovery or price comparison website, you've relied on web scraping to provide you a service.
I believe Kayak has agreements with the sites they scrape though. So it's a different type of "scraping", really.
Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably other, better options. APIs, data set downloads, etc. https://commoncrawl.org/
I wish web.archive.org had an index by someone like common crawl. There is lots of great stuff on archive.org
web.archive.org has a CDX index, similar to Common Crawl.
Since I use both of these archives together, I wrote this code to iron out the differences between them:
Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably other, better options. APIs, data set downloads, etc. https://commoncrawl.org/
Common Crawl is missing far too many URLs for it to be useful in a real world scenario.
Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably other, better options. APIs, data set downloads, etc. https://commoncrawl.org/
Common Crawl is missing far too many URLs for it to be useful in a real world scenario.
Haha I love that people forget how google/bing are out there scraping everything and anyone who scrapes anything for any other reason is a "bad guy". You can get around some web scraping blockers by just setting your user agent as Googlebot too which I find funny...
No they don't, Google and Bing respect robots.txt. Most websites would open it up to them because they need the traffic, so it's a type of scraping that is beneficial. Any other scraping, especially when ignoring robots.txt, is unsolicited. And if said website takes additional advanced anti-scraping measures, and you persist in bypassing that too, then to me you're clearly unethical, even if it's technically legal. "…
> No they don't, Google and Bing respect robots.txt.
Common Crawl is missing far too many URLs for it to be useful in a real world scenario.
But can't you add to their index?
No. You can add to the Wayback Machine at web.archive.org via their "save page now" interface... Common Crawl is attempting to be a sample of the web, and doesn't take url suggestions.
I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…
> people who are trying to build businesses on top of other people's innovation and data
How would scraping, say, reddit, differ from the business model of Reddit itself?
> those that are doing it to my platforms are doing so solely to steal data
I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opi…
Google is web scrapper number one, as any search engine. Making web scrapping illegal mean making search engine illegal. You do not want information to be public and/or free? Put it under login and charge for it. You want to prevent people to reuse the data you publish to build other (potentially competitive) products, then use licensing and copyright, and the law. However, banning a technological mean because what a…
What if Google didn't scrape websites automatically, and waited till users submit their domains to them, to mark that they want to be scraped? I think in that case, most users would still submit their domains there, because they want to come up in Google search. You might want your website to be scraped by some people/companies and not by others, but not have to put everything behind a login screen (which some determined scrapers would still try to breach in some way).
It would be relatively easy to solve this problem if the original supplier wanted the problem to be solved. Instead of releasing a batch of inventory at a certain time, run a raffle over a week or two and then randomly select folks to allow to purchase the item.
Wouldn’t a queuing system be more fair?
By queuing, do you mean first come first serve?
No, that causes the problem. That encourages people to use bots to be the first one to purchase the moment the inventory is released.
I don't understand how a random raffle would ever not be fair (with the assumption that one person gets only one entry)