51–60 of 114 posts
Anecdotal data. I released a couple of things on product hunt. Popularity-wise one did will, the other went nowhere. Financially, it was completely the opposite. Boring stuff is very successful. I haven't seen a "popular" product hunt thing that I am willing to pay for in ages!! People pay for pain meds, product hunt featured products are colorful vitamins.
Earlier quoted context omitted.
It really depends. There are plenty of legitimate uses for scraping (for example, I've been involved with academic research that involved scraping Twitter search results), and it's only really feasible to collect the amount of data you need using scraping plus paid proxies. That being said, there are also a number of nefarious paid proxy services which offer residential IPs (read: are usually botnets).
What is legitimate to a user is not the same as what is legitimate to a site owner. The legitimate way would probably be to use the Twitter API.
Fair warning: This is a blog post advertisement for ScrapingBee. The data is still interesting. The most interesting chart is one of the last: Proportion of Failures over time. As expected, more recent product links are less likely to 404 or 5xx. Going back to 2014, almost 1/3 of the featured links give a 4xx or a 5xx response. That’s a lot! More surprising, links as recent as 2020 show a 1/4 failure rate. Those proj…
Sure, but it's no different than any other blog post from a company. And framing it that way is quite disingenuous since the post pretty much only sticks to the topic and doesn't overtly promote their product.
My startup is still around, [1] and we posted on PH one or two other times when we launched new products. Even though we had some powerful hunters (thanks to our early presence on the site), I found it took too much time to be worthwhile for follow-on product releases. I'd be interested to know if others have had the same experience, or if they have tips for how to get a meaningful bump out of subsequent posts.
Earlier quoted context omitted.
It is definitely super annoying that companies are allowed to spy on us and do all kinds of crazy things with our data, all using computers and automation and "bots" and such, but individuals are increasingly not allowed to use automation to help us out online. Seems rather one-sided. On the other hand, I get that abuse is a huge problem. I do wish at least bots operating at roughly human request rates & daily total…
Sometimes the scraping situation gets kinda ironic. I worked at a large eRetailer/marketplace and obviously we scraped our major competitors just as they scraped us (there are four major marketplaces here). So each company had a team to implement anti-scraping measures and defeat competitor's defences. Instead of providing an API everyone decided to spend time and money on this useless weapons race.
Earlier quoted context omitted.
"Nefarious" is a strong word. Courts have repeatedly ruled that scraping data that is otherwise available publicly is legal. You may not personally agree with the ethics, but there are a lot of people who do.
I agree it's a strong word, which is why I said borderline nefarious. However, it's not that far off from a DDOS tool. At least in the United States, sounds like the jury is still out on the legality: https://www.reuters.com/technology/us-supreme-court-revives-... , but my perspective was more from an ethics standpoint anyway.
> We consider a 2XX (Success) and 3XX (Redirection) status codes successful I feel like this is flawed, especially considering 1/2 of the successful responses were 3XX. It's possible that they had just linked a short URL that was a redirect, but it's also possible that the product was shuttered and a redirect put in place to a replacement product, the company homepage, or even an acquiring company. I don't think ther…