Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are Product Hunt's featured products still online today?

scrapingbee.com

Post reply on HN
Print view View original

41–50 of 114 posts

Re: Are Product Hunt's featured products still online today?

#41

Post by whakim »

BlewisJS wrote: post #16

Unrelated to the article - is it just me or is this scrapingbee product borderline nefarious? From the homepage: > Thanks to our large proxy pool, you can bypass rate limiting website, lower the chance to get blocked and hide your bots! > Scrapingbee helps us to retrieve information from sites that use very sophisticated mechanism to block unwanted traffic, we were struggling with those sites for some time now and I'…

It really depends. There are plenty of legitimate uses for scraping (for example, I've been involved with academic research that involved scraping Twitter search results), and it's only really feasible to collect the amount of data you need using scraping plus paid proxies. That being said, there are also a number of nefarious paid proxy services which offer residential IPs (read: are usually botnets).

Re: Are Product Hunt's featured products still online today?

#42

Post by bydlocoder »

brimble wrote: post #30
stickfigure wrote: post #19

Earlier quoted context omitted.

No more nefarious than the measures websites put up to avoid scrapers? This just rehashes the Linkedin vs Hiq case: https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn (not a user, but I do some amount of scraping through other means)

It is definitely super annoying that companies are allowed to spy on us and do all kinds of crazy things with our data, all using computers and automation and "bots" and such, but individuals are increasingly not allowed to use automation to help us out online. Seems rather one-sided. On the other hand, I get that abuse is a huge problem. I do wish at least bots operating at roughly human request rates & daily total…

Sometimes the scraping situation gets kinda ironic. I worked at a large eRetailer/marketplace and obviously we scraped our major competitors just as they scraped us (there are four major marketplaces here). So each company had a team to implement anti-scraping measures and defeat competitor's defences. Instead of providing an API everyone decided to spend time and money on this useless weapons race.

Re: Are Product Hunt's featured products still online today?

#43

Post by vorpalhex »

brimble wrote: post #30
stickfigure wrote: post #19

Earlier quoted context omitted.

No more nefarious than the measures websites put up to avoid scrapers? This just rehashes the Linkedin vs Hiq case: https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn (not a user, but I do some amount of scraping through other means)

It is definitely super annoying that companies are allowed to spy on us and do all kinds of crazy things with our data, all using computers and automation and "bots" and such, but individuals are increasingly not allowed to use automation to help us out online. Seems rather one-sided. On the other hand, I get that abuse is a huge problem. I do wish at least bots operating at roughly human request rates & daily total…

A company my previous employer partnered with once asked us to integrate with them.. via scraping and using bots to fill out forms.

Which would have been fine except they also imposed terribly low rate limits with no ability to check them.

We eventually pulled the partnership since it was more work than value.

Re: Are Product Hunt's featured products still online today?

#44

Post by NicoJuicy »

fjabre wrote: post #40
BlewisJS wrote: post #16

Unrelated to the article - is it just me or is this scrapingbee product borderline nefarious? From the homepage: > Thanks to our large proxy pool, you can bypass rate limiting website, lower the chance to get blocked and hide your bots! > Scrapingbee helps us to retrieve information from sites that use very sophisticated mechanism to block unwanted traffic, we were struggling with those sites for some time now and I'…

Nefarious? Then they should arrest Google first, it is the king of web scrapers.

Robots.txt

Re: Are Product Hunt's featured products still online today?

#45

Post by ianwootten »

jastr wrote: post #36

> there's actually proportionally less failures in Product Hunts busiest period This is a really interesting post! I think there's a little survivorship bias. As Product Hunt grew 2015-2017, users posted old projects of theirs which were already popular and successful.

Glad you enjoyed the post - I hadn't considered this.

Re: Are Product Hunt's featured products still online today?

#46

Post by sasquatch69 »

PragmaticPulp wrote: post #3

Fair warning: This is a blog post advertisement for ScrapingBee. The data is still interesting. The most interesting chart is one of the last: Proportion of Failures over time. As expected, more recent product links are less likely to 404 or 5xx. Going back to 2014, almost 1/3 of the featured links give a 4xx or a 5xx response. That’s a lot! More surprising, links as recent as 2020 show a 1/4 failure rate. Those proj…

Fair warning: this response is an advertisement for ycombinator

Re: Are Product Hunt's featured products still online today?

#47

Post by jedberg »

> We consider a 2XX (Success) and 3XX (Redirection) status codes successful

I feel like this is flawed, especially considering 1/2 of the successful responses were 3XX. It's possible that they had just linked a short URL that was a redirect, but it's also possible that the product was shuttered and a redirect put in place to a replacement product, the company homepage, or even an acquiring company. I don't think there is an easy way to tell based just on the response code, and I'm not sure you could even programmatically determine it unless you had samples of what the pages looked like on launch day (maybe compare today vs the Internet Archive?).

Re: Are Product Hunt's featured products still online today?

#48

Post by BlewisJS »

paxys wrote: post #39
BlewisJS wrote: post #16

Unrelated to the article - is it just me or is this scrapingbee product borderline nefarious? From the homepage: > Thanks to our large proxy pool, you can bypass rate limiting website, lower the chance to get blocked and hide your bots! > Scrapingbee helps us to retrieve information from sites that use very sophisticated mechanism to block unwanted traffic, we were struggling with those sites for some time now and I'…

"Nefarious" is a strong word. Courts have repeatedly ruled that scraping data that is otherwise available publicly is legal. You may not personally agree with the ethics, but there are a lot of people who do.

I agree it's a strong word, which is why I said borderline nefarious. However, it's not that far off from a DDOS tool.

At least in the United States, sounds like the jury is still out on the legality: https://www.reuters.com/technology/us-supreme-court-revives-..., but my perspective was more from an ethics standpoint anyway.

Re: Are Product Hunt's featured products still online today?

#49

Post by sydthrowaway »

71a54xd wrote: post #14

My favorite un-ironic ProductHunt product was an app that would let you map where you cried. Absolutely nutty. I miss peak product hunt :(

ProductHunt was simply a way for bootcamp graduates to express themselves during the frontend JS hype cycle of 2014-2018

Re: Are Product Hunt's featured products still online today?

#50

Post by BlewisJS »

whakim wrote: post #41
BlewisJS wrote: post #16

Unrelated to the article - is it just me or is this scrapingbee product borderline nefarious? From the homepage: > Thanks to our large proxy pool, you can bypass rate limiting website, lower the chance to get blocked and hide your bots! > Scrapingbee helps us to retrieve information from sites that use very sophisticated mechanism to block unwanted traffic, we were struggling with those sites for some time now and I'…

It really depends. There are plenty of legitimate uses for scraping (for example, I've been involved with academic research that involved scraping Twitter search results), and it's only really feasible to collect the amount of data you need using scraping plus paid proxies. That being said, there are also a number of nefarious paid proxy services which offer residential IPs (read: are usually botnets).

What is legitimate to a user is not the same as what is legitimate to a site owner. The legitimate way would probably be to use the Twitter API.
Post reply on HN