Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon2csv: Amazon products scraper to CSV (no API token required)

github.com

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#31

Post by AznHisoka »

RhodesianHunter wrote: post #11
Jdam wrote: post #4

The issue with those tools is that Amazon changes the product layout very often and heavily conducts A/B tests. I’ve once even heard that computer vision is the most stable way to scrape Amazon. I guess this library will stop working rather soon.

>I’ve once even heard that computer vision is the most stable way to scrape Amazon At a former employer we scraped Amazon many millions of times per day with very simple old tools that rarely needed updating.

Same here. Scraping their search results page is easy if you have a bunch of IPs. No manipulation or workarounds needed(ie headless browsers, ensuring your http headers look like a real user).

I have not scraped a ton of actual individual product pages though so cant testify about scraping that. I do remember it might have been harder.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#32

Post by gsich »

kull wrote: post #20
zeusk wrote: post #8

Earlier quoted context omitted.

why aren't Larry and Sergey behind bars, then? Scraping publicly available information is far from illegal. Also, Interestingly only Alibaba's bots are completely blocked from crawling: https://www.amazon.com/robots.txt

Check amazon api T&C, also try to do the same with Craigslist and see how long you they will let you do it. scraping data is always a shady business if you do it without a permission of content owner

I don't need to check the T&C to view the page.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#33

Post by ZoomStop »

AznHisoka wrote: post #30
yoaviram wrote: post #23

Why not use the API? Disclaimer: I'm the author of python-amazon-simple-product-api [1] [1] https://github.com/yoavaviram/python-amazon-simple-product-a...

Are you referring to the Product Advertising API? Doesnt that require you to have a quota of affiliate sales to keep using it? I can’t find where they state this requirement but I remembered they were very sneaky about disclosing this. If you dont have any affiliate sales after X months, your API key will stop working.

Currently you have to be a member of their affiliate program to get API access. To become a full "member" you have to be a prospect who generates three referral sales (iirc) within a 30 day period. So once in you have the API, but getting in isn't as easy as filling out a form. From there you can get your API rate limits increased from the default 1x call per second up to 10 based on your prior 30 day affiliate sales.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#34

Post by k__ »

yoaviram wrote: post #23

Why not use the API? Disclaimer: I'm the author of python-amazon-simple-product-api [1] [1] https://github.com/yoavaviram/python-amazon-simple-product-a...

Sometimes this isn't possible.

I wrote an app that is basically a new UI for the Amazon products. It runs entirely on the client. The Amazon API simply didn't work in that setup.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#35

Post by hobofan »

Jdam wrote: post #4

The issue with those tools is that Amazon changes the product layout very often and heavily conducts A/B tests. I’ve once even heard that computer vision is the most stable way to scrape Amazon. I guess this library will stop working rather soon.

Search results scraping on Amazon is fairly stable.

What's more difficult is product page scraping, because there you have hundreds of different variations. Some from A/B testing and a lot just being specific things that show up for certain product categories (e.g. video games).

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#36

Post by bufferoverflow »

I remember trying to build a scraper for Amazon. I quickly discovered that there are many types of item pages, and they change over time too. A/B testing probably. Just to get the price of the product out of their HTML markup reliably was a nightmare, I had to build a huge tree of if-this-then-maybe-that logic.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#38

Post by raitucarp »

yoaviram wrote: post #23

Why not use the API? Disclaimer: I'm the author of python-amazon-simple-product-api [1] [1] https://github.com/yoavaviram/python-amazon-simple-product-a...

Man, looks great. I also build something similar in node.js. I implement everything what documentation said (complete implementation). ICYMI:

https://github.com/Ribhnux/piranhax

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#39

Post by chx »

irfan wrote: post #29
amingilani wrote: post #21

Earlier quoted context omitted.

I'm sorry, I have trouble understanding your question but if you mean how we ship from Amazon to Pakistan, and how we got people to use our service: we worked out a pipeline to get products from the US to Amazon, and advertising + word-of-mouth. Also: + There's no direct way to buy 90% of products from Amazon since they don't ship to Pakistan + Our service is the only in the country that gives a fixed price at checko…

I disagree with the last point. Services like MyUS, Viabox etc. are cheaper than Nazdeeq in many cases.

I do not know Nazdeeq but parent claims they deal with importing and last I checked neither MyUS nor any other random package forwarder deal with customs which can be a real PITA. (I have some experience with importing things into ... legally interesting ... countries, from 1990 till I left in 2006 I, as an individual have imported a lot of computers and parts into Hungary. It was ... fun.)
Post reply on HN