Live data from Hacker News

Amazon2csv: Amazon products scraper to CSV (no API token required)

github.com

21–30 of 60 posts

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#21

Scraping Amazon is fun and all, but when you start overdoing it they rate-limit your IP and show you my worst nightmare: the Dogs of Amazon (a 500 page with pictures) Why do I know this? Because I'm the CTO at Nazdeeq.com where we let users buy Amazon products from countries where they don't ship easily, like Pakistan. Edit: totally open to partnerships in more countries

I'm from Brazil and what you said made me curious, not sure why, but Amazon here didn't catch. How did you solve problems like logistics and interest from the public?

I'm sorry, I have trouble understanding your question but if you mean how we ship from Amazon to Pakistan, and how we got people to use our service: we worked out a pipeline to get products from the US to Amazon, and advertising + word-of-mouth. Also:

+ There's no direct way to buy 90% of products from Amazon since they don't ship to Pakistan

+ Our service is the only in the country that gives a fixed price at checkout in PKR

+ Our customer service is excellent

+ We're one of the cheapest options available, as long as the competition imports products legally.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#22
post #20
post #8

Earlier quoted context omitted.

why aren't Larry and Sergey behind bars, then? Scraping publicly available information is far from illegal. Also, Interestingly only Alibaba's bots are completely blocked from crawling: https://www.amazon.com/robots.txt

Check amazon api T&C, also try to do the same with Craigslist and see how long you they will let you do it. scraping data is always a shady business if you do it without a permission of content owner

If you’re scraping.. you’re not using the API.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#24
post #20
post #8

Earlier quoted context omitted.

why aren't Larry and Sergey behind bars, then? Scraping publicly available information is far from illegal. Also, Interestingly only Alibaba's bots are completely blocked from crawling: https://www.amazon.com/robots.txt

Check amazon api T&C, also try to do the same with Craigslist and see how long you they will let you do it. scraping data is always a shady business if you do it without a permission of content owner

It is anything but shady. They can send you a C&D or file a suit and seek injunction but there is no way they can get you in trouble with the law for scraping publicly available data.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#25

Earlier quoted context omitted.

I just took a glance, but nothing seemed too off. Do you care to elaborate?

Sure. I'm not really trying to criticize the code, it's just that a lot of this looks foreign and unconventional to me. 1. requests.Session() is a class. IDK what request.session() invokes (see https://github.com/tducret/amazon-scraper-python/blob/master... ). 2. Isn't one of the points of using Session() that it'll persist stuff like cookies and headers? So why is it re-defining the headers multiple times? (e.g. bot…

1. It just does return Session(), that's easy enough to find out[1].

2. It doesn't really matter and maybe it's so they are kept closer since they are modified, Session merges your call provided and its own headers (yours take precedence) and it still handles the cookies if you provide own headers. Session also has the benefit of connection pooling so it's quicker to do more than one request with it[2] (normal get, post, etc. in requests module go through request function in the end which actually constructs a Session for that single request).

3. What's wrong there? It's just a default argument. Strings aren't mutable so it avoids this pitfall[4]. Is the " quote a problem here? It's a matter of taste/style. PEP8 is silent on it[3] and just say to pick a convention and there seems to be one here. Some people (me too) also use single quotes for non-human readable strings and double quotes for human readable strings.

4. If you mean here[5] then there is a len just above it to catch the 'expected' error/missing element, just the .text part is unchecked. As for the general lack of checks - I don't put them into my GreaseMonkey or random Python one off scraping code either. Site layout is invariant of a certain version of a scraper script so if some field is missing or something like that then the website layout must have changed and the entire script probably needs to be reworked (or the field is not always present there in the first place so the script is also useless in that particular scraping case) and might as well crash (or if its used by someone they can catch the exception). Either way (crash or catch) when something you expected to surely be there is missing the results are not coming or might be wrong. That code as it is now anticipates that there might not be such an element but if there is it must have the expected field. If the site has been observed to always work like that (certain element might be missing but surely has that field when its there) then script just works like that and guards against the first expected possibility (missing element) but not the second (missing field) since if how site is laid out, how data is stored in elements, etc. changed significantly, then the script also needs changes or risks producing bad or incomplete output (arguably worse as a default than a loud failure would be, it also depends on what you're doing and what the scrape is for).

I'd assume most users and programmers would rather get an error than have script return an empty list (despite there being content up there) just because the layout changed. The only other solution (other than return a wrong result by design and hide the errors or log them somewhere where no one cares to read anyway) would be to catch such exceptions somewhere high and either pack them into a new exception that is thrown with more information (what URL, what element content was exactly, entire response text, etc.) but that's probably too much care/work for such a one off script OR throw your entirely own ones from some low place, but it's vanity then because Python exceptions point really strongly to where they were thrown and in what call stack so it's just as clear what was broken without the need to add lots of checks yourself and throw a "element X is missing field Y that should always be there" message.

[1] - https://github.com/requests/requests/blob/master/requests/se...

[2] - http://docs.python-requests.org/en/master/user/advanced/#ses...

[3] - https://www.python.org/dev/peps/pep-0008/#string-quotes

[4] - http://www.effbot.org/zone/default-values.htm

[5] - https://github.com/tducret/amazon-scraper-python/blob/master...

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#26

Scraping Amazon is fun and all, but when you start overdoing it they rate-limit your IP and show you my worst nightmare: the Dogs of Amazon (a 500 page with pictures) Why do I know this? Because I'm the CTO at Nazdeeq.com where we let users buy Amazon products from countries where they don't ship easily, like Pakistan. Edit: totally open to partnerships in more countries

Hi Amin, your platform seems nice. Just wanted to give you a heads-up that your website is being classified as ["phishing" by Avast](https://i.imgur.com/SmuuRfD.png). I think if you replace "Amazon" in the url with something else it should work fine. Best of luck!

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#27
post #8
post #6

It is also illegal to scrape AZ, since if you scrape it , it means you don’t own this content and you are just stilling products data added to the site by produsts proper owners.

why aren't Larry and Sergey behind bars, then? Scraping publicly available information is far from illegal. Also, Interestingly only Alibaba's bots are completely blocked from crawling: https://www.amazon.com/robots.txt

> Scraping publicly available information is far from illegal.

The scraping itself may not be (although I'm pretty sure here in Belgium there is a law against collecting other people's data), but what you do with it may not be legal.

You could make a case for making any kind of profit generated from scraping data illegal. Don't get me wrong, I love scraping things myself.

Also find it amazing there are companies out there like Crawlera that can do serious scraping work and openly flaunt deploying tech to get around whatever scraping blockers are out there.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#28

Scraping Amazon is fun and all, but when you start overdoing it they rate-limit your IP and show you my worst nightmare: the Dogs of Amazon (a 500 page with pictures) Why do I know this? Because I'm the CTO at Nazdeeq.com where we let users buy Amazon products from countries where they don't ship easily, like Pakistan. Edit: totally open to partnerships in more countries

Are you using the API or web scraping? We never really had problems with IP banning if the traffic looks like a real user.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#29

Earlier quoted context omitted.

I'm from Brazil and what you said made me curious, not sure why, but Amazon here didn't catch. How did you solve problems like logistics and interest from the public?

I'm sorry, I have trouble understanding your question but if you mean how we ship from Amazon to Pakistan, and how we got people to use our service: we worked out a pipeline to get products from the US to Amazon, and advertising + word-of-mouth. Also: + There's no direct way to buy 90% of products from Amazon since they don't ship to Pakistan + Our service is the only in the country that gives a fixed price at checko…

I disagree with the last point. Services like MyUS, Viabox etc. are cheaper than Nazdeeq in many cases.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#30

Why not use the API? Disclaimer: I'm the author of python-amazon-simple-product-api [1] [1] https://github.com/yoavaviram/python-amazon-simple-product-a...

Are you referring to the Product Advertising API?

Doesnt that require you to have a quota of affiliate sales to keep using it? I can’t find where they state this requirement but I remembered they were very sneaky about disclosing this. If you dont have any affiliate sales after X months, your API key will stop working.

Post reply on HN