Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon2csv: Amazon products scraper to CSV (no API token required)

github.com

Post reply on HN
Print view View original

1–10 of 60 posts

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#7

Post by staticautomatic »

dec0dedab0de wrote: post #5
staticautomatic wrote: post #2

Am I the only one who thinks this is rather weird, or at least unconventional code for a scraper in Python?

I just took a glance, but nothing seemed too off. Do you care to elaborate?

Sure. I'm not really trying to criticize the code, it's just that a lot of this looks foreign and unconventional to me.

1. requests.Session() is a class. IDK what request.session() invokes (see https://github.com/tducret/amazon-scraper-python/blob/master...).

2. Isn't one of the points of using Session() that it'll persist stuff like cookies and headers? So why is it re-defining the headers multiple times? (e.g. both GET and POST in the same session have their own respective but identical headers).

3. Is the use of `arg=""` idiomatic? For example in https://github.com/tducret/amazon-scraper-python/blob/master...

4. Using raw list indices without some kind of helper function to catch index and other errors when parsing is not really a good idea in scraping (e.g. `selection[0].text.strip()`.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#8

Post by zeusk »

kull wrote: post #6

It is also illegal to scrape AZ, since if you scrape it , it means you don’t own this content and you are just stilling products data added to the site by produsts proper owners.

why aren't Larry and Sergey behind bars, then? Scraping publicly available information is far from illegal.

Also, Interestingly only Alibaba's bots are completely blocked from crawling: https://www.amazon.com/robots.txt

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#9

Post by mygo »

Jdam wrote: post #4

The issue with those tools is that Amazon changes the product layout very often and heavily conducts A/B tests. I’ve once even heard that computer vision is the most stable way to scrape Amazon. I guess this library will stop working rather soon.

> I guess this library will stop working rather soon.

Don’t really see that as a dealbreaker. So the library will need maintenance. Normal for libraries to need updates in order to keep up with changes. It works today, and it will work whenever it’s updated. Better than nothing and for many use cases that’s good enough.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#10

Post by smt88 »

kull wrote: post #6

It is also illegal to scrape AZ, since if you scrape it , it means you don’t own this content and you are just stilling products data added to the site by produsts proper owners.

Why would the owner of a product want to keep their product info a secret?
Post reply on HN