11–20 of 60 posts
The issue with those tools is that Amazon changes the product layout very often and heavily conducts A/B tests. I’ve once even heard that computer vision is the most stable way to scrape Amazon. I guess this library will stop working rather soon.
At a former employer we scraped Amazon many millions of times per day with very simple old tools that rarely needed updating.
Earlier quoted context omitted.
I just took a glance, but nothing seemed too off. Do you care to elaborate?
Sure. I'm not really trying to criticize the code, it's just that a lot of this looks foreign and unconventional to me. 1. requests.Session() is a class. IDK what request.session() invokes (see https://github.com/tducret/amazon-scraper-python/blob/master... ). 2. Isn't one of the points of using Session() that it'll persist stuff like cookies and headers? So why is it re-defining the headers multiple times? (e.g. bot…
Earlier quoted context omitted.
Sure. I'm not really trying to criticize the code, it's just that a lot of this looks foreign and unconventional to me. 1. requests.Session() is a class. IDK what request.session() invokes (see https://github.com/tducret/amazon-scraper-python/blob/master... ). 2. Isn't one of the points of using Session() that it'll persist stuff like cookies and headers? So why is it re-defining the headers multiple times? (e.g. bot…
Its a good thing its open source, submit a PR!
The issue with those tools is that Amazon changes the product layout very often and heavily conducts A/B tests. I’ve once even heard that computer vision is the most stable way to scrape Amazon. I guess this library will stop working rather soon.
>I’ve once even heard that computer vision is the most stable way to scrape Amazon At a former employer we scraped Amazon many millions of times per day with very simple old tools that rarely needed updating.
Earlier quoted context omitted.
>I’ve once even heard that computer vision is the most stable way to scrape Amazon At a former employer we scraped Amazon many millions of times per day with very simple old tools that rarely needed updating.
Are you able to share some details? How often did you have to get new IP addresses? What about user agent? Were the scapers "straight to the point" like amazon2csv (ie: make a request directly to the search page) or did they have randomized behavior (eg: re-use sessions from time to time; click a random link on the page; start from the homepage...)? Did the scrapers ever went against amz's robots.txt directives (eg:…
For example, Scrapinghub's Crawlera (the guys behind the Scrapy python lib)
Earlier quoted context omitted.
Sure. I'm not really trying to criticize the code, it's just that a lot of this looks foreign and unconventional to me. 1. requests.Session() is a class. IDK what request.session() invokes (see https://github.com/tducret/amazon-scraper-python/blob/master... ). 2. Isn't one of the points of using Session() that it'll persist stuff like cookies and headers? So why is it re-defining the headers multiple times? (e.g. bot…
Its a good thing its open source, submit a PR!
Why do I know this? Because I'm the CTO at Nazdeeq.com where we let users buy Amazon products from countries where they don't ship easily, like Pakistan.
Edit: totally open to partnerships in more countries
Scraping Amazon is fun and all, but when you start overdoing it they rate-limit your IP and show you my worst nightmare: the Dogs of Amazon (a 500 page with pictures) Why do I know this? Because I'm the CTO at Nazdeeq.com where we let users buy Amazon products from countries where they don't ship easily, like Pakistan. Edit: totally open to partnerships in more countries
It is also illegal to scrape AZ, since if you scrape it , it means you don’t own this content and you are just stilling products data added to the site by produsts proper owners.
Why would the owner of a product want to keep their product info a secret?
It is also illegal to scrape AZ, since if you scrape it , it means you don’t own this content and you are just stilling products data added to the site by produsts proper owners.
why aren't Larry and Sergey behind bars, then? Scraping publicly available information is far from illegal. Also, Interestingly only Alibaba's bots are completely blocked from crawling: https://www.amazon.com/robots.txt