Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon2csv: Amazon products scraper to CSV (no API token required)

github.com

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#41

Post by always_good »

yasoob wrote: post #26
amingilani wrote: post #17

Scraping Amazon is fun and all, but when you start overdoing it they rate-limit your IP and show you my worst nightmare: the Dogs of Amazon (a 500 page with pictures) Why do I know this? Because I'm the CTO at Nazdeeq.com where we let users buy Amazon products from countries where they don't ship easily, like Pakistan. Edit: totally open to partnerships in more countries

Hi Amin, your platform seems nice. Just wanted to give you a heads-up that your website is being classified as ["phishing" by Avast]( https://i.imgur.com/SmuuRfD.png ). I think if you replace "Amazon" in the url with something else it should work fine. Best of luck!

Reminds me of how nobody could see one of my user's avatars because the url (a hash) had started with an "ad" segment (for bucketing), as in "/avatars/ad/ad3adb33f". So adblockers blocked it.

My protest against such a ridiculous heuristic was to not fix it.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#42

Post by amingilani »

irfan wrote: post #29
amingilani wrote: post #21

Earlier quoted context omitted.

I'm sorry, I have trouble understanding your question but if you mean how we ship from Amazon to Pakistan, and how we got people to use our service: we worked out a pipeline to get products from the US to Amazon, and advertising + word-of-mouth. Also: + There's no direct way to buy 90% of products from Amazon since they don't ship to Pakistan + Our service is the only in the country that gives a fixed price at checko…

I disagree with the last point. Services like MyUS, Viabox etc. are cheaper than Nazdeeq in many cases.

Hey Irfan! I'm sorry you feel that way, but as someone else here pointed out, we handle everything from clearing customs to last-mile delivery (to your doorstep), which is why our pricing may seem expensive because we declare all costs upfront.

A lot of our customers have similar horror stories where their goods get stuck in customs because they didn't realize clearing is a thing.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#43

Post by amingilani »

chx wrote: post #39
irfan wrote: post #29

Earlier quoted context omitted.

I disagree with the last point. Services like MyUS, Viabox etc. are cheaper than Nazdeeq in many cases.

I do not know Nazdeeq but parent claims they deal with importing and last I checked neither MyUS nor any other random package forwarder deal with customs which can be a real PITA. (I have some experience with importing things into ... legally interesting ... countries, from 1990 till I left in 2006 I, as an individual have imported a lot of computers and parts into Hungary. It was ... fun.)

You covered it perfectly! :)

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#44

Post by amingilani »

yasoob wrote: post #26
amingilani wrote: post #17

Scraping Amazon is fun and all, but when you start overdoing it they rate-limit your IP and show you my worst nightmare: the Dogs of Amazon (a 500 page with pictures) Why do I know this? Because I'm the CTO at Nazdeeq.com where we let users buy Amazon products from countries where they don't ship easily, like Pakistan. Edit: totally open to partnerships in more countries

Hi Amin, your platform seems nice. Just wanted to give you a heads-up that your website is being classified as ["phishing" by Avast]( https://i.imgur.com/SmuuRfD.png ). I think if you replace "Amazon" in the url with something else it should work fine. Best of luck!

Thank you Yasoob! Dammit, again? I already had them white-label our site once but I'll look into this again. Thank you!

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#45

Post by amingilani »

dewey wrote: post #28
amingilani wrote: post #17

Scraping Amazon is fun and all, but when you start overdoing it they rate-limit your IP and show you my worst nightmare: the Dogs of Amazon (a 500 page with pictures) Why do I know this? Because I'm the CTO at Nazdeeq.com where we let users buy Amazon products from countries where they don't ship easily, like Pakistan. Edit: totally open to partnerships in more countries

Are you using the API or web scraping? We never really had problems with IP banning if the traffic looks like a real user.

Neither, actually, we're using a heavily configured reverse-proxy.

This means that, unfortunately, all the traffic has to go through our own servers.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#47

Post by amingilani »

always_good wrote: post #41
yasoob wrote: post #26

Earlier quoted context omitted.

Hi Amin, your platform seems nice. Just wanted to give you a heads-up that your website is being classified as ["phishing" by Avast]( https://i.imgur.com/SmuuRfD.png ). I think if you replace "Amazon" in the url with something else it should work fine. Best of luck!

Reminds me of how nobody could see one of my user's avatars because the url (a hash) had started with an "ad" segment (for bucketing), as in "/avatars/ad/ad3adb33f". So adblockers blocked it. My protest against such a ridiculous heuristic was to not fix it.

It makes sense why you would choose to do that and I can certainly empathize, but in the interest of user experience I try to fix these problems because my customers deserve a good user experience.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#49

Post by staticautomatic »

FRex wrote: post #25
staticautomatic wrote: post #7

Earlier quoted context omitted.

Sure. I'm not really trying to criticize the code, it's just that a lot of this looks foreign and unconventional to me. 1. requests.Session() is a class. IDK what request.session() invokes (see https://github.com/tducret/amazon-scraper-python/blob/master... ). 2. Isn't one of the points of using Session() that it'll persist stuff like cookies and headers? So why is it re-defining the headers multiple times? (e.g. bot…

1. It just does return Session(), that's easy enough to find out[1]. 2. It doesn't really matter and maybe it's so they are kept closer since they are modified, Session merges your call provided and its own headers (yours take precedence) and it still handles the cookies if you provide own headers. Session also has the benefit of connection pooling so it's quicker to do more than one request with it[2] (normal get, p…

Thanks for humoring my post with a real answer.

1. I never realized there was a function that just returned an instance of the class. Should've just looked it up myself.

2. I was wrong and misread the header stuff.

3. There's nothing wrong with it. It's just a convention I'm not accustomed to seeing or using. Admittedly, there are lots of ways to skin a cat with optional and default args.

4. Yeah I understand what you're saying. I guess it's a fine "greasemonkey" approach. Just more susceptible to DOM changes and code errors than I'm comfy with even for a rudimentary implementation.

I agree you'd usually want to get an error than an empty list but I think it's a little more complicated than just whether you want an error or an empty list. Sometimes you want the error but don't get it, which is why I tend to write more code around checking stuff and catching exceptions. I think the best example is you might not see an index error but the list item that's returned for your specified index isn't actually what you wanted because the DOM changed or you wrote code against one page that broke on another one you thought would be identical.

Re: Amazon2csv: Amazon products scraper to CSV (no API token required)

#50

Post by xenomachina »

kull wrote: post #19
smt88 wrote: post #10

Earlier quoted context omitted.

Why would the owner of a product want to keep their product info a secret?

Ex. People take products data and copy to eBay then try to dropship getting products from your fba. People pay big money for nice photos of products and then somebody just comes and takes it as their own

> People pay big money for nice photos of products and then somebody just comes and takes it as their own

This raises an interesting question: if someone had a product on Amazon and had product photos they took, does Amazon still allow other sellers to use the same listing? In other words, does the seller agreement allow Amazon to reuse your (potentially expensive to produce) product photos on your competitor's product listing?

Post reply on HN