Nice. From my experience I've found Parsel [1] (used by scrapy) to be an easier to use HTML parsing library than Beautiful Soup. That's just imo.
Amazon2csv: Amazon products scraper to CSV (no API token required)
51–60 of 60 posts
Re: Amazon2csv: Amazon products scraper to CSV (no API token required)
#52Earlier quoted context omitted.
why aren't Larry and Sergey behind bars, then? Scraping publicly available information is far from illegal. Also, Interestingly only Alibaba's bots are completely blocked from crawling: https://www.amazon.com/robots.txt
> Scraping publicly available information is far from illegal. The scraping itself may not be (although I'm pretty sure here in Belgium there is a law against collecting other people's data), but what you do with it may not be legal. You could make a case for making any kind of profit generated from scraping data illegal. Don't get me wrong, I love scraping things myself. Also find it amazing there are companies out…
>
LinkedIn had multiple layers of scraping detection systems deployed, and went to significant efforts to block their data from getting scraped[1].
Last year, they were ordered by a Federal court explicitly to allow scraping of content and remove systems that were designed to impede and block scraping efforts[2].
There's no clear law (in the US) directly aimed at scraping, and repurposing anti-hacking laws brought up the murky definition of what is unauthorized access. If a judge clarifies explicitly that scraping is not unauthorized access (which happened in this case, although needs to stand up to appeal[3]), then entities that are interested in preventing scraping have lost one of their core legal underpinnings. It demonstrates why companies like Crawlera have been able to flaunt the type of serious scraping work they do, because it's hard to bully people with a legal argument that has been debunked and affirmed as debunked on appeal. So it's better to avoid the risk of setting that precedence entirely until you can't avoid it.
[1] https://techcrunch.com/2016/08/15/linkedin-sues-scrapers/
[2] https://www.reuters.com/article/us-microsoft-linkedin-ruling...
[3] https://www.courthousenews.com/linkedin-takes-data-scraping-...
Re: Amazon2csv: Amazon products scraper to CSV (no API token required)
#53Earlier quoted context omitted.
Are you referring to the Product Advertising API? Doesnt that require you to have a quota of affiliate sales to keep using it? I can’t find where they state this requirement but I remembered they were very sneaky about disclosing this. If you dont have any affiliate sales after X months, your API key will stop working.
Currently you have to be a member of their affiliate program to get API access. To become a full "member" you have to be a prospect who generates three referral sales (iirc) within a 30 day period. So once in you have the API, but getting in isn't as easy as filling out a form. From there you can get your API rate limits increased from the default 1x call per second up to 10 based on your prior 30 day affiliate sales…
Last I looked that limit was per IP address.
Re: Amazon2csv: Amazon products scraper to CSV (no API token required)
#54The company I work for (zinc.io) has this: https://zincapi.com/
We brand it as an ordering API, but we also offer retrieving the product data (item details/pricing.) We put a LOT of engineering resources into data quality and maintenance, as the API is core to our flagship product, PriceYak. If you have questions or want a token, email adam@zinc.io and mention this post.
Re: Amazon2csv: Amazon products scraper to CSV (no API token required)
#55Why not use the API? Disclaimer: I'm the author of python-amazon-simple-product-api [1] [1] https://github.com/yoavaviram/python-amazon-simple-product-a...
The API comes with a TOS that severely restricts what you can do with the data.
Re: Amazon2csv: Amazon products scraper to CSV (no API token required)
#56Scraping Amazon is fun and all, but when you start overdoing it they rate-limit your IP and show you my worst nightmare: the Dogs of Amazon (a 500 page with pictures) Why do I know this? Because I'm the CTO at Nazdeeq.com where we let users buy Amazon products from countries where they don't ship easily, like Pakistan. Edit: totally open to partnerships in more countries
In the Philippines there's something quite similar called Galleon. They've been recently acquired but I think they might be open to partner. They've expanded to Thailand, if I'm not mistaken.
Re: Amazon2csv: Amazon products scraper to CSV (no API token required)
#57Why not use the API? Disclaimer: I'm the author of python-amazon-simple-product-api [1] [1] https://github.com/yoavaviram/python-amazon-simple-product-a...
Hi @yoaviram
That was the initial plan. Your project seems to be great, but I was not allowed to become an Amazon affiliate... So it became impossible for me to use your library :(
(I told the full story on my (French) blog (Amazon email snapshot included) > https://www.tducret.com/scraping/2018/06/05/amazon2csv-ou-co...)
By the way, thank you very much to the HackerNews community. Your support was GREATLY appreciated. I'd love to share other projects with you (maybe linkedin2csv, netflix2csv...). Any other ideas?
Re: Amazon2csv: Amazon products scraper to CSV (no API token required)
#58Earlier quoted context omitted.
Ex. People take products data and copy to eBay then try to dropship getting products from your fba. People pay big money for nice photos of products and then somebody just comes and takes it as their own
> People pay big money for nice photos of products and then somebody just comes and takes it as their own This raises an interesting question: if someone had a product on Amazon and had product photos they took, does Amazon still allow other sellers to use the same listing? In other words, does the seller agreement allow Amazon to reuse your (potentially expensive to produce) product photos on your competitor's produ…
Yes they allow it. It often leads to "product listing hijacking": another seller sends a counterfeit/similar product to amazon using the UPC of your (private label) product. Product & review pages are then shared between all these different products. AFAIK the only way to combat this is purchasing the other product, document how if differs from your product, report it all to amazon and hope they act before too much damage is done to your product listing.
Re: Amazon2csv: Amazon products scraper to CSV (no API token required)
#59Earlier quoted context omitted.
I disagree with the last point. Services like MyUS, Viabox etc. are cheaper than Nazdeeq in many cases.
Hey Irfan! I'm sorry you feel that way, but as someone else here pointed out, we handle everything from clearing customs to last-mile delivery (to your doorstep), which is why our pricing may seem expensive because we declare all costs upfront. A lot of our customers have similar horror stories where their goods get stuck in customs because they didn't realize clearing is a thing.
Ok I had my first horror story with custom clearance today. Going to give nazdeeq.com a try next time :-)
Re: Amazon2csv: Amazon products scraper to CSV (no API token required)
#60Earlier quoted context omitted.
Hey Irfan! I'm sorry you feel that way, but as someone else here pointed out, we handle everything from clearing customs to last-mile delivery (to your doorstep), which is why our pricing may seem expensive because we declare all costs upfront. A lot of our customers have similar horror stories where their goods get stuck in customs because they didn't realize clearing is a thing.
Ok I had my first horror story with custom clearance today. Going to give nazdeeq.com a try next time :-)
Haha, we'd love to have you! Throw my name at the support channel, and I should appear to help you out if you have any trouble :)