Is this really within the TOS of Shopify?
Show HN: I scraped 25M Shopify products to build a search engine
171–180 of 286 posts
Re: Show HN: I scraped 25M Shopify products to build a search engine
#172What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…
Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…
>Mongo will run me about $1,500 / month at the current CPU level. AWS all in will be about $700. I'm currently not storing the image files, so that reduces the cost as well.
It will probably cost you just $100 to rent a server from Hetzner and do the same thing. I would also use Redis or another kind of cache to hit the DB less.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#173Why not manticore as backend? Much better perf than ES, less memory intense, sql syntax. Just fantastic all round!!
Re: Show HN: I scraped 25M Shopify products to build a search engine
#174What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…
>I have always used standard python tools like selenium, bs4 and the like
There's nothing to scrap. You just download a JSON, the site owners kindly put on your disposal.
Scraping is a more complex process, where you have to work around rate limiting and captchas. For the tool I built I wrote tens of thousands of lines of code and I still find daily issues I have to deal with if I want to scrap a particular web page, issues I don't always have the time to solve.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#175Is this really within the TOS of Shopify?
Does it matter? Or you can't do anything not explicitly allowed by law?
Shopify is the company spotted in so many shenanigans, so anything that undermines it's business I would personally welcome very well.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#176Earlier quoted context omitted.
We had no capital, so advertising or solutions that basically involved "throwing money at the problem" were off the table for us. We spent time posting in forums helping people find items they were looking for, and we had a few posts here on HN that generated short-lived, explosive traffic bursts. I remember those days we had posts get picked up on HN, it was always an exciting night! We were looking at influencers a…
looks like simptoms of no market. maybe you were solving a problem already solved by amazon ? most shops on shopify also use amazon
>looks like simptoms of no market. maybe you were solving a problem already solved by amazon ? most shops on shopify also use amazon
FAANGS get around this by creating problems that they will offer to solve.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#177Earlier quoted context omitted.
looks like simptoms of no market. maybe you were solving a problem already solved by amazon ? most shops on shopify also use amazon
Many shops do double list, this is true. However, I don't think its a solved problem. There are many people who do not want to shop on Amazon for their own reasons. There are also people who want to shop locally, and Amazon provides no mechanism to do so (that I'm aware of). There are also many smaller shops who simply cannot afford to list on Amazon, as there are considerable fees associated with running a successfu…
To me it seems like a small market. And worse, it's hard to conquer that small market since it's very fragmented. Even if you had money for advertisements, it still would have been hard.
On the plus side, though, if you had the skills to build that platform, you certainly have the skill to build a more profitable and easier to monetize platform.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#178What technology did you use to build the scraper and how did you get around the usual challenges (anti bot, ip banning, etc) with scraping large amounts of data?
Scraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end. Here's an example: https://www.wildfox.com/products.json
What’s the trade off using js for this? Would it have been much faster to use go or something?
Re: Show HN: I scraped 25M Shopify products to build a search engine
#179The Terms page goes to "Jaggi Enterprises", "A Modern Investment Fund. We buy, build, and invest in software companies with recurring revenue.".
So maybe this is not really something a guy built for his wife, but some anonymous startup that googled "Which terms rank best on Hacker News" and then wrote the "I did ... my wife .." story?
Re: Show HN: I scraped 25M Shopify products to build a search engine
#180On the page where you show details about the product, I would like to have it include the same product from other Shopify stores by doing an image similarly comparison.
And then highlight how the price compares.
For example, here are some pretty crazy red shoes. But they are too expensive for me. Would be interesting to see if this is the only store selling these shoes, or if someone else has the same shoes much cheaper.
https://www.searchagora.com/products/vasco-4-47fb0f87-5b89-4...