[flagged]
OP said in another comment:
> ... narrowed down the focus to stores to US only and between $100k - $1m in revenue to keep the initial data set manageable (and the CPU / Storage costs reasonable).
Show HN: I scraped 25M Shopify products to build a search engine
91–100 of 286 posts
Re: Show HN: I scraped 25M Shopify products to build a search engine
#92Earlier quoted context omitted.
https://www.shopify.com/robots.txt lists a lot of sitemap files, which tend to be a good starting point.
It seems sort of questionable to use the list of things to not scrape as a starting point for scraping.... I mean, I get it's not actually enforced.
[flagged]
Re: Show HN: I scraped 25M Shopify products to build a search engine
#93Searching is slow (kinda expected that right now), but after clicking a product and then hitting back, I have to wait for the search again.
Not at computer so I didn't check the headers, but maybe allow the client to cache the response for a short time so it doesn't need to load search results again.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#94This is great - just a couple UI things bugging me. 1. When clicking "Open" on a product, the user should be able to open that in a separate tab. Currently that's not possible; I'm sure because it's being delivered in a single page (can't check now because you're getting hugged to death by HN). 2. When the server's slow, as it just was, there should be some kind of waiter / loader to immediately show the user that th…
Thanks for sharing this! Definitely wasn't expecting this level of traffic so didn't account for some front-end loading experiences. Implementing these now.
For 3, thinking to let the back button work the same as the "x", that way a user can return to where they are in a search result regardless of what they click on.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#95That's funny, I made a domain-specific version of this for canadian coffee deals. https://beangrid.mcconomy.org/
Which coffee seems to hit the best in Canada (your take). I find the espresso in Canada hasn't been as good as the coffee brands in the US but I'm open to possibilities.
Also like the project!
Re: Show HN: I scraped 25M Shopify products to build a search engine
#96Built the same thing a while back while collecting a lead list for sales. Not bothered to keep data updated but was a fun thing to build in a couple days. (disclaimer mobile experience is meh cause it was a fun project) https://zensear.ch How did you find list of all Shopify stores? I ended up just checking every .com, .net, etc as I didn't find an easy way to figure it out directly from shopify.
In another comment, OP writes: > Bought an initial list of 2m stores for a few hundred dollars from a website called "Built With". Think they are used for building sales outreach lists. Then narrowed down the focus to stores to US only and between $100k - $1m in revenue to keep the initial data set manageable (and the CPU / Storage costs reasonable).
[deleted]
Re: Show HN: I scraped 25M Shopify products to build a search engine
#97cool project. You might have notice, but there's a non-trivial amount of fraud on shopify (fake shops, info stealers, etc). Might be interesting to look at that dataset and explore a bit =) it's quite fascinating
What are the telltales, for spotting those?
And how aggressively does Shopify verify/police them?
Re: Show HN: I scraped 25M Shopify products to build a search engine
#98What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…
Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…
I'm biased, but I'd recommend exploring Typesense for search.
It's an open source alternative to Algolia + Pinecone, optimized for speed (since it's in-memory) and an out-of-the-box dev experience. E-commerce is also a very common use-case I see among our users.
Here's a live demo with 32M songs: https://songs-search.typesense.org/
Disclaimer: I work on Typesense.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#99How are you planning to monetize this? You mentioned you are spending around $2K just to run it. Is there a commission strategy or ads? Or populate with your products at one point so you sell your own thing?
Re: Show HN: I scraped 25M Shopify products to build a search engine
#100What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…
Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…
> I'm currently not storing the image files, so that reduces the cost as well.
I wonder if someone catches on and replaces all your image URLs to the fuzzy testicle egg cup[0], will that negatively impact reputation?
