Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I scraped 25M Shopify products to build a search engine

searchagora.com

Post reply on HN
Print view View original

211–220 of 286 posts

Re: Show HN: I scraped 25M Shopify products to build a search engine

#211

Post by adentranter »

tehlike wrote: post #124
pencildiver wrote: post #80

Earlier quoted context omitted.

Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…

Disclaimer: I am building https://pricetracker.wtf You may want to look at Hetzner, and cut your costs by about 90%. Feel free to reach me, email in profile.

hey! this is cool, I take it you are based in the US?

How long have you been working on this?

Re: Show HN: I scraped 25M Shopify products to build a search engine

#212

Post by pencildiver »

senecaso wrote: post #130

I hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we m…

Thanks for sharing this! If you're up for it, I'd love to talk more about your experience, especially the technical tooling. Working as fast as I can to understand the right way to approach the tech, as there are tradeoffs with performance and price. I'm at support @ searchagora .com

Re: Show HN: I scraped 25M Shopify products to build a search engine

#213

Post by pencildiver »

sanketgoyal11 wrote: post #210

How did you find the list of shopify stores and names?

Found a list on a site called "Built With". Mentioned this in another comment but I think it's meant for building sales outreach email lists.

Once you have the store URL, you can get general store information at [Base URL]/meta.json

Here's an example: https://wildfox.com/meta.json

Re: Show HN: I scraped 25M Shopify products to build a search engine

#214

Post by tehlike »

adentranter wrote: post #211
tehlike wrote: post #124

Earlier quoted context omitted.

Disclaimer: I am building https://pricetracker.wtf You may want to look at Hetzner, and cut your costs by about 90%. Feel free to reach me, email in profile.

hey! this is cool, I take it you are based in the US? How long have you been working on this?

On and off for a year, with more time allocated since June. Yes I am in California.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#215

Post by timthelion »

wingerlang wrote: post #206
DeathArrow wrote: post #168

Earlier quoted context omitted.

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

> In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Anecdotally, I guess, I'd say extremely popular. I never search for products anywhere else.

Yeah, here in Czechia I always look at https://www.heureka.cz/ first.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#216

Post by RagnarD »

jesterson wrote: post #175
RagnarD wrote: post #171

Is this really within the TOS of Shopify?

Does it matter? Or you can't do anything not explicitly allowed by law? Shopify is the company spotted in so many shenanigans, so anything that undermines it's business I would personally welcome very well.

It matters when you get sued.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#217

Post by timthelion »

SushiHippie wrote: post #205
pbmonster wrote: post #191

Earlier quoted context omitted.

Amazon has no direct presence in Switzerland, but you can order a fraction of its products from neighboring countries. Many products are not available, mainly because nobody wants to deal with customs once the product crosses the EU boarder. Amazon itself never moved into Switzerland in the first place for many reasons (small market, unusual customs situation, relatively high salary for warehouse workers), and in the…

If you wouldn't have said that it's basically the Amazon in Switzerland, I'd have thought that this is some blogspam dropshipping site...

Amazon is a blogspam drop shipping site in Europe

Re: Show HN: I scraped 25M Shopify products to build a search engine

#219

Post by dangoodmanUT »

pencildiver wrote: post #80
screye wrote: post #57

What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…

Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…

Yo fuck mongo just use RDS or some digitalocean DB. Or really just use opensearch/elasticsearch, or even typesense (don't bother with raft it's so broken) or meilisearch

Re: Show HN: I scraped 25M Shopify products to build a search engine

#220

Post by dangoodmanUT »

hackideiomat wrote: post #187
pitched wrote: post #118

Shopify has tried a few times to build a tool like this but hasn’t ever managed to get any traction. I think that missing any curation at all could be what eventually kills it. Their current attempt is https://shop.app and a query for red shoes is mostly red shoes.

> a query for red shoes is mostly red shoes well I get mostly black shoes lol Edit: ah no, they just use half a page for shoe shops first with black shoes as logo??

ads baby
Post reply on HN