Show HN: I scraped 25M Shopify products to build a search engine

Post by KomoD » Thu, Dec 14, 2023, 7:29 PM UTC

pencildiver wrote: post #80

screye wrote: post #57

What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…

Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…

Post by callmeed » Thu, Dec 14, 2023, 8:56 PM UTC

senecaso wrote: post #258

callmeed wrote: post #167

I built this a couple years ago (now defunct) for the same reason :) The public JSON endpoints on shopify stores make it pretty easy to get the data. You mentioned using Mongo but it sounds expensive. I honestly think you could do this with just elastic or even postgres full text search and save money. Here's a pro tip + feature you should implement: Shopify has a semi-hidden hack where you can link directly to check…

I didnt know about the link to checkout. That's a slightly nicer user experience for sure. Still, its confusing for users who want to do more shopping at the same time. I had users who clicked on a number of items, clicked "add to cart" in each one (all different shops), and then couldn't figure out how to checkout on the main site afterwards! Obviously people were looking for a more complete one-stop-shopping experi…

Post by ljm » Thu, Dec 14, 2023, 10:23 PM UTC

altdataseller wrote: post #154

ljm wrote: post #84

Earlier quoted context omitted.

2.2k/mo right off the bat is pretty steep, especially if you're paying that while the search response reliably takes over 10 seconds. Why would you shovel 1.5k into MongoDB's pockets right off the bat? Especially when ElasticSearch is much better suited to what you're trying to do?

Sounds like someone drank the Mongo kool-aid. You absolutely do not need Mongo, let alone Mongo Atlas. 25 million documents with ecommeece products is measly and should fit in a single 600 GB server

Post by mfrye0 » Thu, Dec 14, 2023, 11:01 PM UTC

neeleshs wrote: post #164

mfrye0 wrote: post #115

Earlier quoted context omitted.

If you're already on AWS, I recommend switching to postgres for now. For context, I have 3 RDS instances, each multi zone, with the biggest instance storing several billion records. My total bill for all 3 last month was $661. Postgres has full text search, vector search, and jsonb. With jsonb you can store and index json documents like you would in Mongo. - https://www.postgresql.org/docs/current/textsearch.html - h…

how big is the disk for the biggest instance?

Post by berkes » Thu, Dec 14, 2023, 11:19 PM UTC

stef25 wrote: post #225

berkes wrote: post #199

Earlier quoted context omitted.

> site called "Built With", Do you have Alink. And are they any good?

Google ? https://builtwith.com/