111–120 of 286 posts
What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…
What's your revenue model? I see you expanded on the details of your $1.5K monyhly cost, but failing to see how you make money? Affiliates fees?
I wanted to first prove that people would actually use this / find value in it. Fortunately a few merchants have reached out already via email to talk through the business model so this will likely evolve as we learn more.
What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…
Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…
Postgres has full text search, vector search, and jsonb. With jsonb you can store and index json documents like you would in Mongo.
- https://www.postgresql.org/docs/current/textsearch.html - https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-rd...
Aside: The ending of the 1948 "The Red Shoes" was funny to me, but I think I was a little loopy after slogging thru it. I don't know if I recommend it or not.
That's funny, I made a domain-specific version of this for canadian coffee deals. https://beangrid.mcconomy.org/
Which coffee seems to hit the best in Canada (your take). I find the espresso in Canada hasn't been as good as the coffee brands in the US but I'm open to possibilities. Also like the project!
This isn't worth the cost or effort. Shopify already has an internal tool with this functionality that they are planning to publicize.
Earlier quoted context omitted.
Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…
2.2k/mo right off the bat is pretty steep, especially if you're paying that while the search response reliably takes over 10 seconds. Why would you shovel 1.5k into MongoDB's pockets right off the bat? Especially when ElasticSearch is much better suited to what you're trying to do?