71–80 of 286 posts
What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…
Some observations:
- Don't use infinite scrolling, it's an outdated UI practice that leads to bad user experience. It also makes the footer entirely unviewable.
- Clicking on a product card image does not reliably open up the product. I have to randomly click on it a few times (Chrome, Brave)
- Clicking on product card image and title leads to different actions, this is a bit unexpected, should show some hint of the difference.
- The product page pop up will reset the search list when closed, this messes up my search navigation, breaks the flow of browsing.
Cool project but Shopify already has this. https://shop.app https://shop.app/search/results?query=red%20shoes
How did you find list of all Shopify stores? I ended up just checking every .com, .net, etc as I didn't find an easy way to figure it out directly from shopify.
Great project. If you continue to crawl the data, be sure to save it so you can detect price changes a la camelcamelcamel.
For all of Amazon's faults, the fact that they tolerate CCC does drive a lot of my online purchases there. CCC used to track other sites, and was eventually blocked on all of them. If more sites want my business, showing their pricing history (either from internal data, or by letting someone build the DB) would go a long way.
What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…
For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json.
Built a crawler in simple Javascript to run through a list that I bought on a site called "Built With", access their JSON file with the product listing data, and scrape the exact data we want for Agora. Then storing it in Mongo and, currently, using Mongo Atlas Search (i.e. saw they released Vector Search but haven't looked at it). It has been a process of trial and error to pick the right data fields that are required for the front-end experience but not wanting to increase the size of the data set drastically. And after initially using React, switched to NextJS to make it easier to structure URLs of each product listing page.
Mongo will run me about $1,500 / month at the current CPU level. AWS all in will be about $700. I'm currently not storing the image files, so that reduces the cost as well.
A few improvement that has helped so far:
- Having 2 separate Search Indexes, one for the 'brand' and on for the 'product'. There's a second public JSON file that is available on all Shopify stores with relevant store data at [Base URL]/meta.json For example: https://wildfox.com/meta.json
- Removing the "tags" that are provided by store owners on Shopify. I believe these are placed for SEO reasons. These were 1 - 50 words / product so removing these reduced the data size we're dealing with. The tradeoff is that they can't be used to improve the search experience now.
Hope this helps. Still wrapping my head around all of this.