What was the process for scraping 25M products ?
I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale.
Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ?
______________
A recommendation for how to improve search.
Your base captions will be pretty bad. You can use spot instances on a smaller GPU machine to run a dense captioning model (https://portal.vision.cognitive.azure.com/demo/dense-caption...) and generate captions for all your images.
Then for search, a simple vector store index would be a great retrieval solution here. It is better to do search using those as well.
Both are pretty cheap and can be done reliably within 20-30 lines of code each in python. 3rd party tools for these are pretty stable.