151–160 of 286 posts
Hey, I have a Shopify store that sells e-paper calendars / smart screens. I tried to search for it but I could not find it. What should I do so your crawler can find me? https://shop.invisible-computers.com
Cool project! As you scale, you may benefit from these two projects I maintain, and the Big Tech uses :) https://github.com/unum-cloud/usearch - for faster search https://github.com/unum-cloud/uform - for cheaper multi-lingual multi-modal embeddings Feel free to reach out with feedback and feature requests!
I hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we m…
What plans did you have for generating revenue from the site? (Serious question - given your low costs it would seem like a tiny amount of revenue would gave been enough.)
Earlier quoted context omitted.
Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…
2.2k/mo right off the bat is pretty steep, especially if you're paying that while the search response reliably takes over 10 seconds. Why would you shovel 1.5k into MongoDB's pockets right off the bat? Especially when ElasticSearch is much better suited to what you're trying to do?
Hey, I have a Shopify store that sells e-paper calendars / smart screens. I tried to search for it but I could not find it. What should I do so your crawler can find me? https://shop.invisible-computers.com
Hi, you could drop an email to onboard@peppyhop.com and we will be happy to onboard you. Please add target geography like you would like to target Indian market or US market
I hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we m…
Im curious why you consider lack of users to be the problem. I would have described it as lack of revenue. What plans did you have for generating revenue from the site? (Serious question - given your low costs it would seem like a tiny amount of revenue would gave been enough.)
Earlier quoted context omitted.
We had no capital, so advertising or solutions that basically involved "throwing money at the problem" were off the table for us. We spent time posting in forums helping people find items they were looking for, and we had a few posts here on HN that generated short-lived, explosive traffic bursts. I remember those days we had posts get picked up on HN, it was always an exciting night! We were looking at influencers a…
looks like simptoms of no market. maybe you were solving a problem already solved by amazon ? most shops on shopify also use amazon
Still, there were problems with our solution that if addressed may have provided a better market fit. If we had had more runway, we would have worked to address them, but that simply wasn't in the cards.
Built the same thing a while back while collecting a lead list for sales. Not bothered to keep data updated but was a fun thing to build in a couple days. (disclaimer mobile experience is meh cause it was a fun project) https://zensear.ch How did you find list of all Shopify stores? I ended up just checking every .com, .net, etc as I didn't find an easy way to figure it out directly from shopify.
Hey, I have a Shopify store that sells e-paper calendars / smart screens. I tried to search for it but I could not find it. What should I do so your crawler can find me? https://shop.invisible-computers.com
https://www.searchagora.com/products/invisible-calendar-6266...
Thinking that we should have a page where store owners can submit their URL to be crawled.
What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…
Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…