"There's about 25 million products on Agora right now." How many stores are represented in index.
If you check the "about" section you can see how many merchants were added, over 640k at the moment
Show HN: I scraped 25M Shopify products to build a search engine
241–250 of 286 posts
Re: Show HN: I scraped 25M Shopify products to build a search engine
#242Earlier quoted context omitted.
Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…
Yo fuck mongo just use RDS or some digitalocean DB. Or really just use opensearch/elasticsearch, or even typesense (don't bother with raft it's so broken) or meilisearch
We’ve interacted before on Twitter and GitHub, and I want to address your point about Raft in Typesense since you mention it explicitly:
I can confidently say that Raft in Typesense is NOT broken.
We run thousands of clusters on Typesense Cloud serving close to 2 Billion searches per month, reliably.
We have airlines using us, a few national retailers with 100s of physical stores in their POS systems, logistic companies for scheduling, food delivery apps, large entertainment sites, etc - collectively these are use cases where a downtime of even an hour could cause millions of dollars in loss. And we power these reliably on Typesense Cloud, using Raft.
For an n-node cluster, the Raft protocol only guarantees auto-recovery for a failure of up to (n-1)/2 nodes. Beyond that, manual intervention is needed. This is by design to prevent a split brain situation. This not a Typesense thing, but a Raft protocol thing.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#243I'm a Shopify store owner myself. I saw there is a $99 per month to get your product verified, how would this compete in terms of CPC with a traditional channel such as google ads or meta ads?
Re: Show HN: I scraped 25M Shopify products to build a search engine
#244Love it! Some improvements are needed on search but is an amazing MVP, I'll use this for my late christmas shopping
Re: Show HN: I scraped 25M Shopify products to build a search engine
#245I'm a Shopify store owner myself. I saw there is a $99 per month to get your product verified, how would this compete in terms of CPC with a traditional channel such as google ads or meta ads?
Not the OP but counting on an average of $3 dollars per click in the US across those channels, I'd say this pricing is way more effective with the amount of searches the site is getting.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#246Cool project! As you scale, you may benefit from these two projects I maintain, and the Big Tech uses :) https://github.com/unum-cloud/usearch - for faster search https://github.com/unum-cloud/uform - for cheaper multi-lingual multi-modal embeddings Feel free to reach out with feedback and feature requests!
I've been following your work for a while, was really excited to play with UDisk but I guess that got dropped in favor of AI solutions?
Due to constrained resources we had to prioritize the smaller and simpler projects - USearch, UForm, UCall, and on the personal side - StringZilla, and SimSIMD. That’s a lot for a team without revenue and VC funding.
I am still actively thinking about open-sourcing UDisk. Thanks for keeping tabs on us :)
Re: Show HN: I scraped 25M Shopify products to build a search engine
#247Cool project! As you scale, you may benefit from these two projects I maintain, and the Big Tech uses :) https://github.com/unum-cloud/usearch - for faster search https://github.com/unum-cloud/uform - for cheaper multi-lingual multi-modal embeddings Feel free to reach out with feedback and feature requests!
Can't believe I missed this. Taking a look at both repos now. The further I get into this space, the less I feel like I know. Appreciate you sharing, will reach out!
Don’t worry, our solutions aren’t well known. They are either used by enthusiasts, or the Big Tech, and the latter don’t like mentioning that :)
Re: Show HN: I scraped 25M Shopify products to build a search engine
#248Incredible. Would love to connect with you. Where can I find you LOL
Re: Show HN: I scraped 25M Shopify products to build a search engine
#249Incredible. Where can I connect with you? Want to pick your brain & swap some thoughts :)
Re: Show HN: I scraped 25M Shopify products to build a search engine
#250Any Unicode input (Japanese or Greek text for example) currently causes a 500 error.