Show HN: I scraped 25M Shopify products to build a search engine • Hacker News Board

Re: Show HN: I scraped 25M Shopify products to build a search engine

#221

Post by vargr616 » Thu, Dec 14, 2023, 12:54 PM UTC

olivermuty wrote: post #185

DeathArrow wrote: post #168

Earlier quoted context omitted.

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

What do you consider the local Amazon variant? And which country?

bol.com in the Netherlands

Re: Show HN: I scraped 25M Shopify products to build a search engine

#222

Post by dangoodmanUT » Thu, Dec 14, 2023, 12:54 PM UTC

ashvardanian wrote: post #140

Cool project! As you scale, you may benefit from these two projects I maintain, and the Big Tech uses :) https://github.com/unum-cloud/usearch - for faster search https://github.com/unum-cloud/uform - for cheaper multi-lingual multi-modal embeddings Feel free to reach out with feedback and feature requests!

I've been following your work for a while, was really excited to play with UDisk but I guess that got dropped in favor of AI solutions?

Re: Show HN: I scraped 25M Shopify products to build a search engine

#223

Post by pencildiver » Thu, Dec 14, 2023, 1:12 PM UTC

ashvardanian wrote: post #140

Cool project! As you scale, you may benefit from these two projects I maintain, and the Big Tech uses :) https://github.com/unum-cloud/usearch - for faster search https://github.com/unum-cloud/uform - for cheaper multi-lingual multi-modal embeddings Feel free to reach out with feedback and feature requests!

Can't believe I missed this. Taking a look at both repos now. The further I get into this space, the less I feel like I know. Appreciate you sharing, will reach out!

Re: Show HN: I scraped 25M Shopify products to build a search engine

#224

Post by jesterson » Thu, Dec 14, 2023, 1:14 PM UTC

RagnarD wrote: post #216

jesterson wrote: post #175

Earlier quoted context omitted.

Does it matter? Or you can't do anything not explicitly allowed by law? Shopify is the company spotted in so many shenanigans, so anything that undermines it's business I would personally welcome very well.

It matters when you get sued.

Living in fear is pathetic

Re: Show HN: I scraped 25M Shopify products to build a search engine

#225

Post by stef25 » Thu, Dec 14, 2023, 1:19 PM UTC

berkes wrote: post #199

pencildiver wrote: post #80

Earlier quoted context omitted.

Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…

> site called "Built With", Do you have Alink. And are they any good?

Google ?

https://builtwith.com/

Re: Show HN: I scraped 25M Shopify products to build a search engine

#226

Post by stef25 » Thu, Dec 14, 2023, 1:28 PM UTC

satvikpendem wrote: post #16

pencildiver wrote: post #7

Earlier quoted context omitted.

Scraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end. Here's an example: https://www.wildfox.com/products.json

How did you detect that it was a Shopify store?

Looked in the source of a random Shopify store, there are 200+ occurrences of "shopify", that's a clue :)

Re: Show HN: I scraped 25M Shopify products to build a search engine

#227

Post by virtuosarmo » Thu, Dec 14, 2023, 1:28 PM UTC

I believe Shopify built their own app / website where you can search for products exclusively from Shopify merchants. https://shop.app/

Re: Show HN: I scraped 25M Shopify products to build a search engine

#228

Post by pencildiver » Thu, Dec 14, 2023, 1:39 PM UTC

dangoodmanUT wrote: post #218

Super cool!!!

Thanks! Laughed audibly at your comment in the other thread about Mongo haha

Looking into all options right now. A tradeoff between current stability, price, and performance.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#229

Post by amne » Thu, Dec 14, 2023, 1:42 PM UTC

DeathArrow wrote: post #207

olivermuty wrote: post #185

Earlier quoted context omitted.

What do you consider the local Amazon variant? And which country?

Emag in Romania. I hate it, they bought most of the competition, they did a lot of anticompetitive things, but it's really easy to buy from them.

At some point, a couple of years ago when they introduced marketplace, I actually thought they are aiming for an "exit" to Amazon. They really got the service part of e-commerce nailed down. Merchants quality is and always will be an issue, but it is the same as on Amazon.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#230

Post by jross225 » Thu, Dec 14, 2023, 1:50 PM UTC

heh, I used to work on the data team at Shopify. I built something similar to search internal dbs for secret santa gifts based on some weird criteria. Scraping might have a large margin of error because a lot of products tend to be ephemeral.

Neat project though!