Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I scraped 25M Shopify products to build a search engine

searchagora.com

Post reply on HN
Print view View original

231–240 of 286 posts

Re: Show HN: I scraped 25M Shopify products to build a search engine

#231

Post by asdadsdad »

pencildiver wrote: post #108
asdadsdad wrote: post #83

cool project. You might have notice, but there's a non-trivial amount of fraud on shopify (fake shops, info stealers, etc). Might be interesting to look at that dataset and explore a bit =) it's quite fascinating

I've definitely noticed that already. Any advice on how to spot that? Another challenge is that there are products sold on the original site and third-party marketplaces, both of which could sell on Shopify. So need to find a way to automatically detect the type of store.

you might check this for inspiration (https://seguranca-informatica.pt/shopping-trap-the-online-st...).

I used to have a huge IOC collection, but now stopped tracking them.

things like HTML markup, pricing patterns, IPs might inform on specific clusters of fraudsters.

I don't think Shopify cares tbh

Re: Show HN: I scraped 25M Shopify products to build a search engine

#232

Post by ttt3ts »

slimebot80 wrote: post #158
ttt3ts wrote: post #78

Built the same thing a while back while collecting a lead list for sales. Not bothered to keep data updated but was a fun thing to build in a couple days. (disclaimer mobile experience is meh cause it was a fun project) https://zensear.ch How did you find list of all Shopify stores? I ended up just checking every .com, .net, etc as I didn't find an easy way to figure it out directly from shopify.

Nice. But can I ask what motivated you? I don't see any affiliate details in the links - do you monetise at all?

Wanted to play with typesense.

The tech behind building something like this isn't hard. Marketing and traffic is. No point in monetizing with no users.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#233

Post by stef25 »

pencildiver wrote: post #51
quickthrower2 wrote: post #44

I like it. I need to be able to filter search to if it will deliver to my country. It desperately needs some indication that your action is being processed, like a spinner, when you search.

Absolutely. Working on a "ships to" filter and enhancing the 'price' filter. Also fixing the loading experience as we speak. Wasn't expecting this level of traffic so didn't account for slow server speed with the front-end experience.

Or even "ships from" / "located in".

I'm in Europe and don't want to deal with custom hassles or delays from shipping. Etsy and Reverb both have this option which I never fail to use.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#234

Post by pencildiver »

jross225 wrote: post #230

heh, I used to work on the data team at Shopify. I built something similar to search internal dbs for secret santa gifts based on some weird criteria. Scraping might have a large margin of error because a lot of products tend to be ephemeral. Neat project though!

Agreed on the large margin of error. Working on a bot to store and convert the images to webp to improve performance. Having the bot do a check for any images that don't exist and removing those listing. Will likely also need to triangulate this with a 404 check. Recently added an option for users to mark a product as "sold out" on the search results which will help as well.

Unrelated, but what was the "weird criteria" for the secret santa exchange? Half joking but also helps with figuring out filters :)

Re: Show HN: I scraped 25M Shopify products to build a search engine

#235

Post by pencildiver »

asdadsdad wrote: post #231
pencildiver wrote: post #108

Earlier quoted context omitted.

I've definitely noticed that already. Any advice on how to spot that? Another challenge is that there are products sold on the original site and third-party marketplaces, both of which could sell on Shopify. So need to find a way to automatically detect the type of store.

you might check this for inspiration ( https://seguranca-informatica.pt/shopping-trap-the-online-st... ). I used to have a huge IOC collection, but now stopped tracking them. things like HTML markup, pricing patterns, IPs might inform on specific clusters of fraudsters. I don't think Shopify cares tbh

Super helpful, thank you.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#236

Post by Oras »

pencildiver wrote: post #80
screye wrote: post #57

What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…

Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…

Take a look at TypeSense. Faster, better filtering, and much much cheaper if you’re going the cloud version

Re: Show HN: I scraped 25M Shopify products to build a search engine

#237

Post by Karl-Heinz »

stef25 wrote: post #225
berkes wrote: post #199

Earlier quoted context omitted.

> site called "Built With", Do you have Alink. And are they any good?

Google ? https://builtwith.com/

Berkes wanted to do good by sharing a provision with the OP, in case he/she buys something at builtwith.

We all know how to Google. :)

Re: Show HN: I scraped 25M Shopify products to build a search engine

#238

Post by alvarome »

slimebot80 wrote: post #158
ttt3ts wrote: post #78

Built the same thing a while back while collecting a lead list for sales. Not bothered to keep data updated but was a fun thing to build in a couple days. (disclaimer mobile experience is meh cause it was a fun project) https://zensear.ch How did you find list of all Shopify stores? I ended up just checking every .com, .net, etc as I didn't find an easy way to figure it out directly from shopify.

Nice. But can I ask what motivated you? I don't see any affiliate details in the links - do you monetise at all?

It looks like from the "become a merchant" you can directly boost your products which I assume will prioritize them and hence drive more traffic to your Shopify store

Re: Show HN: I scraped 25M Shopify products to build a search engine

#239

Post by alvarome »

system2 wrote: post #99

How are you planning to monetize this? You mentioned you are spending around $2K just to run it. Is there a commission strategy or ads? Or populate with your products at one point so you sell your own thing?

It looks like from the "become a merchant" you can directly boost your products which I assume will prioritize them and hence drive more traffic to your Shopify store

Re: Show HN: I scraped 25M Shopify products to build a search engine

#240

Post by selcuka »

ttt3ts wrote: post #81
selcuka wrote: post #46

Earlier quoted context omitted.

I don't think that would work as many people also use Cloudflare etc. You may try using BuiltWith which is a paid service: https://trends.builtwith.com/websitelist/Shopify

It works fine. Just issue a HEAD request when you are unsure and rotate proxies a lot l. Takes a bit of infra but definitely possible.

I mean simply querying nameservers won't work.
Post reply on HN