Show HN: I scraped 25M Shopify products to build a search engine • Hacker News Board

Re: Show HN: I scraped 25M Shopify products to build a search engine

#251

Post by senecaso » Thu, Dec 14, 2023, 3:40 PM UTC

avereveard wrote: post #202

senecaso wrote: post #143

Earlier quoted context omitted.

we didn't, no, but we spent a lot of time sifting through our catalog, and there was a _tremendous_ amount of crap in there. We manually curated and purged shops that were obviously just dropshipping or looked like out-right scams.

Can't you sample ten random product then ask a llm to rate the shop on a scale from drop shipped to artisanal as a first approximation?

I doubt it would be that easy, but, ya, using some form of automation is necessary. We devised a few rudimentary way to filter out the chaff, and it did quite well to remove the garbage. Still some would slip through, so it still required vigilance to remove them when you happen to see them.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#252

Post by senecaso » Thu, Dec 14, 2023, 3:49 PM UTC

DeathArrow wrote: post #168

senecaso wrote: post #130

I hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we m…

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

We were intentionally limiting the number of products and shops we were indexing due to opex. We needed to keep it low enough to provide ourselves with enough runway to keep things floating for longer.

pricerunner is another site which operates in a similar space. We had plans to build out the price tracking and a number of other features, so that we would appeal more to users who had your use cases. Sadly, we weren't getting enough traction. We did have regular users from the EU, but we simply couldn't seem to get in front of enough eyeballs for it to matter. At least at first, I expect that a large amount of your traffic to a new site like this has to be driven by Google, and we failed on that front as well. I'm not an SEO expert, so there were likely many things we did wrong or didn't even do which lead to this situation.

re: a sales platform, that's a pretty big challenge to take on, which would require massive investment up front. Not sure thats a viable route for most. We did have plans to address the "without hoping to various websites" problem, as we identified that as problematic for users very early on. The solution was relatively simple, but required more money to build out. We simply ran out of funds before we could get there.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#253

Post by xnx » Thu, Dec 14, 2023, 3:58 PM UTC

calebegg wrote: post #61

xnx wrote: post #22

Earlier quoted context omitted.

https://www.shopify.com/robots.txt lists a lot of sitemap files, which tend to be a good starting point.

It seems sort of questionable to use the list of things to not scrape as a starting point for scraping.... I mean, I get it's not actually enforced.

Since ~2009 many crawlers recognize "Sitemap:" directives in robots.txt to link to sitemaps: https://en.wikipedia.org/wiki/Robots.txt#Sitemap

Re: Show HN: I scraped 25M Shopify products to build a search engine

#254

Post by qdequelen » Thu, Dec 14, 2023, 4:06 PM UTC

Hey, I'm the CEO of Meilisearch. If your issue is performance, I would love to give you a try with Meilisearch. You'll be able to create an "as you type" experience with our engine that responds in less than 50ms!

Re: Show HN: I scraped 25M Shopify products to build a search engine

#255

Post by qdequelen » Thu, Dec 14, 2023, 4:08 PM UTC

pencildiver wrote: post #7

misterbwong wrote: post #2

What technology did you use to build the scraper and how did you get around the usual challenges (anti bot, ip banning, etc) with scraping large amounts of data?

Scraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end. Here's an example: https://www.wildfox.com/products.json

Did you only get the schema.json?

Re: Show HN: I scraped 25M Shopify products to build a search engine

#256

Post by ctocoder » Thu, Dec 14, 2023, 4:28 PM UTC

how did you get a list of the 25 million stores to crawl?

Re: Show HN: I scraped 25M Shopify products to build a search engine

#257

Post by senecaso » Thu, Dec 14, 2023, 4:28 PM UTC

DeathArrow wrote: post #170

senecaso wrote: post #156

Earlier quoted context omitted.

Our business model revolved around referrals, so lack of users directly translated to lack of revenue. While its true that even if we had millions of users but none of them were buying sponsored items we would have had a revenue problem, that wasn't the problem we were facing, as the few users we did have were in fact purchasing sponsored items.

Then the problem seem to be the lack of users. Have you tried having an YouTube channel, TikTok, Facebook, Twitter, blog and explain daily how you built the website, how your platform is going to help users?

we did have channels on various sites, yes. However, its difficult to maintain a steady stream of content there for people to consume. Not only that, but you have the same discoverability problems as you do for the main site. Also, a blog outlining how you built the site may be of limited value. At least my experience on that front was it would generate short-lived bursts of traffic, but wouldnt generate returning users. So I think those articles were mostly appealing to technical users, and not necessarily users who were looking to do some shopping. Of course technical users do also shop, but after reading a technical article, they probably arent looking to immediately shop, and without some other mechanism putting the site in front of them again when they needed to shop, we would miss the opportunity.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#258

Post by senecaso » Thu, Dec 14, 2023, 5:06 PM UTC

callmeed wrote: post #167

I built this a couple years ago (now defunct) for the same reason :) The public JSON endpoints on shopify stores make it pretty easy to get the data. You mentioned using Mongo but it sounds expensive. I honestly think you could do this with just elastic or even postgres full text search and save money. Here's a pro tip + feature you should implement: Shopify has a semi-hidden hack where you can link directly to check…

I didnt know about the link to checkout. That's a slightly nicer user experience for sure. Still, its confusing for users who want to do more shopping at the same time. I had users who clicked on a number of items, clicked "add to cart" in each one (all different shops), and then couldn't figure out how to checkout on the main site afterwards! Obviously people were looking for a more complete one-stop-shopping experience than I was providing at the time.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#259

Post by konschubert » Thu, Dec 14, 2023, 5:38 PM UTC

pencildiver wrote: post #159

konschubert wrote: post #8

Hey, I have a Shopify store that sells e-paper calendars / smart screens. I tried to search for it but I could not find it. What should I do so your crawler can find me? https://shop.invisible-computers.com

You’re live on Agora: https://www.searchagora.com/products/invisible-calendar-6266... Thinking that we should have a page where store owners can submit their URL to be crawled.

Cool, thanks!

Re: Show HN: I scraped 25M Shopify products to build a search engine

#260

Post by philippemnoel » Thu, Dec 14, 2023, 5:55 PM UTC

mfrye0 wrote: post #115

pencildiver wrote: post #80

Earlier quoted context omitted.

Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…

If you're already on AWS, I recommend switching to postgres for now. For context, I have 3 RDS instances, each multi zone, with the biggest instance storing several billion records. My total bill for all 3 last month was $661. Postgres has full text search, vector search, and jsonb. With jsonb you can store and index json documents like you would in Mongo. - https://www.postgresql.org/docs/current/textsearch.html - h…

You can even do Elastic-level full text search in Postgres with pg_bm25 (disclaimer: I am one of the makers of pg_bm25). Postgres truly rules, agree on the rec :)