Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I scraped 25M Shopify products to build a search engine

searchagora.com

Post reply on HN
Print view View original

201–210 of 286 posts

Re: Show HN: I scraped 25M Shopify products to build a search engine

#201

Post by TekMol »

ltbarcly3 wrote: post #181
TekMol wrote: post #179

The Terms page goes to "Jaggi Enterprises", "A Modern Investment Fund. We buy, build, and invest in software companies with recurring revenue.". So maybe this is not really something a guy built for his wife, but some anonymous startup that googled "Which terms rank best on Hacker News" and then wrote the "I did ... my wife .." story?

Jaggi is a fake it until you make it fake portfolio. Most of the companies it runs are just lorem ipsum fake sites. I think it is likely true that this is a solo dev.

You mean the site is not owned by Jaggi?

Then why would the terms and privacy links go to Jaggi?

Re: Show HN: I scraped 25M Shopify products to build a search engine

#202

Post by avereveard »

senecaso wrote: post #143
MuffinFlavored wrote: post #141

Earlier quoted context omitted.

> We spent time posting in forums helping people find items they were looking for, Did you run any analytics on how much overlap there was across Shopify sites on "similar items" (Alibaba resellers/dropshippers)?

we didn't, no, but we spent a lot of time sifting through our catalog, and there was a _tremendous_ amount of crap in there. We manually curated and purged shops that were obviously just dropshipping or looked like out-right scams.

Can't you sample ten random product then ask a llm to rate the shop on a scale from drop shipped to artisanal as a first approximation?

Re: Show HN: I scraped 25M Shopify products to build a search engine

#203

Post by zo1 »

berkes wrote: post #195
DeathArrow wrote: post #168

Earlier quoted context omitted.

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

Unfortunately many of these "comparison" websites have a businesses model built on affiliate fees. It doesn't take much imagination to predict which products show up as "best" or "cheapest". And the fairer ones have to keep playing cat and mouse with shops lowering pricing when they detect a scraper coming by. Or employ tricks to make their shipping seem free, lowering their overall price on the comparison platform.

Many if not all are like that. It's like everyone wants to take advantage of the lack of perfect information in the marketplace, as opposed to actually being helpful for consumers.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#204

Post by das_keyboard »

calebegg wrote: post #61
xnx wrote: post #22

Earlier quoted context omitted.

https://www.shopify.com/robots.txt lists a lot of sitemap files, which tend to be a good starting point.

It seems sort of questionable to use the list of things to not scrape as a starting point for scraping.... I mean, I get it's not actually enforced.

Not really sure why all the answers here are flagged, but you may be mistaken.

The robots.txt does not exclusively list what not to scrape.

It provides information on which parts are allowed and wich are not (disallowed).

It also provides sitemaps for crawlers as a starting point with more information (eg. which sites are available and how often are they updated, etc.)

Re: Show HN: I scraped 25M Shopify products to build a search engine

#205

Post by SushiHippie »

pbmonster wrote: post #191
olivermuty wrote: post #185

Earlier quoted context omitted.

What do you consider the local Amazon variant? And which country?

Amazon has no direct presence in Switzerland, but you can order a fraction of its products from neighboring countries. Many products are not available, mainly because nobody wants to deal with customs once the product crosses the EU boarder. Amazon itself never moved into Switzerland in the first place for many reasons (small market, unusual customs situation, relatively high salary for warehouse workers), and in the…

If you wouldn't have said that it's basically the Amazon in Switzerland, I'd have thought that this is some blogspam dropshipping site...

Re: Show HN: I scraped 25M Shopify products to build a search engine

#206

Post by wingerlang »

DeathArrow wrote: post #168
senecaso wrote: post #130

I hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we m…

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

> In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are.

Anecdotally, I guess, I'd say extremely popular. I never search for products anywhere else.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#207

Post by DeathArrow »

olivermuty wrote: post #185
DeathArrow wrote: post #168

Earlier quoted context omitted.

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

What do you consider the local Amazon variant? And which country?

Emag in Romania. I hate it, they bought most of the competition, they did a lot of anticompetitive things, but it's really easy to buy from them.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#208

Post by twelve40 »

system2 wrote: post #106
twelve40 wrote: post #43

Earlier quoted context omitted.

hm in this case, he is driving business to these websites. Quite the opposite from OpenAI and scrapers you are talking about, no?

He is putting a high amount of stress on Shopify servers. OpenAI is already in trouble with data scraping practices. Hypothetically if this site becomes successful one day, people will do whatever it takes to become #1 on the search results page and will drive business to only a handful of companies.

You're right in that it probably would be nice to have a civilized opt-out button like Google. But that's not hard to add (especially hypothetically if this site becomes successful). In my mind, total opposite of OpenAI thieving bulk data and giving 0 back to anybody.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#209

Post by pencildiver »

TekMol wrote: post #201
ltbarcly3 wrote: post #181

Earlier quoted context omitted.

Jaggi is a fake it until you make it fake portfolio. Most of the companies it runs are just lorem ipsum fake sites. I think it is likely true that this is a solo dev.

You mean the site is not owned by Jaggi? Then why would the terms and privacy links go to Jaggi?

OP here. Yup, I am in the process of starting a holding company LLC for my software products and small investments. Just went ahead and deleted 2 from the Investments page that are not launched yet but still in-development (just had landing pages up for those). Wasn't planning on releasing the Jaggi site yet, as I'm still wrapping my head around the holding company structure / it's new to me.

Agora has been a side project of mine. TBH in retrospect, I wish I would have given this post more thought as the servers / search performance wasn't prepared for any significant traffic. So definitely didn't game HN.

Post reply on HN