Live data from Hacker News

Show HN: I scraped 200M Shopify products to build a search engine

searchagora.com

41–44 of 44 posts

Re: Show HN: I scraped 200M Shopify products to build a search engine

#41
post #8

Earlier quoted context omitted.

Finding products that aren't on Amazon or other top sites is a problem for consumers. Google Shopping is pretty bad. Connecting buyers direct to merchants is a win and something they all want. Every single entity that sells online is trying to boost their revenue share for direct to consumer as a mandate. Because they get more margin and they get to know their customer for ongoing marketing. The pandemic exposed many…

You refute my point and then agree with me that it can be expensive. I worked for largest eCommerce employer twice and have built scraping and search systems at scale. I know a thing or two in eCommerce shopping sites. Do you know how much infra and people it takes to keep pricing, product, variant data up-to-date? Hint: It's significant capex investment. Problem is entry point funnel in the shopping journey. If all…

You never even commented that it can be expensive just said it's a terrible idea because it's "lose-lose" and a "traffic problem".

Anyway, I refuted ALL of your points and even supplied angles for keeping costs down while driving value to defined audiences.

I personally scraped and ML'd all AirBnB listings and calendar data every day for over a year to build a product. It didn't cost that much in time and $. Got a shit ton of value out of it. Someone even built a nice business doing exactly what I was doing.

If you look on the other thread for the OP, someone had done 100m products for $550/mo. You also must not understand that shopify and woocommerce have structured data available because you are talking about people being needed to keep product data up to date. They aren't scraping the frontend.

Personally I'm a riches in niches guy and a more defined audience and product set would be the way to go off of what they've started.

The ocean doesn't need to be boiled to provide value to consumers and to merchants here.

Re: Show HN: I scraped 200M Shopify products to build a search engine

#42

How do you plan to drive customer traffic to this site? As others mentioned, it's bare bones, raw search engine. I think these days, consumers need something more than just a bare choice because it's too much. People get paralyzed when they are presented with multiple options. I think if you could develop something that works similar to interest or Instagram, that would be more interesting, especially for female cons…

Generally I'm of the belief that there are 2 types of users, with different acquisition strategies: 1. Users who want to find a very specific product, with the intent to purchase. This assumes that the search functionality and quality filters are working very well. User comes to Agora, finds exactly what they are looking for, and continues with their day. We save them time and provide them with a lot of immediate val…

This man really knows what he is talking about.

I am one of such customers, either looking for that Peripheral / Tech with exact criteria, or I am just browsing around for new releases and recommendations (mostly on blogs & tech forums).

I am not interested in the toy or makeup advertised by the top post on instagram. I am interested in products with long review by a pro reviewer and some honest discussion around it from random customers.

Re: Show HN: I scraped 200M Shopify products to build a search engine

#43

Earlier quoted context omitted.

> You enter criteria/product name and you get results Not on Amazon. At least not since 2020 or so.

GP is correct. There is not much value in search itself. Are you serious about Amazon? I can absolutely find everything I need on Amazon. Who needs exact search when there are better alternatives through recommendation, people also buy etc? I am curious to know — if you think search on amazon as broken as you think, why do you think people keep buying from Amazon? They have strong logistic network. But, to even start…

Amazon (as of today, in Europe, for me) is a Wish with better logistics and replacement policy.

If I search for a product like a milk former or a wine opener or a phone case, I already expect the results to be 80% copycat crap and the reviews fake.

Adjusted ratings by various fake spotter tools usually bring down 4.5+ ratings down to 3.0–3.5.

That the search engine absolutely refuses sometimes to include words I want and exclude words I don’t want is the icing on the cake.

If I didn’t forget the renewal date, I’d have quit Prime (which I subscribed to since its introduction) already last year.

Post reply on HN