Live data from Hacker News

Show HN: I scraped 25M Shopify products to build a search engine

searchagora.com

51–60 of 286 posts

Re: Show HN: I scraped 25M Shopify products to build a search engine

#51

I like it. I need to be able to filter search to if it will deliver to my country. It desperately needs some indication that your action is being processed, like a spinner, when you search.

Absolutely. Working on a "ships to" filter and enhancing the 'price' filter.

Also fixing the loading experience as we speak. Wasn't expecting this level of traffic so didn't account for slow server speed with the front-end experience.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#52
post #37
post #22

Earlier quoted context omitted.

https://www.shopify.com/robots.txt lists a lot of sitemap files, which tend to be a good starting point.

Did this suddenly get changed? Nothing but "# ,: # ,' | # / : # --' / # \/ />/ # /" is shown now.

Weird. I think it did change. Google cache shows a 2229 line file: https://webcache.googleusercontent.com/search?q=cache%3Ahttp...

Re: Show HN: I scraped 25M Shopify products to build a search engine

#53

Awesome! It would be good to listen to the enter key when typing in a search query. Your privacy and terms links point to what appears to be the saas code framework you used (just a guess). I was looking for your contact/email so I can ask you some questions.

Enter key should work, but the loading speed is very slow right now. Fixing that right now :)

You can reach out at support @ searchagora.com. Would love to talk!

Re: Show HN: I scraped 25M Shopify products to build a search engine

#55

That's funny, I made a domain-specific version of this for canadian coffee deals. https://beangrid.mcconomy.org/

Super cool project (especially as a coffee lover myself)!

The fun part was figuring out how I was going to put the site up without hosting ;)

Re: Show HN: I scraped 25M Shopify products to build a search engine

#56
I'm sorry, but I have to question where this heartfelt story about looking out for your wife is in any way real?

The website certainly doesn't look like a side project, it has a fully fledged system for merchants to advertise on Agora for a fee, an affiliate system offering $50 commissions to onboard merchants and the ToS and Privacy policy link to a website with the following mission statement:

> We buy, build, and invest in software companies with recurring revenue and product-led growth.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#57
What was the process for scraping 25M products ?

I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale.

Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ?

______________

A recommendation for how to improve search.

Your base captions will be pretty bad. You can use spot instances on a smaller GPU machine to run a dense captioning model (https://portal.vision.cognitive.azure.com/demo/dense-caption...) and generate captions for all your images.

Then for search, a simple vector store index would be a great retrieval solution here. It is better to do search using those as well.

Both are pretty cheap and can be done reliably within 20-30 lines of code each in python. 3rd party tools for these are pretty stable.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#58
post #27

Great project. If you continue to crawl the data, be sure to save it so you can detect price changes a la camelcamelcamel.

For all of Amazon's faults, the fact that they tolerate CCC does drive a lot of my online purchases there. CCC used to track other sites, and was eventually blocked on all of them. If more sites want my business, showing their pricing history (either from internal data, or by letting someone build the DB) would go a long way.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#59
post #52
post #37

Earlier quoted context omitted.

Did this suddenly get changed? Nothing but "# ,: # ,' | # / : # --' / # \/ />/ # /" is shown now.

Weird. I think it did change. Google cache shows a 2229 line file: https://webcache.googleusercontent.com/search?q=cache%3Ahttp...

Seems it might be looking at the referrer. Loading https://www.shopify.com/robots.txt from clicking the link shows the weird line while opening it in a private browser window shows the right one.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#60

I'm sorry, but I have to question where this heartfelt story about looking out for your wife is in any way real? The website certainly doesn't look like a side project, it has a fully fledged system for merchants to advertise on Agora for a fee, an affiliate system offering $50 commissions to onboard merchants and the ToS and Privacy policy link to a website with the following mission statement: > We buy, build, and…

OP here. Yup, haven't launched the holding company yet but idea is to have an LLC for all my software projects. Still a work-in-progress in both thinking and execution. Agora specifically came from a personal need and is obviously still an MVP project.

Spun up a Merchant Page and Affiliate Program page in a few hours on Webflow using a template. There is a merchant dashboard built but the 'affiliate program' is a test.

Post reply on HN