101–110 of 286 posts
Have Swedish family. Searched dala because family wants traditional Christmas ornaments. Sure enough, there were several results that were 10x cheaper than what I could find on the first page of Big Search Company. Great job!
[flagged]
I wouldn't be happy to hear someone calling something I worked on "dogshit" to be honest. I learned from his work today and appreciate his approach. It doesn't hurt to be kind.
Searching is slow (kinda expected that right now), but after clicking a product and then hitting back, I have to wait for the search again. Not at computer so I didn't check the headers, but maybe allow the client to cache the response for a short time so it doesn't need to load search results again.
What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…
Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…
Earlier quoted context omitted.
You really didn't think about this. Search engine crawlers are specifically designed to ignore sites that have the right setups. In other words: they try to be good citizens of the web. Data scrapers on the other hand are known to steal others IP and use it to jump start their own services. When you do that you're using all the bandwidth of the site while devaluing everything they've built. They will have invested si…
hm in this case, he is driving business to these websites. Quite the opposite from OpenAI and scrapers you are talking about, no?
Cool! But how did you get the initial dataset of 643,000+ Shopify stores (data as per your “About” page) in the first place, to then scrape the products from their /products.json feeds? Or did you just try a huge list of domain names at random?
Shopify sites also have shop-name.com/products.json which has URLs that point to cdn.shopify.com
cool project. You might have notice, but there's a non-trivial amount of fraud on shopify (fake shops, info stealers, etc). Might be interesting to look at that dataset and explore a bit =) it's quite fascinating
Another challenge is that there are products sold on the original site and third-party marketplaces, both of which could sell on Shopify. So need to find a way to automatically detect the type of store.
I like it. I need to be able to filter search to if it will deliver to my country. It desperately needs some indication that your action is being processed, like a spinner, when you search.
Absolutely. Working on a "ships to" filter and enhancing the 'price' filter. Also fixing the loading experience as we speak. Wasn't expecting this level of traffic so didn't account for slow server speed with the front-end experience.