Ask HN: Working with CPG Retail Data

would it be legal to sell

I've been a senior leader at a CPG shopper marketing and retail technology company out of Bentonville, AR, primarily supporting Walmart, but also working on Target, Amazon, and a few dozen other regional grocery stores, and can confidently tell you that you are not crazy, there is no unified solution, and if you stay in this industry you will be chasing the dream of a simpler and less painful solution forever. :P

You mention retailer web portals being the only source you can get some of the data you need and that it's very manual. Historically Walmart has had the same situation with their Retail Link application that vendors can download sales data from (this has finally changed recently with the release of Walmart Luminate). No API. Just an antiquated UI that you can create scheduled reports that export excel files. To that end, for the last 20+ years there has been a vibrant industry in Northwest Arkansas (where Walmart is based) of startups creating web crawler based systems that create, schedule, and download reports from Retail Link and then ingest them into a system that is more conducive to powering the analytics demands of vendors. This niche industry of web crawler based data and analytics companies has created a market (just taking some broad but somewhat confident guesses here) of dozens if not hundreds of millions of dollars of capital over the last 20 years. You asked if there's "an opportunity here to solve these problems", there certainly is and has been for a while, but unfortunately it's a painful path.

I feel pretty confident in stating that the "data strategy" in the CPG realm has been, is currently, and for the foreseeable future will be: persistence, resilience, and creativity. Keep trying to build something better despite the constraints, don't give up when it feels like an uphill battle, and never accept the apparent constraints of the technical environment. I've reverse engineered so many things that we would eventually turn into a product to provide value to our customers. Nothing feels better than telling someone you figured out a way around something that they were told was not possible.

Be creative. Be persistent. If I'm being completely honest, get weird. The solutions you'll need will likely not fit perfectly into best practices. You'll definitely help the company you work for today, but you'll also likely help your own career in the long run (if you choose to stay in the retail industry). It may feel like a pain in the ass today, but if you apply your persistence and knowledge to it, these challenges will be the thing that make you super valuable and versatile engineer that people want to hire in the future.

At Databoutique.com we’re trying to solve the web data accessibility problem with a marketplace which connects web data sellers and buyers. Buyers can get the data with three clicks, on S3 bucket or download from the website. It’s pre scraped, quality checked and legal compliant. If a website is not listed, you can ask to sellers to provide it. Sellers deliver data using standard data structures and make their price. Far from perfect since we launched three months ago, but working on it.

This is a huge help, thank you. I've been using the Postman Interceptor extension, but OpenAPI-devtools looks tailor made for my use case. The hard part will still be figuring out the auth headers/calls (EDIT: the dev has added auth support since the HN post!)

Luckily I haven't had to use those desktop apps in 7 years, and today everything is through their web portal + .csv exports. Export restrictions still apply, but at least it's a standard format.