Ask HN: Working with CPG Retail Data

I work in the consumer packaged goods (CPG) industry. Specifically the US market. Think of the sorts of items you buy on a consistent basis at a grocery, big box, club, or drug store. Everything from bagged salad to cough syrup.

Over the past several years I’ve worked at a mid-size company and moved from a basic analyst role to wearing several data hats. Mostly taking on data engineering and business intelligence tasks. The hot thing right now is Power BI reporting, so I build out data pipelines, create data models, and design Power BI reports. This has lead to a lot of career success at my company, but lately I’ve been more and more frustrated by the seemingly antiquated data practices in the CPG industry.

My company is a bit unique in that we are not a single retailer or manufacturer. We work with CPG brands and retailers across the entire country. This means we rely heavily on syndicated retail data from providers like Circana (merger between IRI and The NPD Group) and NielsenIQ (NIQ). They get retail scan data from almost every CPG retailer in the country. Except some retailers are exclusive to one platform, so you’ll never have a complete picture without both.

However, Circana and NIQ do not make it easy to extract what I consider medium-size data. Everything is a portal with data presented across various “dashboards” and rudimentary no-code report creator web apps. The minute you ask about an API or data transfer service they wonder why you would ever want to leave their platform. And when you do convince your company to purchase large data extract access or similar (why do we need that when we have “analysts” who can extract hundreds of small data pieces through the web portal?), you find out it’s incredibly fragile, inflexible, slow, and unreliable. The pricing for access is never transparent either.

In some cases, I can access a retailers’ data directly (for the manufacturers my company works with) though their web portal and reverse engineer an API. But that’s typically a limited data set and more supply chain focused.

Has anyone developed a successful data strategy in the CPG realm? Is there an opportunity here to solve these problems? Are these normal problems in tech/other industries?

I don’t have experience in this but it sounds like obstacles put in place precisely to stop the bulk data extraction you speak of, and force lock in.

Off that’s the case you’ll find it very difficult to change.

I’ve found in other business domains, often things that make no sense and have a fairly easy solution are that way because of lock in strategies.

Part of the problem is that those 3p data providers aren’t sophisticated, but the other factor to consider is that the account managers and support workers assigned to you are less sophisticated. I work at a major retailer and one of the 3P vendors you mentioned does partner with us on providing data pipelines that we query directly. Feel free to pm me, I’m always happy to learn more about other people’s experiences in our industry

Not specific to the CPG realm, but I've had some very good luck working with the OpenAPI-devtools Chrome Extension[1] (previous discussion here on hackernews[2]) to discover the underlying APIs of various sites that I want to scrape data from.

[1] https://github.com/AndrewWalsh/openapi-devtools

[2] https://news.ycombinator.com/item?id=38012032

Yes, I've been in precisely that situation with precisely those vendors. We would get data in some weird proprietary format that was only readable in the vendor's awful desktop application.

We ended up spending a couple months reverse engineering these files (they ended up looking like database page tables or something) to finally break out of that ecosystem and get the data in our own databases.

It was a lot of work but absolutely worth it in the end. I have a feeling just the ability to natively read and convert those files is worth a lot but it's also the kind of thing that seems like it might be hard to sell...