Ask HN: How do I improve our data infrastructure?

Well, when several people are working on the same project they "share" the transformed data by connecting to the same EC2 instance. The way data is transformed is via 4 scripts, 2 notebooks and a bunch of manual operations, so no one really wants to touch that. I spent my 4th day working with a contractor to write a Makefile that reproduces all the steps for ONE project.

I talk about adding infrastructure in my original post, but I'm very well aware that my time is currently better spent consolidating the existing as much as I can so the clients can get correct results faster.

Thank you for taking the time to answer thoroughly !

> If it is only used by data science team then you can rip apart the solution and work towards a more logical infrastructure where all you are doing is cleansing, normalizing and deriving features and these become your central feature repository which your team can pull and build models.

It is only used by data scientists. What do you mean by a feature repository? How would you organize it so people can push new features? This sounds very interesting.

> you are talking about denormalization as something you are planning to do, that should have been the starting point of any HDFS/SPARK/Parquet based solution.

It is something that we have to do, but the table have been dumped as is in S3 and every project rebuilds the whole derived dataset regularly. Since these operations are very brittle (a lot of manual work and even transformations performed in notebooks), this is something people dread doing. I am trying to secure this at the moment by writing Makefiles that remove human intervention, but at the end of the day I would like to avoid people spend hours waiting for new data when they need it.

> I can suggest tools for explorations, data quality check etc.

I would appreciate it. Put simply, we get data about the evolution of the stock of clients, transactions with their clients, product descriptions, etc. that is dumped into S3 (I scheduled a chat with people upstream to see what happens). We have 3,4 projects for each client. What currently happens is every team writes the same code to build features in their separate repositories, this code is re-executed every time new data arrives (weekly). These features are then used in prediction models.

Besides the brittleness of the process, I found that people are reluctant to analyse the data because it takes an unreasonable amount of time.

"I was just hired as the first permanent data scientist..."

You hope.

"I was thinking about building a [another] pipeline... "

You're fired.

Shouldn't you first deliver some new insight based on your data analysis skills?

In this case no one technically competent was here to manage them. There were no expectations.

Like others have said, the technical infrastructure is usually a manifestation of the people processes of the corporation. I think it's valuable to kinda ignore the technical stuff initially, and instead first understand the requirements of your customers. It's totally possible that the current system, as weird as it is, might satisfy your customers' requirements best. Unlikely, but it's possible.

But given that they hired you, chances are they know the current system isn't great, and they didn't possess the domain knowledge to fix it. I'd guess you & your company are aligned high-level that change is needed. It's just a matter of making sure you can align your ideas with the short- and long-term goals of the company, usually with a convincing story explaining how your technical changes drive business value.

For example, you mention that data exploration is hard, and I'm inferring this is a problem because you have multiple consultants independently scouring your datasets. If so, you could communicate to your customers that you can reduce consultant onboarding time from 5 days to 2 days (made that up) if you invested in aggregated datasets or a centralized data warehouse. If you can translate this to a dollar figure (like consultant hourly rate), that's even better.

As for what part you tackle first, I'd suggest finding a problem everyone knows about, but is straightforward for you to solve. Goal is to display immediate value, and gain the trust of the people around you. You don't solve the systemic problem immediately, but the trust you gain is currency you use months from now to really invest in the system. Because truth is, higher-ups rarely value invisible things like data quality or maintainability, they respond very positively to shiny new graphs and numbers.

FWIW I don't know if it's just me, but I feel like the bulk of data science is the ugly pipeline and architectural decisions you're facing now. I read people doing interesting modeling & machine learning work, but I keep wondering how much work went into getting the data into a modeling-ready state. I haven't worked at a company where the % of data team effort going to pipelines is less than, say, 80%.