Ask HN: How do I improve our data infrastructure?

The code is not pretty to look at, but this is not our biggest problem. We inherited a weird infrastructure: a mix of files in HDF5 and Parquet format dumped in S3, read with Hive and Spark.

Here are the current issues:

- The volume does not require a solution that is this complex (we’re talking 100Gb max accumulated over the past 4 years)

- It’s a mess: every time we onboard a new person we have to spend several days explaining where the data is.

- There is no simple way to explore the data.

- Data and code end up being duplicated: people working on several projects that require the same subset write their own transformation pipeline to get the same results.

Am I the only person here who finds it completely insane?

I was thinking about building a pipeline to dump the raw data in a Postgres and then build other pipelines to denormalize and aggregate the data for each project. The difficulty with this, and any data science project is to find the sweet spot between data that is fine-grained enough to allow to compute features, but fast enough to query to train models. I was thinking that in a first iteration, data scientists would explore their denormalized, aggregated data and create their own feature with code. As the project matures we could tweak the pipeline to compute the features. Do you have any experience with this?

Finally, I love data science and I really don’t want to end up being the person who writes pipelines for everyone. Everyone else is a consultant, and they don’t have any incentive to care about the long-term impact of architecture choices: their management only evaluates delivery (graphs, model metrics, etc.). How do I go about raising awareness?

Without much info to go as to how the infrastructure is complex, the first thing I would suggest is to get rid of operations overhead by going with a hosted service. AWS EMR, Databricks, Qubole, etc. offer a service with S3+Hive+Spark.

The reason for using the tools might not just be volume. It could also be due to vicissitude (large number of different data sources using spark for pre processing), or scaling, or something else. Try to understand why the existing solution is used before planning a migration.

If you are determined to go with a new pipeline, build a prototype for a small subset of data science tasks you have and carefully evaluate the pro and cons vs the existing approach for that subset.

If you can quickly set up a simple version to test with any of the rest of the team that are willing, that'd be my first port of call. A simple ETL job to pull in fresh data from your S3 store and push it into Postgres. Then you can hopefully get everyone else on board with how much easier this setup will make their lives, and build a consensus that this migration would be a good idea.

A quick idea re. Postgres and 'quick enough' vs. 'fast enough' - what are your requirements around data freshness for the models. Can you take data from a read replica or even your DB backups (extra points for letting you test your backups) for your model training workloads to keep load off the main instance?

W.r.t. the 'It takes several days to explain where all the data is' - in the first instance, I'd draw up a shared spreadsheet listing all this stuff. Yes it's best case going to be eventually consistent with the actual data you hold, but it should cut down the time spent regurgitating the same information over and over and gives everyone a central point to store it. This will be much easier and faster to implement than a full blown data catalog, which you could look at doing once you've got the 'duct tape' solution going.

If you want to chat, email is in profile.

Its manifestation in the data world, from an awareness perspective might benefit from similar solutions. I imagine there is plenty of advice about that. Personally I think refactoring is not a management concern - why should they care, and why should you expect them to understand? It's a technical problem and so I think you need to convince your fellow techies of the benefits to them. What might they be?

If you don't have a data engineering function, then one way to promote a better organisation is to make each data scientist an expert in one subset of the data (if that is possible). Then it becomes their responsibility to service any data requests, and they will automate and refactor when that responsibility becomes onerous, and not before, which should be a good way to self regulate your resources.

I have used standards and conventions within data platforms that I have written, but I feel that they are not as important as discovering what you need for your particular job and situation. It seems you have a handle on main factors to consider. Of the top of my head the various tradeoffs are to do with computation time, storage space, code complexity, infra costs, costs of refactoring, future proofing etc. And if in doubt, use the lean and agile approach!

You want Sales - logon here, select [this] from [that] You want forecasts - email the angry VP and ask for the latest spreadsheet

Once you know where things are - then you can think about rebuilding as you need.