Ask HN: How do I improve our data infrastructure?

Before recommending any solution, you need to ask yourselves the following:

1. Data ownership: do you own the data, or do other department rely on it ? Or worse, are the data customer data for which you need to guarantee some kind of clear audit trail and access control ?

2. What are the data ? Structured, semi-structured (Log data ), unstructured (images, sound, etc.).

3. What is the data for ? Analysis, training some models, viz ?

4. Can you put the data in the cloud, or do you need to store it on prem ? Questions to consider: data ownership, regulatory constraints, budget for cloud, IT quality in your company, etc.

5. Are the data write once, or are they often modified ?

Generally, I would try to create a single source of truth, but the difficulty would depend a lot on the answers of the above question. If the data are not often modified, then it is much easier to do it: you keep a single source of truth as whatever format is currently used, and you create a pipeline to create derived data (e.g. parquet/hdf5) as simple as possible first. You make sure that the derived data are RO if you can (technically and "politically").

This way you decouple the SST from the format used downstream, at which point you have much more latitude to improve things. If the SST format sucks, you can change it w/o impacting downstream users. You can also "export" the data into different format for different usages, including a DB which is indeed nice to build app/dashboards/etc. on top of. I avoid distributed platforms like the plague, especially if it is managed by the data team.

The difficulty of that decoupling phase depends a lot on the questions above. If you can use the cloud, you don't need an IT team, and backups are much easier to manage, as long as you have the budget for it (and the budget will be small for that amount of data). Another difficult is data consistency: you can often decouple w/o completely consistency (e.g. format consistency is enough, values consistency is not strictly required).

The choice of technologies is in my experience completely secondary to the problems above.

I work on an ELT process for something that's doing that about now on SQL Server, and not much query tuning is needed tbqh.

It's currently one step up from people running notebooks locally and having no shared space for the data.

I know of a local company whose data solution consists of dumping into Segment > S3 files > Pentaho (IIRC) > RedShift, and then using two different BI solutions, depending on the analyst. It needs two full-time data engineers just to keep it alive.

Now the funny part: a dump of their production database is less than 2GB and that isn't going to change any time soon: they don't make that much data to begin with, and their business model doesn't scale.

The argument used for building this new infrastructure is that users used to query directly into the production database and that would allegedly slow down their web app. So they decided they should take an "industry standard" path of handling data. C-levels were too afraid to "just use SQL" and instead asked "what is Amazon doing?".

It is an absolute mess and costed three months of the engineering team just to set up the application to generate the right events, but at least business people has access to data without having to stop an engineer in the hallway.

I don't think this will ever come back to bite anyone in an interview because the fact the dataset has less than 2GB will never come up: interviewers charitably assume that it wasn't overkill or that the person isn't padding the resume.

I frankly believe that a lot of places are like that. We criticize web developers all the time for over-engineering simple apps, but everyone is doing the same in other areas, we just can't see it like we do with web apps.