Ask HN: How do I improve our data infrastructure?

Parquet also supports much better access mechanisms, like being able to deserialize a single column without having to read in entire rows.

But like you mentioned, 1TB of data in a traditional database isn't that bad.

I know exactly how you feel. I was put in the same position and ended up spending 2 years trying to clean up the data and set up the warehouse and handle all the requests for exploring the data.

I had a background in AI and was fully blocked on doing anything I wanted.

4 Years later I build a company to make this easy for people.

I would do the following: 1) Setup a Redshift Instance 2) Use Fivetran to Dump all the data into that Redshift cluster. 3) Leverage my startup Narrator.ai to model and use that data.

Now your team can use your modeled data and you have a clean time-series data structure to do the DS algorithms you want.

Reach out and we can talk about this in details ahmed@narrator.ai.

[...]

> I was thinking about building a pipeline

[...]

> How do I go about raising awareness?

Don't.

You don't get it yet.

Everything there is there for a reason.

Some of it may be technical reasons, some of it may be delivery reasons, some of it may be people reasons, team reasons, political reasons, etc.

You just landed a sweet perm job in a field you love. Don't ruin it by becoming that cliched new hire that sees all their problems and knows how to make it all better. You may be right. Technically. They may even encourage you.

But you could also be wrong technically. You could step on toes politically and end up sidelined. You could end up biting off more than you can chew and end up becoming responsible for the bigger mess later.

May advice is to stop, take a deep breath, look around, appreciate what you've already achieved by getting there, get to know your colleagues, get to know the company, get to really understand the system so when the next new hire comes in you can explain the reason behind everything (maybe getting to know the detailed history of the system and why it formed the way it did?), and make sure you know absolutely everything you can before changing everything you can.

Yeah, maybe you'll feel some parts of your job suck for a while (I wish this was easier, it's stupid that I have to do all this work to get form A to B when I can see a better way), but if you give yourself more time to learn, you're really doing yourself a favor in the long run.

And you're a perm now, in a big corp, doing data science. Relax, you got it made, right?

So chill out and take a deep breath and enjoy your new workplace and everything about it (not just the stack in front of you), and if you still want to make changes somewhere down the line, start small, bit off a tiny little piece you can chew, and succeed with that small improvement before moving forward with anything more.

Also think of it differently, if you do end up being the one responsible for reinventing the whole stack, then milk that project for everything you can. It's a big corp play so you have to do that in a big corp way. There'll be meetings, committees, decisions, stakeholders, teams formed, responsibilities. You could even parlay this project into some greater responsibility and title for yourself, maybe even use it to boost your career. SO think of it not like you are trying to understand a technical problem, but you are trying to understand a piece of (and through it, the enitrety of), your whole new organization, with all that entails: the people, the team relationships, how decisions get made, etc. So enjoy playing that game, because you are happy to be in a big corp, so the sort of benefits that can bring you is what you want, right?

It's not a startup. And if you feel technically unsatisfied, use the time to learn some new languages or skills, or kick some side projects down the road for your own benefit.