Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is no data engineering roadmap

alasdairb.com

Post reply on HN
Print view View original

81–90 of 133 posts

Re: There is no data engineering roadmap

#81

Post by camgunz »

xmprt wrote: post #22
rectang wrote: post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

> Data engineering can be unsatisfying if you thrive on writing reliable systems This is what I needed for me to stop considering data engineering. Not that I was thinking about it a lot. I feel very happy with my current job but the thought of the code I write being so brittle as to paralyze me from making any changes terrifies me.

We build pipelines in staging schemas, and we only promote them to prod once they pass tests. This is pretty analogous to SWE (it's where we got the idea) and is a pretty crucial part of us meeting the requirement where we don't put inaccurate data in front of customers. It's--of course--only as good as the tests, but that's every reliable system.

Re: There is no data engineering roadmap

#82

Post by 2-718-281-828 »

Simon_O_Rourke wrote: post #64

Here's the thing with data engineering, if you do an absolutely superb job you end up being largely invisible to the business. Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected. The issues I encountered working as a data engineer many…

> if you do an absolutely superb job you end up being largely invisible to the business

openly communicate how difficult some feature, project or solution is going to be. estimate time then multiply with pi and round up. then proudly present how you got it done despite obstacles possibly even in less time.

also good - create problems - then solve them.

Re: There is no data engineering roadmap

#84

Post by infosecb »

Simon_O_Rourke wrote: post #64

Here's the thing with data engineering, if you do an absolutely superb job you end up being largely invisible to the business. Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected. The issues I encountered working as a data engineer many…

> Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected.

Sounds a lot like Cybersecurity.

Re: There is no data engineering roadmap

#85

Post by BiteCode_dev »

Ntrails wrote: post #75
BiteCode_dev wrote: post #74

Earlier quoted context omitted.

This is a lesson I keep repeating to our juniors: when you do something notable, celebrate it, and make sure to include the whole team, the users and the boss in the celebration. It can be as simple as an email, but sometimes, making a few chat messages or even calls. It takes times, it seems silly, but you get a fantastic ROI on this. Because now people see your work as valuable. It's concrete to them. They understa…

> It keeps them in the loop, give them context and perspective, and help them take better decisions Other side of the coin: "I just spent 2 weeks optimising a daily process to be twice as fast" might well end up in me wondering why anyone would care about the 1 minute reduction in delivery time for daily cadence data. It might also result in me being stroppy about work I cannot get resourced that has obvious $ value.…

That's good feedback to have. If your boss/client step in and says "we don't need that", this allows you to re-calibrate the team priorities and shows you need to get better at extracting the real need in the future.

Re: There is no data engineering roadmap

#86

Post by xmcqdpt2 »

baazaa wrote: post #70

The lack of a roadmap is disastrous in terms of professional quality IMO. A lot of orgs have data scientists who are good at coding and maths who can't do their work because the data engineers, who got their jobs because they kinda knew SQL a decade ago, are incompetent. I'm sure this is less of a problem in tech companies which are skimming the cream of the crop, but in other organisations it's apparent that a third…

I've been in such an org. The problem is that few decent coders want to do data engineering work, because they'd rather build software and because data engineering pays less and is kind of a dead end in a lot of org.

You end up with a lot of DBAs that self-style as Data Engineers.

Re: There is no data engineering roadmap

#87

Post by nologic01 »

You cant have a roadmap if you dont have a destination, or at least a direction for travel for the immediate next steps.

Remarkably the article doesnt event try to define data engineering, so how you can possibly have a roadmap to it.

"SQL" is too vague and jargonistic. Could be just "data analyst". The Engineering aspect is supposed to add a more fundamental data flow approach, less business context driven and more close to the metal. This is the skillset of big data framework (streaming data) nosql etc.

We could say it puts more focus on performance, but even that is imprecise because a lot of SQL expertise is also about performant designs and for many people would claim that e.g graph DB's are not essential

Re: There is no data engineering roadmap

#88

Post by BiteCode_dev »

tropianhs wrote: post #78
BiteCode_dev wrote: post #74

Earlier quoted context omitted.

This is a lesson I keep repeating to our juniors: when you do something notable, celebrate it, and make sure to include the whole team, the users and the boss in the celebration. It can be as simple as an email, but sometimes, making a few chat messages or even calls. It takes times, it seems silly, but you get a fantastic ROI on this. Because now people see your work as valuable. It's concrete to them. They understa…

Or just do this 1) Break something on purpose. 2) Make sure the business notice it (usually a spectacular cost increase does the job). 3) Come in and fix the issue 4) Make a cost comparison quick chart before/after 5) Become famous

You don't need to do it artificially.

If you look long enough in most code bases, you will eventually find a spectacular problem.

Case in point, I was working for several months for a client, and I had a slow afternoon, so I decided to convert some calculation to numpy, see if we gained any free perf.

We got a x100 local speed up, which was very fishy. Gaining speed is common with vectorization, but two orders of magnitude is a lot.

So I looked at the original algo. There was a glaring mistake in it, that I fixed by my numpy code without noticing, and just changing that in the pure Python algo, without using numpy at all, made it X50 faster.

I could then call my client, and celebrate the good news. Not "there was a mistake", no. But "we found margin for progress".

Re: There is no data engineering roadmap

#89

Post by steveBK123 »

Simon_O_Rourke wrote: post #64

Here's the thing with data engineering, if you do an absolutely superb job you end up being largely invisible to the business. Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected. The issues I encountered working as a data engineer many…

This is exactly why I found DE is not a long term career path. I work in data, I occasionally do DE roles, but if you aren't moving over to the app side or business/domain side, you are leaving a lot of money on the table long term.

The funny thing, being in data, if you take the time to understand it.. you are tremendously valuable to the business/domain side.

Otherwise you 1000% risk becoming part of the background infra. No one celebrates when the faucet dispenses water or the power switch works. They do lose their collective #$#@ when the water or power fails though. Data Engineering has a tendency to be treated that way.

DQ is a big part of it as well. Everyone wants it, no one wants to pay the hours/dollars to do it.. No one is sure what they want, but probably what you propose is too simple. Can't AI do it? Etc.

Re: There is no data engineering roadmap

#90

Post by aerzen »

rectang wrote: post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

> joins being logically correct

What do you mean by this?

Post reply on HN