Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is no data engineering roadmap

alasdairb.com

Post reply on HN
Print view View original

91–100 of 133 posts

Re: There is no data engineering roadmap

#91

Post by blowski »

(Summary of this article):

> Data engineering career roadmaps are a waste of time because they're full of things you don't need to learn. Here instead is an over-simplified data engineering career roadmap.

Roadmaps are useful when there's a lot you could learn, you're not sure you have a complete list, a lot of it isn't essential, you don't know what order to do it in, and you don't know what resources to learn from. Many juniors are in that place. To get into a company where they can learn on-the-job or from a mentor is extremely difficult. So a good roadmap is a pretty good alternative to that.

Saying "there cannot be a roadmap" or "I didn't need a roadmap, neither do you" is unehlpful at best, damaging at worst. To be an effective junior data engineer needs more than just knowing SQL.

Re: There is no data engineering roadmap

#92

Post by baazaa »

xmcqdpt2 wrote: post #86
baazaa wrote: post #70

The lack of a roadmap is disastrous in terms of professional quality IMO. A lot of orgs have data scientists who are good at coding and maths who can't do their work because the data engineers, who got their jobs because they kinda knew SQL a decade ago, are incompetent. I'm sure this is less of a problem in tech companies which are skimming the cream of the crop, but in other organisations it's apparent that a third…

I've been in such an org. The problem is that few decent coders want to do data engineering work, because they'd rather build software and because data engineering pays less and is kind of a dead end in a lot of org. You end up with a lot of DBAs that self-style as Data Engineers.

I'm in Australia and data engineers here are overpaid, afaict slightly better paid on average than software engineers who are better paid than data people.

So you end up with bizarro stuff where a $150k AUD data engineer who basically can't code or even set-up FTP telling some maths PhD data scientist on $100k what he can or can't do. Managers, unable to gauge technical ability, assume the data engineer must be highly skilled because they're getting paid so much.

The wages and lack of skill I think are symptomatic of there being no real entry-point in data eng. There's a very slow osmosis of people from genuinely skilled areas into data eng, the main drawback being that the entire culture of data eng is defective with so many unskilled hacks.

Re: There is no data engineering roadmap

#93

Post by sanderjd »

_bk90 wrote: post #54
sanderjd wrote: post #38

Earlier quoted context omitted.

I would certainly prefer it if that were the case, but it largely is not, in my experience.

It seems like your experience doesn't overlap with data engineering, in that case.

Ok. I think this is just going to end up being "no true scotsman". I'll say "in many companies, data engineers are more doing BI data preparation and presentation than building distributed processing infrastructure or whatever and they are compensated more like analysts than general software engineers" and you'll say "well that's not data engineering".

My argument is not an "ought" argument, it's an "is". I agree with you that data engineering "ought" to be, if anything, a more difficult specialized practice of software engineering, because you need all the software engineering skills and then also specialized data skills. But I'm saying that what it often "is", is more like a specialization of BI and analyst roles.

I think the OP is a pretty good demonstration of this. If "all you need is SQL", does that sound more like a specialization of a software engineer skillset or an analyst skillset? I think the latter... And sure you can say, "well the article is wrong, if all you need is SQL, that's not data engineering", but we're just back to "no true scotsman"; I believe it is common to see the role this way.

Re: There is no data engineering roadmap

#94

Post by eatonphil »

BiteCode_dev wrote: post #85
Ntrails wrote: post #75

Earlier quoted context omitted.

> It keeps them in the loop, give them context and perspective, and help them take better decisions Other side of the coin: "I just spent 2 weeks optimising a daily process to be twice as fast" might well end up in me wondering why anyone would care about the 1 minute reduction in delivery time for daily cadence data. It might also result in me being stroppy about work I cannot get resourced that has obvious $ value.…

That's good feedback to have. If your boss/client step in and says "we don't need that", this allows you to re-calibrate the team priorities and shows you need to get better at extracting the real need in the future.

I completely agree with your initial comment and your response here. Transparency is a great feedback loop. And it's always good to celebrate what you do.

Re: There is no data engineering roadmap

#95

Post by eek2121 »

rectang wrote: post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

I personally don't have an issue with SQL. I started my career a couple decades ago as a SQL guy and I still use it occasionally.

If you wanted to, you can write tests for the queries. I've done so in the past.

Re: There is no data engineering roadmap

#96

Post by parpfish »

> People used to say “Software Engineering is not an entry level role”.

Something that I ran into myself and have seen other data-people struggle with is that it is hard to get good experience working with data when you don't have access to data.

Online SQL tutorials and classes typically have pretty clean/small toy datasets that are only really useful for teaching you the correct syntax. Going from these simple queries that are all ~10 lines to a production system with queries that are 100s of lines is jarring.

Re: There is no data engineering roadmap

#97

Post by hulahoof »

baazaa wrote: post #92
xmcqdpt2 wrote: post #86

Earlier quoted context omitted.

I've been in such an org. The problem is that few decent coders want to do data engineering work, because they'd rather build software and because data engineering pays less and is kind of a dead end in a lot of org. You end up with a lot of DBAs that self-style as Data Engineers.

I'm in Australia and data engineers here are overpaid, afaict slightly better paid on average than software engineers who are better paid than data people. So you end up with bizarro stuff where a $150k AUD data engineer who basically can't code or even set-up FTP telling some maths PhD data scientist on $100k what he can or can't do. Managers, unable to gauge technical ability, assume the data engineer must be highl…

I work in the aus data space, I’ve met a lot of intelligent PHD data scientists, but very rare is it to meet one who can scale their work efficiently across the org. The amount of time wasted on models that never make it to production is ridiculous.

Re: There is no data engineering roadmap

#99

Post by parpfish »

nevi-me wrote: post #77

I view data engineering as a hybrid of SQL and programming. I'd go as far as saying that it's a mixture of various disciplines. In the end it depends on what you'd be doing as a data engineer. My experience transitioning into the role/ecosystem in the last decade (I'm only 33) + hiring data analysts + scientists + engineers has left me with this impression: Data engineering isn't so much an entry-level career, simila…

> It was a lot of night-time and weekend tinkering to get to where I am.

I think that this gets at the difficult part of hiring junior candidates for any role -- you want the people that are capable of learning a new thing if they're given some time to learn, but it's hard to find a signal for 'capable of and willing to learn'.

Re: There is no data engineering roadmap

#100

Post by pixl97 »

eek2121 wrote: post #95
rectang wrote: post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

I personally don't have an issue with SQL. I started my career a couple decades ago as a SQL guy and I still use it occasionally. If you wanted to, you can write tests for the queries. I've done so in the past.

At the end of the day with SQL, is what are you testing....

I support an application that does authentication and does URL/component registration based on the IP/hostname/FQDN of the machine at startup. It shoves some of this stuff off into a table for URL validation as requests might come in from a different node but need to verify that host was part of the cluster.

In client systems this start up was almost always insanely fast, until one day for one of our clients it wasn't. Startup process was in a restart loop with a timeout where I had never seen one before. Started it in debug mode without a timeout and it was taking minutes to start up. Look at the SQL side and this thing is just processing more IP records than I'd ever seen in the app and the registration table size was huge.

Turns out this machine had went though some massive number IPv6 addresses in a very short period of time. The way the SQL was written for this query wasn't great, but never needed to be because most instances of this software may have 20 records in the table at most. That's the one 'fun' thing about SQL that's hard to test for.

Post reply on HN