Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is no data engineering roadmap

alasdairb.com

Post reply on HN
Print view View original

61–70 of 133 posts

Re: There is no data engineering roadmap

#61

Post by benrutter »

I think the idea of a data engineering roadmap is flawed because there isn't a single cookie cutter "data engineer" role to aim for, just like if wouldn't make sense to talk about a single route into becoming a "software engineer".

I'm a data engineer but spend most of my time writing code in dask, and use fairly little of SQL. I've been in roles before though where long and complex SQL routines was the main focus.

My advice that nobody asked for: focus on 1) general programming skills which are transferable anywhere and 2) do what you personally find interesting. Like SQL? Sweet plenty if people will pay to do that. Obsessed with using scala and spark in combo? Someone'll probably pay you to do that too.

Don't focus on being a "data engineer" or any other label- just enjoy programming and find roles that have more of what makes you happy.

Re: There is no data engineering roadmap

#62

Post by benrutter »

rectang wrote: post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

I see a lot of replies saying something to the effect that "SQL is actually really testable, people just don't do it" which I'm sure is true, but probably less helpful if you're having to work on a tech stack that doesn't offer unit testing (I'm thinking something like Azure Data Factory or even some setups/configurations of airflow).

Would love to hear from anyone out there who has a good test cycle for predominantly SQL based code? What tools are you using? How does the process work out for you? What are some pitfalls you found along the way?

Re: There is no data engineering roadmap

#63

Post by benrutter »

claytonjy wrote: post #6

I agree with so much in this article: there's no narrow path, SQL is king, and enough with the damn influencers. But how many entry-level DE jobs will hire folks without programming experience? I've never heard of such a thing in the US, perhaps the UK market is different? If you're new to DE you're unlikely to have any experience managing a database or using common ETL tooling, so your transferable skills are basica…

I've worked in a big company that had explicit "junior data engineer" roles, but even those would expect some programming experience - either through a bootcamp, degree or related role like data analysis.

But basically yeah, as a UK data engineer, I'd say what you've said is pretty much right on- programming language + SQL will get you hired much more than "I've used this ETL tool"

Re: There is no data engineering roadmap

#64

Post by Simon_O_Rourke »

Here's the thing with data engineering, if you do an absolutely superb job you end up being largely invisible to the business. Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected.

The issues I encountered working as a data engineer many moons ago, is that in a business that's either agnostic to data quality or doesn't invest in reliability, the data engineers tend to become the face of failure. If a Kafka queue begins acting up, or a data ingestion jobs fails because of some upstream data source changes, then it's usually up to the data engineer to...

1. Notice that something's gone wrong.

2. Figure out who in the organization is going to be affected and let them know.

3. Run any triage and comms with the business keeping them in the loop until it's fixed.

In my experience, like the guy who only gives good news to the dictator, certain execs/leadership would quite happily let data engineers take the lead for outages, only swooping back in once everything was up and running again to demand a report on "how this happened" and some glib advice to "take time to stop this happening again".

While it can be technically rewarding as a career move, it's got to be highlighted that it's only fun and games until you hit your first major outage.

Re: There is no data engineering roadmap

#65

Post by code_biologist »

benrutter wrote: post #62
rectang wrote: post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

I see a lot of replies saying something to the effect that "SQL is actually really testable, people just don't do it" which I'm sure is true, but probably less helpful if you're having to work on a tech stack that doesn't offer unit testing (I'm thinking something like Azure Data Factory or even some setups/configurations of airflow). Would love to hear from anyone out there who has a good test cycle for predominantl…

I gave a positive review of dbt's testing capabilities in this other comment: https://news.ycombinator.com/item?id=36720013

This is what the actual generic tests look like. There's also support for one-off SQL-based tests: https://docs.getdbt.com/reference/resource-properties/tests#...

This article covers a lot of ground as to how to actually use dbt tests in practice. I'm particularly glad they mentioned the dbt-utils and dbt-expectations libraries: https://www.datafold.com/blog/7-dbt-testing-best-practices

The dbt ecosystem is also a strength. This article has good examples of the type of tooling and libraries in the ecosystem: https://meltano.com/blog/top-5-of-dbt-packages-and-tools-in-...

I have no affiliation with any of the above linked companies, just a happy dbt user! I'll also add that I don't use the dbt cloud, just invoke the dbt cli in a backend task runner.

Re: There is no data engineering roadmap

#67

Post by makeitdouble »

Simon_O_Rourke wrote: post #64

Here's the thing with data engineering, if you do an absolutely superb job you end up being largely invisible to the business. Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected. The issues I encountered working as a data engineer many…

TBF, in most companies that have a traditional core business (let's say, selling pharma products), IT is a cost center and your run of the mill engineer is seen as the plumber who's suppose to make everything seamless, as you describe.

Short of solving a pain point in a spectacular way, the whole IT dept will be the face of stuff crashing, security issues and budget burning.

Re: There is no data engineering roadmap

#68

Post by scary-size »

Simon_O_Rourke wrote: post #64

Here's the thing with data engineering, if you do an absolutely superb job you end up being largely invisible to the business. Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected. The issues I encountered working as a data engineer many…

This hits home. I communicate a lot more with up and downstream stakeholders as a data engineer. Be it for designing future changes and how those affect others. Or, as you said, triaging outages/delays, figuring out where the missing dependency is. Less problem solving by coding.

The SQL thing the article mentions is also true. Even if you work with higher level tech like Spark. SQL gives you a good feel for joins. I also like the fast feedback cycles you get from SQL compared to something Scala/Python based.

Re: There is no data engineering roadmap

#69

Post by ghiculescu »

kozikow wrote: post #58
ghiculescu wrote: post #34

Earlier quoted context omitted.

i think you're probably onto something, what do you think the equivalent to SQL would be in product management?

Jira? Powerpoint?

Nah. Being good at Jira makes you worse at product management.

Re: There is no data engineering roadmap

#70

Post by baazaa »

The lack of a roadmap is disastrous in terms of professional quality IMO. A lot of orgs have data scientists who are good at coding and maths who can't do their work because the data engineers, who got their jobs because they kinda knew SQL a decade ago, are incompetent.

I'm sure this is less of a problem in tech companies which are skimming the cream of the crop, but in other organisations it's apparent that a third-rate data scientist or software engineer is probably much better at their job than a second-rate data engineer. And indeed if I were running an IT dept I'd probably just fill the data engineering roles with decent coders and tell them to reskill a bit.

Post reply on HN