Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is no data engineering roadmap

alasdairb.com

Post reply on HN
Print view View original

1–10 of 133 posts

Re: There is no data engineering roadmap

#2

Post by catsarebetter »

Now that I think about it you're right. All the maps for dev careers so far:

1. Staff eng map - https://staffeng.com/stories/

2. Bootstrappers - Indiehackers

3. Ecommerce - StarterStory

4. Software Engineering - /u/softwareengineering

5. Higher salary - Blind, reddit

6. Technical interviews - Leetcode forums

Looks like the shape of the solution can be forums, books, articles, what do you think you'll do?

Re: There is no data engineering roadmap

#6

Post by claytonjy »

I agree with so much in this article: there's no narrow path, SQL is king, and enough with the damn influencers.

But how many entry-level DE jobs will hire folks without programming experience? I've never heard of such a thing in the US, perhaps the UK market is different? If you're new to DE you're unlikely to have any experience managing a database or using common ETL tooling, so your transferable skills are basically SQL and python, and that's what every interview process is going to screen you for.

Re: There is no data engineering roadmap

#7

Post by rectang »

> All you need is l̶o̶v̶e̶ SQL

I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated.

Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which when it gets changed breaks in unpredictable ways. And the cultural traditions of testing and validation are underdeveloped because SQL has no core testing story — people count on joins being logically correct, but don’t go to the trouble to prove that true. Vector operations are just hard to get right in any ecosystem and if you don’t stay humble and apply skepticism to your work, it’s going to be wrong a lot.

I try to do stuff in pandas or similar — which has its own problems, but at least allows for testable library code. But in the field of data engineering, that’s swimming against the tide.

Re: There is no data engineering roadmap

#8

Post by chrismarlow9 »

Good article but a bit short. Anyway two books I can recommend for data engineering:

Designing data intensive applications-will give you a good overview of tools and the theories, algorithms, data structures behind it for different types of DB.

High performance browser networking-throwing this in because it can help extend the book above outside of a data center to the last mile. Sometimes you can save yourself a bunch of headaches on the data server side by doing things client side and caching/prefetching more than what the user asked for.

Re: There is no data engineering roadmap

#10

Post by jerrygenser »

rectang wrote: post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

I've done written heavily tested code in python. But also heavily tested code in SQL. When I was doing data engineering 5 years ago, dbt was a new tool and I used that to write tests, ci/cd, etc. in data engineering. It was a game changer.
Post reply on HN