There is no data engineering roadmap • Hacker News Board

Re: There is no data engineering roadmap

#101

Post by pixl97 » Fri, Jul 14, 2023, 1:39 PM UTC

parpfish wrote: post #96

> People used to say “Software Engineering is not an entry level role”. Something that I ran into myself and have seen other data-people struggle with is that it is hard to get good experience working with data when you don't have access to data. Online SQL tutorials and classes typically have pretty clean/small toy datasets that are only really useful for teaching you the correct syntax. Going from these simple quer…

And dealing with database sizes that have millions/hundreds of millions of rows.

Moving to a system where your indexes are larger than main memory and some really expensive queries may need to spill to temp tables if you don't keep your data sizes under control is also jarring. Big transactions getting deadlocks, and how to read said deadlock graphs and learn how to deal with said locking when you write your SQL starts to become a full time job.

Re: There is no data engineering roadmap

#102

Post by tm-guimaraes » Fri, Jul 14, 2023, 1:40 PM UTC

_bk90 wrote: post #30

sanderjd wrote: post #26

Earlier quoted context omitted.

The problem is that if you have all the skills of a software engineer, lots of people would pay you more to do different things than they will to do "data engineering", which is often a lot about cobbling different tools together and doing a bunch of custom transformations and validations that are more tedious than creative.

A data engineer is as much a software engineer as a back-end engineer or front-end engineer and paid equivalently.

Same as “game developer”

But that’s not how markets work.

Also a job title doesn’t really say much. On all of these 4 roles there’s people doing trivial work, as there’s people doing very deep technical work. Same with pushing the envelope on tech.

Re: There is no data engineering roadmap

#103

Post by maxlamb » Fri, Jul 14, 2023, 1:46 PM UTC

thenerdhead wrote: post #98

Data engineering reminds me of the George Carlin bit on how names change, but the things don't. Database management -> Data management -> Data architecture -> Database administration -> Big Data engineering -> Data engineering.

Database administration is more like a "sysadmin of database". IMO not equivalent to the other ones. Data Architecture is like software architecture, so more like high level system design. I do agree though that data engineering is equivalent to what data management referred to in the past.

Re: There is no data engineering roadmap

#104

Post by ambicapter » Fri, Jul 14, 2023, 1:58 PM UTC

infosecb wrote: post #84

Simon_O_Rourke wrote: post #64

Here's the thing with data engineering, if you do an absolutely superb job you end up being largely invisible to the business. Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected. The issues I encountered working as a data engineer many…

> Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected. Sounds a lot like Cybersecurity.

Cybersecurity jobs get paid very well, though.

Re: There is no data engineering roadmap

#105

Post by itsoktocry » Fri, Jul 14, 2023, 1:58 PM UTC

BiteCode_dev wrote: post #88

tropianhs wrote: post #78

Earlier quoted context omitted.

Or just do this 1) Break something on purpose. 2) Make sure the business notice it (usually a spectacular cost increase does the job). 3) Come in and fix the issue 4) Make a cost comparison quick chart before/after 5) Become famous

You don't need to do it artificially. If you look long enough in most code bases, you will eventually find a spectacular problem. Case in point, I was working for several months for a client, and I had a slow afternoon, so I decided to convert some calculation to numpy, see if we gained any free perf. We got a x100 local speed up, which was very fishy. Gaining speed is common with vectorization, but two orders of mag…

>I could then call my client, and celebrate the good news. Not "there was a mistake", no. But "we found margin for progress".

This is great framing.

Bugs in code are not really "mistakes"; they will occur. Finding and fixing them is a positive.

Re: There is no data engineering roadmap

#106

Post by phillipcarter » Fri, Jul 14, 2023, 2:05 PM UTC

ghiculescu wrote: post #34

phillipcarter wrote: post #5

Really enjoyed this article. Made me feel a tad inspired to write a similar one, "There is no Product Management roadmap", tailored for the many people looking to break into that discipline. Loooots of parallels.

i think you're probably onto something, what do you think the equivalent to SQL would be in product management?

It's a little more loosie goosie, but there's really no single tool. The intuition is that a good PM on a team helps guide things along at all stages such that when anyone looks back, they think "this never would have happened without them driving this". But the details of that aren't easily moldable into some framework or tool you can use for other projects, teams, or organizations.

Re: There is no data engineering roadmap

#107

Post by Ensorceled » Fri, Jul 14, 2023, 2:18 PM UTC

A data engineer's "hammer" may be SQL but the data engineer's JOB is gluing all that SQL together into a process that works efficiently and reliably.

You need to know how those other tools work together, whether it's DBT or Airflow or ... or ...

I do agree with ignoring "influencers" they are almost always trying to tie you into their stack or product and don't necessarily have your best interests at heart.

Re: There is no data engineering roadmap

#108

Post by yesiamyourdad » Fri, Jul 14, 2023, 2:21 PM UTC

Maybe it's just the positions I've ended up in, but my DE roles have required a TON of ops as well. You can build all these pipelines, but you will be absolutely buried in operational work if you can't get the automation squared away. With the role I've been in for the last 2 years, the most significant things we've done were done in Terraform, Ansible, and Prometheus. Sure, there's lots of SQL and Python, some Java, some Scala, but the biggest issue has been getting the system to where we get useful and actionable alerts and downstream data consumers get notified about late or missing data without requiring manual intervention from the data engineers.

Re: There is no data engineering roadmap

#109

Post by iblaine » Fri, Jul 14, 2023, 3:03 PM UTC

sanderjd wrote: post #26

iblaine wrote: post #13

The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data Engineers excel, I think, is to have all the skills of a software engineer + the ability to think like a data analyst. SQL is definitely part of that journey, and more skills should be mastered if you’re going to do things like use Kafka, airflow or spark.

The problem is that if you have all the skills of a software engineer, lots of people would pay you more to do different things than they will to do "data engineering", which is often a lot about cobbling different tools together and doing a bunch of custom transformations and validations that are more tedious than creative.

Agreed. The job title of Data Engineer gets abused. Some DE's are Analysts, others are Backend SWEs. Their goals are aligned, to create value out of data, and their methods may differ greatly.

Re: There is no data engineering roadmap

#110

Post by jmye » Fri, Jul 14, 2023, 3:04 PM UTC

Sterm wrote: post #49

jmye wrote: post #39

Earlier quoted context omitted.

They’re speaking to entry-level people, who will not be designing bespoke solutions. SQL is exactly the right recommendation.

In the same way that VB was to front end developers 20 years ago.

Sure, the language that has underpinned data work for the last few decades and has relevance regardless of what shiny framework you put on top of it is just like VB twenty years ago. Good comparison.