Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is no data engineering roadmap

alasdairb.com

Post reply on HN
Print view View original

51–60 of 133 posts

Re: There is no data engineering roadmap

#51

Post by chrismarlow9 »

SOLAR_FIELDS wrote: post #46
bigger_cheese wrote: post #35

Earlier quoted context omitted.

>a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform data and present it to end users for analysis (BI) in such a way that it’s not painful or arduous to consume it, and ensuring that it is clean and accurate 100% this. I basically work in this role in m…

One really interesting aspect of this that I’ve had some revealing conversations with the data engineers in my org is how to best expose the firehose of data for people in BI tooling. We use dbt in my org, and Metabase as the BI tool, and a lot of thought is put into how to create a clearinghouse that serves the needs of the organization. The current pattern that has been of interest is to ELT into what the data engi…

I like this and I think it's where modern AI will shine the most. Like a clippy but for data.

The (outside of this scope) question is what happens when you feed that decision back into the system. I think the "recursive AI" question has been exhausted though.

Re: There is no data engineering roadmap

#52

Post by quickthrower2 »

alexleeds wrote: post #4

I was going to say, ‘Bullsh*t Learn SQL.’ Then I read the post.

Then learn what sequence of actions your data store(s) takes based on that SQL. Relational engines are great but can need tuning (add indexes, de/normailze, materialized views or not) and then key/value ones are going to be doing scans from the entire cluster generally unless you get the cluster key in there and ideally the primary key too.

Re: There is no data engineering roadmap

#53

Post by di456 »

bjornasm wrote: post #19
di456 wrote: post #12

Earlier quoted context omitted.

It is important, but when the shift to cheap compute and storage hit an inflection point it became possible to build out wide tables that combine the characteristics of facts and dimensions, and real data modeling took a back seat. That possibility led to bias toward moving quickly with less emphasis on making the data model sustainable. I'm seeing the pendulum start to swing the other way, where the complexity of th…

Interesting, this makes me think of a process I am going through, where I have a couple of very wide tables where I feel a need to build fact tables simply to get a better understanding of the data and the domain they are from. I have already stumbled a couple of times as the groupings was not as expected at all. Think this was the last push for me to build those fact tables once and for all, to get an overview and a…

Sounds like you are on the right track.

A non-technical benefit is there won't be so much context to keep in your head when triaging data issues in the future.

A database table is very similar to the separation of concerns problem of core software engineering. A table really just is an abstraction for storing and fetching various data points.

Finding the right balance is part art and part science.

Re: There is no data engineering roadmap

#54

Post by _bk90 »

sanderjd wrote: post #38
_bk90 wrote: post #30

Earlier quoted context omitted.

A data engineer is as much a software engineer as a back-end engineer or front-end engineer and paid equivalently.

I would certainly prefer it if that were the case, but it largely is not, in my experience.

It seems like your experience doesn't overlap with data engineering, in that case.

Re: There is no data engineering roadmap

#55

Post by appplication »

rectang wrote: post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

DE is an inherently less tame domain than traditional SWE. The ground is more likely to shift under your feet. IMO the real problem with testing data engineering is that the data changes at a cadence that is independent to your version control/CI/CD cadence. You can have airtight unit and integration tests that get wrecked when the data schema changes without warning, the distribution of the data shifts, etc.

And you can certainly say “I’m going to enforce x rules on ingest and if it doesn’t match it doesn’t pass”. Depending on your system you may end up with a lot of on call or self-enforced down time or data loss. You might white list a schema and then later find out you’ve been missing out on 3 months of high value data because your upstream provider added a new columns. Alternatively, you can let things float through, monitor changes, but don’t let it plug up your system. This is also a risky game to play, with its own set of downsides.

Striving for comprehensive testing is still super important though. I find probably 60-80% of my time is spent tooling test frameworks that let us address the various edge cases. We use pyspark, and the amount of test-oriented tooling around that is, at least to me, surprisingly underdeveloped.

Re: There is no data engineering roadmap

#56

Post by jochem9 »

atomicnumber3 wrote: post #37
vjust wrote: post #32

Earlier quoted context omitted.

You made great points. Data Engineering cannot claim to support test-driven development to half the extent of the rest of software engineering. Frequently DE projects have no concept of Dev, Test, Staging and then Prod. Quite often its dev and then straight to Prod. SQL of course is to partly or fully blame for this. My last job for an large insurance company, they happily set a best practices of 90% test coverage (w…

In my experience another big problem is that it's just expensive. For Spark jobs for instance, it's very common for it to run fine on the small test dataset on your laptop but then when you release it to run for 2 days on the prod dataset you end up having to do tuning there with pretty long turnaround times. And that then extends to why staging and even dev aren't very useful - the scale is part of the equation. Eve…

Writing performing transformation code is one of the critical skills that a data engineer needs to master. It's a combination of experience with and knowledge about the underlying technology. How it actually processes data (e.g. learning how database pages work or going beyond dataframes and getting hands on experience with Spark RDDs).

Re: There is no data engineering roadmap

#57

Post by jvans »

bigger_cheese wrote: post #35
SOLAR_FIELDS wrote: post #15

Earlier quoted context omitted.

Interesting. I work in the space currently (an ELT company) and I wouldn’t characterize the skill sets you mentioned as critical to a data team or a data driven organization. They are helpful, sure, but a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform…

>a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform data and present it to end users for analysis (BI) in such a way that it’s not painful or arduous to consume it, and ensuring that it is clean and accurate 100% this. I basically work in this role in m…

I imagine the number of people who fit the venn of data engineer with signal analysis knowledge/skills is pretty small. Is hiring for that kind of role difficult? How would you recommend people with skills in SWE/DE become hireable in the signal processing world?

Re: There is no data engineering roadmap

#58

Post by kozikow »

ghiculescu wrote: post #34
phillipcarter wrote: post #5

Really enjoyed this article. Made me feel a tad inspired to write a similar one, "There is no Product Management roadmap", tailored for the many people looking to break into that discipline. Loooots of parallels.

i think you're probably onto something, what do you think the equivalent to SQL would be in product management?

Jira? Powerpoint?

Re: There is no data engineering roadmap

#59

Post by code_biologist »

appplication wrote: post #55
rectang wrote: post #7

> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…

DE is an inherently less tame domain than traditional SWE. The ground is more likely to shift under your feet. IMO the real problem with testing data engineering is that the data changes at a cadence that is independent to your version control/CI/CD cadence. You can have airtight unit and integration tests that get wrecked when the data schema changes without warning, the distribution of the data shifts, etc. And you…

I've recently done my first implementation of dbt and though its fundamental Jinja+yaml nature does show through a bit, I was really pleasantly charmed by the data testing. The catalog of data assertions (dbt-utils and dbt-expectations really fill out the bare bones default tests) is generic across data sources, intermediate tables, or output tables, and is typically 3-5 lines of yaml for any given test. Hits all the basics: nullness, uniqueness, freshness, value cardinality, value distribution, row count, foreign key validity, etc.

I still don't like the yaml completely, but it makes adding data assertions so easy there is no excuse not to.

This project is pretty black and white, but I'm glad dbt supports test failure severity to lower on call load (error vs warning) when I have the need.

Just wanted to add a positive review for dbt, that it made it fun to do better data testing!

Re: There is no data engineering roadmap

#60

Post by code_biologist »

SOLAR_FIELDS wrote: post #46
bigger_cheese wrote: post #35

Earlier quoted context omitted.

>a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform data and present it to end users for analysis (BI) in such a way that it’s not painful or arduous to consume it, and ensuring that it is clean and accurate 100% this. I basically work in this role in m…

One really interesting aspect of this that I’ve had some revealing conversations with the data engineers in my org is how to best expose the firehose of data for people in BI tooling. We use dbt in my org, and Metabase as the BI tool, and a lot of thought is put into how to create a clearinghouse that serves the needs of the organization. The current pattern that has been of interest is to ELT into what the data engi…

Have you evaluated Superset or Lightdash against Metabase? If so I'd love to hear about your experience. I'll shortly be helping a client company migrate BI off Looker and haven't gotten my hands dirty with the options yet.
Post reply on HN