Data Engineering Design Patterns
dedp.online
Data Engineering Design Patterns
1–10 of 50 posts
Re: Data Engineering Design Patterns
#2DAMA-DMBOK2 covers this very comprehensively
Re: Data Engineering Design Patterns
#3DAMA-DMBOK2 covers this very comprehensively https://www.dama.org/cpages/body-of-knowledge
Data engineering is cool and new while data management is old school and enterprise.
Specifically, data engineering in some tech companies is truly a revenue driver, so it makes data engineering in other organizations be viewed as a cost center so much, even if it is the same work at most organizations.
Re: Data Engineering Design Patterns
#4Could be interesting once there's more content, in its current state the content is mostly just definitions.
Re: Data Engineering Design Patterns
#5DAMA-DMBOK2 covers this very comprehensively https://www.dama.org/cpages/body-of-knowledge
Data engineering is cool and new while data management is old school and enterprise. Specifically, data engineering in some tech companies is truly a revenue driver, so it makes data engineering in other organizations be viewed as a cost center so much, even if it is the same work at most organizations.
How do you define the two terms?
Re: Data Engineering Design Patterns
#6The "please let me know" https://www.dedp.online/appendix/feedback> link at the bottom of the page gives an ugly Apache 404
heh, I would send feedback about that but ...
---
update: so, it seems the Feedback in the sidebar https://www.dedp.online/appendix/feedback.html> works so it was just a missing page extension. While reading that I discovered the GitHub repo for the project is private, which explains why I couldn't find it, either
Re: Data Engineering Design Patterns
#7Earlier quoted context omitted.
Data engineering is cool and new while data management is old school and enterprise. Specifically, data engineering in some tech companies is truly a revenue driver, so it makes data engineering in other organizations be viewed as a cost center so much, even if it is the same work at most organizations.
How do you define the two terms?
Data engineering versus data management?
Data engineering is nominally more pipeline oriented and less concerned with the governance & people side of things, but good data engineering people end up driving a lot of data management work because that's what makes the data engineering less painful (eliminate root cause of data errors and annoying data requests) and data overall more useful and valuable.
Re: Data Engineering Design Patterns
#8Earlier quoted context omitted.
Data engineering is cool and new while data management is old school and enterprise. Specifically, data engineering in some tech companies is truly a revenue driver, so it makes data engineering in other organizations be viewed as a cost center so much, even if it is the same work at most organizations.
How do you define the two terms?
Data Engineering is an engineering displine -- it can involve anything from data ingestion, transformation, storage, enrichment, aggregation up to presentation in operational reports. But it's still a manufacturing process with "data" as an input and "data" as its output.
Data Management is an organization discipline -- it is about how the enterprise manages data as an asset and how data is embedded in the organization. This includes data governance issues like common data models, and a chain of command (which person/role is responsible for which piece of data), but also second-tier data processes such as quality control and data valuation.
Re: Data Engineering Design Patterns
#9Could be interesting once there's more content, in its current state the content is mostly just definitions.
Author of the book here — 100%. The idea is to release early and update on the go. You might wonder if it's worth delving into the book in its current, unfinished form. I'd say it depends.
There's quite a bit of content around general DE knowledge. However, the anticipated design patterns are still in the works. Suppose that's what you're most excited about. In that case, you'll find the beginnings of exploring the patterns of `caching` and `ad-hoc querying` in the first Convergent Evolution chapter, but otherwise, you need to wait for more.
Re: Data Engineering Design Patterns
#10The "please let me know" https://www.dedp.online/appendix/feedback > link at the bottom of the page gives an ugly Apache 404 heh, I would send feedback about that but ... --- update : so, it seems the Feedback in the sidebar https://www.dedp.online/appendix/feedback.html > works so it was just a missing page extension. While reading that I discovered the GitHub repo for the project is private, which explains why I co…
Author of the book here — yes, I used Mdbook (https://github.com/rust-lang/mdBook), and the links strangely end with `.html. as you correctly updated. Initially, I wanted to change it but left it as it was.