Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Engineering Design Patterns

dedp.online

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Data Engineering Design Patterns

#41

Post by thegabriele »

alexyz12 wrote: post #27

As a Data Engineer, I feel like a fixer. People think their data is incorrect and they ask me to fix it. It's similar to fixing someone's computer when they think its broken. Incredibly valuable but also somewhat unfulfilling in that you are rarely innovating/inventing like a SWE. There you are building tools that can change how people act/think. On the other hand, as a SWE/inventor you aren't desperately needed like…

I work as DE in a local medium business and since there's no giant all-in-one ERP my role is essentialy build solutions to improve (or create) analytics to act upon. It's working and even if i'm not at spaceX, i feel like i'm inventing something. But, since the size of it all..i also fix things (software and hardware).

Now, the problem is to convince other business that they might need my services.

Re: Data Engineering Design Patterns

#42

Post by articsputnik »

emlos wrote: post #23
maddynator wrote: post #22

Does anyone know what tech is used here to create the online book?

looks like mdBook https://github.com/rust-lang/mdBook

Yep, it is. Super happy with it. I added some custom themes and links to my second brain, but other than that, all mdBook. I also have more notes about writing a book with Markdown and self-publish at https://www.ssp.sh/brain/create-a-book-on-markdown/.

Re: Data Engineering Design Patterns

#43

Post by FranzFerdiNaN »

qsort wrote: post #14
JAlexoid wrote: post #11

It's interesting to note, that when I was first called a DE - it was just software engineer in the data domain. As in writing full software, that happen to focus on data. Just 6 years ago I would be tinkering with PrestoDB code, looking at optimizing the scheduler and building Hadoop extensions. Between that and today the field swung to people who came from BI, with considerably less software engineering background.…

The BI world is honestly kind of weird. You have people who are at the intersection of "understands databases, the relational model, query optimization etc. at the level of a very senior SWE" ∩ "needs to be told how git works in the year of our lord 2023".

Im in this post and i dont like it lmao.

Re: Data Engineering Design Patterns

#44

Post by free_flintshire »

fifilura wrote: post #39
opportune wrote: post #24

Earlier quoted context omitted.

A lot of “data engineers” are former db analysts and such that don’t know much of anything technically outside of SQL and even that might be something they only are “certified” to know rather than actually good at. It’s basically becoming a title I’d associate with being low-skill. I used to be a “software engineer in data” and never call myself a data engineer because people would think I don’t know how to write/mai…

Ok I guess you neew to find the worplace that suits you. Just that I have had some recent experience with muti threaded java k8s services reading daily "streams" from Kafka. When a SQL query would have solved it.

Many such cases in the data space of CV building over problem solving

Re: Data Engineering Design Patterns

#45

Post by _aaed »

moandcompany wrote: post #33

I used to give a lightning talk at Google called "Life of a Data Engineer" (~6+ years ago), where I tried to explain to Google Software Engineers and Product Managers the concept of a "Data Engineer" in other parts of industry, as well as Google Cloud Customers. In that talk, I outlined three (3) primary archetypes which still exist today: - Data Engineering for Business Intelligence (often a rebranding of yesterday'…

https://f.hubspotusercontent10.net/hubfs/2866765/Whitepapers...

So people don't have to give their details for the survey

Re: Data Engineering Design Patterns

#46

Post by fifilura »

free_flintshire wrote: post #44
fifilura wrote: post #39

Earlier quoted context omitted.

Ok I guess you neew to find the worplace that suits you. Just that I have had some recent experience with muti threaded java k8s services reading daily "streams" from Kafka. When a SQL query would have solved it.

Many such cases in the data space of CV building over problem solving

I guess you have to decide what you want on your CV. "I solve problems" or "I build complex stuff".

Re: Data Engineering Design Patterns

#47

Post by fud101 »

rockwotj wrote: post #29
icedchai wrote: post #25

Earlier quoted context omitted.

Don't forget the person who thinks they really have "big data" and needs all this massive infrastructure. Eventually, one discovers it's a few gigs of CSV files that fits in RAM on a laptop.

I feel like duckdb has been making waves here helping people realize you don’t need snowflake/bigquery/etc for all ypur datasets, but you can still get the nice feature set of those systems

What is duckdb

Re: Data Engineering Design Patterns

#48

Post by datadrivenangel »

fud101 wrote: post #47
rockwotj wrote: post #29

Earlier quoted context omitted.

I feel like duckdb has been making waves here helping people realize you don’t need snowflake/bigquery/etc for all ypur datasets, but you can still get the nice feature set of those systems

What is duckdb

Embedded column store database. SQLite but much much faster for aggregations.

Re: Data Engineering Design Patterns

#49

Post by anonu »

This is a pretty cool resource and lots of effort to put it together. I spent ten minutes reading the first few pages. I have to admit it's a bit hard to read. I would describe it as fairly verbose with lots of preamble before actually saying what you want to say. I also think this is very fixable with some editing and iterations. Just my 2 cents.

Re: Data Engineering Design Patterns

#50

Post by bomewish »

kozikow wrote: post #17
JAlexoid wrote: post #11

It's interesting to note, that when I was first called a DE - it was just software engineer in the data domain. As in writing full software, that happen to focus on data. Just 6 years ago I would be tinkering with PrestoDB code, looking at optimizing the scheduler and building Hadoop extensions. Between that and today the field swung to people who came from BI, with considerably less software engineering background.…

It's the least defined role. Currently, I am in a funny situation when all teams agree we need an additional data engineer. But basically: - Sales and finance want more of business intelligence analyst - Devs want more of a backend engineer - ML researchers want a data analyst proficient in ETL to do pipelines on the training dataset All of those 3 have only one thing in common - they need to know SQL very well. I've…

I’m puzzled by this. Knowing SQL seems like a minor technical matter. You can learn whatever you need to do (or use gpt4 to quickly do stuff). Why does one need the expertise in syntax rather than just a general strong foundation in software and data stuff?
Post reply on HN