Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Become a Data Engineer in 2021

khashtamov.com

Post reply on HN
Print view View original

121–130 of 142 posts

Re: How to Become a Data Engineer in 2021

#121

Post by markus_zhang »

master_yoda_1 wrote: post #97

IMHO first you need to become a programmer then you can become a data engineer. So if you need to start by learning data structure then you are doing something wrong. Also the topics suggested in "Algorithms & Data Structures" could easily be skipped, the information is drastically misleading. We should seriously have some fact checker, otherwise this kind of bullshit article keep trending on HN and people keep wasti…

Can you recommend something else? I'm preparing to go through a full CS education.

Re: How to Become a Data Engineer in 2021

#122

Post by markus_zhang »

snidane wrote: post #85

In data engineering your goal is "standardization". You can't afford every team using their unique tech stack, their own databases, coding styles, etc. People leave the company all the time and you as a data engineer always end up with their mess which now becomes your responsibility to maintain. You'd at least be grateful if those people had used the same methods to code stuff as you and your team so that you wouldn…

Why would no one want to do it though? It's way more interesting than my current BA job. I managed to squeeze a bit into the role by opting to do some ETL and it's already very rewarding.

I'd definitely do it even if it's the same pay as BA.

Re: How to Become a Data Engineer in 2021

#123

Post by fantod »

dibujante wrote: post #105

"In order to undestand how these systems work I would recommend to know the language in which they are written. The biggest concern with Python is its poor performance hence the knowledge of a more efficient language will be a big plus to your skillset." What? The Apache stack that's written in Scala recompiles all your code into JVM bytecode, regardless of what language you've written it in. Yes, that includes Scala…

> In order to undestand how these systems work I would recommend to know the language in which they are written. The biggest concern with Python is its poor performance hence the knowledge of a more efficient language will be a big plus to your skillset.

I think these two sentences are sort of orthogonal to one another. The first, I interpret as saying that it's useful to understand Scala if you're using Spark, essentially because of the law of leaky abstractions [1]. I think you're responding to the second sentence and in that case I agree.

[1] https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

Re: How to Become a Data Engineer in 2021

#124

Post by master_yoda_1 »

markus_zhang wrote: post #121
master_yoda_1 wrote: post #97

IMHO first you need to become a programmer then you can become a data engineer. So if you need to start by learning data structure then you are doing something wrong. Also the topics suggested in "Algorithms & Data Structures" could easily be skipped, the information is drastically misleading. We should seriously have some fact checker, otherwise this kind of bullshit article keep trending on HN and people keep wasti…

Can you recommend something else? I'm preparing to go through a full CS education.

Coding and building teach you more than taking a course or watching a video. If you don't have any programming background, you can enroll in some coursera or udacity courses to start with. Then go through this course http://web.stanford.edu/class/cs106x/, the course reader is really good. After that for data engineering read this book https://www.amazon.com/Designing-Data-Intensive-Applications.... Also learn some sql. Take some data, feed into sql light db, and ask question and convert question into query. Becoming good at this takes some time. Be patience. The learning curve is like hokey stick, initial phase might have a dip but it accelerate in the later phase. BY ANY CHANCE DO NOT JOIN A BOOTCAMP.

Re: How to Become a Data Engineer in 2021

#125

Post by pjmlp »

Mauricebranagh wrote: post #48
josephmosby wrote: post #16

(source for everything following: I recently hired entry-level data engineers) The experience required differs dramatically between [semi]structured transactional data moving into data warehouses versus highly unstructured data that the data engineer has to do a lot of munging on. If you're working in an environment where the data is mostly structured, you will be primarily working in SQL. A LOT of SQL. You'll also n…

It does rather depends what sort of data I bet a data engineer at CERN or JPL has quiet a different set of required skills to say Google or a company playing at data science because its the next big thing. I should imagine at CERN etc knowing which end of soldering iron gets hot might still be required in some cases. I recall back in the mumble extracting data from b&w film shot with a high speed camera, by projectin…

When I was there, almost 20 years ago, it was all about C++, Python and Fortran, with GUIs being done in Java Swing.

I bet it is still mostly the same, just using Web GUIs nowadays.

Re: How to Become a Data Engineer in 2021

#126

Post by pjmlp »

StreamBright wrote: post #2

2021? More like 2010. Hadoop is getting deprecated rapidly and more companies split their write and read workloads. Separated storage and compute is also popular. Scala is not used that much, I think it is not worth the time investment. More and more companies go for Kotlin instead of Java when these want to tap into the Java ecosystem.

Kotlin's future is tied with Android, on the JVM it will be another Scala in 5 years time.

If JetBrains gets lucky, they might manage to create a cross-platform Kotlin eco-system as they are trying hard to push, as means to sell InteliJ licenses.

Lets see if it doesn't end like like Typesafe.

Re: How to Become a Data Engineer in 2021

#127

Post by imcoconut »

u678u wrote: post #53

Incidentally does anyone have resources for SMALL data? EG a few MB of a time, but requires the same ETL, scheduling, traceability. I'd love some lite versions of big-data tools but needs to be simple, small and cheap.

No reason you cant still use airflow for the orchestration/depenendecy management/scheduling. And for the processing and storage - Pandas and sqlite.

Also would highly highly recommend looking into kedro (which has airflow integration, or you could just run your pipelines with crontab)

Re: How to Become a Data Engineer in 2021

#128

Post by markus_zhang »

master_yoda_1 wrote: post #124
markus_zhang wrote: post #121

Earlier quoted context omitted.

Can you recommend something else? I'm preparing to go through a full CS education.

Coding and building teach you more than taking a course or watching a video. If you don't have any programming background, you can enroll in some coursera or udacity courses to start with. Then go through this course http://web.stanford.edu/class/cs106x/ , the course reader is really good. After that for data engineering read this book https://www.amazon.com/Designing-Data-Intensive-Applications... . Also learn some…

Thanks. I'm actually in middle of cs106 and indeed the reader is pretty good. I also have the intensive book which I read a bit but think I need more technical muscle for it.

Re: How to Become a Data Engineer in 2021

#129

Post by Grimm1 »

markus_zhang wrote: post #118
Grimm1 wrote: post #66

Earlier quoted context omitted.

I spent way too large a portion of my last position teaching developers about indexes, query plans and underlying join types and their impact on performance and memory consumption.

Just curious do you have such knowledge for columnar database such as Vertica?

Not Vertica though that looks very interesting. I do have a lot of experience with Redshift though. The difficulty is most implementations of data warehouses are fairly bespoke, even down to query plan and execution so knowledge on Redshift may not completely transfer to Vertica for instance.

Re: How to Become a Data Engineer in 2021

#130

Post by markus_zhang »

Grimm1 wrote: post #129
markus_zhang wrote: post #118

Earlier quoted context omitted.

Just curious do you have such knowledge for columnar database such as Vertica?

Not Vertica though that looks very interesting. I do have a lot of experience with Redshift though. The difficulty is most implementations of data warehouses are fairly bespoke, even down to query plan and execution so knowledge on Redshift may not completely transfer to Vertica for instance.

Thanks. But how does one approach to learn the internals for these things? It's not like MySQL or SQL Server or PostgreSQL that we have tons of books and very detailed documentation. For Vertica we only have a doc, no books, just provided as is.

It seems to be the norm for everything that takes flight around 2010. Of course many are open sourced so those are OK I guess.

Post reply on HN