Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Become a Data Engineer in 2021

khashtamov.com

Post reply on HN
Print view View original

51–60 of 142 posts

Re: How to Become a Data Engineer in 2021

#51

Post by darth_avocado »

alexpetralia wrote: post #49
darth_avocado wrote: post #44

We want all these skills, yet, we'll give you a separate title and pay you less than a software engineer. Meanwhile front end software engineers are still software engineers and get high pay.

I don't think data engineers are paid less than software engineers.

They are. I should know. I've worked as one for years including big tech companies. For e.g. FB has a lower pay than SWE, lower RSUs etc. and you can only get SWE pay if you transition into one, and that requires you to go through an interview process internally.

Re: How to Become a Data Engineer in 2021

#52

Post by garciasn »

prions wrote: post #29

SQL proficiency is important but I wouldn't say it supersedes programming experience. To me, Data Engineering is a specialization of software engineering, and not something like an analyst who writes SQL all day. As DE has evolved, the role has transitioned away from traditional low code ETL tools towards code heavy tools. Airflow, Dagster, DBT, to name a few. I work on a small DE team. We don't have the human power…

TL;DR: SQL is still a DE's best and most ubiquitous tool/language in the space--hands down.

I've led DE teams for the last decade. I have lived through shifts in toolsets, languages, etc. Regardless of platform, languages, model types, etc, etc, etc, etc, the one constant has been SQL with some sort of of scripting around it.

Right now, it seems Python is the big wrapper language, whether it's via dag or some other means but that's just the preferred method TODAY. Considering SQL has been around for decades and has outlasted just about every other language and system, many of which have opted for a SQL-like interface on top of their system, I would highly recommend DEs be very strong there.

Re: How to Become a Data Engineer in 2021

#53

Post by u678u »

Incidentally does anyone have resources for SMALL data? EG a few MB of a time, but requires the same ETL, scheduling, traceability. I'd love some lite versions of big-data tools but needs to be simple, small and cheap.

Re: How to Become a Data Engineer in 2021

#54

Post by red_hare »

darth_avocado wrote: post #51
alexpetralia wrote: post #49

Earlier quoted context omitted.

I don't think data engineers are paid less than software engineers.

They are. I should know. I've worked as one for years including big tech companies. For e.g. FB has a lower pay than SWE, lower RSUs etc. and you can only get SWE pay if you transition into one, and that requires you to go through an interview process internally.

That's totally contrary to my experience where Data Engineers are considered specialized SWE and paid more.

But I've never worked for FB.

Re: How to Become a Data Engineer in 2021

#55

Post by musingsole »

u678u wrote: post #53

Incidentally does anyone have resources for SMALL data? EG a few MB of a time, but requires the same ETL, scheduling, traceability. I'd love some lite versions of big-data tools but needs to be simple, small and cheap.

Most orgs working with small data I've seen will just fall back to the tutorial version of some big data tools (often times just eating the unused infrastructure cost for something like Hadoop when they're generating biweekly reports). Most project managers have a dream of their project scaling up and want to be prepared should a dream become a reality. And if you "under-engineer" (by which I mean specifically engineer for the problem the company is facing), you'll get called out by every armchair developer for not going with the "obvious, best solution."

I'm not bitter; you're bitter. /s

Re: How to Become a Data Engineer in 2021

#56

Post by kfk »

u678u wrote: post #53

Incidentally does anyone have resources for SMALL data? EG a few MB of a time, but requires the same ETL, scheduling, traceability. I'd love some lite versions of big-data tools but needs to be simple, small and cheap.

I have been working with small data few years. We built an internal library to move data in/out of systems and the schedule this into jobs. We mostly leverage S3 and Spectrum. The major complexities we found were in scheduling, proxies and fetching raw data from legacy applications.

Re: How to Become a Data Engineer in 2021

#57

Post by darth_avocado »

red_hare wrote: post #54
darth_avocado wrote: post #51

Earlier quoted context omitted.

They are. I should know. I've worked as one for years including big tech companies. For e.g. FB has a lower pay than SWE, lower RSUs etc. and you can only get SWE pay if you transition into one, and that requires you to go through an interview process internally.

That's totally contrary to my experience where Data Engineers are considered specialized SWE and paid more. But I've never worked for FB.

You are right in the sense that if you look at average SWE salaries and data engineering salaries, the average salary is higher for data engineers. Because the starting salaries for data engineers tend to be higher because of all the skills that are needed and there's plenty of SWE positions that require more than just a degree in CS. But if you start comparing salaries at maybe a senior level (4-5 yoe+), the salaries for SWEs start becoming a lot more than DEs. And again, I've worked in different companies, big and small, this holds true for all companies that have "Data Engineer" titles. There's of course companies like Netflix where you are a data engineer but still get a SWE title and get paid the same.

Re: How to Become a Data Engineer in 2021

#58

Post by molsongolden »

zaptheimpaler wrote: post #46

Somewhat outdated view. This may be the current stack, but its outdated now and is slowly being replaced. The new view is not big data pipelines and ETL jobs, its lambda architecture, live aggregations/materialized views and simple SQL queries on large data warehouses that hide the underlying details. The batch model may still apply to ML I guess, but I'm no expert there.

Any resources/guides you'd recommend?

Re: How to Become a Data Engineer in 2021

#60

Post by StreamBright »

smattiso wrote: post #30
StreamBright wrote: post #2

2021? More like 2010. Hadoop is getting deprecated rapidly and more companies split their write and read workloads. Separated storage and compute is also popular. Scala is not used that much, I think it is not worth the time investment. More and more companies go for Kotlin instead of Java when these want to tap into the Java ecosystem.

Are you working in this field? I am looking for a consultant to setup a modern data processing pipeline for a data driven hardware product I am building.

Yes, I am moving companies to their next data pipeline, that is my specialty. I have added my email to my profile, you can reach out to me.
Post reply on HN