Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Become a Data Engineer in 2021

khashtamov.com

Post reply on HN
Print view View original

31–40 of 142 posts

Re: How to Become a Data Engineer in 2021

#31

Post by dominotw »

knur wrote: post #20
somurzakov wrote: post #14

advanced proficiency in SQL and in any scripting language of your choice (C#/powershell, python) is enough to be a data engineer on any technical stack: windows/linux, on-prem/cloud, vendor specific/opensource, literally anything.

I disagree. That's not enough these days. If you want to build anything mildly interesting, you need to have a solid background on software engineering (building data pipelines in Spark, Flink, etc. goes way beyond knowing SQL), you need to really understand your runtime (e.g. the JVM, and how to tune it when working with massive amounts of data), you need a bit of knowledge about infrastructure, because some of the…

> building data pipelines in Spark, Flink, etc. goes way beyond knowing SQL

What if you build you data pipelines in sql? curious if you have an example of a data pipeline that needs spark?

Re: How to Become a Data Engineer in 2021

#32

Post by sseppola »

dominotw wrote: post #22
josephmosby wrote: post #16

(source for everything following: I recently hired entry-level data engineers) The experience required differs dramatically between [semi]structured transactional data moving into data warehouses versus highly unstructured data that the data engineer has to do a lot of munging on. If you're working in an environment where the data is mostly structured, you will be primarily working in SQL. A LOT of SQL. You'll also n…

Agree 100% with this comment, Old stack: Hadoop, spark, hive, hdfs. New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflake/redshift

Can you elaborate more on the "roles" of the "new stack"? To me dbt/dataform and airflow/dagster are quite similar, so why do you need one of each? fivetran/stitch/singer are all new

Re: How to Become a Data Engineer in 2021

#33

Post by st1x7 »

pgoggijr wrote: post #21
dominotw wrote: post #5

I've been in this space last 6 yrs or so and my scala usuage has gone down to zero. Not worth learning scala.

This is an anecdote - plenty of firms are using Scala in their data engineering stacks and it's a great tool for the job. While maybe not strictly necessary per se, it's a great way to get a foot in the door, and provides a great way to foster advanced type systems and functional programming (I personally find it to be a really fun language to write in to boot).

> plenty of firms are using Scala in their data engineering stacks

Isn't that just a result of everyone being into Spark a few years ago?

Re: How to Become a Data Engineer in 2021

#34

Post by somurzakov »

knur wrote: post #20
somurzakov wrote: post #14

advanced proficiency in SQL and in any scripting language of your choice (C#/powershell, python) is enough to be a data engineer on any technical stack: windows/linux, on-prem/cloud, vendor specific/opensource, literally anything.

I disagree. That's not enough these days. If you want to build anything mildly interesting, you need to have a solid background on software engineering (building data pipelines in Spark, Flink, etc. goes way beyond knowing SQL), you need to really understand your runtime (e.g. the JVM, and how to tune it when working with massive amounts of data), you need a bit of knowledge about infrastructure, because some of the…

I believe what you described is a job of Platform Engineer/Systems Engineer/Data lake Architect, especially JVM aspect of it. The interesting job is in the beginning when you build the cluster initially, or do major extension, after that the ops/maintenance is usually outsourced to cheap labor offshore - so this kinda job is personally not for me.

spark has dataframe API which is similar to pandas api and can be learned in one day, especially if you know python.

same for Airflow and other frameworks, it just a fancy scheduler that anyone can pick up in a couple days.

Re: How to Become a Data Engineer in 2021

#35

Post by sails »

disgruntledphd2 wrote: post #24
dominotw wrote: post #22

Earlier quoted context omitted.

Agree 100% with this comment, Old stack: Hadoop, spark, hive, hdfs. New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflake/redshift

Huh, what replaces Spark in those lists? For my money, its the best distributed ML system out there, so I'd be interested to know what new hotness I'm missing.

Snowflake I suppose for the average ML use case. Not for your high-performance ML, but for your average data scientist, maybe?

Edit: I may be wrong[1], would be curious to know what users who've used Spark AND Snowflake would add to the conversation.

[1] https://www.snowflake.com/blog/snowflake-and-spark-part-1-wh...

Re: How to Become a Data Engineer in 2021

#36

Post by marcinzm »

sails wrote: post #35
disgruntledphd2 wrote: post #24

Earlier quoted context omitted.

Huh, what replaces Spark in those lists? For my money, its the best distributed ML system out there, so I'd be interested to know what new hotness I'm missing.

Snowflake I suppose for the average ML use case. Not for your high-performance ML, but for your average data scientist, maybe? Edit: I may be wrong[1], would be curious to know what users who've used Spark AND Snowflake would add to the conversation. [1] https://www.snowflake.com/blog/snowflake-and-spark-part-1-wh...

Snowflake hits its limits with complex transformations I feel. Not just due to using SQL. It's "type system" is simpler than Spark's which makes certain operations annoying. There's a lack of UDFs for working with complex types (lists, structs, etc.). Having to write UDFs in Javascript is also not the greatest experience.

Re: How to Become a Data Engineer in 2021

#38

Post by dominotw »

marcinzm wrote: post #36
sails wrote: post #35

Earlier quoted context omitted.

Snowflake I suppose for the average ML use case. Not for your high-performance ML, but for your average data scientist, maybe? Edit: I may be wrong[1], would be curious to know what users who've used Spark AND Snowflake would add to the conversation. [1] https://www.snowflake.com/blog/snowflake-and-spark-part-1-wh...

Snowflake hits its limits with complex transformations I feel. Not just due to using SQL. It's "type system" is simpler than Spark's which makes certain operations annoying. There's a lack of UDFs for working with complex types (lists, structs, etc.). Having to write UDFs in Javascript is also not the greatest experience.

> There's a lack of UDFs for working with complex types (lists, structs, etc.). Having to write UDFs in Javascript is also not the greatest experience.

We load our data into SF in json and do plenty of list/struct manipulation using their inbuilt functions[1]. I guess you might have write a UDF if you are doing something super weird but inbuilt functions should get you pretty far 90% of the time.

https://docs.snowflake.com/en/sql-reference/functions-semist...

Re: How to Become a Data Engineer in 2021

#40

Post by somurzakov »

disgruntledphd2 wrote: post #24
dominotw wrote: post #22

Earlier quoted context omitted.

Agree 100% with this comment, Old stack: Hadoop, spark, hive, hdfs. New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflake/redshift

Huh, what replaces Spark in those lists? For my money, its the best distributed ML system out there, so I'd be interested to know what new hotness I'm missing.

distributed ML != Distributed DWH.

Distributed ML is tough to train because of very little control over train loop. I personally prefer using single server trainkng even on large datasets, or switch to online learning algos that do train/inference/retrain at the same time.

as for snowflake, I havent heard of people using snowflake to train ML, but sbnowflake is a killer in managed distribited DWH that you dont have to tinker and tune

Post reply on HN