21–30 of 142 posts
I've been in this space last 6 yrs or so and my scala usuage has gone down to zero. Not worth learning scala.
While maybe not strictly necessary per se, it's a great way to get a foot in the door, and provides a great way to foster advanced type systems and functional programming (I personally find it to be a really fun language to write in to boot).
(source for everything following: I recently hired entry-level data engineers) The experience required differs dramatically between [semi]structured transactional data moving into data warehouses versus highly unstructured data that the data engineer has to do a lot of munging on. If you're working in an environment where the data is mostly structured, you will be primarily working in SQL. A LOT of SQL. You'll also n…
Old stack: Hadoop, spark, hive, hdfs.
New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflake/redshift
I've been in this space last 6 yrs or so and my scala usuage has gone down to zero. Not worth learning scala.
This is an anecdote - plenty of firms are using Scala in their data engineering stacks and it's a great tool for the job. While maybe not strictly necessary per se, it's a great way to get a foot in the door, and provides a great way to foster advanced type systems and functional programming (I personally find it to be a really fun language to write in to boot).
What job can this do that can't be done via sql. dealing with unstructured data?
(source for everything following: I recently hired entry-level data engineers) The experience required differs dramatically between [semi]structured transactional data moving into data warehouses versus highly unstructured data that the data engineer has to do a lot of munging on. If you're working in an environment where the data is mostly structured, you will be primarily working in SQL. A LOT of SQL. You'll also n…
Agree 100% with this comment, Old stack: Hadoop, spark, hive, hdfs. New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflake/redshift
For my money, its the best distributed ML system out there, so I'd be interested to know what new hotness I'm missing.
Earlier quoted context omitted.
Agree 100% with this comment, Old stack: Hadoop, spark, hive, hdfs. New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflake/redshift
Huh, what replaces Spark in those lists? For my money, its the best distributed ML system out there, so I'd be interested to know what new hotness I'm missing.
I was comparing it for "traditional" data engineering stack that used spark for data munging, transformations ect.
I don't have much insight into ML systems or how spark fits there. Not all data teams are building 'ml systems' though. Parent comment wasn't referring to any 'ml systems', not sure why that would be automatically inferred when someone mentions data stack .
2021? More like 2010. Hadoop is getting deprecated rapidly and more companies split their write and read workloads. Separated storage and compute is also popular. Scala is not used that much, I think it is not worth the time investment. More and more companies go for Kotlin instead of Java when these want to tap into the Java ecosystem.
Hadoop is still widely used in enterprises (especially in banks), if you have experience working with Hadoop ecosystems it is a big plus anyway.
There is also some trends:
https://trends.google.com/trends/explore?date=today%205-y&ge...
For GCP, our stacks tend to be Composer (Airflow), BigQuery, Cloud Functions, and Tensorflow. There's the occasional Hadoop/Spark platform out there, but clients using those tend to have older platforms.
(source for everything following: I recently hired entry-level data engineers) The experience required differs dramatically between [semi]structured transactional data moving into data warehouses versus highly unstructured data that the data engineer has to do a lot of munging on. If you're working in an environment where the data is mostly structured, you will be primarily working in SQL. A LOT of SQL. You'll also n…
> I agree with a few other commentators here that Hadoop/Spark isn't being used a lot in their production environments I guess I'm the odd-man out because that's all I've used for this kind of work. Spark, Hive, Hadoop, Scala, Kafka, etc.
As DE has evolved, the role has transitioned away from traditional low code ETL tools towards code heavy tools. Airflow, Dagster, DBT, to name a few.
I work on a small DE team. We don't have the human power to grind out SQL queries for analysts and other teams. Our solutions are platforms and tools we build on top of more fundamental tools that allows other people to get the data themselves. Think tables-as-a-service.
2021? More like 2010. Hadoop is getting deprecated rapidly and more companies split their write and read workloads. Separated storage and compute is also popular. Scala is not used that much, I think it is not worth the time investment. More and more companies go for Kotlin instead of Java when these want to tap into the Java ecosystem.