[deleted]
How to Become a Data Engineer in 2021
41–50 of 142 posts
Re: How to Become a Data Engineer in 2021
#42...and nothing of basic statistics? Data Science people want to know about your data pipeline and have some quantification of the quality of that data. Also, monitoring data pipelines for data integrity often relies upon a statistical test. You don't need to go as far as Bayesian but you do need to understand when a median goes way off or if it bi-modal, etc.
That should be assumed in the "engineer" part of the role.
Yeah I would definitely expect an engineer to have a grasp of basic statistics such as mean, median, mode and be able to interpret statistical graphs on a basic level (Modality, skewness, shape,
etc) .
Re: How to Become a Data Engineer in 2021
#43(source for everything following: I recently hired entry-level data engineers) The experience required differs dramatically between [semi]structured transactional data moving into data warehouses versus highly unstructured data that the data engineer has to do a lot of munging on. If you're working in an environment where the data is mostly structured, you will be primarily working in SQL. A LOT of SQL. You'll also n…
> I agree with a few other commentators here that Hadoop/Spark isn't being used a lot in their production environments I guess I'm the odd-man out because that's all I've used for this kind of work. Spark, Hive, Hadoop, Scala, Kafka, etc.
I should have specified more thoroughly.
I am not seeing Spark being chosen for new data eng roll-outs. It is still very prevalent in existing environments because it still works well. (used at $lastjob myself)
However - I am still seeing a lot of Spark for machine-learning work by data scientists. Distributed ML feels like it is getting split into a different toolkit than distributed DE.
Re: How to Become a Data Engineer in 2021
#44We want all these skills, yet, we'll give you a separate title and pay you less than a software engineer. Meanwhile front end software engineers are still software engineers and get high pay.
Re: How to Become a Data Engineer in 2021
#45Earlier quoted context omitted.
Agree 100% with this comment, Old stack: Hadoop, spark, hive, hdfs. New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflake/redshift
Can you elaborate more on the "roles" of the "new stack"? To me dbt/dataform and airflow/dagster are quite similar, so why do you need one of each? fivetran/stitch/singer are all new
Airflow allows for more complex transformations of data that SQL may not be suited for. DBT is largely stuck utilizing the SQL capabilities of the warehouse it sits on, so for instance, with Redshift you have a really bad time working with JSON based data with DBT, Airflow can solve this problem. That's one example, but last I was working with it we found DBT was great for analytical modeling type transformations but from getting whatever munged up data into a useable format in the first place Airflow was king.
We also trained our analysts to write the more analytical DBT transformations which was nice, shifted that work onto them.
Don't get me wrong though, you can get really far with just DBT + Fivetran, in fact, it removes like 80% of the really tedious, but trivial ETL work. Airflow is just there for the last 20%
(Plus you can then utilize airflow as a general job scheduler)
Re: How to Become a Data Engineer in 2021
#46Somewhat outdated view. This may be the current stack, but its outdated now and is slowly being replaced. The new view is not big data pipelines and ETL jobs, its lambda architecture, live aggregations/materialized views and simple SQL queries on large data warehouses that hide the underlying details. The batch model may still apply to ML I guess, but I'm no expert there.
Re: How to Become a Data Engineer in 2021
#47(source for everything following: I recently hired entry-level data engineers) The experience required differs dramatically between [semi]structured transactional data moving into data warehouses versus highly unstructured data that the data engineer has to do a lot of munging on. If you're working in an environment where the data is mostly structured, you will be primarily working in SQL. A LOT of SQL. You'll also n…
Great points. It depends on where the business is at, the scale of their data, how processed their data is, and the timeliness/accuracy requirements of that data.
Re: How to Become a Data Engineer in 2021
#48(source for everything following: I recently hired entry-level data engineers) The experience required differs dramatically between [semi]structured transactional data moving into data warehouses versus highly unstructured data that the data engineer has to do a lot of munging on. If you're working in an environment where the data is mostly structured, you will be primarily working in SQL. A LOT of SQL. You'll also n…
It does rather depends what sort of data I bet a data engineer at CERN or JPL has quiet a different set of required skills to say Google or a company playing at data science because its the next big thing.
I should imagine at CERN etc knowing which end of soldering iron gets hot might still be required in some cases.
I recall back in the mumble extracting data from b&w film shot with a high speed camera, by projecting it on to graph paper taped to the wall and manualy marking the position of the "object"
Re: How to Become a Data Engineer in 2021
#49We want all these skills, yet, we'll give you a separate title and pay you less than a software engineer. Meanwhile front end software engineers are still software engineers and get high pay.
I don't think data engineers are paid less than software engineers.
Re: How to Become a Data Engineer in 2021
#50Earlier quoted context omitted.
> I agree with a few other commentators here that Hadoop/Spark isn't being used a lot in their production environments I guess I'm the odd-man out because that's all I've used for this kind of work. Spark, Hive, Hadoop, Scala, Kafka, etc.
I should have specified more thoroughly. I am not seeing Spark being chosen for new data eng roll-outs. It is still very prevalent in existing environments because it still works well. (used at $lastjob myself) However - I am still seeing a lot of Spark for machine-learning work by data scientists. Distributed ML feels like it is getting split into a different toolkit than distributed DE.
I guess it depends on what jobs you're looking for. There's a lot of exiting companies/teams (like mine) looking to hire people but we're on the "old stack" using Kafka, Scala, Spark, etc. We don't do any ML stuff but I'm on the pipeline side of it. The data scientists down the line tend to use Hive/SparkSQL/Athena for a lot of work but I'm much less involved with that.
Not all jobs are new pasture and I think that's forgotten very frequently.