Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Become a Data Engineer in 2021

khashtamov.com

Post reply on HN
Print view View original

91–100 of 142 posts

Re: How to Become a Data Engineer in 2021

#91

Post by victor106 »

disgruntledphd2 wrote: post #24
dominotw wrote: post #22

Earlier quoted context omitted.

Agree 100% with this comment, Old stack: Hadoop, spark, hive, hdfs. New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflake/redshift

Huh, what replaces Spark in those lists? For my money, its the best distributed ML system out there, so I'd be interested to know what new hotness I'm missing.

I agree with this.

Along with ML it is also a very high performance extract and transformation engine.

Would love to hear what other tech that are being used to replace Spark.

Re: How to Become a Data Engineer in 2021

#92

Post by cjalmeida »

sonofaragorn wrote: post #76
dominotw wrote: post #19

Earlier quoted context omitted.

SQL has taken over the space completely. 90% of data munging and transforms happen via SQL. I would learn python. Its the number one language outside sql.

What do you use if you need to process hundreds of GBs of data?

PySpark or Dask

Re: How to Become a Data Engineer in 2021

#93

Post by markus_zhang »

mulmen wrote: post #90
markus_zhang wrote: post #86

Earlier quoted context omitted.

This. I also think modern columnar databases and other techniques somehow makes Kimball to be obsolete or relaxed some how, but I could be very wrong. For example we use Vertica and DBA told us that Vertica loves wide tables with many columns, which doesn't look very Kimball to me. This gives me some trouble as I'm not really show how to model data properly.

> For example we use Vertica and DBA told us that Vertica loves wide tables with many columns, which doesn't look very Kimball to me. I have heard advice like this from colleagues and frankly I don't buy it. It certainly isn't gospel. I think it's an oversimplification. Columnar stores love star schemas. You can get away with a single table model too but you still need some kind of dimensional or at least domain-base…

I think Kimball has one benefit that single table doesn't have i.e. it's much more business intuitive. For example it's really difficult for me now to explain to our new hire why two completely unrelated fields are stuck in one table and one is NULL for 95% of the time.

The wide table thing is also kind of certified by someone inside of Vertica so I guess it does make sense. I'm not actually working in data team so I don't have the CS knowledge to prove or disprove it. Fortunately our new data lead is going to convert everything into Kimball so we are going to have solid proof eventually.

Re: How to Become a Data Engineer in 2021

#94

Post by runT1ME »

I've been approached about various data engineering jobs over the last couple years and the job descriptions have varied wildly. It has been everything from:

1. SQL/analytics wizard, capable of building out dashboards and quickly finding insights in structured data. Oracle/MSSQL/PostGres etc. Maybe even capable of FE development.

2. Pipeline expert, capable of building out data pipelines for transforming data, Flink, Spark, Beam on top of Kafka/Kinesis/Pubsub run from an orchestration engine like Airflow. Even this could span from using mostly pre-built tools wiring together things with a bit of python to move data from A to B, to the other exteme of full fledge Scala engineer writing complex applications that run on these pipelines.

3. Writing infrastructure software for big data pipelines, customizing Spark/Beam/Flink/Kafka and/or writing custom big data tools when out of the box solutions don't work or scale. Some overlap with 2, but really distinguished by it being a full fledged software engineer specializing in the big data ecosystem.

So, are all three of these appropriate to call Data Engineer? Is it mainly #1 and people are getting confused? I would certainly fall into the #3, so I'm always surprised when people approach me about 'SQL transform' type jobs.

Re: How to Become a Data Engineer in 2021

#95

Post by mrslave »

somurzakov wrote: post #40
disgruntledphd2 wrote: post #24

Earlier quoted context omitted.

Huh, what replaces Spark in those lists? For my money, its the best distributed ML system out there, so I'd be interested to know what new hotness I'm missing.

distributed ML != Distributed DWH. Distributed ML is tough to train because of very little control over train loop. I personally prefer using single server trainkng even on large datasets, or switch to online learning algos that do train/inference/retrain at the same time. as for snowflake, I havent heard of people using snowflake to train ML, but sbnowflake is a killer in managed distribited DWH that you dont have t…

> sbnowflake is a killer in managed distribited DWH that you dont have to tinker and tune

How do Snowflake (and Redshift, mentioned above) compare with CitusDB? I really like the PostgreSQL experience offered by Citus. I've been bit by too many commercial databases where the sales brochure promises the product does X, Y, and Z, only to discover later that you can't do any of them together because reasons.

Re: How to Become a Data Engineer in 2021

#96

Post by Grimm1 »

ABeeSea wrote: post #83
ianbutler wrote: post #69

Earlier quoted context omitted.

Glue is both more of a pain in the butt than regular old spark with pyspark and way more expensive, from my experience I would seriously question someone suggesting to use it. We could have been using it wrong, but porting our Glue scripts to standard EMR after our initial POC saved us over 10x the cost and it was substantially faster.

Both pricing and start-up times are significantly better in Glue 2.0 (assuming one can migrate). But even on Glue 1.0, orchestrating an ETL process with with several dozen jobs is a non-trivial amount of configuration and labor. (Jobs failures, job restarts, paging, job run history, cloudwatch logs, re-usable infrastructure as code when creating a new jobs, permissions and security, etc) that the increased cost is mo…

We're crawling and processing TBs of web data, we just use some python workers, Airflow, SQS and trigger a few scheduled EMR jobs easy peasy. Restarts and what not are handled by kubernetes at the container level and by Airflow at the code level. Airflow bakes in permissions and managing jobs. Glue left us a lot to be desired in that area, and $400-600 per ingest can't beat $30 bucks for the time the EMR cluster is up and since we use Kube for everything already it wasn't much a hassle to continue using it here. I'm sure in your case it makes sense, and in ours it didn't and this is why technology is crazy :P

Re: How to Become a Data Engineer in 2021

#97

Post by master_yoda_1 »

IMHO first you need to become a programmer then you can become a data engineer. So if you need to start by learning data structure then you are doing something wrong. Also the topics suggested in "Algorithms & Data Structures" could easily be skipped, the information is drastically misleading. We should seriously have some fact checker, otherwise this kind of bullshit article keep trending on HN and people keep wasting their time on learning LSM tree (what the fuck is that in the first place).

Re: How to Become a Data Engineer in 2021

#98

Post by tomnipotent »

zaptheimpaler wrote: post #46

Somewhat outdated view. This may be the current stack, but its outdated now and is slowly being replaced. The new view is not big data pipelines and ETL jobs, its lambda architecture, live aggregations/materialized views and simple SQL queries on large data warehouses that hide the underlying details. The batch model may still apply to ML I guess, but I'm no expert there.

This is true for only a very limited subset of data producers that need real-time or near real-time data included in ML models. For 99% of the rest, batch processing is just fine and considerably more economical.

Re: How to Become a Data Engineer in 2021

#99

Post by mulmen »

markus_zhang wrote: post #93
mulmen wrote: post #90

Earlier quoted context omitted.

> For example we use Vertica and DBA told us that Vertica loves wide tables with many columns, which doesn't look very Kimball to me. I have heard advice like this from colleagues and frankly I don't buy it. It certainly isn't gospel. I think it's an oversimplification. Columnar stores love star schemas. You can get away with a single table model too but you still need some kind of dimensional or at least domain-base…

I think Kimball has one benefit that single table doesn't have i.e. it's much more business intuitive. For example it's really difficult for me now to explain to our new hire why two completely unrelated fields are stuck in one table and one is NULL for 95% of the time. The wide table thing is also kind of certified by someone inside of Vertica so I guess it does make sense. I'm not actually working in data team so I…

Yeah I think this is a common misconception with columnar stores. That if they like wide tables (they enable wide tables) that must mean the wider the better.

Sorting (or partitioning) is one of the most powerful optimizations in your toolbox. But only when optimized for some kind of access pattern. When you combine domains to get more width you have to make a compromise on the sorting. Then the wheels come off.

You still need different tables for clicks and orders and payments, even if they are very wide. You may or may not physically conform your dimensions in pure Kimball style but logically you (should) still start there.

Re: How to Become a Data Engineer in 2021

#100

Post by dh5 »

darth_avocado wrote: post #51
alexpetralia wrote: post #49

Earlier quoted context omitted.

I don't think data engineers are paid less than software engineers.

They are. I should know. I've worked as one for years including big tech companies. For e.g. FB has a lower pay than SWE, lower RSUs etc. and you can only get SWE pay if you transition into one, and that requires you to go through an interview process internally.

AFAIK a "data engineer" at FB specifically is not quite equivalent to "software engineer specializing in distributed systems" which this article refers to--i.e. there is a totally separate job track for SWEs in general. The "data engineer" job title is definitely not standardized between companies.
Post reply on HN