Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Become a Data Engineer in 2021

khashtamov.com

Post reply on HN
Print view View original

71–80 of 142 posts

Re: How to Become a Data Engineer in 2021

#71

Post by otter-in-a-suit »

prions wrote: post #29

SQL proficiency is important but I wouldn't say it supersedes programming experience. To me, Data Engineering is a specialization of software engineering, and not something like an analyst who writes SQL all day. As DE has evolved, the role has transitioned away from traditional low code ETL tools towards code heavy tools. Airflow, Dagster, DBT, to name a few. I work on a small DE team. We don't have the human power…

I wholeheartedly agree with the "specialization" comment.

Unless you are in a position where you can entirely rely on managed tools that do the work for you and all effort is centered around managing the data, rather than the holistic view of your data pipelines (Talend ETL, Informatica - the "pre-Hadoop" world, if you will, and maybe some modern tools like Snowflake), then a good Data Engineer needs a deep understanding of programming languages, networking, some sysadmin stuff, distributed systems, containerization, statics, and of course a good "architect" view on the ever-growing zoo of tools and languages with different pros and cons.

Given that at the end of the day, most "Data Pipelines" run on distributed Linux machines, I've seen and solved endless issues with Kernel and OS configurations (noexec flags, ulimits, permissions, keyring limits ...), network bottlenecks, hotspotting (both in networks and databases), overflowing partitions, odd issues on odd file systems, bad partition schemes, a myriad of network issues, JVM flags, needs for auditing and other compliance topics, heavily multi-threaded custom implementations that don't use "standard" tools and rely on language features (goroutines, multiprocessing in Python, Threadpools in Java ...), encoding problems, various TLS and other security challenges, and of course, endless use of GNU tools and other CLI-fun and I would not necessarily expect for a pure SQL use case (not discounting the fact that SQL is, in fact, very important).

Not to mention that a lot of jobs / workflows Data Engineers design and write tend to be very, very expensive, especially on managed Clouds - generally a good idea to make sure everything works and your engineers understand what they are doing.

Re: How to Become a Data Engineer in 2021

#72

Post by marcinzm »

disgruntledphd2 wrote: post #65
somurzakov wrote: post #40

Earlier quoted context omitted.

distributed ML != Distributed DWH. Distributed ML is tough to train because of very little control over train loop. I personally prefer using single server trainkng even on large datasets, or switch to online learning algos that do train/inference/retrain at the same time. as for snowflake, I havent heard of people using snowflake to train ML, but sbnowflake is a killer in managed distribited DWH that you dont have t…

So do I, theoretically at least. But Spark is super cool and actually has algorithms which complete in a reasonable time frame on hardware I can get access to. Like, I understand that the SQL portion is pretty commoditised (though even there, SparkSQL python and R API's are super nice), but I'm not aware of any other frameworks for doing distributed training of ML models. Have all the hipsters moved to GPUs or someth…

>I'm not aware of any other frameworks for doing distributed training of ML models.

Tensorflow, PyTorch (not sure if Ray is needed) and Mxnet all support distributed training across CPUs/GPUs in a single machine or multiple machines. So does XGBoost if you don't want deep learning. You can then run them with KubeFlow or on whatever platform your SaaS provider has (GCP AI Platform, AWS Sagemaker, etc.).

edit:

>All the replies to my original comment seem to forget that, or maybe Snowflake has such functionality and I'm unaware of it.

Snowflake has support for custom Javascript UDFs and a lot of built in features (you can do absurd things with window functions). I also found it much faster than Spark.

Re: How to Become a Data Engineer in 2021

#73

Post by u678u »

musingsole wrote: post #55
u678u wrote: post #53

Incidentally does anyone have resources for SMALL data? EG a few MB of a time, but requires the same ETL, scheduling, traceability. I'd love some lite versions of big-data tools but needs to be simple, small and cheap.

Most orgs working with small data I've seen will just fall back to the tutorial version of some big data tools (often times just eating the unused infrastructure cost for something like Hadoop when they're generating biweekly reports). Most project managers have a dream of their project scaling up and want to be prepared should a dream become a reality. And if you "under-engineer" (by which I mean specifically engine…

Yeah I'm bitter. I want to bring structure, tools and discipline to small scale data gathering, but the big tools are just too time-consuming to get up and keep running with just a few hours.

Re: How to Become a Data Engineer in 2021

#74

Post by mywittyname »

smattiso wrote: post #27
mywittyname wrote: post #12

For GCP, our stacks tend to be Composer (Airflow), BigQuery, Cloud Functions, and Tensorflow. There's the occasional Hadoop/Spark platform out there, but clients using those tend to have older platforms.

What is your product? I am looking for a consultant to help me setup a good process for a data driven product hardware product.

The work I do is almost entirely Google Anaytics/Ads related. So probably not what you're looking for, but if so, leave your email and I'll reach out!

Re: How to Become a Data Engineer in 2021

#75

Post by disgruntledphd2 »

marcinzm wrote: post #72
disgruntledphd2 wrote: post #65

Earlier quoted context omitted.

So do I, theoretically at least. But Spark is super cool and actually has algorithms which complete in a reasonable time frame on hardware I can get access to. Like, I understand that the SQL portion is pretty commoditised (though even there, SparkSQL python and R API's are super nice), but I'm not aware of any other frameworks for doing distributed training of ML models. Have all the hipsters moved to GPUs or someth…

>I'm not aware of any other frameworks for doing distributed training of ML models. Tensorflow, PyTorch (not sure if Ray is needed) and Mxnet all support distributed training across CPUs/GPUs in a single machine or multiple machines. So does XGBoost if you don't want deep learning. You can then run them with KubeFlow or on whatever platform your SaaS provider has (GCP AI Platform, AWS Sagemaker, etc.). edit: >All the…

> Snowflake has support for custom Javascript UDFs and a lot of built in features (you can do absurd things with window functions). I also found it much faster than Spark.

UDF support isn't really the same, to be honest. You're still prisoner of the select from pattern. Don't get me wrong, SQL is wonderful where it works, but it doesn't work for everything that I need.

I completely agree that it's faster than Spark, but it's also super-expensive and more limited. I suspect it would probably be cheaper to run a managed Spark cluster vs Snowflake and just eat the performance hit by scaling up.

Tensorflow, PyTorch (not sure if Ray is needed) and Mxnet all support distributed training across CPUs/GPUs in a single machine or multiple machines. So does XGBoost if you don't want deep learning.

I forgot about Xgboost, but I'm a big fan of unsupervised methods (as input to supervised methods, mostly) and Spark has a bunch of these. I haven't ever tried to do it, but based on my experience of running deep learning frameworks and distributed ML, I suspect the combination of both to be exponentially more annoying ;) (And i deal mostly with structured data, so it doesn't buy me as much).

> You can then run them with KubeFlow or on whatever platform your SaaS provider has (GCP AI Platform, AWS Sagemaker, etc.).

Do people really find these tools useful? Again, I'm not really sure what SageMaker (for example) buys me on AWS, and their pricing structure is so opaque that I'm hesitant to even invest time in it.

Re: How to Become a Data Engineer in 2021

#76

Post by sonofaragorn »

dominotw wrote: post #19
switch007 wrote: post #11

Earlier quoted context omitted.

What languages are worth learning?

SQL has taken over the space completely. 90% of data munging and transforms happen via SQL. I would learn python. Its the number one language outside sql.

What do you use if you need to process hundreds of GBs of data?

Re: How to Become a Data Engineer in 2021

#77

Post by dominotw »

sonofaragorn wrote: post #76
dominotw wrote: post #19

Earlier quoted context omitted.

SQL has taken over the space completely. 90% of data munging and transforms happen via SQL. I would learn python. Its the number one language outside sql.

What do you use if you need to process hundreds of GBs of data?

sql on snowflake

Re: How to Become a Data Engineer in 2021

#78

Post by marcinzm »

disgruntledphd2 wrote: post #75
marcinzm wrote: post #72

Earlier quoted context omitted.

>I'm not aware of any other frameworks for doing distributed training of ML models. Tensorflow, PyTorch (not sure if Ray is needed) and Mxnet all support distributed training across CPUs/GPUs in a single machine or multiple machines. So does XGBoost if you don't want deep learning. You can then run them with KubeFlow or on whatever platform your SaaS provider has (GCP AI Platform, AWS Sagemaker, etc.). edit: >All the…

> Snowflake has support for custom Javascript UDFs and a lot of built in features (you can do absurd things with window functions). I also found it much faster than Spark. UDF support isn't really the same, to be honest. You're still prisoner of the select from pattern. Don't get me wrong, SQL is wonderful where it works, but it doesn't work for everything that I need. I completely agree that it's faster than Spark,…

>UDF support isn't really the same, to be honest. You're still prisoner of the select from pattern. Don't get me wrong, SQL is wonderful where it works, but it doesn't work for everything that I need.

Not sure how it's different from what you can do in Spark in terms of data transformations. Taking a list of objects as an argument basically allows your UDF to do arbitrary computations on tabular data.

> I forgot about Xgboost, but I'm a big fan of unsupervised methods (as input to supervised methods, mostly) and Spark has a bunch of these.

That's true, distributed unsupervised methods aren't done in most other places I know of. I'm guessing there's ways to do that with neural network although I haven't looked into it. The datasets I deal with have structure in them between events even if they're unlabeled.

>I completely agree that it's faster than Spark, but it's also super-expensive and more limited. I suspect it would probably be cheaper to run a managed Spark cluster vs Snowflake and just eat the performance hit by scaling up.

I used to do that on AWS. For our use case, Athena ate its lunch in terms of performance, latency and cost by an order of magnitude. Snowflake is priced based on demand so I suspect it'd do likewise.

Re: How to Become a Data Engineer in 2021

#79

Post by darth_avocado »

somurzakov wrote: post #63
darth_avocado wrote: post #57

Earlier quoted context omitted.

You are right in the sense that if you look at average SWE salaries and data engineering salaries, the average salary is higher for data engineers. Because the starting salaries for data engineers tend to be higher because of all the skills that are needed and there's plenty of SWE positions that require more than just a degree in CS. But if you start comparing salaries at maybe a senior level (4-5 yoe+), the salarie…

experienced data engineers should graduate to data architects/ML engineers and this way they can get on par with SWE, pls correct me if I am wrong.

You're right, but most companies do not have those positions formalized and therefore you're expected to do those as part of your job, but not gain the financial benefits. Also, there is a big disparity in these titles and what the duties entail, which inherently again feeds into the problem.

Re: How to Become a Data Engineer in 2021

#80

Post by hermitcrab »

u678u wrote: post #53

Incidentally does anyone have resources for SMALL data? EG a few MB of a time, but requires the same ETL, scheduling, traceability. I'd love some lite versions of big-data tools but needs to be simple, small and cheap.

Take a look at our https://www.easydatatransform.com tool. It is a drag and drop data munging tool for datasets up to a few million rows. It runs locally on Windows or Mac. You should be able to install it and start transforming your data within a few minutes. It doesn't have a built in scheduler (yet), but you can run it from the command line.

Excel Power Query is also quite lightweight. But is pretty klunky in my (biased) opinion.

Post reply on HN