Goodbye, data science

Mainstream data scientists don't do data science, they are either ML engineers or data analysts who use ML Python libraries and promoted to data scientists with bigger paycheck. With machine learning and AI being the trend, the job is sexy and it is a chance for the company to market itself as it uses AI and cutting-edge tech. I worked for a data solutions company and while I work together with data scientists to propose a design to our clients, it was the AI part narrated by the data scientists that makes clients ready to throw money. Even if the requirements seem impossible or at least very difficult to achieve. These projects often fail because the data scientists couldn't reach the accuracy goal written in the contract and the project ends up in the trash. These data scientists eventually leave the company or get fired, only to find another job within a short time with even bigger salaries.

Now for the data engineering part; I wish the OP all the best with his career, but he is still in the honeymoon period and didn't witness the misery of being a data engineer.

- The job can get very repetitive very quickly, unless he'll work on infrastructure, his tasks will be mainly focused on maintaining existing ETL pipelines or building ones, and both are labor tasks. You'll end up a data plumber who makes sure data goes from A to B and then C and that's it. You'll seldom find something new or revolutionary to work on and you'll keep using the same tools as long as you're in the same company. Even when moving to a new job, you'll pretty much be hired because of your knowledge of the same data warehouse or ETL tool you used in the previous job.

- Data engineering is an underappreciated job. If things go right, nobody pats you in the shoulder. When shit gets loose, you'll be the one to clean up the mess. What makes it worse is, you can't leave this mess long because data is a snowball effect; if you leave it unprocessed, you'll end up with more data clogging your pipelines and what can be fixed within an hour can quickly take long nights and even days to resolve, and do you know what this means? Managers won't have their fancy dashboards updated and they'll start panicking.

- Data engineering is not rewarding. Again, you're doing your job. No one cares.

- Data engineering has nothing to show for. Yes, data is being crunched and processed and baked. It's the data analyst who builds the fancy dashboards for managers, and the data scientists who create fancy graphs for managers, and the ML engineers who create fancy products for managers. You're just a plumber who, instead of fixing toilets and sinks, fixes data pipelines.

- Just recently, data engineering salaries are rising thanks to low supply and higher demand thanks to better awareness from CTOs and heads of data about the importance of the role, but until a few years ago, they were paid less than a software engineer.

The definition of Standard deviation is in chapter 1 of Stats 101. https://www.google.com/search?q=standard+deviation&tbm=isch Apparently, asking Stats 101 chapter 1 question of a so called "Data Scientist" is too much of an irrelevant question!

> expect people to have very high Stats skills

Or as you have made apparent, expect people to have ZERO stats skills!

Some of the innumerate activities I have observed in "expert" data scientists and ML engineers who have years of experience without once thinking about sample sizes

1. Using A/B tests to accept the Null hypothesis instead of rejecting it

2. Squandering away 30M $ in annual revenue because they wanted to avoid a situation/meeting in which they might look like they don't understand statistics. This is hilarious because they simply nodded their head as if they understand all the calculations and then simply dropped any other meetings or followups and left 30M $ on the table

3. Not refreshing a key revenue generating model for 18 months because the were "trying to figure out" why the AUC was improving when the performance on "golden set data" was dropping

4. Using thresholding and aggregation to produce poor quality distorted training data of rich perfectly sampled data

5. Trying to use A/B tests to estimate impact even when the control and variant are not independent

All of the above at FAANGS! My coworkers in a non FAANG company were much more sophisticated. These are the kind of candidates a "build recommendations for youtube" interview selects. Template appliers.

The list of stupidities goes on and on! But yeah, none of them think that a basic understanding of statistics is necessary for work. The good thing about Javascript engineers is that they don't have an understanding of Statistics and are aware of it. However the DS/MLEs are unskilled and unaware of it.