4 years ago I moved from a role where I primarily wrote C# as an architect on a web application, to an architect helping to build a data warehouse. The contrast in tooling, discipline and information available to build anything in the data world is so stark it had me questioning my career decisions. Sure, you can read Kimball and Inmon and I'm sure there are a handful of others out there - but there are drastically f…
Why do you think that coded ETL is winning over the click-and-drag variant? I'd say the latter makes things a lot easier no?
Genuine question: why is there so much pure teeming hatred for data scientists in this comment thread? Almost every comment comes off as full of snark and vitriol against data scientists.
Yes, the tide is turning now... who came up with the term "data scientist" anyway? It's a made up profession. If you need someone who understands statistics, get a statistician, or maybe a mathematician. If you need someone that designs and writes computer programs, get a computer programmer. But a "data scientist"? No, thanks.
> It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. What? That runs counter to my experience at every company where I've either seen data engineers or worked as one. My observations of how management treats the two groups is this: Data engineers ("blacksmiths"): Blacksmiths are paid less. People think of them as less highly educated. Th…
> reduced their QA or eliminated it wholesale This is usually the case only for companies that work on low risk applications (I.e. not safety related or critical industries) or have been lulled into complacency (sometimes, ironically, “we haven’t had a major issue so obviously QA isn’t needed” when strong QA is precisely why they didn’t see issues)
> low risk applications (I.e. not safety related or critical industries)
My anecdotal experience is with Palantir (software used in war zones). Between 2016-2018, they eliminated most of the 150-200 QA people they had. Testing is now done by devs and users with the help of sound CD principles like blue/green deployments.
Just my personal experience, but the people at my company titled "Data Engineers" basically can only be trusted to (very slowly) move data around, while the people titled "Data Scientists" have to do all the cleaning work to make the data suitable for analysis and modeling, in addition to doing that analysis and modeling. Is the point here that data scientists are doing too much of the work that should be handled by…
That sounds... very weird. Something is for sure wrong in your org. Anyway, the point, I think, is that data engineers and data scientists should be integrated in one team and not operate in different silos.
> What ends up happening is you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team Not to disparage the amazing data scientists I've worked with, but I've been on teams where this is very much the approach to operationalizing models. It's basically, "Here's the sklearn model and some fragile featurization scripts we built. Can you take this to prod ASAP?" The problem I've seen is th…
This is 100% my experience as a data scientist. The engineering support we get is restricted to submitting a ticket for database access or moving data from one system to another. Wouldn't dream of involving an engineer in a data science project team, because I have no evidence that they have any experience or expertise in anything other than tickets to move data around.
Yeah, because moving data around (which is hardly the entire responsibility of data engineer) is not useful at all
> reduced their QA or eliminated it wholesale This is usually the case only for companies that work on low risk applications (I.e. not safety related or critical industries) or have been lulled into complacency (sometimes, ironically, “we haven’t had a major issue so obviously QA isn’t needed” when strong QA is precisely why they didn’t see issues)
> low risk applications (I.e. not safety related or critical industries) My anecdotal experience is with Palantir (software used in war zones). Between 2016-2018, they eliminated most of the 150-200 QA people they had. Testing is now done by devs and users with the help of sound CD principles like blue/green deployments.
That’s fair, because there’s process elements in place to mitigate risk. I read your comment to say “get rid of QA” not “get rid of the separate QA team”. My comment was more pointed to those who feel testing is inherently wasteful. In your example, the testing and configuration is tightly controlled to mitigate the same risk as a QA than (although there may be something to be said about the best practice of having the QA team be independent). Where I get nervous is when testing gets cut in hazardous systems because of cost or schedule. I personally wouldn’t want to get on a plane or autonomous car built like that. My own personal anecdotal experience is that organizations that were cavalier about QA on safety critical hardware/software inevitably had their comeuppance
What must be communicated to management: It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. Magicians will be magicians, always hustling (bullshitting), but they will never have the value and job security of the blacksmith. The blacksmith can see the fruits of her own labour, whilst the magician must lie to herself and others in order to…
> It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. What? That runs counter to my experience at every company where I've either seen data engineers or worked as one. My observations of how management treats the two groups is this: Data engineers ("blacksmiths"): Blacksmiths are paid less. People think of them as less highly educated. Th…
I work at a place with a very high count of PhDs. Some of them write code. All of them view writing code as something menial and unimportant and its shows in the resulting work, which from my experience is atrocious. Of course I understand that YMV, but I will forever be skeptical of anyone writing code with a PhD after working here.
Np. There is a common trend in the industry where a company hires on a data scientist, doesn't know the data prerequisites (specifically labeled data), the data scientist struggles, after a while the company fires the data scientist. This leaves the company with a bad taste in their mouth. In recent years I tend to get hired on as a specialist to help fix this. (And yes, I've been the first engineer hired on too.) Wh…
Thank you for your comments! They are very insightful. To piggyback a bit: Assuming you are a competent data "analyst" who wants to become a data engineer, how would you go about it? Is "go back to school and get a CS degree" the answer? I suppose this question is very broad, but I am curious if a practitioner like you has an opinion. --- To give some context: I recently graduated with a STEM PhD, and looking to move…
Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…
The data science field has been flooded with PhDs with nowhere else to go that have no background in engineering, and sadly often have a very poor understanding of both machine learning and statistics. Companies were in a rush hire "data scientists" and boot camps like Insight were more than happy to pump out very impressive PhDs with just enough understanding to build a Keras model. I've worked in industry awhile do…
> The data science field has been flooded with PhDs with nowhere else to go that have no background in engineering, and sadly often have a very poor understanding of both machine learning and statistics.
I am a PhD student in a non-engineering field. I've been taking as many math and stats courses as I can, but what other courses should I be trying to take if I want to excel as a data scientist? Software engineering CS type courses?