81–90 of 367 posts
Earlier quoted context omitted.
You and me both, friend. Except I normally get down-voted to oblivion for saying the opposite.
I guess depends on what valuable means. I imagine most comp scientists are less replaceable than most software engineers, so point for compsci.
Earlier quoted context omitted.
You and me both, friend. Except I normally get down-voted to oblivion for saying the opposite.
I guess depends on what valuable means. I imagine most comp scientists are less replaceable than most software engineers, so point for compsci.
Obviously this is a huge generalization but I think it's a useful way to think about it. And when I say scientist, I mean "Professor of CS" not "24 year old with a BS in CS".
Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…
Not to disparage the amazing data scientists I've worked with, but I've been on teams where this is very much the approach to operationalizing models. It's basically, "Here's the sklearn model and some fragile featurization scripts we built. Can you take this to prod ASAP?"
The problem I've seen is that DS & DE teams were in different parts of the org and had their own sprints that were in no way connected. So they kept chucking models over the wall and we kept trying to faithfully operationalize. Once we convinced leadership that we had to collaborate from the get-go, things went a whole lot better. It also improved the working relationship of engineers and scientists.
I learned a hell of a lot from the scientists; they learned how to write better code. They also learned what code they didn't need to write because I could do it faster or better than them, leaving them to focus on more important things. It was pretty amazing to find what manual processes they would setup in lieu of proper (or even any) engineering support. Again, these are amazingly smart people, but they were being square-pegged into a lot of round-hole engineering tasks.
Now, the much more frustrating issue I had was being in a very data-heavy organization and being told by a distinguished engineer (my skip-level) plus my direct manager that, "data engineering isn't a real discipline." I left that org very shortly thereafter.
Why is data scientist a profession in IT but not for example computer scientist? Many IT professionals studied computer science but they don't call themselves scientists in their line of work.
Then the federal government came up with this title something like "Secretary of Data Scientist" and the guy who made up the term worked at that.
He clearly articulates team roles: from Biz Devs, marketers, PMs, UX designers, UI designers, Web Developers, API Engineers, Data scientists, Applied researchers, Platform/Data engineers, QA engineers, DevOps Engineers.
Then he talks about different ways to increase agility by combining these roles into generalists empowered to iteratively explore the "pyramid of data value" until the right product-market fit is found.
Building Data-science Intensive Web Applications is inherently waterfall, not agile, and I find this book to be a fascinating reference.
[1] https://www.oreilly.com/library/view/agile-data-science/9781...
The problem that I've seen is often "data scientists" are expected to be the equivalent of full-stack engineers (or maybe more accurately: one-man CTO shops)—to understand data architecture, understand business architecture, ensure data quality, build data into product, build dashboards, derive insights, posit hypotheses, set strategy, and drive business value. Thus many "data scientists" are juiced-up report-builder…
Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…
I feel seen. At a previous job, our output after some cleaning and transforming was a pg_dump for the data scientists to load. We had little visibility of what they did to that database once they got it.
Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…
Companies were in a rush hire "data scientists" and boot camps like Insight were more than happy to pump out very impressive PhDs with just enough understanding to build a Keras model.
I've worked in industry awhile doing DS work and have been astounded at the number of PhDs that both don't know how to write Python that doesn't live in a notebook and throw away years of disciplined experimentation experiences to just throw keras models at data until the needle moves.
There do exist excellent data scientists out there, who are both very solid software engineers and really know their stuff mathematically, but I've found most of these people can't reliably find jobs because the people interviewing them know so little that good data scientists will be penalized for answer a stock question correctly.
The field has been so flooded with amateurs that have no idea what they're doing, that potential mentors have been driven out, and now it's just a mess. To get a job doing DS if you do know what you are doing you have to play a weird game where you guess the incorrect answer the interviewer has in mind.
Earlier quoted context omitted.
These are incredibly disparate skill sets. Of course anyone would want to hire someone like this, and far more would claim to possess such a broad skill set, but in practice it is extremely rare. You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy…
> You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy), technical skills (application programming, advanced databases, advanced machine learning, Excel modeling) and finally some business domain knowledge. This is pretty much the bare minimum requ…
However, it's likely that our experiences simply diverge here.
Earlier quoted context omitted.
The aspiration that GP was getting to was that less cleaning is required as a result of better data engineering, I believe.
Correct. If you build your instrumentation correctly, then you don't really need to do any "cleaning." Doesn't mean you might not need to do transformation for different uses but ideally wouldn't need to, for example change data types like turning a bool into an int.
Unfortunately, data engineers rarely deal with purely in-house data. You're gonna be pulling data from a variety of data sources. I can assure you that if you're pulling from government data sources, you're gonna have a hell of a time. Speaking from direct experience, my team is probably going to spend $10M/year just trying to keep a government dataset in order, because they won't do it themselves. I'm talking lawyers, legal analysts, data engineers, data scientists, data entry personnel, etc.. just to fix data that should have never been broken in the first place.
It shouldn't be a shock that cleaning the data is the path of least resistance for many.