No post body was provided.
Every modeler is supposed to be a great Python programmer
51–60 of 131 posts
Re: Every modeler is supposed to be a great Python programmer
#52I loved Python. It was easy to learn, very powerful, has libraries for everything... Then I started supporting researchers and scientists who wrote "python code" to run simulations etc. Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it. But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting…
That sounds similar to my experience with the Python ecosystem (of which I have a profound dislike at the moment). If you haven't tried Mamba yet I encourage you to give it a shot; for me it has had a pretty significant impact on the time I have to spend sitting around waiting for Conda packages to install.
To make this more clear as I have had the same experience and want more people to enjoy it: Mamba is more or less a drop-in replacement for the conda tool with the guts rewritten in C++ and as a result is much faster to resolve dependencies -- I want to say an order of magnitude faster or more. It's even noticeably faster at downloading and installing. You just
conda install -c conda-forge mamba
Re: Every modeler is supposed to be a great Python programmer
#53My hypothesis is that few companies are big enough to be able to dedicate many people to _exclusively_ modeling.
If a modeler has 4 weeks to spend on learning new things, odds are most companies would benefit more from the modeler learning how to do basic parts of the operational (python) part of their job (which needs to happen for models to be useful work) vs spending 4 weeks diving deeper on some aspect of modeling (which may or may not yield percentage points of improvement on some problem).
Re: Every modeler is supposed to be a great Python programmer
#54This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…
I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…
No post body was provided.
Re: Every modeler is supposed to be a great Python programmer
#55Everybody wants to work on the fun stuff. Turns out in software, getting something to actually deliver value (meaning convert it from prototype or proof of concept to something running in production) is 90% not fun stuff. Most people learn that in their first job. As a software engineer the only way I’d become a “data scientist/trader’s servant” is if I’m getting paid exorbitantly. Otherwise it’s the worst kind of wo…
I strongly disagree, most software engineers will not have excellent knowledge of modelling, and someone who is mainly a data scientist will not be able to compete with a dedicated engineer that studied distributed systems, kubernetes, and knows undefined C behaviour by heart.
When building a house, the guy that lay concrete are welders are different people, Anastesiologists and surgeons are not interchangeable, etc.
Many Industries will produce code, just like many people often need to nail two wooden bits together, but that does not make them a qualified structural engineer.
Re: Every modeler is supposed to be a great Python programmer
#56This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…
I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…
Personally, I'd like to do both the data science and the engineering. I'd also like to be in a team where each member does both. I at least "feel" like these two skills are close enough spiritually to one another that you can find a team composed entirely of people that do both.
Re: Every modeler is supposed to be a great Python programmer
#57Earlier quoted context omitted.
I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…
Why do you want them to write code at all, then? Why not just task them with writing user stories around the parameterized functions they need, and just let you figure out how to implement it all?
Assuming you're writing actually detailed and complete user stories and those user stories are for functionality that basically amounts to "just math" (i.e. no UI or API standards or whatever to worry about alongside the business logic) I imagine it's not much more difficult or time consuming to just write the code rather than describing it in a user story.
Having one person write user stories and a second person, who understands the domain less than the first person write the actual code would be twice as much work for a worse result.
Re: Every modeler is supposed to be a great Python programmer
#58This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…
That’s what makes technical interviews even more frustrating. Most jobs day to day are like 5-10% solving deep technical puzzles and 90-95% fiddling with tooling and automating things that are incredibly frustrating or time consuming. So ya I’m not sure how to solve your silly brain teaser, but I have written custom test frameworks to automate the tedium away to save the team hundreds of man hours. It took a deep und…
Yep, I always tell people that the "plumbing" is by far the most important part of most software projects.
The actual business problems basically solve themselves if you can write all the glue between them in a way that doesn't add extra cognitive load.
Re: Every modeler is supposed to be a great Python programmer
#59Earlier quoted context omitted.
Well, in purely software shops, there's often people dreaming up the 'what to do' and a different group of people actually writing the code. Same with systems design, we have 'architects.' This should be no different for statistical modelers or other disciplines. Employers are just cheap.
I've never worked on more painful codebases than when the "architects" don't have to bother writing actual code and so are ignorant of all the edge cases and special-case business rules that turn their pretty pictures into a horrific ball of mud.
The "architects" I've known also had a habit of giving you their grand perfect plan in a meeting that lasts no more than an hour or so, disappearing for the next 6 months without communicating with the people building the thing at all, and then being surprised when the final codebase looks nothing like the perfect system they'd designed in their head.
Architects have to be involved in building the thing they're architecting.
(It feels like the blueprint analogy fits here - actual building architects write blueprints, and then builders go off and build the house based on those blueprints. The code we write is effectively the blueprint, not the house. So what the hell are our architects making?
We need to treat software "architects" more like building site foremen than actual architects)
Re: Every modeler is supposed to be a great Python programmer
#60I loved Python. It was easy to learn, very powerful, has libraries for everything... Then I started supporting researchers and scientists who wrote "python code" to run simulations etc. Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it. But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting…
Conda may suck in some ways but it solved a big problem; installing software without root accesss. It effectively removed os dependencies allowing for operating system changes. HPC operating systems were effectively ossified prior to that. Notebooks on the other hand are a god damned abomination. Obligatory past discussion; https://news.ycombinator.com/item?id=17856700
Couldn't you just do that with pip?