Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every modeler is supposed to be a great Python programmer

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

61–70 of 131 posts

Re: Every modeler is supposed to be a great Python programmer

#61

Post by fbdab103 »

GeneralMayhem wrote: post #30
bigger_cheese wrote: post #26

Earlier quoted context omitted.

I work in an Industrial smelter (I'm a Materials/Chem Eng I look at outputs of models don't have a lot to do with developing them) the way the data pipelines are setup here the models are deployed as something called a "pickle file" I think this is something very specific to Python - maybe the equivalent would be a .dll file in C++ world. From point of view of the integration process the pickle file is just a black b…

Pickles and .dlls solve completely different purposes. A .dll, which stands for "dynamic link library", is code - it's how you package parts of your implementation in certain environments (Windows). A pickle file is data - it's a way of serializing out the contents of the state of a data structure, using some magic built into Python reflection, and then loading it back later. Pickling is very convenient for its purpo…

>the serialization format is derived from the implementation and not very stable, so if the implementation changes - sometimes in subtle ways like upgrading your Python version - the serialized data isn't readable anymore.

Could be wrong, but I do not believe this is true anymore. There are pickle versions, but I believe the format has been standardized such that you should be able to freely move between Python releases.

Re: Every modeler is supposed to be a great Python programmer

#62

Post by lmm »

lr4444lr wrote: post #37
mynameisash wrote: post #21

Earlier quoted context omitted.

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

Why do you want them to write code at all, then? Why not just task them with writing user stories around the parameterized functions they need, and just let you figure out how to implement it all?

What's the best way to write those user stories? Certainly not plain English; you'd probably want to use some kind of pseudocode. Ah, but it would be good if you could run that pseudocode and see what the results are in the happy path, without worrying too much about what happens when something goes wrong. We have that, it's called Python, so that tends to be how these user stories are expressed.

Re: Every modeler is supposed to be a great Python programmer

#63

Post by lmm »

polalavik wrote: post #10
version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

That’s what makes technical interviews even more frustrating. Most jobs day to day are like 5-10% solving deep technical puzzles and 90-95% fiddling with tooling and automating things that are incredibly frustrating or time consuming. So ya I’m not sure how to solve your silly brain teaser, but I have written custom test frameworks to automate the tedium away to save the team hundreds of man hours. It took a deep und…

> It took a deep understanding of VERY specific tools (shout out to ASIC EDA tools). But that doesn’t matter on a technical interview. So you hire somebody who can Leetcode, but can’t figure out how to fit all the pieces of the actual job together.

What would you suggest testing? If you ask about the specifics of a given framework, you'll get someone who's memorized that framework but can't actually think. If you get someone who can code and has a decent level of general intelligence, they'll generally be able to learn a new API.

Re: Every modeler is supposed to be a great Python programmer

#64

Post by ilyt »

mynameisash wrote: post #21
version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

Right but they still need to know some code and talk programming language.

The "here is some shitty code that does the job, make it good and production-ready" requires far less communication and domain knowledge than them telling you how exactly it should work and you translating that into code

Re: Every modeler is supposed to be a great Python programmer

#65

Post by FridgeSeal »

mynameisash wrote: post #21
version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

I mean, this is probably how things should be done.

Having the specialists of each field appropriately resourced, focused and aligned streamlines everyone’s life. The problem comes in that most places either get the scientists to pull double duty and we end up in the scenario scientists are pushing messy, subpar code out the door that nobody else wants to touch because they’re unsupported and operating out of their depth, and devs don’t want to touch it because they’re not keen to fix someone else’s hacky code and because they probably have other priorities and the science stuff likely isn’t well integrated into the rest of the other code-delivery and monitoring systems.

Re: Every modeler is supposed to be a great Python programmer

#66

Post by ajoseps »

Tcepsa wrote: post #38
bluedino wrote: post #33

I loved Python. It was easy to learn, very powerful, has libraries for everything... Then I started supporting researchers and scientists who wrote "python code" to run simulations etc. Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it. But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting…

That sounds similar to my experience with the Python ecosystem (of which I have a profound dislike at the moment). If you haven't tried Mamba yet I encourage you to give it a shot; for me it has had a pretty significant impact on the time I have to spend sitting around waiting for Conda packages to install.

Yeah I was looking for this comment. +1 for mamba

Re: Every modeler is supposed to be a great Python programmer

#67

Post by FridgeSeal »

fbdab103 wrote: post #36
FridgeSeal wrote: post #7

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas. C++ is out for different reasons I suspec…

>Tried putting R into production recently? It’s a frustrating and brittle experience. Oh man, can I sign onto this rant. While Python has spent a decade+ trying-and-failing to standardize on one of a dozen tools to properly manage dependencies, at least it is trying. R is still global-namespace, no-pinning by default. Sorta-kinda you can squint where renv is going, but still needs a lot of development. I had some hop…

> Oh man, can I sign onto this rant.

Please, be my guest hahaha.

I think my issue with so many of these tools is that they’re fundamentally meant to be used interactively. R (renv/etc) doesn’t have a package manager separate from the runtime, so you can’t just “yarn/pip/cargo/etc install” and have your dependencies setup prior to your script running, you have to invoke manual R commands and hope everything goes off ok, it doesn’t spontaneously pick a different directory to install your stuff in, or have a tantrum about permissions because it wants to install in some root dir.

Re: Every modeler is supposed to be a great Python programmer

#68

Post by polalavik »

p1necone wrote: post #58
polalavik wrote: post #10

Earlier quoted context omitted.

That’s what makes technical interviews even more frustrating. Most jobs day to day are like 5-10% solving deep technical puzzles and 90-95% fiddling with tooling and automating things that are incredibly frustrating or time consuming. So ya I’m not sure how to solve your silly brain teaser, but I have written custom test frameworks to automate the tedium away to save the team hundreds of man hours. It took a deep und…

Yep, I always tell people that the "plumbing" is by far the most important part of most software projects. The actual business problems basically solve themselves if you can write all the glue between them in a way that doesn't add extra cognitive load.

Great way to paraphrase it. I’ve found all jobs to be plumbing. The actual task of writing the code to do xyz was always straightforward or 5 minutes of googling. What no Googling is going to solve for you though is how to fit A B and C of your company’s process together into a manageable solution. Manageable being another keyword here. It’s easy to write a lot of code. It’s hard to keep all the pieces of it organized and flexible to future use cases/needs.

Re: Every modeler is supposed to be a great Python programmer

#69

Post by bfung »

In Silicon Valley back in 2014, when the seeds of ML/DS started to get traction, all the people with Data Science titles knew how to write Java webapps or Hadoop MR jobs to ingest, clean, transform, model/analyze, and serve results that went into production.

The specialization def. has narrowed scope in the last ~10ish years, but the original roots were that: Java + stats + database knowhow.

So yes, learn some production level skills. Having far too specialized people also runs the risk of lost-in-translation models that only work in the original implementation, until edge cases show up and model is out of date.

Re: Every modeler is supposed to be a great Python programmer

#70

Post by analog31 »

mynameisash wrote: post #21
version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

Interestingly, I'm a physical scientist in a business that makes hardware and software. I'm up front with people, that I'm not an engineer, and nobody expects me to be one. I write code to support prototyping and testing, but don't expect it to go into production. Nobody wants my janky prototype. I produce a theory of operation that covers a proposed design, as well as an outline of the basic manufacturing and service processes needed to make it work. The prototype sometimes helps confirm that the technical requirements for the product can be satisfied at an early stage of a project.

When I do share code and things that I've designed, it's usually for tooling not product, e.g., a script that helps calculate non-obvious design parameters and tolerances. Often, my code is used to test hardware components before the official software is ready.

In my case, I'm on call, though not 24/7, because the business as a whole isn't. For instance I'm available to diagnose supplier and production problems, and deal with weird issues that emerge in the field.

Post reply on HN