Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every modeler is supposed to be a great Python programmer

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

91–100 of 131 posts

Re: Every modeler is supposed to be a great Python programmer

#91

Post by GauntletWizard »

ClumsyPilot wrote: post #55
opportune wrote: post #18

Everybody wants to work on the fun stuff. Turns out in software, getting something to actually deliver value (meaning convert it from prototype or proof of concept to something running in production) is 90% not fun stuff. Most people learn that in their first job. As a software engineer the only way I’d become a “data scientist/trader’s servant” is if I’m getting paid exorbitantly. Otherwise it’s the worst kind of wo…

I strongly disagree, most software engineers will not have excellent knowledge of modelling, and someone who is mainly a data scientist will not be able to compete with a dedicated engineer that studied distributed systems, kubernetes, and knows undefined C behaviour by heart. When building a house, the guy that lay concrete are welders are different people, Anastesiologists and surgeons are not interchangeable, etc.…

No, but they'd sure as heck have some overlap - and a working knowledge of python happens to be that overlap.

Re: Every modeler is supposed to be a great Python programmer

#92

Post by disgruntledphd2 »

bfung wrote: post #69

In Silicon Valley back in 2014, when the seeds of ML/DS started to get traction, all the people with Data Science titles knew how to write Java webapps or Hadoop MR jobs to ingest, clean, transform, model/analyze, and serve results that went into production. The specialization def. has narrowed scope in the last ~10ish years, but the original roots were that: Java + stats + database knowhow. So yes, learn some produc…

2014 is pretty late for this, more like 2011. I joined FB in 2023 and data people were mostly writing hive at that point. Agreed on the stats and db part, that's never gone away.

Re: Every modeler is supposed to be a great Python programmer

#93

Post by andy800 »

FridgeSeal wrote: post #7

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas. C++ is out for different reasons I suspec…

> wish there was some more variety

Kotlin is coming up on the outside. Late to the party, but the data science ecosystem is progressing nicely.

Re: Every modeler is supposed to be a great Python programmer

#94

Post by Fiahil »

selimthegrim wrote: post #23
Fiahil wrote: post #12

Yes, that’s expected of every scientist I am working with. The reason is quite simple and have little to do with Engineering work being more expensive than Scientific one (it’s actually the opposite): get a nice problem to solve or a great model to build, and very soon, you have an ivory tower completely disconnected from the original objective. To be fair, I don’t ask scientists to understand concurrency issues in p…

If they wanted to learn, where would you point them?

Side-by-side coding with a backend engineer (web or other), is great !

Anything else involving practice is going to work as well :)

Re: Every modeler is supposed to be a great Python programmer

#95

Post by subradios »

FridgeSeal wrote: post #65
mynameisash wrote: post #21

Earlier quoted context omitted.

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

I mean, this is probably how things should be done. Having the specialists of each field appropriately resourced, focused and aligned streamlines everyone’s life. The problem comes in that most places either get the scientists to pull double duty and we end up in the scenario scientists are pushing messy, subpar code out the door that nobody else wants to touch because they’re unsupported and operating out of their d…

The big thing is prestige, it takes some pretty big brains to write good, clean production infrastructure that doesn't fall over every five minutes.

Those big brains are absolutely capable of learning the math and doing the creative work of modeling.

They want to do the creative work.

Which means that if you create a prestige gradient and don't let your engineers do interesting things they themselves might get a paper out of, you lose good engineers and get bad engineers instead.

This is also a constant problem. In DevOps and SRE, the very best devops and sre people are absolutely incredibly mercurial and mercenary - because they've been fed total lies about building systems again and again when the net job description is "hey ops person, this application I wrote is misbehaving can you do advanced troubleshooting on this server for me do I can go do leetcode".

Re: Every modeler is supposed to be a great Python programmer

#96

Post by jononor »

fbdab103 wrote: post #61
GeneralMayhem wrote: post #30

Earlier quoted context omitted.

Pickles and .dlls solve completely different purposes. A .dll, which stands for "dynamic link library", is code - it's how you package parts of your implementation in certain environments (Windows). A pickle file is data - it's a way of serializing out the contents of the state of a data structure, using some magic built into Python reflection, and then loading it back later. Pickling is very convenient for its purpo…

>the serialization format is derived from the implementation and not very stable, so if the implementation changes - sometimes in subtle ways like upgrading your Python version - the serialized data isn't readable anymore. Could be wrong, but I do not believe this is true anymore. There are pickle versions, but I believe the format has been standardized such that you should be able to freely move between Python relea…

The base serialization is stable and compatible across Python versions. However pickle files can serialize entire Python objects, including from custom and third party code. Compatibility of serialization then depends on the versions of the code that implements those objects, which is often not well managed.

Re: Every modeler is supposed to be a great Python programmer

#97

Post by jononor »

programmarchy wrote: post #76

Is there anything like Blender Nodes, but for data modeling? It's an amazingly powerful system. [1] The learning curve is still fairly steep in terms of having to learn all the nodes and how they interact, but you can't make a syntax error. [1] https://www.youtube.com/watch?v=7EeIsUErzLE #nodevember - Simon Thommes - Procedural Shader Showreel (Blender)

RapidMiner is one. In the physics and electronics/physics domain there is also Simulink.

It can often hard to integrate these into production code though, as they are designed primarily as interactive desktop applications, with per seat licensing.

Re: Every modeler is supposed to be a great Python programmer

#98

Post by xLaszlo »

He should find a SWE and partner up. As more clients enter the ML space there are more less sophisticated ones who has less of a concept of cross functional teams. Rather than educate themselves these clients have unrealistic demands.

While I am a huge advocate of DSea writing better code going fullstack is unrealistic for them. What does it even mean to integrate? Fastapi? Docker? Helm charts? Monitoring and observability? SRE? The list is endless.

This is a classic case of "the client doesn't care the want business value".

If as a DS you want to get better at writing code join our Code Quality for Data Science (CQ4DS) discord:

https://discord.com/invite/8uUZNMCad2

Re: Every modeler is supposed to be a great Python programmer

#99

Post by zmachinaz »

Not sure that I would recommend actually hiring that guy. He does not seem to understand that modelling is only one part of the equation. If you model something non-trivial, usually you can not just hand over your R scripts to someone else and say: Please implement that in python/C. You have implementation constraints which feed back into the modelling itself, like latency or scalability. Furthermore, good luck in letting someone translate your non-trivial math into another language hoping that he won't break it in some subtle or non-subtle way. Its just far more efficient if you have someone who can actually do the prototype directly in python or C, and let then a pro-developer optimize specific parts of the code.

Re: Every modeler is supposed to be a great Python programmer

#100

Post by lp4vn »

I think nobody here really pointed out a very relevant issue that's completely widespread at least with tech job market: companies don't want to pay people.

You don't see this kind of problem in other established professions, you don't expect an accountant to be able to perform the job of a lawyer neither you expect a nurse to be able wear the hat of a nutritionist.

Now with the technological professions, let's use the term knowledge professions as an umbrella term, companies take advantage of the fact that these professions have not been around for that long and are not that established to keep expanding their rol of responsibilities.

We see that all the time with tech companies. It's not rare that you're supposed to know the frontend, backend, testing, devops, some of them even domain knowledge and the list keeps expanding even though sometimes they entail different sets of skills. The salary, not surprisingly, doesn't grow proportionally to the list of requirements. Companies don't want specialized people anymore, they want someone who will quickly pick the job of other people when/if they finish theirs.

That's what I believe the author's rant was about. He has been looking for a job in his field, he is not a software engineer. Yet people are expecting him to be a professional developer on the top of being a professional data modeler.

Post reply on HN