Why scientists are turning to Rust

In my PhD I was analyzing petabytes of structured data from particle detectors that required relatively complex algorithms just to reconstruct, before any physics analysis is ever started, and the analyses were just as complex. My experiment’s total code base was at least 100kloc of C++ and Fortran if not more, and used a farm with over 20k cores for massive distributed processing, single jobs regularly taking hundreds or thousands of cores and terabytes of memory.

I’m currently working on distributed GPU accelerated hydrodynamic simulations where a single input is in the ~1-10Gb range of structured mesh data and the calculation requires ~200+ Gb of GPU memory.

I really don’t see how excel would be a viable option for basically anything I have worked on in my scientific career, beyond my undergraduate toy analyses.

- I had none of those examples in mind (for example by foreign code I was referring to code written in a different project, not to using a different language). No, I don't need you to give me code snippets on how to do my actual list.

- Obviously Julia can do everything that Python can. And most of it will look ergonomic, short and simple. The difficult part isn't whether the language can do it once you know how, it's how quickly you can get to figuring it out when you don't know it.

- So what makes the small things difficult is really the mix of unhelpful error messages and the limited resources online.

This is my overall point - if you take someone who hasn't written code in either Python or Julia, give them access to online search and ask them to complete almost any task, they'll do it faster and with less effort in Python.

So yes you can easily start hacking something in fortran or c++, and the cognitive load is probably much smaller, but then you have the risk of being eaten by your bugs, or of not working on some problems just because the tool you make is so badly designed you give up doing the necessary changes.

Fortran is a great language for legacy products, with code written 50 years ago still working fine. But it has also gained new features over the years, keeping it ahead of competitors in terms of its ability to harvest all a machine can deliver.

This is all for technical work like numerical modelling. Scientists are a very practical lot, switching from Fortran to C (usually C++) when that makes sense, then to python, R, etc when that makes sense, and so forth.

The original article was about scientists switching to rust. I don't know anyone who has done that. I do know many who are exploring Julia, though.

Bear in mind that scientists tend to be quite good at technical work. Learning a new language is not a significant problem for them. But scientific programming is about much, much more than programming. The essence of scientific programming is in the underlying mathematics and science, and Fortran just happens to be pretty good at those things.

Technically they're infrastructure software engineers without the title, but most people don't know what that is, so it can be easier to call them data engineers. The company doesn't give titles like that, they're just "the cloud people" with a vanilla software engineer title.

Unless you're a BI / business analyst, there are probably software engineers that work with AWS / gCloud at your company, who setup the database(s) and do the backend. What you want to do is called an IT request, where you request they help out when you need to productionize. It's bad form for a data scientist or any other kind of researcher to have the keys to the live production servers. While it can be fun to learn that stuff, there is a barrier of entry, specially monitoring software and unit tests.

This is more on the PM (or DPM (data project manager)) side but: https://www.datascience-pm.com/domino-data-science-lifecycle... can give a good map of the process. Many who do research are on their own, not apart of larger teams and are disconnected, so they don't know what options they have. Maybe that roadmap can help, depending on your situation.

When it comes to productionization, I've had too many years of software engineers rewriting my models and adding bugs left and right. The second they want to rewrite it from a notebook into a py file (or equivalent language, usually Java), I get cautious. I find the best process skips these issues and the software engineers will thank you for it: Automate most of the productionization process. Using notebooks as an example, python has libraries that can import notebooks, so you don't have to copy paste code into a py file. Instead an OOP wrapper (which they love) can be written with the interface they like. They can do the IO like streaming data or batching or whatever. They can do the unit tests and monitoring part. What you can do is write the function calls inside of the interface. Usually you want a .predict() method, similar to how ML algos work, but .run() is popular too. Inside of that, put all of the functions from the notebook you want called and in what order. In metaphor it's like writing a header file in C or C++. This py file will then import the notebook and call those functions.

The beauty of this is at least four fold: 1) no new bugs 2) no explaining how the model works, because you're doing that part and 3) If you want to update the model, you can update the notebook and then no work needs to be done in the py file. It's importing the notebook, so it will import the changes. It's fully automated. 4) You can have EDA, plots, documentation, all that stuff that usually takes up 40-60% of the notebook, not be required to remove it, because it will not get called.

The only downside is that every cell in the notebook that the py file will call needs to be wrapped in a function. However, this is a good idea by default, because you'll get global variable type bugs when you run the cells out of order if your cells are not wrapped in a function, so to avoid that you're probably already writing functions in your notebooks.

*Sometimes code has to go fast, and notebooks tend to use dataframes, which is fine for 99% of the work out there. In that case, instead of having the software engineers rewriting my model, because of the problems mentioned above, I will instead 1) Write libraries in C/C++ that accelerate parts of the notebook. And if that can't be done I will 2) write the model directly while prototyping in a language like Rust, C, C++, or Java. I'd rather write it than have someone port it. Most will not do this, because it's beyond the average researcher's skill, but it alleviates so many problems, and I'm not afraid of statically typed languages.