Lot of discussion over on /r/rust https://www.reddit.com/r/rust/comments/k4jdvw/why_scientists...
I bought your book and it's been helpful. Thanks!
Glad to hear it!
Why scientists are turning to Rust
61–70 of 105 posts
Re: Why scientists are turning to Rust
#62Earlier quoted context omitted.
Rust is easier than C++ for this kind of work because of its packages. In some ways Rust is closer to Python for prototyping an idea to see if it works. Rust gets challenging when you have to deal with large systems, which is not a problem for research. Researchers / scientists will sometimes not even write functions. It's that small of a scope.
> Researchers / scientists will rarely even write functions. It's that small of a scope Scientific computing can mean so many different things that if feels like some of us are talking past each other in this thread. The person in the article talks about needing to get out every ounce of performance and writing an application that identifies genome variants at scale. You're talking code so small that it doesn't need…
The code I'm talking about needs to run fast (or be type safe) or you wouldn't consider using Rust. But internally in a library or a function that gets ran over and over again and needs to run super fast often isn't a lot of lines of code. So what I mean by a researcher may not even write a function is, they write the meat of the algorithm, it gets handed off to software engineers, and they turn it into a library or OOP it or whatever, and then run it on servers.
My current project in Python is massive compared to a normal one atm and it's about 2-3 pages of code in Jupyter. In Rust, if the libraries are there, it would be about the same size. I have 8 functions atm. It's also the end of the project and I'm in talks with data engineers to wrap it in OOP. They'll handle the IO. Or at least, that's how I've always done it and I've done a project in Rust.
Re: Why scientists are turning to Rust
#63Earlier quoted context omitted.
Which part of Julia is difficult?
Compared to Python? Just about everything starting from simple things like how to create a simple data structure, how to convert a variable to a different type, how to understand any error message that you get, how to open a file or modify a visualisation and extending all the way up to overarching concepts like the typing system, importing foreign code and managing dependencies.
How is making a structure in Julia harder than in python?
Re: Why scientists are turning to Rust
#64Earlier quoted context omitted.
Excel is far too slow for large scale datasets.
Is that true? Python is pretty slow too. I can’t imagine many scientists would really mind leaving their work running overnight.
I think you are...dramatically underestimating the size of many "modern" datasets and what most scientists want to do with them.
We record (electrical) neural activity from behaving animals. The raw data is about 1 Gb/minute, and we collect hours of it per day, 5-7 days/week. The lab next door has microscopes that produce image sets that are in the Gb to Tb range (big swathes of human brains, imaged at micron resolution). An RNAseq experiment is ~20 Gb range; other genomic things are similar, maybe a bit smaller.
All of this would be intractable in Excel, even over a long weekend :-) On top of that, Excel often doesn't do much of what you would need: there's basically no support for signal processing or image analysis or genomic work.
Re: Why scientists are turning to Rust
#65Earlier quoted context omitted.
> Researchers / scientists will rarely even write functions. It's that small of a scope Scientific computing can mean so many different things that if feels like some of us are talking past each other in this thread. The person in the article talks about needing to get out every ounce of performance and writing an application that identifies genome variants at scale. You're talking code so small that it doesn't need…
The code I'm talking about needs to run fast (or be type safe) or you wouldn't consider using Rust. But internally in a library or a function that gets ran over and over again and needs to run super fast often isn't a lot of lines of code. So what I mean by a researcher may not even write a function is, they write the meat of the algorithm, it gets handed off to software engineers, and they turn it into a library or…
[deleted]
Re: Why scientists are turning to Rust
#66Earlier quoted context omitted.
> Researchers / scientists will rarely even write functions. It's that small of a scope Scientific computing can mean so many different things that if feels like some of us are talking past each other in this thread. The person in the article talks about needing to get out every ounce of performance and writing an application that identifies genome variants at scale. You're talking code so small that it doesn't need…
The code I'm talking about needs to run fast (or be type safe) or you wouldn't consider using Rust. But internally in a library or a function that gets ran over and over again and needs to run super fast often isn't a lot of lines of code. So what I mean by a researcher may not even write a function is, they write the meat of the algorithm, it gets handed off to software engineers, and they turn it into a library or…
I'm a little jealous you work somewhere where there are data engineers to hand off to!
Re: Why scientists are turning to Rust
#67Does anyone know why scientists aren’t just using Excel? Nothing against Python/Julia but Excel seems like it would be my first choice (paired with some CSV files).
Excel doesn't scale, isn't reproducible, and has behaviors that silently destroy data
https://genomebiology.biomedcentral.com/articles/10.1186/s13...
https://www.theverge.com/2020/8/6/21355674/human-genes-renam...
Re: Why scientists are turning to Rust
#68Earlier quoted context omitted.
Excel is far too slow for large scale datasets.
Is that true? Python is pretty slow too. I can’t imagine many scientists would really mind leaving their work running overnight.
Python itself is rather slow indeed, but more often than not, it doesn't matter: for many data science projects, python is only used as a glue to control the processing and/or pass data between low-level C++ implementation that can chew through data at great speed, (e.g using parallelism).
Examples that immediately come to mind include : numpy, tensorflow, etc ...
Even heavy-duty I/O can be made to crank with python if you do it properly.
Re: Why scientists are turning to Rust
#69I don't think scientists should worry themselves with the cognitive burden of Rust. It's hard enough to figure out how to do things in any language as a full-time software developer, never mind someone that just needs a program for their research, etc.
Scientists are currently writing their code in C++ in many cases. If you have needs that require the performance benefits of C++, why not do it in a language that has a cohesive design (and is actually intuitive) instead of learning the pile of layers of decades-old debt that is C++?
I question if scientists ever should have to tackle the ins and outs of software architecture and performance optimization. Never mind the daunting task of running software.
I don’t really see Rust being altogether different to someone outside the software industry. It’s yet another complex knowledge domain that can be abused and intractable like any other.
Re: Why scientists are turning to Rust
#70I thought scientists were turning to Julia.
Many were but many turned back to whatever they were coming from. Julia it's an interesting language with rather odd design choices though.
In my opinion, the major hurdle to a wider adoption is that it was/is oversold. They claim c-like processing speeds, and in many cases they can be reached, but not off the bat. You need lots of hacks to get to those speeds.