201–210 of 333 posts
This is so true I don't think I ever read something so true. It's not even scientists vs software developers. It's people who are really into software development and clean code. They say the program needs a total rewrite and proceed to add 20 layers of inheritance and spreading out every function over 8 files. Ever since I make sure to repeat my mantra every week to developers: How maintainable code is is measured i…
Earlier quoted context omitted.
(Ab)use of any paradigm (I'll need a shower for using that word) can result in nightmares. Inheritance has its place and it is definitely useful in more than "few specialised cases". It can get out of hand and it can become a nightmare. Composition has its place and it is definitely not better than inheritance except in "few specialised cases". It can also result in nightmare, just wait till adoption of Rust and go i…
>Writing clean and maintainable code should be the best practice and writing obfuscated code for performance and security should be reserved for "few specialised cases" Except, we can (fairly objectively) reason about performance and security, while 'clean code' and 'maintainability' are arbitrary, with vague guidelines at best. Throwing out those first characteristics in name of the latter ones is just irrational. (…
You are assuming that the only things that matter are those that can be objectively measured (and measured simply and straightforwardly, with well-known metrics today).
Developer frustration, which will increase when having to deal with messy, unmaintainable code, is a real thing, even if it's harder to measure than performance and security. Not only does it create real stress and thus harm to the developers, it also slows development in ways that are going to be much less consistent and predictable than what's needed to write clean, maintainable code in the first place.
(Also, of course, there are at least some fairly well-accepted standards of clean, maintainable code, even if some aspects of those aren't entirely agreed on by everyone, and painting them as completely arbitrary, subjective things is just wrong.)
Earlier quoted context omitted.
I basically agree (except I will take static typing over dynamic typing any day). I work in computational chemistry, and scientists here don’t necessarily have problems actually coding (we’ve been doing it for 70+ years). But the “other stuff” is taking more and more time. Before, you wrote Fortran, put the files on disks or whatever, and sent them around. Now, you need to know: C++ and Python (and maybe Fortran too)…
> (except I will take static typing over dynamic typing any day) Why? What exact typesystem do you prefer over Python's dynamic typing? A lot of its idioms, and probably e.g. NumPy/SciPy infrastructure would be about impossible with current static typesystems (look at the mess that is C++ scientific/ndarray libraries). Ditto for much of the autodiff and GPU stuff like pytorch. Julia could perhaps get there, but the i…
> What exact typesystem do you prefer over Python's dynamic typing?
I just want to know what various objects are laying around in my code so I don't have to keep it in my head.
For example, if you are using a database library like psycopg, you run a query and execute it. What kind of object is returned from the query execution function? How do you check if it was null? You have to go looking at documentation, and often what you find is examples of what you can do. But those examples often don't include absolutely everything, so you have to go looking at the code itself. But library code in python is often pretty arcane.
With Fortran/C++/Rust/etc, you get all of that basically for free in your editor's autocomplete. And the compiler will check to make sure you didn't do anything truly dumb.
Dynamic typing also encourages some frustratingly bad habits. Lots of code out there will have functions that change return type depending on what arguments are provided. And occasionally someone will forget to return something in some branch and you won't know until you hit an edge case (during runtime) and suddenly you have a None coming from somewhere.
My projects tend to be in Python now, but I recently started a side project in C++ again and found it very refreshing.
> probably e.g. NumPy/SciPy infrastructure would be about impossible
Partly true, but maybe not as true as you think. I've been using the nlohmann JSON library for C++, and it is amazing. The code almost looks pythonic, and its type conversions are done automatically (although at runtime of course).
For example,
for(const auto &[key, value] : some_json["x"].items())
double d = value["other_key"];
This is partly because, in my opinion, some "best practices" are superstitions. Some practice was best because of some issue with 80s era computing, but is now completely obsolete; problem has been solved in better ways or has completely disappeared thanks e.g. to better tooling or better, well, practices. e.g. Hungarian notation. Yet it is still passed down as a best practice and followed blindly because that's what…
[flagged]
Two more to the scientists' tab: 1. No tests of any kind. "I know what the output should look like." Over time people who know what it should look like leave, and then it's untouchable. 2. No regard to the physical limits of hardware. "We can always get more RAM on everyone's laptops, right?". (You wouldn't need to if you just processed the JSONs one at a time, instead of first loading all of them to the memory and t…
I'm a scientist programmer working in a field comprised by biologists and computer scientists, and what I've experienced is almost exactly the opposite of the author. I've found the problems that biologists cause are mostly: * Not understanding dependencies, public/private, SCM or versioning, making their own code uninstallable after a few months * Writing completely unreadable code, even to themselves, making it imp…
> * Not understanding dependencies, public/private, SCM or versioning, making their own code uninstallable after a few months
Ultimately, if there were a simple way to get data in the correct state in an os-independent, machine independent (from raspberry pi to HPC the code should always work), concise, and idempotent way - people would use it. There isn't. But the certainly could be.
The solution we desperately need is a basically a pull request to a simple build tool (make, Snakemake, just, task, etc) that makes this idempotent and os-independent setup simple. Snakemake works on windows and Unix, so that's a decent start.
One big point is matching data outputs to source code and input state. *Allowing ipfs or torrent backends to Snakemake can solve this problem.*
The idea would be to simply wrap `input/output: "/my/file/here"` in `ipfs()`, wherein this would silently check if the file is locally cached to return, but if not go to IPFS as a secondary location to check for the file, then if the file isn't at either place, calculate it with the run command specified in Snakemake. It's useful to have this type of decentralized cache, because it's extremely common to run commands that may take several months on a supercomputer that give files that may only be a few MBs (exchange correlation functional) or only a few GBs (NN weights) so downloading the file is *immensely* cheaper to do than re-running the code - and the output is specified by the input source code (hence git commit hash maps to data hash).
The reason IPFS or torrent is the answer here is for several reasons: 1) The data location is specied by the hash of the content - which can be used to make a hash map of git commit hashes of source code state that map to data outputs (the code uniquely specifies the data in almost all cases, and input data can be included for the very rare cases it doesn't) 2) The availability and speed of download scales with popularity. Right now, were at the mercy of centralized storage systems, wherein the download rate can be however low they want it to be. However, LLM NN weights on IPFS can be downloaded very fast when millions of people *and* many centralized storage providers have the file hosted. 3) The data is far more robust to disappearing. Almost all scientific data output links point to nothing (MAG, sra/geomdb - the examples are endless). This is for many reasons such as academics moving and the storage location no longer being funded, accounts being moved, or simply the don't have enough storage space for emails on their personal Google drive and they delete the database files from their research. However, these are often downloaded many times by others in the field - the data exists somewhere - so it just needs to be accessible by decentralizing the data and allowing the community to download the file from the entire community which has it.
One of the important aspects to include in this buildtool would be to ensure that, every time someone downloads a certain file (specified by the git commit hash - data hash map) or uploads a file after computing it, they host the file as well. This way the community grows automatically by having a very low resource and extremely secure IPFS daemon host all of the important data files for different projects.
Having this all achieved by the addition of just 6 characters in a Snakemake file might actually solve this problem for the scientific / data science community, as it would be the standard and hard to mess up.
The next issue to solve would be popularize a standard way to get a package to work on all available cores/gpu/resources, etc on a raspberry pi to HPC without any changes or special considerations. Pyspark almost does this, but there's still more config than desirable for the community, and the requirement of installing OS-level dependencies (Java stuff) to work on python can often halt it's use completely (if the package using pyspark is a dependency of a dependency of a dependency, wet lab biologists [the real target users] *will not* figure out how to fix that problem if it doesn't "just work"[TM])
Earlier quoted context omitted.
It's a decent measure of complexity: It's not that "opening files" themselves is work-intensive. But having a lot of files smells of overengineered code. One long, yet simple function has less cognitive overhead than spreading the function across multiple classes or functions or call hierarchies (themselves spread over multiple files).
> One long, yet simple function has less cognitive overhead than spreading the function across multiple classes or functions or call hierarchies Not if you are encapsulating and naming effectively... Why read 100 lines when you can read 20 and find concerns in one routine you are concerned with? Function calls can be expensive. However, optimization can come whenever you need it, and if what you need is one call vs 5…
When you only have to superficially skim the code, that works.
If there are incorrect abstractions, such as logging, transaction logic or manual error handling mixed with "well named function calls", then it is already very problematic even to skim.
If you have to debug, it quickly becomes torture. Especially if state is involved and shared between multiple methods or classes.
If you have to reimplement the code: you're probably fucked.
TL;DR: a counter-productive rant against software engineers, claiming that bad code from software engineers is worse than bad code from scientists. Did you consider hiring an experienced software engineer as a lead?