Why Rust is taking the data engineering world by storm

Why Rust is taking the data engineering world by storm
kerkour.com

Is this true? I love Rust and use it daily, yet I often hear from data engineering peers that Rust has too much friction for most data engineering projects, because of the borrow checker, lifetimes, compile times, inability to express functions and traits in some ways, async futures, and ABI/FFI issues with external data processing code.

Happy to be corrected about all the above, though.

It's memory safe and compatible with C code

Then it concludes with a pitch to read the authors learn rust book for 35 dollars

like, i see the authors point about backend data systems things like databases, query engines and storage. for the *low level* systems side of data engineering, yeah i see what you’re saying.

for the higher level *take super hacky code from data scientists and turn it into a production grade pipeline* i just don’t see it right now.

the thing is, right now at least, i would take the data science python code, clean it up then make it run via pyspark (or just multiprocessing). it’s still the fast happy path in my experience.

this means i don’t have to rewrite the whole thing in rust from scratch, and the data science team can continue to contribute to it with a bit of hand holding (i have zero confidence in an average data scientist learning rust).

also, i can run it on a single machine, or over a cluster when we start getting more data throughput. and i can set up autoscaling so i don’t have to pay attention when that’s happened.

(caveat: unless there’s some super simple python->rust processing library that basically mimics pyspark that i probably don’t know about?)

so, like, yeah, i see the point from the low level side of things. but at the higher level… i don’t think it’s there… yet.

better title might have been “why rust is taking the low level data systems world by storm”

addendum: i am actually learning rust at the moment (not for data stuff) and it has struck me as a decent language for data engineering pipeline work because of the emphasis on well defined structures (strong typing). is basically defining a concrete data model. i will check out the authors autumn of rust thingy blog post series at some point. might be helpful.