Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing a JPEG Decoder in Rust – Part 2: Implementation I

mht.technology

Post reply on HN
Print view View original

71–80 of 86 posts

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#71

Post by vertex-four »

wahern wrote: post #67
vertex-four wrote: post #62

Earlier quoted context omitted.

I don't think it is - less, bash, gcc, systemd, most interpreters, and Firefox will all pretty much just crash if malloc() fails - they're varying levels of things you'd want a "systems language" for. Unless you're trying to define systems language to mean one used for kernel or bare-metal embedded development (both cases where even in C, you skip the standard library and write your own memory allocation functions et…

I'm not saying that aborting on OOM never makes sense. I'm saying there are many situations where you shouldn't abort on OOM, and those situations overlap considerably with the situations in which a systems language is ideal[1]. So a systems language that doesn't allow for robust and fine-grained OOM handing isn't much of a systems language. Example: Basically any highly concurrent network daemon that multiplexes man…

On the other hand... who actually disables over-committing on their Linux servers or the machines they run image editors on so that you could actually handle out-of-memory errors? I've never come across anyone who's changed the setting unless told to by a specific piece of software (almost always database software).

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#72

Post by steveklabnik »

wahern wrote: post #70
steveklabnik wrote: post #65

Earlier quoted context omitted.

To be clear, _every_ panic has to have the same implementation; you can't choose unwind vs abort for some panics vs others. The default allocator crate always does an abort, both before and after this change.

When building an app, does changing the default allocator effect all imported libraries? And is that allocator used for boxing, when boxing requires dynamic allocation?

Currently, "the allocator" is global in Rust. That is, the standard library uses liballoc to do all of its heap allocation, and it's liballoc that aborts on OOM.

You can of course write code that does anything, including use some other mechanism than liballoc.

There's motion on several fronts here:

The first, and currently unstable, is swapping out the implementation of liballoc for a different one. (we ship jemalloc and a pass-through to the system allocator with Rust)

The second, and currently in the "working on an RFC" phase, is per-object allocators.

Both of these are unstable because we're not 100% sure of the interface we want to stabilize yet, it's still a work in progress.

  > when boxing requires dynamic allocation?
Boxing always implies dynamic allocation.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#73

Post by nnethercote »

vertex-four wrote: post #62
wahern wrote: post #60

Earlier quoted context omitted.

That's all irrelevant for a language which is supposed to be a systems programming language, however that term is defined.

I don't think it is - less, bash, gcc, systemd, most interpreters, and Firefox will all pretty much just crash if malloc() fails - they're varying levels of things you'd want a "systems language" for. Unless you're trying to define systems language to mean one used for kernel or bare-metal embedded development (both cases where even in C, you skip the standard library and write your own memory allocation functions et…

> Firefox will all pretty much just crash if malloc() fails

It's not nearly as simple as that. A general approach we try to use is the following.

- Small allocations are infallible (i.e. abort on failure) because if a small allocation fails you're probably in a bad situation w.r.t. memory.

- Large allocations, especially those controlled by web content, are fallible.

The distinction between small and large isn't clear cut, but anything over 1 MiB or so might be considered large. You certainly don't want to abort if a 100 MiB allocation fails, and allocations of that size aren't unusual in a web browser.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#74

Post by technion »

pslam wrote: post #53

If you scan through the source (on github), the most notable thing is the lack of the "unsafe" keyword. I've seen too many people basically transliterate from C to Rust, along with all the unsafe operations. This one is pretty much unoptimized, but still seems to be performant enough, and doesn't do anything "unsafe". That's not to say you could just throw this into the back-end of a public-facing website. It's still…

Given the history of vulnerabilities in tools like ImageMagick, that panic might still be a better position than an exploitable memory bug.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#75

Post by quotemstr »

vertex-four wrote: post #43
quotemstr wrote: post #8

Earlier quoted context omitted.

Yes: now we all forget how to correctly manage memory. Rust: where resources are unlimited and aborting when you run out of them is okay

The vast majority of programs that you run on a day to day basis will abort when they run out of memory. On a default Linux system, there's not even any way to prevent that - you need to faff with the config to make it actually return errors when allocating memory.

Malloc can always fail, even when you have overcommit enabled. You can always run out of address space. It's pretty common on mobile, which is still predominantly 32-bit.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#76

Post by wahern »

nnethercote wrote: post #73
vertex-four wrote: post #62

Earlier quoted context omitted.

I don't think it is - less, bash, gcc, systemd, most interpreters, and Firefox will all pretty much just crash if malloc() fails - they're varying levels of things you'd want a "systems language" for. Unless you're trying to define systems language to mean one used for kernel or bare-metal embedded development (both cases where even in C, you skip the standard library and write your own memory allocation functions et…

> Firefox will all pretty much just crash if malloc() fails It's not nearly as simple as that. A general approach we try to use is the following. - Small allocations are infallible (i.e. abort on failure) because if a small allocation fails you're probably in a bad situation w.r.t. memory. - Large allocations, especially those controlled by web content, are fallible. The distinction between small and large isn't clea…

That approach may make sense on the desktop, but it doesn't make sense for network servers. On a network server allocations tend to be small. Your 10,001st request might fail not because of a huge allocation but because it just pushed you over the line.

Aborting all 10,000 requests provides really poor QoS. You can get away with that if you're Google, because 10,000 requests is still miniscule relative to your total load across "the cloud". For almost everybody else it will hit hard, and among other things makes you more susceptible to DoS.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#77

Post by wahern »

vertex-four wrote: post #71
wahern wrote: post #67

Earlier quoted context omitted.

I'm not saying that aborting on OOM never makes sense. I'm saying there are many situations where you shouldn't abort on OOM, and those situations overlap considerably with the situations in which a systems language is ideal[1]. So a systems language that doesn't allow for robust and fine-grained OOM handing isn't much of a systems language. Example: Basically any highly concurrent network daemon that multiplexes man…

On the other hand... who actually disables over-committing on their Linux servers or the machines they run image editors on so that you could actually handle out-of-memory errors? I've never come across anyone who's changed the setting unless told to by a specific piece of software (almost always database software).

I do. Not only do I disable overcommit, I disable swap.

Most software is riddled with buffer overflows and other exploits, and yet it's rare that you come across an intruder while he's installing his rootkit. That doesn't mean it's not happening, just that people are ignorant about it; and that things can appear normal even with rootkits installed.

Like buffer bloat, people can be experiencing a problem without even realizing it's a problem. When software crashes under load they just think that it's _normal_ to crash under load.

Or when it crawls to a snails pace under load because it's swapping like mad, they think that's normal, even though QoS would have been much better if the software failed the requests it couldn't serve rather than slowing everybody down until they _all_ timeout, sometimes even preventing administrators from diagnosing and fixing the problem.

OOM provides back pressure. Back pressure is much more reliable and responsive than, e.g., relying on magic constants for what kind of load you _think_ can be handled.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#78

Post by bjz_ »

gmfawcett wrote: post #50
cfallin wrote: post #48

Earlier quoted context omitted.

Not at all -- there's a lot of commonality in the language designs. Core language features: both have algebraic datatypes like ML ("case classes" in Scala, enums in Rust), and a 'match' statement that makes working with them easy. Both have pervasive destructuring that works with match arms, 'let' bindings, etc. Data structures and mutation: both encourage a pragmatic version of immutability -- use values and provide…

I'm aware that Rust's bootstrap compiler was written in Ocaml. And yes, there are lots of ML inspirations in the language -- but not enough to make an ML language. Where are the module functors, for example? Many would consider that an essential element of an ML language, and it is a stretch to call a language an ML family member without them. I don't want this to devolve into a No True Scotsman debate, I just feel t…

Module functors were an SML feature - not really in the original 'ML' language. I would put Haskell in the ML family, and it doesn't have module functors. Rust has type inference, and a core type syntax and semantics that is heavily inspired by it.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#79

Post by bjz_ »

daconvergence wrote: post #6

Rust code looks very similar to ES6 or Typescript. The great CS language convergence has begun!

There's a huge difference between concrete syntax and static/dynamic semantics. I wish more developers would stop judging languages by their covers, but perhaps that is too much to ask...

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#80

Post by conradev »

technion wrote: post #74
pslam wrote: post #53

If you scan through the source (on github), the most notable thing is the lack of the "unsafe" keyword. I've seen too many people basically transliterate from C to Rust, along with all the unsafe operations. This one is pretty much unoptimized, but still seems to be performant enough, and doesn't do anything "unsafe". That's not to say you could just throw this into the back-end of a public-facing website. It's still…

Given the history of vulnerabilities in tools like ImageMagick, that panic might still be a better position than an exploitable memory bug.

If you run it in a spawned thread, the thread will actually just unwind and disappear if it panics (by default).

But to detect panics, you can use `afl.rs` to fuzz a parser: https://github.com/frewsxcv/afl.rs#user-content-trophy-case

Post reply on HN