Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust: Not So Great For Codec Implementing

codecs.multimedia.cx

Post reply on HN
Print view View original

71–80 of 283 posts

Re: Rust: Not So Great For Codec Implementing

#71

Post by m-j-fox »

davemp wrote: post #58

> And that’s why C is still the best language for systems programming—it still lets you to do what you mean (the problem is that most programmers don’t really know what they mean) Honestly after a ~1yr of embedded C co-op/intership experience, I'm familiar enough but not too entrenched to say that C is not that great for embedded/systems. When you're dealing at the hardware architecture level you need more detail tha…

Thumbs up for driving more nails into C's coffin. What any pretender to the throne needs most is C interoperability because we're stick with legacy code. You need to be able to just code in C when necessary including the datatype zoo, the unsafety and the ugly macros. If it takes even 5 minutes to wrap a C function for your new language, that's years of work for some codebases.

Re: Rust: Not So Great For Codec Implementing

#72

Post by derefr »

binarycrusader wrote: post #44
Sean1708 wrote: post #11

> And don’t tell me about Bytes crate—it should not be a separate crate I'd be interested to hear the author's reasoning behind this, if it does what they want then why not use it? It's small and well written, so I don't think vetting it should be a problem. The rest of the article seems quite sensible, that comment just strikes me as a little odd.

A little copying is better than a little dependency. https://go-proverbs.github.io/ As a developer at a large software company, every dependency that is not part of the language runtime itself is a pain because legal paperwork and evaluation has to be done for each individual component before I can use it / ship it. NOTE: I am not referring to copying I would be doing, but to the crates thats have little dependencies…

"Dependency" doesn't imply "third-party library." There are plenty of crates that are maintained by the Rust organization itself. You could think of them as a "non-standard library."

(This isn't uncommon; it's also true in, say, Elixir: there are a few useful Hex packages owned by the elixir-lang GitHub org itself. And I believe it's true in Haskell as well.)

Re: Rust: Not So Great For Codec Implementing

#73

Post by dbaupp »

valarauca1 wrote: post #38
Sean1708 wrote: post #11

> And don’t tell me about Bytes crate—it should not be a separate crate I'd be interested to hear the author's reasoning behind this, if it does what they want then why not use it? It's small and well written, so I don't think vetting it should be a problem. The rest of the article seems quite sensible, that comment just strikes me as a little odd.

I agree with the author. Bytes is basically just C++'s `std::string` (kind of don't bite my head off people who've memorized the C++17 standard). Its an ARC backed array [1]. This suggests it should be a fairly fundamental abstraction. Edit: ^^^My C++ is wrong sorry :( Really I disagree with its purpose. In its immutable, non-threaded safe form you can create the same structure by just borrowing a value. This ofc req…

Being an ARC backed array isn't like std::string, and I don't see why having that copy-on-write behaviour implies it is fundamental. Could you expand?

Re: Rust: Not So Great For Codec Implementing

#74

Post by gwbas1c »

I tried Rust for the first time a few months ago. One of the things that really impressed me was that it was very easy to just get started. Even though the language is very different than I'm used to, I was able to "just jump in" without jumping through a lot of hoops. I was working with 3rd party crates very quickly.

One thing that I think does need improvement is the error handling. Far too many idiomatic examples just panic for error handling. There's no good way to just group errors by a higher-level classification and fall through to a generic error handler. As a result, the choice is between panic on error, or very detailed and verbose error handling. There's no middle ground.

Re: Rust: Not So Great For Codec Implementing

#75

Post by floatingatoll »

pcwalton wrote: post #64
binarycrusader wrote: post #44

Earlier quoted context omitted.

A little copying is better than a little dependency. https://go-proverbs.github.io/ As a developer at a large software company, every dependency that is not part of the language runtime itself is a pain because legal paperwork and evaluation has to be done for each individual component before I can use it / ship it. NOTE: I am not referring to copying I would be doing, but to the crates thats have little dependencies…

> A little copying is better than a little dependency. https://go-proverbs.github.io/ I really disagree with that quote. Copying is how you get bugs sticking around in software for all time (for example, doing binary searches in a way that avoids overflow is surprisingly tricky, and the endless copying of naive binary search code is why this bug is so difficult to eradicate). Honestly, that quote is just an excuse to…

I believe they mean, each time a source code is downloaded and used that contains a LICENSE file, a legal review must occur. So if it's bundled into one licensed work "Rust With Lots Of Crates Bundled", then that's one form to fill out, but if it's "Rust" and then "Download And Use Crates", that's one form per addon to fill out.

Re: Rust: Not So Great For Codec Implementing

#76

Post by pcwalton »

floatingatoll wrote: post #75
pcwalton wrote: post #64

Earlier quoted context omitted.

> A little copying is better than a little dependency. https://go-proverbs.github.io/ I really disagree with that quote. Copying is how you get bugs sticking around in software for all time (for example, doing binary searches in a way that avoids overflow is surprisingly tricky, and the endless copying of naive binary search code is why this bug is so difficult to eradicate). Honestly, that quote is just an excuse to…

I believe they mean, each time a source code is downloaded and used that contains a LICENSE file, a legal review must occur. So if it's bundled into one licensed work "Rust With Lots Of Crates Bundled", then that's one form to fill out, but if it's "Rust" and then "Download And Use Crates", that's one form per addon to fill out.

But you're bound by the license even if you copy and paste the code into your project instead of using Cargo.

Re: Rust: Not So Great For Codec Implementing

#77

Post by daniel_rh »

eptcyka wrote: post #10

Whilst unrelated to the article, my complaint about codecs in Rust is that they seem to be slow. Whilst the reason for this might be the immaturity of the libraries that I've used, but they've always been slower than their C counter-parts. The native JPEG decoder spins up 4 threads to decode the same amount of frames at 3x the time as the libjpeg-turbo does. There's a similar story for the FLAC decoder. I don't think…

I'm the author of https://github.com/dropbox/rust-brotli and I certainly ran into the issues mentioned in the article, and the initial version of my brotli decoder and encoder were each almost 10x slower. I also worked around each and every one of them, and the result is something that performs at 80-90% of the speed of the original on average, and some files compress faster with rust than with the original brotli codec.

allocator: I ran into the same situation and abstracted it with a generic allocator: https://github.com/dropbox/rust-alloc-no-stdlib it's an ugly solution, but it works and can allow for significant perf improvements down the line by bundling all same types together

benchmarks: I simply made a test that printed the time

primitive types: it's easy to write generic functions that do this

macro system: I found to be amazing, but I never used it to compact data types

borrow checker: split_at_mut was super helpful...also core::cmp::replace was useful to taking away pieces of a structure and manipulating them, then putting them back

Though it wasn't all roses: to try to get SSE vectorization I had to convert this 6 line short string matching function https://github.com/dropbox/rust-brotli/blob/238c9c539b446d7d...

into this multi hundred line monster with macros and so forth https://github.com/dropbox/rust-brotli/blob/238c9c539b446d7d...

but in the end it was as fast as hand-tuned intrinsics in C

I also found myself scared by dependencies, including onto the std library, so I tried to get everything to remain within the core library. This should allow something as low-level as a codec to be used in kernel space or in another place where a custom allocator is needed.

I also found that "rewriting in safe rust from C" was made easy by corrode https://github.com/jameysharp/corrode since C and rust can be interface compatible, it's easy to go one file at a time and turn it into working rust, then safe rust.

Re: Rust: Not So Great For Codec Implementing

#78

Post by wuch »

jeeyoungk wrote: post #46
myusernameisok wrote: post #36

Earlier quoted context omitted.

> The only legitimate complaint there is that it's tough in Rust to get two arbitrary mutable slices into the same array. Rust wants to be sure they're disjoint, to prevent aliasing. For some matrix manipulation, this is inconvenient. I've only looked into Rust briefly, but I thought this was the point of unsafe blocks? You can roll your own multi-dimensional array data structure using unsafe blocks if you're certain…

Unsafe allows you to dereference raw pointers and foreign-function calls, but you cannot go around the borrow checker, which is what's causing annoyance. That said, you can create raw and dereference raw pointers, which lets you to do the above.

Using unsafe blocks and pointers can silence the borrow checker, but that doesn't mean that code is well defined. For example, using swap (mentioned in the article) with overlapping references would be undefined behaviour.

Re: Rust: Not So Great For Codec Implementing

#79

Post by biokoda »

gwbas1c wrote: post #74

I tried Rust for the first time a few months ago. One of the things that really impressed me was that it was very easy to just get started. Even though the language is very different than I'm used to, I was able to "just jump in" without jumping through a lot of hoops. I was working with 3rd party crates very quickly. One thing that I think does need improvement is the error handling. Far too many idiomatic examples…

> There's no middle ground.

Not at all. This is what the ? operator is for (was try! before).

The idiomatic rust way is to have your own Error type that implements From for other Error types that you may encounter (io::Error, Utf8Error, etc.).

Then you can simply just write let result = dosomething()?; for any operation that may fail. If an error happens, function will return with the error.

I don't think you can do it any cleaner, given the constraints Rust is operating under.

Re: Rust: Not So Great For Codec Implementing

#80

Post by Manishearth »

davemp wrote: post #58

> And that’s why C is still the best language for systems programming—it still lets you to do what you mean (the problem is that most programmers don’t really know what they mean) Honestly after a ~1yr of embedded C co-op/intership experience, I'm familiar enough but not too entrenched to say that C is not that great for embedded/systems. When you're dealing at the hardware architecture level you need more detail tha…

> but if it makes an honest effort to be ergonomic for embedded

There are constant improvements but I wouldn't say we're there yet. I think we will be though, embedded is something a lot of folks care about.

Post reply on HN