Live data from Hacker News

Writing a JPEG Decoder in Rust – Part 2: Implementation I

mht.technology

31–40 of 86 posts

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#31
post #29

I feel like i'm missing something. In the "Why" section he claims that Rust is a performant language with a low-level feel. In this part, he claims that decoding a tiny JPG image took 2 seconds with his code. How is that "performant" by any definition?

Make it work, then optimize.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#32
post #29

I feel like i'm missing something. In the "Why" section he claims that Rust is a performant language with a low-level feel. In this part, he claims that decoding a tiny JPG image took 2 seconds with his code. How is that "performant" by any definition?

A language allowing high performance code doesn't mean it automatically guarantees high performance code. The latter is essentially impossible.

One common pitfall is not turning the optimiser on (with --release for cargo) i.e. benchmarking a debug build rather than a release one, and from a quick glance this code looks like it may be doing a system call to read each and every byte of the file due to the use of .bytes() on an unbuffered file, whereas it would be better to use, say, read_to_end to read many bytes of the file at once.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#33
post #29

I feel like i'm missing something. In the "Why" section he claims that Rust is a performant language with a low-level feel. In this part, he claims that decoding a tiny JPG image took 2 seconds with his code. How is that "performant" by any definition?

Somebody on reddit found the bottleneck: https://www.reddit.com/r/rust/comments/4yinbt/writing_a_jpeg...

After fixing it, it apparently runs in 100 ms.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#34
post #2

I don't know rust, but what happens if you get a 0xff as the last byte here, wouldn't i+1 go out of array bounds? while i

Since Rust people are reading this.

I find myself wanting to walk an iterator pairwise regularly for stuff like this and I'm using:

    for (cur, next) in (& vec).iter().zip(vec.iter().skip(1)) {
        encoded_data.push(cur);
        if cur == 0xff && next == 0x00 {
            // ...
        }
    }
Is there a better way to write this? What I'd really like is the equivalent to Clojure's partition[1]. I know that the stdlib has windows `(partition n (dec n) seq)` and chunks `(partition n n seq)` implemented on slices but I usually want this on iterators for the lookahead-like behavior shown here.

[1] https://clojuredocs.org/clojure.core/partition

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#35
post #29

I feel like i'm missing something. In the "Why" section he claims that Rust is a performant language with a low-level feel. In this part, he claims that decoding a tiny JPG image took 2 seconds with his code. How is that "performant" by any definition?

The code reads the file into a big buffer and then does array operations to index into it, which seems like the opposite of what you want for a good decoder. Using iterators etc. will let you read the file incrementally and will avoid array bounds checking.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#36
post #2

I don't know rust, but what happens if you get a 0xff as the last byte here, wouldn't i+1 go out of array bounds? while i

Since Rust people are reading this. I find myself wanting to walk an iterator pairwise regularly for stuff like this and I'm using: for (cur, next) in (& vec).iter().zip(vec.iter().skip(1)) { encoded_data.push(cur); if cur == 0xff && next == 0x00 { // ... } } Is there a better way to write this? What I'd really like is the equivalent to Clojure's partition[1]. I know that the stdlib has windows `(partition n (dec n)…

I'm not 100% sure, but https://crates.io/crates/itertools might have what you're looking for.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#37
post #2

I don't know rust, but what happens if you get a 0xff as the last byte here, wouldn't i+1 go out of array bounds? while i

Since Rust people are reading this. I find myself wanting to walk an iterator pairwise regularly for stuff like this and I'm using: for (cur, next) in (& vec).iter().zip(vec.iter().skip(1)) { encoded_data.push(cur); if cur == 0xff && next == 0x00 { // ... } } Is there a better way to write this? What I'd really like is the equivalent to Clojure's partition[1]. I know that the stdlib has windows `(partition n (dec n)…

The `.windows(usize)` function almost does this, but it uses a runtime width and so has to yield slices rather than tuples.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#38
post #16

Earlier quoted context omitted.

It's not, but they're both ML-family languages. I had hoped that Rust would be able to offer a similar level of elegance to Scala.

The biggest difference between Scala and Rust here is ".iter()", which is necessary mainly because you need to specify whether the iterator iterates over references (".iter()") or passes results by value (".into_iter()"). Because Java doesn't have value types and has no concept of move semantics, it has no need for the distinction, while Rust does. In effect, the "elegance" of Scala here is really the product of its…

Wouldn't the type guarantees provided by rusts type system allow for different implementations based on whether it was called on value types vs reference types? I thought there was an auto specialization feature created a while ago that did something like that.

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#39
post #29

I feel like i'm missing something. In the "Why" section he claims that Rust is a performant language with a low-level feel. In this part, he claims that decoding a tiny JPG image took 2 seconds with his code. How is that "performant" by any definition?

Author here.

I am having some trouble reproducing the 2 seconds number, but with optimizations I'm getting around 1.5 seconds (without, I'm up to 5!), and as balducien mentions, /u/DroidLogician pointed out a _huge_ performance flaw in the file reading code.

Writing bad code is easy in any language (as I just showed :) - writing _super_ performant code is only possible in some.

EDIT: I made a pretty bad mistake while writing this comment, claiming that fixing the file reading code improved the speed of the decoder _a lot_, which was not true at all. In fact, the difference is barely (if at all) noticeable when decoding lena.jpeg. However, when reading a larger file, the improvement is definitely there. So sorry! Replying to HN comments while drinking and watching a movie turns out not to be a great idea after all :)

Re: Writing a JPEG Decoder in Rust – Part 2: Implementation I

#40
post #29

I feel like i'm missing something. In the "Why" section he claims that Rust is a performant language with a low-level feel. In this part, he claims that decoding a tiny JPG image took 2 seconds with his code. How is that "performant" by any definition?

Somebody on reddit found the bottleneck: https://www.reddit.com/r/rust/comments/4yinbt/writing_a_jpeg... After fixing it, it apparently runs in 100 ms.

It's worth mentioning, those numbers are for my test program which only read a single file, using my and his implementations. In addition, the file read was 20MB. lena.jpeg is 90K.
Post reply on HN