Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zlib-rs is faster than C

trifectatech.org

Post reply on HN
Print view View original

251–260 of 492 posts

Re: Zlib-rs is faster than C

#251

Post by jandrewrogers »

cmrdporcupine wrote: post #77
kccqzy wrote: post #66

Earlier quoted context omitted.

By your line of reasoning, SIMD intrinsics functions should not be marked as unsafe in the first place. Then why are they marked as unsafe?

There's no standardization of simd in Rust yet, they've been sitting in nightly unstable for years: https://doc.rust-lang.org/std/intrinsics/simd/index.html So I suspect it's a matter of two things: 1. You're calling out to what's basically assembly, so buyer beware. This is basically FFI into C/asm. 2. There's no guarantee on what comes out of those 128-bit vectors after to follow any sanity or expectations, so... b…

The example here is trivially safe but more general SIMD safety is going to be extremely difficult to analyze for safety, possibly intractable.

For example, it is perfectly legal to dereference a vector pointer that references illegal memory if you mask the illegal addresses. This is a useful trick and common in e.g. idiomatic AVX-512 code. The mask registers are almost always computed at runtime so it would be effectively impossible to determine if a potentially illegal dereference is actually illegal at compile-time.

I suspect we’ll be hand-rolling unsafe SIMD for a long time. The different ISAs are too different, inconsistent, and weird. A compiler that could make this clean and safe is like fusion power, it has always been 10 years away my entire career.

Re: Zlib-rs is faster than C

#252

Post by rat87 »

sunshowers wrote: post #122
chongli wrote: post #47

Earlier quoted context omitted.

Isn't it the case that once you use unsafe even a single time, you lose all of Rust's nice guarantees? As far as I'm aware, inside the unsafe block you can do whatever you want which means all of the nice memory-safety properties of the language go away. It's like letting a wet dog (who'd just been swimming in a nearby swamp) run loose inside your hermetically sealed cleanroom.

What language is the JVM written in? All safe code in existence running on von Neumann architectures is built on a foundation of unsafe code. The goal of all memory-safe languages is to provide safe abstractions on top of an unsafe core.

I don't think what something was written in should count. Baring bugs it should still be memory safe. But I believe JVM has ffi and as soon as you use ffi you risk messing up that memory safety.

Re: Zlib-rs is faster than C

#254

Post by pjmlp »

YZF wrote: post #18

I found out I already know Rust: unsafe { let x_tmp0 = _mm_clmulepi64_si128(xmm_crc0, crc_fold, 0x10); xmm_crc0 = _mm_clmulepi64_si128(xmm_crc0, crc_fold, 0x01); xmm_crc1 = _mm_xor_si128(xmm_crc1, x_tmp0); xmm_crc1 = _mm_xor_si128(xmm_crc1, xmm_crc0); Kidding aside, I thought the purpose of Rust was for safety but the keyword unsafe is sprinkled liberally throughout this library. At what point does it really stop mat…

It goes both ways, many C folks call files full of inline Assembly and compiler specific extensions, C.

Re: Zlib-rs is faster than C

#255

Post by vlovich123 »

jandrewrogers wrote: post #251
cmrdporcupine wrote: post #77

Earlier quoted context omitted.

There's no standardization of simd in Rust yet, they've been sitting in nightly unstable for years: https://doc.rust-lang.org/std/intrinsics/simd/index.html So I suspect it's a matter of two things: 1. You're calling out to what's basically assembly, so buyer beware. This is basically FFI into C/asm. 2. There's no guarantee on what comes out of those 128-bit vectors after to follow any sanity or expectations, so... b…

The example here is trivially safe but more general SIMD safety is going to be extremely difficult to analyze for safety, possibly intractable. For example, it is perfectly legal to dereference a vector pointer that references illegal memory if you mask the illegal addresses. This is a useful trick and common in e.g. idiomatic AVX-512 code. The mask registers are almost always computed at runtime so it would be effec…

Presumably a bounds check on the mask could be done or a safe variant exposed that does that trick under the hood. But yeah I don’t disagree that it’s “safe SIMD” is unlikely to scratch the itch for various applications but hopefully at least it’ll scratch a lot of them enough that the remaining unsafe is reduced.

Re: Zlib-rs is faster than C

#256

Post by j16sdiz »

jrockway wrote: post #55

Chromium is kind of stuck with zlib because it's the algorithm that's in the standards, but if you're making your own protocol, you can do even better than this by picking a better algorithm. Zstandard is faster and compresses better. LZ4 is much faster, but not quite as small. Some reading: https://jolynch.github.io/posts/use_fast_data_algorithms/ (As an aside, at my last job container pushes / pulls were in the dev…

Chromium supports brotli and zstd

Re: Zlib-rs is faster than C

#257

Post by oneshtein »

throwaway2037 wrote: post #234
masfuerte wrote: post #190

Earlier quoted context omitted.

int average(int x, int y) { return (x+y)/2; }

I assume you are hinting at 'int' is signed here? And, that signed overflow is UB in C? Real question: Ignoring what the ISO C language spec says, are there any modern hardware platforms (say: ARM64 and X86-64) that do not use two's complement to implement signed integers? I don't know any. As I understand, two's complement correctly supports overflow for signed arithmetic. I might be old, but more than 10 years ago,…

AI rewrote to avoid undefined behavior:

  int average(int x, int y) {
    long sum = (long)x + y;
    if(sum > INT_MAX || sum

Re: Zlib-rs is faster than C

#258

Post by vlovich123 »

est31 wrote: post #20
hermanradtke wrote: post #15

Earlier quoted context omitted.

> basically written in C Unsafe Rust still has to conform to many of Rust’s rules. It is meaningfully different than C.

It has also way less tooling available than C to analyze its safety.

The things I’ve seen broadly adopted in the industry (i.e. sanitizers) are equally available in Rust. & Rust’s testing infrastructure is standardized so tests are actually common to see in every library.

Re: Zlib-rs is faster than C

#259

Post by School-Cotton »

YZF wrote: post #18

I found out I already know Rust: unsafe { let x_tmp0 = _mm_clmulepi64_si128(xmm_crc0, crc_fold, 0x10); xmm_crc0 = _mm_clmulepi64_si128(xmm_crc0, crc_fold, 0x01); xmm_crc1 = _mm_xor_si128(xmm_crc1, x_tmp0); xmm_crc1 = _mm_xor_si128(xmm_crc1, xmm_crc0); Kidding aside, I thought the purpose of Rust was for safety but the keyword unsafe is sprinkled liberally throughout this library. At what point does it really stop mat…

> I thought the purpose of Rust was for safety but the keyword unsafe is sprinkled liberally throughout this library.

This is such a widespread misunderstanding… one of the points of rust (there are many other advantages that have nothing to do with safety, but let’s ignore those for now) is that you can build safe interfaces, possibly on top of unsafe code. It’s not that all code is magically safe all the time.

Re: Zlib-rs is faster than C

#260

Post by fooker »

vlovich123 wrote: post #255
jandrewrogers wrote: post #251

Earlier quoted context omitted.

The example here is trivially safe but more general SIMD safety is going to be extremely difficult to analyze for safety, possibly intractable. For example, it is perfectly legal to dereference a vector pointer that references illegal memory if you mask the illegal addresses. This is a useful trick and common in e.g. idiomatic AVX-512 code. The mask registers are almost always computed at runtime so it would be effec…

Presumably a bounds check on the mask could be done or a safe variant exposed that does that trick under the hood. But yeah I don’t disagree that it’s “safe SIMD” is unlikely to scratch the itch for various applications but hopefully at least it’ll scratch a lot of them enough that the remaining unsafe is reduced.

No, a bounds check beats the purpose of simd in these cases
Post reply on HN