71–80 of 87 posts
Given how branchless algorithms are helping optimize not just network transport (compression) and even OS system libs (no citation for this one, but I’ve heard), that I really wish colleges begin teaching this along with DS/Algo course material.
Earlier quoted context omitted.
So now the government needs to compel a corporation to hand over some data, because they are no longer able to read it straight off the wire like they could before. That sounds like a significant improvement to privacy.
People trafficking drugs into Australia were using a secure, encrypted messaging service developed by a private third party provider. They eventually found out that the third party provider was in fact the Australian Federal Police, reading all their messages in clear and in real time. The government only needs to compel a corporation if that corporation has an adversarial relationship with them. We have tried the ce…
"This means that whenever a user visits a website on Cloudflare that has ECH enabled, no one except for the user, Cloudflare, and the website owner will be able to determine which website was visited. Cloudflare is a big proponent of privacy for everyone and is excited about the prospects of bringing this technology to life.' This isn't privacy. This is centralized snooping. It's like Google's approach to third party…
Another HN hot take about the Cloudflare bogeyman. The CDN can't give you content you're asking for without knowing which content you're asking for. This improvement prevents your ISP and the government from reading your packets to get that same information.
I think these are important issues and worth talking about.
Earlier quoted context omitted.
It is about moving the trust. > This means that whenever a user visits a website on Cloudflare that has ECH enabled, no one except for the user, Cloudflare, and the website owner will be able to determine which website was visited. So you must use entity which controls the DNS and this entity makes the request further for actual website. Feels like just worse VPN.
> It is about moving the trust. Trust isn't being moved, though. Cloudflare could, by design, always see what website you were accessing. The difference is with ECH, there is one less party (someone listening in on your internet traffic) that can see which hostname you're accessing.
Let me just stress that the effect of Zstandard on individual end-user latency is a rounding error. No user will ever go: “That was a quick loading web site. Must be Zstandard!”. The effect is solely Cloudflare having to spend x% less bandwidth to deliver the content, saving on their network and server resources.
If it saves them money, great. That also means resources saved, and that also means it's better for the planet, thus better for humanity. I'm failing to see the disadvantage.
A very nice feature of zstd is that it is seekable . So you could map that to HTTP Range requests and go crazy about it.
I don't think so? It's only seekable with an additional index [1], just like most other compression schemes. Having an explicit standard for indices is definitely a plus though. [1] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...
Source?
Earlier quoted context omitted.
I don't think so? It's only seekable with an additional index [1], just like most other compression schemes. Having an explicit standard for indices is definitely a plus though. [1] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...
>just like most other compression schemes Source?
I assume the seeking implies a significantly faster way to skip the first N bytes of the decompressed output without the actual decompression. "Significant" is here defined as a time complexity strictly less than linear time in the size of compressed or decompressed data in that skip, so for the RLE sequence like `10a90b`, seeking to the 50th byte should be able to avoid processing `10a` at all; reading `10a` but not actually emitting 10 copies of `a` doesn't really count.
Virtually every compression algorithm has two main parts, modelling and coding. Modelling either transforms the input into a more compressible form, or estimates how much is particular portion of the input likely. Those results are then coded into one or more sets of symbols with different probabilities, resulting in a compressed bit sequence. Both parts can be made as complex as needed and will likely to destroy any relation between the input and output byte. Many algorithms including Zstandard do come with some framing scheme to aid with data verification, but such framing is generally not enough for actual seeking unless each frame declares both the uncompressed and compressed size and can be independently decompressed from each other. That's what the Zstandard's seekable format actually does: both sizes for every frame, and an implicit promise that a new frame is generated every so often. (Zstandard frames are already defined to be independent.)
There do exist some compression schemes that retain the seekability by design. Texture compression is a good example; a massively parallel nature of GPUs means that a fixed-rate scheme can be much more efficient than a variable-rate and thus inherently sequential scheme. There are also some hybrids that use two interconnected algorithms for CPU and GPU respectively. But such algorithms are not a norm.
Earlier quoted context omitted.
Another HN hot take about the Cloudflare bogeyman. The CDN can't give you content you're asking for without knowing which content you're asking for. This improvement prevents your ISP and the government from reading your packets to get that same information.
The internet is moving towards a place where it might not be possible to self-host anything important without getting DDoS'd. Companies like Cloudflare provide a solution to this problem, but that also creates a crutch that means no effort is expended to solve the problem at the root, which means the day may come when you don't have any option left other than Cloudflare. I think these are important issues and worth t…
Earlier quoted context omitted.
>just like most other compression schemes Source?
Uh, I'm confused why you expect most compression schemes to be necessarily seekable after all. I assume the seeking implies a significantly faster way to skip the first N bytes of the decompressed output without the actual decompression. "Significant" is here defined as a time complexity strictly less than linear time in the size of compressed or decompressed data in that skip, so for the RLE sequence like `10a90b`,…
Wtf, you were the one who brought that up.
Earlier quoted context omitted.
Uh, I'm confused why you expect most compression schemes to be necessarily seekable after all. I assume the seeking implies a significantly faster way to skip the first N bytes of the decompressed output without the actual decompression. "Significant" is here defined as a time complexity strictly less than linear time in the size of compressed or decompressed data in that skip, so for the RLE sequence like `10a90b`,…
>I'm confused why you expect most compression schemes to be necessarily seekable after all. Wtf, you were the one who brought that up.