61–70 of 132 posts
Caches drive the internet. They are orders of magnitude more important than compression. 99.99% of requests hit a local cache. (1) Compression is important too. (1) I worked in a telco. Check it out for yourself!
It's a bit silly to oppose these two technologies... Caches also rely on compression to be efficient, they work hand in hand.
Earlier quoted context omitted.
A nice reversal of the meme where "the supervising professor takes all the credit and the army of grad students do the real work".
This algorithm is known as Huffman encoding.
Please pardon my ignorance... My understanding is that there are 1000s of different compression algorithms, each with their own pros/cons dependent on the type and characteristics of the file. And yet we still try to pick the "generically best" codec for a given file (ex. PNG) and then use that everywhere. Why don't we have context-dependent compression instead? I'm imagining a system that scans objects before compre…
There is some approach you can use sort of like this within PNGs themselves I think. It's been a while so I might be misrembering but effectively each "row" of data that is compressed can be encoded as the difference from the preceding row using 4 or 5 different operations. You can achieve better compression by brute forcing the possible operations used to encode the rows to find the "most compressible" output. Not q…
That's the approach used by tools like optipng and pngcrush.
> I gave up before implementing it but in the stub I left this comment to myself " A heuristic approach is to use adaptive filtering as follows: independently for each row, apply all five filters and select the filter that produces the smallest sum of absolute values per row.".
The same idea can be found in the PNG standard itself: "For best compression of truecolor and grayscale images, we recommend an adaptive filtering approach in which a filter is chosen for each scanline. The following simple heuristic has performed well in early tests: compute the output scanline using all five filters, and select the filter that gives the smallest sum of absolute values of outputs. (Consider the output bytes as signed differences for this test.) This method usually outperforms any single fixed filter choice. However, it is likely that much better heuristics will be found as more experience is gained with PNG." (quoted from the PNG specification, version 1.2)
Caches drive the internet. They are orders of magnitude more important than compression. 99.99% of requests hit a local cache. (1) Compression is important too. (1) I worked in a telco. Check it out for yourself!
So then people complain "oh but The Illuminati can see what websites I'm going to!" Yeah, and they still can with HTTPS, it's called statistical network traffic analysis. Decades of research papers show you can uniquely identify a client going to a random internet server (and the page they're browsing) just by sniffing a bunch of POPs. It's used by law enforcement and Five Eyes to identify internet users around the world. Even protocols that have countermeasures (TOR) don't stand up to it.
Caches drive the internet. They are orders of magnitude more important than compression. 99.99% of requests hit a local cache. (1) Compression is important too. (1) I worked in a telco. Check it out for yourself!
This is why I was flabbergasted by the nonsense internet takeover of HTTPS. You can't cache it. Probably 99% of web traffic is generic public content, but we can't cache any of it, because 1% of it needs to be private. So then people complain "oh but The Illuminati can see what websites I'm going to!" Yeah, and they still can with HTTPS, it's called statistical network traffic analysis. Decades of research papers sho…
Compared to transit, last mile bandwidth is effectively limitless and free. Cache fill at the edge is important, last mile caching not so much.
Earlier quoted context omitted.
This algorithm is known as Huffman encoding.
Yes, and it's named after graduate student Huffman (at the time) instead of after his supervisor/teacher Fano.
Caches drive the internet. They are orders of magnitude more important than compression. 99.99% of requests hit a local cache. (1) Compression is important too. (1) I worked in a telco. Check it out for yourself!
This is why I was flabbergasted by the nonsense internet takeover of HTTPS. You can't cache it. Probably 99% of web traffic is generic public content, but we can't cache any of it, because 1% of it needs to be private. So then people complain "oh but The Illuminati can see what websites I'm going to!" Yeah, and they still can with HTTPS, it's called statistical network traffic analysis. Decades of research papers sho…
Also, a lot of ISPs blackbox caching proxies were buggy and breaking websites.
Quanta magazine and all these pop science websites need to be stopped. (Not because popular science is bad, but because they do it badly, and the clickbait is insufferable)
I agree. They have a breathless tone to them that's quite annoying to me (I work in data compression as an academic, and I found this article uninspiring.) By the way, there was an old Soviet magazine called "Kvant" (Russian for Quantum, I think). I do not know Russian, but I have 2 collected volumes of selected articles from them. [1] [2] Their quality is astonishingly good, and high-level. The difference is this: T…
I'll just say up front that I am not an academic, and I enjoy the breadth of coverage in the Quanta articles. I would liken them to science articles in American Scientist. (Perhaps you don't like that either.) Yes, they are popularized, but they are still technical.
Are you bored by a description of an algorithm you know well? This article clearly describes the process of Huffmann encoding. It's not one of the most amazing discoveries, but is this topic ever going to be exciting? I'd say it's easier to follow their article than either Wikipedia or the top animation hit [1].
There are two other claims you make that appear baseless.
On the first point:
> Quanta magazine articles are written by journalists
The first bio I checked [2] is a Ph.D. mathematician. If they also write, that does not make them less qualified. I'll grant that the second bio I checked [4] was "only" a journalist, but the third was a professor in a named chair [5]. Just clicking, not searching for examples.
On the second point:
> oversimplifies the problem and gives no idea about the actual solution
In your reference [3], it describes the problem clearly and devotes several paragraphs to what looks like a sketch of a solution. Certainly it outlines the ingredients used. (Search for "To see how they arrived at their new upper limit." and "In their proof [...]".)
[1]: https://cmps-people.ok.ubc.ca/ylucet/DS/Huffman.html [2]: https://www.quantamagazine.org/authors/erica-klarreich/ [4]: https://www.quantamagazine.org/authors/kevin-hartnett/ [5]: https://www.quantamagazine.org/authors/stevenstrogatz/
Caches drive the internet. They are orders of magnitude more important than compression. 99.99% of requests hit a local cache. (1) Compression is important too. (1) I worked in a telco. Check it out for yourself!
This is why I was flabbergasted by the nonsense internet takeover of HTTPS. You can't cache it. Probably 99% of web traffic is generic public content, but we can't cache any of it, because 1% of it needs to be private. So then people complain "oh but The Illuminati can see what websites I'm going to!" Yeah, and they still can with HTTPS, it's called statistical network traffic analysis. Decades of research papers sho…
I don’t think the kind of traffic analysis you mention works as well as you think it does for identifying individual pages e.g. which tweet someone is viewing. Moreover it requires a level of technical sophistication that is beyond all but the most advanced countries, countries that tend to have some measure of rule of law.
Earlier quoted context omitted.
This is why I was flabbergasted by the nonsense internet takeover of HTTPS. You can't cache it. Probably 99% of web traffic is generic public content, but we can't cache any of it, because 1% of it needs to be private. So then people complain "oh but The Illuminati can see what websites I'm going to!" Yeah, and they still can with HTTPS, it's called statistical network traffic analysis. Decades of research papers sho…
HTTPS everywhere is not only about privacy but also about integrity. You don't want internet randos (including ISPs) to swap your content with ads, bitcoin miners and put other batshit crazy JavaScript in it. Also, a lot of ISPs blackbox caching proxies were buggy and breaking websites.