Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Unbelievable Demo

brendangregg.com

Post reply on HN
Print view View original

231–240 of 478 posts

Re: An Unbelievable Demo

#231

Post by beagle3 »

versteegen wrote: post #167
sgtnoodle wrote: post #140

Earlier quoted context omitted.

Zip files have all the metadata in a footer rather than a header. As a result, compressed files can be added and overwritten by appending to the file without disturbing already compressed data. Additionally, the "deflate" compression likely does not span across files, so files that did not change from version to version would have a similar compressed byte sequence, regardless of the order they were added to the arch…

> Additionally, the "deflate" compression likely does not span across files Clarification: .zip (unlike .tar.gz for example, or "solid" .7z) compresses each file separately, that's nothing to do with the compression algorithm used. In addition, DEFLATE, the LZ77-based compression which is by far most commonly used in .zip (and also by gzip) has a window size of 32kB (uncompressed). So yes, even if you used DEFLATE on…

On the other hand HIgh voltage sid collection (hvsc) distributes a zipped zip;

Each file is 1k-20k, of which there are 40,000 or so. But they are catalogued in 3-4 deep directories, so if you just zip them, the metadata takes 30% or so of the zip.

But the metadata does compress very well, so they zip it again.

Re: An Unbelievable Demo

#232

Post by kevincox »

maccard wrote: post #112
throwaway3699 wrote: post #75

Earlier quoted context omitted.

Implementing software updates where you don't want to ship entire binaries again (and only the diff) would be one. In some video games the assets are also packed into massive binaries, so you don't want to ship gigabytes of data because you replaced one icon. Sadly many games do this anyway nowadays.

One of the reasons games do this is the data is compressed, so a "patch" might be indistinguishable from a real update. Also, as a dev, you have no idea what version your users are updating _from_. You either need to generate some number of patches for every version you could be updating from, and figure out if you should just download the whole thing again in any of those cases anyway.

There are solutions to this!

The simplest one is generate patches for recent versions, where recent can be years in the past. It is a linear operation but you only run it on release so it probably isn't a huge cost. You can also use some heuristics such as if if diff is >20% of the file just stop and force users still on that version to do a full update.

A second option is using zsync[1]. zsync is basically a precomputed rolling checksum. The client can download this manifest and they download just the parts of the file they need. This way you don't care about the source, if there is any similarity they can save resources.

And of course these can be combined. Generate exact deltas for recent versions and a zsync manifest for fallback.

[1] http://zsync.moria.org.uk/

Side note: One nice thing about zsync is that the actual download happens from the original file using range requests. This is nice for caching as a proxy only needs to cache the new data once. Is there a diff tool that generates a similar manifest for exact diffs? So instead of storing the new data in the delta file it just references ranges of the new file.

Re: An Unbelievable Demo

#233

Post by vidarh »

thomashabets2 wrote: post #166

Heh. I've a few times asked questions, only to be pointed to my own blog for answers. To be fair, yes those articles were on the same topic, but I'm actually trying to take the next step here. :-)

I've found my own blog posts in the search results while searching for something more than once.

The best part is when my own blog posts actually have the answer I was looking for - at which point I start feeling old and forgetful.

Re: An Unbelievable Demo

#234

Post by greyhair »

I had a similar, smaller scale of that happen, in a sort of reverse direction a few years ago. My director dropped a resume on my desk for someone coming from a company I had worked at ten years prior, thinking I might have met them (it was a small organization). I didn't recognize the name, but skimmed through the resume quickly, and their primary claim on their work history was something I invented just six months before leaving the organization. I couldn't believe it. What were the odds of that resume ending on my desk? Basically zero, but what a huge mistake. And they didn't claim they had maintained and extended it, they claimed they had invented it! He didn't get called in for an interview.

Re: An Unbelievable Demo

#235

Post by kevincox »

oblio wrote: post #116
maccard wrote: post #112

Earlier quoted context omitted.

One of the reasons games do this is the data is compressed, so a "patch" might be indistinguishable from a real update. Also, as a dev, you have no idea what version your users are updating _from_. You either need to generate some number of patches for every version you could be updating from, and figure out if you should just download the whole thing again in any of those cases anyway.

> One of the reasons games do this is the data is compressed, so a "patch" might be indistinguishable from a real update. Does this happen with more advanced compression algorithms? I've rsynced zip files of different versions of internal software and the diff was always much, much smaller than the entire package.

In general yes. After the first difference the compressed streams will be basically random compared to each other. However there are numerous things that may avoid this.

For zip files each individual file is compressed independently. So unchanged files and prefixes don't need to be resent, even if once a file changes the entire tail end of it needs to be resent.

Some times compression algorithms "reset" periodically. For example the `gzip --rsyncable` patch. This basically resets the compression stream so that a change will only affect part of the compressed file. This does have a cost in terms of compressed size because the compressor can't deduplicate across resets. However if the resets are infrequent you can maintain fairly good delta transfer with little space overhead.

Additionally some delta transfer tools detect common compression and decompress the file "in transfer", performing the delta checks on the original file.

Re: An Unbelievable Demo

#236

Post by Macha »

alkonaut wrote: post #198
Aachen wrote: post #162

Earlier quoted context omitted.

> having to rapidly re-architect the tool around a non-GPLv3 licensed library ... or just go with it and have it be open source? The old version is already open and free for anyone to request the code of. No rush at that point, you can withhold updates for a little while while you rearchitect this or take the situation as it is and have the next few bugfix releases also fall under GPL until you get around to replacin…

> Quickly removing the code doesn't change the previously released versions' license. But would $BIG_CORP publish source on request for a proprietary product just because they built one version with a GPL library by mistake and later fixed it? Has this been successful, ever?

Isn't this how OpenWRT got started?

Re: An Unbelievable Demo

#237

Post by awwaiid »

Aachen wrote: post #162
Twirrim wrote: post #89

Years ago, I interviewed a candidate for a role on my team. As usual, one of the ways I break the ice with candidates is to get them to talk "war stories". The team he'd worked on had produced a tool that was only ever intended to be used by the team to solve a particular problem they had. It contained proprietary code. Unknown to the team, word had spread about the tool, and others had started to use it, including s…

> having to rapidly re-architect the tool around a non-GPLv3 licensed library ... or just go with it and have it be open source? The old version is already open and free for anyone to request the code of. No rush at that point, you can withhold updates for a little while while you rearchitect this or take the situation as it is and have the next few bugfix releases also fall under GPL until you get around to replacin…

The license of the previous code didn't magically become the GPL altogether, instead it by default became un-distributable. They were required to (a) stop distribution of the existing code since it at best had no clear license, and (b) if they wanted, going forward remedy the license by clearly making it GPL or doing the rewrite of the dependency. Or even reach out to the library author and ask for an LGPL or other alternative - there is sometimes (often?) some flexibility there.

The built-in conflict resolution in the GPL is no-distribution.

Re: An Unbelievable Demo

#238

Post by kevincox »

codeulike wrote: post #153
rubicon33 wrote: post #63

Earlier quoted context omitted.

I don't understand what bsdiff does, or is. I am a software developer and I frankly have no clue what I would ever use bsdiff for! I've read what it does (libraries for building and applying patches to binary files) and still don't really have a sense for what the purpose of this tool is. What are some real life use cases for it? When does a developer need such a tool?

Its old now but here's a description from Chromium about something they use called Courgette which is similar/related to bsdiff https://blog.chromium.org/2009/07/smaller-is-faster-and-safe... The explanation here is pretty fascinating https://www.chromium.org/developers/design-documents/softwar...

The really cool TL;DR here is that courgette "disassembles" the binary before diffing. Basically turning internal references into symbolic references. This way adding an extra instruction to a function won't affect all of the relative addresses in surrounding code.

Re: An Unbelievable Demo

#239

Post by mannykannot »

rodgerd wrote: post #28
galacticaactual wrote: post #21

TIL that people fly around the world demoing DTrace...

At the time, Sun were pushing it heavily as one of the two salvations for Solaris, so I'm not surprised. I have to say that my limited experience in dealing with Sun as a customer mirrors Brendan's comments around a remarkable arrogance, and it probably played no small part in their downfall.

This has me imagining a struggling engineer at Sun, trying to live up to the arrogance and falling short, who finds Brendan's work and decides to save his career by passing it off as his own. Then some high-level executive decides to make it one of the straws that will save the company...

This is a made-up story, so you are free to make up your own ending as to whether this was the world-travelling VIP, and if so, whether he had some partial or complete flashback on hearing Brendan's name. One thing we can be sure of: whoever 'carelessly' stripped the copyright notice out of Brendan's code had seen his name before (and it was, perhaps, the part of the code he was most familiar with!)

Re: An Unbelievable Demo

#240

Post by toyg »

V-2 wrote: post #185
oblio wrote: post #120

Earlier quoted context omitted.

As a non-native English speaker I absolutely hate English accents where consonants absolutely disappear for no good reason. Personally I'd rather have Americans "lean heavily on the R" than act like the letter doesn't exist (rhotic vs non-rhotic). I think it's another factor why American English is more popular than British English (besides the huge economic factor, the US economy being 5x the UK one), since their pr…

I like British accent for its aesthetics, but from pragmatic point of view it's not even a contest (from the perspective of a non-native speaker), I agree. It's not just the r's (in fact, some British accents are rhotic - around their South West, if I'm not mistaken?), there are all these glottal stops and whatnot. But, from my observations at least, there are also big discrepancies related to social class. When I mo…

The thing is that UK dialects are virtually unknown outside the country; they can be very different from Received Pronounciation, but diverge in ways that are still fundamentally predictable for a native English speaker (unless you wander into Scotland or Ireland).

American accents are more familiar because of Hollywood, so they tend to be less surprising; and likely because a lot of them were actually developed by people who learned English as a second language, they are often exaggerate in effect, very clear, and actually more regular (particularly on names, where UK "rules" are anything but).

This said, "deep south" US accents, when pushed hard, can become as inscrutable as certain UK dialects.

Post reply on HN