Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’”

plus.google.com

Post reply on HN
Print view View original

201–210 of 265 posts

Re: Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’”

#201

Post by jdbernard »

sanderjd wrote: post #144
bananas wrote: post #115

Earlier quoted context omitted.

As follows. It's more how the abstraction works. XML: ->[byte stream]->[deserializer]->[bignum] JSON: ->[byte stream]->[json reader]->[string]->[deserializer]->[bignum] The latter is, well, wrong.

How does the [deserializer] step in the XML example know to call into [bignum], and why can't the [json reader] in the JSON example have that knowledge in the same fashion?

Because the XML document has a semantic meaning that is specifically designed for this application. It may even have a schema definition document which formally defines what types to expect. JSON, by contrast, has type definitions imposed on it by its nature as JavaScript code.

Re: Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’”

#202

Post by haberman »

eurleif wrote: post #198
haberman wrote: post #197

Earlier quoted context omitted.

Lack of Turing-completeness can be a feature. Take PDF vs PostScript. The latter is Turing-complete and therefore you cannot jump to an arbitrary page or even know how many pages the document has without running the entire thing first. By limiting expressiveness you also gain static analysis and predictability. It's not about limiting the potential of computers, it's about designing systems that strike the right bala…

>There really is no "better way" -- if JSON was executable then calling such an API would literally be giving it full control of your app and of the user's computer. Of course there's a "better way": running the code in a sandbox. You could do so using js.js[1], for example. (Of course, replacing a JSON API with sandboxed JS code is likely to be a bad idea. But it is possible.) [1] https://sns.cs.princeton.edu/2012/0…

You're right inasmuch as I shouldn't have implied that unsandboxed interpretation is the only option.

But my larger point still stands; the fundamental tradeoff is still "power of the payload" vs "guarantees to the container." Even in the case of sandboxed execution, the container loses two important guarantees compared with non-executable data formats like JSON:

1. I can know a priori roughly how much CPU I will spend evaluating this payload.

2. I can know that the payload halts.

This is why, for example, the D language in DTrace is intentionally not Turing-complete.

Re: Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’”

#203

Post by specialist »

lifeisstillgood wrote: post #41

What I like is the "I dont start prototyping till I have a good mental picture" I am currently stuck on a project I want to start becasue I cannot get it to fit right in my (future) head. And I am glad I am not an idiot for not being able to knock out my next great project in between lattes. (Ok, in direct comparison terms I am an idiot, but at least its not compounded)

  "A change in perspective is worth 80 IQ points."
  
  -- Alan Kay
My biggest hurdle solving new problems is divining a unifying, simplifying metaphor. Once you have the right notion, that Eureka! moment, everything falls into place, like magic.

Like how Kepler was able to fully explain Bache's astronomical data once he realized the planets orbits the sun.

Personal example: I used to write print production software. Placing pages onto much larger sheets of paper that get folded and bound into a book. A task called image positioning aka imposition. It took me years to figure out how to model the problem. Key insight was simulating the work backwards, from binding back to the press. Then when I showed the new solution to my coworkers, the response was "Well, duh."

Re: Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’”

#204

Post by eurleif »

haberman wrote: post #202
eurleif wrote: post #198

Earlier quoted context omitted.

>There really is no "better way" -- if JSON was executable then calling such an API would literally be giving it full control of your app and of the user's computer. Of course there's a "better way": running the code in a sandbox. You could do so using js.js[1], for example. (Of course, replacing a JSON API with sandboxed JS code is likely to be a bad idea. But it is possible.) [1] https://sns.cs.princeton.edu/2012/0…

You're right inasmuch as I shouldn't have implied that unsandboxed interpretation is the only option. But my larger point still stands; the fundamental tradeoff is still "power of the payload" vs "guarantees to the container." Even in the case of sandboxed execution, the container loses two important guarantees compared with non-executable data formats like JSON: 1. I can know a priori roughly how much CPU I will spe…

If excess CPU/non-halting behavior is the issue, you could run the code with a timeout.

Re: Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’”

#205

Post by haberman »

eurleif wrote: post #204
haberman wrote: post #202

Earlier quoted context omitted.

You're right inasmuch as I shouldn't have implied that unsandboxed interpretation is the only option. But my larger point still stands; the fundamental tradeoff is still "power of the payload" vs "guarantees to the container." Even in the case of sandboxed execution, the container loses two important guarantees compared with non-executable data formats like JSON: 1. I can know a priori roughly how much CPU I will spe…

If excess CPU/non-halting behavior is the issue, you could run the code with a timeout.

You could. But that has downsides also:

1. imposing CPU limits incurs an inherent CPU overhead and code complexity.

2. if those limits are hit, you can't tell whether the code just ran too long or whether it was in an infinite loop.

So now if we fully evaluate the options, the choice is between:

1. A purely data language like JSON: simple to implement, fast to parse, decoder can skip over parts it doesn't want, etc.

2. A Turing-complete data format: have to implement sandboxing and CPU limits (both far trickier security attack surfaces), have configure CPU limits, when CPU limits are exceeded the user doesn't know whether the code was in an infinite loop or not, maybe have to re-configure CPU limits.

Sure, sometimes all the work involved in (2) is worth it, that's why we have JavaScript in web browsers after all. But a Turing-complete version of JSON would never have taken off like JSON did for APIs, because it would be far more difficult and perilous to implement.

Re: Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’”

#206

Post by strictfp »

WalterBright wrote: post #170

Back in the bad old DOS days, instead of creating a file format for saving/loading the configuration of the text editor, I simply wrote out the image in memory of the executable to the executable file. (The configuration was written to static global variables.) Running the new executable then loaded the new configuration. This worked like a champ, up until the Age of Antivirus Software, which always had much grief ov…

Thank you for that anecdote, it made my day. Simply awesome.

Re: Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’”

#207

Post by lern_too_spel »

sparkie wrote: post #155
tedchs wrote: post #137

Why reinvent on-disk data formats when you can just make a file of protocol buffers? https://code.google.com/p/protobuf/

Why reinvent binary serialization when you could use ASN.1, or any of the thousand binary serialization formats that pre-date protobufs?

For that specific example, you can find a good discussion here: https://groups.google.com/forum/m/#!topic/protobuf/eNAZlnPKV...

Re: Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’”

#208

Post by vfclists »

What is it with HN commenters and their demented ability to send topics completely of track? I would have thought someone might have examined the code or what Linus is trying to implement and comment about it.

But here we have threads about Lua, why people hate XML and love JSON and all kinds if irrelevant issues which have been well hashed elsewhere ad nauseam. Why not restrict to an analysis of whatever it is Linus developing?

HN is getting truly annoying and sucky, if it isn't so already.

Re: Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’”

#209

Post by WalterBright »

strictfp wrote: post #206
WalterBright wrote: post #170

Back in the bad old DOS days, instead of creating a file format for saving/loading the configuration of the text editor, I simply wrote out the image in memory of the executable to the executable file. (The configuration was written to static global variables.) Running the new executable then loaded the new configuration. This worked like a champ, up until the Age of Antivirus Software, which always had much grief ov…

Thank you for that anecdote, it made my day. Simply awesome.

I learned a heckuva lot from reading the ADVENT Fortran source code. I was floored when I figured out how it was saving its configuration - such a brilliant idea. And in DOS it could be implemented in about 5 lines of simple C code. (Code size was critical in the old 64Kb days.)

The other huge thing I learned from ADVENT was polymorphism. The comment in the source code "the troll is a modified dwarf" was an epiphany for me.

Re: Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’”

#210

Post by WalterBright »

strictfp wrote: post #206
WalterBright wrote: post #170

Back in the bad old DOS days, instead of creating a file format for saving/loading the configuration of the text editor, I simply wrote out the image in memory of the executable to the executable file. (The configuration was written to static global variables.) Running the new executable then loaded the new configuration. This worked like a champ, up until the Age of Antivirus Software, which always had much grief ov…

Thank you for that anecdote, it made my day. Simply awesome.

I forgot to mention, on a floppy disk system, saving the configuration in the exe file made for fast loading of the program, since it didn't need to do extra floppy file operations to load the config.
Post reply on HN