Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Executable Is a SQLite Database

fzakaria.com

Post reply on HN
Print view View original

71–80 of 116 posts

Re: Executable Is a SQLite Database

#71

Post by Joker_vD »

XJ6w9dTdM wrote: post #53
inigyou wrote: post #37

Earlier quoted context omitted.

You can do any of this in-memory wife ELF of course, it's just that nobody does and nobody's written the tools for it (the most obscure tool I know of is `ld -r`, which links two or more .o files and produces a new .o file instead of an executable)

For sure, the main advantage is that since the SQLite file format is easy to use compared to ELF, it's more likely that such experiments would be even tried at all. I got another one: bundling multiple (completely separate) executables with shared dependencies in the same SQLite, selected by argv[0] when called, like what busybox does

Also known as a directory.

Re: Executable Is a SQLite Database

#72

Post by hantusk »

andai wrote: post #32

This whole article is fantastic, but already at the start, the SQLite virtual tables thing is blowing my mind. https://www.sqlite.org/vtablist.html You can "mount" your filesystem (or anything else) as a SQL database, wtf. That's amazing. This sounds like it could be extremely useful.

Check out https://hub.steampipe.io/, available as sqlite/postgtesql extensions

Re: Executable Is a SQLite Database

#73

Post by self_awareness »

> Two processes running the same SELF binary do not share text pages the way a normally-mmap‘d ELF does, because the bytes are copied out of the b-tree rather than mapped.

Isn't this the whole point of executable file formats? To avoid loading everything all at once, share immutable segments with other processes, be extremely fast when loading, and integrate with OS page fault handlers to load data on demand?

I don't understand this project at all.

Even if it's possible to query the executable using SQL, we can always just use the SQL frontend to interface with executable data. I mean, ELF as a backend format for the "executable database". Like osquery interfaces with the system using SQL language. Osquery doesn't convert all OS data structures to sqlite :)

Btw, I think SELF is already "coined" by Signed ELF executables - SELF files, existing for example on Playstation 3.

Re: Executable Is a SQLite Database

#74

Post by garganzol »

stillpointlab wrote: post #59
garganzol wrote: post #21

The copied vs. mapped memory situation is the only deal breaker in this experiment. Otherwise, file format unification would be a big step forward. The PE/COFF executable file format used by Windows (and some older Unix systems) is a relational database as well. The same goes to .NET assembly format - it's a relational database too. The wheel gets reinvented over and over again.

I was thinking that one could just write a loader that goes from SELF->ELF on load. You then pay the cost on startup. Probably not worth it for one-shot programs like ls/cat/etc, but for long running programs and daemons it might be workable.

Another option is to adapt a database engine to support page-aligned blobs, which is probably not that hard to achieve.

Re: Executable Is a SQLite Database

#75

Post by garbagepatch »

inigyou wrote: post #44
HackerThemAll wrote: post #18

Yeah, I have been amazed for an entire life how many tools the IT industry invented during last 50 years to just disguise a database. OK, in '70s, '80s or '90s when compute and storage resources were limited and every bit counted, specialized formats did make sense. But nowadays we'd save enormous efforts by just packaging stuff in SQLite databases. Microsoft's proprietary file formats (Office, Power BI etc.), OpenOf…

At first I thought it was still for a word document to be saved in SQL but then I thought about embedded images and it makes complete sense. Makes more sense for a Publisher/Impress app though as they have more self-contained objects. In Word/Writer the main text is one huge run of characters, how would you sqlize that effectively?

Current ooxml/odt formats use a zip file with xml files inside (plus other files like images). I'd guess one way would be to store the xml itself as a blob and another to use a json field, which Sqlite stores in a more efficient format called jsonb: https://sqlite.org/json1.html#jsonb

Re: Executable Is a SQLite Database

#76

Post by garganzol »

setheron wrote: post #63

(author) I'm enjoying the comments. When I published a short-paper with this idea in academic circles, the feedback wasn't so kind

Don't pay too much attention to the inner-circle feedback. Some people subconsciously use the tone of reception to control you, especially peers/colleagues are prone to this. They see someone having a bright idea, and they genuinely don't like it just because it makes them feel small in comparison. The thing they do next - they try to extinguish the spark by creating an illusion of worthlessness in your mind. By doing so, they are getting rid of a potential competitor on their own way to success. According to psychology, they start feeling normal again by reducing you, this is a natural compensatory mechanism ingrained in human archetypes.

The key thing here is to be able to discern between real and manipulated input information. For this, gut feeling usually works best: it spots that whiny, attention-insisting, importance-seeking, fear-inducing tone of a manipulator, but oftentimes the higher-level nervous system of the brain suppresses that signal (e.g. "How can a well-educated and charismatic person feel a bit off and responsibility-lacking sometimes? It cannot be right, so it must be something with my interpretation of reality. I bet they have the best intentions.")

An even better approach may be not to search for feedback at all, unless it's shared naturally without any strings attached.

Re: Executable Is a SQLite Database

#77

Post by stillpointlab »

garganzol wrote: post #74
stillpointlab wrote: post #59

Earlier quoted context omitted.

I was thinking that one could just write a loader that goes from SELF->ELF on load. You then pay the cost on startup. Probably not worth it for one-shot programs like ls/cat/etc, but for long running programs and daemons it might be workable.

Another option is to adapt a database engine to support page-aligned blobs, which is probably not that hard to achieve.

Of course there are many options, but the benefit of the SQLite approach is the existing tooling, support, etc. Like others have said, one could just write a SQL interface over a set of virtual tables derived from the actual elf (if one just wanted a CRUD-like interface to a binary).

All of the benefits of SQLite disappear once one diverges from the format in any way. At which point it would be better to ask: "what is the best first-principles implementation of this idea?", instead of "what is the minimal change to SQLite to achieve this specific narrow goal?"

Re: Executable Is a SQLite Database

#78

Post by setheron »

garganzol wrote: post #74
stillpointlab wrote: post #59

Earlier quoted context omitted.

I was thinking that one could just write a loader that goes from SELF->ELF on load. You then pay the cost on startup. Probably not worth it for one-shot programs like ls/cat/etc, but for long running programs and daemons it might be workable.

Another option is to adapt a database engine to support page-aligned blobs, which is probably not that hard to achieve.

I might look at this next; with Nix it's easy to explore and rebuild seamlessly ;P

Re: Executable Is a SQLite Database

#80

Post by hedgehog »

setheron wrote: post #78
garganzol wrote: post #74

Earlier quoted context omitted.

Another option is to adapt a database engine to support page-aligned blobs, which is probably not that hard to achieve.

I might look at this next; with Nix it's easy to explore and rebuild seamlessly ;P

I didn't understand the core issue, is it that due to non-alignment of the on-disk data you have to do a copy at load time, but if you could guarantee alignment of the actual blob content then you could use it directly?
Post reply on HN