Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Orthogonal Persistence

github.com

Post reply on HN
Print view View original

11–20 of 38 posts

Re: Orthogonal Persistence

#11

Post by kragen »

082349872349872 wrote: post #9
usrusr wrote: post #4

Earlier quoted context omitted.

Reminds me of the discussions on hn when Intel Optane wasn't quite dead yet. Those always seemed to end with the conclusion that if separation between volatile and persistent memory had not been forced on us by technological reality, it would be a concept we'd better have invented at some point.

I was tangentially involved with an orthogonally persistent OS, and we indeed had had to reinvent a distinct journalling channel and special optionally-volatile storage, for DBMS-style applications.

other reasons to need optionally-volatile storage include secure encryption key generation (reusing randomness often fatally compromises it) and device drivers (if you restore the internal state of your device driver from a checkpoint, but not the state of the device, you will probably crash the system the next time the driver tries to frob the device)

despite this, virtual machine checkpoints in qemu work well enough for many purposes

Re: Orthogonal Persistence

#12

Post by marcosdumay »

qazxcvbnm wrote: post #6

> Transactions are not modular because every function needs to know whether it’s already in a transaction or not, to be conscious of what global entry point in a completely different module owns the transaction. I fail to understand the section about why transactions are unmodular. I've never encountered transaction code where the initiator of the transaction would affect the computation; could anyone elucidate this?

You can't write this in any random function that you don't know who will call:

do $$

begin transaction;

update page set change_count = change_count + 1 where page_id = 1;

if (select change_count = 100 from page where page_id = 1) then

    rollback;
else

    commit;
end if;

$$

Re: Orthogonal Persistence

#16

Post by cmrdporcupine »

usrusr wrote: post #4
convolvatron wrote: post #2

having this as a model would be lovely and I agree wholeheartedly with the exposition. its interesting though to think about what impact this model would have on programming. a lot of our processing and tooling are built around the notion that programs are _almost_ right, and that we can bring them in and out - and hopefully in the process our precious data hasn't been mangled. when we express state directly in progr…

Reminds me of the discussions on hn when Intel Optane wasn't quite dead yet. Those always seemed to end with the conclusion that if separation between volatile and persistent memory had not been forced on us by technological reality, it would be a concept we'd better have invented at some point.

I am not sure about this?

Volatile memory is at this time merely an outgrowth of the uptime of the system. Back when people routinely turned their machines "off and on again", it became part of that convention. But now uptime can be measured in years, and even personal laptops can enter and exit suspended state for weeks on end without clearing volatile memory.

What we have developed in software systems to accommodate this on long running processes is garbage collection.

If the volatile/non-volatile distinction had never developed, all that would have happened is that R&D into garbage collection would have been more intense, and earlier.

In fact Lisp had garbage collection from day 1.

Systems like Smalltalk were also built from the ground up on an image-based model where all reachable state was persistent.

In other words: transient data does not necessitate volatile memory. It necessitates garbage collection, though. (And likely also a distinction in programming between "performant" memory areas and non-performant, assuming our NV storage is the latter.)

In a way, programmers having to deal with their garbage upfront and not relying on "have you tried turning it off and on again?" could have created better software engineering practices earlier? Maybe?

Re: Orthogonal Persistence

#17

Post by jerf »

qazxcvbnm wrote: post #7

> Persistence is Orthogonal to the Data Model, ... I have some experience with a custom data runtime where the persistence is orthogonal to the data model, with silhouettes reminiscent of the described solutions in many of the features of my system, including multiple orthogonal/model-agnostic persistence backends, automatic data synchronisation, persistable executions, automatable schema changes, automatic reactivit…

Or, to put it another like, like visual programming, like "programming languages should be able to wear syntaxes like themes", like "there ought to be some sort of nocode type solution with all the power of conventional programming but easy enough for anyone to pick up", there are reasons why this is not how all programming works already. Good ones and big ones. And none of those reasons are that nobody has had the idea before or put work into implementing it. If you want to succeed with an approach like this, you're going to need to understand them.

To be honest, such experience as I've had with automated persistence has generally actually strongly convinced me of the opposite, that it is a positive good that we do not get persistence everywhere. Consider the understanding that we get from functional programming that state is generally dangerous and to be carefully managed. Pervasive persistence fights hard against that careful management. Now state is not just in your program up until the OS process is terminated, but it's all permanently and automatically persisted. You get a huge new class of bugs involving path dependence on what bits of code were running across what bits of state when, and who ran which versions, and you hit them all the time, and they are nightmares to debug. At least when the program has the courtesy to completely cease existing and leave some particular concrete bit of state behind for the future, and then run through your code to load it back from that location, you have boundaries, and procedures for minimization and reconstruction. I actually shy away from too much automated persistence, and also have a very skeptical eye on the ever-present promise of memory that is as fast as RAM but persists like SSDs... I rather expect the computing world will discover that "rebooting" is not just a crutch, but actually a pretty fundamental and useful tool. However much in theory your software should never need it, in practice it's just too useful.

That said, best of luck to those jousting with this windmill. I'm not saying don't joust, people in general probably don't joust enough, I'm just saying, learn the history of why this hasn't worked before and learn the challenges. Success is at the very least more likely if one learns from the previous efforts.

Re: Orthogonal Persistence

#18

Post by geophile »

That phrase certainly brings back memories, from when I worked at an object-oriented database startup.

The object-orientation was actually pretty unimportant, (except for those products that brought in persistence via inheritance -- so not really orthogonal). No, the point was adding a new storage class to programming languages.

I worked at Object Design, and we had (IMHO) an incredibly elegant approach. In our approach, persistence really was orthogonal to type, for C/C++. If you want a FooBar, you would write "new FooBar(123)". That gives you a FooBar in the heap, disappears at process end (or on deletion), etc. Or you could write "new(db) FooBar(123)", and then on commit (we had transactions of course), the FooBar would be in the database, and accessible by other processes.

A page-faulting mechanism would bring in pages containing locations that your program referenced. That itself was very elegant.

But the really beautiful thing about this architecture was getting it to work in a 32-bit address space. We did some clever things about mapping portions of the address space during the faulting process to make things work transparently. (This problem pretty much disappears with a 64-bit address space.)

Separate from all that, we had a collection library, integrated with an OO query language. E.g., you could have a collections of widgets in your database, write "widgets[: weight It was a "database system" because our VCs wanted it to be. But it really wasn't. It really was a new storage class for C/C++, and later, for Smalltalk and Java.

Object Design also had a spectacularly talented group of engineers, many of whom came from MIT AI Lab/Symbolics.

Re: Orthogonal Persistence

#19

Post by Retr0id »

>servers only see a unindexed random-looking key value store

(quoted from the main readme)

I bet there's some fun attacks waiting to happen, related to watching for specific access patterns. Avoidable I'm sure, but I imagine it'll require awareness from application developers.

Re: Orthogonal Persistence

#20

Post by Retr0id »

Retr0id wrote: post #19

>servers only see a unindexed random-looking key value store (quoted from the main readme) I bet there's some fun attacks waiting to happen, related to watching for specific access patterns. Avoidable I'm sure, but I imagine it'll require awareness from application developers.

idk if the authors are reading this, but here's some feedback on the row encryption scheme:

1. Please use an AEAD!

2. IIUC, the current design exposes the hashes of the data values. This seems undesirable and I think you can avoid it.

Post reply on HN