Live data from Hacker News

Realm is a mobile database: an alternative to SQLite and key-value stores

github.com

31–37 of 37 posts

Re: Realm is a mobile database: an alternative to SQLite and key-value stores

#31
Is there any technical information on how the database works? For a supposed Sqlite replacement, there is a surprising lack of the information. The only reference to concurrency mode I have found is this:

> All changes to an object (addition, modification and deletion) must be done within a write transaction.

> Write transactions incur non-negligible overhead - you should architect your code to minimize the number of write transactions.

So is there a single write lock? If you try to start two transactions, will the second one be delayed or retried? Can reads proceed while the transaction is processing?

> Results instances are live, auto-updating views into the underlying data

So if I am doing x.first, x.second while the object is being updated in a second thread, will I get mixed state?

Is database crash proof? If the phone runs out of battery while in write transaction, will the database get damaged?

... so many questions

Re: Realm is a mobile database: an alternative to SQLite and key-value stores

#32
post #8
post #6

OT: What happened in the database space and why? In 2000 you would have had a hard time naming enough database systems to run out of fingers for counting. But in recent years database system seem to pop out of the ground like JavaScript frameworks. So what changed? Did the idea of a general purpose database system break because the usage scenarios became so diverse that we can no longer pretend that one size fits alm…

Realm has been available for many years and is actually really different from what else is out there by the fact that it is a real object database. It has just been flying under the radar a bit given that it has only been available for mobile devices. I for one think that it is awesome that it is now available for node.js as well. Especially with the new replication functionality that allows you to share live data be…

"Real" object databases — "OODBMS" — uses to be a thing, too. They still survive in some niches, like finance and health, though largely as legacy systems. Products like Objectivity, ObjectStore, POET, Caché (which is built on MUMPS), GemStone (Smalltalk) etc. were once big, expensive products that seemed poised to take over from old-hat SQL databases.

Those databases failed not necessarily because the technology was wrong, although they certainly made some choices that didn't help; those databases were often tightly coupled with opaque binary object serialization formats (with no way of looking into a database without going the binary route) that were in turn tightly coupled with the app's class hiearchy. For example, several of these actually post-processed your code (e.g. your Java bytecode) to insert serialization code, so that when you followed a relationship (e.g. "library.books[0].author"), it would automatically query under the hood and return the author as an object. They ended up being clunky to work with as a result; for example, in the beginning, very few of them had a query language, so the only way to query the data was by following relationships through your graph of objects.

Realm looks interesting as a new approach to the object database design, but at the same time seems a bit limited. It doesn't seem to have schemas or any kind of role-based permissions, and seems to make the same mistake of not including a query language (so you can't build a REPL for it, and every language binding needs its own query builder). The transaction semantics also seem to be entirely undefined (is it read-committed?). From what I can tell, it's also "offline first", in the sense that you have a local database that can sync to/from a server, but you're technically always working on an offline copy of the master data; while nice to have for a mobile app, it does limit what you can use it for. I also don't see any support for fine-grained updates. Every write you do seems to overwrite, which I assume means the last write always wins; how do you deal with conflicts, or implement things like counters, maps or arrays that need to be incrementally updated?

Re: Realm is a mobile database: an alternative to SQLite and key-value stores

#33
post #31

Is there any technical information on how the database works? For a supposed Sqlite replacement, there is a surprising lack of the information. The only reference to concurrency mode I have found is this: > All changes to an object (addition, modification and deletion) must be done within a write transaction. > Write transactions incur non-negligible overhead - you should architect your code to minimize the number of…

It's ACID & Serializable. A write will block all other writes, but multiple reads can occur without overhead or blocking due to MVCC. Reads are consistent within the runloop, but updates to your objects occurs when you start a write transaction, so you know you always work on the latest version of the data. Basically, you are in an implicit read transaction all the time, except when you are in a write transaction :-).

> So if I am doing x.first, x.second while the object is being updated in a second thread, will I get mixed state? So no you will not get mixed state.

> Is database crash proof? If the phone runs out of battery while in write transaction, will the database get damaged?

Yes it is crashproof. Fully ACID.

Any other questions?

This article goes a bit more into the details of threading: https://realm.io/news/threading-deep-dive/ And this one more into the core design: https://realm.io/news/jp-simard-realm-core-database-engine/

Re: Realm is a mobile database: an alternative to SQLite and key-value stores

#34
post #8

Earlier quoted context omitted.

Realm has been available for many years and is actually really different from what else is out there by the fact that it is a real object database. It has just been flying under the radar a bit given that it has only been available for mobile devices. I for one think that it is awesome that it is now available for node.js as well. Especially with the new replication functionality that allows you to share live data be…

"Real" object databases — "OODBMS" — uses to be a thing, too. They still survive in some niches, like finance and health, though largely as legacy systems. Products like Objectivity, ObjectStore, POET, Caché (which is built on MUMPS), GemStone (Smalltalk) etc. were once big, expensive products that seemed poised to take over from old-hat SQL databases. Those databases failed not necessarily because the technology was…

Realm certainly has a schema - anything that made you think it doesn't? It's ACID and Serializable.

It also has a query language. But as one of the core design principles is to be as native to the language and platform as possible, the query language takes different forms. In Java it's a fluent API, in Cocoa it's NSPredicates string based. And the JS string based is roughly identical to that as well.

It's absolutely designed to be offline first. I'm curious as to which limitations you see with that?

Operational Transform is used to resolve conflicts at the property level, so I would say it's much more fine-grained than most other systems that would overwrite entire objects. The specific resolution rules are very simple and intuitive and how to deal with custom needs are described here: https://realm.io/docs/realm-object-server/#conflict-resoluti.... There is ordered lists already and explicit counters are being exposed in the SDKs at the moment.

You might find this article useful as well: https://realm.io/news/eventually-consistent-making-a-mobile-...

Re: Realm is a mobile database: an alternative to SQLite and key-value stores

#35
post #34

Earlier quoted context omitted.

"Real" object databases — "OODBMS" — uses to be a thing, too. They still survive in some niches, like finance and health, though largely as legacy systems. Products like Objectivity, ObjectStore, POET, Caché (which is built on MUMPS), GemStone (Smalltalk) etc. were once big, expensive products that seemed poised to take over from old-hat SQL databases. Those databases failed not necessarily because the technology was…

Realm certainly has a schema - anything that made you think it doesn't? It's ACID and Serializable. It also has a query language. But as one of the core design principles is to be as native to the language and platform as possible, the query language takes different forms. In Java it's a fluent API, in Cocoa it's NSPredicates string based. And the JS string based is roughly identical to that as well. It's absolutely…

I couldn't, and still can't, find any documentation or "list of features" that mention schema support, query language or ACID semantics. The documentation is pretty sparse and disorganized.

(By query language I don't mean an opaque builder based on NSPredicates or whatever, I mean a query language, written as text, which is portable across clients.)

OT is great, but it doesn't have that until it's actually exposed. My pet use case is string edits, which absolutely need to be cleanly conflict-resolvable.

The inconvenience with offline-first is that by definition you're never operating on current data: You have to sync to a central server before the data is visible to others, and you're editing potentially stale data. The latency of inbound syncing is particularly dependent on the volume of changes being sent from upstream, which might require that a client, so as not to be overwhelmed with updates, subscribe to a very small subset of the entire dataset. For example, if you open up a UI that edits a single document, you might want to subscribe to changes to only that document, in order to update the UI in real time. I don't know if that's even possible with Realm, or if you can in fact run it in "classic client/server" (i.e. "online first") mode.

Re: Realm is a mobile database: an alternative to SQLite and key-value stores

#36
post #34

Earlier quoted context omitted.

Realm certainly has a schema - anything that made you think it doesn't? It's ACID and Serializable. It also has a query language. But as one of the core design principles is to be as native to the language and platform as possible, the query language takes different forms. In Java it's a fluent API, in Cocoa it's NSPredicates string based. And the JS string based is roughly identical to that as well. It's absolutely…

I couldn't, and still can't, find any documentation or "list of features" that mention schema support, query language or ACID semantics. The documentation is pretty sparse and disorganized. (By query language I don't mean an opaque builder based on NSPredicates or whatever, I mean a query language , written as text, which is portable across clients.) OT is great, but it doesn't have that until it's actually exposed.…

Why would you want a text-based query language if you are an idiomatic C# programmer? Every database product on that platform is queried through the standard LINQ syntax - that's why we implemented it that way.

Re: Realm is a mobile database: an alternative to SQLite and key-value stores

#37

Earlier quoted context omitted.

I couldn't, and still can't, find any documentation or "list of features" that mention schema support, query language or ACID semantics. The documentation is pretty sparse and disorganized. (By query language I don't mean an opaque builder based on NSPredicates or whatever, I mean a query language , written as text, which is portable across clients.) OT is great, but it doesn't have that until it's actually exposed.…

Why would you want a text-based query language if you are an idiomatic C# programmer? Every database product on that platform is queried through the standard LINQ syntax - that's why we implemented it that way.

I'm not an idiomatic C# programmer, but having a text-based language is really useful in all sorts of situations:

* Interactive REPL (think psql)

* Use outside of C#, e.g. small bash scripts for automation of small tasks such as deleting old records in a cron job

* Copy/paste a into Slack, Github issues etc. without having to drag along any dependent code or worrying that it's not a complete query

Query builders are nice, but eventually they have to compile down to something "neutral" and portable.

Post reply on HN