Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Announcing SurrealDB 1.0

surrealdb.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Announcing SurrealDB 1.0

#31

Post by threeseed »

anonzzzies wrote: post #26
ThinkBeat wrote: post #17

This feels like satire. and written by someone who should have spent more time studying the history of database systems. > Imagine a world where the majority of your backend logic is seamlessly embedded within the database itself This is not a good idea. It has been done many times and never quite caught on because it is not a good idea. From a security perspective it is a nightmare. Or if you do put all the correct…

> never quite caught on It was quite normal in 70s 80s and even 90s; all the ms sql, db2, oracle and as400 systems I encountered in those days had all or almost all logic as stored procs. Very large ones.

During the Java era the standard quickly became ORM instead of stored procedures.

Largely thanks to NeXT showing the way with EOF (the original ORM).

Re: Announcing SurrealDB 1.0

#32

Post by vlovich123 »

mpercival531 wrote: post #22
vlovich123 wrote: post #13

Is there any trick to implement live updates in a scaleable way? In the limit for a naive implementation, every mutation would have to check every subscriber to see if the change is relevant which seems like it would cause large bottlenecks for writes.

Listener callout does not have to be part of the synchronous writes. Say if you keep a changelog, listeners can be asynchronously notified. These checks can be batched and throttled as well to minimize call-outs. I would suspect this is what Google does with Cloud Firestore.

If you’re throttling, then your database can’t actually keep up with broadcasting changes. Throttling only helps ensure the system keeps working even when there’s a sudden spike.

Batching only helps if you’re able to amortize the cost of notifications by doing that, but it’s not immediately clear to me that that there’s an opportunity to amortize since by definition all the book keeping to keep track of the writes would have to happen (roughly) in the synchronous write path (+ require a fair amount of RAM when scaling). The sibling poster made mention of taking read / write sets and doing intersections, but I don’t think that answers the question for several reasons I listed (ie it seems to me like you’d be taking a substantial performance hit on the order of O(num listeners) for all writes even if the write doesn’t match any listener). You could maybe shrink that to log(N) if you sort the set of listeners first, but that’s still an insertion speed of m log(n) for m items and n listeners or O(mn) if you can’t shrink it). That seems pretty expensive to me because it impacts all tenants of the DB, not just those using this feature…

Re: Announcing SurrealDB 1.0

#33

Post by vlovich123 »

jitl wrote: post #20
vlovich123 wrote: post #13

Is there any trick to implement live updates in a scaleable way? In the limit for a naive implementation, every mutation would have to check every subscriber to see if the change is relevant which seems like it would cause large bottlenecks for writes.

One of the guys behind Convex explained the rough idea; I hope I do it justice. The strategy is to break the subscription up into listens based on the read-set ranges of the query. Then you put the individual read-set ranges into a system table that you index. Finally when writes happen, you notify all queries who's read-set intersects with the write-set. For example, say I have a query `SELECT * from block WHERE par…

Is there a talk you can point me to?

There’s several challenges with this approach that come up for me (which unless I’m mistaken is the naive approach of checking each write against a notification set).

The first is that maintaining the read set seems very expensive since it scales with the number of live queries installed. In a multi tenant DB, that seems tricky to manage in terms of memory used.

The second is that the computation of intersection with a read range seems potentially very expensive. Imagine you have a string column. You now have to do a string comparison for every insertion that might hypothetically match the query.

Finally, computing that read set correctly seems challenging (as you mention) and it’s not immediately clear to me it’s always tractable (eg could you do this with arbitrary complicated joins?).

Additionally, in your description, each write has an implicit table scan to do the select to find the intersection. That will tank write throughput even for small total intersection sets (eg there’s a reason LSM databases do deletes by inserting a tombstone instead of checking whether the data exists first, same with the merge operator in RocksDB - a db read in the write path significantly kills performance)

Re: Announcing SurrealDB 1.0

#34

Post by threeseed »

fiddlerwoaroof wrote: post #23
sroussey wrote: post #18

Earlier quoted context omitted.

Right, so could I use it to creat an Airtable clone?

Generating a table schema per airtable spreadsheet sounds like an anti-pattern to me.

I've never seen database that either (a) allows you to create that many schemas at all or (b) doesn't become less stable because of it.

One example of this is that the database may create a directory structure per schema which will then result in file system performance degradation or hitting end user limits.

Re: Announcing SurrealDB 1.0

#35

Post by pavlov »

Is it a database where the only date/time type is a melting clock, and a join is defined as “the chance encounter of a sewing machine and an umbrella on an operating table”?

Re: Announcing SurrealDB 1.0

#36

Post by 10000truths »

ThinkBeat wrote: post #17

This feels like satire. and written by someone who should have spent more time studying the history of database systems. > Imagine a world where the majority of your backend logic is seamlessly embedded within the database itself This is not a good idea. It has been done many times and never quite caught on because it is not a good idea. From a security perspective it is a nightmare. Or if you do put all the correct…

> Or if you do put all the correct isolation around the code to protect the database, then you have basically created an "app server" (old term) inside a database, and it would happily, run outside of the database since in essence it is already doing so.

You can drastically cut down on end-to-end latency by eliminating the network hop from your app server to your DB, especially in cases where you are forced to make multiple back-to-back DB queries per request.

Re: Announcing SurrealDB 1.0

#37

Post by dwb »

khuey wrote: post #30
benrutter wrote: post #7

Earlier quoted context omitted.

Is it claiming anywhere to be open source? I've heard an interview with the creators and they seemed pretty open about which aspects where open source and which aspects where shut down? I could be missing something from their branding but I'd rather see projects with non-open source licenses from the beginning that bait-and-switch license change when they want to become profitable. (Although, I'd much rather just see…

> Is it claiming anywhere to be open source? They have a page on their website where they claim to be "an open source company" ( https://surrealdb.com/opensource )

Certainly debatable copy, but in total fairness that page is a list of actually-Open-Source licensed projects.

Re: Announcing SurrealDB 1.0

#38

Post by hakanito »

ThinkBeat wrote: post #17

This feels like satire. and written by someone who should have spent more time studying the history of database systems. > Imagine a world where the majority of your backend logic is seamlessly embedded within the database itself This is not a good idea. It has been done many times and never quite caught on because it is not a good idea. From a security perspective it is a nightmare. Or if you do put all the correct…

A world where the database is seamlessly embedded in the backend code instead would be more useful. Oh wait…

Re: Announcing SurrealDB 1.0

#39

Post by omneity »

grounder wrote: post #29
omneity wrote: post #16

Strong vibes from RethinkDB. I hope SurrealDB will have a different fate, I'm a big big fan of live queries and change feeds and don't like most existing implementations.

Can you give a quick elaboration on what you mean about the fate of RethinkDB? I'm not familiar with it.

The project didn't really take off as a business and the team joined Stripe. See the "History" section. https://en.wikipedia.org/wiki/RethinkDB

Re: Announcing SurrealDB 1.0

#40

Post by obinnaspeaks »

zamazan4ik wrote: post #19

1.0 version but https://github.com/surrealdb/surrealdb/issues/1548 is still open :)

A lot of the issues here have been resolved in documentation. For example there’s now new docs on permissions and deployment also more information on backups. Others are currently being worked ;)..

The issue is still open as we haven’t resolved all of it yet so makes a good reference.

Post reply on HN