Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turning the database inside-out (2015)

martin.kleppmann.com

Post reply on HN
Print view View original

71–80 of 97 posts

Re: Turning the database inside-out (2015)

#71

Post by oa335 »

dagss wrote: post #39
williamdclt wrote: post #35

Earlier quoted context omitted.

> Something like "int identity" except that the int is assigned during commit, so that you have guarantee that if you see IDs 5 and 7, then 6 will never show up I don't think that's possible, nor is it something you should actually need. If two transactions tx1 and tx2 are concurrent (let's say tx2 begins after tx1 and also finishes after tx1), then tx2 has done some work without access to tx1's data (as tx1 hadn't c…

You explain why I don't need them for a very different usecase than what I refer to. My point is I want a new primitive -- a pub/sub sequence number -- to avoid having Kafka around at all. What Kafka does is "only" to generate and store such a sequence number after all (it orders events on a partition, but the sequence number I talk about is the same thing just different storage format). So also you do need it in the…

It sounds like the Log Sequence Number in Postgres is what you are looking for. If you subscribe to a Postgres publication via logical replication, each commit will be emitted with a monotonically increasing LSN.

Re: Turning the database inside-out (2015)

#72

Post by timacles »

chikere232 wrote: post #36

Isn't this essentially how a modern transactional database works anyway? All mutations end up in the Write Ahead Log (WAL) and you can replicate or back up that to be able to recover the state at a point in time?

Thats how the database works internally, thats not how the interface to the database works.

Apps dont know about the WAL log, and the database has internal logic for processing the WAL. Things like transactions, rollback etc...

Re: Turning the database inside-out (2015)

#73

Post by guskel »

agumonkey wrote: post #70
timacles wrote: post #68

Earlier quoted context omitted.

Who besides Oracle offers this stuff though? Yeah Oracle has a bunch of nice features, it also costs a gajillion dollars that no one besides a large enterprise can afford.

time based snapshots are in datomic and also possible in other dbms (maybe via extensions) for the rest i don't know

XTDB (inspired by datomic) also has bitemporal queries.

Re: Turning the database inside-out (2015)

#75

Post by crabbone »

reubenmorais wrote: post #63
crabbone wrote: post #59

Earlier quoted context omitted.

> perfectly That's a joke, right? If you tried to live in this world, as a human being, not in some abstract database-building sense, you'd be completely lost in the first second of your existence, and would probably die in minutes because your body parts would "forget" how to function properly. We make sense of the world because we have "object permanence", which requires the concept of larger things made up of comp…

> If all you can do is transitions, you don't know upon transitioning if you still have a keyboard you are typing on or not. This also means that if you want to be able to function somehow in this world, then after each transition, you'd have to reassess all properties of all interesting aspects of the world just to make sure they are still there. You're assuming that at every timestep all transitions are equally val…

If you want to build a stateless system with transitions, you are limited to regular expressions (finite automata).

I don't think you'd like to write real-life applications using a regular language. It might be interesting as an experiment, but life without recursion or potentially infinite loops sounds bleak... Just imagine the struggle of creating some generic containers like trees / hash-tables.

I don't think it's impossible, and may be even interesting to see how far one can take such a language to make it useful, but enjoyable this is not...

Re: Turning the database inside-out (2015)

#76

Post by mike_hearn »

timacles wrote: post #68
mike_hearn wrote: post #50

Earlier quoted context omitted.

Depends which industry. If you look at a lot of non-tech industry then they'll use a commercial DB with all those features in place already, rather than hacking up their own data layer. A few years ago I spent some time in the enterprise finance space, and learned some unfashionable tech you don't see talked about on Hacker News much. It left me with a new appreciation for what goes on there. A staggering amount of t…

Who besides Oracle offers this stuff though? Yeah Oracle has a bunch of nice features, it also costs a gajillion dollars that no one besides a large enterprise can afford.

I got curious about this and took a look at some pricing calculators. The results are pretty counter-intuitive.

Compared to PostgreSQL that you run yourself, it's expensive because PostgreSQL you run yourself costs nothing if you assume your time is free. But, how many people want to run it themselves? Especially as Postgres isn't much fun to admin (fiddling with vacuuming, setting up replication by hand, managing major version upgrades etc and you may not be able to scale this way).

So in reality a lot of companies and especially startups these days pay Amazon to run the database for them, and so then the cost question is how much more does it cost for a cloud hosted Oracle DB vs a cloud hosted Postgres DB?

Well, an 8 vCPU hosted RDS Postgres in AWS with 32 GB of memory and 100 GB of storage plus another 200 GB of backup storage - so one less powerful than a local DB on my laptop - costs $1,200/month in US East. That's expensive! AWS doesn't let you scale CPU and RAM independently, so I tried to pick something in the middle. For only 100 GB of data you probably don't need 4 physical cores.

So then I checked the OCI (Oracle Cloud) price calculator and specced out a similar database. I picked autonomous serverless (i.e. fully managed), transaction processing+mixed, autoscaling with 8 ECPUs and same amount of primary/backup storage. They don't let you spec RAM independently, I guess because it's a shared DB so RAM usage is transient and not a VM allocation. The cost came to ~$800/month - that's significantly cheaper than RDS Postgres despite that Oracle DBs have drastically more features. Many of which are optimizations that can reduce your database load anyway, so presumably you need more Postgres cores to match the equivalent performance if those features are used smartly (honestly I haven't ported an app between postgres and oracle so I don't have experience with this).

This is pretty surprising. I'd have expected an Oracle DB to cost more, not less. Auto-scaling is part of it (cost is double RDS if you turn that off), but then again, this is possible because Oracle has more multi-tenant and resource isolation features to begin with so it's reasonable to share a database server and overcommit CPU. With AWS it's a full VM so you have to stop the db server manually if you want to save money. Also OCI is a cheaper cloud than AWS as it has less brand recognition I guess. This feels a bit like Amazon is exploiting people's mental defaults. Lots of devs think AWS and Postgres are the only cloud+db combination that is reasonable to consider, and apparently they charge on that basis?

I haven't specced out what a hosted bare metal cluster would cost. You can't cluster Postgres in the same way anyway (multi-write master with full SQL, no sharding).

Re: Turning the database inside-out (2015)

#77

Post by lamp_book »

Xenoamorphous wrote: post #44

Does any one have some resources where a real, practical example is implemented? Because I can only find fairly theoretical resources but not real world examples. Say, like how some simple CMS would work with a datastore like this. What does the event to update the headline of an article look like? How are integrity constraints enforced, e.g. an article can't reference an author that doesn't exist? Things like that.

Similar enough is double entry bookkeeping and generally I think event sourcing is more common for fraud detection, log analysis for security, etc. - use cases where how the application got in its state is as important as the state.

Re: Turning the database inside-out (2015)

#80

Post by agumonkey »

guskel wrote: post #73
agumonkey wrote: post #70

Earlier quoted context omitted.

time based snapshots are in datomic and also possible in other dbms (maybe via extensions) for the rest i don't know

XTDB (inspired by datomic) also has bitemporal queries.

thanks I couldn't remember that name
Post reply on HN