Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Architecture of Serverless Data Systems

jack-vanlightly.com

Post reply on HN
Print view View original

21–30 of 45 posts

Re: The Architecture of Serverless Data Systems

#21

Post by Dave3of5 »

ddorian43 wrote: post #20
Dave3of5 wrote: post #19

Earlier quoted context omitted.

> No, you get a DBMS and only change the storage underneath I think what you mean is what we have implemented a side channel DBMS which holds a copy which you use for the transactionality. It's a terrible approach I would not do this at all you don't get any benefit from using s3 here. This is not to say you can't use s3 to pull large blob storage off the DB and reference it in the DB I'm talking about the entire DB…

> I'm talking about the entire DB as s3. Yes, it can be done, except for WAL append & COLD GET. You "just" have to re-architect everything in the storage layer. Do you have anything specific in mind besides the 2 things I mentioned?

You haven't given specific of this to be blunt no idea what you mean.

I already said my points about the problems your follow-up comments have not addressed those.

Also there is no such thing in s3 as a cold GET there is a cold startup for a lambda. The latency from s3 on a get is orders of magnitude poorer that EBS or LSSD and should be used "infrequently".

Re: The Architecture of Serverless Data Systems

#22

Post by ddorian43 »

Dave3of5 wrote: post #21
ddorian43 wrote: post #20

Earlier quoted context omitted.

> I'm talking about the entire DB as s3. Yes, it can be done, except for WAL append & COLD GET. You "just" have to re-architect everything in the storage layer. Do you have anything specific in mind besides the 2 things I mentioned?

You haven't given specific of this to be blunt no idea what you mean. I already said my points about the problems your follow-up comments have not addressed those. Also there is no such thing in s3 as a cold GET there is a cold startup for a lambda. The latency from s3 on a get is orders of magnitude poorer that EBS or LSSD and should be used "infrequently".

Imagine using rocksdb backed by S3. You store only sstables in S3. You append the WAL in EBS and archive it in S3. You cache blocks in local SSD. If you want faster WAL, you append the WAL in local-ssd and use multi-az replication in your db and fsync WAL to EBS less frequently than locally.

> What about transactionality, partial updates, running multi document queries, consistency of the whole set of documents.

You do that in another layer on top. The filesystem doesn't provide transactions, yet you do them on a layer on top.

> In terms of scalability there are, limits 3500rps per key prefix.

The S3 metadata is (was?) sharded on key-prefix. You fix this by using more prefixes. By hashing the filenames or something.

> The latency from s3 on a get is orders of magnitude poorer that EBS or LSSD and should be used "infrequently".

Yes, it is. You should use a local SSD for most things.

Re: The Architecture of Serverless Data Systems

#23

Post by dangoodmanUT »

ddorian43 wrote: post #17
Dave3of5 wrote: post #16

A bit too waffling for me to read all but I would like to make a small comment. Why are more and more devs trying to use s3 as a general purpose DB? Working on a system right now where the architects have made this mistake it has insanely poor performance (High latency) and lack any proper ACID compliance. I've now been asked to "make it faster" and the answer is to switch back to an actual DBMS. > Top tier SaaS serv…

> You have to rewrite a whole DBMS on top of s3 itself or use redshift to get these things. No, you get a DBMS and only change the storage underneath. You can't use S3 for appending to WAL though. All those can be fixed besides the latency for a cold GET from S3 and appending WAL to S3.

You defintiely can use S3 for appending to a WAL (I've done it), they have read-after-write consistency

Re: The Architecture of Serverless Data Systems

#24

Post by Cthulhu_ »

fergie wrote: post #6

Two observations: - "serverless" is a really bad name for these systems. As is often commented, some variation of "somebody-elses-server" would be better. - Cost wasn't mentioned in the article, but the cost of renting databases and search-indices is still really high, even though these technologies are no longer the new hotness.

Eh, you can get upset about the industry using inaccurate words but we'd be here for a while then. AI isn't AI but machine learning, cryptocurrency isn't a currency, cloud servers aren't actually floating water vapor, the Dutch political parties that have freedom and/or democracy in their names aren't actually for freedom or democracy, and serverless actually uses servers. Let it go, just use the accepted term so that everyone knows what you mean, but remain aware of what it really means and don't buy into the hype or the FUD.

Re: The Architecture of Serverless Data Systems

#25

Post by amelius »

fergie wrote: post #6

Two observations: - "serverless" is a really bad name for these systems. As is often commented, some variation of "somebody-elses-server" would be better. - Cost wasn't mentioned in the article, but the cost of renting databases and search-indices is still really high, even though these technologies are no longer the new hotness.

Serverless means you have a network cable that ends somewhere abruptly. The open-end will cause reflections, so you can use this technology to store data for brief moments.

Re: The Architecture of Serverless Data Systems

#26

Post by ddorian43 »

dangoodmanUT wrote: post #23
ddorian43 wrote: post #17

Earlier quoted context omitted.

> You have to rewrite a whole DBMS on top of s3 itself or use redshift to get these things. No, you get a DBMS and only change the storage underneath. You can't use S3 for appending to WAL though. All those can be fixed besides the latency for a cold GET from S3 and appending WAL to S3.

You defintiely can use S3 for appending to a WAL (I've done it), they have read-after-write consistency

You're doing it for large OLAP writes. But as soon as you do OLTP or small writes it will become very slow & expensive.

See warpstream as example https://news.ycombinator.com/item?id=37036291

Re: The Architecture of Serverless Data Systems

#28

Post by lysecret »

Dave3of5 wrote: post #16

A bit too waffling for me to read all but I would like to make a small comment. Why are more and more devs trying to use s3 as a general purpose DB? Working on a system right now where the architects have made this mistake it has insanely poor performance (High latency) and lack any proper ACID compliance. I've now been asked to "make it faster" and the answer is to switch back to an actual DBMS. > Top tier SaaS serv…

The serverless pitch is extremely appealing to many companies. Also, many serverless offerings seem like a great choice at the beginning. But at some point, you will want SQL features like joins, groups, secondary indexes, foreign keys etc.

The missing link is really a serverless postgres (which many are working on but nothing has impressed me so far.

Re: The Architecture of Serverless Data Systems

#29

Post by 2mol »

lysecret wrote: post #28
Dave3of5 wrote: post #16

A bit too waffling for me to read all but I would like to make a small comment. Why are more and more devs trying to use s3 as a general purpose DB? Working on a system right now where the architects have made this mistake it has insanely poor performance (High latency) and lack any proper ACID compliance. I've now been asked to "make it faster" and the answer is to switch back to an actual DBMS. > Top tier SaaS serv…

The serverless pitch is extremely appealing to many companies. Also, many serverless offerings seem like a great choice at the beginning. But at some point, you will want SQL features like joins, groups, secondary indexes, foreign keys etc. The missing link is really a serverless postgres (which many are working on but nothing has impressed me so far.

Sorry if this is ignorant, but I never understand what serverless Postgres means. What's different from a hosted Postgres instance? Some scaling characteristics or the fact you interact with it via an API instead of some library, ORM, or plain SQL?

Re: The Architecture of Serverless Data Systems

#30

Post by aserafini »

ahartmetz wrote: post #10
socketcluster wrote: post #5

I'm currently working on a server-less, no-code multi-tenant platform. I'm still unsure if I should aim for full no-code or go for low-code. So far it's possible to build complex apps with it using only HTML tags (web components). Although it also exposes a CRUD interface, I haven't promoted this aspect as I feel it detracts from the huge time-saving and maintenance benefits which come with building apps using only d…

> serverless no-code Heck, at this point, it's time for a software platform called softwareless.

Paper and pen?
Post reply on HN