Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Architecture of Serverless Data Systems

jack-vanlightly.com

Post reply on HN
Print view View original

11–20 of 45 posts

Re: The Architecture of Serverless Data Systems

#11

Post by vbezhenar »

fergie wrote: post #6

Two observations: - "serverless" is a really bad name for these systems. As is often commented, some variation of "somebody-elses-server" would be better. - Cost wasn't mentioned in the article, but the cost of renting databases and search-indices is still really high, even though these technologies are no longer the new hotness.

My opinion is that serverless means you don't have to deal with servers (physical or virtual). You deal with higher-level entities. You don't install Ubuntu, you just provision database from web UI or terraform script. Hardware, OS and some software is managed for you. This allows consumer to deal with less complexity and this allows provider to be more flexible with resource allocations.

Re: The Architecture of Serverless Data Systems

#12

Post by bradgessler »

ahartmetz wrote: post #10
socketcluster wrote: post #5

I'm currently working on a server-less, no-code multi-tenant platform. I'm still unsure if I should aim for full no-code or go for low-code. So far it's possible to build complex apps with it using only HTML tags (web components). Although it also exposes a CRUD interface, I haven't promoted this aspect as I feel it detracts from the huge time-saving and maintenance benefits which come with building apps using only d…

> serverless no-code Heck, at this point, it's time for a software platform called softwareless.

I believe that’s called hardware.

Re: The Architecture of Serverless Data Systems

#13

Post by pdpi »

fergie wrote: post #6

Two observations: - "serverless" is a really bad name for these systems. As is often commented, some variation of "somebody-elses-server" would be better. - Cost wasn't mentioned in the article, but the cost of renting databases and search-indices is still really high, even though these technologies are no longer the new hotness.

EC2, k8s, etc all abstract the hardware away, but they still use the server as a core abstraction. Serverless does away with that abstraction entirely. It’s obviously nonsense from a low-level point of view (as you say: one of many classes of “somebody else’s server” products), but, as far as developer experience goes, it’s pretty accurate.

Re: The Architecture of Serverless Data Systems

#14

Post by bob1029 »

See also - Sql Server Hyperscale. We've been using this for about 18 months now and it feels like it has saved us a lot of hassle.

https://learn.microsoft.com/en-us/azure/azure-sql/database/h...

The only downside we can spot so far is the presence of a 100mb/s throttle for txn log writes in order to satisfy replication requirements. Beyond this, it is indistinguishable from an express instance on a local dev machine. You lose some of the other on-prem features when you go managed, but most new applications don't need that stuff anymore. The message broker pieces are the only ones I miss, but there are a lot of other managed options for that, and you can still DIY with a simple Messages table and 3-4 stored procedures.

On the read & reporting side, I see no downsides. You mostly get OLAP+OLTP in the same abstraction without much headache. If someone really wanted to go absolutely bananas with reporting queries, data mining, AI crap, whatever, you could give them their own geo replica in a completely different region of the planet. Just make sure they aren't doing any non-queries and everything should be fine.

For large binary data, we rely on external blob storage and URLs. The txn log write limit shouldn't feel like much of a restriction if you are using the right tools for each part of the job. Think about how many blob URLs you could fit within 100 megabytes. If you make assumptions about URL structure, you can increase this by a factor of 2-3x.

Re: The Architecture of Serverless Data Systems

#15

Post by m_mueller »

bradgessler wrote: post #12
ahartmetz wrote: post #10

Earlier quoted context omitted.

> serverless no-code Heck, at this point, it's time for a software platform called softwareless.

I believe that’s called hardware.

"a wrapper around somebody else's software on yet another person's servers"

or maybe how about "SaaSaaS"?

Re: The Architecture of Serverless Data Systems

#16

Post by Dave3of5 »

A bit too waffling for me to read all but I would like to make a small comment.

Why are more and more devs trying to use s3 as a general purpose DB?

Working on a system right now where the architects have made this mistake it has insanely poor performance (High latency) and lack any proper ACID compliance. I've now been asked to "make it faster" and the answer is to switch back to an actual DBMS.

> Top tier SaaS services like S3 are able to deliver amazing simplicity, reliability, durability, scalability, and low price because their technologies are structurally oriented to deliver those things. Serving customers over large resource pools provides unparalleled efficiency and reliability at scale

In terms of simplicity using s3 is anything but simple. Sure the CRUD api is simple but there are a bunch of gotchas. What about transactionality, partial updates, running multi document queries, consistency of the whole set of documents. You have to rewrite a whole DBMS on top of s3 itself or use redshift to get these things.

In terms of scalability there are, limits 3500rps per key prefix.

It's actually not lower price than a DBMS when you have a lot of data.

Re: The Architecture of Serverless Data Systems

#17

Post by ddorian43 »

Dave3of5 wrote: post #16

A bit too waffling for me to read all but I would like to make a small comment. Why are more and more devs trying to use s3 as a general purpose DB? Working on a system right now where the architects have made this mistake it has insanely poor performance (High latency) and lack any proper ACID compliance. I've now been asked to "make it faster" and the answer is to switch back to an actual DBMS. > Top tier SaaS serv…

> You have to rewrite a whole DBMS on top of s3 itself or use redshift to get these things.

No, you get a DBMS and only change the storage underneath. You can't use S3 for appending to WAL though.

All those can be fixed besides the latency for a cold GET from S3 and appending WAL to S3.

Re: The Architecture of Serverless Data Systems

#18

Post by Cwizard »

latchkey wrote: post #4
threeseed wrote: post #3

I wasn't aware of 800G/1TB networking before. Very strange world that transferring data between servers is 2x as fast as reading from a PCI bus.

InfiniBand, and unless you're super high up on the food chain (aka: married to NVIDIA), you won't get access to it for at least 52+ weeks after you submit your purchase order.

In case anyone else is wondering what InfiniBand is: https://network.nvidia.com/pdf/whitepapers/IB_Intro_WP_190.p...

Re: The Architecture of Serverless Data Systems

#19

Post by Dave3of5 »

ddorian43 wrote: post #17
Dave3of5 wrote: post #16

A bit too waffling for me to read all but I would like to make a small comment. Why are more and more devs trying to use s3 as a general purpose DB? Working on a system right now where the architects have made this mistake it has insanely poor performance (High latency) and lack any proper ACID compliance. I've now been asked to "make it faster" and the answer is to switch back to an actual DBMS. > Top tier SaaS serv…

> You have to rewrite a whole DBMS on top of s3 itself or use redshift to get these things. No, you get a DBMS and only change the storage underneath. You can't use S3 for appending to WAL though. All those can be fixed besides the latency for a cold GET from S3 and appending WAL to S3.

> No, you get a DBMS and only change the storage underneath

I think what you mean is what we have implemented a side channel DBMS which holds a copy which you use for the transactionality. It's a terrible approach I would not do this at all you don't get any benefit from using s3 here.

This is not to say you can't use s3 to pull large blob storage off the DB and reference it in the DB I'm talking about the entire DB as s3.

Re: The Architecture of Serverless Data Systems

#20

Post by ddorian43 »

Dave3of5 wrote: post #19
ddorian43 wrote: post #17

Earlier quoted context omitted.

> You have to rewrite a whole DBMS on top of s3 itself or use redshift to get these things. No, you get a DBMS and only change the storage underneath. You can't use S3 for appending to WAL though. All those can be fixed besides the latency for a cold GET from S3 and appending WAL to S3.

> No, you get a DBMS and only change the storage underneath I think what you mean is what we have implemented a side channel DBMS which holds a copy which you use for the transactionality. It's a terrible approach I would not do this at all you don't get any benefit from using s3 here. This is not to say you can't use s3 to pull large blob storage off the DB and reference it in the DB I'm talking about the entire DB…

> I'm talking about the entire DB as s3.

Yes, it can be done, except for WAL append & COLD GET. You "just" have to re-architect everything in the storage layer.

Do you have anything specific in mind besides the 2 things I mentioned?

Post reply on HN