Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we built a serverless SQL database

cockroachlabs.com

Post reply on HN
Print view View original

81–90 of 122 posts

Re: How we built a serverless SQL database

#81

Post by reilly3000 »

geoduck14 wrote: post #76
reilly3000 wrote: post #52

Earlier quoted context omitted.

Snowflake is OLAP and billed based on usage time + storage. This looks like it’s a regular OLTP SQL DB and pay by request.

I see. Looking at the docs, the pricing for storing data is measured in Gigs (I think $1 per gig per month) - not Tb. This makes me think it is intended for smallish data. I have a database now with 40 Tb - this would cost $40k a month just to store!

Yikes! I think this is really positioned for side projects and microservices. There is a clear path to migrating to a full instance or Postgres cluster over certain usage levels, but for many cases being able to scale to zero is still economically and operationally valuable.

40Tb is a ton-o-data! Snowflake sounds like a good fit.

Re: How we built a serverless SQL database

#82

Post by emilsedgh »

anamax wrote: post #80
phoboslab wrote: post #37

> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic. The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres. I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even…

During a technical interview many years ago, I commented that the system architecture didn't scale. The interviewer responded "When that matters, I won't even be managing the person whose problem that is."

Isn't that the right mindset?

Re: How we built a serverless SQL database

#83

Post by infogulch »

psadri wrote: post #44
phoboslab wrote: post #37

> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic. The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres. I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even…

Unfortunately, easy to hit that with say GraphQL where each client request can resolve to dozens of db selects vs a single hand written/tuned SQL select.

If you're using GraphQL with SQL -- postgres specifically -- I would say use Hasura, but support between Hasura & CockroachDB seems to have stalled due to missing triggers [0] [1]. CRDB supports a feature called "changefeeds" [2] which is claimed might cover some of Hasura's use-cases, but that's a proprietary extension not present in base PostgreSQL.

[0]: https://github.com/hasura/graphql-engine/issues/678

[1]: https://github.com/cockroachdb/cockroach/issues/28296

[2]: https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-...

Re: How we built a serverless SQL database

#84

Post by reilly3000 »

pier25 wrote: post #75
andydb wrote: post #2

To any who might see this, I'm the author of the blog post, and led the engineering team that built CockroachDB Serverless. I'll be monitoring this thread in case there are any questions you'd like to ask me about it.

How does this serverless offering compare to Fauna?

I would assume the main point is that it’s actual PostgreSQL, not a new query language.

Re: How we built a serverless SQL database

#85

Post by nerdywordy »

andydb wrote: post #2

To any who might see this, I'm the author of the blog post, and led the engineering team that built CockroachDB Serverless. I'll be monitoring this thread in case there are any questions you'd like to ask me about it.

Can this CRDB Serverless offering handle the burst connections of a serverless function based app? Are pooling or query queueing features built in?

Or would users face connection limits at some upper bound until the old function connections get spun down?

Re: How we built a serverless SQL database

#86

Post by YZF »

emilsedgh wrote: post #82
anamax wrote: post #80

Earlier quoted context omitted.

During a technical interview many years ago, I commented that the system architecture didn't scale. The interviewer responded "When that matters, I won't even be managing the person whose problem that is."

Isn't that the right mindset?

This will be somebody else's problem? Ehm, no it's not.

- When the bridge collapses I'll be managing someone else.

- When they find out this airplane has critical design flaws I'll be managing someone else.

- When this software I'm working on is hacked I'll be managing someone else.

Re: How we built a serverless SQL database

#87

Post by anamax »

YZF wrote: post #86
emilsedgh wrote: post #82

Earlier quoted context omitted.

Isn't that the right mindset?

This will be somebody else's problem? Ehm, no it's not. - When the bridge collapses I'll be managing someone else. - When they find out this airplane has critical design flaws I'll be managing someone else. - When this software I'm working on is hacked I'll be managing someone else.

All bridges have a capacity limit.

"Doesn't scale" doesn't imply "inadequate."

Overbuilding is waste.

Re: How we built a serverless SQL database

#88

Post by emilsedgh »

YZF wrote: post #86
emilsedgh wrote: post #82

Earlier quoted context omitted.

Isn't that the right mindset?

This will be somebody else's problem? Ehm, no it's not. - When the bridge collapses I'll be managing someone else. - When they find out this airplane has critical design flaws I'll be managing someone else. - When this software I'm working on is hacked I'll be managing someone else.

It's not a matter of who's problem it is. It's just that a scale-able architecture in most cases is a premature optimization. When building a product, scalability is only one aspect. And in case of most startups and companies, amongst the smaller ones.

I personally interview a lot of people and if they start proposing microservices or k8s or anything trendy like that (before having context), I consider it a negative point.

When hiring, I want someone to take a look at a busines problem, break it down into a smaller pieces. Most of the times, the most important engineering work is coming up with the right data models/data structures.

So yeah, maybe the architecture I have now wont scale up. But at least it'll get the business going. Years later when there are more resources, the software could be rewritten or whatever.

Also, you'r example mentioned "software being hacked". I never said undermine security. Security should always be taken seriously. Security is not a premature optimization. Scalability is.

Re: How we built a serverless SQL database

#89

Post by andydb »

justsomeuser wrote: post #67
andydb wrote: post #2

To any who might see this, I'm the author of the blog post, and led the engineering team that built CockroachDB Serverless. I'll be monitoring this thread in case there are any questions you'd like to ask me about it.

Can I connect using an SSH tunnel (without the SSL cert)?

At this point, only Postgres SSL connections are supported.

Re: How we built a serverless SQL database

#90

Post by mr_toad »

mdekkers wrote: post #79
lumost wrote: post #58

Earlier quoted context omitted.

The answer is unfortunately less clear cut. Particularly if you assume that whoever is tasked with scaling this hypothetical DB doesn't know what they are doing a-priori. The following questions are likely to come up 1) My t3.xl DB is down, how much bigger can I make it? 2) My r3.24xl DB can only handle 100 TPS and now my site is down, what can I do? 3) My 2x r3.24xl DB cluster costs a lot of money, Are other solutio…

> The answer is unfortunately less clear cut. Particularly if you assume that whoever is tasked with scaling this hypothetical DB doesn't know what they are doing a-priori. The answer is very clear-cut: Work with professionals.

The amount of data being collected in the world is growing much faster than the number of database engineers is.
Post reply on HN