Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

D1: Our SQL database

blog.cloudflare.com

Post reply on HN
Print view View original

231–240 of 241 posts

Re: D1: Our SQL database

#231

Post by Thaxll »

manigandham wrote: post #205
Thaxll wrote: post #203

Earlier quoted context omitted.

Why would you use SQLlite once you start dealing with network, just use MySQl or PG. It's just re-inventing the wheel badly, I need to read the details but basically you're using a tool SQLLite that was not designed to be used outside of a single app use case.

What context are you talking about here? For Cloudflare, they're offering it because it's simple and lightweight, and they already have their Durable Objects product which serves as the transaction ordering mechanism and takes care of writes. If you're doing it yourself then sure it's probably not the best fit but that's up to you to decide.

SQLLite was not meant to be used by multiple process so you have to build the missing parts yourself, 100% those have more limitations and issues than regular RDBMS that were built for it.

Re: D1: Our SQL database

#232

Post by infogulch »

discodave wrote: post #211
infogulch wrote: post #98

Earlier quoted context omitted.

Thanks for the clarification, that is what I would expect. Does SQLite support some kind of monotonic transaction id that can be used as a cache coherency key? Say a client writes a new record to the database which returns `{"result": "ok", "transaction_id": 123}`, then to ensure that subsequent read requests are coherent they provide a header that checks that the read replica has transaction_id >= 123 and either wai…

Since it's a relational DB, and supports transactions, you can have a journal table right? I know of a very important system at AWS that did this with MySQL :D

Yes you could do it manually, but it would be nice if the solution didn't require carefully managing update queries so the journal addition isn't missed and increasing write amplification to manually update a journal table when that information probably already exists somewhere in the WAL implementation.

Re: D1: Our SQL database

#234

Post by CRConrad »

rad_gruchalski wrote: post #220
otoolep wrote: post #158

Earlier quoted context omitted.

I agree -- this blog post is light on details. To me the value Cloudflare believes they are offering is mostly ease-of-use, particularly setup. With minimal work you can have a stateful, relational store available to your code. But in terms of actual database functionality, they are not offering anything particularly novel. Of course, I might be missing something. In fact, I don't see anything D1 is doing that is not…

I’ve been looking at rqlite for some time and it’s really great to track the product on github. I believe that the power of what Cloudflare offers here isn’t in the actual database. It’s the packaging and how it sits in their serverless world. Even with rqlite, I still need ip addresses to run a resilient system. As someone who sometimes needs a table here snd there, I really, really don’t want a server. I want a tab…

> It’s the packaging and how it sits in their serverless world. [...] As someone who sometimes needs a table here snd there, I really, really don’t want a server. I want a table to store a thousand records in and that’s it.

Sorry, but I don't get it -- WTF does "serverless" even mean here?

I mean, sorry for jumping on your comment specifically, I know that wasn't primarily what you were talking about here, but... You seem to know what you're talking about, effortlessly encompassing "their serverless world" etc.

The article even mentions that > That makes sense to me; "serverless" means "not having a server". So then you have a local DB; be it SQLite or a DBF or Paradox or MS Access file or whatever. Or even a local DB software "service"; Firebird or MySQL or what have you.

But the term, as it's been bandied about online for the last decade(?) or so (including in this article), seems to pretty obviously actually be about... Remote servers (that's what it talks about replicating between, right?). So what's "serverless" about that???

I've been wondering for a good while now. Anyone who has a short explanation, or link to such, please jump in and enlighten me.

(Otherwise I'll have to conclude it's like "the Cloud", a.k.a. "Someone else's computer". "Serverless" = Someone else's server? :-)

[Edit: Typo.] [Edit: Sigh... Two of them.]

Re: D1: Our SQL database

#235

Post by rad_gruchalski »

CRConrad wrote: post #234
rad_gruchalski wrote: post #220

Earlier quoted context omitted.

I’ve been looking at rqlite for some time and it’s really great to track the product on github. I believe that the power of what Cloudflare offers here isn’t in the actual database. It’s the packaging and how it sits in their serverless world. Even with rqlite, I still need ip addresses to run a resilient system. As someone who sometimes needs a table here snd there, I really, really don’t want a server. I want a tab…

> It’s the packaging and how it sits in their serverless world. [...] As someone who sometimes needs a table here snd there, I really, really don’t want a server. I want a table to store a thousand records in and that’s it. Sorry, but I don't get it -- WTF does "serverless" even mean here? I mean, sorry for jumping on your comment specifically, I know that wasn't primarily what you were talking about here, but... You…

Sure, let me explain my way of thought. I think it’s time to get off of the bandwagon „it’s someone else’s computer”. Sure, internet services don’t live in the void and there’s always a server out there. Are the end of the day, three things in life are for certain: taxes, death, and there is a server.

However, with Cloudflare specifically, you write apps that deal with individual requests. You, as a dev, never ever get exposed to a server. This abstraction goes way further than even AWS Lambda goes. There are no local caches, no temp directories, no mucking around with a shutdown of a function. Every instance of an app deals with exactly one request.

That’s what I mean by „their serverless world”.

Re: D1: Our SQL database

#236

Post by detroitcoder »

slashdev wrote: post #111

For a Cloudflare article, this one is surprisingly light on technical details. And for the product where it most matters. I'm guessing this is a single master database with multiple read replicas. That means it's not consistent anymore (the C in ACID). Obviously reads after a write will see stale data until the write propogates. I'm a bit curious how that replication works. Ship the whole db? Binary diffs of the mast…

Going to be very interesting to see how they glue together R2, edge workers and sqllite. They can manage replication using R2 and make the sqllite process aware of this for eventual consistency. Having edge compute with edge data on a globally consistent data model is the dream.

Re: D1: Our SQL database

#237

Post by CRConrad »

rad_gruchalski wrote: post #235
CRConrad wrote: post #234

Earlier quoted context omitted.

> It’s the packaging and how it sits in their serverless world. [...] As someone who sometimes needs a table here snd there, I really, really don’t want a server. I want a table to store a thousand records in and that’s it. Sorry, but I don't get it -- WTF does "serverless" even mean here? I mean, sorry for jumping on your comment specifically, I know that wasn't primarily what you were talking about here, but... You…

Sure, let me explain my way of thought. I think it’s time to get off of the bandwagon „it’s someone else’s computer”. Sure, internet services don’t live in the void and there’s always a server out there. Are the end of the day, three things in life are for certain: taxes, death, and there is a server. However, with Cloudflare specifically, you write apps that deal with individual requests. You, as a dev, never ever g…

Thanks!

Re: D1: Our SQL database

#238

Post by CRConrad »

sophacles wrote: post #221
hunterb123 wrote: post #182

Earlier quoted context omitted.

I prefer to make the world better through actions. Not pointlessly redefining words that have no ill intent in the first place and harassing people that don't use the words you decided were proper.

Just a thought, what if we made the world a better place through clearer terminology? It's not a major improvement, but it helps reduce friction in communications. Ignoring connotations, master/slave is pretty unclear to me and needs more explanation. I've seen it used to describe: * a coordinator with a worker pool in which the coordinator sends jobs to the workers for actual computation * a designation of where on…

> Ignoring connotations, master/slave is pretty unclear to me and needs more explanation. I've seen it used to describe:

> * a designation of where on the bus a device sits

I see, you're worried people will get this mixed up with Rosa Parks?

Honestly, it feels like the PC brigade is really having to bend over backwards to find arguments. That should tell them something.

> Or we could differentiate the cases with more applicable and descriptive words.

So you get Primary / Secondary or Main / Secondary... For all of your above cases. I don't quite see how that differentiates them.

Re: D1: Our SQL database

#239

Post by Quarrelsome »

endisneigh wrote: post #70
Quarrelsome wrote: post #67

Earlier quoted context omitted.

they should be using SQLite more often than they are.

Why? What use cases are better with SQLite vs Postgres, MySQL, etc?

less things to go wrong. This gives you benefits all the way up the dev stack. Changing an integration test from needing its own db server installed to just a couple of files on disk is a big difference in complexity. You can probably run that test with just a local disk almost infinitely entirely deterministically, conversely as soon as you go onto the network all bets are off.

Granted, if you already have the tooling its less of a big deal but if you don't then you need the tooling, e.g. your build and test machines need to have access to some sql installation somewhere and that the process of maintaining that can be a flaky one.

Re: D1: Our SQL database

#240

Post by archon810 »

einichi wrote: post #126
messe wrote: post #115

Earlier quoted context omitted.

Yep, you're not charged for egress.

B2 to Cloudflare also does not incur egress fees: https://www.backblaze.com/blog/backblaze-and-cloudflare-part... Backblaze B2 customers will be able to download data stored in B2 to Cloudflare for zero transfer fees. This happens automatically once Cloudflare is configured to distribute your B2 files.

Only to Cloudflare though, right? If users download, it'll be billable.

With R2, user downloads are free, aren't they?

Post reply on HN