Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PlanetScale is now generally available

planetscale.com

Post reply on HN
Print view View original

121–130 of 139 posts

Re: PlanetScale is now generally available

#121

Post by shlomi-noach »

ahachete wrote: post #117
ahachete wrote: post #116

Earlier quoted context omitted.

Thank you for your comments, I appreciate it. I'm still not sold, however. I would like to understand the underlying principles, "how this works". I don't need implementation details (happy if they are shared, though) but more on the main principles of operation. Please see my further comments below: > Git is very bad at analyzing SQL diffs. Agreed, nothing against. So PS has built-in a nice SQL diff. Neat! But what…

I think this comment answers most of my questions: https://news.ycombinator.com/item?id=29248306 Can you confirm (PS) this is how it works? From what I understand here, there are "shadow servers", replicating from the production traffic. If so, this is cool. I still see some caveats: * One already mentioned, the scope of migrations is limited to those where both old and new DDL are compatible with the currently runni…

Thank you! Please first see my comments to parent, as they describe how online schema change work within the same server; with PlanetScale branching, we do give you a development branch with which you can play as much as you want, without affecting production. Online schema change kicks in when you deploy your changes to production.

> the scope of migrations is limited to those where both old and new DDL are compatible with the currently running application.

You are absolutely correct, and that is the paradigm. Say, for example, you want to add a column, so you first run the migration that adds the column, and only afterwards can you deploy an application change that actually utilizes that column. Likewise if you want to DROP a column, you first deploy an app change that ceases to reference the column, and only then can you actually drop it.

This paradigm worked very well for the companies I worked with, and makes for both loose and tight coupling between code and database. It's loose when you have your test databases where you can deploy schema changes at will. It's loose in the sense you can take small steps at a time, each isolated from the other (e.g. ADD COLUMN does not require you to make any app changes _yet). Then, it's tight where you couple your code changes with the schema in your git repo. It's tight in that the app never gets too far from the database (normally one change away at any given time, per development branch).

> Being the migration asynchronous, I lose control of when to deploy changes to the application

Great point and absolutely on our radar.

> Not knowing exactly then the cut-over process is going to happen is also potentially a problem.

Again great point and on our radar. To be honest I previously moved away from caring about the exact cut-over time. We designed gh-ost to do just that: stall cut-over until the engineer/developer is happy to sit at their desk. OVer time, we found it was unnecessary. But absolutely there's use cases for both approaches.

Re: PlanetScale is now generally available

#122

Post by ahachete »

shlomi-noach wrote: post #121
ahachete wrote: post #117

Earlier quoted context omitted.

I think this comment answers most of my questions: https://news.ycombinator.com/item?id=29248306 Can you confirm (PS) this is how it works? From what I understand here, there are "shadow servers", replicating from the production traffic. If so, this is cool. I still see some caveats: * One already mentioned, the scope of migrations is limited to those where both old and new DDL are compatible with the currently runni…

Thank you! Please first see my comments to parent, as they describe how online schema change work within the same server; with PlanetScale branching, we do give you a development branch with which you can play as much as you want, without affecting production. Online schema change kicks in when you deploy your changes to production. > the scope of migrations is limited to those where both old and new DDL are compatib…

> Thank you! Please first see my comments to parent

Thank you indeed for the time taken to answer all my comments. Now together with all the information here, I understand how it works, and what the trade-offs are.

If my input serves for anything, I'd strongly recommend to take all the information here and write it in a structured way as part of the documentation. I didn't see there any information as valuable as this one. For me, and possibly many others, knowing this information is required in order to make informed decisions about whether to use this or not; and if so, how and what are the trade-offs (e.g. atomizing the changes such that db changes and code changes are independent, which I agree is in general a good thing, but is something to be clearly aware of).

> Again great point and on our radar. To be honest I previously moved away from caring about the exact cut-over time. We designed gh-ost to do just that: stall cut-over until the engineer/developer is happy to sit at their desk. OVer time, we found it was unnecessary. But absolutely there's use cases for both approaches.

For me it's important as cut-over takes some locks. Sure, for a small amount of time. But these locks may create some problems, so that's why I want to be aware. Most of the time are other DDL changes, which are a non-issue here since you already prevent that. But there could be others related to normal db operation. For example, and this may not apply here but does apply with Postgres, such a lock may queue other locks behind (including read-only queries). And if the cut-over lock is itself blocked by other lock (say an explicit table lock), then everything queues on that table and leads to a lock storm, which in turn may cause effective downtime. That's why when we plan migrations or operations similar as this cutover (for example in Postgres a repack operation, which is essentially rewriting a shadow table, in this case just for the purpose or reducing bloat), we really need to take this into account.

Re: PlanetScale is now generally available

#123

Post by ahachete »

shlomi-noach wrote: post #119
ahachete wrote: post #116

Earlier quoted context omitted.

Thank you for your comments, I appreciate it. I'm still not sold, however. I would like to understand the underlying principles, "how this works". I don't need implementation details (happy if they are shared, though) but more on the main principles of operation. Please see my further comments below: > Git is very bad at analyzing SQL diffs. Agreed, nothing against. So PS has built-in a nice SQL diff. Neat! But what…

Again, thank you for the questions. I am estimating that your database space isn't MySQL, which is just fine of course. Reason I'm asking/guessing, is that in the MySQL space, online schema change toold have been around for over a decade and are the go-to solution for schema changes. A small minority of the industry, based on my understanding as a member of the community, uses other techniques such as rolling migrati…

> I am estimating that your database space isn't MySQL, which is just fine of course.

You are absolutely right :) My background is strongly on Postgres, you can see from my profile more information if you want to.

So yes, I apologize if some of my questions are not applying or become to obvious for cases that are MySQL-based. But for the most part, I believe principles of operation are the same.

> [other comments]

As mentioned, thank you very much for the detailed information. This completes the picture that I was looking for. I will definitely go in more detail for some of the links provided.

This principle of operation is not too different from something I proposed to a Postgres project some time ago (https://github.com/cybertec-postgresql/pg_squeeze/issues/18). This tool indeed is conceptually pretty similar. It's a shame that supporting schema changes is not part of their focus at this point. It wouldn't do throttling either, but it shouldn't be a difficult feature to add, I guess.

For other users here that may be interested in the Postgres world, there are two tools that perform similar operation (creating a shadow table and filling it in the background), but are both focused on rewriting the table to avoid bloat, rather than for doing a schema migration:

* pg_repack (https://reorg.github.io/pg_repack/): the most used one, relies on triggers * pg_squeeze: already mentioned, uses logical replication

Re: PlanetScale is now generally available

#124

Post by shlomi-noach »

ahachete wrote: post #123
shlomi-noach wrote: post #119

Earlier quoted context omitted.

Again, thank you for the questions. I am estimating that your database space isn't MySQL, which is just fine of course. Reason I'm asking/guessing, is that in the MySQL space, online schema change toold have been around for over a decade and are the go-to solution for schema changes. A small minority of the industry, based on my understanding as a member of the community, uses other techniques such as rolling migrati…

> I am estimating that your database space isn't MySQL, which is just fine of course. You are absolutely right :) My background is strongly on Postgres, you can see from my profile more information if you want to. So yes, I apologize if some of my questions are not applying or become to obvious for cases that are MySQL-based. But for the most part, I believe principles of operation are the same. > [other comments] As…

Heh, and in the MySQL space, we use a "trivial" online schema schema migration (that has no actual schema changes) to avoid table bloat :)

Re: PlanetScale is now generally available

#125

Post by shlomi-noach »

ahachete wrote: post #122
shlomi-noach wrote: post #121

Earlier quoted context omitted.

Thank you! Please first see my comments to parent, as they describe how online schema change work within the same server; with PlanetScale branching, we do give you a development branch with which you can play as much as you want, without affecting production. Online schema change kicks in when you deploy your changes to production. > the scope of migrations is limited to those where both old and new DDL are compatib…

> Thank you! Please first see my comments to parent Thank you indeed for the time taken to answer all my comments. Now together with all the information here, I understand how it works, and what the trade-offs are. If my input serves for anything, I'd strongly recommend to take all the information here and write it in a structured way as part of the documentation. I didn't see there any information as valuable as thi…

I really appreciate your feedback. I'll pass on the documentation advice, it's good to have your user perspective.

I hear you on cut-over, and - it's indeed on our radar! I hope to bring good news.

Re: PlanetScale is now generally available

#126

Post by danielrhodes »

dastbe wrote: post #86
danielrhodes wrote: post #79

Earlier quoted context omitted.

This is sort of wait nails people with DynamoDB pricing. If you mess up your query, you end up scanning through tons of data, racking up the cost.

imo at least with dynamodb, its pretty straightforward what any given call is capable of doing, and you can get back capacity usage on every request to track. with these mysql/postgres compatible apis the actual query is abstracted as is its cost.

Yeah the pricing model for SQL based queries is not optimal at all considering you don't control how the query optimizer will use the indexes, and (not trying to assign malice) they aren't very incentivized to improve it.

Re: PlanetScale is now generally available

#127

Post by evanelias »

shlomi-noach wrote: post #114
ahachete wrote: post #94

I'm a bit confused with the "branching" [0] and "non-blocking schema changes" [1] features. I'm confused as they sound like "the next big thing" and I don't see anything special here. Not saying are bad concepts or ideas, the contrary. But not really useful either. Surely I'm missing something, so I would love to hear from the PlanetScale team here if possible. I have a strong and long Postgres operational background…

Hi, engineer at PlanetScale and maintainer for Vitess here. Appreciate your thoughtful comment, a couple answers: > is something that sounds like I could do ... from Git platforms themselves Git is very bad at analyzing SQL diffs. It can show you the textual diff between two CREATE TABLE statements, but it will not know what it takes to get you from _here_ to _there_. It has many parsing issues, like capturing irrele…

Do note, earlier this month I archived the repo linked in [1] as a last resort, directly in response to the repeated behavior of you and your colleagues.

I don't even know where to begin with explaining this to outsiders, but here's a sample: I receive confused support emails from companies and users (always non-paying ones, at that) about Skeefree literally every single week; at no point in our lengthy email discussion last year did you ever disclose that your employer would be basing the entirety of its marketing campaign for its commercial offering around schema management in direct competition with my own bootstrapped products; then more recently in your fork of [1] your coworker is adding functionality that once again directly competes with the functionality in my commercial products.

In brief, repeatedly using my own open source work to compete with me, which in turn is preventing me from ever generating revenue from this work, which is needed for the work to continue.

It seemingly never ends. At this point I'm literally on the verge of throwing all my work in the trash and never touching a database again. I've clearly wasted the last several years of my life and all I get in return is a shout-out on a day-old HN thread, cool cool.

Re: PlanetScale is now generally available

#128

Post by sahaamity »

zild3d wrote: post #23
herodotus wrote: post #7

I am having trouble understanding what a "Serveless Database" is. When I do a search of the term, I get hype, not a definition. For example: "What is Serverless Database? Serverless Database is a prerequisite for Serverless Computing. These are specially designed for the workloads which are unpredictable and can change rapidly. What’s more? This allows you to pay only for the database resources you use, on a second-b…

With a dedicated Mongo DB cluster (server-full), you are paying for a certain cluster size, per hour. It doesn't matter if you read or write any data to it. You're paying for a machine with a specific amount of storage capacity and cpu. Use it or lose it. DynamoDB (considered serverless), you're charged based on the read and write throughput, and how much you have stored (GB-month). If you don't store any data, you'r…

DataStax also launched Serverless Cassandra (Astra) recently.

https://www.datastax.com/products/datastax-astra/pricing

https://thenewstack.io/the-serverless-database-you-really-wa...

Disclaimer : I work for DataStax.

Re: PlanetScale is now generally available

#129

Post by shlomi-noach »

haggy102 wrote: post #111
harshit164 wrote: post #47

Earlier quoted context omitted.

Vitess using range based sharding and with a single command of `Reshard` you can split a range to n sub-ranges. Ref for how command looks like https://vitess.io/docs/user-guides/configuration-advanced/re...

Happy to be wrong but both of you sound like you're commenting without any operational knowledge about either database technology.

Since I feel injustice done here, I'd like to point out that harshit164 is a Vitess maintainer who is my go-to person of reference for these exact topics, and who is authoritative about this database technology.

Re: PlanetScale is now generally available

#130

Post by MapleWalnut »

boiler_up800 wrote: post #24
gh123man wrote: post #14

This seems cool. I am currently shopping around for a cheap managed database for an app I am launching soon (and bootstrapping - so cost is a concern right now). We are starting small but want room to grow. The free tier looks really promising! Especially compared to some of the entry level plans on other cloud providers. Can anyone who has tried Planetscale and compared it to other managed database solutions tell me…

I've tried Aurora and Firebase. I think PlanetScale wins on 2 areas at 2 different scales: branching when you're small, and connections when you're large. With branching it's a feature where every dev can run a command and get their own database version to mess with. They can submit a pull request and as the tech lead I can check it and publish it to prod. Or integrate it with our CI. On connections, it is interestin…

PlanetScale branching only copies the schema, not the data.

from their docs:

> Note: Data from the main database is not copied to development branches.

https://docs.planetscale.com/concepts/branching

Post reply on HN