Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB: How to Use the JSON Schema Validator

percona.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: MongoDB: How to Use the JSON Schema Validator

#11

Post by rooam-dev »

ardy42 wrote: post #8
rooam-dev wrote: post #4

Earlier quoted context omitted.

True, but all those migration scripts/patches/table locks add complexity and inconvenience (less agile).

> True, but all those migration scripts/patches/table locks add complexity and inconvenience (less agile). I don't think so. You could say exactly the same thing about maintaining tests for your code. Explicit schemas and constraints (and the effort that goes along with maintaining them) are very much like having tests for your data. They both help ensure that your code actually works when it needs to.

Last time I worked with PostgreSQL, it would lock entire table while adding a new column. Imagine users table, then you have to schedule this kind of altering outside business hours... which impacts devs (they have to stay late and/or come early) and business potentially, it has to wait till next day to have a feature delivered.

All these are trade-offs and everyone decides what is more important for them.

Re: MongoDB: How to Use the JSON Schema Validator

#12

Post by anarazel »

rooam-dev wrote: post #11
ardy42 wrote: post #8

Earlier quoted context omitted.

> True, but all those migration scripts/patches/table locks add complexity and inconvenience (less agile). I don't think so. You could say exactly the same thing about maintaining tests for your code. Explicit schemas and constraints (and the effort that goes along with maintaining them) are very much like having tests for your data. They both help ensure that your code actually works when it needs to.

Last time I worked with PostgreSQL, it would lock entire table while adding a new column. Imagine users table, then you have to schedule this kind of altering outside business hours... which impacts devs (they have to stay late and/or come early) and business potentially, it has to wait till next day to have a feature delivered. All these are trade-offs and everyone decides what is more important for them.

> it would lock entire table while adding a new column

It does so for the entire << 1ms it takes to add the necessary metadata.

Re: MongoDB: How to Use the JSON Schema Validator

#13

Post by rooam-dev »

anarazel wrote: post #12
rooam-dev wrote: post #11

Earlier quoted context omitted.

Last time I worked with PostgreSQL, it would lock entire table while adding a new column. Imagine users table, then you have to schedule this kind of altering outside business hours... which impacts devs (they have to stay late and/or come early) and business potentially, it has to wait till next day to have a feature delivered. All these are trade-offs and everyone decides what is more important for them.

> it would lock entire table while adding a new column It does so for the entire << 1ms it takes to add the necessary metadata.

If that's the case, then one less problem to deal with, however I remember it was a matter of several minutes.

Re: MongoDB: How to Use the JSON Schema Validator

#14

Post by anarazel »

rooam-dev wrote: post #13
anarazel wrote: post #12

Earlier quoted context omitted.

> it would lock entire table while adding a new column It does so for the entire << 1ms it takes to add the necessary metadata.

If that's the case, then one less problem to deal with, however I remember it was a matter of several minutes.

That's only the case if you add a DEFAULT value for the new column. Up until the soon-to-be-released v11 that required updating existing rows. Without a default it's essentially just inserting a single row in a row into an internal catalog table (pg_attribute). With a default in v11, we basically just store the default for a new column out of line, and reference it when the row is read.

In either case a lock has to be acquired on the table, which means if there's a longrunning transaction, you can end up blocking for a while...

Re: MongoDB: How to Use the JSON Schema Validator

#15

Post by rooam-dev »

anarazel wrote: post #14
rooam-dev wrote: post #13

Earlier quoted context omitted.

If that's the case, then one less problem to deal with, however I remember it was a matter of several minutes.

That's only the case if you add a DEFAULT value for the new column. Up until the soon-to-be-released v11 that required updating existing rows. Without a default it's essentially just inserting a single row in a row into an internal catalog table (pg_attribute). With a default in v11, we basically just store the default for a new column out of line, and reference it when the row is read. In either case a lock has to b…

Ah, yes, i think that was the case. Thanks for extra details!

Re: MongoDB: How to Use the JSON Schema Validator

#16

Post by bsaul »

amarkov wrote: post #10
bsaul wrote: post #7

Earlier quoted context omitted.

good luck writing code for that kind of dataset then. Data has schema by definition otherwise you wouldn't be able to reason about it.

Certainly, but that doesn't mean the schema has to be a strict validation encoded into your storage format. It's a perfectly well-defined programming model to say "well, I'm reading query X with schema Y, and if some rows don't match Y give me nulls instead".

"well, I'm reading query X with schema Y, and if some rows don't match Y give me nulls instead"

Seems like a recipe for disaster to me, but well... A database isn't a "storage format". It's most often the single source of truth for a set of information.

Not being fully sure what data you expect from that source of truth and yet being able to query it is really dangerous. What if you start to update this data after having nullified things you didn't understand ?

Re: MongoDB: How to Use the JSON Schema Validator

#17

Post by amarkov »

bsaul wrote: post #16
amarkov wrote: post #10

Earlier quoted context omitted.

Certainly, but that doesn't mean the schema has to be a strict validation encoded into your storage format. It's a perfectly well-defined programming model to say "well, I'm reading query X with schema Y, and if some rows don't match Y give me nulls instead".

"well, I'm reading query X with schema Y, and if some rows don't match Y give me nulls instead" Seems like a recipe for disaster to me, but well... A database isn't a "storage format". It's most often the single source of truth for a set of information. Not being fully sure what data you expect from that source of truth and yet being able to query it is really dangerous. What if you start to update this data after ha…

Schemaless databases are good for scenarios where the database isn't a source of truth. If you have a table full of e.g. per-second heartbeats from a bunch of deployed services, there's no fundamental underlying truth anyone's trying to gather from it, and you can't afford to run a full schema migration every time someone adds a new metric.

I recognize some people do try to use schemaless databases in the way you're describing, and I agree that's weird and dangerous.

Post reply on HN