Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you version your data?

news.ycombinator.com

Post reply on HN
Print view View original

51–58 of 58 posts

Re: Ask HN: How do you version your data?

#51

Post by webmaven »

hbcondo714 wrote: post #27
webmaven wrote: post #22

Earlier quoted context omitted.

Most data versioning approaches would help bulk consumers of the data, rather than through the API. You could add a separate "changes" feed to help API consumers, or inline version info to the main API results.

Thanks, was thinking about doing a new endpoint for changes too but wanted to see if this topic of data versioning pertains to this or not

It does, but I think what you're really after here is a mechanism for cache invalidation.

Check out PostgreSQL's LISTEN and NOTIFY:

https://www.postgresql.org/docs/9.6/static/sql-listen.html

https://www.postgresql.org/docs/9.6/static/sql-notify.html

A related HN discussion a few years ago: https://news.ycombinator.com/item?id=6689213

Re: Ask HN: How do you version your data?

#52

Post by gtonic »

gregw2 wrote: post #38
webmaven wrote: post #14

Earlier quoted context omitted.

I am pretty familiar with several, such as Alembic, South, Django Migrations, etc. I'm not just interested in versioning schema changes, but data changes as well.

There are database migration tools like Liquibase/datical and, I believe, Flyway that let you specify migrations not just of schema changes (DDL) but also data changes (DML). I have used Liquibase in production for handling data changes for small-scale data (in megabytes) for dev/test/mock data and/or values in preconfigured application-controlling tables, but I wouldn't use it for gigabytes of transactional data.

+1 for Liquibase, too

Re: Ask HN: How do you version your data?

#53

Post by clord »

ComputerGuru wrote: post #23
clord wrote: post #4

I wrote[0] about a method I've used with success in the past. Essentially you use the previous version's hash as the name for the next version. The benefit is that merging the work of multiple developers is easier. I ran the process manually when I did it and would love to hear if someone writes a script that makes the process easier. [0]: https://medium.com/@clord/for-migration-of-schemas-use-versi...

So if you release version 1.0 with hash x, then release version 1.1 with hash y, and then release version 1.0.1 with hash z, 1.0.1 == 1.1?

For branching schemas, yes it gets complicated. But at least you get a nice conflict while merging the branches (at the right place) These conflicts help guide a rewrite of the schema history into something consistent. That's part of it: when a merge happens, you basically have to create a new migration that does the merge too.

and if you never merge the forks, there is no problem. they likely won't share a production database anyway, right?

Re: Ask HN: How do you version your data?

#54

Post by alexatkeplar »

webmaven wrote: post #45
alexdean wrote: post #35

Great question - we invented a system for this at Snowplow, called SchemaVer: http://snowplowanalytics.com/blog/2014/05/13/introducing-sch... SemVer doesn't work for data - for one thing, there is no concept of a "bug" in your data (so patches are meaningless). We have hundreds of companies actively using SchemaVer via the Snowplow ( https://github.com/snowplow/snowplow/ ) and Iglu ( https://github.com/snowplow/iglu/…

Looks quite interesting, though again this is versioning the data schema rather than the data. I think you have a certain amount of fuzziness around the idea of an "interaction" with the data. It would probably help to think about compatibility and breaking changes in terms of reads vs. writes in order to get the determinism you're looking for and better alignment with SemVer. That is, if a client using the previous…

I agree there is some fuzziness. Long after we wrote this I read:

https://www.w3.org/2001/tag/doc/versioning#iddiv371153984

which has a very succinct explanation of forwards and backwards compatibility as it relates to producers and consumers.

It's high time we did a second draft of SchemaVer which explains it in terms of forwards/backwards compatibility; the actual behavior of it (when to bump etc) would barely change.

Re: Ask HN: How do you version your data?

#55

Post by sheepmullet »

webmaven wrote: post #43
austinpray wrote: post #40

Earlier quoted context omitted.

Any change you make to the schema will be a breaking change.

Not so. Adding a new column is not a breaking change for reads, and may not be a breaking change for writes unless the new column is required, has no default value, and cannot be NULL.

That is not really true in practice.

Take for example a GetUserStatistics() call which provides a list of userids and the users last login date.

A client might be using this list to get statistics on system usage.

If you change the codebase to add the concept of a test user and add an isTestUser column to GetUserStatistics() you have broken the contract with your users.

You had an implicit contract based on shared understanding of the data.

Now of course to correctly determine user usage statistics you need to exclude the test users by checking the new column.

Re: Ask HN: How do you version your data?

#56

Post by sheepmullet »

webmaven wrote: post #45
alexdean wrote: post #35

Great question - we invented a system for this at Snowplow, called SchemaVer: http://snowplowanalytics.com/blog/2014/05/13/introducing-sch... SemVer doesn't work for data - for one thing, there is no concept of a "bug" in your data (so patches are meaningless). We have hundreds of companies actively using SchemaVer via the Snowplow ( https://github.com/snowplow/snowplow/ ) and Iglu ( https://github.com/snowplow/iglu/…

Looks quite interesting, though again this is versioning the data schema rather than the data. I think you have a certain amount of fuzziness around the idea of an "interaction" with the data. It would probably help to think about compatibility and breaking changes in terms of reads vs. writes in order to get the determinism you're looking for and better alignment with SemVer. That is, if a client using the previous…

I like the idea but I'd suggest not using semver terminology as it is misleading.

Breaking writes would not be considered a minor change in semver.

Re: Ask HN: How do you version your data?

#57

Post by webmaven »

sheepmullet wrote: post #55
webmaven wrote: post #43

Earlier quoted context omitted.

Not so. Adding a new column is not a breaking change for reads, and may not be a breaking change for writes unless the new column is required, has no default value, and cannot be NULL.

That is not really true in practice. Take for example a GetUserStatistics() call which provides a list of userids and the users last login date. A client might be using this list to get statistics on system usage. If you change the codebase to add the concept of a test user and add an isTestUser column to GetUserStatistics() you have broken the contract with your users. You had an implicit contract based on shared un…

Changing the semantics is entirely besides the point. You can change the semantics of the data without making any schema changes at all!

Re: Ask HN: How do you version your data?

#58

Post by sheepmullet »

webmaven wrote: post #57
sheepmullet wrote: post #55

Earlier quoted context omitted.

That is not really true in practice. Take for example a GetUserStatistics() call which provides a list of userids and the users last login date. A client might be using this list to get statistics on system usage. If you change the codebase to add the concept of a test user and add an isTestUser column to GetUserStatistics() you have broken the contract with your users. You had an implicit contract based on shared un…

Changing the semantics is entirely besides the point. You can change the semantics of the data without making any schema changes at all!

Isn't it the whole point?

As a consumer it is what I care about.

Post reply on HN