Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you version your data?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 58 posts

Re: Ask HN: How do you version your data?

#21

Post by peclink »

clord wrote: post #4

I wrote[0] about a method I've used with success in the past. Essentially you use the previous version's hash as the name for the next version. The benefit is that merging the work of multiple developers is easier. I ran the process manually when I did it and would love to hear if someone writes a script that makes the process easier. [0]: https://medium.com/@clord/for-migration-of-schemas-use-versi...

What a great idea, thanks for sharing!

BTW, why using shell scripts to write migrations is so unpopular? We have lots of language/framework specific solutions or some «language agnostic» tools using plain SQL, both ways are rather limiting.

Re: Ask HN: How do you version your data?

#22

Post by webmaven »

hbcondo714 wrote: post #20

On a somewhat related note, I have an API that returns time series data. The API itself is versioned but not the data when it is added to since the endpoint is the same. How would I go about notifying consumers of the API that the data has been updated and they should call the API to get the latest data? Would versioning the data help with this?

Most data versioning approaches would help bulk consumers of the data, rather than through the API.

You could add a separate "changes" feed to help API consumers, or inline version info to the main API results.

Re: Ask HN: How do you version your data?

#23

Post by ComputerGuru »

clord wrote: post #4

I wrote[0] about a method I've used with success in the past. Essentially you use the previous version's hash as the name for the next version. The benefit is that merging the work of multiple developers is easier. I ran the process manually when I did it and would love to hear if someone writes a script that makes the process easier. [0]: https://medium.com/@clord/for-migration-of-schemas-use-versi...

So if you release version 1.0 with hash x, then release version 1.1 with hash y, and then release version 1.0.1 with hash z, 1.0.1 == 1.1?

Re: Ask HN: How do you version your data?

#24

Post by shneg »

My hack way is to dump the tables to text and commit that. That was done by cron and not coupled with committing the code.

Granted, I was interested in the data within the table not the structure and the size was manageable.

Re: Ask HN: How do you version your data?

#25

Post by nepsilon »

hbcondo714 wrote: post #20

On a somewhat related note, I have an API that returns time series data. The API itself is versioned but not the data when it is added to since the endpoint is the same. How would I go about notifying consumers of the API that the data has been updated and they should call the API to get the latest data? Would versioning the data help with this?

If you can predict when you're going to get new content, just use an HTTP header to tell until when the content can be seen as fresh. Then the client will know when to call the API again.

If not, ask the client to provide an HTTP callback endpoint, and ping it to notify of new fresh data on your side. Using WebSocket is also an option.

Re: Ask HN: How do you version your data?

#26

Post by ratpik »

The most RESTFul way to do this would be to use content negotiation using the `content-type` header set to something like `application/com.vendor.product+json; data-version=2.1; api-version=3.4` where the minor version indicates data changes and the major version indicates schema changes to the data. You can club together the API and data versions into one version like '3.4.2.1' if you can define what a major/minor change to the API means. Exact details on how the client is exposed to the versioning will depend on the product requirements.

In terms of storing the data, we had a system where the content would be zipped after the content developers were done with the authoring and sent to a place which would convert it into appropriate JSON documents with the metadata and versioning information stored in the DB while the the document content could be stored in the cloud or a document database like Mongo or just Postgres. The content authors only knew excel who were trained to follow a schema while writing the content. That was like a low cost CMS. You can update only the content that has a diff or the entire content depending on how well you can identify a diff for the content. The entire content makes it simple.

Re: Ask HN: How do you version your data?

#27

Post by hbcondo714 »

webmaven wrote: post #22
hbcondo714 wrote: post #20

On a somewhat related note, I have an API that returns time series data. The API itself is versioned but not the data when it is added to since the endpoint is the same. How would I go about notifying consumers of the API that the data has been updated and they should call the API to get the latest data? Would versioning the data help with this?

Most data versioning approaches would help bulk consumers of the data, rather than through the API. You could add a separate "changes" feed to help API consumers, or inline version info to the main API results.

Thanks, was thinking about doing a new endpoint for changes too but wanted to see if this topic of data versioning pertains to this or not

Re: Ask HN: How do you version your data?

#28

Post by hbcondo714 »

nepsilon wrote: post #25
hbcondo714 wrote: post #20

On a somewhat related note, I have an API that returns time series data. The API itself is versioned but not the data when it is added to since the endpoint is the same. How would I go about notifying consumers of the API that the data has been updated and they should call the API to get the latest data? Would versioning the data help with this?

If you can predict when you're going to get new content, just use an HTTP header to tell until when the content can be seen as fresh. Then the client will know when to call the API again. If not, ask the client to provide an HTTP callback endpoint, and ping it to notify of new fresh data on your side. Using WebSocket is also an option.

Thanks, unfortunately the new data updates can vary.

Re: Ask HN: How do you version your data?

#29

Post by pnathan »

I consider data to be a first class object. I break it into data in motion and data at rest.

Data in motion - messages - always look something like this:

    message := version timesent field1 field2 ... fieldn
    version := INTEGER
    time-sent := INTEGER
and the parsers know to reject messages with versions greater than what they can parse; depending on the system, they can also be backwards compatible. Time-sent turns out to be a lifesaver in debugging. You might also need TZ of time-sent, depends on the domain.

Versioning data at rest tends to be a little squirrely depending on the domain. Do you migrate data or do you not? what's your uptime? streaming or batch? Sometimes I version the actual table names, sometimes I migrate.. it depends. My preference is for migration to keep a consistent system, but that is not always feasible.

I'm a huge fan of SQL - it defines the data shape and structures the transforms possible on it, along with allowing a strong separation of data and computation. Postgres is my friend; I heavily use foreign keys and constraints on the schema. That way the data is reliable. (if your data isn't reliable, your schema should reflect that too of course). If I need to have multiple versions of data running at the same time, multiple tables or migrating is cleaner than versioning the specific rows. Otherwise you wind up with nulls and driving schema logic out into your code.

Typically I tack a unix time of insert into the rows for later analysis. You might also care to insert the current application name+version into the rows to catch any bugaboos when that changes.

Re: Ask HN: How do you version your data?

#30

Post by vijucat »

Edit: I realized that your question is just about the versioning scheme (the "what"), and my answer is more about the "how". I hope you still find it useful.

After being inspired by Hans Werner's answer here (several terabytes of binary data, 50000+ revisions), I chose Subversion. It's not conventional, but works very well in practice.

https://stackoverflow.com/questions/127692/svn-performance-a...

You get:

a) Natural audit trail & notes on data modification

b) Managed central dump of data => multiple, distributed local copies that you don't need to worry about keeping in sync. Just delete the cache and the data access API (see below) will check it out automatically again when you request the file.

Data access is encapsulated via an API that manages a /home/datadump/ of cached, revisioned files. You refer to the file using it's name + revision number (see below). I guess tags and branches can be used for more natural revison numbers, but I need to investigate whether Subversion's cheap copy works well in practice for this. They might be the perfect solution for you major.minor.patch needs?

User Code --> get_data("", "r=38") --> API checks the cache for file_path/file_name_r=38. If it's not there it checks it out using a read-only user id and puts it in the cache and returns the path to /home/datadump/file_path/file_name_r=38.

Unusual, but works just fine for our purposes of mostly-read-only large files that need a revision history.

An idea that I did not explore was using ZFS or other revisioned file systems. Another "crazy" idea that works just fine for some folks is using Binary blobs in a database; not sure about size limits, though.

Post reply on HN