351–355 of 355 posts
Earlier quoted context omitted.
Most csv utilities support an alternative delimiter. If I need to edit a file by hand, I'll typically pick an uncommon character for the delimiter (pipe "|" works well since it's uncommon). For me, that pretty much entirely eliminates any of the pain with CSV.
More tools should use the ASCII unit and field separator characters intended for this purpose: https://ronaldduncan.wordpress.com/2009/10/31/text-file-form...
Earlier quoted context omitted.
Most csv utilities support an alternative delimiter. If I need to edit a file by hand, I'll typically pick an uncommon character for the delimiter (pipe "|" works well since it's uncommon). For me, that pretty much entirely eliminates any of the pain with CSV.
More tools should use the ASCII unit and field separator characters intended for this purpose: https://ronaldduncan.wordpress.com/2009/10/31/text-file-form...
Earlier quoted context omitted.
>> CSV is a mess. One quote in the wrong place and the file is invalid. > That breaks the other formats too, why pick on CSV? I think it's perhaps badly worded, but the implied (and more important) criticism seems to me to be that CSV makes this kind of error much more likely, with its handling of quotes. Having worked with CSV files that had commas in the data (and sometimes quotes too), I quickly learned that I sho…
Indeed, it’s because people use an editor that isn’t designed for editing csv to edit csv. Every csv files is a text file, but not every text file is a csv file, but people use text editors to edit them. A csv editor would forbid you from entering that quote, automatically add a matching one, ask you for your intentions, or whatever, but it shouldn’t silently corrupt your file. A spreadsheet-like UI, but without form…
VisiData is an interactive multitool for tabular data. It combines the clarity of a spreadsheet, the efficiency of the terminal, and the power of Python, into a lightweight utility which can handle millions of rows with ease.
Earlier quoted context omitted.
What if you are doing both, as in trying to source control your database. A lot of tools (liquibase for one) can use CSV files as the basis of read-mostly lookup tables for example. E.g. if I had a list of "product categories" that I wanted to be part of my database when I bootstrap a new environment to run my software. Liquibase can turn that CSV file into a set of upserts that will run if the file hash has changed.…
Databases are measured in gigabytes and terabytes. If you put only a portion of it in source control, how do you back up the rest of your database? Furthermore, static data is a minor subset of a database. Data by its nature is volatile. Transactions make up 80% of the data. A projection or snapshot/summary is a the summarization of the daily/hourly/minute transactions. If you want a golden copy to bootstrap new envi…
> If you put only a portion of it in source control, how do you back up the rest of your database? I think this is simply a misunderstanding of how these DB change management tools work and what they can do to help you with complex migrations over an applications lifetime.
You still back your DB up like normal.
Your your "mostly static" data is managed by running inserts/updates/deletes when data changes (either manually, or the tool can usually generate them for you), when you actually apply it, the software also records that the migration has been applied in a database change log table. That way when you want to update your database with the latest migrations, it'll only run what hasn't been applied yet.
That allows your standard backup/restore procedures to work just fine.
> If you want a golden copy to bootstrap new environments, I would argue you are better off backing up that golden copy and restoring it using native database tools
So this is essentially what we are doing with liquibase... Using database dumps without any data as our baseline.
Any DDL changes are managed through migration scripts.
There are a number of things that are not managed by migration script, and are instead apply-on-change in our project though.
We found it better to have our views stored as individual files in liquibase, and have them apply on change, because of nested dependencies and other issues with not having a good source of truth for the view definition.
Functions/procedures were another that are best treated as code rather than migrations. It allows you to deal with conflicts between branches with your standard merge tools.
Our "static" data that would only change when there is an application change is managed through csv files managed by liquibase that apply on any change. That needs to be in-sync with the version of the application deployed, so it makes sense to manage that along with the code.
Earlier quoted context omitted.
CSV should be pretty easy to merge, given that it’s line based.
Although it's the easiest, same row cell changes still conflict unnecessarily.
1
45