Why not simply use the ASCII characters dedicated for this? STX,ETX, RS, GS, etc
Time to retire the CSV?
111–120 of 594 posts
Re: Time to retire the CSV?
#112> It's Time to Retire the CSV
> This column obviously contains dates, but which dates? Most of the world
It's time to retire local formats and always write YYYY-MM-DD (which is both the international and the Swedish standard, and the most convenient for parsing and sorting).
> A third major piece of metadata missing from CSVs is information about the file’s character encoding.
It's bloody the time to retire all the character encodings and always use UTF-8 (and update all the standards like ISO, RFC etc to require UTF-8). The last time I checked common e-mail clients like Thunderbird and Outlook created new e-mails in ANSI/ISO codepages by default (although they are perfectly capable of using UTF-8) - this infuriated me.
> If not CSV, then what? ... HDF5
Indeed! Since the moment I discovered HDF5 I wonder why is it not the default format for spreadsheet apps. It could just store the data, the metadata, the formulae, the formatting details and the file-level properties in different dimensions of its structure to make a perfect spreadsheet file. Nevertheless spreadsheet apps like LibreOffice Calc and MS Excel don't even let you import from HDF5.
> An enormous amount of structured information is stored in SQLite databases
Yet still very underused. It ought to be more popular. In fact every time I get CSV data I import it to SQLite to store and process but most of the people (non-developers) have never heard of it. IMHO it also begs to be supported (for easy import and export at least) by the spreadsheet apps. A caveat here is it still uses strings to store dates so the dates still can be in any imaginable format. Fortunately most of the developers use a variation of ISO 8601 conventionally.
And by the way, almost every application-specific file format could be replaced by SQLite or HDF5 for good. IMHO the only cases where custom format make good sense are streaming and extremely resource-limited embedded solutions.
Re: Time to retire the CSV?
#113Earlier quoted context omitted.
Human can't efficiently write or parse XML or json, though. In some scenarios CSV hits the right spot to be accessible to human and computer, and the table can be laid out so that one can sort/grep/awk to quickly gain some insight.
As someone that works with maven and npm.... what? This is valid JSON [["bob", "jones", 1, 22], ["frank", "was", 32, 45]] That's unreadable and unparsable by a human? Not only is JSON often more parsable, because it's structured it also becomes a lot easier to query. I grep and awk xml and json stuff all the time. I also have the added bonus of being able to use `jq` for json content.
Thats just a CSV with extra steps.
Re: Time to retire the CSV?
#114I think there's two conversations here: known, working CSVs as part of a workflow; and CSVs from random, unknown sources.
There's no such thing as schema-less - there's undefined schema
Re: Time to retire the CSV?
#115Earlier quoted context omitted.
As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. I definitely can't do that with .xlsx. There is almost zero friction to bolting CSV export capability to an existing system, which is part of why it's so popular.
> As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. And usually generate garbage for anything but the most trivial case, which really nobody gives a shit about. That's the main reason why CSV absolutely sucks too, you have to waste month diagnosing the broken shit you're given to implement the workarounds necessary to deal with it. > I definitely can't do that with .xlsx…
As soon as you open it in Excel, it's garbage anyway, since it will replace date-like items with nonsense, drop number digits, convert anything it can, reencode monetary unities, and so on.
If you don't open it in Excel, you can have as strict a parser as you want, just like any other format.
But neither is going anywhere anyway.
Re: Time to retire the CSV?
#116Earlier quoted context omitted.
To point 1, I'd argue that a SQLite database is a great next step beyond CSV despite being a binary format.
SQLite also requires external dependency and special execution environment and are not as portable as CSV.
SQLite is included in most operating systems by default. It's also on macOS and iOS.
It also runs on all kinds of embedded devices in addition to personal computers and servers.
Re: Time to retire the CSV?
#117Yes CSV sucks, but beyond that this rant is...quite sloppy. In dismissing YAML and JSON it seems to confuse probability of corruption when using an editor that isn't structure-aware with “lack of precision”, and then treats binary formats as better in this regard. Not only is the terminology bizarre, but the application of the criteria is nonsensical. Binary formats are not less likely to be corrupted when using an editor that isn't specifically aware of the format, and JSON and YAML (especially when accompanied by explicit schemata) are no less amenable to structure (and, when used, schema) aware editors than binary formats are.
> To usurp CSV as the path of least resistance, its successor must have equivalent or superior support for both editing and viewing
Since the author has already ruled (on spurious grounds) human readability and thus the usability of ubiquitous text editors as incompatible with requirements for a successor format, this is simply impossible.
> If Microsoft and Salesforce were somehow convinced to move away from CSV support in Excel and Tableau, a large portion of business users would move to a successor format as a matter of course.
Yeah, sorry, you’ve put the cart before the horse. Neither of those firms are going to do that until the vast majority of users have already migrated off of CSV-based workflows, it would be an insane, user hostile move that would be a bigger threat to their established dominance in their respective markets than anything any potential competitor is likely to do in the foreseeable future.
Re: Time to retire the CSV?
#118I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…
OP and you gave me an idea : "The only true successor of CSV should be forward/backward compatible with any existing CSV variant" If we manage to write a spec that meet this criteria we'll have a powerful standard with easy adoption.
obligatory: https://xkcd.com/927/
Re: Time to retire the CSV?
#119Earlier quoted context omitted.
And CSV can be consumed and produced by spreadsheet software, which is useful to critical in many contexts where CSV is part of the pipeline. There the alternative would be to use xlsx files which… isn't necessarily any better. Until someone gets excel to ingest and produce something in a better format, we're pretty much stuck.
As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. I definitely can't do that with .xlsx. There is almost zero friction to bolting CSV export capability to an existing system, which is part of why it's so popular.
You can write what "looks" like CSV to you, but there are no guarantees it will import correctly.
The problem is 10x worse when you get CSV from one source and rely on another process to load it. I fought this problem for several days going from NetSuite to Snowflake via CSV.
Re: Time to retire the CSV?
#120Earlier quoted context omitted.
> A truly open format is available and accessible. Sqlite? > Applications have a speed increase from using csvs. Sqlite? > The new format should not be grossly larger than the one it is replacing Sqlite it is. -------- Oh, you mean something that Excel can open? Oh yeah, I guess CSV then. But lets not pretend #1 (openness), #2 (speed), and #3 (size) are the issues.
I'd add the constraint that it needs to be understandable by most people. A typical non-tech office worker can use CSV without any trouble. Sqlite has an extremely steep learning curve for someone unfamiliar with databases.
Exactly. Worst case a CSV can be edited and viewed or even created in MS Notepad or any other text editor which is pretty much guaranteed to be on a system. Something to deal with Sqlite? No such luck and the barrier is much higher.