I've never seen an article say so little in so many words.
Time to retire the CSV?
501–510 of 594 posts
Re: Time to retire the CSV?
#502Earlier quoted context omitted.
> There's also Avro, which fails your point #1 (it's a binary format) but that binary format is a lossless alternate encoding of what's canonically a JSON document, and there are both simple CLI tools / and small, free, high-quality libraries that can map back and forth between the "raw" JSON document and the Avro-encoded file. At any time, you can decode the Avro-encoded file to text, to examine/modify it in a text…
As someone who's used Avro [0], it's a pain because Avro records must be ordered, but JSON by definition is unordered. Avro's "JSON" format is more of a JSON-like format. At one point, when I was writing a script that would ingest an Avro record and then output a new one, I had to fiddle with things to make Python use an OrderedDict [1] so the new record would be output in the right order. [0] though the last time I…
I'm not sure what you mean by Avro records "must" be ordered. If you mean that the serialization format specifies the ordering of the fields, then yes, that is true, but that's an advantage in terms of compactness and processing efficiency (https://avro.apache.org/docs/current/spec.html#order). If you don't like it though, there are other formats like protobuf and thrift that have no such requirement, at the cost of ~3 bits per field, which can be a comparatively efficient trade off.
Re: Time to retire the CSV?
#503I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…
I would add 4) editable in vim.
Re: Time to retire the CSV?
#504I guess it's nice we've reached the generation who can rant about this and not even mention that XML solved nearly all the problems mentioned and everybody hated it. It turns out people hate bloat and complexity more than they hate all the ills of CSV put together!
I'll happily take xml over csv any day, but my coworkers and our customers won't!
Re: Time to retire the CSV?
#505Are there really new projects stretching the limits of CSV out in production?
Everyone knows it’s not capable of storing anything more than table data. It was never meant to be much more.
It doesn’t really store objects well. To write the values of a composite object would require following some format order. It would go past human readability and just be a bloated way of writing bytes with readable characters.
Re: Time to retire the CSV?
#506The article's strongest criticism of CSV is that it's easy for someone to mangle it when manually editing. This is true. It's also true for every format. It was weakest when it implied there is no real standard. There is, and it's robust for representing data, even data that includes any combination of commas and double-quotes. The algorithm for creating well-formed CSV from data is straightforward and almost trivial…
You didn't mention handling new lines as data. Excel for example will include these as-is and double quote the cell. It is very easy to overlook edge cases in CSV.
> It is very easy to overlook edge cases in CSV.
That was not a criticism from the original article, and isn't even true.
If you have newlines in your original data, and you "overlooked" this "edge case", then neither JSON nor YAML nor any other format will save you. The same fix applies to them all.
This is really a very poor criticism.
Re: Time to retire the CSV?
#507Earlier quoted context omitted.
Most XML is not human-readable. There's just too much line noise, and most of what's emitted has a weird schema that's difficult to parse using Human Brain 1.0.
I don't think it's much worse than CSV, with it's inflexible structure, not allowing for any formatting to be inserted. As for weird schemas, that really is an implementation specific issue. This example was sketched up in minutes and already reads better than most spreadsheets I've seen over the years! 100 237.87
While weird schemas are an implementation-specific issue, the fact remains that most XML that I've seen in commercial settings is exactly as unreadable as I described. It's harder to go against the grain of the status quo when very few people accompany you on the journey.
Re: Time to retire the CSV?
#508Earlier quoted context omitted.
CSV is far from perfect, but it's nice that I can easily work with them without needing any libraries. All I need is file I/O and the ability to split strings. It doesn't get much simpler. I'll admit though that "import JSON" and then being able to essentially convert the entire file into a dictionary is nice if the data has more structure to it.
> All I need is file I/O and the ability to split strings. ...until there is a newline inside a field. The moronic quoting mechanism of CSV is one half of the problem; people like you, who try to parse it by "just splitting strings" is the other half. The third half is that it's locale dependent and after 30+ years, people still don't use Unicode.
You've assumed an awful lot about my use cases. The data I deal with in .CSV form is always pre-processed and doesn't have any of the minefield occurrences you've mentioned. There can't be a newline or anything like that in an input. In my decade of using .CSV files daily, I've only had one tertiary system where that is a problem.
Also, when doing interactive work, it's a bit different than writing production IT software.
Re: Time to retire the CSV?
#509Earlier quoted context omitted.
> All I need is file I/O and the ability to split strings. ...until there is a newline inside a field. The moronic quoting mechanism of CSV is one half of the problem; people like you, who try to parse it by "just splitting strings" is the other half. The third half is that it's locale dependent and after 30+ years, people still don't use Unicode.
Never write your own parser, especially as just string.split(), and when possible don't use C(omma)SV formats, but C(haracter)SV, aka DSV: https://en.wikipedia.org/wiki/Delimiter-separated_values There are non-printable, non-typable characters specifically defined as separators (ASCII 28-31) with UTF equivalents.
As printed above, I'm not writing software where that would be a problem. In general if I was writing a commercial application or something in production where I don't control all the inputs I would agree 100%, but I'm lucky to not have those problems. I could use a bunch of libraries and additional code to try to catch non-existent errors I don't have, or fix my actual problems and move on.
I appreciate the perspective though for sure. I'm guilty of the same thing on HN, where I assume people have the same uses as me when they're writing code that has to be extremely robust or blazingly fast.
Re: Time to retire the CSV?
#510Earlier quoted context omitted.
Of course there is a old solution in the ANSI character set. File, Record, Group and Unit separator characters
Yes. You could get a long way with a text format in which: -the first line is always a header -fields are separated by Unit separator characters -records are separated by Record separator characters -encoding is UTF8 If you wanted to get fancy you could also have: -comment lines -column metadata (e.g. column 0 is an ISO date, column 2 is text, column 3 is an integer) Both the above could start with a Unicode characte…
But it sounds like a very very good approach.