Also CSV can be queried : https://til.simonwillison.net/sqlite/one-line-csv-operations
A love letter to the CSV format
121–130 of 711 posts
Re: A love letter to the CSV format
#122Earlier quoted context omitted.
I always treat CSVs as comma separated values with new line delimiters. If it’s a new line, it’s a new row.
Do you ever have CSV data that has newlines within a string?
I don't. If I ever have a dataset that requires newlines in a string, I use another method to store it.
I don't know why so many people think every solution needs to to be a perfect fit for every problem in order to be viable. CSV is good at certain things, so use it for those things! And for anything it's not good at, use something else!
Re: A love letter to the CSV format
#123I have to agree.
It was pretty straightforward (although tedious) to write custom CSV data exports in embedded C, with ZERO dependencies.
I know, I know, only old boomers care about removing pip from their code dev process, but, I'm an old boomer, so it was a great feature for me.
Straight out of libc I was able to dump data in real-time, that everyone on the latest malware OSes was able to import and analyze.
CSV is awesome!
Re: A love letter to the CSV format
#124Earlier quoted context omitted.
JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…
What happens when you need to encode the newline character in your data? That makes splitting _either_ CSV or LDJSON files difficult.
The new line character in a JSON string would always be \n. The new line in the record itself as whitespace would not be acceptable as that breaks the one line record contract.
Remember that this does not allow arbitrary representation of serialized JSON data. But it allows for any and all JSON data as you can always roundtrip valid JSON to a compact one line representation without extra whitespace.
Re: A love letter to the CSV format
#125CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…
JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…
Eh, it really isn't. The format does not lend itself to tabular data, instead the most natural way of representing data involves duplicating the keys N times for each record.
Re: A love letter to the CSV format
#126Earlier quoted context omitted.
Set a bigger tab size in the editor, e.g. `:set ts=32` in vim.
Aaaand now I can fit like 3 fields next to each other before they wrap and the field with arbitrary length text still misaligns the field after it
There is at least a chance that the text fields will fit in a tab. If not, a tool like “column” on Linux can be used. There is no chance that a text field will fit inside the width of a comma.
Re: A love letter to the CSV format
#127I think I understand the point being made, but all this reliance on text-based data means we require proper agreement on text encodings , etc. I don't think it's very useful for number-based data anyway, it's a massively bloated way to store float32s for instance and usually developers truncate the data losing about half of the precision in the process. For numerical data, nothing beats packing floats into blobs.
I think binary formats have many advantages. Not only for numbers but other data as well, including data that contains text (to avoid needing escaping, etc; and to declare what character sets are being used if that is necessary), and other structures. (For some of my stuff I use a variant of DER, which adds a few new types such as key/value list type.)
Re: A love letter to the CSV format
#128Earlier quoted context omitted.
CSV is a pseudo-standard anyway, IMO the delimiter should be a configurable option (like it is in Unix cut and those kinds of tools).
> IMO the delimiter should be a configurable option It is. CSV has been character separated vs comma separated for probably decades now. Most tools you'd use to mess with them allow you to specify which separator character is being used.
I agree although it seems to render the distinction made by GP moot, right?
Re: A love letter to the CSV format
#129"the controversial ex-post RFC 4180" I looked at the RFC. What is controversial about it?
Look at how it handles escaping of special characters and particularly new lines (RFC 4180 doesn’t guarantee that a new line is a new record) and how it’s written in 2005 yet still doesn’t handle unicode other than via a comment about ”other character sets”.
> RFC 4180 doesn’t guarantee that a new line is a new record
Correctly. A good parser should step through the line one column at a time, and shouldn't even consider newlines that are quoted.
If you're naively splitting the entire file via newline, that isn't 4180's fault, that is your fault for not following the standard or industry norms.
I'll happily concede the UNICODE point however; but I don't know if that makes it controversial.
Re: A love letter to the CSV format
#130Earlier quoted context omitted.
That would be solved by using the ASCII control chars Record Separator / Unit Separator! I don't get how this is not widely used as standard.
If there were visible well known characters that could be printed for those and keys on a keyboard for inputting them we would probably have RSV files. Because they are buried down in the nonprintable section of the ASCII chart they are a pain for people to deal with. All it would have taken is one more key on the keyboard, maybe splitting the tab key in half.
> If there were visible well known characters that could be printed...
...There would be datasets that include those characters, and so they wouldn't be as useful for record separators. Look into your heart and know it to be true.