441–450 of 594 posts
> CSVs are built for humans, not machines And that's why they're not going away. Trying to add typing to them is probably missing the point that >95% of humans that use them don't need or want to understand that.
When they give me literally anything else I get pissed off and push their task lower in my queue.
This is an example of a genre I like to describe as: programmer objects to a solution that meets everyone's requirements because it doesn't make them feel like a beautiful code-poet. I like elegance as much as anyone. And I think it's a good proxy for other important qualities. But don't prioritize it above building something that actually does the job. Be an engineer.
Ah yes, and then there's https://www.theverge.com/2020/8/6/21355674/human-genes-renam... Not to mention the mess that is exchanging documents between different locales. It's all sunshine and roses until you get your CSVs from an office in a different country (which happens a lot in Europe). CSV gets the job done until it doesn't.
Earlier quoted context omitted.
You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…
>the fact that both of its separators (newlines and commas) can appear as-is inside column values, with a different meaning, if those column-values are quoted, means that there's no way to parallelize CSV processing, because there's no way to read-ahead and "chunk" a CSV purely lexically Yes, this is a major pain. It can be avoided by using Tab separated value (TSV) files, which don't use escaping. But then you can't…
Yet, everbody gets it wrong. Everybody. Thoroughly, fantastically, almost unimaginably wrong.
For example, in the Microsoft world: I come across CSVs in about 5 scenarios, all of them very common, most of them designed to interact: Excel, PowerShell's Export-CSV, SQL Server, Power BI, and the Azure Portal. None of these are obscure. None of these use CSV infrequently. Yet, they're all mutually incompatible!!!
That just blows my mind.
For example, SQL Server will output string ",NULL," to represent a null field instead of just a pair of commas (",,"), so every other tool will convert this to the string "NULL", which is not a null.
PowerShell helpfully outputs the "type" of the value it is outputting, like so:
PS C:\> dir | ConvertTo-Csv
#TYPE System.IO.DirectoryInfo
"PSPath","PSParentPath","PSChildName","PSDrive","PSProvider",...
It just goes on and on.
Different quoting rules. Random ability/inability to handle new lines. Different ways of handling quoted versus unquoted string values. Encoding is UTF-8 by default or not. Handling of quote characters within strings. Etc, etc...
Basically, within one vendor's ecosystem, flagship applications have at best a 50:50 chance of opening arbitrary CSV files.
Don't even get me started on the inherent ambiguities of the format, like interpreting dates, times, or high precision decimals, etc...
Oh, and before I forget: the SQL Server Integration Services team wrote lengthy articles on how their engine can process CSV files faster than the competition. Why is this a feature? Because CSV is a woefully inefficient format and processing it fast is an achievement.
Lastly: By default, SQL Server cannot export table data to a flat data file and round-trip it with full fidelity, in any format. Exchanging just a couple of tables between servers is... not fun.
So yes, a replacement format with an efficient, high-fidelity binary format is long overdue.
It turns out people hate bloat and complexity more than they hate all the ills of CSV put together!
Earlier quoted context omitted.
CSV is still easier to parse because the C++ dudes still refuse to implement some kind of nice operator-overloaded interface like #include std::json myjson("{\"someArray\": [1,2,3,4,{\"a\": \"b\"}]}"); std::cout and the result is we have 50 different rogue JSON libraries instead of an STL solution. Until the STL folks wake up, boost::split can deal with the CSV.
https://github.com/nlohmann/json
> It's Time to Retire the CSV > This column obviously contains dates, but which dates? Most of the world It's time to retire local formats and always write YYYY-MM-DD (which is both the international and the Swedish standard, and the most convenient for parsing and sorting). > A third major piece of metadata missing from CSVs is information about the file’s character encoding. It's bloody the time to retire all the c…
I don't need a library to work with CSV in any language. That alone is a deal breaker is plenty of situations, no matter how widespread the format becomes.
The article's strongest criticism of CSV is that it's easy for someone to mangle it when manually editing. This is true. It's also true for every format. It was weakest when it implied there is no real standard. There is, and it's robust for representing data, even data that includes any combination of commas and double-quotes. The algorithm for creating well-formed CSV from data is straightforward and almost trivial…
It is very easy to overlook edge cases in CSV.
Like
Id(int), Name, LastName, BornAt(date)
1, asd, dsa, 1992-07-12
Earlier quoted context omitted.
You can write what "looks" like CSV to you, but there are no guarantees it will import correctly. The problem is 10x worse when you get CSV from one source and rely on another process to load it. I fought this problem for several days going from NetSuite to Snowflake via CSV.
Can you give an example? The rules for CSV files are so simple I'm struggling to imagine a case where something looks correct but in fact isn't correct.
name,position
"Smith, John＂‚Manager