The least fun csv-like format i have seen was mandated by a Big Spanish Bank. It was original a fixed fieldwith flatfile. Numerals were 0-leftpadded and fixed decimal (decimal point not written). Except that someone decided they needed to open it in Excel so semicolons were added at the end of each field. Then they discovered that some data didn't actually fit in the fixed with fields. So certain fields where variable width.
End result was a file that looked like shit in Excel, and couldn't be parsed by a fixed width parser.
Time to retire the CSV?
81–90 of 594 posts
Re: Time to retire the CSV?
#82CSV will never go away. But surely we can build better systems to handle all the complexities and variability in CSV? If we can handle image recognition with all its complexities, we can handle the more constrained variety in CSVs. There are only so many variations to CSV files, and perhaps a better preprocessor is the answer here versus asking for the near-impossible task of "retiring" CSV.
[deleted]
Re: Time to retire the CSV?
#83XSLX covers a lot of ground in this area, works with the market leading spreadsheet program, but there are also great libraries to encode and decode them in Python, Java and other languages.
I agree, xlsx is pretty easy to deal outside Excel (at least until you hit 1M rows!).
Re: Time to retire the CSV?
#84Surely the successor should be SQLite. Tabular data like CSV, easy to view on different mediums, free and open source, single file databases. It has to be this, right?
Replacing an almost trivial open format with one that requires a specific 85 kLOC codebase to access does not seem like an improvement. SQLite is wonderful, but it's definitely not a replacement for CSV
> Replacing an almost trivial open format
The main feature of CSV is that it's aways broken, not that it's almost trivial.
> with one that requires a specific 85 kLOC codebase to access
sqlite is available essentially everywhere. Every browser uses sqlite extensively internally, so do tons of other software on your standard desktop machine (to say nothing of mobile OS where it's ubiquitous). Using sqlite is not any sort of constraint.
> SQLite is wonderful, but it's definitely not a replacement for CSV
Only in the sense that it's nowhere near as bad and annoying as CSV.
Re: Time to retire the CSV?
#85The whole point of CSV is that it's simple. You don't need and special libraries. If you can write hello world to a file you can make a CSV and more often than not it'll just work. Yes somethings, like DNA, shouldn't be CSV files. But don't blame the tool for bad craftsmanship. There are more things that it works for than doesn't. As a side note dates should be ISO 8601
[deleted]
Re: Time to retire the CSV?
#86Earlier quoted context omitted.
As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. I definitely can't do that with .xlsx. There is almost zero friction to bolting CSV export capability to an existing system, which is part of why it's so popular.
> As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. And usually generate garbage for anything but the most trivial case, which really nobody gives a shit about. That's the main reason why CSV absolutely sucks too, you have to waste month diagnosing the broken shit you're given to implement the workarounds necessary to deal with it. > I definitely can't do that with .xlsx…
People doesn't matter which format is, since it simple work in any spreadsheet software.
Yeah i can generate any file with a bunch of printf, but csv i dont have to read a specification, i its possible to read with a bunch of read without have to use a xml or xlsl library.
Re: Time to retire the CSV?
#87CSV is bad - but convenience is hard to beat. It's the same thing with excel. People abuse it, but you just can't beat the fact that your programmer, CEO, analyst, and secretary can all contribute to the same file.
Yeah, this is pretty much it. The author complains about CSVs being "notoriously inconsistent" as though switching to some other format would magically change that. They're only inconsistent because sometimes lazy programmers do ",".join(mylist) instead of using an RFC4180 compliant CSV writer. Lazy programmers will just use non-compliant methods of creating whatever magic format OP is dreaming about. Case in point:…
>CSVs also are great because you can parse them one row at a time. This makes for a very scale-able and memory-efficient way of processing very large files containing millions of rows.
Even RFC4180-compliant CSVs can be incredibly memory-inefficient to parse. If you encounter a quoted field, you must continue to the next unescaped quote to discover how large the field is, since all newlines you encounter are part of the field contents. Field sizes (and therefore row sizes) are unbounded, and much harder to determine than simply looking for newlines - if you were to naively treat CSV as a "memory-efficient" format to parse, you would create a parser that would be easy to blow up with a trivial large file.
Re: Time to retire the CSV?
#88More correctly titled, "I Don't Like CSV".
There are probably valid uses for CSV, but more often than not, it's the wrong choice. As soon as data has any form of structure to it (and most data does). CSV complicates everything. Even for unstructured data, the problem of escape characters often shows it's ugly head. The moment your data contains a comma, tab, or space, you run into a nasty mess that, in the best case makes your system fail, and in the worst ca…
Just an FYI, JSON and XML both have characters that need to be escaped properly.
Re: Time to retire the CSV?
#89They complain about the loss of metadata, but never get into specifics. I'm surprised they didn't mention the biggest example of this. Excel is forced to guess the type of input data it's reading, and sometimes it guesses wrong and messes up your data. A number of genes had to be renamed because too many genetic markers were being interpreted as dates: https://www.theverge.com/2020/8/6/21355674/human-genes-renam...
Re: Time to retire the CSV?
#90HDF5 is efficient for large datasets and has its use cases but being a binary (and fairly easy to corrupt) it isn't a reasonable general replacement for CSV. It also has limitations such as it cannot be read while it is being written.