201–210 of 594 posts
Earlier quoted context omitted.
By this argument UTF8 can't exist. And yet here it is. PS: I never said 100% forward/backward compatible with all variant at the same time and without any noticeable artifact. I meant compatible in a non blocking way.
What are you talking about? UTF8 is a single well-defined specification, and detecting that data is definitely not UTF8 is trivial.
Earlier quoted context omitted.
> As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. And usually generate garbage for anything but the most trivial case, which really nobody gives a shit about. That's the main reason why CSV absolutely sucks too, you have to waste month diagnosing the broken shit you're given to implement the workarounds necessary to deal with it. > I definitely can't do that with .xlsx…
As soon as you open it in Excel, it's garbage anyway, since it will replace date-like items with nonsense, drop number digits, convert anything it can, reencode monetary unities, and so on. If you don't open it in Excel, you can have as strict a parser as you want, just like any other format. But neither is going anywhere anyway.
No, you can not. Because the CSV format is so fuzzy you can very easily parse incorrectly and end up with a valid parse full of garbage.
Trivially: incorrect separator, file happens to not contain that separator at all, you end up with a single column. That's a completely valid file, and might even make sense for the system. Also trivially: incorrect encoding, anything ascii-compatible will parse fine as iso-8859-*. Also trivially: incorrect quoting / escaping, might not break the parse, will likely corrupt the data (because you will not be stripping the quotes or applying the escapes and will store them instead).
It's like you people have never had to write ingestion pipelines for CSVs coming from randos.
It's probably not news to anyone who works with data that CSVs are inferior to other file formats like parquet. I think the valuable insight here is that there needs to be a meme / movement that CSV is bad or deprecated. That's what's actually going to put the nails in its coffin, not private griping from developers when they get CSVs. I'm all for it. Down with CSV :)
The thing is, I need a hefty library to write parquet. I can write a CSV with a couple fprintf statements and a loop. A parser isn't much harder if we can make assumptions about CSV dialect ahead of time. If I'm trying to make an exportable format for a data logger with an SD card running on an ARM microcontroller, it doesn't get much easier than CSV. Sure, I could save space by rolling my own binary format, but then…
that is... if you're following the RFC 4180 guidance for CSV. microsoft has another internal standard that occasionally gets referenced.
Earlier quoted context omitted.
1. open(2) doesn't care about the file format 2. Yes, mixing formatting with data slows down data processing, don't do it. 3. Excel is not the replacement for CSV, and CSV is not a compact format. I mean, maybe if you are used to XML it is, but otherwise, just no. Yes, there is truly a problem.
> CSV is not a compact format That's only true if you're trying to send all your data in a single, monolithic CSV. If you're sending multiple CSVs, you're capable of representing data as well as a relational data store. Which is to say, you're representing your data using a system of data normalization specifically designed to minimalize data duplication. A single CSV represents a single table, and in most cases with…
No, that's true in general. There are exceptional cases where it might not be true, but it's true in general.
> Yes, you do have situations where you're storing losing data density due to using plain text strings, but that's not a limitation particularly unique to CSV for data serialization formats.
That there are other inefficient data serialization formats does not mean that it is not an inefficient data serialization format.
> Additionally, it is a problem that can largely be mitigated by simple text compression.
"Simple text compression"? Not sure what you are referring to there. If you are speaking of generalized compression, there's a bunch of information theory pointing out the problem with that thinking.
> Furthermore, once you switch to a non-text representation, you're limiting yourself to whatever that data representation is. It's easy to represent an arbitrary precision decimal number in plain text. It's hard to find a binary representation that universally represents the same data regardless of the system on the other end. Again, that's not a problem unique to CSVs.
It's pretty easy to represent arbitrary precision decimal numbers in non-text formats and they mean the same thing regardless of the system on the other end, but let's say you were right. Great, now represent arbitrary precision binary numbers (you know, the kind that computers work with), in text formats.
> If you're working with an API, object by object, then JSON is certainly going to be better,
Let me stop you right there. It is not "certainly going to be better". JSON is generally worse.
> If you're working with bulk data of many disparate, unrelated, complex objects, however, or where you're transferring and entire system, you're not going to do much better than CSV.
There's a whole world out there of formats that are better than CSV at this job, and in many cases they are A LOT better than CSV. Even Avro, for all its pitfalls, is MUCH better.
Earlier quoted context omitted.
You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…
If you interpret "CSV" as purely comma seperated values then maybe. But in my bubble "CSV" means textfiles that are separated by some separator. Be it tabs, spaces, commas, or any other ASCII character. Some are more usable then others, if you have commas in your data then use tabs. If you have tabs use Form Feed or Record Separator or vertical tabs ... and so on. Of course this is not always applicable, since you so…
When I see CSV parsers like https://www.papaparse.com/ that even try to support comments and empty lines in the format, I wonder if it'd really be that bad to just raise an error on anything that doesn't fit RFC 4180[1], with explanation on where and how the file is corrupted. Push the issue to the writers of such files.
Then I remember the Robustness Principle[2] and I chill a little.
Microsoft -> The name of the format is 'comma separated values' not 'semicolon separated values'!
I'm going to get downvoted, but I'll ask, who cares? You may have perfectly valid reasons to not like CSV, but it's choice at the end of the day to use it or not. Maybe you didn't make the choice, and inherited a system that does, but I much rather read an article about how you changed minds and/or switched to something more advantageous for you and your system versus what amounts to a rant. It's not time to retire C…
Earlier quoted context omitted.
> A truly open format is available and accessible. Sqlite? > Applications have a speed increase from using csvs. Sqlite? > The new format should not be grossly larger than the one it is replacing Sqlite it is. -------- Oh, you mean something that Excel can open? Oh yeah, I guess CSV then. But lets not pretend #1 (openness), #2 (speed), and #3 (size) are the issues.
I'd add the constraint that it needs to be understandable by most people. A typical non-tech office worker can use CSV without any trouble. Sqlite has an extremely steep learning curve for someone unfamiliar with databases.
Writing code that can accept and parse arbitrary CSV files is a whole different thing. If I had to do that, I'd be yelling too.
Similar story can be said for JSON, markdown, etc, where standard is inadequate, non-existent, or in mutual competition.