Please don't. CSV is one of the worst file formats ever conceived.
Use (compressed) line-delimited JSON if you need a file of records.
Consider Using CSV
91–100 of 112 posts
Re: Consider Using CSV
#92Earlier quoted context omitted.
>CSV is superior to JSON for tabular data Can't you just do this? { "columns": ["col1", "col2", "col3"], "data": [ [1, 2, 3], [4, 5, 6], [7, 8, 9] ] } That's valid JSON but it's human-readable and human-editable rows of comma-separated data, just like CSV.
You can. But I don't see how that is superior to the equivalent CSV.
Until you try to either cram it into Excel, work with different encodings or pass it around different software platforms (even your lang vs JS), no difference.
CSV has been abused a lot to make it work on conflicting use-cases, JSON handles a lot of misshaps happened with delimiter-separated record formats, like new-lines or bring-your-own-character encoding.
Re: Consider Using CSV
#93Earlier quoted context omitted.
It's not a separator character, but at least vim and emacs acknowledge the page feed character. A pittance, I suppose.
> It's not a separator character, Isn't it? I thought all the separator characters (0x1e, 0x1f, 0x1c) were specifically for delimiting records, fields and units. What are they for?
They’re saying that “page feed character” (I’m guessing form feed) is acknowledged by Emacs, in contrast to those separator characters.
I think it’s used to mark sections in Emacs Lisp code.
Re: Consider Using CSV
#94Earlier quoted context omitted.
Yes, I feel like this would've been more helpful generalized as "Consider DSV" (delimiter-separated values) than CSV specifically, because of the interop issues that often come up. I'd have also mentioned using Parquet.
but which delimiter. if you choose pipe ok, now you have to make sure nobody typed a pipe into the input field or spreadsheet, and you cannot store unix commands if you choose tab, ok, now people will get confused when they try to edit the text file to replace tabs with spaces, and now you have trouble putting code snippets into data fields because they have tabs. this is the problem and it's why xml/json exist. in m…
Both pipe and tab are infinitely better for so-called human-readable data compared to comma. Comma doesn’t even work well for numbers since some locales use comma as the decimal separator. And a data format can’t be “human-readable” if you’re not allowed to write numbers in the way that you’re used to write them.
Re: Consider Using CSV
#95Earlier quoted context omitted.
I want to use parquet more frequently, but it creates new problems that do not exist if I dump to CSV. Last I looked, there were not any good GUIs that would let someone quickly browse the data. Now it is just a blob lacking introspection. CSV has issues, but it is universal.
Not a GUI tool but try Visidata for looking inside Parquet files (and other tabular formats) https://www.visidata.org/
A bit round-about, but the slick way I discovered is to take a detour through DuckDB. DuckDB offers parquet bindings which you can link through a kind of foreign data interface and then query through SQL. Using this, you can then just browse parquet files through DBeaver or your IDE of choice. Hardly an out of the box solution I can offer to a random collaborator, but fantastic for your savvy analyst.
Re: Consider Using CSV
#96> It's only 77 bytes, with 29 for the header and 24 for each line. At 100,000 entries, this list would be 2.4 MB (that's ~63% less than the JSON). If size is really the issue but you still want schema enforcement protobuf is the way to go.
I'd go with sqlite instead. Also, there are specialized formats like Parquet
Re: Consider Using CSV
#97Earlier quoted context omitted.
Not a GUI tool but try Visidata for looking inside Parquet files (and other tabular formats) https://www.visidata.org/
A bit round-about, but the slick way I discovered is to take a detour through DuckDB. DuckDB offers parquet bindings which you can link through a kind of foreign data interface and then query through SQL. Using this, you can then just browse parquet files through DBeaver or your IDE of choice. Hardly an out of the box solution I can offer to a random collaborator, but fantastic for your savvy analyst.
[deleted]
Re: Consider Using CSV
#98CSV looks deceptively simple. It is far too easy to just write(','.join(whatever)), which sort of works, until it doesn’t, and then someone, sometimes I, has to sort out the resulting mess. PLEASE use a proper CSV library (Python comes with a CSV module in the standard library), or at least implement the entire format according to the RFC from the outset, even if you think you won’t need it!
This is a matter of developer education. The correct way to create and parse CSV files is to use a third-party library. They can get complicated. A field in a CSV can contain commas and quotes. In some cases, a single field can contain a line-feed, and you'll need to ensure the parser you use supports that. This would allow an entire CSV file to be embedded inside the field of a CSV field. At a minimum, a parser must support Excel's default parser logic.
But, if you pick the right parser and generator, then you're ok with using it.
Re: Consider Using CSV
#99I'm definitely in the "Just use JSON for most things" camp, but I'm wondering, why would you ever choose CSV for interfacing microservices over protobuf?
Isn't protobuf basically CSV but with good libraries at the interface point and standards around how to deserialize the streams?
Re: Consider Using CSV
#100As much as I like and use CSV for database work, it has a problem with being poorly specified. The most common problems are when processing CSVs produced elsewhere which might not enclose text fields with quotes and thus have issues with data that includes commas and multi-line data.
There is a spec (RFC 4180 [1]) but it's definitely not widely followed. Worse, for a lot of data there's no problems for potentially years , until your numbers get too big or the first time a quote or comma gets in the data. In my experience one of the biggest barriers I run into -- and the primary reason I hate using CSV -- is Microsoft Excel. It misinterprets numbers as dates, it convers big numeric identifiers to…
Regarding excel and rfc4180…
I avoided CSV for quite a while because I had excel-vs-CSV compatibility concerns like this.
However, when I tested this for myself a few years back, Excel output to my surprise was rfc4180 or darn near it (it might use CRLF rather than LF?) It emitted commas and quotes the same way as the rfc for all the test cases I checked.
That said, I agree with you Excel is problematic as an input source. Usually the problems are the humans who touch the data in excel, but what I’ve found is the automation problems tend to be with Excel parsing and interpreting incoming data (before it goes to CSV.) Exponents, trimming leading zeros, etc. as you say. But if the data is confirmed good in excel before being emitted, the CSV it emits is decent.
Counterexamples welcome.