Earlier quoted context omitted.
A manager: I wish I could have a CSV inside my CSV. Any sane person: NO!
base64 encoded fields it is.
That was the right answer. You are hired!
Friends don't let friends export to CSV
251–260 of 459 posts
Re: Friends don't let friends export to CSV
#252Earlier quoted context omitted.
Does it have nesting operators? I want to embedd ASCII within my ASCII fields. So I can have a table within my table.
It kind of does. See `man ascii` * FS (0x1C) file separator * GS (0x1D) group separator * RS (0x1E) record separator * US (0x1F) unit separator I've never seen these in the wild though.
They‘re used a lot in barcodes, e.g. for delimiting the different fields of a driving license.
Re: Friends don't let friends export to CSV
#253Earlier quoted context omitted.
"Always no"? What. Really not sure what you mean here when you agree that the possibility of it working exists. I get the sentiment -- when I request data through FOIA, I will almost always request it as "an excel format" because I know that I'll at least be able to import it. CSV is much less of a guarantee and will have issues -- missing quotes, wrong delimiters, inconsistent column counts, things like that. So req…
> when I request data through FOIA Fascinating. Can you share any details? Did you ever think to share some of your interesting finds here on HN as a submission?
Every now and then, yep :)
https://mchap.io/that-time-the-city-of-seattle-accidentally-...
Re: Friends don't let friends export to CSV
#254"You give up human readable files, but what you gain in return is..." Stop right there. You lose more than you gain. Plus, taking the data out of [proprietary software app my client's data is in] in csv is usually easy. Taking the data out in Apache Parquet is...usually impossible, but if it is possible at all you'll need to write the code for it. Loading the data into [proprietary software app my client wants data p…
In my experience, human readable file formats are a mistake. As soon as people can read a single file they think that that's the entire format and that it's okay to write it by hand or write their own code for it. And when everyone writes code based on the just what they've personally seen about a format, everyone is sad. This is why not a single piece of software on earth uses the CSV RFC. This is why people hand yo…
I think a takeaway could also be not to give people options when making a human-readable format. "you always need quotes, they're not optional" solves the comma problem. "the delimiter is always a comma" solves the delimiter problem. json has also fared better than csv, I'd say.
Re: Friends don't let friends export to CSV
#255This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…
It's a premature optimization issue. If you don't have special requirements like IO throughput, or mission critical data accuracy guarantees, be biased towards picking the format that anyone can easily open in a spreadsheet.
You can open out easily, but just as easily it can be wrong. So with this bias you'd still not export csv, you'd use xls
Re: Friends don't let friends export to CSV
#256Earlier quoted context omitted.
Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…
> in the real world the spec is essentially "Can you import it to Excel?" And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.
[deleted]
Re: Friends don't let friends export to CSV
#257Or export to CSV correctly and test with Excel and/or LibreOffice. Honestly CSV is a very simple, well defined format, that is decades old and is “obvious”. I’ve had far more trouble with various export to excel functions over the years, that have much more complex third-party dependencies to function. Parsing CSV correctly is not hard, you just can’t use split and be done with it. This has been my coding kata in eve…
> Honestly CSV is a very simple, well defined format, that is decades old and is “obvious”
This is the problem though. Everything thinks it is "obvious" and does their own broken implementation where they just concatenate values together with commas, and then outsources dealing with the garbage to whoever ends up with the file on their plate.
If a more complex, non-obvious format was required, instead of "easy I'll just concatenate values" they might actually decide to put engineering into it (or use a library)
Re: Friends don't let friends export to CSV
#258The reason CSV is popular is because it is (1) super simple, and (2) the simplicity leads to ubiquity. It is extremely easy to add CSV export and import capability to a data tool, and that has come to mean that there are no data tools that don't support CSV format. Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Pa…
> But Protobuf is not a splitable format, which means it is not Big Data friendly, unlike Parquet, Avro and CSV. What, when I worked at Google concatenating protobuf strings was a common way to concatenate protobufs, they are absolutely splittable. People might not know it but there is a reason they are designed like they are, it is to handle big data as you say. If you mean you can't split a single protobuf, sure, b…
> designed like they are
I laughed. We don't call somebody writing a poor varint serializer for in-house use, then discovering that it doesn't handle negative numbers and floats that well so slapping a couple of hotfixes on top of it that make it necessary to have a protocol specification file, "design".
Re: Friends don't let friends export to CSV
#259Every single use I've ever seen of CSV would be improved by the very simple change to TSV. Even Excel can handle it. It is far safer to munge data containing tabs (convert to spaces, etc), than commas (remove? convert to dots? escape?). The better answer is to use ASCII separators as Lyndon Johnson intended, but that turns out to be asking a lot of data producers. Generating TSV is usually easier than generating CSV.
BTW is Excel better in reading UTF8 files? Last time I checked it can't read it without errors if you just open the file with it without using the import function an explicitly select UTF8.
The trick is using UTF-8 with BOM.
Re: Friends don't let friends export to CSV
#260Earlier quoted context omitted.
Eh. I much prefer to produce and consume line delimited JSON. (Or just raw JSON). Its easy to parse, self descriptive and doesn't have any of CSV's ambiguity around delimiters and escape characters. Its a little harder to load into a spreadsheet, but in my experience, way easier to reliably parse in any programming language.
JSON(newline delimited or full file) is significantly larger than csv. With csv the field name is mentioned once in the header row. In JSON every single line repeats the field names. It adds up fast, and is more of a difference than between csv to parquet.
Compression will make the size overhead disappear instantly.
I do see your point - I know that its less efficient, and its not the best format if you're handling it every day or using it for huge data sets. But for a quick and dirty handoff between programs its lovely. It takes ~5 lines to parse in just about any programming language. And you can do so without pulling in any extra dependencies.
Looking at the downvotes I can see that its a controversial choice. But I stand by it.