Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

391–400 of 459 posts

Re: Friends don't let friends export to CSV

#391

Post by rkaveland »

smcin wrote: post #355
rkaveland wrote: post #261

Author here. I see now that the title is too controversial, I should have toned that down. As I mention in the conclusion, if you're giving parquet files to your user and all they want to know is how to turn it into Excel/CSV, you should just give them Excel/CSV. It is, after all, what end users often want. I'm going to edit the intro to make the same point there. If you're exporting files for machine consumption, pl…

Well what would be a more accurate title? "CSV format should only be for external interchange or archival; columnar formats like Parquet or Arrow better for performance"? People are busy; instead of hinting "something more robust than CSV", mention the alternatives and show a comparison (load time/search time/compression ratio) summary graph. (Where is the knee of the curve?) There's also an implicit assumption to ea…

> Well what would be a more accurate title? "CSV format should only be for external interchange or archival; columnar formats like Parquet or Arrow better for performance"?

Something more boring, like "Consider whether other options make more sense for your data exports than CSV". Plenty of people have suggested other good options in comments on this submission, such as for example sqlite. I think the post comes off as if I'm trying to sell a particular file format for all uses cases, when what I had in mind when writing it was to discourage using CSV as a default. CSV has a place, certainly, but it offloads a lot of complexity on the people who are going to consume the data, in particular, they need to figure out how to interpret it. This can't necessarily be done by opening the file in an editor and looking at it, beyond a certain size you're going to need programming or great tools to inspect it anyway.

I was given an initial export of ~100 poor quality CSV files totaling around 3TB (~5-6 different tables, ~50 columns in each) in size a few years back, and had to automate ingestion of those and future exports. We could've saved a lot of work if the source was able to export data in a friendlier format. It happened more than once during that project that we were sent CSVs or Excel sheets that had mangled data, such as zip codes or phone numbers with leading 0s removed. I think it is a good thing to inform people of these problems and encourage the use of formats that don't necessity guessing data types. :shrug:

> People are busy; instead of hinting "something more robust than CSV", mention the alternatives and show a comparison (load time/search time/compression ratio) summary graph. (Where is the knee of the curve?)

This might be an interesting thing to follow up later, but would require a lot more work.

Re: Friends don't let friends export to CSV

#392

Post by mardifoufs »

burntsushi wrote: post #365
vmchale wrote: post #356

Earlier quoted context omitted.

its parser is buggy! https://github.com/BurntSushi/xsv/issues/337 (I ran into this issue myself)

I just responded to that. It isn't the parser that's a buggy. The parser handles the quotes just fine. If it didn't, that would be a serious bug in the `csv` crate that oodles of users would run into all the time. There would be forks over it if it had persisted for that long. The problem is that `xsv table` doesn't print the parsed contents. It just prints CSV data, but with tabs, and then those tabs are expanded to…

Ok this might sound stupid, and a bit unrelated, but you make so many great tools that I can't help but ask. How do you start planning and creating for a tool that needs to "follow standards"(in this case I know CSV is under specified but still!), is it by iteration or do you try to set and build a baseline for all the features a certain tool needs? Or do you just try to go for modularity from the get go even if the problem space is "smaller" for stuff like csv for example.

Re: Friends don't let friends export to CSV

#393

Post by mardifoufs »

maxcoder4 wrote: post #323
prmoustache wrote: post #289

Earlier quoted context omitted.

> if there was a strict governing body for it that wasn't a Webster-style "whatever people are speaking is the new definition of correct". There is no way it can work. People don't care about governing bodies when they speak a language.

They kind of do. In my language there exists a central "governing" body that decides what is correct, and some "incorrect" regionalisms are disappearing because of it.

Are you referring to french? Because, if anything, french in France has an insane quantity of slangs and has an extremely emergent vocabulary. Much more so than any English speaking country I can think of. Quebec isn't really influenced by the Académie française yet has a much more "correct" usage of the language generally speaking.

Maybe it's a totally different language but still it goes to show that even a very prestigious central authority doesn't make a language better or less prone to diverge. Regardless of the reason, French is evolving much more quickly than English.

Re: Friends don't let friends export to CSV

#395

Post by alex_suzuki »

euroderf wrote: post #288
alex_suzuki wrote: post #252

Earlier quoted context omitted.

They‘re used a lot in barcodes, e.g. for delimiting the different fields of a driving license.

That sounds like the premise for an utterly fascinating deep dive.

It‘s quite the rabbit hole, I can assure you.

Re: Friends don't let friends export to CSV

#396

Post by nmz »

outop wrote: post #216
nmz wrote: post #74

Earlier quoted context omitted.

oh boy. here's where it breaks - supporting "" (single) - Supporting newlines in "", oops, now you can't getline() and instead need to getdelim() - Supporting comments # (why is this even a thing) - Supporting multiple "" in a field - Escaping " with "" or \" - length based csv, so all fields are seekable. It's a mess, which one's your csv?

The vast majority of CSVs do not have strings which include either quotes or newlines. No CSV I have ever encountered has comments.

Google contacts does.

Re: Friends don't let friends export to CSV

#397

Post by ianburrell »

brian-bk wrote: post #173
josephg wrote: post #80

Earlier quoted context omitted.

Eh. I much prefer to produce and consume line delimited JSON. (Or just raw JSON). Its easy to parse, self descriptive and doesn't have any of CSV's ambiguity around delimiters and escape characters. Its a little harder to load into a spreadsheet, but in my experience, way easier to reliably parse in any programming language.

JSON(newline delimited or full file) is significantly larger than csv. With csv the field name is mentioned once in the header row. In JSON every single line repeats the field names. It adds up fast, and is more of a difference than between csv to parquet.

That is only if the JSON uses objects. JSON arrays map much better to CSV. In that case, only adding brackets to the front and end of each line.

The ability of JSON to do both objects and arrays is useful, for example the first line can be an object or array of objects describing the fields. Then there is less confusion between schema lines and data lines like there is with CSV.

Re: Friends don't let friends export to CSV

#398

Post by deathanatos »

Dalewyn wrote: post #211
deathanatos wrote: post #181

Earlier quoted context omitted.

> Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone. While I'm not a fan of USV, I do believe it is type-able on both macOS and Linux¹. The IME's character picker in both OSes contains all of the necessary characters, I think. (I use "␤" in commit messages, occasionally. That was a typed ␤, not copy/pasted from a Google query or such.) It would be quite tedious , I do admit…

Any character within reason can certainly be entered by way of Character Map in Windows or its equivalent in Linux or MacOS, but if you're arguing that then you don't understand the crux of my argument: Convenience is a virtue. There is value in the delimiter simply being a key on practically any keyboard in existence. Anything that involves something more complicated than just pushing a single button on a keyboard (…

> Anything that involves something more complicated than just pushing a single button on a keyboard (this includes IMEs)

My point is that this is merely a more stringent argument; it's now "on a keyboard, and cannot involve dead keys, etc." … which now excludes classic CSV, too, which requires two keys to enter a double quote. (Shift+')

Again, it does require more keys, and it is certainly not convenient, but n keys to me is still n keys. The real question is why one isn't using a proper tool to enter the data, and is instead encoding it by hand, which, again, even for a classic CSV, is basically something I've never done. (… because why would one?)

Re: Friends don't let friends export to CSV

#399

Post by deathanatos »

al_borland wrote: post #201
deathanatos wrote: post #181

Earlier quoted context omitted.

> Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone. While I'm not a fan of USV, I do believe it is type-able on both macOS and Linux¹. The IME's character picker in both OSes contains all of the necessary characters, I think. (I use "␤" in commit messages, occasionally. That was a typed ␤, not copy/pasted from a Google query or such.) It would be quite tedious , I do admit…

Even if the csv isn’t being typed out by hand, when importing into Excel and the delimiter needs to be manually entered, because it isn’t one of the common ones Excel has a radio button for… it is nice to be able to easily type it.

While I can see a convenience argument for the somewhat contrived case of typing an entire file out by hand, entering the character once for the import does not seem like a great bar.

> it is nice to be able to easily type it.

Again, that's where an IME is helpful; on the OSes I mention, it's "␞" is:

  macOS: ⌘+^+Space, "record", , Enter
  Linux: Super+e, "record", Space, Space, , Enter
The process is highly visual on both, so you're getting feedback about whether you're about to hit the right character, or not.

(And like, if you have the file, you can always ^C ^V the character, if say you don't know how to IME, or you don't know what the name of the char is, etc.…)

Re: Friends don't let friends export to CSV

#400

Post by unsupp0rted »

reportgunner wrote: post #389
unsupp0rted wrote: post #343

I wish I could get Excel to stop converting Product UPCs to scientific notation when opening CSVs. Also some UPCs start with 0 Worst is when Excel saves the scientific notation back to the CSV, overwriting the correct number.

1. Open a blank workbook 2. Enable the legacy Text Import Wizard as per [0] 3. Go to Data -> Get Data -> Legacy Wizards -> From Text (Legacy) 4. Set config based on your CSV file, typically select "Delimited" and "My data has headers" enabled 5. Click Next and pick the delimiter, typically "Comma" 6. Click Next and click the columns with UPCs, select "Text" in the "Column data format" area 7. Click Finish (I'm not sa…

Wow! I had no idea you could set data format on legacy text columns during import. I had thought the column preview was just that- not a selectable radio button that you can then apply column data formatting to.

Thanks for the instructions! It's cumbersome as heck, but it's better than nothing.

Post reply on HN