Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

291–300 of 459 posts

Re: Friends don't let friends export to CSV

#291

Post by stephenr »

samatman wrote: post #155
msla wrote: post #119

Earlier quoted context omitted.

> In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator Heh. Ah, HN. Always good for a laugh line.

Yeah, hon hon hon and all, but one of my (US) bank statements exports a CSV which uses commas in numbers in the US fashion, so $1,500 and the like. Writing a custom CSV munger to intake that into ledger-csv was... fun, but then again, only had to do it once.

Surely if they're putting commas in values they were quoting the values though?

Re: Friends don't let friends export to CSV

#292

Post by wiredfool »

lolive wrote: post #240
josefx wrote: post #235

Earlier quoted context omitted.

Does it have nesting operators? I want to embedd ASCII within my ASCII fields. So I can have a table within my table.

A manager: I wish I could have a CSV inside my CSV. Any sane person: NO!

Been there, done the Interop.

Re: Friends don't let friends export to CSV

#293

Post by sam_goody »

I have all SQL exported to CSV and committed to git once a day (no, I don't think this is the same as WAL/replication).

Dumping to CSV is built into MySQL and Postgres (though MySQL has better support), is faster on export and much faster on import, doesn't fill up the file with all sorts of unneeded text, can be diffed (and triangulated by git) line by line, is human readable (eg. grepping the CSV file) and overall makes for a better solution than mysqldumping INSERTs.

In Docker, I can import millions of rows in ~3 minutes using CSV; far better than anything else I tried when I need to mock the whole DB.

I realize that the OP is more talking about using CSV as a interchange format or compressed storage, but still would love to hear from others if my love of CSV is misplaced :)

Re: Friends don't let friends export to CSV

#294

Post by iainmerrick »

prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

This article seems written by someone who never had to work with diverse data pipelines

I think that's a little unfair, it sounds like the author does have a decent amount of experience working with real-world CSV files:

I remember spending hours trying to identify an issue that caused columns to "shift" around 80% into a 40GB CSV file, and let me tell you, that just isn't fun.

Re: Friends don't let friends export to CSV

#295

Post by prox »

Pikamander2 wrote: post #277
diogocp wrote: post #129

Earlier quoted context omitted.

> in the real world the spec is essentially "Can you import it to Excel?" And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.

I've been amazed by how much better LibreOffice is at importing CSVs in a sane manner than Excel. Its CSV import prompt is nothing short of the gold standard and puts Excel to shame. Also, even if the CSV format is completely valid, Excel will still find a way to misinterpret some of your cells in baffling ways, destroying the original data in the process if you don't notice it in time.

Heck yes, LibreOffice shines when it comes to that. Excel always threw me curveballs a lot.

Re: Friends don't let friends export to CSV

#296

Post by iainmerrick »

bruce511 wrote: post #285
lolive wrote: post #240

Earlier quoted context omitted.

A manager: I wish I could have a CSV inside my CSV. Any sane person: NO!

Well you might be wrong, but EDI in general and HL7 specifically allow 3 levels of "fields in fields in field". As long as your parser copes, and as long as you have appropriate structures to import into, its no big deal.

So now when I'm exporting data I need to know what nesting level it's going to live at so I can generate the correct separators?

I really think that might be the worst idea I've heard for a while!

Re: Friends don't let friends export to CSV

#297

Post by iainmerrick »

hn_throwaway_99 wrote: post #183
albert_e wrote: post #179

Earlier quoted context omitted.

Everyone of us was a beginner at some point. The first time we came across CSV format we likely typed it in notepad by hand. A lot of issues with CSVs are also sometimes troubleshooted by hand-- by manually fixing a quote or a comma. There is value is the ability to do this level of editing and troubleshooting.

> The first time we came across CSV format we likely typed it in notepad by hand. Again, I'm not saying CSVs aren't edited by hand in a text editor, I'm saying they aren't created from scratch in a text editor, even by beginners. USVs are easy to edit in a text editor, too, and I tried viewing and editing USVs with a couple different fonts and had no problems.

If the separators can't easily be typed, how do you add a new cell?

Re: Friends don't let friends export to CSV

#298

Post by iamflimflam1 »

prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

Every integration I’ve ever worked on has started off with high ideas of APIs and nice data standards. And has eventually devolved into “can we just put a CSV file on an FTP site…”. With the inevitable, “it’s not really CSV…”

Re: Friends don't let friends export to CSV

#299

Post by groestl »

Dalewyn wrote: post #162
hn_throwaway_99 wrote: post #158

Earlier quoted context omitted.

I can't remember the last time I, or anyone I've ever worked with for that matter, ever typed up a CSV from scratch. The whole point of USV is that the delimiters can't normally be typed so you don't have to worry about escaping. USV supports displayable delimiters (see https://github.com/SixArm/usv ), so for the much more common case of editing an existing CSV in a text editor, you can just copy and paste.

I've valued the virtue of CSVs being readable by any text editor known to man, and I've occasionally edited them by hand. The pure simplicity of reading and typing commas trumps any value provided by more esoteric configurations. As for escaping, that's for the subsequent programmers (which could also be me) to figure out. If it is me, I'll deal with it because it keeps things simple.

> I've occasionally edited them by hand.

Yeah, usually when the quoting was f'up.

Re: Friends don't let friends export to CSV

#300

Post by JonChesterfield »

XML. Not CSV, not Parquet (whatever that is), not protobufs. Export the data as XML, with a schema. Not json or yaml either. You can render the XML into whatever format you want downstream.

The alternative path involves parsing csv in order to turn it into a different csv, turning json into yaml and so forth. Parsing "human readable" formats is terrible relative to parsing XML. Go with the unambiguous source format and turn it into whatever is needed in various locations as required.

Post reply on HN