Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

321–330 of 711 posts

Re: A love letter to the CSV format

#321

Post by nukem222 »

derriz wrote: post #166
koolba wrote: post #63

Earlier quoted context omitted.

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

> It’s that libraries and tools to parse CSV tend to suck. Whereas JSON is the lingua franca of data. This isn't the case. An incredible amount of effort and ingenuity has gone into CSV parsing because of its ubiquity. Despite the lack of any sort of specification, it's easily the most widely supported data format in existence in terms of tools and language support.

> An incredible amount of effort and ingenuity has gone into CSV parsing because of its ubiquity.

Yea and it's still a partially-parseable shit show with guessed values. But we can and could have and should have done better by simply defining a format to use.

Re: A love letter to the CSV format

#322

Post by jaza »

Qem wrote: post #2

9. Excel hates CSV It clearly means CSV must be doing something right. This is one area where LibreOffice Calc shines in comparison to Excel. Importing CSVs is much more convenient.

This. I got burnt by the encoding and other issues with CSV in Excel back in the day, I've only used LibreOffice Calc (on Linux) for viewing / editing CSVs for many years now, it's almost always a trouble-free experience. Fortunately I don't deal much with CSVs that Excel-wielding non-devs also need to open these days - I assume that, for most folks, that's the source of most of their CSV woes.

Re: A love letter to the CSV format

#323

Post by nukem222 »

sundarurfriend wrote: post #88
andrepd wrote: post #24

Earlier quoted context omitted.

That would be solved by using the ASCII control chars Record Separator / Unit Separator! I don't get how this is not widely used as standard.

I remembered seeing a comment like this before, and... comment: https://news.ycombinator.com/item?id=26305052 comment: https://news.ycombinator.com/item?id=39679662 "ASCII Delimited Text – Not CSV or Tab Delimited Text" post [2014]: https://news.ycombinator.com/item?id=7474600 same post [2024]: https://news.ycombinator.com/item?id=42100499 comment: https://news.ycombinator.com/item?id=15440801 (...and many more.) "Th…

Excel needs to default its export to this. Unfortunately excel is proprietary software and therefore fucked.

Re: A love letter to the CSV format

#324

Post by bobmcnamara »

k_bx wrote: post #156

I've recently been developing a raspberry pi based solution which works with telemetry logs. First implementation used an SQLite database (with WAL log) – only to find it corrupted after just couple of days of extensive power on/off cycles. I've since started looking at parquet files – which turned out to not be friendly to append-only operations. I've ended up implementing writing events into ipc files which then pe…

There's definitely a place for it. I ran into the same problem with a battery powered event logger. Basically alternate between sleep-until-event and sample-until-event-over.

SQLite was fine until the realities of that environment hit.

0) I need to save the most data over time and my power budget is unpredictable due to environmentals. 1) When should I commit? SQLite commit per insert slows down, impacts battery life, impacts sample rate. Practically you could get away with batching all data for a small period. 2) SQLite is slow to repair databases. Partially written file would often take longer to repair than we had battery to run.

CSV based format filled that niche. First column was line-column count to support firmware upgrades. Last column is line-checksum. Another column indicating if this line was the last for an event. Parser skips corrupted lines/entries.

Re: A love letter to the CSV format

#327

Post by tubthumper8 »

zeroimpl wrote: post #293
rcbdev wrote: post #153

Earlier quoted context omitted.

Same for JSON though. What Python considers a valid JSON might not be that if you ask a Java library.

Example? I know there's some ambiguity over whether literals like false are valid JSON, but I can't think of anything else.

That _shouldn't_ be ambiguous, `false` is a valid JSON document according to specification, but not all parsers are compliant.

There's some interesting examples of ambiguities here: https://seriot.ch/projects/parsing_json.html

Re: A love letter to the CSV format

#328

Post by bobmcnamara »

hajile wrote: post #65

The argument against JSON isn't very compelling. Adding a name to every field as they do in their strawman example isn't necessary. Compare this CSV field1,field2,fieldN "value (0,0)","value (0,1)","value (0,n)" "value (1,0)","value (1,1)","value (1,n)" "value (2,0)","value (2,1)","value (2,n)" To the directly-equivalent JSON [["field1","field2","fieldN"], ["value (0,0)","value (0,1)","value (0,n)"], ["value (1,0)","…

> JSON parsers will not only output the expected values every time

Unless you need appendability, but then you should probably just use NDJSON/JSONL for a lot of cases.

Re: A love letter to the CSV format

#329

Post by metalliqaz »

achr2 wrote: post #326

Using ascii 'US' Unit Separator and 'RS' Record Separator characters would be a far better implementation of a CSV file.

and of course you can do that if you wish, as many CSV libraries allow arbitrary separators and escapes (though they usually default to the "excel compatible" format)

but at least in my case, I would not like to use those characters because they are cumbersome to work with in a text editor. I like very much to be able to type out CSV columns and rows quickly, when I need to.

Re: A love letter to the CSV format

#330

Post by nukem222 »

dylan604 wrote: post #239
FridgeSeal wrote: post #235

Earlier quoted context omitted.

CSV tooling has had to invest enormous amounts of effort to make a fragile, under-specified format half-useful. I would call it ubiquitous, I would call the tooling that we’ve built around it “impressive” but I would by no means call any of it “good”. I do not miss dealing with csv files in the slightest.

Microsoft Windows has had to invest enormous amounts... Apple macOS has had to invest enormous amounts... Pick your distro of Linux has had to invest enormous amounts... None of them a perfect and any number of valid complaints can be said about any of them. None of the complaints make any of the things useless. Everyone has workarounds. Hell, JSON has had to invest enormous amounts of effort...

[flagged]
Post reply on HN