Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

461–470 of 594 posts

Re: Time to retire the CSV?

#461

Post by politician »

astine wrote: post #365
woodrowbarlow wrote: post #301

Earlier quoted context omitted.

how're,you,handling,quotes?'

If you're manually generating your own CSV files, you probably know what kind of data you are generating and consequently whether your data is going to contain commas. If commas and newlines don't exist in your data, then you can safely ignore quoting rules when generating CSV files. I know that I've generated CSVs in the past and rather than figuring out the correct way to quote the strings, I just removed any incon…

You use the right ANSI characters - the record and field separators (30,31) - and avoid hacks like comma, pipe, and newline.

Re: Time to retire the CSV?

#462

Post by anodyne33 »

In e-Discovery our metadata is exchanged in csv format almost exclusively with one caveat, the defacto delimiters are Pilcrow and Thorn. We've solved the quote/comma in data problem, but as soon as we start mixing character sets, someone writes a load file in ANSI all hell breaks loose.

Re: Time to retire the CSV?

#464

Post by politician »

jedimastert wrote: post #439
AussieWog93 wrote: post #404

Earlier quoted context omitted.

>If you think you can do better than CSV, let's see your proposal. I've got one! It's basically the same as regular CSV, but everything is UTF-8, the columns and lines are delineated by dedicated UTF-8 "delineator" codepoints (if they aren't defined in the spec, find reasonable surrogates and use them), and therefore nothing ever needs to be escaped. More human readable than regular CSV, less prone to error and just…

fun fact: ascii (and utf-8) has separators! There's a "file separator", "group separator", "record separator", and "unit separator".|

This whole “problem” with CSV boils down to ignorance of the purpose of ANSI characters below 32. This has been a solved problem for decades.

Re: Time to retire the CSV?

#465

Post by meepmorp »

unnouinceput wrote: post #276
osigurdson wrote: post #99

Earlier quoted context omitted.

Take an Excel file and change the extension to .zip, then extract the contents. You will see that it is a collection of XML files. Therefore it should be reasonable to conclude that this approach can work for Excel sized datasets. However it is not particularly readable/diff-able if this is part of your use case.

Correction: the new xlsx is a zip file, the old xls format is true binary.

Do you mean the OLE-based format, or the really old one?

Re: Time to retire the CSV?

#466

Post by sigzero »

dragonwriter wrote: post #139
Twisell wrote: post #53

Earlier quoted context omitted.

OP and you gave me an idea : "The only true successor of CSV should be forward/backward compatible with any existing CSV variant" If we manage to write a spec that meet this criteria we'll have a powerful standard with easy adoption.

> If we manage to write a spec that meet this criteria we'll have a powerful standard with easy adoption. So, a binary format consisting of: (1) a text data segment (2) and end of file character (3) a second text data segment with structured metadata describing the layout of the first text data segment, which can be as simple (in terms of meaning; the structure should be more constrained for machine readability) as “…

No, not binary.

Re: Time to retire the CSV?

#467

Post by sigzero »

masklinn wrote: post #178
salamandersauce wrote: post #120

Earlier quoted context omitted.

Exactly. Worst case a CSV can be edited and viewed or even created in MS Notepad or any other text editor which is pretty much guaranteed to be on a system. Something to deal with Sqlite? No such luck and the barrier is much higher.

> Worst case a CSV can be edited and viewed or even created in MS Notepad or any other text editor which is pretty much guaranteed to be on a system. Good luck getting that garbage to be ingestible by anything. Odds are the system you're trying you shove it in will start by choking on the UTF8 BOM notepad insists on, then it will choke on the record separators, then on the incorrect escaping (or quoting). > Something…

SQLite is not the answer. At all.

Re: Time to retire the CSV?

#469

Post by theamk »

masklinn wrote: post #55
Sanzig wrote: post #44

Earlier quoted context omitted.

As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. I definitely can't do that with .xlsx. There is almost zero friction to bolting CSV export capability to an existing system, which is part of why it's so popular.

> As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. And usually generate garbage for anything but the most trivial case, which really nobody gives a shit about. That's the main reason why CSV absolutely sucks too, you have to waste month diagnosing the broken shit you're given to implement the workarounds necessary to deal with it. > I definitely can't do that with .xlsx…

I have a script which generates a CSV file using a bunch of print statements. The columns are hostnames and some numbers, so it is never going to contain commas or newlines. This will be perfectly valid CSV every time.

That’s why CSV is absolutely beautiful - there is a huge number of applications that people really care about, and their data is constrained enough that there is not need to care about CSV escaping and need for any third party libraries.

Creating XSLX file by hand is possible, but this will be a large amount of code and I wouldn’t include this in my script, it would need to be a separate library - which means build system support, learning the API etc...

Re: Time to retire the CSV?

#470

Post by RodgerTheGreat »

dheera wrote: post #386
7thaccount wrote: post #366

Earlier quoted context omitted.

CSV is far from perfect, but it's nice that I can easily work with them without needing any libraries. All I need is file I/O and the ability to split strings. It doesn't get much simpler. I'll admit though that "import JSON" and then being able to essentially convert the entire file into a dictionary is nice if the data has more structure to it.

CSV is still easier to parse because the C++ dudes still refuse to implement some kind of nice operator-overloaded interface like #include std::json myjson("{\"someArray\": [1,2,3,4,{\"a\": \"b\"}]}"); std::cout and the result is we have 50 different rogue JSON libraries instead of an STL solution. Until the STL folks wake up, boost::split can deal with the CSV.

A string split function is a poor choice if there's any possibility the CSV file contains quoted fields. Robust handling of both CSV and JSON requires a parser. In my experience, CSV can actually be trickier than JSON to parse because there are so many edge cases, alternatives, and ambiguities.
Post reply on HN