Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

181–190 of 711 posts

Re: A love letter to the CSV format

#181

Post by sadcodemonkey »

Someone1234 wrote: post #107
fragmede wrote: post #54

Earlier quoted context omitted.

I want to push Sqlite as a data interchange format! it has the benefit of being well defined, and can store binary data, like images for product pictures inside the database. not a good idea if you're trying to serve users behind a web app, but as interchange, better than a zip file with filenames that have to be "relinked".

For context: I have a LOT of experience of interchange formats, like "full time job, every day, all day, hundreds of formats, for 20-years" experience. Based on that experience I have come to one key, but maybe, counter-intuitive truth about interchange formats: - Too much freedom is bad . Why? Generating interchange data is cheaper than consuming it, because the creator only needs to consider the stuff they want to…

I love the wisdom in this comment!

Re: A love letter to the CSV format

#182

Post by lgas »

koolba wrote: post #131
packetlost wrote: post #125

Earlier quoted context omitted.

Eh, it really isn't. The format does not lend itself to tabular data, instead the most natural way of representing data involves duplicating the keys N times for each record.

You can easily represent it as an array: [“foo”,”bar”,123] That’s as tabular as CSV but you now have optional types. You can even have lists of lists. Lists of objects. Lists of lists of objects…

Typing isn't optional in JSON, every value has a concrete type, always.

Re: A love letter to the CSV format

#183

Post by hajile »

derriz wrote: post #166
koolba wrote: post #63

Earlier quoted context omitted.

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

> It’s that libraries and tools to parse CSV tend to suck. Whereas JSON is the lingua franca of data. This isn't the case. An incredible amount of effort and ingenuity has gone into CSV parsing because of its ubiquity. Despite the lack of any sort of specification, it's easily the most widely supported data format in existence in terms of tools and language support.

Can you point me to a language with any significant number of users that does NOT have a JSON library?

I went looking at some of the more niche languages like Prolog, COBOL, RPG, APL, Eiffel, Maple, MATLAB, tcl, and a few others. All of these and more had JSON libraries (most had one baked into the standard library).

The exceptions I found (though I didn't look too far) were: Bash (use jq with it), J (an APL variant), Scratch (not exposed to users, but scratch code itself is encoded in JSON), and Forth (I could find implementations, but it's very hard to pin down forth dialects).

Re: A love letter to the CSV format

#184

Post by calfuris »

Suppafly wrote: post #97
TrackerFF wrote: post #41

Earlier quoted context omitted.

Have you tried using the "from text/csv" importer under the data tab? Where it will import your data into a table. Because that one will import ISO 8601 timestamps just fine.

This, it's dumb but Excel handles csv way better if you 'import' it vs just opening it. I use excel to quickly preview csv files, but never to edit them unless I'm OK only ever using it in Excel afterwards.

Even in that case I'd be hesitant to open a CSV file in excel. The problem is that it will automatically apply whatever transformation it thinks is appropriate the moment you open the file. Have a digit string that isn't semantically a number? Too bad, it's a number now, and we're gonna go ahead and round it. You didn't really need _all_ of the digits of that insurance policy number, did you?

They did finally add options to turn off the common offenders, but I have a deeply ingrained distrust at this point.

Re: A love letter to the CSV format

#185

Post by ndsipa_pomu »

orthoxerox wrote: post #174
andrepd wrote: post #24

Earlier quoted context omitted.

That would be solved by using the ASCII control chars Record Separator / Unit Separator! I don't get how this is not widely used as standard.

But what if one of your columns contains arbitrary binary data?

You'd likely need to uuencode it or similar as CSV isn't designed for binary data.

Re: A love letter to the CSV format

#186

Post by gthompson512 »

benwilber0 wrote: post #114
koolba wrote: post #63

Earlier quoted context omitted.

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

JSON is a textual encoding no different than CSV. It's just that people tend to use specialized tools for encoding and decoding it instead of like ",".join(row) and row.split(",") I have seen people try to build up JSON strings like that too, and then you have all the same problems. So there is no problem with CSV except that maybe it's too deceptively simple. We also see people trying to build things like URLs and q…

> It's just that people tend to use specialized tools for encoding and decoding it instead of like ",".join(row) and row.split(",")

You really super can't just split on commas for csv. You need to handle the string encodings since records can have commas occur in a string, and you need to handle quoting since you need to know when a string ends and that string may have internal quote characters. For either format unless you know your data super well you need to use a library.

Re: A love letter to the CSV format

#187

Post by otikik »

derriz wrote: post #166
koolba wrote: post #63

Earlier quoted context omitted.

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

> It’s that libraries and tools to parse CSV tend to suck. Whereas JSON is the lingua franca of data. This isn't the case. An incredible amount of effort and ingenuity has gone into CSV parsing because of its ubiquity. Despite the lack of any sort of specification, it's easily the most widely supported data format in existence in terms of tools and language support.

Meanwhile, Excel exports to CSV as “semicolon separated values” depending on your OS locale

Re: A love letter to the CSV format

#188

Post by freehorse »

simonw wrote: post #154
koolba wrote: post #131

Earlier quoted context omitted.

You can easily represent it as an array: [“foo”,”bar”,123] That’s as tabular as CSV but you now have optional types. You can even have lists of lists. Lists of objects. Lists of lists of objects…

Right - the JSON-newline equivalent of CSV can look like this: ["id", "species", "nickname"] [1, "Chicken", "Chunky cheesecakes"] [2, "Dog", "Wagging wonders"] [3, "Bunny", "Hopping heroes"] [4, "Bat", "Soaring shadows"]

This misses the point of standardization imo because it’s not possible to know a priori that the first line represents the variable names, that all the rows are supposed to have the same number of elements and in general that this is supposed to represent a table. An arbitrary parser or person wouldn’t know to guess since it's not standard or expected. Of course it would be parsed fine but the default result would be a kind of structure or multi-array rather than tabular.

Re: A love letter to the CSV format

#189

Post by meepmorp »

hajile wrote: post #169
packetlost wrote: post #151

Earlier quoted context omitted.

You're missing my point: basically nothing spits out data in that format because it's not ergonomic to do so. JSON is designed to represent object hierarchies, not tabular data.

CSV is lists of lists of fixed length. JSON is lists of lists of any length and groups of key/value pairs (basically lisp S-expressions with lots of unnecessary syntax). This makes it a superset of CSV's capabilities. JSON fundamentally IS made to represent tabular data, but it's made to represent key-value groups too. Why make it able to represent tabular data if that's not an intended use?

> CSV is lists of lists of fixed length.

I'd definitely put that in my list of falsehoods programmers believe about CSV files.

Re: A love letter to the CSV format

#190

Post by kentm »

derriz wrote: post #166
koolba wrote: post #63

Earlier quoted context omitted.

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

> It’s that libraries and tools to parse CSV tend to suck. Whereas JSON is the lingua franca of data. This isn't the case. An incredible amount of effort and ingenuity has gone into CSV parsing because of its ubiquity. Despite the lack of any sort of specification, it's easily the most widely supported data format in existence in terms of tools and language support.

I’ve found that the number of parsers that don’t handle multiline records is pretty high though.
Post reply on HN