Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

421–430 of 594 posts

Re: Time to retire the CSV?

#421

Post by anigbrowl »

asdff wrote: post #284
turtlebits wrote: post #119

Earlier quoted context omitted.

You can write what "looks" like CSV to you, but there are no guarantees it will import correctly. The problem is 10x worse when you get CSV from one source and rely on another process to load it. I fought this problem for several days going from NetSuite to Snowflake via CSV.

Can you give an example? The rules for CSV files are so simple I'm struggling to imagine a case where something looks correct but in fact isn't correct.

Me, a naive idiot: CSV is simple I will write my own exporter because I am clever

Me, 20 minutes later: Heh that was easy I am a genius

Me, 21 minutes later: Unicode is ruining my life T_T

Don't get me wrong, I really like CSV because it's so primitive and works so well if you are disciplined about it. But it's easy to get something working on a small dataset and forget all the other possibilities only to faceplant as soon as you step outside your front door. In the case above my experience with dealing with CSV data from other people made me arrogant, when I should have just taken a few minutes to learn my way around a mature library.

Re: Time to retire the CSV?

#422

Post by zz865 »

As an aside, XML really was great though, its a pity it fell out of fashion. Schemas, XPath, so many great solutions that we're still messing with in JSON, YAML etc etc.

Re: Time to retire the CSV?

#423

Post by amyjess »

cbsmith wrote: post #138
derefr wrote: post #74

Earlier quoted context omitted.

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

> There's also Avro, which fails your point #1 (it's a binary format) but that binary format is a lossless alternate encoding of what's canonically a JSON document, and there are both simple CLI tools / and small, free, high-quality libraries that can map back and forth between the "raw" JSON document and the Avro-encoded file. At any time, you can decode the Avro-encoded file to text, to examine/modify it in a text…

As someone who's used Avro [0], it's a pain because Avro records must be ordered, but JSON by definition is unordered. Avro's "JSON" format is more of a JSON-like format. At one point, when I was writing a script that would ingest an Avro record and then output a new one, I had to fiddle with things to make Python use an OrderedDict [1] so the new record would be output in the right order.

[0] though the last time I touched it was in 2014, so things might've changed since

[1] as per [0], this was before Python dicts were ordered by default

Re: Time to retire the CSV?

#424

Post by lallysingh »

Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

Really, just use the right delimeters to start: https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text

Ascii #31 instead of commas, Ascii #30 instead of newlines. Now those characters can go into your values.

If that's no good, zstd-compressed proto.

Re: Time to retire the CSV?

#426

Post by Evidlo »

qwerty456127 wrote: post #354
titzer wrote: post #261

Earlier quoted context omitted.

I, too, am a fan of ISO 8601. In an insane world of date formats, it's the only sane choice. https://en.wikipedia.org/wiki/ISO_8601

Almost nobody is a fan of the actual ISO 8601. It requires a big 'T' to separate the date from the time - people consider this ugly and rarely implement this in the wild. People mostly use RFC 3339 which allows using a space instead of the 'T'. Both also require colons to separate hours and minutes and this makes it impossible to use in file names if you want to support accessing them from Windows. I personally use t…

Are ISO standards ever updated? Is there any chance we might see an ISO 8601 date variant which is meant for filenames?

Re: Time to retire the CSV?

#427

Post by derefr »

jerf wrote: post #328
turtles_ wrote: post #271

Earlier quoted context omitted.

Indeed, this has already been done: http://ndjson.org/ To be fair it's not an objectionable format. Using line breaks to separate objects makes it streamable, and you don't need to enclose the whole thing in an array to make it a valid JSON document.

That is not quite a CSV replacement. I use it for things with objects and stuff all the time. To be a CSV replacement you really need to add that each line must be a JSON array, and that it can only have scalars in it (no sub-arrays or objects). That would be a decent enough replacement for CSV itself. Not perfect, but the CSV "standard" is already a nightmare at the edge anyhow and honestly a lot of it can't be fixe…

> that it can only have scalars in it (no sub-arrays or objects)

I see CSV files that contain JSON arrays/objects in their fields all the time. Mainly from exporting Postgres tables that contain json/jsonb-typed columns. Are you saying that these aren't valid CSVs?

Re: Time to retire the CSV?

#428

Post by anigbrowl »

rutthenut wrote: post #63
habitue wrote: post #7

Earlier quoted context omitted.

Objectively, CSV is terrible for humans despite being a plaintext format. No one reads CSVs: they're incomprehensible since the columns are not aligned with the headings. (You might be drawing an analogy with JSON, which is often human readable because it puts the keys right there next to the values). The best that can be said for its simplicity is that it's easy to write code that can dump data out in CSV format (an…

Surprised you would think that JSON is more human-readable, as the layout of that makes it visually quite appalling - not so bad when through a pretty-printer utility, admittedly, but may as well go for XML as another use case

For ages I would just dump json files in a browser and let it auto-pretty print it for a first look.

A while back I discovered https://dadroit.com/ which is small (in feature set) but perfectly formed (nicely designed and extremely fast, even on huge files).

Re: Time to retire the CSV?

#429

Post by segmondy »

shock-value wrote: post #302
rjh29 wrote: post #225

There isn't enough love for TSV imo. Unlike commas, tabs rarely appear in input data, so you don't need complicated quoting or escaping rules. You just have tab and newline as special characters. Processing this data is extremely fast. Grepping for particular values is also very fast, as you can use the tabs as anchors when searching. TSV allows you to do stuff on a single machine and GNU parallel that people would n…

Strong agree. TSV is better than CSV always (unless you need to interface with an external system that doesn't accept it). For those not aware, TSV and CSV differ by more than just the delimiting character. TSV has a dead-simple specification: https://www.iana.org/assignments/media-types/text/tab-separa... . CSV does not have a standard spec and implementations differ quite a bit, but often in subtle ways.

There's no difference between TSV and CSV but the separator, most good libraries allow you to use any character you wish as the separator.

Here's the RFC for CSV - https://datatracker.ietf.org/doc/html/rfc4180

Re: Time to retire the CSV?

#430

Post by EthicalHacker03 »

Use fixed width files instead... along with a good old COBOL copybook. Nothing beats parsing fixed width records performance-wise. You can auto-generate staging and persistence tables, ETL jobs and quality control with just the copybook as input.
Post reply on HN