Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

251–260 of 594 posts

Re: Time to retire the CSV?

#251
post #239

Earlier quoted context omitted.

Nope, that's CSV without the drawbacks of CSV. That's CSV that can have special characters and doesn't suffer from delimiter problems. When someone says "Maybe we can fix CSV" this is what you should do instead of trying to "fix" CSV.

Interesting. This is JSON++ somehow. What should it be called? Line-oriented JSON? Row-JSON?

It's just JSON. It's not an extension to the spec, it's a subset of the spec. If anything, you could say it's JSON--.

Re: Time to retire the CSV?

#252

What are good alternatives to CSV that provide all of these features: - easy to parse - easy to edit with a generic text editor - easy to edit with a widely available GUI, like LibreOffice - allow adding more data with only append operations

I recall ESR's book about Unix conventions and culture mentioned something I think he called stanza-based. A file format where each line is a value and records separated by empty lines. Don't think Office tools can work with that though, except as regular text files.

Re: Time to retire the CSV?

#253
post #44

Earlier quoted context omitted.

As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. I definitely can't do that with .xlsx. There is almost zero friction to bolting CSV export capability to an existing system, which is part of why it's so popular.

You can write what "looks" like CSV to you, but there are no guarantees it will import correctly. The problem is 10x worse when you get CSV from one source and rely on another process to load it. I fought this problem for several days going from NetSuite to Snowflake via CSV.

This is true, but a lot of data processing takes place in a context where frictionless export functionality is more important than a 100% guarantee of import compatibility. I'd rather ingest city = ",CHANGSHA,HUNAN" (real example!) than ingest nothing at all because my vendor doesn't have time to integrate a JSON serializer.

Re: Time to retire the CSV?

#254
post #74

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

Commas, quotes, colons, newlines, and hyphens (JSON and YAML separators) can occur in content as well. How is that any different from separators occurring in csvs?

In practice, I used TSVs a lot, as tabs do not usually occur in most data. Alternatively, you could use pipes (|) or control characters as field or row separators.

Re: Time to retire the CSV?

#255
post #74

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

JSON/JSONL is not self-describing for many of the most common tabular data use cases. There's no date type or currency type.

If you are going to displace a standard, it has to be significantly better than the old.

Re: Time to retire the CSV?

#256

Lack of types is my biggest gripe with CSV. I think just being able to specify types in column headers would be a win. E.g. “column1:int,column2:string,column3:datetime”. Type inference has bit me too many times.

Maybe have a format based on postgres types? They are fairly comprehensive, well defined and already have a text representation.

Re: Time to retire the CSV?

#257

  U+FEFF"aaa","b CRLF
  bb","cc"c" CRLF
  zzz,yyy,xxx
Just an example of the wonderful world of CSV \(.)/ which I've used quite a lot, because it enables quick and easy (dirty) data dumps/ exchanges. But using it as a data exchange format between multiple parties often leads to problems, such as above.

Re: Time to retire the CSV?

#258
CSVJ open standard addresses many common problems of CSV. It is basically a CSV where rows are encoded as a JSON.

Unfortunately it is not widely adopted yet and the language support is yet to be improved.

Re: Time to retire the CSV?

#259

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

Parquet (and a few other competitors) have open-source implementations in many languages, are much faster to load than csv, are natively compressed, are strongly typed and don't require parsing...

There are few reasons to continue using csv in this day and age.

Re: Time to retire the CSV?

#260

Earlier quoted context omitted.

SQLite also requires external dependency and special execution environment and are not as portable as CSV.

SQLite is included in most operating systems by default. It's also on macOS and iOS. It also runs on all kinds of embedded devices in addition to personal computers and servers. https://en.wikipedia.org/wiki/SQLite#Operating_systems

[deleted]
Post reply on HN