Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

171–180 of 594 posts

Re: Time to retire the CSV?

#171

Post by ab111111111 »

Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

I'd add:

4) Changes to the replacement format should be human-readable in a diff

Re: Time to retire the CSV?

#173

Post by bluedino »

cogman10 wrote: post #140
Semaphor wrote: post #98

Earlier quoted context omitted.

Both are also not a good fit for columnar data at all.

I disagree. Json, in particular, can be nearly as compact as CSV by storing the data as an array of arrays. [[1,2,3], [4,5,6]] It's easy to make a structured data interchange format mimic and unstructured format. It's impossible to go the other way around without severe problems.

Except that won't open in Excel

Re: Time to retire the CSV?

#174

Post by crdrost »

CSV parsing is not as bad as it sounds and can be a decent fizzbuzzy interview question.

The real problem with CSV is the lack of vision of data formats in general. Inside your programming language you need to say that you want to read a CSV? You need to import a different library and change both the parse code and your consumption code to instead do JSON? You need to change everything to use futures in order to stream the results? Are you out of your damn mind?

So now that disk space is so cheap it would make sense for any file format to just begin with a single few-kilobytes line that defines the parser for the coming file. Could be sandboxed, WASM-y or something, could be made printable of course... Sure, it might not be possible to get the full SQLite library in there but you could at least make it free to switch between JSON and BSON and CSV without having to recompile the software or force someone to design modular file input systems. Somehow the only flexible containers that hold different data structures are video container formats that do not care about what codec you used. They "get it"— can the rest of us?

Re: Time to retire the CSV?

#176

Post by Twisell »

Macha wrote: post #103
Twisell wrote: post #53

Earlier quoted context omitted.

OP and you gave me an idea : "The only true successor of CSV should be forward/backward compatible with any existing CSV variant" If we manage to write a spec that meet this criteria we'll have a powerful standard with easy adoption.

The issue is CSV is not compatible with CSV. It's not possible to write a spec that covers all CSV files in the world. CSV means things that are mutually incompatible in the less common cases, and the only way to really parse them correctly is to know which variant generated it. But you can't even tag that variant in the file by your criteria, as existing CSV parsers won't understand it.

By this argument UTF8 can't exist. And yet here it is.

PS: I never said 100% forward/backward compatible with all variant at the same time and without any noticeable artifact. I meant compatible in a non blocking way.

Re: Time to retire the CSV?

#177

Post by qayxc »

CobaltFire wrote: post #165
qwerty456127 wrote: post #112

> It's Time to Retire the CSV > This column obviously contains dates, but which dates? Most of the world It's time to retire local formats and always write YYYY-MM-DD (which is both the international and the Swedish standard, and the most convenient for parsing and sorting). > A third major piece of metadata missing from CSVs is information about the file’s character encoding. It's bloody the time to retire all the c…

Honestly I constantly see dates argued about and people state various formats that are still confusing. 4-2-2 of any variety can be confused. Why not 2-3-4 or 4-3-2 (DD-MMM-YYYY or YYYY-MMM-DD)? I’ve never understood why that isn’t more widely used.

What would MMM look like? 02-005-2022?

Usually MMM refers to the 3-letter shorthand of the month, e.g. "APR" or "OCT", but I guess that's not what you meant because it couldn't be used internationally.

Re: Time to retire the CSV?

#178

Post by masklinn »

salamandersauce wrote: post #120
ARandomerDude wrote: post #108

Earlier quoted context omitted.

I'd add the constraint that it needs to be understandable by most people. A typical non-tech office worker can use CSV without any trouble. Sqlite has an extremely steep learning curve for someone unfamiliar with databases.

Exactly. Worst case a CSV can be edited and viewed or even created in MS Notepad or any other text editor which is pretty much guaranteed to be on a system. Something to deal with Sqlite? No such luck and the barrier is much higher.

> Worst case a CSV can be edited and viewed or even created in MS Notepad or any other text editor which is pretty much guaranteed to be on a system.

Good luck getting that garbage to be ingestible by anything. Odds are the system you're trying you shove it in will start by choking on the UTF8 BOM notepad insists on, then it will choke on the record separators, then on the incorrect escaping (or quoting).

> Something to deal with Sqlite? No such luck and the barrier is much higher.

SQLite is already present in most if not all environments (usually multiple times too, you have at least one instance of sqlite per browser on your machine, and probably a system one as well, and a few others), so it's really at most a matter of providing SQLite clients by default.

Re: Time to retire the CSV?

#179

Post by smhenderson »

radmuzom wrote: post #28

Every few years an article like this pops up. I find it tiring - because they are primarily from a software engineer's viewpoint who is probably trying to write a parser and needs to handle the edge cases. As a data scientist, I receive and process around 75GB of CSV every day - of course I don't process it manually. Our processes have been running a few years now and millions of dollars of revenue rides on it. I don…

That's pretty much how I feel as well. People that complain about all the variations in CSV "formats" out there aren't wrong but in practice, for me anyway, most of the files I work with in any given situation come from the same source or a small set of sources and use the same variant.

If/when I get a data source introduced into my workflow that differs from this variant I come up with a routine to normalize it, integrate that into my workflow and move on.

Post reply on HN