Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

431–440 of 594 posts

Re: Time to retire the CSV?

#431

Post by anigbrowl »

zz865 wrote: post #422

As an aside, XML really was great though, its a pity it fell out of fashion. Schemas, XPath, so many great solutions that we're still messing with in JSON, YAML etc etc.

I really wanted to like XML but every book I picked up about it was so awful, and every XML browser program I tried so unbelievably slow, that I just gave up trying to figure it out when it was still current. I didn't need it for work; I just thought the concept was cool and it'd be a fun thing to know. Perhaps people were so eager to prove it was powerful that they ended up making it seem like something for masochists.

Re: Time to retire the CSV?

#432

Post by qwerty456127 »

Evidlo wrote: post #426
qwerty456127 wrote: post #354

Earlier quoted context omitted.

Almost nobody is a fan of the actual ISO 8601. It requires a big 'T' to separate the date from the time - people consider this ugly and rarely implement this in the wild. People mostly use RFC 3339 which allows using a space instead of the 'T'. Both also require colons to separate hours and minutes and this makes it impossible to use in file names if you want to support accessing them from Windows. I personally use t…

Are ISO standards ever updated? Is there any chance we might see an ISO 8601 date variant which is meant for filenames?

> Are ISO standards ever updated?

Yes, ISO standards are updated as regularly as it makes sense for standards to change (obviously they wouldn't really be standards if they were updated more often).

Wikipedia says ISO 8601 was first published in 1988, with updates in 1991, 2000, 2004, and 2019.

The C language is another example. The latest stable revision (C17) is ISO/IEC 9899:2018.

> Is there any chance we might see an ISO 8601 date variant which is meant for filenames?

Hardly.

Re: Time to retire the CSV?

#433

Post by romwell »

turtlebits wrote: post #119
Sanzig wrote: post #44

Earlier quoted context omitted.

As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. I definitely can't do that with .xlsx. There is almost zero friction to bolting CSV export capability to an existing system, which is part of why it's so popular.

You can write what "looks" like CSV to you, but there are no guarantees it will import correctly. The problem is 10x worse when you get CSV from one source and rely on another process to load it. I fought this problem for several days going from NetSuite to Snowflake via CSV.

> but there are no guarantees it will import correctly.

What do you mean, there are "no guarantees"? You are in charge! You know what data you're dumping, you can see if it imports well. You can tailor your use case.

That's not the same as getting a CSV from some dump, where you have limited (if any) control over the behavior.

Re: Time to retire the CSV?

#434

Post by romwell »

masklinn wrote: post #202
marcosdumay wrote: post #115

Earlier quoted context omitted.

As soon as you open it in Excel, it's garbage anyway, since it will replace date-like items with nonsense, drop number digits, convert anything it can, reencode monetary unities, and so on. If you don't open it in Excel, you can have as strict a parser as you want, just like any other format. But neither is going anywhere anyway.

> If you don't open it in Excel, you can have as strict a parser as you want, just like any other format. No, you can not. Because the CSV format is so fuzzy you can very easily parse incorrectly and end up with a valid parse full of garbage. Trivially: incorrect separator, file happens to not contain that separator at all, you end up with a single column. That's a completely valid file, and might even make sense for…

>It's like you people have never had to write ingestion pipelines for CSVs coming from randos.

That's because this is not what this thread is about.

The comment you're responding to is not about CSVs coming from "randos". It's for the case where that rando is you, so you can make sure the problems you mention don't happen on the generation side of CSVs.

Re: Time to retire the CSV?

#435

Post by jedimastert »

qwerty456127 wrote: post #112

> It's Time to Retire the CSV > This column obviously contains dates, but which dates? Most of the world It's time to retire local formats and always write YYYY-MM-DD (which is both the international and the Swedish standard, and the most convenient for parsing and sorting). > A third major piece of metadata missing from CSVs is information about the file’s character encoding. It's bloody the time to retire all the c…

I don't need a library to work with CSV in any language. That alone is a deal breaker is plenty of situations, no matter how widespread the format becomes.

Re: Time to retire the CSV?

#436

Post by shakna »

dtparr wrote: post #370
unnouinceput wrote: post #276

Earlier quoted context omitted.

Correction: the new xlsx is a zip file, the old xls format is true binary.

To be fair, xlsx came out with Office 2007, so it's not exactly 'new' anymore. Perhaps at this point it's reasonable for 'excel file' to mean the one that's been the default for 14 years?

The xlsx document is based on the OpenDocument format, which actually came out in '05, so it's a wee bit older than that.

Re: Time to retire the CSV?

#437

Post by jedimastert »

rendall wrote: post #397

The article's strongest criticism of CSV is that it's easy for someone to mangle it when manually editing. This is true. It's also true for every format. It was weakest when it implied there is no real standard. There is, and it's robust for representing data, even data that includes any combination of commas and double-quotes. The algorithm for creating well-formed CSV from data is straightforward and almost trivial…

I wonder why escape dquotes with another dquote instead of a backslash or something?

Re: Time to retire the CSV?

#438

Post by 08-15 »

7thaccount wrote: post #366
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

CSV is far from perfect, but it's nice that I can easily work with them without needing any libraries. All I need is file I/O and the ability to split strings. It doesn't get much simpler. I'll admit though that "import JSON" and then being able to essentially convert the entire file into a dictionary is nice if the data has more structure to it.

> All I need is file I/O and the ability to split strings.

...until there is a newline inside a field.

The moronic quoting mechanism of CSV is one half of the problem; people like you, who try to parse it by "just splitting strings" is the other half. The third half is that it's locale dependent and after 30+ years, people still don't use Unicode.

Re: Time to retire the CSV?

#439

Post by jedimastert »

AussieWog93 wrote: post #404
pwinnski wrote: post #215

This reads like a joke. If you think you can do better than CSV, let's see your proposal. Hint: you probably can't, and if you could, you probably couldn't get Excel to export it, so you still probably can't. "The status quo is bad, more recent popular formats aren't good enough either, but I don't actually have a specific proposal that's better than all of the above" is a lot faster to read than that article, and sa…

>If you think you can do better than CSV, let's see your proposal. I've got one! It's basically the same as regular CSV, but everything is UTF-8, the columns and lines are delineated by dedicated UTF-8 "delineator" codepoints (if they aren't defined in the spec, find reasonable surrogates and use them), and therefore nothing ever needs to be escaped. More human readable than regular CSV, less prone to error and just…

fun fact: ascii (and utf-8) has separators! There's a "file separator", "group separator", "record separator", and "unit separator".|

Re: Time to retire the CSV?

#440

Post by contravariant »

jedimastert wrote: post #437
rendall wrote: post #397

The article's strongest criticism of CSV is that it's easy for someone to mangle it when manually editing. This is true. It's also true for every format. It was weakest when it implied there is no real standard. There is, and it's robust for representing data, even data that includes any combination of commas and double-quotes. The algorithm for creating well-formed CSV from data is straightforward and almost trivial…

I wonder why escape dquotes with another dquote instead of a backslash or something?

Because then you'd have to escape backslashes.
Post reply on HN