Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

151–160 of 594 posts

Re: Time to retire the CSV?

#151

Post by megous »

masklinn wrote: post #70
pdonis wrote: post #65

Earlier quoted context omitted.

> An xlsx file is just a bunch of XML files in a zip. A bunch of XML files with opaque formats that MS constantly makes changes to to make its competitors have to keep chasing the format.

Even if it were true it wouldn't matter a whit to the production side of the format, which is what "produce CSVs using fprintf" is: excel can consume them all.

As someone who tried, Excel's handling of CSV files was the reason to abandon the idea and generate XLSX.

Libreoffice handles normal UTF-8 encoded, quoted value CSV files fine. Excel not so much.

Re: Time to retire the CSV?

#152

Post by dylan604 »

dragonwriter wrote: post #124
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

> The author is not forced to use csvs. The author appears to be a consultant selling data prep/transformation services. As long as the market is using CSVs, he’s forced to use CSVs, at least as end-of-pipeline inputs and outputs. Of course, “people optimize their workflows for something other than making my job easy” is a common, but also rarely persuasive in motivating action from others with different jobs, compla…

The consultant should be less concerned about what makes their life/job easier, but what makes their client's job easier by working with the consultant. If the client has to jump through hoops to serve consultant1 data in a format they are not used to, but consultant2 will accept the data they have already available or can be exported in a format familiar to them, the client will prefer consultant2.

Re: Time to retire the CSV?

#153

Post by masklinn »

slunk wrote: post #137
masklinn wrote: post #55

Earlier quoted context omitted.

> As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. And usually generate garbage for anything but the most trivial case, which really nobody gives a shit about. That's the main reason why CSV absolutely sucks too, you have to waste month diagnosing the broken shit you're given to implement the workarounds necessary to deal with it. > I definitely can't do that with .xlsx…

> That's the main reason why CSV absolutely sucks too [...] Is it? I think you're absolutely right that naive points of view like the one you're responding to will lead to avoidable bugs, but I'm not so sure the problem is CSV so much as people who assume CSV is simple enough to parse or generate without using a library.

> I'm not so sure the problem is CSV so much as people who assume CSV is simple enough to parse or generate without using a library.

The simplicity of CSV is what tells people that they can parse and generate it without a library, and even more so that that's a feature of CSV. You just had to read the comments disagreeing with me to see exactly that.

Re: Time to retire the CSV?

#154

Post by cbsmith »

Sunspark wrote: post #102
derefr wrote: post #74

Earlier quoted context omitted.

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

The formats you describe sound interesting, but in the big corporate/government world the barriers to adoption are high. For example, the software package Cognos which is owned by IBM and connects to database servers. To extract data from it, you have a choice of csv, Excel or pdf. The pdf is only used if you have done a perfect job creating a report in Cognos and don't need to edit/adjust it further in Excel. So rea…

Cognos is a BI tool, and the formats you are talking about are for rendering reports, not data. If you're picking data formats based on being able to load & edit them in to Excel, you're already doomed.

Re: Time to retire the CSV?

#155

Post by masklinn »

megous wrote: post #151
masklinn wrote: post #70

Earlier quoted context omitted.

Even if it were true it wouldn't matter a whit to the production side of the format, which is what "produce CSVs using fprintf" is: excel can consume them all.

As someone who tried, Excel's handling of CSV files was the reason to abandon the idea and generate XLSX. Libreoffice handles normal UTF-8 encoded, quoted value CSV files fine. Excel not so much.

Did you misread / misunderstand my comment somehow? Hint: my comment is not about generating CSV for excel to consume.

Re: Time to retire the CSV?

#156

Post by megous »

xtracto wrote: post #78
fighterpilot wrote: post #32

Earlier quoted context omitted.

I'm a human and I love CSVs. What other format can I open in a spreadsheet, access through cat/grep/awk, and easily load into any programming language? Any other format has to sacrifice one of these three things, and that's bad. It's this trifecta that makes it so versatile and human friendly.

> access through cat/grep/awk, and easily load into any programming language Until the CSV fields contain commas themselves. Even if fields are surrounded by "".

Then you just switch to a 3-line php script (fgetcsv() in a loop) and continue on... :)

Re: Time to retire the CSV?

#157

Post by masklinn »

turtlebits wrote: post #119
Sanzig wrote: post #44

Earlier quoted context omitted.

As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. I definitely can't do that with .xlsx. There is almost zero friction to bolting CSV export capability to an existing system, which is part of why it's so popular.

You can write what "looks" like CSV to you, but there are no guarantees it will import correctly. The problem is 10x worse when you get CSV from one source and rely on another process to load it. I fought this problem for several days going from NetSuite to Snowflake via CSV.

> I fought this problem for several days going from NetSuite to Snowflake via CSV.

Yeah if you see a CSV import feature without a billion knobs you know you're in for a world of hurt.

If you see a CSV import feature with a billion knobs, you're probably still in a world of hurt.

Re: Time to retire the CSV?

#158

Post by gsich »

derefr wrote: post #74
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

If you interpret "CSV" as purely comma seperated values then maybe. But in my bubble "CSV" means textfiles that are separated by some separator. Be it tabs, spaces, commas, or any other ASCII character. Some are more usable then others, if you have commas in your data then use tabs. If you have tabs use Form Feed or Record Separator or vertical tabs ... and so on.

Of course this is not always applicable, since you sometimes don't control the format you get your data.

Re: Time to retire the CSV?

#159

Post by osrec »

I can't think of another more efficient format for tabular data that's also human readable with a simple text editor... I'm personally not retiring the CSV any time soon!

Re: Time to retire the CSV?

#160

Post by dragonwriter »

dylan604 wrote: post #152
dragonwriter wrote: post #124

Earlier quoted context omitted.

> The author is not forced to use csvs. The author appears to be a consultant selling data prep/transformation services. As long as the market is using CSVs, he’s forced to use CSVs, at least as end-of-pipeline inputs and outputs. Of course, “people optimize their workflows for something other than making my job easy” is a common, but also rarely persuasive in motivating action from others with different jobs, compla…

The consultant should be less concerned about what makes their life/job easier, but what makes their client's job easier by working with the consultant. If the client has to jump through hoops to serve consultant1 data in a format they are not used to, but consultant2 will accept the data they have already available or can be exported in a format familiar to them, the client will prefer consultant2.

Sure, but if everyone is blocked from using the format consultant1 opposes because the industry wakes up one day and abandons it utterly, then consultant2 no longer has an advantage, which is why someone who would be consultant1 except that they don't want to lose business to consultant2 lobbies for it.
Post reply on HN