Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

211–220 of 594 posts

Re: Time to retire the CSV?

#211

Post by Macha »

Twisell wrote: post #201
masklinn wrote: post #184

Earlier quoted context omitted.

What are you talking about? UTF8 is a single well-defined specification, and detecting that data is definitely not UTF8 is trivial.

And yet it is forward/backward compatible with ASCII and non blocking against all it's ill defined variants.

ASCII was well defined, CSV was not. Therefore they could take the highest bit, which they could know that was unused per the ASCII spec, and use that to encode their extra UTF-8 information.

Also UTF-8/ascii compatibility is unidirectional. A tool that understands ASCII is going to print nonsense when it encounters emoji or whatever in UTF-8. Even the idea that tools that only understand ASCII won't mangle UTF-8 is limited - sure dumb passthroughs are fine, but if it manipulates the text at all, then you're out of luck - what does it mean to uppercase the first byte of a flag emoji?

Re: Time to retire the CSV?

#212

Post by cbsmith »

tw04 wrote: post #198
cbsmith wrote: post #154

Earlier quoted context omitted.

Cognos is a BI tool, and the formats you are talking about are for rendering reports , not data. If you're picking data formats based on being able to load & edit them in to Excel, you're already doomed.

The entire article is about replacing CSVs for exchanging data exported from Excel... so why wouldn't he be picking data formats based on being able to load and edit them in Excel? If you're trying to solve this problem in a way that EXCLUDES Excel, you're already doomed. The business world will laugh at you and continue on their merry CSV way. >The biggest and most thorny problem to solve is the people problem: how…

> The entire article is about replacing CSVs for exchanging data exported from Excel...

No, it's not. It's about replacing CSVs for exchanging data. It mentions that CSVs often are the product of someone exporting data from a spreadsheet or doing a table dump, and how just doing that tends to create a ton of problems, but Excel is an example, not the subject matter of the article.

> The business world will laugh at you and continue on their merry CSV way.

The business world pays me a lot of money to teach them not to use CSVs.

Re: Time to retire the CSV?

#213

Post by brushfoot »

qayxc wrote: post #192
_moof wrote: post #127

This is an example of a genre I like to describe as: programmer objects to a solution that meets everyone's requirements because it doesn't make them feel like a beautiful code-poet. I like elegance as much as anyone. And I think it's a good proxy for other important qualities. But don't prioritize it above building something that actually does the job. Be an engineer.

Ah yes, and then there's https://www.theverge.com/2020/8/6/21355674/human-genes-renam... Not to mention the mess that is exchanging documents between different locales. It's all sunshine and roses until you get your CSVs from an office in a different country (which happens a lot in Europe). CSV gets the job done until it doesn't.

That's not a problem inherent to CSV the format, though, any more than using Word to edit JSON means smart quotes are a problem with JSON. It's about knowing or not knowing the tools you're using.

Unexpected behavior is a potential problem with any tool or format, certainly no less so with the kinds of solutions the article is proposing.

Re: Time to retire the CSV?

#214

Post by pessimizer »

zabzonk wrote: post #61

As the author of a CSV munging tool (CSVfix) I think most of the problems with CSV could be fixed if people producing CSV output, and people reading CSV input obeyed the rules of the RFC. Sadly, most people don't, and any textual output or input is routinely described as CSV, when it is nothing of the sort - even to the extent of not being comma-separated!

If you're referring to http://csvfix.byethost5.com/csvfix15/csvfix.html, thank you from the bottom of my heart. You were my pick for best tool for dealing with CSVs when a bunch of others (some of which sadly ended up with better adoption) were competing for the prize.

Re: Time to retire the CSV?

#215

Post by pwinnski »

This reads like a joke.

If you think you can do better than CSV, let's see your proposal. Hint: you probably can't, and if you could, you probably couldn't get Excel to export it, so you still probably can't.

"The status quo is bad, more recent popular formats aren't good enough either, but I don't actually have a specific proposal that's better than all of the above" is a lot faster to read than that article, and says the same thing.

Re: Time to retire the CSV?

#216

Post by ho_schi »

XML? Which is often not human readable, overly complex and therefore requires enormous amounts of processing power to be parsed.

JSON - and also CSV - are ubiquitous because they are actually human readable and usually quick to process. JSON is strictly defined it just doesn't over comments or structures, like XML. CSV is loosely defined but better than another competing standard ;)

A good data interchange format must be human readable, JSON and CSV are. Proprietary stuff not. And if you feel the need for speed? Seriously? Okay, then think about a binary format.

I prefer also the .conf format (so called INI) over other complex stuff for application settings.

Re: Time to retire the CSV?

#217

Post by da_chicken »

cbsmith wrote: post #204
da_chicken wrote: post #169

Earlier quoted context omitted.

> CSV is not a compact format That's only true if you're trying to send all your data in a single, monolithic CSV. If you're sending multiple CSVs, you're capable of representing data as well as a relational data store. Which is to say, you're representing your data using a system of data normalization specifically designed to minimalize data duplication. A single CSV represents a single table, and in most cases with…

> That's only true if you're trying to send all your data in a single, monolithic CSV. No, that's true in general. There are exceptional cases where it might not be true, but it's true in general. > Yes, you do have situations where you're storing losing data density due to using plain text strings, but that's not a limitation particularly unique to CSV for data serialization formats. That there are other inefficient…

> No, that's true in general. There are exceptional cases where it might not be true, but it's true in general.

Okay, put your money where your mouth is. Prove it.

Re: Time to retire the CSV?

#218

Post by cogman10 »

bluedino wrote: post #173
cogman10 wrote: post #140

Earlier quoted context omitted.

I disagree. Json, in particular, can be nearly as compact as CSV by storing the data as an array of arrays. [[1,2,3], [4,5,6]] It's easy to make a structured data interchange format mimic and unstructured format. It's impossible to go the other way around without severe problems.

Except that won't open in Excel

Correct. That'd be a moving goalpost. The original claim was that JSON doesn't work well with columnar data.

If the primary usecase is to take data give it to a non-programmer to evaluate it, then CSV isn't terrible. Add a transformer that's ran when someone manually requests to see the data.

However, for machine to machine communication, CSV should never be used.

Re: Time to retire the CSV?

#219

Post by pwinnski »

CobaltFire wrote: post #165
qwerty456127 wrote: post #112

> It's Time to Retire the CSV > This column obviously contains dates, but which dates? Most of the world It's time to retire local formats and always write YYYY-MM-DD (which is both the international and the Swedish standard, and the most convenient for parsing and sorting). > A third major piece of metadata missing from CSVs is information about the file’s character encoding. It's bloody the time to retire all the c…

Honestly I constantly see dates argued about and people state various formats that are still confusing. 4-2-2 of any variety can be confused. Why not 2-3-4 or 4-3-2 (DD-MMM-YYYY or YYYY-MMM-DD)? I’ve never understood why that isn’t more widely used.

YYYY-MM-DD with numeric months sorts in order. Nothing else does.

Re: Time to retire the CSV?

#220

Post by da_chicken »

masklinn wrote: post #164
da_chicken wrote: post #142

Earlier quoted context omitted.

No, SQLite's dynamic data types would silently coerce data just like opening a CSV directly with Excel does. The advantage of CSV is that it's as accurate as your plain text representation of your data can be. Since binary data can be represented by character data, that's 100% accurate. As soon as you introduce a storage format that has made assumptions about the type of data being stored, you've lost flexibility. SQ…

> No, SQLite's dynamic data types would silently coerce data just like opening a CSV directly with Excel does. SQLite's "dynamic data types" coerce data on input, not output. Once the data is in sqlite the way you wanted it, excel has no interpretation to perform, except insofar as really really wanting dates. > The advantage of CSV is that it's as accurate as your plain text representation of your data can be. Yeah…

> SQLite's "dynamic data types" coerce data on input, not output.

That's not relevant. If a data format coerces data when it's stored, it's still not a data format suitable for serialization.

Post reply on HN