Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

291–300 of 594 posts

Re: Time to retire the CSV?

#291

Post by tsimionescu »

Macha wrote: post #211
Twisell wrote: post #201

Earlier quoted context omitted.

And yet it is forward/backward compatible with ASCII and non blocking against all it's ill defined variants.

ASCII was well defined, CSV was not. Therefore they could take the highest bit, which they could know that was unused per the ASCII spec, and use that to encode their extra UTF-8 information. Also UTF-8/ascii compatibility is unidirectional. A tool that understands ASCII is going to print nonsense when it encounters emoji or whatever in UTF-8. Even the idea that tools that only understand ASCII won't mangle UTF-8 is…

To be fair, there is basically no way to manipulate arbitrary text at all without mangling it, UTF-8-aware or not. What does it mean to take the first 7 characters of a UTF-8 string which might contain combinator characters and left-to-right special chars? What if the text uses special shaping chars, such as arranging hieroglyphs in cartouches? You basically need a text-rendering aware library to manipulate arbitrary strings.

Re: Time to retire the CSV?

#292

Post by postalrat »

unnouinceput wrote: post #273
karteum wrote: post #264

Earlier quoted context omitted.

I personally believe that at least SQLite matches all those criteria : "1) A truly open format is available" : sqlite is open-source, MIT-licensed, and well specified (even though I am usually not so happy with its weak typing approach, yet in this case this precisely enables a 100% correspondance between CSV and sqlite since CSV has also no typing at all...) "2) Applications have a speed increase from using csvs" :…

sqlite files are binary files, so a non-starter. If I open them in Notepad I can't read the values like I can with CSV.

"If I open them in Notepad"

Well there's your problem.

Re: Time to retire the CSV?

#293

Post by lmilcin »

I call this "criticizing something without understanding why it succeeded in the first place". Also "writing blog posts while blind to other users' use cases".

Do you really think there were no other "portable" formats to exchange data and SQLite is the first one?

The reason CSV is so popular has nothing to do with technical superiority which it obviously lacks.

The reason CSV is so popular is because it is dead simple and extremely easy to integrate within practically any conceivable workflow.

And because it is text format which you can trust, even if you have no other tools, you can inspect and edit in a text editor which is exactly the reason why other formats like INI, XML, JSON or YAML are so popular.

How would you:

a) Run shell command on a locked (can't install anything) PROD server to find the files and write a report of the file location, date of change and file size so that it can be easily processed by something else? Return the data in stream on the same SSH connection?

b) Quickly add a line to an existing startup script to get a report of when the script is starting and stopping so you can import data to excel. You just need it for couple of days, then you delete it.

c) Produce a report you can send to your coworker knowing they don't know how to program? And you don't want spending time explaining to them how to make use of your superior file format?

d) You talk to another team. You need to propose a format and specify a report with as little effort as possible. You know they are not advanced technically so you keep things simple?

e) You are solving an emergency and have to cross-check some data with another team. You need to produce a file that they will be able to read and process. You are stressed for time so you don't want to start a committee right now to find out shared technology. What would be safe choice without asking the other team for options?

f) A manager asked you for some data. You want to earn quick kudos. How will you send the data to get your kudos rather than irritating him?

These are all real life situations where CSV is being used.

As much as I understand technical arguments for SQLite, not every person is a developer. And all that technical superiority is worth nothing when they struggle to make use of the file you sent them.

I would say, the news of CSV's death are greatly exaggerated.

Re: Time to retire the CSV?

#294

Post by wombatpm »

lmilcin wrote: post #263
srcreigh wrote: post #116

Earlier quoted context omitted.

SQLite is included in most operating systems by default. It's also on macOS and iOS. It also runs on all kinds of embedded devices in addition to personal computers and servers. https://en.wikipedia.org/wiki/SQLite#Operating_systems

Just because most popular operating systems have libraries, doesn't yet mean it is portable. Can I click on it and open it in Excel? If not then it is not portable for me and for a lot of other people. The main reason I use CSV is to produce reports that I can either open myself or send to other people so that they can click on it and open themselves and immediately start hacking away. Excel is still corporate lingua…

Try using tabs. Still open in excel with less problems

Re: Time to retire the CSV?

#295

Post by lufte »

johnrob wrote: post #34

I find JSON “array of arrays” to be a better solution while also remaining human-readable. Especially when formatted as one array per line, analogous to CSV.

Parsing a CSV can be done in a line-per-line basis, while a an array of arrays in JSON is not valid until you reach the end.

How would any existing JSON parser handle 75GB of data in a single array of arrays?

Re: Time to retire the CSV?

#296

Post by karmakaze »

The premise is that CSVs are in widespread use and not suitable for their used purposes.

The conclusion is that there are many binary formats which are more suitable, except that none are used widely enough, and the post even goes on to not make a recommendation. Finally it says that we have to accept this sad state:

"Ultimately, even if it could muster the collective will, the industry doesn’t have to unify around a single successor format. We only need to move to some collection of formats that are built with machine-readability and clarity as first-order design principles. But how do we get there?"

Re: Time to retire the CSV?

#297

Post by WorldMaker »

bluedino wrote: post #173
cogman10 wrote: post #140

Earlier quoted context omitted.

I disagree. Json, in particular, can be nearly as compact as CSV by storing the data as an array of arrays. [[1,2,3], [4,5,6]] It's easy to make a structured data interchange format mimic and unstructured format. It's impossible to go the other way around without severe problems.

Except that won't open in Excel

Excel actually has really good JSON import… it's just "hidden" in the Data tab on the ribbon and users don't want to learn how to use it.

I feel so much of what keeps CSV in use is that it's a format that you can relatively easily generate without pulling in a lot of library dependencies and relatively quickly generate something that users can just "double click it opens in Excel".

What the Excel team could really gift to developer humanity at this point is some dumb file extension Excel registers like XLJSON that you could just rename a JSON file to and Excel opens it like a spreadsheet on double click.

Re: Time to retire the CSV?

#298

Post by MrPowers »

Here are the main advantages of Parquet over CSV (post author made these points more abstractly):

* Parquet stores schema in the metadata, so schema inference isn't required (schema inference is expensive for big datasets)

* Parquet files are columnar so individual columns can be grabbed for analyses (Spark does this automatically). This is a huge performance improvement.

* Row groups contain min/max info for each column, which allows for predicate pushdown filtering data skipping. Highly recommend playing with PyArrow + Parquet to see the metadata that's available.

* Columnar file formats are easier to compress. Binary files are way smaller than text files like CSV even without compression.

I wrote a blog post that shows how a Parquet query that leverages column pruning and predicate pushdown filtering can be 85x faster than an equivalent CSV query: https://coiled.io/parquet-column-pruning-predicate-pushdown/

CSVs are great for small datasets when human readability is an important feature. They're also great when you need to mutate the file. Parquet files are immutable.

It's easy to convert CSVs => Parquet/Delta/Avro with Pandas/Dask/Spark.

The world is already shifting to different file formats. We just need to show folks how Parquet is easy to use and will greatly increase their analysis speeds & they'll be happy to start using it.

Small nit: the article implies Apache Arrow is a file format. It's a memory format.

Re: Time to retire the CSV?

#299

Post by shock-value »

alexpw wrote: post #231

1) CSV can often be human readable at a glance (more so than most other formats, depending on that data), and that makes it appear deceptively simple and compact. Possibly due to that, I'd bet most of us have been bit by a writer/reader that doesn't respect the RFC rules. 2) I ask for TSV, whenever convenient. It's been more reliable, and I don't have a comprehensive why, but I think it's slightly more resilient to w…

TSV is a dead simple and fully-specified format. Here it is: https://www.iana.org/assignments/media-types/text/tab-separa...

CSV is a mish-mash of different, complicated, and under-specified standards and/or implementations.

Re: Time to retire the CSV?

#300

Post by Tagbert »

da_chicken wrote: post #142
cls59 wrote: post #104

Earlier quoted context omitted.

To point 1, I'd argue that a SQLite database is a great next step beyond CSV despite being a binary format.

No, SQLite's dynamic data types would silently coerce data just like opening a CSV directly with Excel does. The advantage of CSV is that it's as accurate as your plain text representation of your data can be. Since binary data can be represented by character data, that's 100% accurate. As soon as you introduce a storage format that has made assumptions about the type of data being stored, you've lost flexibility. SQ…

Excel's default insistence on trying to coerce the data into what it expects the data to be is the source of many of the problems of people using CSV. Excel will even ignore quotes around values if it thinks that the value inside might be a number. I often work with CSV files that contain 17 digit IDs that use numeric characters "01233998742021925" which Excel will convert lossily into scientific notation rather than leaving it as a string. There are ways to override that but they are tedious and people don't do it by default.

I'd be satisfied if Excel was just a little less aggressive about it's data coercion on import.

Post reply on HN