Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

381–390 of 594 posts

Re: Time to retire the CSV?

#381

Post by cdcarter »

jerf wrote: post #194
derefr wrote: post #74

Earlier quoted context omitted.

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

"Self-describing formats like JSON Lines are big... but when you compress them, they go back to being small." For CSV file replacements, I'd expect something like "one JSON array per line, all values must be JSON scalars". In that case, it's not much larger than a CSV, especially one using quotes already for string values. But this demonstrates the problem with JSON for CSV, I suppose. Is each line an object? Is it w…

JSON Lines is a specified format that answers those questions. https://jsonlines.org/ Seems like it qualifies to the level of authority you're requiring.

Re: Time to retire the CSV?

#382

Post by radus »

shadowwolf007 wrote: post #241
radmuzom wrote: post #28

Every few years an article like this pops up. I find it tiring - because they are primarily from a software engineer's viewpoint who is probably trying to write a parser and needs to handle the edge cases. As a data scientist, I receive and process around 75GB of CSV every day - of course I don't process it manually. Our processes have been running a few years now and millions of dollars of revenue rides on it. I don…

Yeah - I used to lead a department that would process somewhere around 10TB of CSV formatted data per day. The edge cases are a hassle but they don't become less of a hassle from a business perspective by switching to json or really any other format. We tried an experiment of using more json and eventually gave it up because it wasn't saving any time at a holistic level because the "data schema" conversations massive…

Would DuckDB (https://duckdb.org/) work as your file format with enforced column types?

Re: Time to retire the CSV?

#383

Post by [deleted] »

asdff wrote: post #237
gsich wrote: post #158

Earlier quoted context omitted.

If you interpret "CSV" as purely comma seperated values then maybe. But in my bubble "CSV" means textfiles that are separated by some separator. Be it tabs, spaces, commas, or any other ASCII character. Some are more usable then others, if you have commas in your data then use tabs. If you have tabs use Form Feed or Record Separator or vertical tabs ... and so on. Of course this is not always applicable, since you so…

Those other things have different names like TSV

[deleted]

Re: Time to retire the CSV?

#384

Post by nicoburns »

divbzero wrote: post #373
hermitcrab wrote: post #332

Earlier quoted context omitted.

>the fact that both of its separators (newlines and commas) can appear as-is inside column values, with a different meaning, if those column-values are quoted, means that there's no way to parallelize CSV processing, because there's no way to read-ahead and "chunk" a CSV purely lexically Yes, this is a major pain. It can be avoided by using Tab separated value (TSV) files, which don't use escaping. But then you can't…

ASCII has special delimiters 0x1E Record Separator and 0x1F Unit Separator to avoid conflicting with values, but they have never gained widespread adoption.

I worked on a project that used these in a proprietary file format. It was actually quite nice. There was no escaping required because they simply stripped those characters out of the actual content.

Re: Time to retire the CSV?

#385

Post by rendall »

the_only_law wrote: post #361
xtracto wrote: post #78

Earlier quoted context omitted.

> access through cat/grep/awk, and easily load into any programming language Until the CSV fields contain commas themselves. Even if fields are surrounded by "".

I recall one of my old Phones could dump SMS messages as CSV. I wanted to be build and interface that would let me search through my messages. Turns out when you have data that contains both commas and quotes things get screwey real quick. You could have quoted data like: 123,ABC,”,””,456 Where ,” is column data. I think that standard method is to double quote the field, but the dump sure wasn’t doing that for me, pl…

> 123,ABC,”,””,456

That is badly formatted CSV, assuming ” is a double-quote. Double-quote (") is how you escape commas, so the 3rd entry will make a parser barf, since there is no close quote.

It should be 123,ABC,"","",456

> Turns out when you have data that contains both commas and quotes things get screwey real quick.

Not really. It's pretty logical. If the entry has even one comma, put quotes around it. If it also has quotes, double each of them. Otherwise don't worry about it.

> what happens when the data is something like abc””,?

abc"", becomes "abc"""","

Re: Time to retire the CSV?

#386

Post by dheera »

7thaccount wrote: post #366
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

CSV is far from perfect, but it's nice that I can easily work with them without needing any libraries. All I need is file I/O and the ability to split strings. It doesn't get much simpler. I'll admit though that "import JSON" and then being able to essentially convert the entire file into a dictionary is nice if the data has more structure to it.

CSV is still easier to parse because the C++ dudes still refuse to implement some kind of nice operator-overloaded interface like

    #include 

    std::json myjson("{\"someArray\": [1,2,3,4,{\"a\": \"b\"}]}");
    std::cout
and the result is we have 50 different rogue JSON libraries instead of an STL solution. Until the STL folks wake up, boost::split can deal with the CSV.

Re: Time to retire the CSV?

#387

Post by mivade »

dheera wrote: post #386
7thaccount wrote: post #366

Earlier quoted context omitted.

CSV is far from perfect, but it's nice that I can easily work with them without needing any libraries. All I need is file I/O and the ability to split strings. It doesn't get much simpler. I'll admit though that "import JSON" and then being able to essentially convert the entire file into a dictionary is nice if the data has more structure to it.

CSV is still easier to parse because the C++ dudes still refuse to implement some kind of nice operator-overloaded interface like #include std::json myjson("{\"someArray\": [1,2,3,4,{\"a\": \"b\"}]}"); std::cout and the result is we have 50 different rogue JSON libraries instead of an STL solution. Until the STL folks wake up, boost::split can deal with the CSV.

https://github.com/nlohmann/json

Re: Time to retire the CSV?

#388

Post by da_chicken »

cpx86 wrote: post #349
da_chicken wrote: post #226

Earlier quoted context omitted.

> But this demonstrates the problem with JSON for CSV, I suppose. Is each line an object? How is that not a problem with every data serialization format? It does me no real good if I have an XML schema and a corresponding file. If I don't know what those elements and attributes represent I'm not really any better off. It's not like JSON or XML can meaningfully be marshaled back into objects for use generically withou…

> That's why Microsoft got away with proprietary date formats in System.Text.Json. What's proprietary in it? It follows ISO 8601-1:2019 and RFC 3339 according to the docs.

Sorry, that should be System.Runtime.Serialization.Json. System.Text.Json is the newer class that replaced it.

In .Net Framework 4.6 and earlier, the only built-in JSON serializer in the .Net Framework was System.Runtime.Serialization.Json.DataContractJsonSerializer.

You can still see it. If you're on Windows 10, run Windows Powershell v5.1 and run:

  Get-Item C:\Windows\System32\notepad.exe | Select-Object -Property Name, LastWriteTime | ConvertTo-Json
You'll see this output:

  {
    "Name":  "notepad.exe",
    "LastWriteTime":  "\/Date(1626957326200)\/"
  }
Microsoft didn't fix their weird JSON serialization until quite late. They may have back ported it to the .Net Framework, but they've deleted that documentation. Powershell v6 and v7 include the newer classes that are properly behaved. This is why Json.NET used to be so popular and ubiquitous for C# and ASP applications. It generated JSON like most web applications do, not the way Microsoft's wonky class did. Indeed, I believe it may be what System.Text.Json is based on.

Re: Time to retire the CSV?

#389

Post by hermitcrab »

divbzero wrote: post #373
hermitcrab wrote: post #332

Earlier quoted context omitted.

>the fact that both of its separators (newlines and commas) can appear as-is inside column values, with a different meaning, if those column-values are quoted, means that there's no way to parallelize CSV processing, because there's no way to read-ahead and "chunk" a CSV purely lexically Yes, this is a major pain. It can be avoided by using Tab separated value (TSV) files, which don't use escaping. But then you can't…

ASCII has special delimiters 0x1E Record Separator and 0x1F Unit Separator to avoid conflicting with values, but they have never gained widespread adoption.

While these have clear advantages over comma and CR for parsing (no more escaping!), they aren't at all convenient for manual editing.

Re: Time to retire the CSV?

#390

Post by proverbialbunny »

turtlebits wrote: post #109
radmuzom wrote: post #28

Every few years an article like this pops up. I find it tiring - because they are primarily from a software engineer's viewpoint who is probably trying to write a parser and needs to handle the edge cases. As a data scientist, I receive and process around 75GB of CSV every day - of course I don't process it manually. Our processes have been running a few years now and millions of dollars of revenue rides on it. I don…

It'll work until someone upstream upgrades their CSV writing library and your process breaks. But it's still writing CSV.

The same argument could be made for all other data formats.

CSV is like the C of data formats. It's incredibly stable yet simple enough you can make your own variant if you need to.

Post reply on HN