Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

561–570 of 594 posts

Re: Time to retire the CSV?

#561
post #547

Earlier quoted context omitted.

As with pretty much everything else computing, the world suffers because Microsoft has been dumping terrible tools on it for decades, and people just take their garbage as the way things have to be.

Excel has its advantages, but it is funny the tools people choose to work with. The number of times Access would make more sense...

Didn't they sunset Access in lieu of Power BI Apps or whatever the hell they're calling their shot at no code these days?

Re: Time to retire the CSV?

#562
post #531

Earlier quoted context omitted.

I guess my point is that CSVs or any text files aren't really "human readable". You still need some application top view them.

In some incredibly pedantic sense, sure. But the difference is that every computer comes with what a human needs to read a CSV file. That's not true of an SQLite database.

Could it be?

Re: Time to retire the CSV?

#563
post #56

Earlier quoted context omitted.

Which format would you propose if I am writing a .NET tool to produce a file to be consumed by Excel?

I would expect .NET to have excellent .xlsx capabilities? Most libraries have an easy way to write those from tabular data.

Why would you expect that?

Re: Time to retire the CSV?

#564
post #349

Earlier quoted context omitted.

> That's why Microsoft got away with proprietary date formats in System.Text.Json. What's proprietary in it? It follows ISO 8601-1:2019 and RFC 3339 according to the docs.

Sorry, that should be System.Runtime.Serialization.Json. System.Text.Json is the newer class that replaced it. In .Net Framework 4.6 and earlier, the only built-in JSON serializer in the .Net Framework was System.Runtime.Serialization.Json.DataContractJsonSerializer. You can still see it. If you're on Windows 10, run Windows Powershell v5.1 and run: Get-Item C:\Windows\System32\notepad.exe | Select-Object -Property N…

Oh that one - yeah I've always steered clear of DataContractJsonSerializer. Never understood why they did it so weird.

To be fair, RFC 3339 wasn't even published back when this class was implemented (in .NET 3.5) so I guess they just went with whatever worked for their needs. ¯\_(ツ)_/¯

Re: Time to retire the CSV?

#565
post #88

Earlier quoted context omitted.

Just an FYI, JSON and XML both have characters that need to be escaped properly.

Correct, and those methods of escaping are clearly defined in both specs. In CSV, there is no universal escaping system. It's all over the board. Any variable length text data format is going to run into issues when special characters are used in the data. CSV has no definition of what should happen when that occurs.

The characters to escape are clearly defined in both specs, but that doesn't mean there's a universal escaping system. Just google "how to escape JSON" and you will see the wealth of users struggling to understand what and how to accomplish this. There will always be people struggling with escape characters, I don't see it being any harder or easier in any format.

Re: Time to retire the CSV?

#566

Earlier quoted context omitted.

Mainly Ctrl+F and diffs, in my case. Sometimes you need to look up a single code in a 50 MB file. And sometimes you need a quick check to see if one line or a million lines changes. It's "exception not the rule" type stuff... but it sure comes in handy to be able to check this stuff quick with basic text tools than have to run it through some binary parser. Same as JSON. But unlike protobufs for example.

Interesting. I would reach for `grep` and `diff` in those cases, so it's not much of a leap to imagine similarly using a tool for a binary format. After all, I only need the textual representation of the thing I'm searching for, not the entire dataset.

Sure, but those specific tools need to be written for each specific binary format.

And honestly they usually simply don't exist. In the vast majority of cases, in my experience.

Re: Time to retire the CSV?

#567

Earlier quoted context omitted.

Excel has its advantages, but it is funny the tools people choose to work with. The number of times Access would make more sense...

Didn't they sunset Access in lieu of Power BI Apps or whatever the hell they're calling their shot at no code these days?

https://www.microsoft.com/en-ww/microsoft-365/access

Re: Time to retire the CSV?

#568

Earlier quoted context omitted.

No, they really do. But 99% isn't 100%. Compare Python's csv parser with Go's. The former will prefer a parse for any input, and this is done in a fairly consistent way that at least gives one the opportunity to deal with "malformed" csv. The latter is a strict parser and coughs up an error upon almost any deviation from RFC 4180. I couldn't use the latter in practice because of it. It would just choke on too many cs…

> No, they really do. But 99% isn't 100%. Neither is 0. Python’s csv library does not handle any problem, it applies the separators / quoting / escapes it was given and that’s that. It gives the developer the opportunity to handle things, but if the developer isn’t aware nothing is handled.

If you're just going to quote me out of context and not bother responding to the actual substance of my comment (which was a high level but concrete comparison of two different CSV parsers, demonstrating an important qualitative difference with respect to csv parsers solving problems), then please, don't bother responding.

Re: Time to retire the CSV?

#569
post #334

Earlier quoted context omitted.

Many JSON Parsers already have an incremental forward read mode (think SAX-style parser if you are familiar with XML parser styles) where you can ask for each array inside that array of arrays one at a time as it reads them. If something unexpected happens at the end of the large outer array such as syntax error you can decide at that point if you rollback what you've already read/operated on or not. JSON.Parse() in…

Interesting. I don't know of any similar option in Python's json module, which is the one that I mostly use, but good to know this exists.

Yes, Python's bundled json module does not support that style of parsing. I saw several suggestions in a quick search to try NAYA [1] if you need that in Python (it's a no dependency Python 3 library) or one of the C-backed wrapper libraries NAYA mentions at the bottom of its Readme if you can afford native dependencies.

[1] https://github.com/danielyule/naya

Re: Time to retire the CSV?

#570
post #59

Earlier quoted context omitted.

Human can't efficiently write or parse XML or json, though. In some scenarios CSV hits the right spot to be accessible to human and computer, and the table can be laid out so that one can sort/grep/awk to quickly gain some insight.

As someone that works with maven and npm.... what? This is valid JSON [["bob", "jones", 1, 22], ["frank", "was", 32, 45]] That's unreadable and unparsable by a human? Not only is JSON often more parsable, because it's structured it also becomes a lot easier to query. I grep and awk xml and json stuff all the time. I also have the added bonus of being able to use `jq` for json content.

Late for the response but: JSON naturally encourages data to structural in ways that are often nested. In your example, if there's one or two additional layers of context (say, affiliation of the two persons), maintaining a CSV-like layout will be more cumbersome and unnecessary. I don't see many CSV or tabular-like JSON/XML in the wild and they really shouldn't be used that way.
Post reply on HN