Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

571–580 of 594 posts

Re: Time to retire the CSV?

#571

Post by da_chicken »

cpx86 wrote: post #564
da_chicken wrote: post #388

Earlier quoted context omitted.

Sorry, that should be System.Runtime.Serialization.Json. System.Text.Json is the newer class that replaced it. In .Net Framework 4.6 and earlier, the only built-in JSON serializer in the .Net Framework was System.Runtime.Serialization.Json.DataContractJsonSerializer. You can still see it. If you're on Windows 10, run Windows Powershell v5.1 and run: Get-Item C:\Windows\System32\notepad.exe | Select-Object -Property N…

Oh that one - yeah I've always steered clear of DataContractJsonSerializer. Never understood why they did it so weird. To be fair, RFC 3339 wasn't even published back when this class was implemented (in .NET 3.5) so I guess they just went with whatever worked for their needs. ¯\_(ツ)_/¯

I'd be quicker to believe that it's because 2007 was still in the middle of Steve Ballmer's Microsoft, where embrace-extend-extinguish was their de jure practice.

Re: Time to retire the CSV?

#572

Post by 08-15 »

7thaccount wrote: post #508
08-15 wrote: post #438

Earlier quoted context omitted.

> All I need is file I/O and the ability to split strings. ...until there is a newline inside a field. The moronic quoting mechanism of CSV is one half of the problem; people like you, who try to parse it by "just splitting strings" is the other half. The third half is that it's locale dependent and after 30+ years, people still don't use Unicode.

You've assumed an awful lot about my use cases. The data I deal with in .CSV form is always pre-processed and doesn't have any of the minefield occurrences you've mentioned. There can't be a newline or anything like that in an input. In my decade of using .CSV files daily, I've only had one tertiary system where that is a problem. Also, when doing interactive work, it's a bit different than writing production IT soft…

So you're not actually parsing CSV and your comment was off-topic. Thank you for the clarification.

Re: Time to retire the CSV?

#573

Post by amyjess »

cbsmith wrote: post #502
amyjess wrote: post #423

Earlier quoted context omitted.

As someone who's used Avro [0], it's a pain because Avro records must be ordered, but JSON by definition is unordered. Avro's "JSON" format is more of a JSON-like format. At one point, when I was writing a script that would ingest an Avro record and then output a new one, I had to fiddle with things to make Python use an OrderedDict [1] so the new record would be output in the right order. [0] though the last time I…

I'm not sure what you mean by Avro records "must" be ordered. If you mean that the serialization format specifies the ordering of the fields, then yes, that is true, but that's an advantage in terms of compactness and processing efficiency ( https://avro.apache.org/docs/current/spec.html#order ). If you don't like it though, there are other formats like protobuf and thrift that have no such requirement, at the cost o…

The problem is that fields aren't ordered in the JSON spec, so fields being ordered in Avro's dialect of JSON automatically makes it non-standard JSON which makes it difficult to use standard tooling with Avro JSON.

Re: Time to retire the CSV?

#574

Post by fuhrysteve »

newbie2020 wrote: post #532
fuhrysteve wrote: post #71

Earlier quoted context omitted.

Yeah, this is pretty much it. The author complains about CSVs being "notoriously inconsistent" as though switching to some other format would magically change that. They're only inconsistent because sometimes lazy programmers do ",".join(mylist) instead of using an RFC4180 compliant CSV writer. Lazy programmers will just use non-compliant methods of creating whatever magic format OP is dreaming about. Case in point:…

Hah, I use the join function when I write them. Can you elaborate upon why that’s bad?

Doesn't require a whole lot of elaboration.

",".join(["a,b","c","d\ne","\"f\",g\""])

yields:

a,b,c,d

e,"f",g"

Try opening that in any csv reader.

Re: Time to retire the CSV?

#575

Post by pbreit »

I could not disagree more. Perhaps the author is writing about a small niche area where the data is more complicated and/or typing is useful.

I'd instead rather see much, much more CSV as well as flattening data models to make data more suitable for tabular representation.

Handling newlines and quotes is really not that difficult.

Re: Time to retire the CSV?

#577

Post by 7thaccount »

08-15 wrote: post #572
7thaccount wrote: post #508

Earlier quoted context omitted.

You've assumed an awful lot about my use cases. The data I deal with in .CSV form is always pre-processed and doesn't have any of the minefield occurrences you've mentioned. There can't be a newline or anything like that in an input. In my decade of using .CSV files daily, I've only had one tertiary system where that is a problem. Also, when doing interactive work, it's a bit different than writing production IT soft…

So you're not actually parsing CSV and your comment was off-topic. Thank you for the clarification.

Define parsing. I'm still going through GB of data in thousands of files and building complex reports, and data structures for scientific analysis. Just because I don't need thousands of lines of code to navigate edge cases doesn't mean I'm not parsing .CSV files.

Re: Time to retire the CSV?

#578

Post by Fwirt »

hermitcrab wrote: post #545
Fwirt wrote: post #542

Earlier quoted context omitted.

The other advantage not being discussed here is that once you have a csv parser, it can be trivially generalized to use any character as a delimiter, and you can usually find a character that isn't found in the data you're working with unless you're working with binary data. I work on a legacy system where passing delimited files around is usually the integration method of the day, and if you can't delimit with comma…

>it can be trivially generalized to use any character as a delimiter But then you have the additional problem of trying to work out what the delimiter is. You can take an educated guess, but this isn't going to be 100% reliable.

This comes back to "picking the right tool for the job". If you're ingesting csvs/tsvs, you're expecting a given format anyway. If you expect the wrong delimiter, the worst thing that happens is you detect that the file has the wrong number of columns and fail out. Even if you're using a more structured format, if you ingest a file containing the wrong dataset, your process will still fail because the correct fields aren't there. No matter what your format is, you can't recover from somebody sending you the wrong file.

Re: Time to retire the CSV?

#579

Post by davidinosauro »

stickfigure wrote: post #541
lmm wrote: post #520

Earlier quoted context omitted.

> If you want something better than CSV to take off, at the very least it needs to solve the common pain points of CSV. JSON doesn't - all it brings to the table is the ability to distinguish number vs text. That's a yawn. It brings the ability to parse in parallel - that's a big deal. And while number vs text might not be a huge difference in theory, in practice it eliminates what, 95% of real-world parse problems?

At least in my experience, I don't see a lot of trouble with number vs string in CSV data. You convert the string to a number or you don't. The pain points are usually dates or currencies - the same problem I usually have with JSON, because there's no standard format. I think you could fix most of the pain of CSV simply by adding a second header row which defines the type of the column, using a common vocabulary of t…

> At least in my experience, I don't see a lot of trouble with number vs string in CSV data.

I definitely had my fair share of trouble with locale-defined number format. Importing a column where a thousand is spelled "1.000", any integer between 1000 and 999999 would be wrongly parsed as a float between 1 and 999, while for any other number (like "0,1" for one tenth, "1.000.000" for a million, "1.000,56" for a thousand euros and change) the parser would give up and keep the string.

I usually have had more luck importing as text, then doing some string replacement of separators before finally converting to number.

Re: Time to retire the CSV?

#580

Post by lmm »

stickfigure wrote: post #541
lmm wrote: post #520

Earlier quoted context omitted.

> If you want something better than CSV to take off, at the very least it needs to solve the common pain points of CSV. JSON doesn't - all it brings to the table is the ability to distinguish number vs text. That's a yawn. It brings the ability to parse in parallel - that's a big deal. And while number vs text might not be a huge difference in theory, in practice it eliminates what, 95% of real-world parse problems?

At least in my experience, I don't see a lot of trouble with number vs string in CSV data. You convert the string to a number or you don't. The pain points are usually dates or currencies - the same problem I usually have with JSON, because there's no standard format. I think you could fix most of the pain of CSV simply by adding a second header row which defines the type of the column, using a common vocabulary of t…

> I don't see a lot of trouble with number vs string in CSV data. You convert the string to a number or you don't.

The trouble is that a lot of existing software (not just Excel) won't properly roundtrip text that looks like numbers and will e.g. strip the leading zero from phone numbers, or worse, change the last digit to make a number that exists in floating point.

> The pain points are usually dates or currencies - the same problem I usually have with JSON, because there's no standard format.

Hmm, I've never had a problem with ISO8861 dates - a string is either a valid date or not, it's very rare for someone to "accidentally" put data in ISO8861 format when it's not actually a date. Dates without timezones can cause problems, but that's more of a semantic issue than a serialization issue. What are the problems that you get?

I can see how currencies could be an issue with the lack of a standardised fixed-precision type. But in my experience they're an order of magnitude less common than issues with phone numbers, postal codes, and the like.

> I think you could fix most of the pain of CSV simply by adding a second header row which defines the type of the column, using a common vocabulary of types. TEXT, NUMBER, ISO8861-DATE, etc.

I'm sure you could. But at that point you're defining a new and incompatible format - you have to make it incompatible, or otherwise people will open these files with a tool that doesn't understand the header format and you're back to square 1 - so you'll pay all the same adoption costs as a completely new format. So it make sense to fix all the issues we can - and a format which can be split and parsed is definitely a major improvement for many use cases.

Post reply on HN