Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSVs Are Kinda Bad. DSVs Are Kinda Good

matthodges.com

Post reply on HN
Print view View original

91–100 of 133 posts

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#91

Post by lazide »

vaylian wrote: post #58
lazide wrote: post #39

Earlier quoted context omitted.

How many get edited or inspected in notepad at some point in their life? Nearly all of them (for any given workflow).

It is nice that text editors are abundantly available and that they can be used for the task. But once the CSV columns get too wide and irregular, then you probably want to reach for a dedicated spreadsheet program, because it is otherwise too hard to figure out which column you are currently reading. There is still room between a text editor and a full-blown spreadsheet program. New DSV editors could emerge when the…

At the point someone is using a different format, they’ll likely pick something explicitly structured. Like everything from JSON, to Yaml, to Protobufs, or hell even XML.

DSV seems like worst of both worlds. Not really structured, AND also not really viewable/editable by lowest common denominator tooling.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#92

Post by mjevans »

usrbinbash wrote: post #61
snapcaster wrote: post #55

Earlier quoted context omitted.

No it isn't in the real world. It's very much your problem if you're the team consuming these files. Try to go tell the head of accounting they need to make all their data rfc4180 compliant see how that goes

> Try to go tell the head of accounting they need to make all their data rfc4180 compliant see how that goes Fun fact: I did. And not just for accounting systems, but all sorts of data ingestion pipelines. Did it work every time? No. Did it work in many cases? Yes. Is that better? Absolutely. Here is the thing: If I accept broken CSV, where do I stop? What's next? Next thing my webservice backends have to accept brok…

I draw the line at, the source keeps changing how it's broken.

If things are broken, but in a predictable, standard for that source way... uggh but at least it's their standard and if some tweak gets the common tools working for that one standard then everyone can move on and be happy.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#93

Post by ElevenLathe »

disgruntledphd2 wrote: post #87
ElevenLathe wrote: post #84

Earlier quoted context omitted.

I personally find that this happens (a lot) less often than with commas or quote characters.

That's fair, but it only takes one to mess up the rest of the file.

Agreed, that's why it's not good for production processes.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#94

Post by bigbuppo »

brunokim wrote: post #45

I've read a comment here some years ago of someone discovering ASCII field delimiters and excited to use them. They then discovered that those characters are only used in three places: the ASCII spec, their own code, and the data from the first client where he tried to use this solution. Any file format needs a well-specified escape strategy, because every file format is binary and may contain binary data. CSV is kin…

To me it's wild that the problem was solved back in the early 1960s (and really, well before that) but everyone just ignored it because of reasons and now we're stuck with a sub-optimal solution.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#95

Post by [deleted] »

bathtub365 wrote: post #56
throw0101d wrote: post #35

Earlier quoted context omitted.

>> […] Excel […] > Don't use broken tools. Tell that to your accounting and finance department and let us know how the message is received. > If you're forced to use Excel, then it's not really you in control, is it? In which case the up-thread's advice to "just produce strict RFC 4180-compliant CSV data" is worthless. "Just." We're stuck with whatever CSVs we get, so 'just' doing X is not an option.

The up-thread’s comment (emphasis mine): > And *if you are in control of producing data*, just produce strict RFC 4180-compliant CSV data The point of the comment was that you likely aren’t in control of producing the data, so the article’s recommendation of using an entirely different format is likely also invalid. I’m not sure what you are arguing against as you seem to actually agree with them.

[deleted]

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#96

Post by dsevil »

Original author writes: >>> "Quick aside: I first learned about these ASCII delimiters while working with .fec files. For whatever reason, the Federal Election Commission in the United States also decided that they needed to ditch the comma, but they landed on using ASCII character 28 which is supposed to be used as a file separator not a field saparator. I have no idea why they picked that one when 31 was right there. Anyway, the FEC also has a tool called fs2comma.exe that turns it back into a CSV format, and a couple of years I filed a FOIA request and got the source code."

I can only speculate on this but in Perl, for fake multimensional arrays à la `$foo{$x,$y,$z}`[^1], Perl uses ASCII character 28 (U+001C INFORMATION SEPARATOR FOUR) as its default subscript separator. Perl borrowed this feature from AWK, which uses the same character by default for the same purpose.

Based on Perl, I initally used that same character for that same purpose in a project or two. I cannot speculate on why Aho, Weinberger, and/or Kernighan chose that character. (On or before 1977.)

[^1]: Not to be confused with nested array (or hash) references in Perl, a truer form of multimensional arrays: `$foo->[$x]->{$y}->[$z]`

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#97

Post by [deleted] »

usrbinbash wrote: post #67
SigmundurM wrote: post #63

Earlier quoted context omitted.

> Don't accept broken CSV. If people keep ignoring standards, thats their problem. From the very memo you link to (RFC 4180): > Implementors should "be conservative in what you do, be liberal in what you accept from others" (RFC 793 [8]) when processing CSV files.

Oh, I am nothing but liberal when it comes to CSV: Clients get the liberty to either have their requests processed, or get a 400 BAD REQUEST And yes, I am aware that the standard says this. My counter question to that is: How much client-liberty do I have to accept? Where do I draw the line? How much is too much liberty? And the answer is: there is no answer. Wherever any system draws that line, it's an arbitrary dec…

[deleted]

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#98

Post by Kon-Peki »

disgruntledphd2 wrote: post #81
Kon-Peki wrote: post #53

Earlier quoted context omitted.

> Excel If you’re deep in the Excel world, chances are extremely high that you also have access to SSMS, which has a really, really good data import tool that makes short work of nasty CSV files. The output of this tool doesn’t even have to be SQL Server, it will use any ODBC driver you’ve got installed; you can send the data to Excel or even a new, properly formatted CSV. And if you want a repeatable package, there…

> If you’re deep in the Excel world, chances are extremely high that you also have access to SSMS, Can you provide more information about this? Like, I'd reinstall windows if this would actually work on messy excel/CSV data.

There is a screenshot here [1]. The trick is to choose the generic "import data" task rather than "import flat file". Then you have a wizard that lets you play around with the specifications, change things up, look at the errors, go back to tweak things, go forwards to try again, etc. The only improvement I can think of is if you could save what you did as a standalone SSIS package (or whatever).

[1] https://www.mssqltips.com/sqlservertutorial/9248/import-and-...

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#99

Post by fragmede »

teddyh wrote: post #6

The article talks about reading and parsing CSV data of unknown variants, but then skips to the solution being using a different format altogether. But you can only switch to a different format if you are producing data, not if you are reading it! And if you are in control of producing data, just produce strict RFC 4180-compliant CSV data and everybody will be able to read it just fine. There is no need to make your…

Sqlite isn't a standard per-se, but outputting an Sqlite db file, if you're writing the export code, is easy enough, and enough of a standard that I dare say you'd be doing fine to output .db files marked as sqpite.
Post reply on HN