Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

Post reply on HN
Print view View original

201–210 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#201

Post by oneeyedpigeon »

olalonde wrote: post #30
mikestew wrote: post #5

Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-charac…

Agreed. A few days ago, I learned that spaces are illegal between separators in CSV. For example, `"val1", "val2"` is illegal (it should be `"val1","val2"`). Kind of unintuitive given that in most languages, non-delimited spaces are insignificant.

I wonder if a 'whitespace separated values' format would make sense, assuming a) the separator is 'any run of whitespace except for a single space (ascii 32)' b) all whitespace in values is folded to a single space (which covers 99.99% of CSV usage). This would make parsing trivial, visual formatting (e.g. alignment) possible, and escaping separators a non-issue.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#202

Post by oneeyedpigeon »

rgarcia wrote: post #172

How about everyone just started following the CSV spec? https://tools.ietf.org/html/rfc4180 Doesn't allow for tab-delimited or any-character-delimited text and handles "Quotes, Commas, and Tab" characters in fields.

I love the way that, in the frickin' formal spec, the presence of a header row is ambiguous, so every tool that ever deals with CSV has to ask a human whether or not a header row is present. Great design decision, that.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#203

Post by WiseWeasel »

Eleutheria wrote: post #75

EDI is actually a wonderful and simple ASCII format for complex documents in use for over 30 years. The underlaying mapping formats for specific industries are a pain to parse but everything is easily formatted using stars or pipes as field separators ST|101 NAM|john|doe ADR|123 sunset blv|sunrise city|CA DAT|20140326|birthday Ah, the joy of simplicity.

It's like a PSV turned on its side, where the first column is 'header', and the following ones are 'value 1', 'value 2',...\n I guess that makes sense when you want to package non-list information, as long as there aren't any un-escaped newlines in the values.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#204

Post by mistercow »

JoshTriplett wrote: post #22

Leaving aside the pain of displaying and typing such characters... > Then you have a text file format that is trivial to write out and read in, with no restrictions on the text in fields or the need to try and escape characters. Phrases like that lead to lovely security bugs.

Yes, but simple filtering is a lot easier than escaping.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#205

Post by haddr »

oneeyedpigeon wrote: post #200
claudius wrote: post #140

Earlier quoted context omitted.

TSV is nicer for output (on stderr/out or a logfile), so tends to crop op if you want to parse the output/logfile of something. I haven’t seen Excel in use at my workplace yet.

Surely only if you're dealing with fixed-width fields.

ohhm this is pain in the ass format. srsly everything can happen there...

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#206

Post by MichaelGG »

mbreese wrote: post #51
Pxtl wrote: post #45

Earlier quoted context omitted.

I don't think CSV is a rigidly-defined format - I'm sure some implementations will happily accept spaces between the comma and the opening quote.

This is one reason why I prefer tab delimited files... The format is pretty simple with few edge cases. There really only one caveat to worry about - fields with tab characters. And that's extremely rare in my field. CSV on the other hand has a few different variations.

Can you explain how tab is any easier than comma? If you have to deal with escaping a character, then certainly it doesn't matter which character it is? For the general case, that is.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#207

Post by MichaelGG »

scintill76 wrote: post #99
sigil wrote: post #78

Meh. What if some data has ASCII 28-31 in it? If you're not using a "real" escaping mechanism, and instead relying on the assumption that certain characters don't appear in your data, then I don't see anything wrong with using \t and \n (ie TSV). Either way, you know your data, and you're using whatever fits it best. If you need something that's never, ever going to break for lack of escaping, might I suggest doing p…

I think the answer is that those shouldn't occur within your data. If you're dealing with binary data, why are you using a text-based file format? If your data is textual, it shouldn't have control character delimiters within it, as they are reserved for that context. So, strip them out of your data if you have to. If you think they need to be preserved or escaped, IMO you're doing something wrong.

What if your delimited file needs to contain other delimited files?

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#208

Post by chinpokomon »

spoiler wrote: post #161
hnal943 wrote: post #118

Earlier quoted context omitted.

Living in the US, I've never heard of an AltGr key until this discussion.

I'm from Croatia, and we have the AltGr key. However, I discovered that Alt + Control = AltGr when I needed to use it at work[1], so it's simply a shortcut, I think. [1]: We have (I recently switched to an UK keyboard, because it suits me better) keyboards at work, because the Croatian (all slavic languages, to be honest) is horrendously counterproductive for programming. Google the layout, and you'll realise why. An…

Perhaps, but I think it still goes against the original intent. Ctrl-~ or Ctrl-^ should give you a record separator (RS) and Ctrl-Del or Ctrl-_ should give you a unit separator (US). For the same reason Ctrl-m or Ctrl-M should give you carriage return (CR). This is because ASCII values from 00-1F are control characters and effectively grounded the most significant bits 7 and 6. Shift similarly would toggle or ground bit 6, depending on the implementation.

What happened was that the Ctrl key became synonymous with "command" after Teletype, so it became more about doing something. Think about Ctrl-x, Ctrl-c, and Ctrl-v as an example, but you still see some relics like Ctrl-d as End of Transmission (EOT) to close a shell or terminal. Alt is like a shift, but it is actually closer to the Fn key on most laptop keyboards. It was an alternative function of that particular key, so where the shift key provided you with an alternate case, Alt was more akin to an entirely different key... it isn't Alt plus an 'a' key, it is Alt-a.

AltGr was like another Alt key. It was originally there to allow you to enter an alternate glyph, especially line drawing characters available in extended ASCII, B0-DF. I thought it was a mapping closer to flipping the most significant bit to 1, but it doesn't exactly overlay the lower ASCII range, so that might be another change that evolved on the way to the modern keyboard.

To your original point, Microsoft Windows will now usually treat the chord Ctrl-Alt as AltGr. I don't know if that is with all layouts, or just those keyboards that lack AltGr. I find that most Linux distributions tend to follow Microsoft's lead and provide similar mappings but now they even repurposed the Win key as Meta or sometimes called Super. So it is likely that Ctrl-Alt is commonly the equivalent of AltGr.

For the propose of this discussion, I think it'd be better if Ctrl could be used to type these text separators, but the way modern operating systems map their modern keyboards, it might be difficult to ever reach consensus on how this should be done.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#209

Post by kibibu »

stingraycharles wrote: post #120
mbauman wrote: post #28

Earlier quoted context omitted.

It's a shame. The solution is in the charset, but tools never developed to use it so we don't use it. But I'd wager that if tools had historically supported them, then the situation would be no different than it is with tab. There are representational glyphs for tab, return, and others (⇥, ↵), and editors can show them in 'show whitespace' modes. There could be representational glyphs for these control characters, to…

Is this a problem that's solved with a "movement" ? I'm willing to contribute parsers and perhaps a text editor plugin if other people are willing to chip in. But indeed, this is a problem that's solvable by making the tools support it. Since most of the tools are OSS, this is mostly solvable.

I'd suggest that the most common tool used to generate and manipulate tabular data is Excel, which has terrible CSV export.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#210

Post by chinpokomon »

pjbrunet wrote: post #145
jrochkind1 wrote: post #138

Earlier quoted context omitted.

> Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. And this is in fact how they show up in vim (or at least my fairly uncustomized vim), using ^ to stand for ctrl as was once conventional: `^_` and `^^` The legacy MARC binary format still used for library data uses ascii 29, 30, and 31 -- although for reasons with probably some bizarre historical definition uses them D…

Back in the day with a numeric keypad you could type Alt-(number) to get any ASCII character, but not sure if that still works. In Firefox apparently Alt-1 takes you to the first tab, Alt-2 second tab...

On Windows yes, just proceed the number with a 0 to get Unicode. See the "How to enter Unicode characters in Microsoft Windows" page [1] for more details, including one of my favorite tricks, using Alt-X in WordPad. It is especially useful when you want to know what the code point is for a Unicode character.

[1] http://www.fileformat.info/tip/microsoft/enter_unicode.htm

Post reply on HN