Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

Post reply on HN
Print view View original

251–260 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#251

Post by eggie »

radicaledward wrote: post #127
dxbydt wrote: post #57

Don't do this. Tsv has won this race, closely followed by Csv. Anything else will cause untold grief for you and fellow data scientists and programmers. I say this as someone who routinely parses 20gb text files, mostly Tsv's and occasionally Csv's for a living. The solution you are proposing is definitely superior but isn't going to get adopted soon.

I was surprised to see you list tsv as more common than csv. I encounter csv's on a pretty regular basis, but I don't think I've had to parse a tsv in the past 3 or 4 years. As a junior web developer, I don't have much experience though. 9 times out of 10, the csv is coming from or going to Excel, or a system that was designed to support Excel. If you don't mind my asking, what types of data do you regularly work wit…

There tends to be less overhead in TSV. Unless you want to represent text that has embedded tabs it seems unnecessary. It works with standard *nix tools. Not a bad compromise and part of the reason that people whose standard "file" is 100Gb prefer it.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#252

Post by ithkuil »

chinpokomon wrote: post #217
ithkuil wrote: post #192

Earlier quoted context omitted.

If only they had included an official escape character in ascii ... oh wait

That ESC is to allow more control characters to be defined than what was initially put into the standard. Consider for instance how ANSI colors are written to the terminal. They are escape sequence control (ESC? I'm not sure if the original designer meant to be meta) characters, identified by the escape sequence (ESC)[.

I'm pretty sure that's later development. Otherwise it would be called "prefix" or "extend".

Unfortunately I cannot quickly find a better source than:

"The "escape" character (ESC, code 27), for example, was intended originally to allow sending other control characters as literals instead of invoking their meaning."

http://en.wikipedia.org/wiki/ASCII

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#253

Post by seszett »

dagw wrote: post #231
awwaiid wrote: post #220

Earlier quoted context omitted.

I fixed my keyboard -- @,#,^,{,}.... no shift by default. Have to press shift to get 2,3,6,[,],9,0,... Must be annoying to have to press shift to get those symbol chars while you're coding, eh?

I think it's french keyboards that are like that by default. Symbols are the default and you have to press the Shift-key to get numbers.

Yeah, and some people complain that they have to use shift to get numbers.

I just think people will criticise their local layout no matter what. I use both a French AZERTY and a Québec QWERTY everyday (at work/home) and I think they are simply equally good for both typing French and for coding. The Québec keyboard (maybe actually Canadian multilingual or something) might have an edge because it more easily allows typing accented letters in uppercase, but on the other hand it doesn't let me type the € sign, so...

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#254

Post by EmielMols »

spoiler wrote: post #161
hnal943 wrote: post #118

Earlier quoted context omitted.

Living in the US, I've never heard of an AltGr key until this discussion.

I'm from Croatia, and we have the AltGr key. However, I discovered that Alt + Control = AltGr when I needed to use it at work[1], so it's simply a shortcut, I think. [1]: We have (I recently switched to an UK keyboard, because it suits me better) keyboards at work, because the Croatian (all slavic languages, to be honest) is horrendously counterproductive for programming. Google the layout, and you'll realise why. An…

Windows used to have a "Czech (Programmers)" keyboard layout. As I remember, it was one of the few layouts that _just worked_. First thing I would do on new installs is make it the only available layout.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#255

Post by XorNot »

Pitarou wrote: post #199

While we're on the subject, we should probably be using control code 16 (Data Link Escape) instead of the backslash character to escape strings. The problem is, of course, that we can't see it (no glyph) and we can't "touch" it (no key for it) so people won't use it. Ultimately, we're all still stick-wielding apes.

All this is making me realize that we also could have easily avoided having confusion about what's a command line argument separator and what's part of a file name if some of these were keyboard keys. "Field separators" to break up your command line, and regular spaces as just regular spaces? Hell yes please.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#256

Post by andrelaszlo »

Pxtl wrote: post #10

I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…

It's pretty easy using Emacs. Just type C-q C-_ or C-q C-^, alternatively C-x 8 RET [037|036] RET.

Emacs makes it really easy to insert arbitrary unicode as well, even if you don't remember the code point. C-x 8 RET s n o w m a n RET will insert a ☃ (snowman).

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#257

Post by csixty4 »

csixty4 wrote: post #174
baddox wrote: post #4

My guess is that TSV/CSV won out simply because anyone can easily type those characters from any standard keyboard on any platform.

TSV/CSV characters were also pretty much guaranteed to exist no matter what kind of terminal was used, and not cause any side-effects. No doubt some teletypes & dumb terminals used those FS, GS, RS etc. characters for special features since they weren't likely to appear in printed data. And I know those characters are used for other things in PETSCII and ASCII. 0x1C, the File Separator in ASCII, is used to turn text…

Sorry, that should say "PETSCII and ATASCII"

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#258

Post by eli »

chinpokomon wrote: post #212
eli wrote: post #35

Earlier quoted context omitted.

Depends on your platform. Even today a Unix/Linux line ending is almost always just LF. See: http://en.wikipedia.org/wiki/Newline

Unix = LF Mac = CR DOS = CRLF The DOS line endings are inherited from previous systems, and while they precisely convey carriage and paper movement of a printer it can be a pain to deal with today.

Only old Macs. They switched to POSIX-style LF with OS X.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#259

Post by jablan »

flux_w42 wrote: post #165
markeganfuller wrote: post #100

Earlier quoted context omitted.

I'm looking at my English keyboard and it has Alt Gr...

Here in Belgium we have to AltGr-the hell out of our keyboard during development. These characters can only be entered using the AltGr key on a Belgian Azerty keyboard (be-latin1): '|' '@' '#' '^' '{' '}' '[' ']' '\' and '~' Its not realy a problem, as long as you're used to it :-)

So most Belgian developers are not switching layouts when coding? I thought most of us non-US developers are "bi-lingual" when it comes to keyboard layouts.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#260

Post by srg0 »

jablan wrote: post #259
flux_w42 wrote: post #165

Earlier quoted context omitted.

Here in Belgium we have to AltGr-the hell out of our keyboard during development. These characters can only be entered using the AltGr key on a Belgian Azerty keyboard (be-latin1): '|' '@' '#' '^' '{' '}' '[' ']' '\' and '~' Its not realy a problem, as long as you're used to it :-)

So most Belgian developers are not switching layouts when coding? I thought most of us non-US developers are "bi-lingual" when it comes to keyboard layouts.

That's nothing. Standard Italian layout lacks tilde and backtick completely… and one needs three fingers to type “{” or “}”. I use only US layout for programming.

Actually, I switch between three layouts on my machine (the third one is for my mother tongue), and I also use AltGr for typographic characters.

Post reply on HN