Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

Post reply on HN
Print view View original

101–110 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#101

Post by flomo »

chiph wrote: post #18
Falling3 wrote: post #15

This reminds me of a depressing bug I run into frequently. I do a bit of work integrating with an inventory management program. Their main method of importing/exporting information is via CSV. The API also imports and exports via CSV, except whoever wrote the code that handle the imports decided not to use any sort of sensible library. Instead they use a built-in function that splits the string based on commas with a…

I deal with a vendor who occasionally sends us files without the double-quote character escaped. I feel your pain.

I used to work in an industry where different vendors passed around massive CSV files. If there was a way to abuse CSV, someone had done it, no two of them were exactly alike.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#102

Post by chiph »

wglb wrote: post #52
chiph wrote: post #21

Earlier quoted context omitted.

Trivia: Carriage return and Line feed are separate characters because they used to be separate operations for devices like Teletypes. Want double-spaced text? CRLFLF. Working with a slow device? CRCRCRLF to give the carriage time to return. Baudot4Life, yo.

Or DEL characters more likely.

The operators could have sent LTRS (all holes punched) but they never did -- their finger was already on CR, so they would just hit it a couple of times. Same net effect - delay until the carriage could return.

Which, BTW, was an indication that your machine needed service. The spring should have been wound tight enough and the track clean & oiled well enough to get the carriage back to the first column in time to not drop any characters. A pneumatic piston ("dash pot") slowed the carriage down as it approached the first column so it wouldn't crash into the stops and get damaged.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#103

Post by tbrownaw »

Istof wrote: post #66
dxbydt wrote: post #57

Don't do this. Tsv has won this race, closely followed by Csv. Anything else will cause untold grief for you and fellow data scientists and programmers. I say this as someone who routinely parses 20gb text files, mostly Tsv's and occasionally Csv's for a living. The solution you are proposing is definitely superior but isn't going to get adopted soon.

TLDR: it's superior, but don't do it...

It is strictly less expressive, because it can't handle nesting. This makes it inferior.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#106

Post by sigil »

scintill76 wrote: post #99
sigil wrote: post #78

Meh. What if some data has ASCII 28-31 in it? If you're not using a "real" escaping mechanism, and instead relying on the assumption that certain characters don't appear in your data, then I don't see anything wrong with using \t and \n (ie TSV). Either way, you know your data, and you're using whatever fits it best. If you need something that's never, ever going to break for lack of escaping, might I suggest doing p…

I think the answer is that those shouldn't occur within your data. If you're dealing with binary data, why are you using a text-based file format? If your data is textual, it shouldn't have control character delimiters within it, as they are reserved for that context. So, strip them out of your data if you have to. If you think they need to be preserved or escaped, IMO you're doing something wrong.

It's nice to be able to use the unix toolset (grep, cut, sort, join, etc) on all kinds of data, not just strictly "textual" data.

They're often my tool of last and only resort when dealing with very large datasets. Sure, you could wait for that dump of all of wikipedia to import into a nice indexed and queryable database, but why not start grepping it immediately? Maybe you want to sort by a key that's textual, but there's satellite data that's non-textual. sort(1) is a pretty amazing program in terms of resource usage; it parallelizes, it makes efficient use of available memory and disk when merge-sorting.

Anyway, there are plenty of examples!

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#107

Post by saalweachter »

__david__ wrote: post #86
stronglikedan wrote: post #53

Earlier quoted context omitted.

You are correct in that it does not solve a problem. Furthermore, the article tries to create a problem with CSV that does not exist. > CSV breaks depending on the implementation on Quotes, Commas and lines CSV does not break; the implementation is broken if it doesn't parse CSV properly. With a proper implementation, CSV solves every problem that will arise from this method.

The problem with CSV is that it looks so simple that nobody ever uses a real library to do it—they just roll their own. So you end up with a million implementations that are all buggy in various different ways. If you receive a CSV formatted file you can never be sure if it's actually good, valid CSV, or some invalid crap from that some programmer that reinvented the wheel because it was "so easy".

And as a side effect, if you are relying on lots of data files provided by other people, you inevitably end up with a library 57,000 parsers, 55,000 of which are for different, slightly broken CSV files.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#108

Post by commandar »

Eleutheria wrote: post #75

EDI is actually a wonderful and simple ASCII format for complex documents in use for over 30 years. The underlaying mapping formats for specific industries are a pain to parse but everything is easily formatted using stars or pipes as field separators ST|101 NAM|john|doe ADR|123 sunset blv|sunrise city|CA DAT|20140326|birthday Ah, the joy of simplicity.

HL7 used in healthcare is similar.

Just grabbing the first few segments from the example message in the wiki article:

  MSH|^~\&|MegaReg|XYZHospC|SuperOE|XYZImgCtr|20060529090131-0500||ADT^A01^ADT_A01|01052901|P|2.5
  EVN||200605290901||||200605290900
  PID|||56782445^^^UAReg^PI||KLEINSAMPLE^BARRY^Q^JR||19620910|M||2028-9^^HL70005^RA99113^^XYZ|260 GOODWIN CREST DRIVE^^BIRMINGHAM^AL^35209^^M~NICKELL’S PICKLES^10000 W 100TH AVE^BIRMINGHAM^AL^35200^^O
The delimiters are defined at the beginning of the opening MSH (message header) segment. HL7 is zero-indexed, but your zero index is always the segment label, so it's easy for non-technical people to count naturally to get the field identifier without having to explain counting n-1 to them.

The one exception to that is the MSH segment. Things get a little screwier there because the first instance of the field delimiter is also counted as a full field in the spec, so it tends to trip people up. So even though "^~\&" above looks like it should be MSH.1, it's actually MSH.2, etc.

The delimiters used in the wiki example are the most common you encounter, but some systems do things differently because reasons. The primary HIS at my hospital uses colons and semicolons, for example (and I want to poke out my eyes with ice picks every time I have to look at the messages coming from it as a result). But since it's all defined right in the message header, it's trivial to convert between delimiters when you need/want to.

Either way, this is how the vast majority of electronic medical records are transmitted today.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#109

Post by Crito »

markeganfuller wrote: post #100
Pxtl wrote: post #37

Earlier quoted context omitted.

> Precisely what makes them valuable is their difficulty to type. Again, if they'd caught on you'd imagine there would be some eventual convention in text-editors for what keybind would be used to enter them. Too bad the AltGr key (intended for entering rarely-used glyphs) doesn't appear on pure-English keyboards.

I'm looking at my English keyboard and it has Alt Gr...

Qwerty keyboards intended specifically for the US market (opposed to the UK for instance) frequently do not have AltGr keys. The thinkpad I'm using right now doesn't have one, and the Das Professional I have next to me doesn't have it either. Come to think of it, I'm not sure if I've ever owned a keyboard with an AltGr key..

To get around this, I have taken to using xmodmap to turn my right alt key into altgr.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#110

Post by pacaro »

csixty4 wrote: post #65

Pick databases have used record marks, attribute marks, value marks, sub-value marks, and sometimes sub-sub-value marks in ASCII 251-255 since the late 1960s. Like the control characters this blog post recommends, the biggest obstacle for Pick developers working on modern terminals is how on Earth to enter or display these characters. There's also the question of how to work with them in environments that strip out n…

Awesome! I was going to make a comment about Pick but you beat me to it. The challenge we had with Pick style involved customers using codepages that required these characters in text.

Encodings aside, the principle of having a hierarchy of delimiters can be hugely powerful

Post reply on HN