I thought in-band signalling was generally considered harmful these days?
+++
Why isn’t there a decent file format for tabular data?
81–90 of 355 posts
Re: Why isn’t there a decent file format for tabular data?
#82I never understood why the ASCII separator characters aren't used more. It seems like we're one simple text editor feature away from having easy display and modification. Is there some historical reason for not doing that?
ASCII separators (mnemonics FS, GS, RS, US) are difficult for most users to type, and have no obvious/standardized visual representation.
My editor, and lots of editors now, put spaces instead of tabs when I press the tab key. No reason something similar couldn't happen in CSV modes/apps.
Re: Why isn’t there a decent file format for tabular data?
#83I thought in-band signalling was generally considered harmful these days? +++
You mean, everybody prefers FTP to HTTP?
Re: Why isn’t there a decent file format for tabular data?
#84Earlier quoted context omitted.
I think CSV is crappy because commas are so common in real data. For almost all scenarios I've had to work with, I'd have been perfectly happy with TSV where literal Tab was a disallowed character. No escaping histrionics required.
Not being able to include Tabs and Carriage Returns in your data can be a problem though.
The ideal would be a format that avoids characters that are common in data, but still allows a way to encode rare characters that are part of data.
I think that was the motivation for multi-character delimiters like or {{ }} in templating languages.
Re: Why isn’t there a decent file format for tabular data?
#85> No escaping. If you want to put \u001F or \u001E in your data – tough you > can’t. Use a different format. > It would be reasonably compact, efficient to parse and easy to manually edit > (Notepad++ shows the unit separator as a ‘US’ symbol). Is it me or it won't be human readable because of the lack of new lines?
I wonder if there is some hacky way around that. For example, an LF (line feed) directly after a RS character is not considered data (but will break the line when shown in a text editor).
Re: Why isn’t there a decent file format for tabular data?
#86I never understood why the ASCII separator characters aren't used more. It seems like we're one simple text editor feature away from having easy display and modification. Is there some historical reason for not doing that?
ASCII separators (mnemonics FS, GS, RS, US) are difficult for most users to type, and have no obvious/standardized visual representation.
Who is typing out CSVs anyway. You’d either use a spreadsheet tool or do it in code.
Re: Why isn’t there a decent file format for tabular data?
#87I mean, in business contexts at least, Excel is what we end up using nearly all the time because it's ubiquitous. This doesn't mean there aren't issues with it; obviously it's nonFree, and it's notorious for making assumptions about data (see recent coverage of scientists literally renaming a gene (I think?) because Excel kept assuming it was a date or something).
But Excel as a tool is on so many desktops that you can assume access, and so it gets used.
I have a strong sense that this may be the biggest reason why no other format has taken over. It's "good enough" that it steals oxygen from any competing idea.
Re: Why isn’t there a decent file format for tabular data?
#88Earlier quoted context omitted.
But usually it will be 1 character vs 4. So that adds a lot of space that doesn’t add much value. It’s also harder to read.
I would say invisible control characters would be even harder to read.
US and RS characters are visible in some editors, e.g. Notepad++.
Re: Why isn’t there a decent file format for tabular data?
#89I think it’s because csv is good enough. All the standards I’ve seen haven’t been worth the effort to implement. So since csv, with all its flaws, is good enough it crowds out other open standards. People complain about it, but it’s not really much of a challenge to use csv. I’d also prefer it over the crap (rdf, xml, even schemad json) proposed by people who value more structure. It’s easier for me to just make clea…
CSV is fine. If you care about edge cases, implement RFC 4180: https://www.rfc-archive.org/getrfc.php?rfc=4180 If you don't, then split each line on ",". Problem solved. If you find tab delimited easier to read (as I do), then check out the IANA spec for TSV files: https://www.iana.org/assignments/media-types/text/tab-separa... It's easier to parse than CSV. Unfortunately, you have to decide how to handle newlines an…
> If you don't, then split each line on ",". Problem solved.
And the millionth bad CSV parser is born.
Re: Why isn’t there a decent file format for tabular data?
#90Why not sqlite? You can read/write to a sqlite file in pretty much any language using decent bindings aside from maybe browser JS. I've worked with third party sqlite files several times and it usually isn't that bad. You can't just use any text editor to work with them, but the sqlite CLI is very easy to install and use. If you want a GUI, DB Browser seems to work very well.