321–330 of 355 posts
The proposal is TSV but using ^_ instead of ^I (tab) to separate columns and ^^ instead of ^M (CR) or ^J (LF) to separate rows. So instead of being unable to put ^I and ^J in your table cells you are now unable to put ^_ and ^^ in your table cells. That sounds exactly as good, or bad, as TSV. So, okay? Sure, do that if you like, that sounds fine. Emacs TAGS files, Info files, and BABYL mailboxes https://quimby.gnus.o…
US and RS characters are much less likely to appear in textual data than Tab or LF. Also I believe this was the original purpose for US and RS in ASCII. So I'm not sure it is 'weird' to use them for that (although it does seem to be uncommon).
SQLite is one of the few file formats that's recommended by the US Library of Congress for archival storage: https://www.loc.gov/preservation/digital/formats/fdd/fdd0004... See also this page on the SQLite website (they're understandably very proud of this): https://www.sqlite.org/locrsf.html I think it's a fantastic format for archiving and distributing data.
All I'd want is a CLI tool that can dump an SQLite file so that it actually looks like a table (using "|", "-", "+", etc.), and maybe also accept grep-style filter specs (per specified column[s] of course).
Earlier quoted context omitted.
All I'd want is a CLI tool that can dump an SQLite file so that it actually looks like a table (using "|", "-", "+", etc.), and maybe also accept grep-style filter specs (per specified column[s] of course).
This sounds like an interesting weekend project. Are you saying you can't find anything for this?
I think it’s because csv is good enough. All the standards I’ve seen haven’t been worth the effort to implement. So since csv, with all its flaws, is good enough it crowds out other open standards. People complain about it, but it’s not really much of a challenge to use csv. I’d also prefer it over the crap (rdf, xml, even schemad json) proposed by people who value more structure. It’s easier for me to just make clea…
Excel supports a lot more tabular formats these days, but you have to search the ribbon for their importers. It's JSON support is actually surprisingly good, for instance.
Sometimes I think the best thing that the Excel team could do to make the state of the world of tabular formats for us better as developers is just to add a bunch of new file extensions that you can rename files to. For instance, if you could just rename a JSON file to end with something like .XLJSON and a Business Analyst or Accountant can just double click to open in Excel, that would open up a lot more nicer tabular format options beyond CSV with better developer ergonomics.
Earlier quoted context omitted.
I prefer tabs because my data can often have commas in it. Seeing tabs isn’t an issue as I also have invisible characters visible in all of my editors. But having your delimiter not be allowed in the record (as in \t is disallowed), makes parsing so much easier. CSV is a bear to parse because you have to read each value from a buffer to handle the quoting. line.strip().split(‘\t’) Is so handy. Batteries included are…
Given your example looks very much like python you could use the CSV module which is built into the standard library and handles all of this for you in a standards compliant manner. You really do want to use a library to parse CSV since there are a number of corner cases. For example, your example code may not read a whole row since rows can have newlines in them.
Right… you absolutely need to use a library. I’ve written the parsers for CSV which handle the edge cases (at least RFC edge cases). But I don’t want to include a library to be able to do it. Most of my scripts are small, and adding libraries makes them more difficult to move around. So, I shy away from CSV as a format as a result.
I’m okay with not allowing new lines and tabs in my fields. It’s a valid trade off for my data.
Yes, many of my scripts are Python, but not all, so adding new libraries for different languages is more difficult for me to remember as opposed to just splitting a line by a record separator.
Also, many of my files require having comments in the file’s header. Another reason why CSV can’t be easily used, as this isn’t part of the RFC.
Earlier quoted context omitted.
We’re talking about a tabular data file format. If you want to include arbitrary binary, use a binary data file. Or base64 encoded data. Most datasets you’d use data like this for are small enough to fit into memory, so let’s not get carried away. (I happen to use tab delimited files to store data that can’t fit into memory, but that’s okay too)
Yes. I think we're agreeing. I was responding to this "Any binary allowed between the quotes.". Binary data can't generally be dropped directly into a text format without some kind of organized encoding.
In retrospect, the idea of random binary data enclosed in quotes is what I’m mainly responding to — which I think we can all agree is a bad idea. (If you need to do that, encode it!)
Seems like the problem here is there is several high quality and well-developed formats, but the author and the commenters here dismiss them because of the different trade-offs they make. csv -- Simple for simple use cases, text-based, however many edge cases, feature lacking etc xlsx -- Works in excel, ubiquitous format with a standard, however complicated and missing scientific features sqlite -- Designed for relat…
As best I can tell no one has mentioned Recutils[0]? It is a little bizarre that csv has never really been nailed down, but yea it's all about trade-offs. [0] https://www.gnu.org/software/recutils/
[0]https://ronaldduncan.wordpress.com/2009/10/31/text-file-form...
Earlier quoted context omitted.
Gotta wonder why the format isn't just a column separator char, a row separator char, and then all the data guaranteed not to have those two chars. Then you could save the thing by finding any two chars that aren't used in the data. I guess this is why we have a zillion formats.
ASCII Code 29: Group Separator ASCII Code 30: Record Separator ASCII Code 31: Unit Separator https://theasciicode.com.ar/ascii-control-characters/record-...