Use gzip for compressing. If you want to stream, use following syntax:
[\n
{ ... },\n
{ ... },\n
{ ... },\n
...\n
With this simple trick you can stream easily..
Consider Using CSV
51–60 of 112 posts
Re: Consider Using CSV
#52Use gzip for compressing. If you want to stream, use following syntax: [\n { ... },\n { ... },\n { ... },\n ...\n ]\n With this simple trick you can stream easily..
in client-side you can parse bunch of items by just wrapping with [ and ]
Re: Consider Using CSV
#53Use gzip for compressing. If you want to stream, use following syntax: [\n { ... },\n { ... },\n { ... },\n ...\n ]\n With this simple trick you can stream easily..
in client-side you can parse bunch of items by just wrapping with [ and ]
and well, there's also BSON: https://github.com/mongodb/js-bson
Re: Consider Using CSV
#54Earlier quoted context omitted.
It appears, but its not. I have not found single program so far that conforms only to this RFC and nothing else. From the RFC itself: Status of This Memo This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited.
> I have not found single program so far that conforms only to this RFC and nothing else. Wouldn't that be impossible, given that parsers have to accept all kind of bizarro CSV flavors? Maybe more importantly, do you know of a single program or single CSV library that doesn't support reading or writing CSV as defined by the RFC?
Yeah, any of them. Just add new line in the "cell" and then go jump from the bridge.
Re: Consider Using CSV
#55> It's only 77 bytes, with 29 for the header and 24 for each line. At 100,000 entries, this list would be 2.4 MB (that's ~63% less than the JSON).
If size is really the issue but you still want schema enforcement protobuf is the way to go.
Re: Consider Using CSV
#56A constrained format based on JSONL with each record being a tuple of number/string/bool/null could better defined than CSV and looks almost like it. The benefit being, almost any json library could work with it, or could be made to one line at a time and it can be parallelized as newlines only exist as the delimiter.
["hello",5,false,1,2,2.334,null]
["world",12,true,1,2,2.334,null]
Re: Consider Using CSV
#57I usually prefer a binary encoding. More efficient on the wire, easier to parse and generate, and with no ambiguity. We have 2 control codes given to us by the teletype era that have the perfect meaning for this kind of data: 0x1E Record Separator 0x1F Unit Separator
and because its a single byte, its fast, no need to tokenize.
You also have 2 more the group separator and the file separator. so you could represent a tree with it.
Re: Consider Using CSV
#58Earlier quoted context omitted.
As Wikipedia puts it, "CSV is widely used to refer to a large family of formats that differ in many ways". If there's a canonical standard, it appears to be RFC4180: https://www.rfc-editor.org/rfc/rfc4180
It appears, but its not. I have not found single program so far that conforms only to this RFC and nothing else. From the RFC itself: Status of This Memo This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited.
An "Internet Standard" is just a designation that has been given to an RFC that has been blessed in a certain way. See https://www.rfc-editor.org/ for more details, but the set of designations is:
* Uncategorised
* Historic
* Experimental
* Informational
* Best Current Practice
* Proposed Standard
* Draft Standard
* Internet Standard
Being an "Internet Standard" is kinda special, but not especially so. For example, IMAP4, originally specified in RFC-3501 in March 2003, updated many times since, and revised in RFC-9051 in August 2021, is still a "Proposed Standard" without an STD designation, nearly 20 years and dozens of interoperable implementations later.
"Rough consensus and running code" is how things get done.
RFC-4180 is plenty good enough a "standard" for people to decide to interoperate over. They just have to decide to do so.
(Note also that HTML5 is not an "Internet Standard" according to the IETF et al. The last version to get an RFC was HTML 2 in RFC-1866, designated "Historic". And interoperability was an issue for a while with later versions of HTML during the "Best viewed in Internet Explorer/Netscape Navigator" wars. To get interoperability like we eventually did, you don't need an "Internet Standard"; you just need implementers who want to interoperate, and are willing to favour it over lock-in, and even over strict backwards-compatibility.)
(Also, the "and nothing else" clause in your comment confuses me. Why not support other formats/variants also? "Be liberal in what you accept" is certainly something that you probably want to avoid if you're designing a new format/protocol that no-one else is using yet, but if you're working with a decades-old format that was traditionally poorly-specified, with millions of documents out in the wild, it's probably the best way to allow existing users to move forward.)
Re: Consider Using CSV
#59Earlier quoted context omitted.
Well the obvious solution would be ASCII 0x1D (Group Separator)! Accept, no one actually uses those ASCII characters. Kind of bums me out that UNIX basically skipped out on them.
It's not a separator character, but at least vim and emacs acknowledge the page feed character. A pittance, I suppose.
> It's not a separator character,
Isn't it? I thought all the separator characters (0x1e, 0x1f, 0x1c) were specifically for delimiting records, fields and units.
What are they for?
Re: Consider Using CSV
#60> It's only 77 bytes, with 29 for the header and 24 for each line. At 100,000 entries, this list would be 2.4 MB (that's ~63% less than the JSON). If size is really the issue but you still want schema enforcement protobuf is the way to go.
protobuf is terrible! Now you have to rely on Google-scale tools to generate code for whatever language(s) you want to read or write the data in, and this becomes quite the encumbrance.