11–20 of 77 posts
Earlier quoted context omitted.
Basically it's the same limitations as CSV. At least you could use something less likely to appear in data as record sepator (like 0x1E) Otherwise it's an interesting idea!
0x1E is the record separator, in ASCII precisely for this purpose. Too bad it’s not popular, here we’re stuck with inferior TSV/CSV
Earlier quoted context omitted.
Basically it's the same limitations as CSV. At least you could use something less likely to appear in data as record sepator (like 0x1E) Otherwise it's an interesting idea!
0x1E is the record separator, in ASCII precisely for this purpose. Too bad it’s not popular, here we’re stuck with inferior TSV/CSV
Earlier quoted context omitted.
yeah, this is a limitation from the TSV format this is based on - there is an extension to the format that supports storing binary blobs - ref: https://github.com/Hafthor/zsvutil?tab=readme-ov-file#nested...
can't you just do quoting?
> Any escaping or encoding of these characters would make the format less human-readable, harder to parse and could introduce ambiguity and consistency problems.
Found the wording of "could introduce ambiguity and consistency problems" a bit odd, but guess they mean that even if things are specified precisely (so there's no ambiguity) not everyone would follow the rules or something? And they want to play nice with other tools following the TSV "standard"
Earlier quoted context omitted.
yeah, this is a limitation from the TSV format this is based on - there is an extension to the format that supports storing binary blobs - ref: https://github.com/Hafthor/zsvutil?tab=readme-ov-file#nested...
Basically it's the same limitations as CSV. At least you could use something less likely to appear in data as record sepator (like 0x1E) Otherwise it's an interesting idea!
Most such formats support efficient querying by skipping the disk read step entirely when a chunk of data is not relevant to a query. This is done by splitting the data into segments of about 100K rows, and then calculating the min/max range for each column. That is stored separately in a header or small metadata file. This allows huge chunks of the data to be entirely skipped if it falls out of range of some query predicate.
PS: the same compression ratio advantages could be achieved by compressing columns stored as JSON arrays, but such a format could encode all Unicode characters and has a readily available decoder in all mainstream programming languages.
It is simple, but how do you access the price in row #1234567890? If your data doesn't have this many records and can fit into RAM, a basic NLJSON or CSV will work just as well.
It is simple, but how do you access the price in row #1234567890? If your data doesn't have this many records and can fit into RAM, a basic NLJSON or CSV will work just as well.
What is NLJSON?
It also wouldn't be that hard to make it seekable. All you would have to do is make each tsv file two columns: record-id, value.