I wrote my own converter a few years ago, then ended up needing it again last week. It’s one of those things you don’t always need but it’s handy to have when you do. https://github.com/baltimore-sun-data/csv2json
Show HN: Transform a CSV into a JSON and vice versa
21–30 of 111 posts
Re: Show HN: Transform a CSV into a JSON and vice versa
#22I recently heard about a tool called Miller that helps convert between JSON and CSV among other formats: https://github.com/johnkerl/miller
mlr --c2j cat documents.csv > documents.jsonl
Re: Show HN: Transform a CSV into a JSON and vice versa
#23Slightly OT: I've realized that CSVs are dramatically more information-dense than the equivalent JSON, and actually make a pretty reasonable API response format if your dataset is large and fits into the tabular shape. They can be a fraction of the size, mainly because keys aren't duplicated for every item.
Hell is other people's CSVs.
Re: Show HN: Transform a CSV into a JSON and vice versa
#24Slightly OT: I've realized that CSVs are dramatically more information-dense than the equivalent JSON, and actually make a pretty reasonable API response format if your dataset is large and fits into the tabular shape. They can be a fraction of the size, mainly because keys aren't duplicated for every item.
Shouldn’t really matter too much if the response is being compressed. If you’re rendering the table in the DOM, the response size is the least of your issues.
This is the real answer. All of the other answers are suggesting various changes to the JSON structure to eliminate key repetition, but this is irrelevant under compression.
Where it becomes relevant is if each record is stored as a separate document so you can't just compress them all together. Compressing each record separately won't eliminate the duplication, so you're better off with either a columnar format (like a typical database) or a schema-based format (like protobuf.)
Re: Show HN: Transform a CSV into a JSON and vice versa
#25CSV is more of a rumor than a standard, plus JSON can have a tree structure. It is a fun idea to think about and may be useful in some narrow cases, but will fail in almost all but those most trivial of structures.
> CSV is more of a rumor than a standard This reminds me of something my boss at a previous job would say: "I am morally opposed to CSV." Why? Because we worked at an NLP company, where we would frequently have tabular data featuring commas, which means if we used CSV we'd have a lot of overhead involving quoting all our CSV data. Instead my boss preferred TSV (T = tab) as our preferred tabular data format, which was…
PSV (Pipe) is also good, tabs can rarely show up in some sets of data like mail addresses if humans key them in. I usually go with one or the other if I have a choice.
Re: Show HN: Transform a CSV into a JSON and vice versa
#26CSV is more of a rumor than a standard, plus JSON can have a tree structure. It is a fun idea to think about and may be useful in some narrow cases, but will fail in almost all but those most trivial of structures.
> CSV is more of a rumor than a standard This reminds me of something my boss at a previous job would say: "I am morally opposed to CSV." Why? Because we worked at an NLP company, where we would frequently have tabular data featuring commas, which means if we used CSV we'd have a lot of overhead involving quoting all our CSV data. Instead my boss preferred TSV (T = tab) as our preferred tabular data format, which was…
I always try to use ascii char 31 (unit seperator) if I'm computationally generating csvs for shunting around data internally.
Re: Show HN: Transform a CSV into a JSON and vice versa
#27Earlier quoted context omitted.
It would be much nicer for the consumer to just de-dupe the keys in your json than to serve an annoying format like CSV. Your JSON could basically be a matrix with a header row, there's nothing forcing you to duplicate keys. { header: [...columnNames], rows: [...values2DArray]}
Make rows 1 dimensional. You don’t need the second dimension, it’s implied by header length. Once you do this, the JSON gzips down to about the same size as CSV, according to the last time I tested this IIRC.
Heck, you could do a single 1-D list (no object), and just give the header count as the first element, which would be even more compact.
Re: Show HN: Transform a CSV into a JSON and vice versa
#28Earlier quoted context omitted.
Shouldn’t really matter too much if the response is being compressed. If you’re rendering the table in the DOM, the response size is the least of your issues.
This is the real answer. All of the other answers are suggesting various changes to the JSON structure to eliminate key repetition, but this is irrelevant under compression. Where it becomes relevant is if each record is stored as a separate document so you can't just compress them all together. Compressing each record separately won't eliminate the duplication, so you're better off with either a columnar format (lik…
To parse it, you need to check the keys. If there can’t be other keys, you can just use an array which is stable on JSON and you can save on keys.
So you just have an array of arrays. Or even a huge array and every X elements, it’s a new record.
If each one has 2 keys,
[
{
key1: ‘a’,
key2: ‘b’
},
{
key1: ‘a’,
key2: ‘b’
}
]
[
[
‘a’,
‘b’
],
[
‘a’,
‘b’
]
]
[
‘a’,
‘b’,
‘a’,
‘b’
]Re: Show HN: Transform a CSV into a JSON and vice versa
#29Earlier quoted context omitted.
> CSV is more of a rumor than a standard This reminds me of something my boss at a previous job would say: "I am morally opposed to CSV." Why? Because we worked at an NLP company, where we would frequently have tabular data featuring commas, which means if we used CSV we'd have a lot of overhead involving quoting all our CSV data. Instead my boss preferred TSV (T = tab) as our preferred tabular data format, which was…
I always try to use ascii char 31 (unit seperator) if I'm computationally generating csvs for shunting around data internally.
it's crazy how our progenitors had the wisdom and foresight to reserve FOUR distinct delimiters for us 28-31 file/group/record/unit but webdevs are just nope we'll go with commas and crlfs and when that doesn't work we'll JSON
Re: Show HN: Transform a CSV into a JSON and vice versa
#30Built something similar on codepen quite a few years ago. Not sure where I came up with the format, seems a bit wild looking at the v. nice dot notation used here, but possibly more useful/efficient for variable data models, also takes into account data types:
https://codepen.io/theprojectsomething/pen/OwppWW
Note: click the Toggle Info to read the "spec" (groan) :)