Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Transform a CSV into a JSON and vice versa

jsonmatic.com

Post reply on HN
Print view View original

11–20 of 111 posts

Re: Show HN: Transform a CSV into a JSON and vice versa

#11

Post by anonytrary »

brundolf wrote: post #2

Slightly OT: I've realized that CSVs are dramatically more information-dense than the equivalent JSON, and actually make a pretty reasonable API response format if your dataset is large and fits into the tabular shape. They can be a fraction of the size, mainly because keys aren't duplicated for every item.

It would be much nicer for the consumer to just de-dupe the keys in your json than to serve an annoying format like CSV. Your JSON could basically be a matrix with a header row, there's nothing forcing you to duplicate keys.

  { header: [...columnNames], rows: [...values2DArray]}

Re: Show HN: Transform a CSV into a JSON and vice versa

#12

Post by quantumofalpha »

brundolf wrote: post #2

Slightly OT: I've realized that CSVs are dramatically more information-dense than the equivalent JSON, and actually make a pretty reasonable API response format if your dataset is large and fits into the tabular shape. They can be a fraction of the size, mainly because keys aren't duplicated for every item.

If you'd use a column-oriented format like {"col1":["a","b","c",...],"col2":[1,2,3...],...}, it's about the same density, no?

Re: Show HN: Transform a CSV into a JSON and vice versa

#13

Post by amichal »

OJFord wrote: post #5
brundolf wrote: post #2

Slightly OT: I've realized that CSVs are dramatically more information-dense than the equivalent JSON, and actually make a pretty reasonable API response format if your dataset is large and fits into the tabular shape. They can be a fraction of the size, mainly because keys aren't duplicated for every item.

Yeah, lists of objects are pretty crap, because they're almost always homogeneous but unenforcédly so; it's not just a size issue but a parsing (or not - usage) issue too. You could approximate CSV in a JSON response like: { "columns": ["a", ..., "z"], "rows": [[1, ..., 26], ..., [11, 2266]] } Or: { "a": [1, ..., 11], ... "z": [26, ..., 2266] } which I've never seen, but would save space, and sort of enforced in the…

i've done the first when serializing an arbitrary table of trace data in a jsonb column. Did it to make it compact and then realized that if all i wanted to do was show it in a UI it was much easier to parse as a html table, and only marginally harder to present an array of objects.

Re: Show HN: Transform a CSV into a JSON and vice versa

#16

Post by inopinatus »

OJFord wrote: post #5
brundolf wrote: post #2

Slightly OT: I've realized that CSVs are dramatically more information-dense than the equivalent JSON, and actually make a pretty reasonable API response format if your dataset is large and fits into the tabular shape. They can be a fraction of the size, mainly because keys aren't duplicated for every item.

Yeah, lists of objects are pretty crap, because they're almost always homogeneous but unenforcédly so; it's not just a size issue but a parsing (or not - usage) issue too. You could approximate CSV in a JSON response like: { "columns": ["a", ..., "z"], "rows": [[1, ..., 26], ..., [11, 2266]] } Or: { "a": [1, ..., 11], ... "z": [26, ..., 2266] } which I've never seen, but would save space, and sort of enforced in the…

A columnar structure is definitely popular in the analytics community, although there are binary formats that are faster to scan again and can be mmap'd for zero-copy access.

The fundamental problem with CSV is that it has no canonical form or formal construction. Even the RFC documents it by example and historical reference, rather than from first principles, and does so with liberal use of "maybe". Consequently being very easy to fling about as a human but much harder to reason about in the abstract, and this is most evident when you get bogged down in the gritty details of writing a CSV importer for your application.

Re: Show HN: Transform a CSV into a JSON and vice versa

#17

Post by amyjess »

th0ma5 wrote: post #15

CSV is more of a rumor than a standard, plus JSON can have a tree structure. It is a fun idea to think about and may be useful in some narrow cases, but will fail in almost all but those most trivial of structures.

> CSV is more of a rumor than a standard

This reminds me of something my boss at a previous job would say: "I am morally opposed to CSV."

Why? Because we worked at an NLP company, where we would frequently have tabular data featuring commas, which means if we used CSV we'd have a lot of overhead involving quoting all our CSV data. Instead my boss preferred TSV (T = tab) as our preferred tabular data format, which was much simpler for us to parse since we didn't really deal with any fields that had \t in them.

Re: Show HN: Transform a CSV into a JSON and vice versa

#18

Post by scrollaway »

inopinatus wrote: post #16
OJFord wrote: post #5

Earlier quoted context omitted.

Yeah, lists of objects are pretty crap, because they're almost always homogeneous but unenforcédly so; it's not just a size issue but a parsing (or not - usage) issue too. You could approximate CSV in a JSON response like: { "columns": ["a", ..., "z"], "rows": [[1, ..., 26], ..., [11, 2266]] } Or: { "a": [1, ..., 11], ... "z": [26, ..., 2266] } which I've never seen, but would save space, and sort of enforced in the…

A columnar structure is definitely popular in the analytics community, although there are binary formats that are faster to scan again and can be mmap'd for zero-copy access. The fundamental problem with CSV is that it has no canonical form or formal construction. Even the RFC documents it by example and historical reference, rather than from first principles, and does so with liberal use of "maybe". Consequently bei…

+1 - If you are looking for something JSON-compatible, JSON Lines (one json object per line - https://jsonlines.org/) is pretty popular as well.

You could store a CSV in JSONL very easily. In fact, jsonlines' website shows it as its first example: https://jsonlines.org/examples/

And if you wanted something that is json file-wide, you can just add some commas and wrap in [] for a list of rows.

Re: Show HN: Transform a CSV into a JSON and vice versa

#19

Post by earthboundkid »

amyjess wrote: post #17
th0ma5 wrote: post #15

CSV is more of a rumor than a standard, plus JSON can have a tree structure. It is a fun idea to think about and may be useful in some narrow cases, but will fail in almost all but those most trivial of structures.

> CSV is more of a rumor than a standard This reminds me of something my boss at a previous job would say: "I am morally opposed to CSV." Why? Because we worked at an NLP company, where we would frequently have tabular data featuring commas, which means if we used CSV we'd have a lot of overhead involving quoting all our CSV data. Instead my boss preferred TSV (T = tab) as our preferred tabular data format, which was…

Lol, so instead of having an actually working solution (escaping), you had a still broken solution that just didn’t blow up as often so you could ignore it until it caused a crash.

Re: Show HN: Transform a CSV into a JSON and vice versa

#20

Post by earthboundkid »

anonytrary wrote: post #11
brundolf wrote: post #2

Slightly OT: I've realized that CSVs are dramatically more information-dense than the equivalent JSON, and actually make a pretty reasonable API response format if your dataset is large and fits into the tabular shape. They can be a fraction of the size, mainly because keys aren't duplicated for every item.

It would be much nicer for the consumer to just de-dupe the keys in your json than to serve an annoying format like CSV. Your JSON could basically be a matrix with a header row, there's nothing forcing you to duplicate keys. { header: [...columnNames], rows: [...values2DArray]}

Make rows 1 dimensional. You don’t need the second dimension, it’s implied by header length. Once you do this, the JSON gzips down to about the same size as CSV, according to the last time I tested this IIRC.
Post reply on HN