Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Transform a CSV into a JSON and vice versa

jsonmatic.com

Post reply on HN
Print view View original

101–110 of 111 posts

Re: Show HN: Transform a CSV into a JSON and vice versa

#101

Post by 867-5309 »

hnlmorg wrote: post #76
867-5309 wrote: post #59

Earlier quoted context omitted.

that's just CSV with unnecessary square brackets and whitespace

As someone who's written parsers for both CSV and jsonlines, I can assure you that you could not be further from the truth: 1. The whitespace is optional. It's just put there for illustrative purposes. 2. Whitespace in CSV can actually corrupt the data where some parsers make incompatible assumptions vs other CSV parsers. Eg space characters used before or after commas -- do you trim them or include them? Some will d…

thanks for your reply, it was really informative. I'm not a fan of CSV so will delve deeper into jsonlines as an alternative next time it crops up

Re: Show HN: Transform a CSV into a JSON and vice versa

#102

Post by hnick »

hnlmorg wrote: post #76
867-5309 wrote: post #59

Earlier quoted context omitted.

that's just CSV with unnecessary square brackets and whitespace

As someone who's written parsers for both CSV and jsonlines, I can assure you that you could not be further from the truth: 1. The whitespace is optional. It's just put there for illustrative purposes. 2. Whitespace in CSV can actually corrupt the data where some parsers make incompatible assumptions vs other CSV parsers. Eg space characters used before or after commas -- do you trim them or include them? Some will d…

CSV is terrible, and I'd never write my own parser, but there are datasets where tab or pipe can never appear and just using @line = split(/|/, $data) or similar in another language is so convenient for quick and dirty scripting.

Re: Show HN: Transform a CSV into a JSON and vice versa

#104

Post by hnlmorg »

th0ma5 wrote: post #98
hnlmorg wrote: post #76

Earlier quoted context omitted.

As someone who's written parsers for both CSV and jsonlines, I can assure you that you could not be further from the truth: 1. The whitespace is optional. It's just put there for illustrative purposes. 2. Whitespace in CSV can actually corrupt the data where some parsers make incompatible assumptions vs other CSV parsers. Eg space characters used before or after commas -- do you trim them or include them? Some will d…

https://xkcd.com/927/

That's the nice thing about jsonlines, it's not creating a new competing standard. It's just making better use of an existing standard (JSON).

Re: Show HN: Transform a CSV into a JSON and vice versa

#105

Post by hnlmorg »

hnick wrote: post #102
hnlmorg wrote: post #76

Earlier quoted context omitted.

As someone who's written parsers for both CSV and jsonlines, I can assure you that you could not be further from the truth: 1. The whitespace is optional. It's just put there for illustrative purposes. 2. Whitespace in CSV can actually corrupt the data where some parsers make incompatible assumptions vs other CSV parsers. Eg space characters used before or after commas -- do you trim them or include them? Some will d…

CSV is terrible, and I'd never write my own parser, but there are datasets where tab or pipe can never appear and just using @line = split(/|/, $data) or similar in another language is so convenient for quick and dirty scripting.

Not just datasets where tab or pipe can never appear, but also that quotation marks aren't used and new lines can never appear (in CSV a row of data can legally span multiple lines because you're not supposed to escape char 12 (or '\n' as it appears in C-like documents).

I do get the convenience of CSV and I've used it loads in the past myself. But if ever you're dealing with data of which the contents of it you cannot be 100% sure of, it's safer to use a standard that has strict rules about how to parse control characters.

Re: Show HN: Transform a CSV into a JSON and vice versa

#106

Post by hnick »

hnlmorg wrote: post #105
hnick wrote: post #102

Earlier quoted context omitted.

CSV is terrible, and I'd never write my own parser, but there are datasets where tab or pipe can never appear and just using @line = split(/|/, $data) or similar in another language is so convenient for quick and dirty scripting.

Not just datasets where tab or pipe can never appear, but also that quotation marks aren't used and new lines can never appear (in CSV a row of data can legally span multiple lines because you're not supposed to escape char 12 (or '\n' as it appears in C-like documents). I do get the convenience of CSV and I've used it loads in the past myself. But if ever you're dealing with data of which the contents of it you cann…

TSV/PSV generally don't allow newlines and commas/quotes are not special so are fine. Though Excel doesn't always play nice, but if you care about data integrity, you won't open it in Excel anyway.

Re: Show HN: Transform a CSV into a JSON and vice versa

#107

Post by hnlmorg »

hnick wrote: post #106
hnlmorg wrote: post #105

Earlier quoted context omitted.

Not just datasets where tab or pipe can never appear, but also that quotation marks aren't used and new lines can never appear (in CSV a row of data can legally span multiple lines because you're not supposed to escape char 12 (or '\n' as it appears in C-like documents). I do get the convenience of CSV and I've used it loads in the past myself. But if ever you're dealing with data of which the contents of it you cann…

TSV/PSV generally don't allow newlines and commas/quotes are not special so are fine. Though Excel doesn't always play nice, but if you care about data integrity, you won't open it in Excel anyway.

AFAIK TSV and PSV aren't specs, they're just an alternative delimiters for CSV. To that end most TSV and PSV parsers will be CSV parsers which match on a different byte ('\t' or '|' as opposed to ','). Which means if the parser follows spec (which not all do) then they will allow newlines and quotes too.

I'm not saying your use case is isn't appropriate though. eg if you're exporting from a DB who's records have already been sanitised and wanting to do some quick analysis then TSV/PSV is probably fine. But if you aren't dealing with sanitised data that doesn't contain \n, \" or others, then there is a good chance that your parser will handle them differently to your expectations -- and even a slim chance that your parser might just go ahead and slightly corrupt your data rather than warn you about differing column lengths et al. So it's definitely worth being aware that TSV and PSV suffer from all the same weaknesses as CSV.

Re: Show HN: Transform a CSV into a JSON and vice versa

#108

Post by earthboundkid »

anonytrary wrote: post #52
earthboundkid wrote: post #20

Earlier quoted context omitted.

Make rows 1 dimensional. You don’t need the second dimension, it’s implied by header length. Once you do this, the JSON gzips down to about the same size as CSV, according to the last time I tested this IIRC.

I edited-in the "2DArray" because I thought it was confusing... But you're right, just calculate offsets. The dominating term is still quadratic, and the term you mentioned is linear. It could be worth it for a scaled org like Google! I wonder which parses faster. I guess CSV does but then the consuming code would still have to parse the strings into JS primitives...

I haven't tested this, but my guess is that because the browser built in JSON.parse will be faster than whatever CSV parser you can write in JS just because it's precompiled to native code. Then the question becomes how long does it take to do the unpacking loop, but it should be pretty quick.

I'd love it if someone did a benchmark though.

Re: Show HN: Transform a CSV into a JSON and vice versa

#109

Post by earthboundkid »

dragonwriter wrote: post #27
earthboundkid wrote: post #20

Earlier quoted context omitted.

Make rows 1 dimensional. You don’t need the second dimension, it’s implied by header length. Once you do this, the JSON gzips down to about the same size as CSV, according to the last time I tested this IIRC.

Heck, you could do a single 1-D list (no object), and just give the header count as the first element, which would be even more compact.

Smart idea.

Re: Show HN: Transform a CSV into a JSON and vice versa

#110

Post by rahimnathwani »

quantumofalpha wrote: post #12
brundolf wrote: post #2

Slightly OT: I've realized that CSVs are dramatically more information-dense than the equivalent JSON, and actually make a pretty reasonable API response format if your dataset is large and fits into the tabular shape. They can be a fraction of the size, mainly because keys aren't duplicated for every item.

If you'd use a column-oriented format like {"col1":["a","b","c",...],"col2":[1,2,3...],...}, it's about the same density, no?

This is the default format used by pandas.DataFrame.to_dict()

I usually need a less dense version, e.g. to send to a jinja2 template, so mostly use to_dict(orient='index').

Post reply on HN