Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Transform a CSV into a JSON and vice versa

jsonmatic.com

Post reply on HN
Print view View original

51–60 of 111 posts

Re: Show HN: Transform a CSV into a JSON and vice versa

#51

Post by 867-5309 »

this can turn an HTML table into JSON with the option to download the JSON, or it can turn JSON into an HTML table with no option to download a CSV -- where does CSV come into this?

also, clearly javascript is a bit too ambitious for the job when e.g. PHP could provide the intended functionality with two lines of code: foreach($arrays as $values){echo implode(',', $values) . "\n";} echo json_encode($arrays);

also, CSV is more for storing rigidly-structured uniform columns and rows, whereas JSON is more for storing loosely-structured varying objects, otherwise you're redeclaring column headings in every array, which wouldn't make much difference for gzipped transport but still wasteful and verbose nonetheless. column headings are usually the first line of a CSV

Re: Show HN: Transform a CSV into a JSON and vice versa

#52

Post by anonytrary »

earthboundkid wrote: post #20
anonytrary wrote: post #11

Earlier quoted context omitted.

It would be much nicer for the consumer to just de-dupe the keys in your json than to serve an annoying format like CSV. Your JSON could basically be a matrix with a header row, there's nothing forcing you to duplicate keys. { header: [...columnNames], rows: [...values2DArray]}

Make rows 1 dimensional. You don’t need the second dimension, it’s implied by header length. Once you do this, the JSON gzips down to about the same size as CSV, according to the last time I tested this IIRC.

I edited-in the "2DArray" because I thought it was confusing... But you're right, just calculate offsets. The dominating term is still quadratic, and the term you mentioned is linear. It could be worth it for a scaled org like Google!

I wonder which parses faster. I guess CSV does but then the consuming code would still have to parse the strings into JS primitives...

Re: Show HN: Transform a CSV into a JSON and vice versa

#53

Post by mananaysiempre »

cowsandmilk wrote: post #7
OJFord wrote: post #5

Earlier quoted context omitted.

Yeah, lists of objects are pretty crap, because they're almost always homogeneous but unenforcédly so; it's not just a size issue but a parsing (or not - usage) issue too. You could approximate CSV in a JSON response like: { "columns": ["a", ..., "z"], "rows": [[1, ..., 26], ..., [11, 2266]] } Or: { "a": [1, ..., 11], ... "z": [26, ..., 2266] } which I've never seen, but would save space, and sort of enforced in the…

I’ll note, your second item here is a structure of arrays, which is often higher performance in practice when you are only interested in certain portions of the data. For this reason, your second structure is how I serialize code I’m interacting with in C.

... Also known as a very simple case of a “column-oriented database”, of which are several at various scales from Metakit[1] to Clickhouse[2]. It’s a neat way to have columns which are sparsely populated, required to accommodate large blobs, numerous but usually not accessed all at once, or frequently added and deleted.

Nothing’s perfect, of course: you can’t stream records in such a format, so no convenient Unix-style tooling.

[1]: http://www.equi4.com/metakit.html [2]: https://yandex.com/dev/clickhouse/

Re: Show HN: Transform a CSV into a JSON and vice versa

#55

Post by hnlmorg »

cerved wrote: post #31
brundolf wrote: post #9

Earlier quoted context omitted.

Sometimes you fetch a large dataset and only show one page at a time in the DOM, or render it as a line in a chart or something. At a previous workplace we had CSV responses in the hundreds of megabytes.

That sounds incredible inefficient What was the rational for such enormous single payloads?

Without knowing more about the application, I'd guess probably caching and/or scaling. If you only need 1 payload then that can be statically generated and cached in your CDN. Which in turn reduces your dependence on the web servers so few nodes are required and/or you can scale your site more easily to demand. Also compute time is more expensive than CDN costs so there might well be some cost savings there too.

Re: Show HN: Transform a CSV into a JSON and vice versa

#56

Post by osullip »

I run a software company and we have a challenge when it comes to these types of conversion tools.

If there is any data that is a) not publicly accessible or b) contains personal information, I cannot authorise the use of a web based third party tool. There is just too much risk that some bad actor uses this as a method to soak up data.

I would love to verify /validate that all of the processing is local and have some way to certify if this hasn't changed.

Re: Show HN: Transform a CSV into a JSON and vice versa

#57

Post by jarofgreen »

At work we work an a Python Library to do this, and much more:

PyPi: https://pypi.org/project/flattentool/

Source: https://github.com/OpenDataServices/flatten-tool

Docs: https://flatten-tool.readthedocs.io/en/latest/

It converts JSON to CSV and vice versa but also Spreadsheet files, XML ...

It has recently had some work to make it memory efficient for large files.

Work, BTW, is an Open Data Workers Co-op working on data and standards. We use this tool a lot directly, but also as a library in other tools. https://dataquality.threesixtygiving.org/ for instance - this is a website that checks data against the 360 Giving Data Standard [ https://www.threesixtygiving.org/ ].

Re: Show HN: Transform a CSV into a JSON and vice versa

#58

Post by hnlmorg »

hnick wrote: post #25
amyjess wrote: post #17

Earlier quoted context omitted.

> CSV is more of a rumor than a standard This reminds me of something my boss at a previous job would say: "I am morally opposed to CSV." Why? Because we worked at an NLP company, where we would frequently have tabular data featuring commas, which means if we used CSV we'd have a lot of overhead involving quoting all our CSV data. Instead my boss preferred TSV (T = tab) as our preferred tabular data format, which was…

PSV (Pipe) is also good, tabs can rarely show up in some sets of data like mail addresses if humans key them in. I usually go with one or the other if I have a choice.

CSV (and it's derivatives) made some sense 20 years ago but these days if you want to make your lives easier with tables you're better off using jsonlines.

  ["Name", "Session", "Score", "Completed"]
  ["Gilbert", "2013", 24, true]
  ["Alexa", "2013", 29, true]
  ["May", "2012B", 14, false]
  ["Deloise", "2012A", 19, true]
Plenty of tools, including some I've written myself, support it.

https://jsonlines.org/examples/

https://murex.rocks/docs/types/jsonl.html

Re: Show HN: Transform a CSV into a JSON and vice versa

#59

Post by 867-5309 »

hnlmorg wrote: post #58
hnick wrote: post #25

Earlier quoted context omitted.

PSV (Pipe) is also good, tabs can rarely show up in some sets of data like mail addresses if humans key them in. I usually go with one or the other if I have a choice.

CSV (and it's derivatives) made some sense 20 years ago but these days if you want to make your lives easier with tables you're better off using jsonlines. ["Name", "Session", "Score", "Completed"] ["Gilbert", "2013", 24, true] ["Alexa", "2013", 29, true] ["May", "2012B", 14, false] ["Deloise", "2012A", 19, true] Plenty of tools, including some I've written myself, support it. https://jsonlines.org/examples/ https://…

that's just CSV with unnecessary square brackets and whitespace
Post reply on HN