Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Transform a CSV into a JSON and vice versa

jsonmatic.com

Post reply on HN
Print view View original

81–90 of 111 posts

Re: Show HN: Transform a CSV into a JSON and vice versa

#81

Post by jarofgreen »

contravariant wrote: post #79
jarofgreen wrote: post #57

At work we work an a Python Library to do this, and much more: PyPi: https://pypi.org/project/flattentool/ Source: https://github.com/OpenDataServices/flatten-tool Docs: https://flatten-tool.readthedocs.io/en/latest/ It converts JSON to CSV and vice versa but also Spreadsheet files, XML ... It has recently had some work to make it memory efficient for large files. Work, BTW, is an Open Data Workers Co-op working on d…

What are the advantages of this tool in comparison with e.g. pandas' json_normalize?

Flatten-tool has more options and functions than just that one pandas functions (But I haven't done a full comparison to all Pandas functions. I wasn't around when the tool was started so I can't say what analysis was done at the time.)

For instance I note with interest their examples on nested data and arrays. We have various different ways you can work with arrays, so you can design user-friendly spreadsheets as you want and still get JSON of the right structure out: https://flatten-tool.readthedocs.io/en/latest/examples/#one-... (Letting people work on data in user-friendly spreadsheets and converting it to JSON when they are done is one of the big use cases we have)

Re: Show HN: Transform a CSV into a JSON and vice versa

#84

Post by true_religion »

lcabral wrote: post #37
MonaroVXR wrote: post #35

Earlier quoted context omitted.

How did you figure this out?

The page has a link at the bottom to the GitHub project where you check the dependencies...

Yes but it’s not a library. It’s an entire website. It even uses Firebase.

Re: Show HN: Transform a CSV into a JSON and vice versa

#85

Post by true_religion »

hughcrt wrote: post #41
ianschmitz wrote: post #39

Earlier quoted context omitted.

There's nothing wrong with web-vitals...

There's nothing wrong with web-vitals, and it's included in create-react-app, which the author used.

I agree, though for the sake of argument Facebooks tolerance for tracking and fingerprinting far exceeds anyone else’s on the internet so their stamp of approval for web vitals is meaningless.

Re: Show HN: Transform a CSV into a JSON and vice versa

#87

Post by aae42 »

oweiler wrote: post #86

Why doesn't the library transform the csv into a an array of json objects?

i was wondering if there would be an option for this in the UI somewhere, but it doesn't look like there is

i wonder if it requires a "key" column to serve as the dictionary key

EDIT: i kind of wouldn't mind a HN discussion on which is better... ruby is my go-to scripting language, so i find this structure very natural, i've found when trying to do the same things with Go, i much prefer things to be structured more like an array of objects

would be interesting to hear the merits of both

Re: Show HN: Transform a CSV into a JSON and vice versa

#88

Post by aae42 »

scrollaway wrote: post #18
inopinatus wrote: post #16

Earlier quoted context omitted.

A columnar structure is definitely popular in the analytics community, although there are binary formats that are faster to scan again and can be mmap'd for zero-copy access. The fundamental problem with CSV is that it has no canonical form or formal construction. Even the RFC documents it by example and historical reference, rather than from first principles, and does so with liberal use of "maybe". Consequently bei…

+1 - If you are looking for something JSON-compatible, JSON Lines (one json object per line - https://jsonlines.org/ ) is pretty popular as well. You could store a CSV in JSONL very easily. In fact, jsonlines' website shows it as its first example: https://jsonlines.org/examples/ And if you wanted something that is json file-wide, you can just add some commas and wrap in [] for a list of rows.

i've known about json lines for a bit (had to make a utility that parsed some output of a program that used jsonlines and was irritated it didn't use the json spec)...

but for some reason just this post made me realize something... i've always been irritated by CSV log files (or space delimited), but would prefer something more structured like JSON, but JSON log files are pretty obnoxious for the reasons mentioned in the comments, a lot of data duplication...

JSON lines for log files seems like a great fit, not sure why i didn't realize this until now, i suppose it was the context of the discussion!

Re: Show HN: Transform a CSV into a JSON and vice versa

#89

Post by quickthrower2 »

gfody wrote: post #29
fellowniusmonk wrote: post #26

Earlier quoted context omitted.

I always try to use ascii char 31 (unit seperator) if I'm computationally generating csvs for shunting around data internally.

it's crazy how our progenitors had the wisdom and foresight to reserve FOUR distinct delimiters for us 28-31 file/group/record/unit but webdevs are just nope we'll go with commas and crlfs and when that doesn't work we'll JSON

JSON is probably popular because JS was popular and it’s a subset. And being a programming language that’s is typed by humans, so those special characters would not have naturally featured.

For data, if we are going to use reserved characters, maybe we just use protobuf and let the serialisation code take the strain.

Re: Show HN: Transform a CSV into a JSON and vice versa

#90

Post by quickthrower2 »

th0ma5 wrote: post #43
earthboundkid wrote: post #19

Earlier quoted context omitted.

Lol, so instead of having an actually working solution (escaping), you had a still broken solution that just didn’t blow up as often so you could ignore it until it caused a crash.

Escaping breaks often as well. The general problem is that the data is inline with the format. Parquet files or something that has clear demarcation between data and file format are more ideal but probably nothing is perfect or future proof. Or accepting of past mistakes either.

You can write a perfectly isomorphic escaping printer/parser
Post reply on HN