Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simdjson – Parsing Gigabytes of JSON per Second

github.com

Post reply on HN
Print view View original

111–120 of 202 posts

Re: Simdjson – Parsing Gigabytes of JSON per Second

#111

Post by badeu »

xnormal wrote: post #77

Any chance of something similar for CSV? (full RFC-4180 including quotes, escaping etc). Terabytes of "big data" get passed around as CSV.

I've developped a full RFC compliant CSV parser with Python bindings and supporting SSE4 to AVX-512 instruction sets, however i'm struggling with my hierarchy to open-source it at the moment.

But, the goal of my message is not to tease you with an unavailable code. It's just to say it is a lot more simpler to write a CSV parser than a JSON parser.

So, do not hesitate to write one yourself ! It's easy and a nice way to introduce yourself to SIMD instructions.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#112

Post by elizabeth_olu »

raphlinus wrote: post #3

This is very cool. Meanwhile, in the xi-editor project, we're struggling with the fact that Swift JSON parsing is very slow. My benchmarking clocked in at 0.00089GB/s for Swift 4, and things don't seem to have improved much with Swift 5. I'm encouraging people on that issue to do a blog post. [1]: https://github.com/xi-editor/xi-mac/issues/102

https://www.elizabethblog.com.ng/2019/02/la-police-released-...

Re: Simdjson – Parsing Gigabytes of JSON per Second

#113

Post by blaisio »

xnormal wrote: post #77

Any chance of something similar for CSV? (full RFC-4180 including quotes, escaping etc). Terabytes of "big data" get passed around as CSV.

It's probably relevant to mention https://github.com/BurntSushi/rust-csv. It uses a state machine (which seems to be the author's expertise) to parse CSVs really fast. Based on some other work, you can do better if you use some of the new SIMD instructions.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#114

Post by osmarks »

uasm wrote: post #106
lifthrasiir wrote: post #93

Earlier quoted context omitted.

C is one of extreme cases; that's why Cap'n'proto works pretty well in C++ and its cousins for example (it amortizes the decoding cost to accessors, and accessors are really cheap in those languages). There are many languages and implementations where decoding cost is not as significant.

> "C is one of extreme cases" I would say it's the other way around. We've had the knowledge and tools to build performant, scalable and highly maintainable systems for a while now. The learning curve is there, but that's part of the trade. We've been too occupied with reducing the entry barrier though - the end result being people shoving JSON into places it should have never been in. JSON can absolutely be a part o…

Because the idea of Xi is that it can support different frontends for different platforms, and that probably wouldn't work out to well if they all had to be in C.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#115

Post by glangdale »

kristjansson wrote: post #82
chubot wrote: post #13

Earlier quoted context omitted.

I agree that's a good strategy for big JSON. Do you know of any such "lazy" parsers? I think the problem is that to extract arbitrary keys, you really need to parse the whole thing, although you don't need to materialize nodes for the whole thing. But if you have big JSON with a given schema, you may be able to skip things lexically. You basically need to count {} and [], while taking into account " and \ within quot…

It’s not exactly the lazy parser you describe, but Sparser[1] builds filters to exclude json lines/files that can’t contain what you’re looking for, and only parses those that might. The Morning Paper’s writeup[2] from last year provides a good summary [1]: http://www.vldb.org/pvldb/vol11/p1576-palkar.pdf [2]: https://blog.acolyer.org/2018/08/20/filter-before-you-parse-...

This work is somewhat orthogonal to ours as it assumes that you can locate JSON records without doing parsing; if I remember correctly, it groups JSON records as lines. If your JSON has been formatted to conform to this, I suppose it would be quite effective.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#117

Post by BFLpL0QNek »

Jerry2 wrote: post #45
glangdale wrote: post #20

One of the two authors here. Happy to answer questions. The intent was to open things but not publicize them at this stage but Hacker News seems to find stuff. Wouldn't surprise me if plenty of folks follow Daniel Lemire on Github as his stuff is always interesting.

I'm writing an IoT library for devices with tiny microprocessors and have been sending data as JSON or BSON (binary JSON). On the backend, I've been storing reports from IoT devices into a database (MariaDB on AWS). How crazy would it be to just store all the data as JSON files on disk (or S3 bucket) and then batch process them when I need to perform data analysis on them? If a million devices sends dozens of status…

Depending on your size / budget / needs Snowflake may interest you. https://www.snowflake.com/product/architecture/.

I haven't used it but have been given a presentation by them on it, and it was very very good.

They store data in S3 and use FoundationDB for indexes. You can feed it JSON and it'll index it and let you query it on a massive scale shockingly fast.

Obviously they are not aimed at small hobby projects but if your project has money / serious product depending on your needs it's well worth looking at.

On the S3 cheaper / smaller end you can batch up data daily / weekly etc. So the landing bucket acts as a queue that gets processed creating daily batch files from the small files aggregated together. You can then take the daily batches to create weekly batches etc etc, essentially partitioning. This will reduce the total number of files needed to query. If you use deterministic names based on how you plan to query this can also reduce the number of files you need to list / parse. When batching / re-partitioning the data you can also use the Apache Parquet format to compress a little better + also import in some of the querying tools out there.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#118

Post by dkersten »

lifthrasiir wrote: post #69
saurik wrote: post #58

Earlier quoted context omitted.

... claims the project whose contributor is here claiming that they are "struggling" with JSON performance. Yeah... "in the noise". LOL.

You misread the rationale. He is arguing that, with all conditions same, the difference between binary formats and JSON would be in the noise. It is often the case that the object construction is more costly than the JSON parsing, and you can't fix that with binary formats. As a minimal and extremely non-scientific benchmark, I've constructed a simple fixed data structure that encodes to JSON (using Python `json` mod…

> with all conditions same, the difference between binary formats and JSON would be in the noise.

But, seemingly, in this case the conditions aren't the same.

Re: Simdjson – Parsing Gigabytes of JSON per Second

#119

Post by wongarsu »

adrianN wrote: post #44

I feel like if you need to parse Gigabytes per second of JSON, you should probably think about using a more efficient serialization format than JSON. Binary formats are not much harder to generate and can save a lot of bandwidth and CPU time.

I have in the past parsed terabytes of JSON. The specific use case was analysing archived Reddit comments. The Reddit API uses JSON, and somebody [1] runs a server that just dumps them in a file, one line of JSON per comment, and offers them for download (compressed, obviously). So now you end up with Gigabytes of small JSONs per month, and anything you do will be quickly dominated by JSON parsing time.

You could store them in some binary format, but the API response format changed over the years with various fields being added and removed, and either your binary format ends up not much better than JSON or you end up reencoding old comments because the API changed.

1: http://files.pushshift.io/reddit/

Re: Simdjson – Parsing Gigabytes of JSON per Second

#120

Post by dkersten »

glangdale wrote: post #80
avodonosov wrote: post #74

Earlier quoted context omitted.

Any chance to have a similar thing for s-expressions? I parse GBs of them and Common Lisp reader is very slow.

Probably not too hard. It would come down to how easy it is to detect quoting conventions so you don't accidentally parse () chars in strings. JSON is medium-easy. I don't know where the canonical definition of s-expressions you're using comes from (is it just Common Lisp?) so I don't know how this works. We'd like to have some more examples of formats people care about - I'm interested in generalizing this work. So…

As a clojure user, I care about EDN, but its probably too niche to spend your time on.

https://github.com/edn-format/edn

Post reply on HN