81–90 of 375 posts
I think parsing once into a faster format (sqlite3 or parquet) would be more beneficial. https://simdjson.org/
Misleading title, it's 90% of the faster time. It's about 45% faster. It's actually a little bit interesting, if you are interested in how we use language. You could argue that now you now get 90% more work done in the same amount of time, and that would align with other 'speed' units that we commonly use (miles per hour, words per minute, bits per second). However, the convention in computer performance is to measur…
Earlier quoted context omitted.
https://github.com/01mf02/jaq?tab=readme-ov-file#performance jaq runs 5x faster on my machine in some cases
jaq is nice. It just loses, performance-wise, to the final step in this article, and it can't do the second mentioned workload (yet) so I didn't include it.
Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON. > * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corrup…
A userland package manager like Gentoo Prefix could be used to install a custom build of this and still get security updates.
Gentoo linux is essentially made specifically for people like this, to be able to optimize one’s own linux rig for one’s specific usecase. After initial setup, it’s pretty simple and easy to use, I remember making a ton of friends at matrix’s Gentoo Linux channel, was fun times. https://www.gentoo.org/ Fun fact, initial ChromeOS was basically just custom Gentoo Linux install, I’m not sure if they still use Gentoo Lin…
That's true but worth noting that "optimize" here doesn't necessarily refer to performance.
I've been using Gentoo for 20 years and performance was never the reason. Gentoo is great if you know how you want things to work. Gentoo helps you get there.
O3 is a bad idea if the package is important. While that’s a dated opinion based on older experiences (Gentoo, various server software, etc), I suspect it’s still the easiest way to get exposed to compiler bugs.
If you're going to offer unsupported FUD, I would prefer you direct the FUD at NDEBUG and the fear that it could have disabled a load-bearing assertion.
Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON. > * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corrup…
Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON. > * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corrup…
This is certainly true. Also, by replacing the allocator and changing compiler flags, you're possibly immunizing yourself from attacks that rely on some specific memory layout.