Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zigler: Zig NIFs in Elixir

github.com

Post reply on HN
Print view View original

81–90 of 93 posts

Re: Zigler: Zig NIFs in Elixir

#81

Post by jerf »

bmitc wrote: post #28
gioazzi wrote: post #26

Earlier quoted context omitted.

And then there are port drivers which are the worst of both worlds! Can crash the BEAM and need much more ceremony than NIF to set up but they’re pretty nice to do in Zig[1] as well [1]: https://github.com/borgoat/er_zig_driver

That's true. Haha! There's another option and that's setting up an Erlang node in the other language. The Erlang term format is relatively straightforward. But I'm honestly not sure of the benefit of a node versus just using a port.

The Erlang term format is straightforward, but if you want to set up another node in another language you need to correctly implement/emulate process linking, binaries, and some other stuff too, it's not just a matter of writing a socket to accept and emit Erlang terms.

It's not impossibly large but it's not something one does on a lark either; if there isn't support in your language already it's hard to justify this over any of the many, many message busses supported by both Erlang and other languages that don't have so many requirements.

Re: Zigler: Zig NIFs in Elixir

#82

Post by polvalente »

Cyph0n wrote: post #44
abrookewood wrote: post #43

Earlier quoted context omitted.

Why would anyone use a NIF instead of a Port then?

IPC/shared memory overhead?

Yeap, this is a big one. In Nx we have some facilities for doing zero-copy stuff that only really work if you have, say, Evision and EXLA running on the same OS process.

We do have IPC handles that could enable this over, say, ports, but then there's a whole other discussion on pointers vs ipc handles

Re: Zigler: Zig NIFs in Elixir

#83

Post by sodapopcan »

drawnwren wrote: post #13
alberth wrote: post #10

Earlier quoted context omitted.

Hence why Rustler is of so much interest since it provides more protections against this happening. Discord is a big Erlang + Rustler user.

Is any of this code open source? As an outsider, I'm kind of at a loss for why anyone wants this or what you kids are doing over there and how offended I should be by it.

TL;DR: Erlang/Elixir/etc are high level languages and the virtual machine they run on, the BEAM, is optimized for speedy IO but is not so great when it comes to intensive CPU tasks. You'll want to write the latter in a good systems language which is what libraries like this provide (you get C bindings out of the box, I believe).

Re: Zigler: Zig NIFs in Elixir

#84

Post by drawnwren »

jhgg wrote: post #14
drawnwren wrote: post #13

Earlier quoted context omitted.

Is any of this code open source? As an outsider, I'm kind of at a loss for why anyone wants this or what you kids are doing over there and how offended I should be by it.

https://github.com/discord/sorted_set_nif

Awesome! Thank you! My sarcasm got downvoted heavily (poe's law), but I was genuinely interested.

Re: Zigler: Zig NIFs in Elixir

#85

Post by filmor »

throwawaymaths wrote: post #77
el_oni wrote: post #57

Earlier quoted context omitted.

Rustler catches panics before they crash the VM and raises them on the elixir side as an exception. So your process might crash but the vm wont

This is terrible, actually. And I've run into it, causing a memory leak.

How so? The whole point of unwinding is to gracefully clear up on panics, how did it peak for you?

It's also not like there is much of a choice here. Unwinding across FFI boundaries (e.g. out of the NIF call) is undefined behaviour, so the only other option is aborting on panics.

Re: Zigler: Zig NIFs in Elixir

#86

Post by throwawaymaths »

filmor wrote: post #85
throwawaymaths wrote: post #77

Earlier quoted context omitted.

This is terrible, actually. And I've run into it, causing a memory leak.

How so? The whole point of unwinding is to gracefully clear up on panics, how did it peak for you? It's also not like there is much of a choice here. Unwinding across FFI boundaries (e.g. out of the NIF call) is undefined behaviour, so the only other option is aborting on panics.

Yes. Abort early in unit tests, core dump so it never makes it to prod

Re: Zigler: Zig NIFs in Elixir

#87

Post by filmor »

sodapopcan wrote: post #75
depr wrote: post #71

Earlier quoted context omitted.

That is not what I meant. I looked at sorted_set_nif which doesn't seem to compile on OTP 26 (we're at 27 now), and fastglobal which has a very old PR with 3 approvals has not been merged. Elixir libraries may not change _much_ but core libraries like telemetry, Ecto, ExDoc, Jason, still get either minor or patch releases all the time. If libraries get regular updates even if they are minor, it indicates they are in…

Ah ya, I do see how the optics of this could give off that impression. I don't use this library myself, but the issue is with Elixir 1.15.7 & OPT 26.1.26 which is VERY different than "It doesn't work on OTP 26." Certain patch versions of Elixir and OTP have caused problems before (sorry, I don't have a citation) and this particular issue looks like it's related to dependencies not syncing up on the config change? I d…

Rustler wasn't properly forward compatible (only with regard to the build process, a compiled library will work just fine on any newer OTP) until 0.29. They are using 0.22, upgrading Rustler will be enough to get rid of this issue for all future OTP versions.

Re: Zigler: Zig NIFs in Elixir

#88

Post by sodapopcan »

filmor wrote: post #87
sodapopcan wrote: post #75

Earlier quoted context omitted.

Ah ya, I do see how the optics of this could give off that impression. I don't use this library myself, but the issue is with Elixir 1.15.7 & OPT 26.1.26 which is VERY different than "It doesn't work on OTP 26." Certain patch versions of Elixir and OTP have caused problems before (sorry, I don't have a citation) and this particular issue looks like it's related to dependencies not syncing up on the config change? I d…

Rustler wasn't properly forward compatible (only with regard to the build process, a compiled library will work just fine on any newer OTP) until 0.29. They are using 0.22, upgrading Rustler will be enough to get rid of this issue for all future OTP versions.

Thank you for the full story here as I just gave the issue a cursory glance. As someone quite ingrained in Elixir, I see an issue referencing specific patch versions of Elixir and OTP and immediately understand it's very specifically targeting that specific Elixir/OTP combo. But depr brings up a good point that not everyone is immediately going to understand this, especially newcommers to the language and it’s generally hard not to just read the headline.

Re: Zigler: Zig NIFs in Elixir

#89

Post by filmor »

throwawaymaths wrote: post #86
filmor wrote: post #85

Earlier quoted context omitted.

How so? The whole point of unwinding is to gracefully clear up on panics, how did it peak for you? It's also not like there is much of a choice here. Unwinding across FFI boundaries (e.g. out of the NIF call) is undefined behaviour, so the only other option is aborting on panics.

Yes. Abort early in unit tests, core dump so it never makes it to prod

The panic is converted to an Erlang error exception. You have to explicitly ignore it to make unit tests pass in spite of it.

I am still interested in the situation you observed.

Re: Zigler: Zig NIFs in Elixir

#90

Post by zelphirkalt »

derefr wrote: post #15

Does anyone actually enjoy using these systems that encourage you to embed programming-language X code in programming-language Y heredocs? I always find actually doing that — and then maintaining the results over time — to be quite painful: you don't get syntax highlighting inside the string; you can no longer search your worktree reliably using extension-based filtering; etc. I personally find the workflow much more…

Actually literate programming might be a tool to get you syntax highlighting back. You could write one block of code in one language and the other one in another language and make one include the other in some place. Both blocks annotated to be their specific language, inside the prose. Emacs for example syntax highlights each block according to its corresponding programming language. It also allows you to edit blocks in separate buffers. Another way could be to switch the syntax highlighting of ones editor temporarily, but then syntax of the surrounding prose and other block might interfere.
Post reply on HN