Does it still count as a Dirac delta when it’s a discrete distribution? (The distributions in TFA are not continuous - they are things like a roll of 1d6 etc)
NoiseLang: Where N = 5 is a Dirac delta
21–30 of 57 posts
Re: NoiseLang: Where N = 5 is a Dirac delta
#22Does it still count as a Dirac delta when it’s a discrete distribution? (The distributions in TFA are not continuous - they are things like a roll of 1d6 etc)
Yes, a Dirac delta is just "all the weight on one point", and that works fine on a die.
For the scope of the language it never even comes up, because Noise is a simulator, it does not evaluate densities, it draws samples.
The point is that every value goes through the same operators. Add them, compare them, pass them to a function, put one in the condition of an if. You can even use a random variable to define another random variable:
bias ~ unif(0, 1) flips ~[10] bernoulli(bias) // bernoulli just took a distribution where a number normally goes.
and in if-stataments:
DistributionC = if DistributionA But you right, dirac only applies to continuous functions, in Noise is only refers to the dirac measure. I found this article a fun/nerd to make my point that everything "acts" as a distribution from the DX perspective, but under the hood 5 is just 5.
And a constant collapses back to a plain integer in the graph anyway, so 5 costs nothing.
Re: NoiseLang: Where N = 5 is a Dirac delta
#23Interestingly, shading languages started out like this - way before consumer GPUs. I remember encountering this idea written in a book written by Ed Catmull of Pixar fame (can't find the title sorry, but it was written in the 80s), but generally comes from signal processing as a way of avoiding aliasing artifacts.. The core idea is to make programming, which is a discrete and discontinuous domain, into a well-behaved…
note that similar concepts appear in mathematics. Generally the term for it is a "mollified" function.
applied to the step function, you would get a smooth cutoff function
https://en.wikipedia.org/wiki/Mollifier#Smooth_cutoff_functi...
this is also related somewhat to the notion of differentiable programming. RELU is (roughly) the same as x * step(x). In differentiable programming one can replace it with smooth approximations, cf "softplus"
https://arxiv.org/pdf/2403.14606
That book also has a chapter on control flow, which is very similar to what you're talking about.
Unrolling an if statement into x = b (result of one branch) + (1-b) (result of the other branch) is also incredibly common in cryptography. If `b` is a "secret" variable, an if statement may leak the value of it via the branch predictor/speculative execution. The way around this is to compute both branches, and then select them with the above arithmetic expression. This mostly works, though compilers are tediously smart, and so one often has to be careful how with how you precisely do it.
Re: NoiseLang: Where N = 5 is a Dirac delta
#24I'd have just written this as a Python library that lazily evaluates expression via numpy personally. The API is useful, language is not
that has some technical limitations. For example, their impl can compile to wasm, which makes giving an online interpreter simpler/lighter weight than relying on running python in the browser.
Re: NoiseLang: Where N = 5 is a Dirac delta
#25Does it still count as a Dirac delta when it’s a discrete distribution? (The distributions in TFA are not continuous - they are things like a roll of 1d6 etc)
[deleted]
Re: NoiseLang: Where N = 5 is a Dirac delta
#26I started this about 9 years ago and never finished it. The idea comes from a course in my telecom degree called "Señales Aleatorias y Ruido" (Random Signals and Noise), I spent so many evenings writing probability by hand, and every time I wanted to check a result with a computer it was a ton of boilerplate. The engine is Rust, the JIT is built on Cranelift, there is also a WASM backend so everything runs in the bro…
Very cool. I have a couple of questions:
1. It's obvious that you are a big fan of Cranelift. I'd be interested to hear more about your experience in practical terms. For example could you share any insight about use cases where it is best suited, and where it might be better to look elsewhere? Did you hit any pain points? What was its killer feature for NoiseLang?
2. You wrote: "My favorite trick is in the RNG. Generating random numbers is a serial dependency chain, so instead of fighting that, the kernel runs four independent streams at once and lets the out-of-order core overlap them. This trick ended up beating a hand-written SIMD kernel!" What does this mean exactly? you just ran a scalar kernel 4-wide using SIMD instructions? or you interleaved 4 scalar copies of the same algorithm? did you generate the code or just duplicate the streams by hand?
Re: NoiseLang: Where N = 5 is a Dirac delta
#27I started this about 9 years ago and never finished it. The idea comes from a course in my telecom degree called "Señales Aleatorias y Ruido" (Random Signals and Noise), I spent so many evenings writing probability by hand, and every time I wanted to check a result with a computer it was a ton of boilerplate. The engine is Rust, the JIT is built on Cranelift, there is also a WASM backend so everything runs in the bro…
Very cool. I have a couple of questions: 1. It's obvious that you are a big fan of Cranelift. I'd be interested to hear more about your experience in practical terms. For example could you share any insight about use cases where it is best suited, and where it might be better to look elsewhere? Did you hit any pain points? What was its killer feature for NoiseLang? 2. You wrote: "My favorite trick is in the RNG. Gene…
I was exploring ways to speed up this language, the naive implementation is just a interpreter executed in rust, which can just do so much. Once thing i explored was to compile the program graph into WASM, then execute WASM. The idea is the the WASM runtime would JIT program and run faster than any interpreter I could write myself. During this exploration, i found that I could use the JIT optimizer directly and skip the WASM step.
What noiselang does it parse, convert to a execution graph then carefully use the Cranelift api to describe the program, and under the hood, it will find different opetimization and generate byte code that runs directly in the host CPU.
The reason for it to be killer is that it allows NoiseLang to run as native speeds, with very little compiler/optimization work. it's a very simple repo.
For the RNG, this was a discovery myself, when profiling, i found the many benchmarks were limited by the speed of the RNG itself, ie, if i could genenrate random numbers faster, the simulation would be faster. xoshiro's next number is computed from its current state. So to get number N+1 you must have finished number N. It's a chain: A → B → C → D. Your CPU can run maybe 6 integer operations per cycle, but a chain only ever offers it one to run. Five of the six lanes sit empty.
I tried to use SIMD to speed this up, but still hit the limit, even if using SIMD, it still had to wait for the next number, a massive speed up came from realizing that i can keep four independent xoshiro256++ states and emits four samples per loop iteration, i += 4. Since the four state-update chains share no registers, the out-of-order core issues them in the same cycles instead of stalling on one serial chain.
SIMD gives you more work per instruction. But I wasn't short on work, I was waiting. A 2-wide xoshiro still needs state N before it can compute state N+1, so the chain is the same length and I wait at every link, I just get two numbers per link instead of one.
And each link costs more. xoshiro rotates a 64-bit word every round, and NEON has no 64-bit rotate, so that becomes three instructions instead of one. Twice the numbers, three times the wait.
Four streams wins because it leaves the chain alone. It just runs four of them at once, and the CPU was already idle enough to overlap them for free.
Re: NoiseLang: Where N = 5 is a Dirac delta
#28I'd have just written this as a Python library that lazily evaluates expression via numpy personally. The API is useful, language is not
People do this kind of computation using numpy all the time. If you weren’t developing a new language with a new syntax, there really isn’t much of a library to write. At that point it’s just using numpy.
Re: NoiseLang: Where N = 5 is a Dirac delta
#29Does it still count as a Dirac delta when it’s a discrete distribution? (The distributions in TFA are not continuous - they are things like a roll of 1d6 etc)
Yes, a Dirac delta is just "all the weight on one point", and that works fine on a die. For the scope of the language it never even comes up, because Noise is a simulator, it does not evaluate densities, it draws samples. The point is that every value goes through the same operators. Add them, compare them, pass them to a function, put one in the condition of an if. You can even use a random variable to define anothe…
I like the “everything is a distribution” approach a lot[1]. Looks great. I’m looking forward to actually having a chance to mess around with it.
[1] And feels philosophically like the unification in the underlying maths between discrete and continuous probability that you get when you apply measure theory
Re: NoiseLang: Where N = 5 is a Dirac delta
#30I started this about 9 years ago and never finished it. The idea comes from a course in my telecom degree called "Señales Aleatorias y Ruido" (Random Signals and Noise), I spent so many evenings writing probability by hand, and every time I wanted to check a result with a computer it was a ton of boilerplate. The engine is Rust, the JIT is built on Cranelift, there is also a WASM backend so everything runs in the bro…
how much did Ai cost for you?