Interestingly, shading languages started out like this - way before consumer GPUs.
I remember encountering this idea written in a book written by Ed Catmull of Pixar fame (can't find the title sorry, but it was written in the 80s), but generally comes from signal processing as a way of avoiding aliasing artifacts..
The core idea is to make programming, which is a discrete and discontinuous domain, into a well-behaved band limited signal. Otherwise you get aliasing (or jaggies), which can happen even INSIDE a surface, if the shader's like that.
The code idea for this is the step function which is the integral of the dirac delta. step(x) returns 1 for all x >0 and 0 otherwise.
Step is not a well-behaved function in the sense, that it changes infinitely quickly at x=0. But once we know what we want, we can replace it with something like that, that's well behaved.
Consider the example pseudocode
color = x> 5? green:blue;
can be rewritten as
color = blue + step(x-5)(green-blue)
With the two being equivalent.
Now if we put the code into a shader, we get jaggies. So to combat the value changing infinitely fast, we go for a function that's like step, but changes smoothly* from 0 to 1 around x=0. Enter smoothstep:
color = blue + smoothstep(x-(5+EPSILION),(x-EPSILON), x)*(green-blue)
And so we defined a 'transition zone' of +-EPSILON(an arbitrary number). While any smooth function can work, smoothstep is chosen because it has a smooth first and second derivative (meaning even if you want to get the rate of change, something that often pops up in computer graphics, the result will be still well behaved).
Pixar's Renderman shading language (which is remarkably similar to GLSL/HLSL/C), used to do this automatically for you. Essentially it could take arbitrary code peppered with if statements, and turn it into a continuous function.
Which is kinda cool imo.
It's also a cool trick in the age of AI. Since you have a function that's well-behaved, you can do things like gradient descent to train an AI to synthetize a function for you. You can even say, that you don't need exact results, you can accept some error.
In this case your program optimization problem can be reframed from doing idempotent transformations on the list of instructions, to getting a program that generates a target function whose error is no greater than some (mathematical) reference function.
I started this about 9 years ago and never finished it. The idea comes from a course in my telecom degree called "Señales Aleatorias y Ruido" (Random Signals and Noise), I spent so many evenings writing probability by hand, and every time I wanted to check a result with a computer it was a ton of boilerplate.
The engine is Rust, the JIT is built on Cranelift, there is also a WASM backend so everything runs in the browser too.
Full disclosure, I could only finish it now because of AI agents. In my experience they are amazing at the runtime and the numerical code, but pretty bad at language design, so I kept that part for myself.
oh! that is very interesting. I was not aware of I could simulate markov chains with Approximate Bayesian, I have some good reading to do this weekend! indeed, expressions like P(D == 8 | D > 3) are already natively supported: https://noiselang.com/play/#x=conditional_bayes
Fair! My thinking was that PM of a single tone signal (the one i use in the demo is equivalent to FM, but shifted a bit). And implementing real FM for decoding is a lot more noisy, but I will add some callout in the article.
Truth be told, you motivated me to write the exact FM with the differenciation, maybe. Could be interesting to simulate PM vs FM for non single tone signals, to see how FM does even better!
I know haha I wanted to be transparent about this, I have been coding since 9 years old, 32 years old now. I have nothing to prove other than it would have been impossible for me find time to complete this project without help, also a Toy language. Not trying to replace anything people use today :) it's a cute project
Nice! I’ve dabbled with something similar on my own lately (originally wrote/vibed to explain some concepts that came up when discussing D&D) at diceplots.com - different approach, keeping the distributions exactly analytical at every step, never sampling.
My system is blocking that site as it is on the HaGeZi blocklist. I don't have any further information, and I'm not expressing an opinion on the site. An alternative might be https://noiselang.com, which is not on the blocklist.
mmmh i can't see the domain blocked in the list, it's my personal blog, i don't even have tracking other than server-side stats. could it be because using netlify dns?
Firstly, I'm not intending any slight on you personally! In fact this might be more of an issue for you interacting with the site than for people just reading an article.
Interestingly, shading languages started out like this - way before consumer GPUs.
I remember encountering this idea written in a book written by Ed Catmull of Pixar fame (can't find the title sorry, but it was written in the 80s), but generally comes from signal processing as a way of avoiding aliasing artifacts..
The core idea is to make programming, which is a discrete and discontinuous domain, into a well-behaved band limited signal. Otherwise you get aliasing (or jaggies), which can happen even INSIDE a surface, if the shader's like that.
The code idea for this is the step function which is the integral of the dirac delta. step(x) returns 1 for all x >0 and 0 otherwise. Step is not a well-behaved function in the sense, that it changes infinitely quickly at x=0. But once we know what we want, we can replace it with something like that, that's well behaved.
Consider the example pseudocode
can be rewritten as color = blue + step(x-5)(green-blue)With the two being equivalent.
Now if we put the code into a shader, we get jaggies. So to combat the value changing infinitely fast, we go for a function that's like step, but changes smoothly* from 0 to 1 around x=0. Enter smoothstep: color = blue + smoothstep(x-(5+EPSILION),(x-EPSILON), x)*(green-blue)
And so we defined a 'transition zone' of +-EPSILON(an arbitrary number). While any smooth function can work, smoothstep is chosen because it has a smooth first and second derivative (meaning even if you want to get the rate of change, something that often pops up in computer graphics, the result will be still well behaved).
Pixar's Renderman shading language (which is remarkably similar to GLSL/HLSL/C), used to do this automatically for you. Essentially it could take arbitrary code peppered with if statements, and turn it into a continuous function.
Which is kinda cool imo.
It's also a cool trick in the age of AI. Since you have a function that's well-behaved, you can do things like gradient descent to train an AI to synthetize a function for you. You can even say, that you don't need exact results, you can accept some error.
In this case your program optimization problem can be reframed from doing idempotent transformations on the list of instructions, to getting a program that generates a target function whose error is no greater than some (mathematical) reference function.
I started this about 9 years ago and never finished it. The idea comes from a course in my telecom degree called "Señales Aleatorias y Ruido" (Random Signals and Noise), I spent so many evenings writing probability by hand, and every time I wanted to check a result with a computer it was a ton of boilerplate.
The engine is Rust, the JIT is built on Cranelift, there is also a WASM backend so everything runs in the browser too.
Full disclosure, I could only finish it now because of AI agents. In my experience they are amazing at the runtime and the numerical code, but pretty bad at language design, so I kept that part for myself.
It's a toy language. Ask me anything!
Definitely going to play around with this, thanks for posting.
I know MCMC isn’t your goal, but seems like this could be used for ABC-MCMC (as is?)
Would also be nice to have an option to plot using a KDE vs histograms.
(Also your FM example seems to be technically PM)
oh! that is very interesting. I was not aware of I could simulate markov chains with Approximate Bayesian, I have some good reading to do this weekend! indeed, expressions like P(D == 8 | D > 3) are already natively supported: https://noiselang.com/play/#x=conditional_bayes
Fair! My thinking was that PM of a single tone signal (the one i use in the demo is equivalent to FM, but shifted a bit). And implementing real FM for decoding is a lot more noisy, but I will add some callout in the article.
Truth be told, you motivated me to write the exact FM with the differenciation, maybe. Could be interesting to simulate PM vs FM for non single tone signals, to see how FM does even better!
Be warned - by using AI like this you've made yourself a lightning rod for the people who really really really dislike AI.
I know haha I wanted to be transparent about this, I have been coding since 9 years old, 32 years old now. I have nothing to prove other than it would have been impossible for me find time to complete this project without help, also a Toy language. Not trying to replace anything people use today :) it's a cute project
WOWW!!!!!
Reminds me of Haskell’s monad-bayes: https://monad-bayes.netlify.app/
oh! that's awesome, i had no idea haskell could express this things
It might be worth looking into probabilistic programming languages. I'm out of date, but I remember webppl, stan, anglican, pymc (a python library).
Seems worth an investigation and maybe mention on the article.
if you read the website, the author explicitly mentions stan in the comparison at the end ;^)
Nice! I’ve dabbled with something similar on my own lately (originally wrote/vibed to explain some concepts that came up when discussing D&D) at diceplots.com - different approach, keeping the distributions exactly analytical at every step, never sampling.
My system is blocking that site as it is on the HaGeZi blocklist. I don't have any further information, and I'm not expressing an opinion on the site. An alternative might be https://noiselang.com, which is not on the blocklist.
mmmh i can't see the domain blocked in the list, it's my personal blog, i don't even have tracking other than server-side stats. could it be because using netlify dns?
Firstly, I'm not intending any slight on you personally! In fact this might be more of an issue for you interacting with the site than for people just reading an article.
There are multiple versions of the list. The authoritative site appears to be https://github.com/hagezi/dns-blocklists, and making a fairly random choice, I used the "medium" version of the "Threat intelligence feed", and specifically the one marked "Link" for AdBlock. That took me to https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/adb..., and manualmeida.dev does appear in that list.
The software I'm using is Little Snitch on a Mac, but since the entry is in the list, that's not the problem.
This is great! thank you for the digging, gonna figuring out how it end up there
This reminds me of https://mc-stan.org
I'd have just written this as a Python library that lazily evaluates expression via numpy personally. The API is useful, language is not