Let's start with a small program, where a is a whole number from 0 through 100:
let result = if a > 4 { 10 } else { 12 }; We're going to build a neural network that gives the same result for those 101 inputs. Neural networks are usually trained, which means their numbers are learned from examples. We won't train anything. We'll construct the numbers directly from the program.
The construction comes from Cheng Lou's handwired, a JavaScript project that turns ordinary code into neural-network operations, and this same comparison is the first example in its README. I ported handwired to Rust as handwired-rs, which this series uses.
Why would anyone want this? Not for speed: three arithmetic steps in place of one if is not a better way to write an if, and this post only demonstrates the mechanism. What it gives us is a program that is a fixed list of arithmetic steps, the same for every input, and such a list can be reasoned about and checked on every input we declare. By the end, we'll have done both for all 101.
Two building blocks: weighted sums and ReLU
The first building block is a weighted sum: multiply the input by a constant and add a constant offset. 5 - a is one, with a multiplied by -1 and an offset of 5.
The second is ReLU, short for rectified linear unit: relu(x) = max(0, x). It keeps a positive number and turns a negative one into 0.
Why would we need the second one at all? Because weighted sums alone can't make a decision. A weighted sum of a weighted sum is again a weighted sum: c * (d * a + e) + f is (c * d) * a + (c * e + f), a constant times a plus an offset. So any number of them in a row is a single straight line in a, and no straight line is 12 for every a up to 4 and 10 for every a from 5 on.
How do we turn a > 4 into 0 or 1?
Our first goal is a number, let's call it flag, that is 1 when a > 4 and 0 otherwise. handwired-rs builds it as two ReLUs in a chain:
inner = relu(5 - a) flag = relu(1 - inner) Does that really behave like a > 4? Let's try the two numbers right at the threshold. For a = 4, inner = relu(1) = 1, and then flag = relu(0) = 0. For a = 5, inner = relu(0) = 0, and then flag = relu(1) = 1. So far so good.
What about the other 99 whole numbers? We don't have to try them one by one. From 0 to 4, a is at most 4, so 5 - a is at least 1, which makes inner at least 1 and 1 - inner 0 or negative, and ReLU turns that into a flag of 0. From 5 to 100, 5 - a is 0 or negative, so inner is 0 and flag = relu(1) = 1. That's an algebraic argument: it covers every whole number in the range without trying any of them.
Turning the flag into 10 or 12
Now we have a number where the program had a decision, so the choice can become arithmetic:
result = 12 - 2 * flag For flag = 0 this is 12, and for flag = 1 it is 10, the two values of the original if. The flag selects the output by multiplication instead of by a jump.
Naming the parts: weights, biases and layers
Each of our three steps is a weighted sum, and the first two are followed by a ReLU. In neural-network terms, the multipliers (-1, -1 and -2) are weights, the offsets (5, 1 and 12) are biases, and each step is a layer. So we've built a network of three layers with one step in each, and it runs the same three operations in the same order for every input. handwired calls such a fixed list of operations a circuit, and that's the name you'll see in the code below.
In the figure, the chain is on the right, and flag for a from 0 to 10 is on the left: dots for the whole numbers, a dashed line for what happens in between.
That dashed slope between 4 and 5 should make you a little suspicious. What is it doing there?
What happens between whole numbers?
Everything we argued so far relied on a being a whole number. So let's break that on purpose and feed in a = 4.5. Before reading on, what do you expect: 10, 12, or something else?
Here is the walk through the chain: inner = relu(0.5) = 0.5, flag = relu(0.5) = 0.5, and result = 12 - 1 = 11. Neither branch. Inputs strictly between 4 and 5 produce a blend of 10 and 12, and that's the dashed slope in the figure.
So the network's correctness is guaranteed for the declared inputs, the whole numbers from 0 to 100. How do we tell the code what those inputs are?
Declaring the inputs in Rust, and testing them
Two facts describe our inputs. a lies between 0 and 100, which handwired-rs calls its range. The allowed values are spaced one apart, which it calls its grid. The comparison needs the grid: it works because no allowed input falls strictly between 4 and 5.
In the program below, look for assume_range and assume_grid, which declare those two facts, and emit_compare, which writes the two ReLUs for us. TensorExecutor runs the compiled circuit, and the loop counts the inputs where it differs from the if. Before you run it, what should it print? If our algebra is right, zero wrong, and 11 for 4.5.
use handwired_rs::{Circuit, Relation, TensorExecutor, emit_compare}; fn main() { let mut circuit = Circuit::new(1); let a = circuit.source(0); circuit.assume_range(a, 0.0, 100.0); // a is between 0 and 100 circuit.assume_grid(a, 1.0); // in steps of 1 let four = circuit.constant(4.0); let flag = emit_compare(&mut circuit, a, Relation::Greater, four); let result = circuit.affine(flag * -2.0 + 12.0); let program = circuit.compile(Some(&[result])); let mut executor = TensorExecutor::new(&program); let mut wrong = 0; for a in 0..=100 { let expected = if a > 4 { 10.0 } else { 12.0 }; if executor.run(&[a as f32])[0] != expected { wrong += 1; } } println!("{} operations, {wrong} of 101 inputs wrong", program.live_operation_count()); println!("a = 4.5 gives {}", executor.run(&[4.5])[0]); } I ran it in a new binary crate after cargo add handwired-rs@0.8.0, with Rust 1.97 on x86-64 Linux, and it prints:
3 operations, 0 of 101 inputs wrong a = 4.5 gives 11 Both expectations hold. The three operations are our three steps: for a "greater than" comparison on a grid of 1, emit_compare builds relu(1 - relu(5 - a)), which I checked in its source at v0.8.0. "0 of 101 wrong" is a test result, and it checks this implementation on exactly the declared inputs. The last line shows that the executor doesn't refuse 4.5. It computes the blend, just as we did by hand.
There is one thing we quietly assumed all along, though. Our algebra used exact arithmetic. Does the computer's arithmetic agree?
Why does the floating-point arithmetic agree?
The program runs on f32, 32-bit floating-point numbers, which round some results. So why wasn't anything rounded here? Two facts. Floating-point addition and multiplication round only when the exact answer can't be stored. And an f32 can store every whole number up to a limit far above 100. Every value here, from the input through each intermediate product and sum (such as the -1 * a inside 5 - a) to the result, is a whole number between -100 and 100. So every exact answer can be stored, and nothing is rounded. That's the numerical argument, and the test agrees with it.
Let's take stock. A one-line if became a circuit of three numerical operations, with weights written down from the program, an argument that covers every declared input, and a test that ran all of them. I ported handwired to Rust because I wanted to write the logic as ordinary Rust functions and have a compiler turn them into circuits like this one, so the original function could stay around as the test. Later posts get to that compiler.
Aside: handwired's README writes the flag as relu(a - 4) - relu(a - 5), two ramps subtracted, which gives the same 0 or 1 on whole numbers. We used the chained form because handwired-rs builds it.
Our numbers here never went past 100. What happens when they grow? In the next post we'll find out where that limit on whole numbers is, and what happens to the arithmetic beyond it.
