The code walkthrough: build and train the model, in this tab

Build the vocabulary

The complete set of characters the model can ever read or write — computed, not chosen:

const vocab = [...new Set(corpus)].sort();

Text → token IDs (and back)

Each character gets its row number in the vocabulary as its ID. Encoding is a lookup; decoding is a join — and the round trip must be perfect:

const stoi = Object.fromEntries(vocab.map((c, i) => [c, i]));
const encode = text => [...text].map(c => stoi[c]);
const decode = ids  => ids.map(i => vocab[i]).join("");

Initialize the weights — all of them random

Now the model. This walkthrough uses the 107,200-parameter sibling of the main model (2 layers, 2 heads, 64-wide, context 64) so training is watchable in a browser tab. Every weight starts as a small random number — std 0.02, the init lesson:

// every weight starts as a small random number (std 0.02 — the init lesson)
const randomWeight = () => 0.02 * gaussianRandom();

const model = {
  tokenEmbeddings:    randomMatrix(vocabSize, width),       // 47 rows × 64
  positionEmbeddings: randomMatrix(contextLength, width),   // 64 seats × 64
  layers: [],
};
for (let l = 0; l < numLayers; l++) {
  model.layers.push({
    queryMatrix:  randomMatrix(width, width),
    keyMatrix:    randomMatrix(width, width),
    valueMatrix:  randomMatrix(width, width),
    mlpExpand:    randomMatrix(width, 4 * width),   // 64 → 256
    mlpCompress:  randomMatrix(4 * width, width),   // 256 → 64
    // ...plus an output projection and two layernorms
  });
}

Ask the newborn model a question

Feed it the start of kural #1 and ask: what comes next? The attention core — dot products, causal mask, softmax, blend — runs exactly as the write-up describes:

// attention, for the token at position t:
const query = multiply(queryMatrix, x[t]);       // "what am I looking for?"

const scores = [];
for (let past = 0; past <= t; past++) {          // causal: past + self only
  const key = multiply(keyMatrix, x[past]);      // "what do I offer?"
  scores[past] = dotProduct(query, key) / Math.sqrt(headSize);
}

const shares = softmax(scores);                  // scores → % summing to 100

let blended = zeros(width);
for (let past = 0; past <= t; past++) {
  const value = multiply(valueMatrix, x[past]);  // "what do I contribute?"
  blended = add(blended, scale(value, shares[past]));   // weightage × value
}

The free correctness test: loss should be ln(47)

Before any training, a correctly wired model must be exactly as good as random guessing. We can predict its loss on paper — ln(47) = 3.8501 — and then measure it:

const probs = softmax(logits);                  // 47 numbers that sum to 1
const loss  = -Math.log(probs[correctNextId]);  // the surprise at the truth
// gave the truth 100% → loss 0;  gave it 1/47 → loss ln(47) = 3.85
// ...averaged over every position of every sequence in the batch

Take a single training step

Grab a random batch, forward, measure loss, trace blame backward (the hand-written backpropagation), and nudge every weight with Adam. One step barely moves the needle — press it a few times and watch:

const batch = randomWindows(corpusIds, batchSize, contextLength);

const loss      = forward(model, batch);   // predict at every position at once
const gradients = backward(model, batch);  // blame per weight — traced by hand,
                                           // no autograd anywhere
for (const weight of allWeights(model)) {
  // Adam, simplified: momentum smooths the nudge, variance scales it
  weight.momentum = 0.9 * weight.momentum + 0.1 * gradients[weight];
  weight.value   -= learningRate * weight.momentum / Math.sqrt(weight.variance);
}

Now let it run

Training is just step 7, thousands of times. Watch the loss fall from 3.85 and, every 25 steps, watch the same model try to write. A few minutes gets it from noise to verse-shaped:

while (training) {
  const loss = model.trainStep();
  if (model.stepNum % 25 === 0) show(model.sampleIds([NL], 100, 0.8));
}

— samples will appear here every 25 steps —

Write a verse

The recursive loop from the write-up: predict, roll the dice, append, repeat. Adjust the temperature and sample the model exactly as it is right now — the longer step 8 has run, the better this gets. (Generating pauses training; press Start again after.)

for (let k = 0; k < n; k++) {
  const logits = model.predict(out.slice(-T));
  const next = rollDice(softmax(logits / temperature));
  out.push(next);                          // feed it back in
}

What you just did

You loaded a corpus, derived a vocabulary, tokenized text, initialized ~107k random numbers, verified the newborn model guesses uniformly, and then nudged those numbers a few hundred times — and verse structure appeared. Nothing else happened. That is the entire recipe, and the 815k-parameter model behind the main page — and every frontier model — is this exact loop with bigger numbers.

The unabridged engine (~400 lines, gradient-checked against numerical derivatives) and the whole project are on GitHub.