A 45-year-old computer learns to invent computer names

Last Sunday I spent the day at an exhibit table with my Tandy Color Computers, rotating through demos: a rock-paper-scissors game that learns how you play, a sentence completer, a melody that finishes the bar you give it. The one I could have left up all day was a screen of sixteen Star Trek episode titles, none of which were ever filmed. I could tell who the Star Trek fans were. They’d stop, read TRIBBLES OF ARCHONS off the green screen, and laugh.

The demonstrations run on a machine I got for Christmas 45 years ago, with 32 kilobytes of memory. This series follows the computer-name model from training examples to generated names, with its arithmetic small enough to inspect.

The five posts in this series:

A 45-year-old computer learns to invent computer names: the training data, tokens and context window.
Where the numbers live: the weights, next-token probabilities and how training changes them.
Twenty epochs, then the assembly: what changes over twenty training passes, and how the 6809 implements the updates.
The prompt, the size, the training data: how prompts, model size and training examples affect the output.
Titles, a game, a fact, and a tune: title generation, a game that updates counts, retrieving an editable fact from context, and melody continuation.

Predict, compare, update

I’ve spent the past few years helping people at work figure out what to do with large language models, ever since I started tinkering with them to automate and improve how we do software engineering. The question under most of the other questions is the same one: what is it actually doing in there?

Training repeats a short loop: predict the next token, compare its probabilities with the training target, and adjust the parameters to reduce that prediction error.

The model shares a next-token learning objective with modern generative language models. It does not share their transformer architecture or scale. I wanted an implementation where I could follow each prediction and parameter update. The CoCo was the obvious candidate (it’s fair to say it’s responsible for my entire career), and 6809 assembly language was the only way it was going to fit.

The model has 290 parameters, numbers that we can change to fit it to a purpose.

The training data

Eighteen names of vintage computers, the training names file:

ACORN ARCHIMEDES      ATARI ST              SINCLAIR ZX SPECTRUM
ACORN BBC MICRO       COMMODORE 64          TANDY COLOR COMPUTER
APPLE II              COMMODORE 128         TANDY TRS-80
APPLE LISA            COMMODORE AMIGA       TANDY MODEL 100
APPLE MACINTOSH       COMMODORE PET
ATARI 400             SINCLAIR ZX80
ATARI 800             SINCLAIR ZX81

A computer doesn’t have words. It has numbers. So the first thing I had to decide was how to turn those names into numbered pieces. The pieces are called tokens, and here a token is a whole word. Split the eighteen names into words, sort the distinct words and number them. That gives 28 word tokens and one boundary token. The boundary token marks the end of a name and fills unused positions at the start of the context window.

 0 <END>       8 APPLE       16 LISA        24 TANDY
 1 100         9 ARCHIMEDES  17 MACINTOSH   25 TRS-80
 2 128        10 ATARI       18 MICRO       26 ZX
 3 400        11 BBC         19 MODEL       27 ZX80
 4 64         12 COLOR       20 PET         28 ZX81
 5 800        13 COMMODORE   21 SINCLAIR
 6 ACORN      14 COMPUTER    22 SPECTRUM
 7 AMIGA      15 II          23 ST

COMMODORE is 13. Why? Because it’s thirteenth when we put the list in alphabetical order. That is all thirteen means. It’s a reference, not a quantity, and there’s nothing to be learned from doing arithmetic on it. (Next post is about what the model does instead.)

A word outside this list has no token identifier, so this model cannot accept or generate it. This whole-word tokenizer cannot assemble an unfamiliar word from smaller pieces.

Two words at a time

The model’s input is never a whole name. It is a window, two tokens wide, and the model’s only job is to guess what comes next.

Take COMMODORE AMIGA. The window starts empty, which I write as two END markers, and the first thing to guess is COMMODORE. Then the window slides one step: END, COMMODORE, and the thing to guess is AMIGA. Slide again: COMMODORE, AMIGA, and the right answer is END, the name is over.

Three guesses from one two-word name. Do that for all eighteen and you have 58 examples, each one a pair of tokens and the token that actually followed. That’s the entire training set.

The context window limits which tokens are available for the next prediction. A larger window can contain more text; text omitted from the current context cannot directly affect this prediction.

Watch it work

When the program loads, the 290 parameters are random. I ask it for a name and it draws from those random parameters, and here is what came out, unedited:

II APPLE COMMODORE ACORN 64 II
LISA ATARI ATARI COLOR MACINTOSH TANDY
ARCHIMEDES ARCHIMEDES ARCHIMEDES 400 SINCLAIR COLOR

Nonsense, and the particular kind of nonsense you’d expect from throwing dice: words repeated, no maker at the front, no end in sight.

Then it trains on the 58 examples for twenty passes: 1,160 training steps. Each step predicts the next token, compares that prediction with the training target, and adjusts the parameters. Under the emulator, running at the real machine’s clock rate, that takes a little over a minute. Then I ask for names again, same seed, same request:

SINCLAIR AMIGA
SINCLAIR ATARI
COMMODORE ATARI
TANDY ARCHIMEDES

Sinclair never made an Amiga. Commodore never made an Atari. But it’s interesting to think, if they had how would it be different? Out of 200 draws at that point, 179 were both new (not one of the eighteen, word for word) and the right shape (two to four words, starting with a maker). Training adjusted the probabilities of tokens following each context. The examples contain no explanation of which words name companies or products, but the resulting word-order patterns can still produce plausible product names.

Why stop at twenty?

Why not keep going? Why did I choose twenty? The training loop reports a number called the loss, which measures how wrong the guesses are, and it keeps falling well past twenty. At twenty it’s 1.83. At sixty it’s 1.27. By that measure the model is still getting better.

So why not run it for sixty? Here’s what it draws at sixty, same seed, same request:

SINCLAIR BBC
SINCLAIR II
COMMODORE 64
TANDY MODEL COMPUTER

COMMODORE 64 is in the training data. At sixty, so are 143 of the 200 draws, word for word. At twenty epochs, only 18 of the 200 draws were copies. Longer training increased exact reproduction of the training names and reduced the number of new ones.

That has a name: overfitting.

There’s a trap at the other end too. At epoch zero, before any training, every draw is new, and every one is exactly six words long because six is where I stopped it. The end marker is a token like any other. Before training, END’s probability comes from the random parameters. At the checkpoints shown here, twenty epochs produced the most names that were both new and the right shape: 179 of 200 draws that were not in the training data and still looked like a plausible computer name.

Try it yourself

(Sorry non-mac folks, don’t have an option for you right now, happy to take PRs to get other platforms working! If I get a strong response, I can toss the clankers at it too.)

Everything is on GitHub, at svetzal/coco-llm. With the XRoar emulator installed, one command starts the run from random numbers:

make present EXP=4

It trains at the 1981 clock rate, parks when it’s done, and draws names when you press a key. Nothing is sped up.

The CoCo runs 6809 assembly, in src/6809. A Python reference implementation performs the same arithmetic on the Mac so I can check the assembly’s results. Experiment records hold the data and conclusions; export scripts extract the numbers shown in these posts. The repository map says what each folder holds. The live run is EXP-004, the live training run.

Watch it work, start to finish

Here’s the whole thing, about two minutes, recorded from the emulator running at the real machine’s clock rate. Nothing is sped up. Twenty times through the examples, then PRESS ANY KEY, then the same seed drawn before training and after it, one name at a time. The pause between names is the 6809 doing 29 multiplies and a softmax for every token.

Inside the 6809: checking the training run

Plausible names alone don’t tell me whether the assembly did the arithmetic correctly. I check the numbers training changed. The 290 parameters occupy two bytes each, so the final result is 580 bytes that can be compared directly.

The Mac runs a fixed-point reference model with the same integer arithmetic, starting seed, training examples and twenty epochs. A build script records its final parameters as the expected result. The 6809 program starts from random parameters and performs the training itself; the expected bytes are there for comparison, not loaded as its starting weights.

After training, the verification routine compares every byte. If one differs, it records the location, the actual byte and the expected byte. This checks agreement with the reference implementation, not whether an invented name is useful or true. The live training experiment’s correctness record reports the simulator checks separately from the whole-machine emulator run.

Next post: the training weights, why every word gets three of them, and why it’s useful to think in probabilities.