pong-ish
I trained a tiny neural net to play Pong, and built a way to watch it think. I’m a visual learner, shoot me.
how it works
It gets six numbers, scaled by the court size or the ball’s top speed:
| ball x | how far across the court the ball is |
|---|---|
| ball y | how far down the court the ball is |
| vel x | horizontal speed and direction |
| vel y | vertical speed and direction |
| ball−ai | the ball’s height relative to its own paddle |
| ball−you | the ball’s height relative to your paddle |
Those go through eight neurons using tanh, then one output that sets the paddle’s direction and speed.
The output is between −1 and 1: up, down, or somewhere in between. The game gives it fresh numbers every 16ms. It uses the same weights throughout your match.
6 × 8 + 8 + 8 × 1 + 1 = 65
That’s 56 weights and 9 biases. The whole forward pass is a small TypeScript function.
tanh neurons, and the output that moves its paddle. Green weights are positive, blue negative. Brighter is more active.training it
I used the cross-entropy method. It tries 60 variations of the weights, has each play 24 rallies, keeps the best 10, and repeats. Each run starts from the current model.
It trains against scripted trackers, including one with no reaction delay. The “human” opponents are scripts with extra lag and noisy aim; I haven’t measured a win rate against actual people.
The trainer uses the game’s physics and tests a few court widths. I check it on separate seeded rallies and count wins, losses, and timeouts separately.
trying to make it better
I don’t have much machine learning experience. This was a project for fun, so read what follows as notes from someone learning, not findings.
The net above is still the first one. I tried three ways to beat it and didn’t ship any of them, but each one taught me something.
a bigger net
8 → 16 hidden neurons, same training method, starting from the old weights. The new neurons start at zero, so at first it plays exactly like the old one.
It beat the old model 955–45 head to head. Against the scripted opponents it was more of a trade:
- better against the no-delay tracker
- worse against the strongest “human” script
- about the same on average
Most of its training rallies were against the no-delay tracker, with a few against the strongest “human” script. It got better at the first and worse at the second, so that’s mostly what the weighting bought. 64 neurons didn’t improve at all. Probably random search getting less efficient with more parameters, but I didn’t test that.
writing the answer by hand
To give the net something better to learn from, I first wrote a player by hand. The physics here are simple: between paddle hits the ball moves in a straight line and bounces off the walls, and where it lands on your paddle sets the return angle.
So the player works out where the ball will arrive, tries a few return angles in simulation, and picks whichever one a perfect tracker would miss by the most. 40,000 rallies against the scripts, zero losses.
teaching the net to copy it
Behavior cloning: record game states, label each one with what the hand-written player would do, and fit the net to the labels with gradient descent.
Done naively, it failed badly. The net only saw states the expert visits, so one small mistake put it somewhere it had never been. The standard fix is DAgger: let the net play, have the expert label the states the net actually reaches, retrain on everything so far, repeat.
After a dozen rounds the 16-neuron copy lost a quarter as often as the original (1,640 losses to 6,826 over the same 36,000 rallies). It seemed to pick up the defense but not the attacking returns. Against the no-delay tracker its rallies mostly timed out, and its win rate there fell from 62% to 15%. Another round of the original training method never found a version that was better against every opponent.
what i took away
The net gets good at whatever it’s rewarded for (obviously). Mine mostly played the no-delay tracker, so it got better against that one and a bit worse against others.
I wouldn’t have noticed from the average win rate alone. Two of these nets have nearly the same average and still play veryyy differently against specific opponents.
show as a table
| opponent | original, 8 neurons | bigger, 16 neurons | copy of the hand-written player |
|---|---|---|---|
| average | 89.1% | 90.1% | 86.9% |
| no-delay tracker | 62.5% | 68.1% | 43.1% |
| delayed tracker | 95.6% | 96.7% | 97.0% |
| strongest “human” script | 94.8% | 90.5% | 95.6% |
| middle “human” script | 97.0% | 97.1% | 99.0% |
| weakest “human” script | 95.6% | 98.3% | 99.6% |
So I compare every opponent and court width separately, on rallies the training never saw, and only call something better if it’s better everywhere. None of these were, so the site still runs the first one. :p
caveat
All of these opponents are scripts. Your paddle moves instantly with the mouse, which none of the trackers can do. I added scripts that move like a mouse and read the bounces, and on a wide court every player, including the hand-written one, rarely scored against them. How any of this does against real people is still a guess.
That’s why I need you to play a round!
Finished games get saved (just your paddle’s path and the score) so I can study where it loses and use them to make it better.