Are You Smarter Than A Fruit Fly?

On October 14th I gave a talk at AI Engineer in New York called “Are You Smarter Than A Fruit Fly?” There’s no recording yet. The slides are a web page with a live fly brain behind them, the fly itself is at fruitfly.review, and the code is at github.com/nicknisi/fly-review.

The pitch, from when I first proposed it:

Code review is done poorly in the age of agents. I want to tickle with the idea that maybe we don’t need to be reviewing the code, especially if we’re not the ones writing the code.

The fly

I took Google’s connectome of the male fruit fly, 176,422 neurons, and made it run on a laptop. A fly can’t read code, so Jev answers 30 yes/no questions about what changed, and the answers become a smell. Test results teach it: a reproduced bug is bitter, a harmless change that passes every test is sugar. If the smell alone makes its tongue come out, it approves.

I tried to train the fly on the codebase by using mutation testing to identify good and bad parts of the code and then train the fly to have memories based on that.

Then the room played against it. Three changes drawn at random, hands up to approve or request changes, then the fly’s vote, then the tests. The room got it right all the time. On 301 changes it had never seen, the fly scored 88.3%, where a coin flip gets 50%.

The harder test

That exam had a flaw: every bug changed behavior, and every harmless change didn’t. So the morning of the talk I gave it 24 tickets that are supposed to change what the code does, each with a right change and a plausible mistake. It approved 0 of the 24 correct changes. It learned “did the behavior change,” not “is this change right.”

I was just trying to be provocative. I do think we need to eliminate the current way of code review, but handing off to a fruit fly is not the answer.

What I proposed instead

Teams maintain their own review skill, a classifier decides what’s risky, automated review handles the rest, and only what fails or is too complex escalates to a human, in summary form. That human should understand “what the code is doing and why, and specifically why not other methods.”

Try the fly on your own pull request at fruitfly.review.