Yesterday, I pondered whether generative AI might eventually kill Capture the Flag (CTF) competitions, such as the one my work team ran at DEF CON last weekend. One data point that makes me worry about that possibility is the lifespan of this year’s crossword challenge. Last year’s puzzle held the record as our longest-running contest, with the final available badge claimed after approximately 36 hours. Last weekend’s crossword lasted about three hours.
I don’t believe this year’s puzzle was appreciably easier than the previous one. The obvious difference was the widespread availability and rapidly improving capability of generative AI. As I described on Friday, contestants were photographing the crossword and handing the entire problem to their LLM of choice. Sometimes the answers were hilariously wrong, but AI was nevertheless remarkably effective at accelerating the solve process.
For a short while this year, it appeared that the 36-hour record might stand in perpetuity. But then, “Voice Mail Villains” aka “Challenge 4” dropped online at 3:00 Thursday afternoon during setup day, before DEF CON had officially begun. Unlike some challenges that require contestants to physically visit the Aerospace Village, anyone monitoring our challenge drops on our Discord channel could immediately begin working on it. The tenth and final solve did not arrive until Saturday afternoon, 49 hours and 43 minutes later. Our previous endurance record had been decisively broken.
I would like to claim that this was because Synaptic Rodeo designed an extraordinarily sophisticated challenge that defeated both humans and artificial intelligence through sheer technical brilliance. Unfortunately, I think the actual explanation is considerably more amusing.
Everyone was overthinking it.
This is an appropriate time to direct everyone’s attention to the tagline of this blog. Overthinking is core to the branding of this site. Consequently, perhaps it is no surprise that Synaptic Rodeo created a challenge that caused a convention full of hackers to dramatically overthink a problem.
“Voice Mail Villains” consisted of a WAV audio file containing a wealth of information. Some of the hidden elements were legitimate clues intended to move contestants toward the solution. Others were gratuitous blind alleys placed there… for my own amusement.
Once contestants discovered that information was hidden inside the file, they quite reasonably assumed that everything they found must matter. One discovery led to another, which reinforced the belief that there was still more to discover. Every new artifact became another thread to pull. The increasing availability of sophisticated analytical tools only expanded the number of directions available for investigation.
Eventually, the challenge developed a fascinating feedback loop. Contestants were finding more and more information without necessarily getting any closer to the answer. The additional analysis was not resolving the uncertainty. It was creating more of it.
Within hacker culture, there is a familiar refrain when someone becomes stuck on a problem: “Try Harder.” The phrase can sound dismissive to an outsider, but that isn’t really the point. It reflects an expectation of persistence and self-reliance. Getting stuck is part of learning. Before asking someone else for the answer, people are told to experiment with another tool, research something unfamiliar, reconsider their assumptions, and keep working the problem.
Most of the time, “Try Harder” is excellent advice. However, as Voice Mail Villains stretched through Thursday night, all day Friday, and into Saturday, it became increasingly apparent that insufficient effort was not the problem. People were trying extraordinarily hard. In fact, they were trying so hard that their determination to find increasingly complicated solutions was becoming part of what prevented them from seeing the simpler path through the challenge.
As the challenge designer, that put me in an unusual position. I wanted people to solve it. The objective was never to create something impossible and then congratulate myself for being clever enough to stump everyone. A good CTF challenge should be difficult enough to require meaningful work, but ultimately provide contestants with the satisfaction of figuring it out.
With the clock winding down toward the final hours of DEF CON, I started doling out hints.
On day 2 and day 3, I issued early-morning guided affirmations on the Discord channel. They were intentionally written in the language of meditation, encouraging people to relax, breathe, pay attention to what was around them, and reconsider how they were approaching the problem. In a culture where the traditional instruction is to “Try Harder,” Synaptic Rodeo was effectively telling everyone to stop trying quite so hard.
Persistence is enormously valuable, but persistence and effort are not exactly the same thing. Continuing to attack the same problem with increasing intensity is only one form of persistence. Sometimes the more resilient response is to recognize that an approach isn’t working, disengage from it, and return to the problem from an entirely different direction.
That is much harder than it sounds because effort creates its own investment. Once we have spent significant time pursuing a particular theory, abandoning it feels like throwing away all the work that brought us there. We become increasingly motivated to prove that the path was correct rather than objectively considering whether it is actually taking us anywhere useful. More effort can make us more committed to the assumptions that got us stuck in the first place.
This is a tennis blog, so I am compelled to return to that topic. Fortunately, there is direct linkage here. There are absolutely days when the answer on the tennis court is to try harder. We need to hustle for one more ball, maintain concentration for another set, or remain disciplined when fatigue makes the easier choice of giving up increasingly tempting.
However, there are other moments when trying harder makes the problem worse. We miss a couple of balls and start consciously trying to correct technique. Every error adds another mechanical thought. Soon we are thinking about the grip, preparation, footwork, contact point, swing path, and follow-through while simultaneously trying to play the next point. The harder we work at manufacturing the correct mechanics, the farther we move from muscle memory of the fluid movement we have performed thousands of times.
Tactics can produce the same trap. We decide before a match that an opponent’s backhand is the weakness and continue attacking it long after the evidence suggests that particular strategy isn’t working. Rather than reconsidering the assumption, we conclude that we simply need to execute the strategy better. We try harder at doing the wrong thing.
That is why I increasingly think resilience deserves a more nuanced definition than simply refusing to quit. Genuine resilience requires remaining engaged with a problem while retaining enough flexibility to recognize when the way we are approaching it needs to change. Sometimes persistence means applying more effort. Sometimes it means backing away long enough to see the problem differently.
The people who eventually solved Voice Mail Villains did not succeed because they stopped trying. They prevailed because they managed to direct their focus to the information that deserved their attention and away from distractions they needed to let go. The mere fact that something could be found did not mean that it needed to be pursued.
That may be the most useful lesson I brought home from DEF CON this year. We live in an era when technology makes it increasingly easy to generate more information, perform more analysis, and explore more possibilities. As the first two posts in this series demonstrated, artificial intelligence dramatically amplifies that capability. However, more data and analysis does not automatically produce more understanding. Sometimes it merely gives us more opportunities to become distracted by things that don’t matter.
The same is true in tennis, both on and off the court. Improvement requires curiosity, persistence, analysis, and a willingness to work through difficult problems. Yet there is also considerable value in recognizing when another mechanical thought, another tactical adjustment, or another attempt to force the solution is not the path to victory.
For nearly 50 hours, Voice Mail Villains had people looking for increasingly complicated answers hidden inside an audio file. By the end, my guidance had become considerably simpler than the hacker community’s traditional instruction to “Try Harder.” I was telling people to stop looking and just listen.
