Yesterday, I described watching hundreds of highly intelligent people submit the same obviously wrong answer to one of my DEF CON CTF challenges because an AI model had confidently (but incorrectly) told them it had cracked the correct flag. My concern was not that generative AI got the answer wrong. Rather, it was how readily people surrendered their own judgment to the machine.
Today, I want to explore a different concern that has been increasingly occupying those of us who design challenges for the Aerospace Village. I am beginning to wonder whether generative AI might eventually kill the CTF.
A Capture the Flag competition, almost universally abbreviated as CTF within the hacker community, consists of a series of challenges designed to test problem-solving and technical skills. Depending upon the competition, participants may encounter things like cryptography, steganography, programming, reverse engineering, network security, puzzles, physical clues, or combinations of several disciplines. Successfully completing a challenge ultimately reveals a hidden string of characters known as the “flag.” Submit the correct flag, and you have demonstrated that you solved the problem.
Traditionally, the flag was never really the point. The value comes from everything required to find it. CTF participants encounter unfamiliar problems, research new technologies, write code, experiment with tools, follow approaches that fail, reconsider assumptions, and eventually figure out how all the pieces fit together. The flag simply provides an objective mechanism for determining whether they successfully completed that journey.
Generative AI is fundamentally disrupting that model. Historically, a challenge designer could focus primarily on how difficult a problem would be for a reasonably skilled human with access to conventional tools and the internet. Today, there is another consideration. We also have to determine how difficult the problem will be for that same human with access to increasingly capable artificial intelligence.
I experienced how rapidly that landscape is changing with another one of my challenges this year. I originally created that puzzle roughly six months before DEF CON. Knowing that AI would inevitably be used against it, it was tested against several LLMs. At the time, they weren’t particularly successful. The challenge required enough interpretation and reasoning that I was satisfied contestants would still have to do meaningful work to solve it.
However, about two weeks before we finalized the challenges for DEF CON, I tested it again. The puzzle hadn’t changed, but AI had improved significantly in the interim.
What had been relatively resistant to generative AI six months earlier could suddenly be cracked with surprisingly little difficulty. That sent me back to the challenge development drawing board at precisely the point when I would have preferred to be finished. I added a couple of additional layers specifically intended to prevent contestants from simply feeding the challenge into an LLM and asking for the flag.
Even with those modifications, contestants using AI assistance solved the challenge in roughly an hour. That isn’t necessarily a problem. An hour of productive interaction with a difficult puzzle still represents meaningful engagement. However, the experience illustrates the extraordinary challenge facing CTF designers. We are developing against a technological baseline that can change significantly between the day a puzzle is created and the day it is released.
The challenge also provided another demonstration of the behavior I described yesterday. Completing the first phase revealed an extraordinarily long URL. The information embedded within that string was intended to provide instructions for determining the actual web address. In other words, reaching the long URL wasn’t the solution. Contestants still needed to interpret what they had discovered and figure out where they were really supposed to go.
As I was finishing the challenge, I briefly considered whether I needed to register the domain represented by the instructional URL. Nobody following the intended solution path should ever actually navigate there, so purchasing it seemed unnecessary.
Then I remembered where I was going. If I left that domain unregistered, there was a reasonable possibility that someone at DEF CON would encounter it, register it, and point it somewhere of their choosing. At best, that could confuse contestants. At worst, someone could use it to steer people toward malicious content or software while appearing to be part of our challenge.
Consequently, I registered two domains. The correct one led contestants to the next phase of the puzzle. The incorrect one redirected them to Rick Astley’s Never Gonna Give You Up video on YouTube.
That turned out to be one of my more prescient decisions of the weekend. As contestants began clearing the first stage of the challenge, I was repeatedly regaled with the familiar opening strains from computers and phones around the Aerospace Village as people Rickrolled themselves.
Presumably, at least some of them had handed the output from the first stage to an AI system and blindly followed whatever URL it identified without stopping to consider whether that destination made sense. Much like yesterday’s crossword, the problem wasn’t that AI produced an imperfect response. The human still needed to exercise discernment over what to do with the information it provided.
Those experiences have contributed to a much larger conversation within our team about how CTF challenges need to evolve. One of our more interesting observations is that making a challenge more technical does not necessarily make it more resistant to AI. In fact, we have frequently observed the opposite.
Highly structured technical and mathematical problems are increasingly fertile territory for LLMs. Standard encoding schemes, mathematical relationships, programming problems, and recognizable technical patterns provide AI with precisely the sort of structure it can exploit. Increasing the technical complexity may dramatically increase the difficulty for a human while doing comparatively little to slow down the machine assisting them.
That creates a strange problem for challenge designers. The traditional instinct when a CTF challenge is too easy is to make the technical problem harder. In an AI-enabled environment, doing that may simply increase the advantage enjoyed by contestants who delegate the technical work to an LLM.
Consequently, we have started experimenting with entirely different approaches. For one of my challenges this year, I developed a novel method for encoding digitized characters rather than relying upon a recognizable standard. Other challenges increasingly incorporate information from the physical environment, requiring participants to locate something, observe something, or interact with a person before they possess everything necessary to solve the problem. Scavenger hunts and physical clues suddenly have an important technical property that was fun (but extra) just last year. Now, it is the best anti-AI in existence.
My team is already researching, discussing, and debating techniques for making next year’s challenges resistant to AI without simply making them frustrating for humans. The fact that this has become an explicit design consideration would have seemed somewhat absurd only a few years ago. Today, I think it may be essential to preserving the CTF as a meaningful learning experience.
That brings me back to the uncomfortable question of whether AI is killing the CTF.
I don’t think we know yet. CTFs have always evolved alongside technology, and cybersecurity itself is fundamentally shaped by the continuous interaction between new capabilities and the countermeasures developed in response. Perhaps generative AI is simply another step in that process. Challenge designers will adapt, contestants will develop new approaches, AI will become more capable, and the cycle will continue.
However, there is something fundamentally different about a technology that can potentially eliminate the intellectual work the exercise was created to produce. If a contestant can photograph a challenge, submit it to an LLM, and receive the flag, the CTF technically still functions. Someone solved the challenge and submitted the correct answer. Yet almost everything that made the exercise valuable has disappeared.
That is the part that concerns me. Six months before DEF CON, I had created what I believed was an AI-resistant challenge. Two weeks before DEF CON, I discovered that I hadn’t. By the time the conference began, I had redesigned portions of it specifically to restore some of the intellectual friction that improvements in AI had eliminated.
Perhaps generative AI will not kill the CTF. It may instead force us to become considerably more thoughtful about what CTFs are actually intended to accomplish.
There is a parallel to tennis. Technology can increasingly analyze our strokes, identify weaknesses, recognize tactical patterns, and suggest ways to improve. So can a coach, but AI has a more direct feedback loop. Automated tools have the potential to accelerate learning in ways that would have been unimaginable for recreational players only a few years ago. However, there are aspects of tennis that develop precisely because players are forced to solve problems on their own.
A coach can tell a player where to serve, just as an LLM can tell a hacker what encoding scheme to try. That information can be enormously useful. However, recognizing the pattern independently and learning why the solution works is a different capability. Tennis players ultimately are required to make those judgments themselves while standing on the court.
Perhaps that is the common challenge facing CTF designers, coaches, teachers, and anyone else trying to develop human capability in an increasingly AI-enabled world. We should absolutely use technology to help people learn more efficiently. At the same time, we need to be careful that efficiency does not eliminate the productive struggle through which much of the learning actually occurs.
Tomorrow, I’ll tell you about the one challenge this year that somehow resisted almost everything thrown at it. In the process, it became the longest-running challenge we have ever produced and taught me an entirely different lesson about what happens when people try too hard to solve a problem.
