“Hello, world.” The six to eight weeks leading up to DEF CON are always a bit overwhelming for me. Between my day job, preparations for the conference, and maintaining the daily publishing tempo on this site, discretionary time evaporates. Every year, I fall almost completely off the grid in June and July. Personal emails go unanswered, text messages languish longer than they should, and anything that isn’t immediately pressing gets pushed somewhere into the indefinate future. If you have been waiting for a response from me, my communication lag times should improve considerably throughout the remainder of the year.
DEF CON 34 wrapped up in Las Vegas over the past weekend, which means I am gradually returning to the regularly scheduled programming of life. However, before we get completely back to tennis, this weekend’s “Unplugged” series is going to wander farther off the court than I normally allow this site to stray. I spent much of last weekend at DEF CON, where my day-job team is a significant content provider for the Aerospace Village. Over the next three days, I want to share a few observations from that experience that have somewhat relevant implications for how we learn, solve problems, and ultimately even play tennis.
DEF CON is one of the world’s largest hacker conferences, and its villages create smaller communities focused on particular areas of interest. The Aerospace Village brings together hackers, engineers, researchers, industry, government, and anyone else interested in the security of aviation and space systems. My involvement there is mostly associated with my professional life, although DEF CON has its own culture that tends to blur the boundaries between work, education, and play.
Anonymity has always been an important part of hacker culture, so many people at DEF CON operate under handles rather than their real names. Mine is “Synaptic Rodeo.” I make absolutely no effort to protect that supposedly secret identity. In fact, typing SynapticRodeo.com into a browser will currently deposit you right back here at Fiend at Court. Someday, SynapticRodeo.com will be an independent blog, but not until after I retire.
One of Synaptic Rodeo’s contributions to this year’s Aerospace Village was four challenges associated with an SR-71-themed Capture the Flag competition. I will explain CTFs in considerably more detail tomorrow. For today’s purposes, all that really matters is that participants solved puzzles to obtain hidden “flags” that could be submitted for prizes.
Those prizes provided considerable motivation. Our team produced a limited number of custom SR-71 badges that have become something of a hot commodity at the conference. DEF CON has an entire culture surrounding badges, particularly unusual ones that cannot simply be purchased. The only way to get one of ours was to be among the first to successfully solve an SR-71 challenge. We intentionally created scarcity, attached it to a difficult problem, and then turned thousands of hackers loose trying to obtain one.
My first challenge dropped on setup day during an annual DEF CON tradition colloquially known as “Merch LineCon.” People begin lining up extraordinarily early to enter the official merchandise area and will spend a substantial portion of the day waiting for the opportunity to buy authentic DEF CON shirts and other merchandise. I don’t entirely understand this phenomenon because I largely make my own shirts, but people really love their official DEF CON gear.
From the perspective of someone with a puzzle to distribute, all those people standing around create a remarkably captive audience. For the past two years, we have produced a flyer advertising the Aerospace Village with a 21-by-21 crossword puzzle on the back. That is essentially the same grid size as the one used by the New York Times for its Sunday crossword. We walked the merchandise line, handing out puzzles to give people something to do while they waited and to advertise the Village.
While we were working our way through the line, a young man ran up to me and asked whether it was permissible to use artificial intelligence to solve the challenge. The answer is yes because there is no practical alternative. We cannot detect whether contestants are using an LLM. Thus, prohibiting it would only disadvantage the cheaters willing to follow the rule. Anyone who ignored the restriction would have access to a powerful tool that everyone else had voluntarily agreed not to use.
At the same time, we don’t particularly want our CTF challenges reduced to a competition over who can get the puzzle into their LLM of choice the fastest.
The person who had asked about AI usage then demonstrated why this is becoming such an interesting problem. He had taken a photograph of the crossword with his phone and given it to an AI system. He showed me the result, including the LLM’s confident assertion that it had instantaneously solved the flag.
However, the flag was wrong.
More interestingly, there was no need to solve a single crossword clue to recognize that the answer was incorrect. The instructions on the flyer explicitly stated that the flag contained 32 characters. The AI-generated answer he showed me was significantly shorter. It should have been obvious to even the most casual observer that this was an AI hallucination.
When I later examined the flag submission logs, it was immediately apparent that the young man was far from alone. We received thousands of submissions containing the same incorrect flag. Presumably, many contestants were using AI systems that were converging on the same wrong answer. Some participants became so convinced that the AI-provided answer must be correct that they began asking us to check the verification bot when it rejected their submission.
That reality disturbs me. LLMs are going to get things wrong. Humans do too. Any tool capable of analyzing incomplete or ambiguous information will sometimes produce an incorrect result, and generative AI introduces its own particular failure modes. The more interesting issue was that highly intelligent people attending one of the world’s premier hacker conferences were accepting an answer that contradicted information sitting directly in front of them.
They actually had two independent pieces of evidence that something was wrong. The proposed flag didn’t satisfy the clearly stated 32-character requirement, and the official verification system rejected it. For at least some participants, neither was sufficient to overcome their confidence in the AI-generated response. Instead of reconsidering the answer, they began questioning whether our verification system was functioning correctly.
That experience gets closer to my growing concern about what generative AI may be doing to the way we think. I don’t believe using AI makes someone less intelligent, nor do I believe there is some particular virtue in refusing to use a tool that can make us more productive. I use generative AI myself. Pretending that these tools don’t exist is no more sensible than insisting that everyone perform arithmetic by hand because calculators made the process easier.
However, there is an important difference between using a tool to augment our thinking and allowing it to replace our judgment. The crossword incident demonstrated how easy it can be to cross that line without recognizing that it happened. The participants had everything they needed to evaluate the proposed answer themselves. The problem wasn’t access to information or an inability to reason through it. They simply accepted the machine’s conclusion without performing a very basic sanity check.
That distinction becomes particularly relevant within a CTF because obtaining the flag is only superficially the objective. The real value comes from figuring out how to get there. Participants experiment, research unfamiliar concepts, recognize patterns, follow blind alleys, reconsider assumptions, and gradually develop an understanding of the problem. The flag merely provides evidence that they successfully completed that process.
The SR-71 badge complicates that incentive. People understandably wanted the scarce physical object, and the fastest way to obtain one was to get to the flag. Generative AI offered an opportunity to dramatically shorten the trip. However, optimizing exclusively for the destination can eliminate much of the value that was supposed to come from making the journey.
There is a tennis lesson buried somewhere in all of this. Our sport is increasingly surrounded by technology that can analyze strokes, identify patterns, generate statistics, and provide recommendations that previously required considerable expertise and observation. I regard most of those capabilities as useful, particularly when they give recreational players access to information that would otherwise be unavailable to them.
The danger comes when we stop evaluating what those tools tell us. An application can identify a technical problem that isn’t actually a problem or recommend a tactical adjustment that makes little sense given the opponent standing across the net. The fact that an answer was generated through high-tech analysis does not relieve us of responsibility for deciding whether it makes sense.
That may be the lesson I took away from standing adjacent to the Merch Line watching AI attempt to solve my crossword. The unsettling part wasn’t that the machine got the answer wrong. It was watching capable people trust that answer when the evidence necessary to reject it was printed on the same piece of paper.
As AI becomes increasingly capable of thinking alongside us, learning how to use it will undoubtedly become an important skill. An equally important success factor will be remembering that the judgment about whether its answers make sense still belongs to us.
