When Data Gives the Wrong (Tennis) Solution

Using data to drive decision making is a well established organizational best practice. Over the past couple of years the rationale for decisions that the USTA has made to increase adult tournament participation has centered around the mantra that we must follow the data. As an engineer, I naturally gravitate to data-based decision making. However, I am not convinced that we are effectively doing that in tennis at the moment.

Storytime

During World War II, a lot of US bombers were getting shot down over Germany. The Center for Naval Analysis decided to conduct a data-based examination of the problem by examining the bullet holes and damage from the aircraft that returned after each mission. A clear pattern immediately emerged: The majority of the damage was to the wings and fuselage of the plane.

The initial engineering recommendation was to increase the armor under the wings and fuselage of the plane. The rationale is because that is where the most bullet holes were found. Unfortunately, there was a fundamental flaw in the analysis. (Do you see it?) The data set was missing information about the planes that had been shot down.

The researchers initially failed to recognize that the data they were studying was a map of where the plane could sustain a bullet and survive rather than a picture of where planes typically sustained bullet holes. The aircraft desperately needed reinforcement in the parts of the plane that never returned with damage. Those critical systems turned out to be the cockpit, engine, and certain parts of the tail.

This story is a great example of survivorship bias in action. In a nutshell, it exists any time there is an exclusive focus on people and initiatives that succeed without consideration of data from failure cases. The business world and media are full of examples of survivorship bias in action.

Survivorship Bias and Tennis

Declining participation in adult tournament tennis is now a crisis for the USTA. The Adult Competition Committee has been considering and implementing changes intended to stem or reverse the decline. There are a lot of smart people united by a common purpose engaged in the process. Everybody is trying to to increase participation in adult tournament tennis.

Recently I have been considering the potential impacts of survivorship bias in the data available to the USTA when examining the adult participation problem. There is frustration in the tournament player community at both the senior and NTRP levels that their opinions haven’t been considered or heard. Those important perspectives certainly need to be a part of the data set. However, they cannot be the exclusive source of data. That would be textbook survivorship bias.

From what I have personally been exposed to, the USTA doesn’t have good data for people that used to play tennis but exited the sport somewhere along the way. As an organization, the USTA produces programming and engagement mechanisms for its active members. It is unfortunate, because data from players in the tennis diaspora is essential for identifying potential opportunities to recover tournament participation at the highest levels.

The USTA seems to have marginally better data on the consumer preferences of the general population of new and prospective tennis players. That is where the idea that the consumer is time limited and has a short attention span originated. That data led the organization to roll out shortened format tennis at certain adult events in an attempt to attract new players. Somehow the idea that short formats might be appropriate for the membership at large gained traction.

Decisions versus Experiments

Human nature is a funny thing. When a person or a group makes a decision, they become attached to it. It sometimes becomes so interwoven with self-identity that even the idea that a decision might be bad can be threatening. Our hatred of being wrong is closely related to the fight or flight survival instinct.

One way to insulate ourselves against our own human nature is to create experiments rather than making decisions. A properly designed experiment can never reflect whether the designer was right or wrong. Experiments answer questions and provide valuable insight and data that drives the next round of questions.

Experiments are transitory and allow an organization to evaluate a lot of small things rather than placing big bets on large scale programs. It is an agile way of conducting business that is inherently less risky than broad initiatives. There is no such thing as a big mistake when the experimental approach is taken. In fact, there are no mistakes at all, only valuable lessons learned.

Culturally and organizationally, the USTA is structured for large long term programs rather than experiments. At one point early in his short tenure, outgoing USTA CEO Michael Dowse remarked that the organization needed to learn to move at internet speed. One way to move the culture in that direction is by constructing experiments rather than implementing large programs that everybody desperately hopes will succeed because there is no alternative.

Ask Better Questions

Experiments are essentially structured around answering questions. That means that the ability to frame out good questions that can be answered through experimentation is essential. Each exercise must involve mechanisms for identifying the data that is essential to evaluate the question as well as mechanisms to collect the data.

This week the National Women’s Tennis Organization (NWTO) leadership converges at the National Women’s Senior Clay Court Championships in Houston. There will be many discussions about the best ways to build and support senior tennis. As these conversations occur, survivorship bias and the need to ask better questions intended to to drive us toward experimentation will be front of mind for me.

I will be trying to ask better questions.

---

1. When data gives the wrong solution, Trevor Bragdon, September 7, 2017.
2. Survivorship Bias and the EMS, Glenn “Powder” Carlson President’s Message, Journal of Electromagnetic Dominance, November, 2021.