NUMBERS RULE YOUR WORLD.

THE HIDDEN INFLUENCE OF PROBABILITY AND STATISTICS ON EVERYTHING YOU DO.

by KAISER FUNG.

Acknowledgments.

I would like to acknowledge the guidance and a.s.sistance of Grace Freedson, Michele Paige, Micah Burch, Kate Johnson, Steven Tuntono, Beth McFadden, Talbot Katz, and my editors, John Aherne and Joseph Berkowitz. My two sisters and brother made invaluable contributions as my most plain-spoken critics.

In addition, throughout this project, I was inspired by fans of my Junk Charts blog, www.junkcharts.typepad.com.

Introduction.

This is not another book about "d.a.m.ned lies and statistics." That evergreen topic has inspired masterworks from Darrell Huff, John Allen Paulos, Ed Tufte, and Howard Wainer, among others. From the manipulative politician to the blundering a.n.a.lyst, from the amateur economist to the hard-selling advertiser, we have endless examples of what can go wrong when numbers are misused. Cherry-picking, oversimplifying, obfuscating-we have seen them all. This book takes a different direction, a positive position: I am interested in what happens when things go right, which is to say, what happens when numbers don't don't lie. lie.

The More We Know We Don't Know What will we learn from Bernie Madoff, the New Yorkbased fund managerswindler who impoverished an exclusive club of well-to-do patrons over three decades until he confessed in 2008? Or from the Enron executives whose make-believe accounting wiped out the retirement savings of thousands of employees? Perhaps we ought to know why the reams of financial data, printed statements, and official filings yielded few clues to the investors, auditors, and regulators who fell for the deception.

What will we learn from the Vioxx debacle in which the Food and Drug Administration conceded, five years after blessing its initial release, that the drug had caused ten thousand heart attacks? Perhaps we ought to know why widely available health and medical information and greater scale and sophistication of clinical trials did not spare Vioxx inventor Merck, doctors, or patients from overlooking the deadly side effects.

We ought also to ask why, despite having access to torrents of stock data and company reports, most of us have not made a killing in the stock market. Despite tallying up the nutritional information of every can and every packet of food, most of us have not achieved the hoped-for bodily downsizing. Despite heavy investment in information technology, flight delays and traffic jams continue to get worse. Despite detailed records of our shopping behavior, many companies have but the slightest clue when we call their service centers. Despite failing to arrest cancer in patients during large-scale clinical trials, beta-carotene and vitamin pills keep flying off the pharmacy shelves.

These examples reveal the unpleasant surprise that the modern obsession with measurement has made us none the wiser. We collect, store, process, and a.n.a.lyze more information than ever before-but to what end? Aristotle's wisdom has never been more relevant than it is today: the more we know, the more we know we don't know.

Stories of a Positive Nature We begin to overcome these failures by examining positive examples of how enterprising people are making sensible use of the new information to better our world. In the next five chapters, you will meet engineers who keep the traffic flowing on Minnesota highways, disease detectives who warn us about unsafe foods, actuaries who calculate how much Floridians must pay to insure homes against hurricanes, educators who strive to make standardized tests like the SAT fair, lab technicians who scrutinize blood samples from elite athletes, data miners who think they can detect our lies, lottery operators who face evidence of fraud, Walt Disney scientists who devise ever-clever ways to shorten queues, mathematicians whose ideas have set off the explosion of consumer credit, and researchers who offer the best tips for air travel.

These ten portraits feature some special men and women whose work is rarely celebrated openly. The reason for this neglect is that their achievement is not of invention, for which we shower awards and accolades, but of adaptation, of refinement, of salesmanship, and of perseverance. Their expertise is applied science.

The Statistical Way of Thinking For me, these ten stories ultimately merge into one: all of these exemplary scientists rely on the statistical way of thinking, as distinct from everyday thinking. I organize the stories into five pairs, each dealing with an essential statistical principle.

What is so unconventional about the statistical way of thinking?

First, statisticians do not care much for the popular concept of the statistical average; instead, they fixate on any deviation from the average. They worry about how large these variations are, how frequently they occur, and why they exist. In Chapter 1 Chapter 1, the experts studying waiting lines explain why we should worry more about the variability of waiting time than about its average. Highway engineers in Minnesota tell us why their favorite tactic to reduce congestion is a technology that forces commuters to wait more, while Disney engineers make the case that the most effective tool to reduce wait times does not actually reduce average wait times.

Second, variability does not need to be explained by reasonable causes, despite our natural desire for a rational explanation of everything; statisticians are frequently just as happy to pore over patterns of correlation. In Chapter 2 Chapter 2, we compare and contrast these two modes of statistical modeling by trailing disease detectives on the hunt for tainted spinach (causal models) and by prying open the black box that produces credit scores (correlational models). Surprisingly, these pract.i.tioners freely admit that their models are "wrong" in the sense that they do not perfectly describe the world around us; we explore how they justify what they do.

Third, statisticians are constantly looking out for missed nuances: a statistical average for all groups may well hide vital differences that exist between these groups. Ignoring group differences when they are present frequently portends inequitable treatment. The typical way of defining groups, such as by race, gender, or income, is often found wanting. In Chapter 3 Chapter 3, we evaluate the mixed consequences that occur when the insurance industry adjusts prices to reflect the difference in the amount of exposure to hurricanes between coastal and inland properties, as well as what happens when designers of standardized tests attempt to eliminate the gap in performance between black and white students.

Fourth, decisions based on statistics can be calibrated to strike a balance between two types of errors. Predictably, decision makers have an incentive to focus exclusively on minimizing any mistake that could bring about public humiliation, but statisticians point out that because of this bias, their decisions will aggravate other errors, which are unnoticed but serious. We use this framework in Chapter 4 Chapter 4 to explain why automated data-mining technologies cannot identify terrorist plots without inflicting unacceptable collateral damage, and why the steroid-testing laboratories are ineffective at catching most of the cheating athletes. to explain why automated data-mining technologies cannot identify terrorist plots without inflicting unacceptable collateral damage, and why the steroid-testing laboratories are ineffective at catching most of the cheating athletes.

Finally, statisticians follow a specific protocol known as statistical testing when deciding whether the evidence fits the crime, so to speak. Unlike some of us, they don't believe in miracles. In other words, if the most unusual coincidence must be contrived to explain the inexplicable, they prefer leaving the crime unsolved. In Chapter 5 Chapter 5, we see how this powerful tool was used to uncover extensive fraud in a Canadian state lottery and to dispel myths behind the fear of flying.

These five principles are central to statistical thinking. After reading this book, you too can use them to make better decisions.

The Applied Scientist at Work These stories take a shape that reflects my own experience as a pract.i.tioner of business statistics. They bring out aspects of the applied scientist's work that differ substantively from that of the pure or theoretical scientist.

All the examples involve decisions that affect our lives in one way or another, whether through public policies, business strategies, or personal choices. Whereas the pure scientist is chiefly concerned with "what's new," applied work must deal with "how high," as in "how high would profits go?" or "how high would the polls go?" In addition to purely technical yardsticks, applied scientists have goals that are societal, as with the Minnesota highway engineers; or psychological, as with the Disney queue managers; or financial, as with hurricane insurers and loan officers.

The pursuit of pure science is rarely limited by time; as an extreme example, mathematician Andrew Wiles meticulously constructed his proof of Fermat's last theorem over seven years. Such luxury is not afforded the applied scientist, who must deliver a best effort within a finite time limit, typically in the order of weeks or months. External factors, even the life cycle of green produce or the pipeline of drug innovations, may dictate the constraint on time. What use would it be to discover the cause of an E. coli E. coli outbreak the day after the outbreak dies down? What is the point of developing a test for a designer steroid after dozens of athletes have already gained unfair advantage from using it? outbreak the day after the outbreak dies down? What is the point of developing a test for a designer steroid after dozens of athletes have already gained unfair advantage from using it?

Some of the most elegant achievements in pure science result from judiciously choosing a set of simplifying a.s.sumptions; the applied scientist adapts these results to the real world by noticing and then coping with inconvenient details. If you have read the writings of Na.s.sim Taleb, you will recognize the bell curve as one such simplification that demands refinement in certain situations. Another example, considered in Chapter 3 Chapter 3, is lumping together distinct groups of people when they should be treated differently.

Successful applied scientists develop a feel for the decision-making process: they know the key influencers, they grasp their individual ways of thinking, they comprehend their motivations, and they antic.i.p.ate sources of conflict. Crucially, they repackage their logic-laced messages to impress their ideas upon those who are more comfortable with intuition or emotion than with evidence. Because understanding the context is so valuable to the applied scientist's work, I have included a wealth of details in all of the stories.

To sum up, applied science has measures of success distinct from those used in theoretical science. For instance, Google recognized this distinction by rolling out its famous "20 percent" time policy, which allows its engineers to split their week between pure science projects of their choosing and applied projects (with an 80 percent emphasis on the latter!).

More And there is something extra for those who want more. The Conclusion of this book serves a dual purpose of consolidation and extension. While summarizing the statistical way of thinking, I introduce the relevant technical language in case you should want to cross-reference a more conventional book. To ill.u.s.trate how universal these statistical principles are, I revisit each concept in a new light, harnessing a different story from the one originally selected. Finally, the Notes section contains further remarks as well as my main sources. A complete bibliography is available at the book's link on my website, www.junkcharts.typepad.com.

Numbers already rule your world. And you must not be in the dark about this fact. See how some applied scientists use statistical thinking to make our lives better. You will be amazed how you can use numbers to make everyday decisions in your own life.

1.

Fast Pa.s.ses / Slow Merges The Discontent of Being Averaged Meter mystery If no one likes, why obey?

One car per green, please -HAIKU ABOUT THE M MINNEAPOLISST. P PAUL COMMUTE BY READER OF THE R ROADGUY BLOG.

Heimlich's Chew Chew Train Good film, big buildup, nice queue Twenty-second ride -HAIKU ABOUT D DISNEY BY A ANONYMOUS In early 2008, James Fallows, longtime correspondent at The Atlantic The Atlantic, published an eye-popping piece about America's runaway trade deficit with China. Fallows explained how the Chinese people were propping up Americans' standard of living. The highbrow journal has rarely created buzz on the Internet, but this article beat the odds, thanks to Netizens who sc.r.a.pped Fallows's original t.i.tle ("The $1.4 Trillion Question") and renamed the article "Average American Owes Average Chinese $4,000." In three months, Internet readers rewarded the piece with more than 1,600 "diggs," or positive responses, which is the high-tech way of singing praise. Evidently, the new headline caught fire. Our brains cannot comfortably process astronomical numbers such as $1.4 trillion, but we can handle $4,000 per person with ease. Simply put, we like large numbers averaged averaged.

The statistical average is the greatest invention to have eluded popular acclaim. Everything has been averaged by someone, somewhere. We average people ("average Joe") and animals ("the average bear"). Inanimate things are averaged: to wit, after the terrorist attacks of September 11, 2001, a security dispatch demonstrated how to "weaponize the average water cooler." Economic processes are averaged, as when a market observer in early 2008 proclaimed "the new hope: an average recession," presumably predicting a shallow one that would depart with haste. Even actions cannot escape: when Barack Obama's lawyer interjected on a Clinton conference call during the heated Democratic primary elections of 2008, the media labeled the occasion "not your average conference call."

Can rare items be averaged? You bet. Forbes Forbes magazine told us, "The average billionaire [in 2007] is 62 years old." Surely no one averages uncountable things, you think. Not so quick; the U.S. Census Bureau has devised a methodology for averaging time: on an "average day" in 2006, U.S. residents slept 8.6 hours, worked 3.8 hours, and spent 5.1 hours doing leisure and sporting activities. It is a near impossibility to find something that has not been averaged. So pervasive is the idea that we a.s.sume it to be inborn and not learned, nor in need of inventing. magazine told us, "The average billionaire [in 2007] is 62 years old." Surely no one averages uncountable things, you think. Not so quick; the U.S. Census Bureau has devised a methodology for averaging time: on an "average day" in 2006, U.S. residents slept 8.6 hours, worked 3.8 hours, and spent 5.1 hours doing leisure and sporting activities. It is a near impossibility to find something that has not been averaged. So pervasive is the idea that we a.s.sume it to be inborn and not learned, nor in need of inventing.

Now picture a world without averages. Imagine having the average child, the average bear, and the average such-and-so-forth punched out of our lexicon. We are dumbfounded to learn that such a world did exist once, before a Belgian statistician, Adolphe Quetelet, invented the "average man" (l'homme moyen) in 1831. Who would have thought: such a commonplace idea is younger than the U.S. Const.i.tution!

Before Quetelet, no one had entertained the import of statistical thinking to the social sciences. Up until that time, statistics and probability fascinated only the astronomers who decoded celestial phenomena and the mathematicians who a.n.a.lyzed gambling games. Quetelet himself was first a distinguished astronomer, the founding director of the Brussels Observatory. It was in midlife that he set the ambitious agenda to appropriate scientific techniques to examine the social milieu. He placed the average man at the center of the subject he named "social physics." While the actual methods of a.n.a.lysis used by Quetelet would strike modern eyes as hardly impressive, historians have, at long last, recognized his impact on the instruments of social science research as nothing short of revolutionary. In particular, his inquiry into what made an able army conscript earned the admiration of Florence Nightingale (it is little known that the famous nurse was a superb statistician who became an honorary member of the American Statistical a.s.sociation in 1874). In this body of work also lay the origin of the body ma.s.s index (BMI), sometimes called the Quetelet index, still used by doctors today to diagnose overweight and underweight conditions.

Since the concept of the average man has been so firmly ingrained into our consciousness, we sometimes fail to appreciate how revolutionary Quetelet really was. The average man was literally an invention, for the average anything did not, and does not, physically exist. We can describe it, but we cannot place it. We know it but have never met it. Where does one find the "average Joe"? Which "average bear" can Yogi Bear outsmart? Which call is the "average" conference call? Which day is the "average" day?

Yet this monumental invention constantly tempts us to confuse the imaginary with the real. Thus, when Fallows calculated an average of $4,000 debt to China per American, he implicitly placed all Americans on equal footing, spreading $1.4 trillion evenly among the population, replacing 300 million individuals with 300 million clones of the imaginary average Joe. (Incidentally, the Netizens mistakenly fabricated only 300 million Chinese clones, rhetorically wiping out three-quarters of China's 1.3 billion people. The correct math should have found the average Chinese lending $1,000 to America.) Averaging stamps out diversity, reducing anything to its simplest terms. In so doing, we run the risk of oversimplifying, of forgetting the variations around the average.

Hitching one's attention to these variations rather than the average is a sure sign of maturity in statistical thinking. One can, in fact, define define statistics as the study of the nature of variability. How much do things change? How large are these variations? What causes them? Quetelet was one of the first to pursue such themes. His average man was not one individual but many; his goal, to contrast different types of average individuals. For him, computing averages was a means of measuring diversity; averaging was never intended to be the end itself. The BMI (Quetelet index), for good measure, serves to identify individuals who are statistics as the study of the nature of variability. How much do things change? How large are these variations? What causes them? Quetelet was one of the first to pursue such themes. His average man was not one individual but many; his goal, to contrast different types of average individuals. For him, computing averages was a means of measuring diversity; averaging was never intended to be the end itself. The BMI (Quetelet index), for good measure, serves to identify individuals who are not not average, and for that, one must first decide what the average is. average, and for that, one must first decide what the average is.

To this day, statisticians have followed Quetelet's lead, and in this chapter, we shall explore how some of them use statistical thinking to battle two great inconveniences in modern living: the hour-long commute to and from work and the hour-long wait to get on a theme park ride. A reasonable person, when trapped in traffic or stuck in a long queue, will suspect that whoever was in charge of planning must have fallen asleep on the job. To see why this reaction misplaces the blame, we need to know a little about the statistics of averages. Working with engineers and psychologists, statisticians are applying this knowledge to save us waiting time.

To label Dr. Edward Waller and Dr. Yvette Bendeck Disney World die-hards would be an understatement. On October 20, 2007, they toured every last open attraction in the Magic Kingdom in just under thirteen hours. That meant fifty rides, shows, parades, and live performances. Buzz Lightyear's s.p.a.ce Ranger Spin, Barnstormer at Goofy's Wiseacre Farm, Beauty and the Beast-Live on Stage, Splash Mountain, Mad Tea Party, Many Adventures of Winnie the Pooh, you name it-everything in the park! Nice work if you can manage it, no? Disney buffs know this to be a mission impossible; they feel lucky to visit four major rides on a busy day, not to mention the nonstop walking required within the hundred acres of park area. Waller and Bendeck had help from Len Testa, who devised the Ultimate Magic Kingdom Touring Plan. Testa's plan lays out precise directions for reaching every attraction in the shortest time possible. He warns unsuspecting novices that it "sacrifices virtually all of your personal comfort."

Len Testa is a thirty-something computer programmer from Greensboro, North Carolina. As the patron saint of disgruntled Disney theme parkgoers worldwide, he brought the gift of touring plans, which prescribe routes that guide patrons through a sequence of attractions in the shortest time possible. While the Ultimate Plan grabs attention, Testa creates touring plans for just about every need: for small kids, families, tweens, active seniors, grandparents with small children, and so on. He is mainly looking after rabid Disney fans, ones who are the most loyal-and easily the most demanding-customers. Sampling their typically breathless trip reports, posted on fan websites or relayed to journalists, one frequently comes across affectionate gripes like these: "Going to Disneyland in the summer months is kind of like cruising to the Bahamas during hurricane season. You're just asking for it.""You haven't lived until you've stampeded to s.p.a.ce Mountain as the opening rope drops, alongside thousands of stroller-wielding soccer moms at a full run.""When those gates spring open at 8 A.M A.M., the weak and the semi-comatose will be left in the dust.""We felt we spent more time in lines than on rides-the fact is, we did! When a wait in the line is ninety minutes and the ride is only five minutes, you have to question your sanity!!""I've never really forgiven my brother for that one time he slowed us down with an untimely bathroom break at Disney's Epcot Center five years ago."

These park-goers have plenty of company. Disney's own exit polls reveal long lines as the top source of customer unhappiness. Industry veterans say the average guest dawdles away three to four hours in queues during a visit lasting eight to nine hours; that's one minute of standing around out of every two to three minutes inside the park! Amus.e.m.e.nt Business Amus.e.m.e.nt Business estimated that the national average for wait time at major attractions in a theme park during the summer was sixty minutes-after which patrons get to spend two minutes on the ride. Since a family of four can spend $1,000 or more in a single trip, it is no wonder why some guests are irritated by seemingly interminable lines. estimated that the national average for wait time at major attractions in a theme park during the summer was sixty minutes-after which patrons get to spend two minutes on the ride. Since a family of four can spend $1,000 or more in a single trip, it is no wonder why some guests are irritated by seemingly interminable lines.

These trip reports leave vivid images of heroic maneuvers to avoid lines. A suitable att.i.tude is required: "When I'm in the parks, I'm a Daddy on a mission. . . . In the course of the afternoon, I'll go from one end of the park to the other and ride more rides, wait less in lines, and see more shows and parades than many other park patrons, with or without kids."

So are small sacrifices . . .

"We manage to avoid long lines with an occasional early morning, and hitting popular attractions during parades, mealtimes, and late evenings."

. . . and knowing how to play the system . . .

"The mother behind me told me that they had waited three hours to ride Dumbo during their last visit. [This time,] she took advantage of early admission to let her kid ride three times in a row with no waiting."

. . . and sweet-talking teachers into granting special permission . . .

"Taking your kids out of school [to go to Disney]. Is it worth it? Yes!"

. . . and spotting opportunities that others give . . .

"It does rain in Florida, especially during summer afternoons. The good news is that this tends to scare off some people. My advice: Buy bright-yellow ponchos for $5 each from any of the gift shops. Then keep those kids walking."

. . . while always adapting tactics: "We are starting to think that reverse-reverse psychology might work: Disney opens one park earlier for all their guests so all the guests go to that park. . . . [Everyone else avoids that park, and] therefore we can go to that park because people think it is going to be packed and they avoid it."

Queues happen when demand exceeds capacity. Most large rides can accommodate 1,000 to 2,000 guests per hour; lines form if patrons arrive at a higher rate. If Disney accurately antic.i.p.ated demand, could it not build sufficient capacity? Did the appearance of long lines reflect negligent design? Surprisingly, the answer to both questions is no. The real culprit is not bad design but variability variability. Disney constructs each theme park to satisfy the "design day," typically up to the ninetieth-percentile level of demand, which means, in theory, on nine out of ten days, the park should have leftover capacity. In reality, patrons report long lines pretty much any day of the year.

Worse, statisticians are certain that queues would persist even if Disney had designed for the busiest day of the year. To understand this piece of anti-intuition, we must realize that the busiest-day demand merely conveys the average park attendance, and this number ignores the uneven distribution of patrons, say, from one attraction to another or from one hour to the next. Even if Disney correctly predicted the total number of patrons riding Dumbo on the peak day (which itself would have been a tough a.s.signment), a line would materialize unavoidably because the patrons would appear irregularly during the day, while Dumbo's capacity does not change. Statisticians tell us that it is the variable pattern of when guests arrive, not the average number of arrivals, that produces queues on all but the peak days. Capacity planning can cope with large and static demand, not fluctuating demand. (A theme park that guarantees no lines would require capacity wildly disproportionate to demand, ensuring substantial idle time and unviable economics.) The engineers who figured out these secrets are hailed as heroes by the Disney die-hards, and they work for the Imagineering division, based in several nondescript buildings in Glendale, California, near Los Angeles. They also design new rides, handling not only the thrill factor but also operations management. In the realm of waiting lines, scientists rely heavily on computer simulations as the mathematics of queuing are super complex and frequently irreducible to neat formulas. Think of simulations as turbocharged what-if a.n.a.lyses, run by farms of computers. Thousands, perhaps millions, of scenarios are investigated, covering possible patterns of arrival and subsequent movement of guests around the park. The summary of these scenarios yields reams of statistics, such as the likelihood that the Dumbo ride will reach 95 percent of its capacity on any given day. This creative approach to working around intractable mathematical problems was invented by the Manhattan Project team while building the atomic bomb and also forms the basis of Moneyball Moneyball statistics featured in Michael Lewis's account of how the Oakland Athletics outwitted powerhouse baseball teams with much bigger budgets. statistics featured in Michael Lewis's account of how the Oakland Athletics outwitted powerhouse baseball teams with much bigger budgets.

Wouldn't you know it? The same script plays itself out on our highways: the bane of commuters is not so much long average trip time as it is variable trip time. The statistics paint a harsh reality indeed: the average American worker spent 25.5 minutes traveling to work in 2006, and in addition, more than ten million people endured commutes of over an hour. In total, traffic delays cost $63 billion a year while wasting 2.3 billion gallons of fuel. But these scary numbers miss the mark. Just ask the pileup of readers who sent grievances to Minneapolis Star Tribune Minneapolis Star Tribune. Those truly put off by a long trip to work every day either practice avoidance . . .

"I chose to live in Minneapolis for transportation-related reasons: great access to transit and reverse commutes. . . . If people chose to live in Eden Prairie [an edge city southwest of Minneapolis], then I don't have much sympathy for their complaints about traffic problems."

. . . or have made peace with the inevitable: "Every day, no matter how much traffic there is, it slows down right by McKnight [Road near Maplewood on I-94]. . . . There have been times when we have stopped and had a c.o.ke somewhere because it gets so miserable sitting on the highway."

Commuters know what they are in for, and they take charge of the situation.

If average trip time is not the source of bother, what is? Julie Cross, another Star Tribune Star Tribune reader, articulated this well: reader, articulated this well: "Picking the fastest route for my morning commute from Apple Valley is a daily gamble. Should I chance Cedar Avenue, hoping to hit free-flowing traffic for a 5-minute trip to work in Eagan? Or would Cedar be stop-and-go, making the reliable 10-minute trip on Interstate Hwy. 35E the better bet?"

Pay attention when Cross used the word reliable reliable. She knew well the average length of her trip to work; what troubled her was the larger variability, and thus unreliability, of the Cedar Avenue option. The highway route required ten minutes, with hardly any day-today variation. Now, if the Cedar Avenue option took five minutes without fail, Julie would never consider taking I-35E. Conversely, if the Cedar Avenue stretch took fifteen minutes without fail, Julie would always take I-35E. The only reason Julie Cross anted up each morning was that the Cedar Avenue route might take less time, even though she knew on average it would take longer. In general, if but one of two routes has variable trip times, then the bet is on. (See Figure 1-1 Figure 1-1.) It is tempting to think proper trip planning will beat back any variability in travel time. But just like Disney's problem of fluctuating demand, this beast proves hard to slay. Jim Foti, who pens the Roadguy column at the Star Tribune Star Tribune, learned the lesson firsthand: Figure 1-1 Julie Cross's Morning Commute Problem: Impact of Variable Trip Times Julie Cross's Morning Commute Problem: Impact of Variable Trip Times [image]

"Last week, when Roadguy had his radio gig, he had to head out to Eden Prairie during the evening rush hour. Scarred by memories of backups at the Hwy. 212 exit, he allowed himself plenty of extra time. But the drive, right around 5 P.M P.M., turned out to be as easy and graceful as a computerized animation, and Roadguy reached his destination nearly 20 minutes early."

Drivers find themselves in a no-win situation: arriving twenty minutes early leads to wasted time and even unanswered doorbells, while arriving twenty minutes late spoils other people's moods, wastes their time, and sometimes causes missed connections. This double whammy is on top of any extra time set aside in antic.i.p.ation of traffic. And once again, variability is the culprit. Jim's strategy would have produced fruit if every trip were like the average trip. In reality, a trip that takes fifteen minutes on average may take only ten minutes on good days but eat up thirty minutes on rubbernecking days. If Jim allows fifteen minutes, he will arrive too early most of the time, and too late some of the time. On few days will he finish the trip in exactly fifteen minutes. In short, variable traffic conditions mess up our well-laid schedules, and that ought to upset us more than the average journey time.

After spending decades fighting average congestion levels, traffic engineers at state departments of transportation have come around to the paramount issue of variable trip times. What is the source of such variability? Cambridge Systematics, an influential transportation consultancy, has estimated that bottlenecks, such as three lanes dropping to two and poorly designed interchanges, account for only 40 percent of congestion delay in the United States. Another 40 percent is due to accidents and bad weather. Choke points on highways restrict capacity and cause predictable average delay, while road accidents and weather-related incidents induce variability around the average. They can create extraordinary gridlocks, like this one . . .

A truck carrying 45,000 pounds of sunflower seeds tipped over around 5:45 A.M A.M. and blocked the left two lanes of the freeway for more than 2 hours. Motorists encountered delays of 30 to 45 minutes as they tried to navigate past the scene.

. . . and this one: A dusting of snow Monday was enough to snarl traffic on the freeways. . . . From 5 A.M A.M. to just after 5 P.M P.M., 110 crashes and 20 rollovers were reported on Minnesota roads. . . . A state transportation official had just two words for drivers: Slow down!

The unpredictability of such events makes freeway congestion unavoidable, and delay on some days considerably worse than the average. Not surprisingly, building more roads is the wrong medicine: supplemental capacity can eliminate bottlenecks, at least in the short term, but it does not directly affect reliability. Worse, many transportation experts, including economist Anthony Downs, warn that we cannot build our way out of congestion. In his book Still Stuck in Traffic Still Stuck in Traffic, Downs elegantly espouses his principle of triple convergence, which postulates that as soon as new capacity gets built, commuters shift their behavior in three notable ways to crowd out any planned benefits: those who previously used local roads decide to switch back to freeways, those who previously altered travel times reverse that decision, and those who previously elected to take public transit return to driving. Thus, new thinking is needed.

The Minnesota Department of Transportation (Mn/DOT) has championed an advanced technique called "ramp metering." Ramp meters are stop-go lights installed on highway entrances to regulate the inflow of traffic. "One car per green" is the familiar mantra. Detectors measure the flow of traffic on the freeway; when the flow exceeds 3,900 vehicles per hour, the freeway is deemed "full," and the meters are turned on to hold back cars at the on-ramp. Another detector counts the backup on the ramp; when the backup threatens to spill over to the local area, the metering speed increases to dump traffic onto the freeway faster. According to an operations specialist with Mn/DOT, these controls temporarily delay the onset of congestion on the freeway.

Ramp metering has compiled an impressive record of success in several metropolitan areas. For example, Seattle saw traffic volume swell by 74 percent even as average journey time was halved during peak hours. Not only were more trips completed, but also less time was spent per trip! So incredible was this double bonus that researchers at the University of California, Berkeley, called it "the freeway congestion paradox." Typically, as more vehicles pile onto the same stretch of highway, inducing congestion, we expect travel times to suffer; conversely, with traffic moving more slowly, the volume of vehicles should decline. Such laws of physics seem immutable. How does ramp metering make its magic?

To unravel the paradox, we must first understand why the emergence of congestion is so feared. Statistical evidence has revealed that once traffic starts to pile up, the average speed of vehicles plunges, and oddly, the carrying capacity of the road degrades below its planned level. In one study, as average speed dropped from 60 to 15 miles per hour during peak hours, traffic flow dropped from 2,000 to 1,200 vehicles per hour. This disturbing situation seemed as illogical as if a restaurateur decided to lay off 40 percent of her staff during busy Friday nights, when one would have thought it crucial to run the kitchen at maximum efficiency. In response to the unsettling discovery, the Berkeley researchers recommended a policy of operating freeways at their optimal speeds, typically 50 to 70 miles per hour, for as much time as possible. In ramp metering, they found an ideal way to throttle the influx of vehicles, a means to maintain the condition of the highway just below congestion level. Its purpose is stamping out variability of traffic speed. The gain in terms of reduced travel time and increased traffic flow "far exceeds any improvements that could be achieved by constructing more freeway lanes."

And there is more to ramp meters. They also mitigate the variability of travel time. Two effects are at play here. First, metering ramps regulate speed, which leads directly to more reliable journey times. Second, the rule stipulating one car per green light s.p.a.ces out vehicles as they enter the highway, and this dramatically brings down accident rates. Fewer accidents mean less congestion and fewer unexpected slowdowns. This is nowhere more widely felt than in the North Star State, birthplace of the notorious "Minnesota Merge." Jim Foti's readers again provided the color commentary: "This is a personal peeve of mine, witnessing people slowly, gradually accelerate on to the freeway so that they just barely make it to the speed limit right at the moment they merge on to the freeway; this causes a conga line of people behind that first merger who end up arriving on to the freeway at 50-45-40-35-30 mph. Slow accelerating mergers can be almost as bad as people who stop at the bottom of the ramps waiting for an opening.""[Where ramps are not metered,] groups of two, three, or more cars [are] TAILGATING EACH OTHER ON THE RAMPS, try to merge as a group. What the heck is wrong with their heads? Unless there are four or five car lengths between cars on the expressway already, how do they expect to keep cars in the right lane from braking?""The reason folks don't use their turn signal on the highway is that most of the time there is a jerk in the other lane who closes the gap so you can't get in. . . . [The person above] is 100 percent right. I no longer use my turn signal on the highway because of the gapclosers."

One reason why volume declines in concert with reducing speed is that vehicles get too close together for comfort when roads become congested. Some drivers are then p.r.o.ne to braking frequently, and in so doing, they release "shock waves" of reactive deceleration upstream, further disrupting the flow of traffic. Thus, the leaching of capacity during rush hours results from incompetence, impatience, aggression, and self-preservation.

Mn/DOT was one of the pioneers in ramp metering, installing its first meters in 1969. During a "war on congestion" in the 1990s, the network grew sixfold to become the densest in the nation, encompa.s.sing two-thirds, or 210 miles, of the freeway system in the Twin Cities metropolitan area. Mn/DOT is also the most aggressive among its peers at holding back ramp traffic during peak hours in order to deliver a reliable flow on the freeway. Industry experts regard Minnesota's system of 430 ramp meters as a national model.

At their core, both Disney and Mn/DOT face the scourge of congestion, and they have both come to realize that no amount of capacity expansion can banish the problem of variability due to fluctuating patron arrivals or unpredictable on-road incidents. Besides, expansion projects take time, money, and frequently political will as well as sacrifice from current users for a future, common good. Growing capacity is a necessary but insufficient solution. Statisticians believe that a sound transportation policy should emphasize optimally utilizing available capacity. Finding new ways to do so costs significantly less than constructing new highways and yields quicker returns. Ramp metering is one such solution. Disney managers have concluded that measures to optimize operations, while effective, also are not sufficient; they have gone one step further than freeway engineers. The crown jewel in Disney's operating manual is perception management.

A body of scholarly research supports the view that crowd control is much more than a mathematical problem or an engineering puzzle; it has a human, psychological, touchy-feely dimension. A key tenet of this research-that perceived waiting time does not equal actual waiting time-has been demonstrated in multiple studies. Mirrors in elevator lobbies, for example, distort people's sense of the amount of waiting time; we tend not to count time spent looking at our reflection as waiting time. Accordingly, Disney engineers, or "Imagineers," devote a lot of effort to shaping patrons' perception of waiting times. By contrast, engineering solutions, including ramp metering, tend to target reductions in actual actual waiting times; these efforts may fail because people misjudge how much time they have stood in lines or stalled their cars. waiting times; these efforts may fail because people misjudge how much time they have stood in lines or stalled their cars.

Over the years, Disney has perfected the magic of managing perceptions. Take a walk around the park, and you cannot fail to see their handiwork. The waiting area of Expedition Everest, for instance, is decorated as a Nepalese village, with artifacts and flora brought back from the Himalayas; before getting on the roller-coaster, patrons weave through a yeti museum and encounter cryptic messages that generate excitement. At other sites, when lines get terribly long, "Streetmosphere" performers playing Hollywood characters mill around to entertain guests. Signs show estimated waiting times that "intentionally turn out to be longer than the actual time waited," according to Bruce Laval, a former Disney executive. The next time you telephone customer service and hear that computer voice announcing, "Your expected wait time is five minutes," contrast how you feel if a customer service rep picks up the call after two minutes with your mental state if you are still on hold after eight minutes. Such is the power of this cla.s.sic strategy of underpromising and overdelivering. These and other designs suggest a briskly moving line or divert attention away from the queue.

The superstar of the Disney queue management effort is Fast-Pa.s.s, the proprietary "virtual" reservation system launched in 2000. Arriving at any of the major attractions, guests can either join the "standby" line and wait then and there, or opt to pick up FastPa.s.ses, which ent.i.tle them to return at a designated later time and join an express lane. Since the FastPa.s.s lane always clears at a much higher rate than the standby line, the typical wait will be five minutes or less when FastPa.s.s holders resurface during the prea.s.signed time. To aid guests in their decision, Disney posts the estimated wait time for those choosing the standby line, juxtaposed with the FastPa.s.s return time. Testimony from satisfied patrons points to the unmistakable success of this concept. One a.n.a.lytically minded fan, Allan Jayne Jr., demonstrated why: "How effective is FastPa.s.s? Very. . . . Let's say that FastPa.s.s forced the regular ('standby line') riders to wait on average 1 hours each instead of 1 hour while FastPa.s.s riders don't wait at all. So we have 9,000 people who did not spend any time waiting and 3,000 riders who waited an average of 1 hours each for a total wait of 4,500 hours. That is about six months of waiting compared with 16 months without FastPa.s.s [all 12,000 riders waiting 1 hour each]. Thus FastPa.s.s saved ten months of standing in line!"

Satisfied guests are eager to pa.s.s along their wisdom, as Julie Neal did on her blog: How to Get the Most from FastPa.s.s1. Designate someone as your FastPa.s.s supervisor. This person will hold all your park tickets, go off to get FastPa.s.ses for your entire party throughout the day, and watch the time. h.e.l.lo, Dad?2. Always hold at least one FastPa.s.s, so you're always "on the clock" for at least one attraction. Get one when you get in the park, then others as often as possible throughout the day.3. Don't sweat it if you miss the return time. Disney rarely enforces it, as long as you use your ticket the same day it was issued.4. Use the service for every FastPa.s.s attraction except those you'll be riding before 10 A.M. A.M. or very late at night. or very late at night.

Clearly, FastPa.s.s users love the product-but how much waiting time can they save? Amazingly, the answer is none none; they spend the same amount of time waiting for popular rides with or without FastPa.s.s! It is mistaken to think that FastPa.s.s eliminates waiting, as the above quotation suggested; it is just that instead of standing in line and in the elements, patrons are set free to indulge in other activities, whether on less popular rides or in restaurants, bathrooms, hotel beds, spas, or shops. The time in queue, which is the lag between arriving at the ride to pick up a FastPa.s.s ticket and actually getting on the ride, may in fact be even longer than before. Given that the attractions have the same capacities with or without FastPa.s.ses, it is just not possible to accommodate more guests by merely introducing a reservation system. So Disney confirms yet again that perception trumps reality. The FastPa.s.s concept is an absolute stroke of genius; it utterly changes perceived waiting times and has made many, many park-goers very, very giddy.

Behind the scenes, statisticians run the FastPa.s.s system through a network of computers that count visitors and record wait times. When a new guest arrives, they figure out how long the ride would take to serve all the patrons in front of him or her, including the "virtual" ones now scattered around the park holding dearly to their FastPa.s.s tickets. The guest is then advised a return time later in the day. The line looks short but only because many people in line are not physically present. The new guest does not get to skip ahead. In effect, Disney presents park-goers the Julie Cross gamble: should they accept the reliable reliable FastPa.s.s option, or should they get in the standby line and roll the die? Those in the standby line can get in with minimal waits if they happen to catch a lull in guest arrivals or FastPa.s.s returns, but more often than not, they will suffer hour-plus waits, as the following frustrated patron could attest: FastPa.s.s option, or should they get in the standby line and roll the die? Those in the standby line can get in with minimal waits if they happen to catch a lull in guest arrivals or FastPa.s.s returns, but more often than not, they will suffer hour-plus waits, as the following frustrated patron could attest: "After standing in line for Peter Pan last summer for over an hour watching the FastPa.s.s line moving through constantly, it appeared that the Cast Members were more inclined to let the FastPa.s.s holders have far too much precedence over those of us who were sweating profusely (not to mention not smelling too great after a day at the park). It was aggravating."

Compare that experience with this view from the other line: "A lot of people were trying to figure out who we were. You could feel their stares."

Like ramp metering, FastPa.s.s also works by stamping out variability, in that guests are being s.p.a.ced out as they arrive. When the rate of arrival exceeds the ride's capacity, those picking up FastPa.s.ses agree to return later in the day. At other times, when demand lapses temporarily, standby guests are admitted readily to prevent idle time. In this way, the rides run at full capacity whenever possible. As Professor d.i.c.k Larson, tellingly known as "Dr. Queue," remarked, "Even though Disney's theme park lines get longer each year, customer satisfaction, as measured by exit polls, continues to rise."

Back in Minnesota, perception trumped reality once more: much to the chagrin of Mn/DOT, the transportation department's prized ramp-metering strategy was under siege in the fall of 2000. State senator d.i.c.k Day led a charge to abolish the nationally recognized program, portraying it as part of the problem, not the solution. In his view, decades of ramp metering had come to naught as the Twin Cities continued to be among the most congested in America. The state came dead last, with 71 percent of its urban freeways declared congested in a report by the American Society of Civil Engineers.

Leave it to Senator Day to speak the minds of "average Joes"-the people he meets at coffee shops, county fairs, summer parades, and the stock car races he loves. He saw ramp metering as a symbol of Big Government strangling our liberty: "It's always bothered me-who stops? Who is the first person to stop at a ramp meter in the morning? Why does he stop? He should just go right through it. The first guy is jamming it up, and it ripples back to fifteen to twenty cars." How the senator managed to tap into a deep well of discontent! The Star Tribune Star Tribune readers offered their firsthand accounts: readers offered their firsthand accounts: "The operation of the meters makes no sense. Far too often the meters are on when the traffic is actually very light on the freeway, and in addition, the meters are cycling at a very slow rate.""Why do traffic managers allow the meters to create long lines at 6 P.M P.M. when there are about thirty cars on the freeway and they are moving at 75 miles per hour? What's up with that? Is no one minding the store?"