Metaknowledge
A few weeks ago, I thought I’d do a demonstration of the wisdom of crowds. I was at a bat mitzvah reception and, as a game, the hosts asked each table to guess the number of Skittles in a big Tupperware bowl. I got everyone at our table to write down a guess and then averaged the results. Based on what social scientists have been saying, our collective answer should have been spot-on. Each of us had a vague hunch about how to pack small objects into big boxes, subject to much uncertainty. Taken together, though, our scraps of erudition should have accumulated while the individual errors cancelled out. But my experiment was an abject failure. Our estimate was off by a factor of two. Another table won the cool blinking necklace.Wisdom of crowds is an old concept. It goes back to Ancient Greek and, later, Enlightenment thinkers who argued that democracy is not just a nice idea, but a mathematically proven way to make good decisions. Even a citizenry of knaves collectively outperforms the shrewdest monarch, according to this proposition. What the knaves lack in personal knowledge, they make up for in diversity. In the 1990s, crowd wisdom became a pop-culture obsession, providing a rationale for wikis, crowdsourcing, prediction markets and popularity-based search algorithms.That endorsement came with a big caveat, however: even proponents admitted that crowds are as apt to be witless as well as wise. The good democrats of Athens marched into a ruinous war with Sparta. French Revolutionary mobs killed the Enlightenment. In the years leading up to 2008, the herd of Wall Street forgot the most basic principles of risk management. Then there was my little Skittles contest. It was precisely the type of problem that crowds are supposed to do well on: a quiet pooling of diverse and independent assessments, without any group discussion that a single person might dominate. Nevertheless, my crowd failed.Dražen Prelec, a behavioural economist at the Massachusetts Institute of Technology (MIT), is working on a way to smarten up the hive mind. One reason that crowds mess up, he notes, is the hegemony of common knowledge. Even when people make independent judgments, they might be working off the same information. When you average everyone’s judgments, information that is known to all gets counted repeatedly, once for each person, which gives it more significance than it deserves and drowns out diverse sources of knowledge. In the end, the lowest common denominator dominates. It’s a common scourge in social settings: think of dinner conversations that consist of people repeating to one another the things they all read in The New York Times.In many scientific disputes, too, the consensus viewpoint rests on a much slenderer base of knowledge than it might appear. For instance, in the 1920s and ’30s, physicists intensely debated how to interpret quantum mechanics, and for decades thereafter textbooks recorded the dispute as a lopsided battle between Albert Einstein, fighting a lonely rearguard action against the new theory, and everyone else. In fact, ‘everyone else’ was recycling the same arguments made by Niels Bohr and Werner Heisenberg, while Einstein was backed up by Erwin Schrödinger. What looked like one versus many was really two on two. Very little fresh knowledge entered the discussion until the 1960s. Even today, Bohr and Heisenberg’s view (the so-called Copenhagen interpretation) is considered the standard one, a privileged status it never deserved.Prelec started from the premise that some people’s judgments deserve greater weight than others. By no longer averaging everyone’s judgments equally, you can avoid overcounting redundant or otherwise extraneous information. You already do this all the time, whenever you trust opinions that are expressed with confidence and spurn diffident-sounding ones. There’s something to be said for that kind of trust. In psychology experiments, people who are more accurate at a task – say, remembering a list of words – tend to express more confidence. Unfortunately, the converse isn’t true: confident people aren’t necessarily more accurate. As W B Yeats wrote: ‘The best lack all conviction, while the worst are full of passionate intensity.’ Also, people systematically overestimate the value of their knowledge. A rule of thumb is that 100 per cent confidence means you’re right 70 to 85 per cent of the time. What is needed is a better way to measure the value of a person’s knowledge before putting it into the wisdom-of-crowds mix.The solution, Prelec suggests, is to weight answers not by confidence but by metaknowledge: knowledge about knowledge. Metaknowledge means you are aware of what you know or don’t know, and of where your level of knowledge stands in relation to other people’s. That’s a useful measure of your value to the crowd, because knowledge and metaknowledge usually go together. ‘Expertise implies not only knowledge of a subject matter but knowledge of how knowledge of that subject matter is produced,’ says Aaron Bentley, a graduate student at the City University of New York Graduate Center who studies social cognition.Whereas you might have no independent way to verify people’s knowledge, you can confirm their metaknowledge....MORE
