Ox Weights, Game Shows, Bean Jars, and the Stock Market

How Hard is it to Outguess the Markets?

Is it time to increase our allocation to stocks? Do you think Tesla is a good investment right now? Should we get into innovation stocks to capitalize on coming changes in technology? Should we reduce exposure to stocks ahead of upcoming elections? Should we get out of the market before the next recession hits? I get questions like this from clients every week, and they all are based on the idea that we can know something that the market doesn’t and therefore is not already incorporated into market prices.

Let me ask you some questions. How great is your skill at estimating slaughtered ox weights? Want to try guessing the number of beans in a large jar, estimating the probability of precipitation next weekend, or estimating the value of Amazon stock? I’ll get right to the point: Any given expert is likely to be better than any given amateur in estimating uncertain quantities. But a large group of estimators (like a financial market) will usually provide a more accurate estimate than any given individual, even a trained expert.

According to Standard and Poors, over the last decade, less than 10% of actively-managed investment funds outperformed an unmanaged stock market index, such as the S&P 500. And if an active financial manager is lucky enough to beat the unmanaged stock market index in any given year, they are unlikely to be able to repeat the feat in subsequent years.

In other words, by simply buying low-cost, unmanaged funds that simply mimic the market indexes, we have a very good chance of outperforming almost all funds actively managed by financial experts.

How can that be? One reason is that actively-managed funds have much higher fees (Wall Street rent, hundreds of analysts on staff, huge marketing budgets, etc.) and trading costs (lots of jumping in and out of holdings), and these higher costs are a large drag on investment performance. This is why our own client accounts are typically invested largely in low-cost passive funds (mimicking the unmanaged index) rather than higher-cost actively managed funds.

But another reason that individuals, even trained professionals, tend to underperform the market may be less obvious on the surface: The wisdom of crowds.

It might seem that “the masses” are not very wise at all. At least, that’s what British scientist and statistician Sir Francis Galton thought and set out to prove in 1906. He believed that average citizens were capable of very little, and therefore voting was a poor way of making decisions. He thought a great way to demonstrate this concept was to do a study of ox weight predictions from a large regional livestock exhibition. There had been numerous attendees at the exhibition, some of whom were experienced butchers, farmers, or ranchers, and many who were not. About 800 attendees participated in a competition to estimate the dressed (slaughtered) weight of a (still living) ox.

Sir Galton expected the estimates to be wildly off the mark and to demonstrate in an easy-to-understand way that average citizens were incompetent when it came to making technical assessments. His premise may have been valid on an individual basis, but collectively, his study demonstrated the exact opposite. Although individual ox weight estimates were widely dispersed, the average crowd estimate of the dressed ox weight was 1,197 pounds. The actual weight was 1,198 pounds (Surowiecki, James, The Wisdom of Crowds, Anchor Books, 2005, xiii).

A little closer to the topic at hand (investing), legendary finance Professor Jack Treynor demonstrated this concept to his finance students using large jars filled with beans. He had his students independently submit their estimates of the number of beans in a jar, and he tallied the results.

The first time he performed this experiment, the jar held 810 beans. The class’s mean estimate was 841. Only two of 46 student estimates were closer to the actual number than the consensus estimate of the group. He performed the experiment a second time with a different class. This time, there were 850 beans in the jar, and the average class estimate was 871. Only one individual estimate out of 56 was closer to the true value than the collective estimate of the class. Professor Treynor actually published this methodology in the Financial Analysts Journal. (Treynor, Jack L. “Market Efficiency and the Bean Jar Experiment.” Financial Analysts Journal 43, no. 3 (1987): 50–53.) This experiment has been repeated countless times in finance and economic classes ever since.

Remember the “Who Wants to be a Millionaire” TV game show? When contestants were stumped by a question, they had three single-use “lifelines.” They could randomly eliminate two possible answers, phone a friend (presumably the most knowledgeable person they knew), or ask the audience to vote. The smart individual friends did reasonably well, coming up with the right answer 65% of the time. The audience was right 91% of the time (Surowiecki, 4).

While an individual brain cell may not possess much intelligence, collectively, numerous brain cells produce amazing intelligence. It is impossible for any given cell to be smarter than the whole brain, and it is unlikely that any individual investor will be consistently smarter than the overall markets consisting of millions of participants/investors all trying to solve the same problem.

In other words, market prices already incorporate the knowledge that some companies and industries are going to grow faster than others, the likelihood and impact of a looming recession, and the impacts of interest rates, inflation, corporate profits, and geopolitical events. It is unusual for us to have any special insight that the market does not already possess.

In essence, every single financial trade results from two parties, each possessing exactly the same public information, arriving at exactly opposite conclusions (one to buy and one to sell at the current price). Yet, in aggregate, all of these opposing viewpoints drive the market price to a reasonable equilibrium that is very difficult to beat.

There are numerous academic studies (though maybe not as colorful as the informal examples above) that demonstrate the efficiency of the financial markets in estimating unknown quantities and driving prices to the collective estimate.

This doesn’t mean that the markets are always right (!), but it means that the market estimate is hard to beat. It is like Las Vegas point spreads on sporting events. It is rare for the actual final score to exactly match the pre-game betting point spread. For example, it is unusual for a football team favored by three points to actually win by exactly one field goal. But point spreads, in general, are still (very) hard to beat consistently, with misses in both directions being very close to equal.

So it is with financial markets. Since the vast majority of active investment managers underperform unmanaged market indexes, the old adage that you get what you pay for often does not hold for the high-fee, active management brokerage industry. On average, the higher fees of active management do not pay for themselves. But the irony here is that markets are so efficient and hard to beat because there are so many highly paid and highly trained market participants driving the prices of securities to a (usually) reasonable equilibrium. But passive investors (like us) can ride almost for free, thanks to the enormous amount of thought and analysis by active investors that have established securities prices in a way that is hard to beat.

In light of this, even Warren Buffett, one of the most successful and famous active investors ever, has recommended low-cost passive index funds for his family and for investors in general.

So clearly, passive investment is superior to actively making investment decisions. Except when it’s not. (What!? A plot twist this close to the end of this story!?)

Yes, it is hard to beat the collective analysis of millions of market participants estimating the value of Tesla or Amazon stock and driving the stock price to that average estimate. But there are two broad categories of problems with this premise about the collective wisdom of the markets. First, there are known conditions under which markets are notoriously not efficient. Second, it is not possible from a practical standpoint to be an entirely passive investor. Substantial judgment must still be used.

These topics of when markets are efficient and when they are inefficient are major topics in investment research, and Nobel prizes have been awarded for research on both aspects. On a more practical level, these are also common discussion topics with clients, which is why I decided to address them in a couple of articles.

If you want to venture into the empirical evidence, I’d recommend A Random Walk Down Wall Street and Irrational Exuberance for easy-to-read summaries of opposing viewpoints, with each acknowledging contrary evidence. And if you want to see someone from the traditional brokerage industry get riled up, bring up the topic of efficient markets! But the truth is that the answer here is not black-and-white. Sometimes markets are efficient, and sometimes they're not. But of course it's natural to crave a simplistic answer to a complex issue.

What are the circumstances that tend to contradict the wisdom of crowds and the efficiency of markets? If passive investing is so statistically compelling, why is it so hard from a practical standpoint to construct an entirely passive investment portfolio? We want to be diversified, but in what proportions should we hold various passive index funds? How do human cognitive biases and emotions feed into market (in)efficiency?

In the words we never wanted to see at the end of an episode of our favorite TV show:

Tim