top of page

Are the Markets Too Volatile to Be Considered Rational?

Last newsletter, we talked about the convincing evidence that market prices already reflect all known public information. The evidence shows that very few active managers who set out to beat the market consistently do so. Very few. And those who do rarely repeat the feat consistently. So simply investing in passive, low-cost index funds will, on average, improve investment performance significantly.

Still, there are some circumstances in which the markets are not as efficient as we might expect. As Warren Buffet summarized, “Observing correctly that the market was frequently efficient, they went on to conclude incorrectly that it was always efficient. The difference between these propositions is night and day.”

There are numerous documented examples of market inefficiencies. Too many to discuss here. But most are still hard to exploit for any kind of advantage, at least consistently.

One of the most convincing illustrations of market inefficiency is that markets are much more volatile than the underlying fundamental measures of value, which are actually fairly steady most of the time.

The underlying value is driven by the concept that today’s value of any investment is the sum of the future cash flows expected from that investment (discounted to today’s value):

If each stack of coins above is the present value of each year’s expected cash flows, then if you scooped all of those coins into a pile, that total would be today’s rational value for that investment. It’s hard to predict the size of each future stack of coins for individual companies, but it is much more straightforward for overall markets.

And importantly, changes in the height of the leftmost stack (the next 12 months) have very little impact on the overall value of the entirety of the coins. Yet the daily news cycle and much of the daily trading activity are obsessively focused on that first stack.

Excess Volatility

Does it seem like the markets have been a wild ride this year? Yet overall underlying corporate earnings have hardly varied from their paths at all this year. In fact, through the first three quarters of 2022, earnings have actually grown, and they are expected to continue to grow in 2023.

As Nobel laureate Robert Shiller pointed out in his Nobel acceptance lecture, markets are much more volatile than underlying company fundaments, and this is one of the most convincing arguments against market efficiency:

Source: Shiller, Robert J., Irrational Exuberance, 2015; Princeton University Press, Appendix A, Figure A-1.

Warren Buffet famously expressed this concept more colloquially in his “Mr. Market” analogy. I encourage all who haven’t read it to follow the link, as this is one of the most concise and colorful expressions of this idea of excess volatility in the markets.


Dr. Shiller’s work points out that when “narratives” take hold, a form of groupthink can dominate investment decisions and cause prices to deviate from truly independent estimates. He does not necessarily conclude that this demonstrates irrationality per se but more along the lines of a high school debate in which one side makes a more convincing case than the other.

Remember Professor Treynor of the Bean Jar Experiments? His experiments demonstrated that very few individuals could beat the group estimate of the number of beans in a jar. But variations of his experiments also showed that when participants were allowed to share methodologies and discuss estimates, their accumulated accuracy actually decreased from those that were truly independent estimates. This is similar to Dr. Shiller’s premise. (Treynor, Jack L. “Market Efficiency and the Bean Jar Experiment.” Financial Analysts Journal 43, no. 3 (1987): 50–53.)

When Warren Buffett was asked the most important trait for a successful investment manager, he responded not with "intelligence" or "business acumen" or "accounting analysis skills" but with “independence of thought.” This meshes with academic research on narratives and groupthink.


  • According to AAA, 73% of Americans think they are better-than-average drivers.1

  • 71% of males and 59% of females think they possess above-average intelligence.2

  • 94% of college professors think their teaching abilities are above average.3

  • If you took my Finance classes, you know that when we performed a written confidence test in class, typically 90% to 100% of the class overestimated their own ability to estimate unknown quantities. (Understandable, though, since they clearly had an above-average teacher.)

So of course many people are going to think they can beat the market.

But as we saw previously, only 10% of active investors actually do beat market indexes over a 10-year timeframe. In this uncertain world, I will take 90% odds over 10% odds any time and steer away from expensive active management. The more technical term for overconfidence is illusory superiority. I trust no examples are needed, since we all witness this human trait regularly.

As Shiller points out, if it weren’t for overconfidence, we’d probably have very little trading in the financial markets at all.

When investment legend Peter Bernstein was asked a similar question to the one put to Mr. Buffett, he answered that the most important trait he learned over his investment career was “humility,” essentially the opposite of overconfidence.

Magical Thinking

Most people, if asked, will say they don’t believe in magical thinking, or the ability to change outcomes with their minds or predict things that haven’t happened yet. But as a species, our behavior suggests we think we can do those things.

When asked to part with a lottery ticket for an upfront payment,

people will require four times more if they personally chose the

numbers on the ticket. People will place larger bets on a coin that has not yet been tossed than on an unseen coin that has already been tossed, suggesting that they think they can somehow influence the outcome of the coin toss. (Shiller, Chapter 9)

We probably all know people who think they have exceptional skills at games of pure chance, like slot machines or video poker. Much of the gambling industry is based on these human traits of overconfidence and magical thinking. I’m going to go not very far out on a limb and assert the same about the investment markets.

(This is not to say that there are not those who consistently win gambling games of skill through exceptional proficiency. World poker champion Annie Duke’s book, Thinking in Bets: Making Smarter Decisions When You Don't Have All the Facts applies to everything from investment decisions to medical treatments.)

Market Bubbles

For a short period in the 1600s, a handful of tulip bulbs could buy a home in Holland. In the late 1800s, the equivalent in the U.S. was railroad stocks. In the late 1900’s, it was internet stocks, etc. Each of these caused significant market distortions that, in hindsight, were not rational. Or maybe they seemed to be rational to many participants at the time, due to the cognitive traits mentioned above.


So…where are we? Markets are pretty efficient, but not entirely. In particular, human emotions and cognitive biases seem to amplify market moves and at times, drive them significantly away from their rational values.

What does this mean from a practical standpoint for investment management?

I had not planned on this “efficient markets” topic being a trilogy, but it will need to be. This newsletter is getting too long! So I will wrap up the practical conclusions in the third instalment next time, and a trilogy it will be. (Sorry, Mr. Tolkien -- you’ve got competition.) But please read all three newsletters, because the movie versions might not be as good. 😊

Until next time…


Follow us:

Featured Posts
bottom of page