Cultivating Judgement
Good judgement is needed for any profession of consequence.
As Naval states “in an age of infinite leverage, judgement is the most important skill.”
How do you cultivate judgement? Mostly through experience.
But, there are some frameworks that can help. Below are my notes from Cultivating Your Judgement Skills, from Michael Mauboussin.
People are bad at making predictions
1/ Judgement and decision making is fundamentally about human beings trying to predict what will happen in the future. The decision they make is to typically to maximize gain and minimize pain of this future event.
2/ When Daniel Kahneman, who was awarded the Nobel Prize in Economics in 2002, was asked which of his 131 academic papers was his favorite, he answered “On the Psychology of Prediction”
3/ Most people want to improve their decision making abilities. Research has found that experts who are paid to make predictions in their field – economics, politics, and sports – are not very good at predicting outcomes. So many of us can improve how we make decisions.
Judgement is a skill that can be taught
4/ Philip Tetlock, professor at Wharton, created a ‘Judgement Matrix’, which separates acquiring a new skill into three stages: stage 1: cognitive and error prone like the first time you ride a bike, stage 2: associative with fewer errors such as biking without falling, and stage 3: autonomous and near perfection such as when riding your bike is habitual.
5/ The Judgement matrix identifies three skills to develop to improve decision making: identifying causality, using reference classes, and integrating new information. To improve your decision making, improve these skills.
6/ Ideally, you are improving your judgement of cause and effect (identifying causality), effective at incorporating prior events into your decision (using reference classes), and updating your beliefs when you have new information (integrating new information).
Theories help filter information
7/ Causality is about identifying cause and effect. This is the same thing as a theory, which is an explanation of cause and effect. Clayton Christensen’s, who was one of the most influential professors at HBS, has a three step view of theory building. First is observation: observing the phenomena and recording the results. Essentially documenting and agreeing on the standards so we are saying the same thing. Second is calssification: place the phenomena into categories that allow for distinction, usually based on attributes. Third step is definition: describe the relationship between cateogies and outcomes. Eg the correlation between the two.
8/ Once you have a theory, you test the prediction with actual data to see how error prone the model is. Also you can identify anonlies and update your theory as needed. As theories develop, classification evolves from attributes to circumstances – not just what works but when it works.
9/ Using reference classes can be thought of as building a graph, similar to a family tree. But instead of family members in the boxes, you have two types of variables: the specifics about the case (the inside view) qnd the base rate (the outside view). And the lines represents the relative weights you assign to those variables.
10/ The “inside” view in business typically involves gathering information, considering scenarios, adjusting, and extrapolating what we think will happen into the future. Pyschologists have found that many people dwell too much on the inside view – what we are trying to forecast is more unique than it actually is or that people are more different from one another than they actually are.
11/ The “outside” view considers a specific forecast in the context of a larger reference class. The outside view relies on the similarity between the case at hand and last examples.
Is it luck or is it skill?
12/ Weighting the variables can be tricky. One way to assess whether more weight should be assigned to the inside view or outside view is to assess the activity on the luck-skill spectrum.
13/ The luck-skill continuum is exactly as it sounds: it plots activities along a spectrum based on if they are more luck or skill based. For example, practicing the violin improves your abilities (skill based) whereas playing the lottery does not (luck based. Activities that are heavily skill-based, you should assign most of the weight to the inside view. For activities that are heavily luck-based, you should assign most of the weight to the outside view.
14/ Reversion to the mean — the probability that the next event is near the mean expected outcome — is influenced by the luck-skill continuum. If the activity is more luck based, the rate of reversion towards the mean is greater than a skill based activity.
15/ For example, if you are an Olympic swimmer, your next race in the pool will probably, and consistently be multiple standard deviations higher than the mean and your rate of reversion to the mean is low. However, if you are a fund manager, it is more probable that next year’s returns will be closer to the mean. Yes there are some outliers that continually outperform the market, but the rate of reversion to the mean is higher for the fund manager than the Olympic swimmer.
16/ To estimate where your judgement lands on the luck-skill continuum, you can use the equation Grand Average + Shrinkage Factor (observed average – grand average) = Estimated True Skill.
Grand average is the mean by which to revert and the shrinkage factor is how much to revert the results. A shrinkage factor of 0 means the next occurance will absolutely revert back to the mean and a shrinkage factor of 1 means it most certainly will not.
17/ For example, let’s say you have a fund manager who returns 12%. The market returns 7%. And for simplicity sale, let’s put the shrinkage factor at 0.1. The estimated true skill equation would be = 7% + 0.1 (12% – 7%) = 7.5%
This means that 7.5% out of the 12% return is based on skill. The rest, 4.5%, and most of the incremental gains are from luck.
18/ As you can see, the shrinkage factor can have a signifcicant impact on the Esrimated True Skill score. Selecting the appropriate factor (0-1.0) is critical.
19/ The coeffient of correlation, measuring the linear degree of relationship between two variables, is a decent proxy.
20/ If we go back to our swimming example, if we have a normally distributed population of people, including an Olympic swimmer at a pool. On Monday they all race and we record their time. They come back on Tuesday and swim again. The coefficient of correlation would be near 1.0 — the Olympic swimmer would outperform everyone else both days while slow swimmers would perform poorly both days. Therefore there is little reversion to the mean and most of the weighting should be on the inside view.
21/ However, if we go back to the fund manager with returns that beat the market, we know that over the medium to long term, those results will be close the the overall performance of the market. Therefore the correlation of beating the market is low, the shrinkage factor will be close to 0, the reversion to the mean is high, and the outside view should receive most of the weight in the prediction.
22/ Research has found that most executives and investors use a small sample size of cases and often underweight the sample cases in this outside view. Said another way, they overweight the inside view and believe the activity and decision is more skill-based than it actually is. This is the definition of hubris.
Reduce your bias to reduce your errors
23/ The most effective decision makers use similarity-based forecasting – a set of unbiased cases in which the more relevant ones receive more weight and the less relevant ones sent discarded, but receive a lower weight.
24/ Integrating new information: how well do you update your priors? Most people are terrible at it. Ideally we have subjective prior beliefs that update when we receive new information, however, we have strong confirmation bias. We are more likely to seek out information that confirms our beliefs than that disconfirms them.
25/ a stage 1 thinker is not able to distinguish which new information matters and which does not. They typically stick to their prior beliefs. A stage 2 thinker is able to distinguish what new information is important and incorporates it in their prediction. they are able to improve their prediction as causality is more clear. However, they are unable to assign the appropriate weights to the new information. A stage 3 thinker is able to both identify which new information provided causal clues and weight the variables accurately.
26/ A key variable to assess if the information is causal is feedback — that is timely and accurate. Two ways to improve the feedback loop — to make it more timely and accurate — is to have control of the outcome (eg you can change the design of a product, the price, etc) and the second is highly reversible decisions (eg public equities).