At the start of the year, I made sixty predictions about how 2017 would go. Now it's time to see just how badly I did. The predictions left unmarked were correct, those struck-through were incorrect. I'm only throwing out the Obamacare prediction, because no one can agree on what constitutes a repeal. It is marked in italics.
I will still be living in Kansas: 95%
My parents will have moved into their new house: 90%
I will still be annoyed about it: 70%
I will read 10 books this year: 90%
....12 books: 85%
....15 books: 75%
....20 books: 60%
I will write an average of one blog post per month: 80%
....two blog posts per month: 60%
I will not succeed in a longform fiction writing project this year: 85%
I will not get back together with my ex-girlfriend: 85%
I will not get a new girlfriend: 65%
I will not get a boyfriend: 95%
I will not experience a major political conversion this year: 80%
I will not experience a major religious conversion this year: 95%
I will be able to perform 15 push-ups with ease: 90%
....20 push-ups: 70%
I will not be hospitalized: 95%
I will not begin drinking regularly: 90%
I will not attend a solstice event: 80%
ACADEMIC/PROFESSIONAL
I will still be studying aerospace engineering: 95%
....at the University of Kansas: 93%
....on track to graduate in May 2018: 90%
I will end the year with GPA 3.0 or above: 80%
I will not be planning to attend grad school: 90%
I will not learn Haskell this year: 80%
I will not begin learning Russian this year: 80%
My French will not improve this year: 75%
SCIENCE/SPACE
Human trials of CRISPR announced: 80%
No successful human clones announced: 65%
KIC 8462852 will not be satisfactorily explained: 80%
No strong evidence of extra-terrestrial life will be found: 95%
SpaceX launches a reused rocket: 80%
Reused rocket is successful: 70%
No deaths in space this year: 90%
No country leaves the ISS Treaty: 85%
POLITICS/WORLD
Donald Trump will still be President at the end of 2017: 95%
Mike Pence will still be Vice President: 95%
Obamacare will not have been repealed: 60%
United States will not go to war with a nuclear power: 85%
United States will not enter a major new war (>100 US casualties): 70%
North Korean government will not be overthrown or displaced: 95%
North Korean government will not begin liberal reforms: 93%
Theresa May will still be Prime Minister of the United Kingdom: 90%
Ukraine Conflict will not be resolved in 2017: 85%
META
I will remember to score these predictions: 95%
I will have been overconfident in these predictions: 80%
This last one sort of requires me to do the math first, so let's get to it. Spoiler: it won't be the tie-breaker question.
60% predictions: 2 right and 1 wrong, for a score of 67%
65% predictions: 2 right and 2 wrong, for a score of 50%
70% predictions: 4 right and 4 wrong, for a score of 50%
75% predictions: 2 right and 1 wrong, for a score of 67%
80% predictions: 10 right and 3 wrong, for a score of 77%
85% predictions: 6 right and 0 wrong, for a score of 100%
90% predictions: 8 right and 0 wrong, for a score of 100%
93% predictions: 2 right and 0 wrong, for a score of 100%
95% predictions: 10 right and 0 wrong, for a score of 100%
Graphed, my results look like this:
What does this chart mean? Bear with me here, because this may get confusing.
The orange line represents perfect calibration i.e. 90% of the predictions made with 90% confidence are accurate. The closer to the line, the better calibrated you are as a predictor. Results above the line are underconfident, while results below the line are overconfident. My predictions made with high confidence were underconfident, while those made with somewhat lower confidence nevertheless proved to be overconfident.
As a general rule, underconfidence is preferable to overconfidence. In colloquial terms, it's better to be safe than sorry.
As a general rule, underconfidence is preferable to overconfidence. In colloquial terms, it's better to be safe than sorry.
If we look at individual categories, it's clear that my worst performance was in Academic/Professional predictions, while my best was either Personal or Politics/World, depending on how you choose to count. Technically Meta was 100% correct, but that wasn't really so much of a prediction as an acknowledgement that January!Me had no idea what I was doing.
There's several issues I want to rectify before my 2018 predictions. Firstly, several of my sample sizes were too small. For example, there's only two predictions with 50% calibration. I got both of these wrong, but it's hard to say whether I was dramatically overconfident or just a bit unlucky. Meanwhile, most of my predictions were made with 85%, 90%, 93%, or 95% confidence. Arguably, I was essentially shooting fish in a barrel on these. It would be better to distribute my predictions more evenly.
Furthermore, it may not be a good idea to use even two significant figures for this kind of project. Throwing out the 50% predictions and lumping everything into decades (60% and 65% predictions together, 70% and 75% predictions together....) we get results looking more like this:
with the breakdown as follows:
60–69% predictions: 4 right and 3 wrong, for a score of 57%
70–79% predictions: 6 right and 5 wrong, for a score of 55%
80–89% predictions: 16 right and 2 wrong, for a score of 84%
90–100% predictions: 20 right and 0 wrong, for a score of 100%
This more seriously reflects the confidence gap between high probability outcomes (things more than 80% likely to happen) and possibilities with lower odds. Basically, I scored a lot of probably things too low, and a bunch of improbable things too high. It's an embarrassing sort of mistake to make, but figuring out where I'm under and overconfident is the entirely purpose of this exercise, so mission accomplished.
2018 predictions, hopefully reflecting these insights, coming soon.
