If P then Q(A). But turns out P(laning) is trickier than you thought

This one is mostly for fun (for a wide enough definition of fun).

We’re going to explore the Surprise Exam Paradox from a unique mathematical perspective, then transpose it onto project risk planning, and why that can never be sufficiently covered. Some of the piece will contain deep dives into the math, but you can skip those bits and still enjoy the rest of the ride.

Our structure will be as usual: first half dedicated to the paradox, then a math paper giving a very cool and fresh point of view on it. We’ll use some of the argument structures from the paper to examine project risk analysis planning, play with that a bit, and reinforce the conclusions from an empirical point of view.

Sounds like a nerdy hoot.

Let’s go.

Prologue: The Paradox

You probably met the surprise exam paradox at one point or another (sometimes called the unexpected hanging paradox, but we’ll stick with the PG title version). It goes something like this:

A teacher tells the class there’ll be an exam on one of the days next week, and guarantees it will be a surprise: the students won’t know which day it’s on until they actually come in and see the exam papers on their desks.

The students, being typical students, invest more work in decoding the statement than what would actually be needed to pass the exam. They think it through: the exam can’t be on Friday, because if Thursday evening arrives with no exam yet, the only day left is Friday. This would mean we would know it’s on Friday beforehand, we won’t be surprised. This means it can’t be on Friday.

But now, with Friday eliminated, the days the exam can take place are reduced to Monday through Thursday, and the same argument applies to the new range: if Wednesday evening arrives with no exam yet, Thursday is the only day left, and we’ll know that beforehand, and won’t be surprised. So Thursday is out as well. Then apply the same structure to eliminate Wednesday, Tuesday, and finally Monday. Chill everyone, there can’t be a surprise exam.

Of course the exam is on Wednesday; it catches everyone by surprise, and the entire class fails except for that one guy nobody likes. He aces it, because that guy is just the worst.

Nice, concise little brain twister. It’s been around since the 1940s, stirring up debate not unlike the Monty Hall problem. Almost everyone thinks there’s an obvious flaw in the students’ reasoning, or the way the problem is presented, but they can’t seem to nail down exactly what that is.

Let’s look at what the standard mainstream professional analysis has to say.

Possible answers

The announcement is just false

The most straightforward approach is to formalise the teacher’s announcement and see what breaks under well defined terms. The standard way to do so is to replace the vague notion of a surprise with the well defined term of provability. We’ll give a general descriptive account, then go through a slightly more formalised version for the logic nerds in the room.

The teacher’s announcement becomes a statement saying: the day of the exam is one of the set Friday to Monday. Let’s go through them one by one. IF we can prove that reaching the day in question with no exam yet means the exam must be on that day, THEN the exam can’t be on that day (because proving it => we won’t be surprised by it, and we’re guaranteed to be surprised).

Now run the students’ argument with that formalisation and see how it plays out.

Go through this whole thing and you end up deriving that D is not any day of the week, contradicting what the teacher said (that there will be a surprise exam next week). So S contradicts itself (or is just flat-out false). The teacher said something incoherent. Paradox dissolved. Yay.

Feels somewhat of a cheat, no?

A very cool paper by Shira Kritchman and Ran Raz agrees, and offers a fresher, more nuanced analysis that leads to a far more satisfying result (IMHO). Their approach will also give us a nice hook to build some day-to-day insights on in the 2nd half.

Let’s see what the paper is all about. Well, not all about; we’ll see what the tiny scoped portion of it that is relevant for us is about. If you have even intermediate math/logic knowhow I strongly recommend you read the whole paper. The full text available publicly, it’s pretty approachable as these things go, and very elegant.

Kritchman and Raz don’t accept the standard end conclusion of the mainstream analysis. It’s not that there’s anything wrong with it, per se, it’s just that the end result invalidates what it sets out to show. If S is indeed a contradiction, then everything follows from S (that’s the logical consequence of contradictions). This means that on Thursday night, the students can prove from S that the exam is on Friday (a-OK so far); but they can also prove that the exam is on Monday, and that it’s on Wednesday, and that I write very short and concise post, and that pigs fly, etc. Contradictions have that corrosive effect of logical systems - everything goes.

Well, if the students can prove that the exam is on Friday AND they can also prove the exam is on Monday, would we say the students know it’s Friday? Obviously not, because if they do, they also know it’s on Monday. And while we previously noted that it’s hard or even impossible to define exactly what knowledge is, a group of directly self-contradicting beliefs for sure isn’t knowledge.

But wait, the whole reason we replaced “know” with “prove” was the assumption that in this case, provability was a good enough proxy for knowledge. If provability from S doesn’t imply knowledge, that defeats the whole exercise. It turns out this whole time we were building a beautiful, well articulated answer for the wrong question.

Well, as the saying goes, if at first you don’t succeed, destroy all evidence that you tried. Or try again. Let’s do the latter.

Second attempt: what knowing actually requires

So, if straightforward simple provability isn’t a correct proxy, what would it take for the students to know, on Thursday night, that the exam is on Friday? We’ll track the possibilities Kritchman and Raz enumerate:

1. The students can’t prove it’s on Friday. OK, this is easy - obviously they don’t know.
2. The students can prove it’s on Friday, and they can also prove it’s on some other day. As we noted, this is not knowledge.
3. Finally, The students can prove it’s on Friday, and they cannot prove it’s on any other day.

Looks like the 3rd option does proxy / correspond to what we’d be willing to call knowledge. Derivability isn’t sufficient; we require derivability AND the absence of competing derivations. So let’s explore what this new prerequisite entails.

The students try to envision getting to Thursday night without there being an exam yet. Can they prove the exam will be on Friday? For sure, that’s easy, it’s the only day left available. Can they prove it’s on some other day? Feels obvious they can’t, as all other days will have gone by without an exam in actuality. This satisfies our new, fuller requirement, and we can indeed say that Thursday night they would know the exam will be on Friday. So they won’t be surprised if it were on Friday, so it can’t be on Friday, and we can carry on with the process as before.

Ah, wait. Actually, we can’t just carry on. In order to say the students CAN’T prove the exam will be in any other day, we need to require something from the reasoning system they apply (call it R); namely, we require it to be consistent. As we noted before, if it isn’t consistent, the students could prove anything they want. OK, so what does this mean? It means the students can’t simply derive D ≠ 5 (the exam can’t be on Friday), but only the conditional Con(R) → D ≠ 5 (IF our reasoning system is consistent, THEN the exam can’t be on Friday).

No biggy, just have the students establish the antecedent (the if part), and we’ll get the consequent (the then part). Here Kritchman and Raz’s argument reveals its coolness: the students can’t. No one can. It’s a proven theorem (Gödel’s 2nd incompleteness theorem, to be exact) that no sufficiently rich, consistent, effectively axiomatised system can prove its own consistency.

So the conditional never triggers. It just sits there, unprovable. When the students try the next inductive step, they don’t get to start from “we know the exam isn’t on Friday”, but rather “we know that if the system is consistent the exam isn’t on Friday”. This is quite a bit weaker, and it can’t support the next rung up the ladder.

At the end of the day the students are left with this: if they believe their system is consistent, then they indeed can derive and know beforehand that the exam cannot be held on Friday (A-OK). But it can still be held on any other day of the week, because they cannot prove their system is consistent and trigger the cascading sequence of inferences. The only thing the can be sure of is that Friday is off the table, and that whatever day the exam will actually be on, that one guy everyone hates will ace it (because of course he will).

Before we borrow this nifty approach for our own uses, let’s stop and appreciate how much more satisfying it is. Instead of dissolving the paradox into “the teacher was talking nonsense”, it locates the specific piece of the puzzle the students needed and couldn’t have. It even does so by connecting to something as fundamental as Gödel’s 2nd incompleteness theorem, which is just great in and of itself. The paradox survives as a paradox, with something true at its core that is unreachable from inside the system trying to reach it¹.

So, very nice paper, cool approach, all is well. Let’s build a bridge to transpose it onto a modern software testing project.

The risk register and the mitigation plan

Project management is the perpetual mitigation of risk. A major part of being able to do that is working off a mitigation plan, built beforehand against a prioritised list of identified risks. One might say it’s like preparing for a surprise exam, trying to predict which day it will hit, and making sure you won’t be taken by surprise 🤷‍♂️.

The analogy isn’t all that farfetched, actually. The register isn’t just a passive description, but an announcement that changes behaviour; and has a critical self referring component, usually marked as “No significant risk remains”. That’s a claim being made within the document, talking about what the document’s own analysis managed to turn up. These are interesting enough hooks for us to build on.

Let’s imagine a hypothetical project the likes of which you’ve probably seen. You got 5 weeks before a hard external deadline. The risk register has 31 prioritised items, reviewed every Tuesday in the weekly cadence meeting.

The first couple of weeks are pretty much what you’d expect - the team goes through each line item and finds an adequate mitigation of it. Payments integration: good testers, early cut-off, fallback clause; data migration: dry run, rollback rehearsal, signed reconciliation; performance: load test against a scaled environment, documented headroom, adequate cache mechanisms. Slowly but surely each risk line item gets a mitigation check and goes to green on the plan. All of this is A-OK.

But. Then comes week 3, and the meeting is shorter than usual as no one has anything meaningful to say. By week 4 someone says the magic words: well, if there was anything significant left, we would have raised it by now. A wall of turned off cameras in the Teams meeting agrees; after all, it’s not an unreasonable thing to say.

There were 4 passes through the list, there were additions, and there were mitigations documented. We projected our mind across the closed set of possibilities, and crossed them off, one by one, in order. There can’t be an exam that will catch us by surprise, as we’ve proven we’d know about it beforehand. Or, more to the point: if a surprising failure mode was going to bite us at go-live, we’d be able to see it from here and add it to the list; we can’t see it from here => therefore it isn’t out there.

Naturally, this project (completely hypothetical and totally not real that for sure not one that crashed and burned under my watch or anything like that) was caught by surprise when a significant issue did come up at go live, and didn’t have an appropriate mitigation.

But how could that have happened? We did everything right! Well, we have the walkthrough of the surprise exam analysis show us the way. The strict mathematical proof becomes looser (or straight up invalid) in our human scenario, so we’ll borrow more of the structure than the mechanical provable derivations; that would be a good enough guide to gain some interesting insights.

Let’s explore the different ways we can translate the mathematical mechanism into human form, while maintaining the general structure of the analysis.

Straightforward translation: consistency

First let’s try a direct translation. In this translation, the risk and mitigation register was great, and each line-item made sense by itself. However, for the register to be a predictive instrument you can reason and derive conclusions from, we also require internal consistency between line items. We’re relying on Con(R) (consistency of the Risk register), but we never made sure that’s actually the case.

So for example, assume there were two line items:

• MIG-04, the migration cutover, whose rollback procedure assumes the cache is flushed cold at switchover so that stale pre-migration rows cannot be served.
• PLAT-11, the platform cutover, whose sequencing assumes the cache is pre-warmed before traffic returns, because a cold cache under the Monday morning spike will take the session tier down, performance-wise.

Each line item makes total sense by itself, and each has a well thought out, procedurally cashed out mitigation. Both were reviewed. Both mitigations were signed off by an owner who understood their own scope completely. However, they are mutually exclusive, in the sense that a formal mitigation plan that relies on both can’t be fully executed. A list the contains them both is internally inconsistent.

Go-live: cache pre-warmed per PLAT-11, migration rollback triggered per MIG-04, warm cache serving pre-migration rows to live customers for a full hour before anyone noticed. By then bad data in downstream artifacts has already been sent out to customers, and the company is in regulatory breach (hypothetical breach, of course, as we noted this is for sure not something that crashed and burned under my watch).

OK, so that’s the scenario. Now, the week before, if you had asked the register “are we covered on cutover”?, it would have answered yes, for sure. Twice even; but in two mutually incompatible ways, each with a green status and a named owner and a link to a rehearsal report. The register asserted a mitigation state where the cache would be cold and warm. It could prove the exam is on two different days at the same time. Derivability from the analysis had stopped implying knowledge, and there was no way² to tell from inside, because all the artifacts still looked right.

Being a direct translation, this plays nicely with the paradox analysis we went through. It does, however, come off as somewhat contrived. Inconsistency may invalidate a mathematical analysis, for sure; and Gödel proves consistency can’t be proven from within a system, yes. But human affairs are by their nature messier, and maybe inherently inconsistent to begin with.

Before we proceed to explore more human-sized translations, there’s still a lesson to be learned even from this contrived scenario. If your risk register is assembled from independently owned line items, each reviewed separately only by its own owner, there won’t be any mechanism anywhere in it to prevent a similar scenario. If that’s the process, inconsistency isn’t the unlucky exception, or a result of the inherent chaos of human activity, but something that could and should have been avoided with some better care and attention.

That wraps the consistency translation. It gave us some insight, yes, but a limited one, as usually consistency isn’t the issue at hand. What usually kills the risk register isn’t the Con(R), but the Com(R), as in assuming its completeness.

A more human-sized translation: completeness

Let’s go back to the week-4 meeting. In a more plausible scenario, when everyone says we would’ve spotted any significant remaining items, the test manager would still wants some catch-all assurance. How does that usually play out? Usually everyone agrees to add a new line item called “buffer”, size it at a sensible-sounding 15% of the effort, and then mark the list as done. Again, seems reasonable. So how can this risk register be caught by surprise?

Well, just look at what the buffer line actually scopes: the risks we have not identified, sized at 15%. This pretends to do the literally impossible task of naming something that’s defined as unnameable. The moment it’s written down, it’s on the register. It has a size, an owner, RACI line and a slot in the steering pack. It has been enumerated. Which doesn’t mean the unenumerated residual disappeared, just that quietly moved under it.

The truly unknown residual is always exactly one line below the lowest named thing on your list, so adding a named line for it just relocates it rather than capturing it. This is made obvious once you stop and think about those 15%. 15% of what, exactly? Well, it’s 15% of the identified work, but it’s used to scope the unidentified work. It measures the wrong thing.

Cashed out in Rumsfeld-ian terms, for our analysis to be complete it needs to not only claim that there are no known unknowns left, but that the unknown unknowns are adequately scoped and covered. Not that there is a catchall line item (that’s easy), but that it’s an adequate catchall line item. And that’s not a claim you could ever be in a position to make, because being in a position to make it would mean knowing the size of the thing you’ve defined as unknowable.

We assume we have that covered with the catchall buffer, but establishing it’s an adequate mitigation would require validating the analysis completeness from inside the analysis, which is the move that isn’t available³. What the team is entitled to say is Com(R) → nothing significant is left (Com being complete). What was actually said was just the consequent, nothing significant is left.

So the contingency gets marked as buffer (legitimate), then as slack (concerning), then as the reason we’re confident (red flag!).

Then everything crashes and burns (hypothetically, etc.).

A point about prioritisation order

If you’re still after a more robust math connection, one can be found through the reliance on prioritisation as an ordered sequence.

In the surprise exam paradox, the students’ inductive reasoning relied on the days of the week being a finite, linearly ordered sequence. The risk prioritisation register similarly relies on the risks and their mitigations being ordered, so that a threshold of significance can be drawn below one of them.

There’s a critical difference between the two cases. For the students, the ordering is given. The days of the week are an ordered sequence as a brute fact of reality. That’s just the way things are, and if for some reason they stop being like that, the exam is the least of anyone’s problems.

There is no equivalent brute fact of the world making risks a linearly ordered sequence. After all, what about ties? Or context shifts due to new information? Or inter-dependency that causes one risk’s mitigation to reprioritise another risk, which itself changes the original risk mitigation and so on? What if the risk order changes as we get closer to go-live?

There is a self-referencing and even circularity, both from the line items to themselves, and from the line items to the threshold under which we stop developing the list. Worse still, the prioritisation ordering is produced by the model whose soundness is the very thing in question. So we use the model to order the risks, then use the ordering to decide when to stop interrogating the model. The stopping rule is derived from the thing it’s supposed to be checking.

This too can be thematically and technically linked to our paradox, with some deeper links available to the brave of heart (the connection is through Chaitin’s proof and Berry expressions, both mentioned in Kritchman and Raz’s paper).

Forget about human-sized math, let’s double down on the human part

Mapping the math directly or thematically onto human practice is a fun thought experiment. But if we’re after more concrete and common failure modes, we need to move from math theorems to the plethora of empirical data about the human part of the equation.

How we generate the risk register

There’s a general well documented psychological quirk where when people sit and thoroughly, exhaustively generate a risk register, up to the point where they struggle to come up with additional line items, the process paradoxically produces more confidence than a when a shorter, partial list is generated. Humans are weird that way.

Magne Jørgensen explored this effect specifically in the software industry in his 2010 paper Identification of more risks can lead to increased over-optimism of and over-confidence in software development effort estimates. A mouthful of a title, for sure, but you pretty much know what you’re gonna get going in.

His hypothesis was deliberately scoped: that there are situations where more work on risk identification produces more optimism and more confidence instead of the intended grounded realism. He tested it through 4 experiments in different countries and settings. His scoped results had (IMHO) muddled statistical significance, but they are still informative. Unlike Kritchman and Raz’s paper, I can’t share the link to the full article text, but look it up yourself. It’s an interesting read.

One experiment is specifically relevant for our purpose (experiment C). Jørgensen had the “MORE” group (identifying more risk line items) tag each identified risk with probability and severity, in the order they came up with them. The first risk people named was rated high-severity in 15 responses. The last risk they named was rated high-severity in 3. Read: people generated risk line items in descending order of severity. Shocker, I know. Wait till you hear he also found water is wet.

A triviality, for sure. But watch how this triviality becomes a critical part in a mechanism of failure. People are in the room, generating risk line items. They generate the list in a roughly descending order of severity, so the last line item they look at is the smallest thing, and that’s the impression each meeting ends on (at least in my experience).

This induces the implicit assumption that ALL that remains is small. This might be amplified further by the general pattern where the harder it becomes to dig out the next risk item, the more confident people get and less risky the project feels (that well documented human quirk we began with). Meaning that the effort of finding the next item gets read as evidence there’s nothing left.

Which connects nicely to our prioritisation order point from before. Remember that a register’s linear ordering is generated by the same model whose soundness is in question, and that this is a circularity. Turns out it’s not only a circularity, but one of the cogs in the mechanism. The ordering that lets you draw a significance threshold is also the ordering that guarantees your final impression is formed by the least alarming item you could generate.

The list doesn’t just permit false confidence. Working in descending order is a mechanism that manufactures false confidence, one diminishing item at a time.

How we consume the risk register

Another failure mechanism doubles down on similar effect from the other side. An older 1978 paper by Fischhoff, Slovic and Lichtenstein⁴ shows how consuming a risk register that have been cut-off creates biased notions of completeness, and distorts relative line item importance.

They gave people some basic fault trees (e.g. “a car failure to start causes a delay”), with major branches pruned out of some versions. In one experiment, subjects were largely insensitive to what had been removed. When shown a tree without branches that the control group marked for over 40% of the failures, subjects put just over 20% into the “all other problems” catchall. Worse, taking one branch and merely presenting it as two separate branches increased how important people judged it to be (though the branches combined scope didn’t change).

Notice what happens here: a meaningless inflation of the enumerated items increases their perceived importance, while making the residual catchall category less important (even though its contents hasn’t changed). So consuming a more granular register makes every listed risk feel weightier, which makes mitigating them feel more consequential, while the “everything else” catchall becomes an ever diminishing stub at the bottom of the list.

Put differently, attention migrates into the list of known unknowns. The list gets handled. And the unknown unknowns that were never on it are shoved into a small line item, then into a buffer, then into slack, then into the reason we’re confident. What we now know should be read as: into a red flag.

As an amusing sidenote, the one and only footnote in the technical report is such perfected irony I had to quote it as is:

To our chagrin, the first starting failure encountered by anyone connected by this study was not included in our fault tree: an ignition key not turning because the steering wheel lock had caught the ignition switch.

<chef_kiss.png />

Crushed from both ends

Let’s assembled all the pieces:

• Fischhoff is about presentation, how a cut-off risk tree drawn by someone else primes and biases your view.
• Jørgensen is about generation, what happens when you build the list yourself.

When read together, you get the perfect storm: whoever draws the tree controls what feels important; the residual unknown is the one thing nobody can correctly estimate, but everyone claims to have scoped; and if you generate the tree yourself you also come away disproportionately reassured by its smallest item, mistaking its size to be a hard ceiling on the size of the unknown.

The actual real world empirical data makes this concrete and vindicates the analysis, with projects exploding far beyond their risk mitigation buffer⁵. Not any of the projects I was in, mind you. Just speaking hypothetically, of course.

Epilogue

So, what now? Should we stop planning for risk mitigation? Of course not. Nor should you stop trying to figure out when a surprise exam is most likely to take place. Both theory and our actual psychology rig the game against us, but playing is still the better option.

What’s left is a habit. Imagine it’s the Tuesday when the register goes green and the room feels relieved. Relief is the wrong response; the students were relaxed too, their reasoning was excellent, and the exam was still on Wednesday. Instead of relief, make a habit of saying the quiet part out loud.

Don’t say “no significant risk remains”; but rather “conditional on our analysis being consistent/complete/good enough, no significant risk remains”. Sounds stupid, but this creates a dynamic where thinking about the antecedent doesn’t seem a completely ridiculous waste of time. Concretely, it may push you to take two green items whose owners have never been in a room together, and ask whether both of their mitigations can be true at the same time.

It’s still the case that If P(lanning), then Q(A). Just make sure the mechanisms that go into the P are good enough.