Kuhn! At the Software Industry - Part 1 - Setting the scene

In this two-part series we’ll trace the dynamics of a generic software project’s descent into crisis testing, through the lens of Kuhn’s normal-science versus crisis-science distinctions and what they suggest about methodology as such.

The first part will walk through an overview of Kuhn’s work, and suggest a mapping of his vocabulary onto a modern software environment. The second part will add some much needed nuance through two post-Kuhnian voices making their cases for and against method.

Setting the stage - your dashboard is a Wicked-themed bright green, while the customers are opening ticket after ticket full of impossible behaviours. We’re looking for an account of how a competent team gets here that doesn’t trivially reduce to “somebody should have written more test cases”.

Let’s boogie.

Prologue: Popper sets the stage

A repeating claim in the blog (well, I expect it to repeat) is that if someone knows any philosophy of science at all, they probably know about Popper and his concept of refutation. That’s doubly so in the testing world, as a handwaving summary of Popper usually maps quite well to a vague notion of what testing boils down to.

Let’s start with a quick rundown of Hume’s riddle of induction, which is the problem Popper was reacting to. Simply put, Hume famously claimed no finite number of observations can support a universal claim, or any kind of projection from the observed instances to the next unobserved instance.

So take the textbook version of this from Hempel: even if every piece of copper that was ever observed conducted electricity, this supposedly gives no rational justification to conclude that the next piece of copper will also conduct electricity, let alone generalise to the rule that all copper conducts electricity. Note - Hume claimed there’s NO justification. Zero.

Induction, according to Hume, has no rational justification. Yet science, which appears to be built out of and on top of induction, has an excellent track record. Not to mention that if induction is broken, even getting out of bed becomes straight up risky - how can you know you won’t just float up to space? Without induction, all bets are off¹.

Throughout the centuries there’ve been countless attempts to answer Hume, and we’ll undoubtedly cover a lot of them in the blog. Popper’s answer is somewhat radical and can be summarised as “Hume is right, but it doesn’t matter”. According to Popper, science doesn’t (or rather, shouldn’t) run on induction at all.

Popper’s view is that science, in as much as we see its aim as the pursuit of true universal theories, can’t pursue this goal inductively (since no logically valid principle of induction has ever been devised, i.e. Hume’s riddle). He therefore proposes a method for our acceptance and practical demarcation efforts - a method that relies solely on deduction. Remember that logically, no amount of finite evidence can confirm a universal claim through deduction; but a single piece of counter evidence can refute a universal claim deductively (Quine will have something to say about that, but let’s keep it simple).

So the foundations of the scientific method he proposed (“science”, being a somewhat open-ended term that only gets us so far) are conjecture and refutation. In VERY handwaving broad strokes, it would go something like this: Propose something bold, derive what it logically forbids, go looking for the forbidden thing. Found it? Great. As the theory and the observation can’t both be right, if we accept the forbidden thing, we have to reject the theory as having been falsified. Didn’t find it? Well, we take the theory not to have been falsified for now. Not proven, mind you, just not yet falsified. Popper set this up as a normative standard for science, alongside other considerations the will be discussed more seriously in another article. For now, a sufficient first approximation of Popper’s view for our needs is that an empirical theoretical system should be deemed scientific if it can be refuted. Meaning that it forbids a state of affairs we have the means to show actually is the case. If nothing could falsify it, Popper excludes it as an empirical statement of science.

(The above is a far better version of the initial text I wrote, which was an extreme TL;DR to the point of being libel. Special thanks to Jonathan Clapsaddle for volunteering his time to make this article less of a caricature of the ideas it references [to be clear, even the above is a shortened version of what Jonathan specified])

Popper’s main concept is EXTREMELY popular, for several reasons. First and foremost - it’s great. Provocative, relatively fresh concept, aptly put. Not without its problems, of course, but overall, great. In addition, it compresses well to two sentences without appearing to lose anything. As these things always are, it does lose a lot under the surface. Even the above summary doesn’t do justice to the full breadth of Popper’s ideas, let alone the bumpersticker version most people know. And finally, it casts the scientist as a hero. From this point of view, the scientist isn’t tediously accumulating small detailed results, but rather bravely submitting their most cherished ideas to criticism and feedback.

Software testers are especially susceptible both to the compression and the flattery. The whole grammar of the profession is Popperian:

• Testing cannot demonstrate the absence of defects, only their presence.
• A passing test proves nothing; a failing test proves something specific.
• We are the function that tries to falsify the claim the development team is making.
• <your_favorite_overhyped_dramatic_platitude />

Cool story bro. Now let’s see how it all falls apart in practice.

Kuhn and the paradigm

Popper was not the only one making normative claims on what Science (with a capital S) is or should be. The mainstream philosophy of science at the time logically reconstructed scientific theories into rational arguments, supporting evidences and justified conclusions and judged them accordingly. This whole approach of prioritising armchair normative philosophical analysis over the descriptive reality of how science was actually done got turned on its head in the early 1960s.

Thomas Kuhn published his most famous work - The Structure of Scientific Revolutions - in 1962, and forever changed the lens science was analysed through. As a philosophy of science course I once took noted, some say it’s the most revolutionary philosophical book of the 20th century. Not a claim one would have an easy time rigorously defending, but still a claim that’s hard to straight out deny. Kuhn’s work is also part of the select few that had a massive effect on other fields of thought, from social sciences to management practices.

Kuhn’s work presents itself as a work of history first, with the philosophy as somewhat of a residue of the historical account. And while Kuhn’s work broadly undermined the relevance of armchair normative philosophical analysis, it first and foremost demolishes Popper’s point of view, specifically.

Kuhn invited us to stop worrying about what’s supposed to happen, and instead to start looking at what actually happens. How do scientists actually spend their days? Well, they do lots of different things, but none of them is trying to refute the ruling theory of their field. Quite the opposite, actually.

Kuhn noticed actual working scientists take an enormous inherited apparatus entirely for granted, and spend their careers on very scoped, detailed problems that the apparatus makes tractable. All of this happens under the ruling theory or within it, never in opposition to it.

If any anomalous results are encountered, they are far more likely to be attributed to bad technique, bad luck or some misunderstanding than to an issue with the theory. In effect, it’s not the scientists who are testing the theory, but the theory that is testing them. Meaning that if an experiment goes wrong, it reflects on the scientists involved, not the theory they follow.

Kuhn called this overall inherited apparatus a paradigm. Wait, no, he actually called the ruling theory itself a paradigm. Oh, hold on, no, it’s the set of shared values for judging what makes a good theory. Or is it one concrete solved problem that everybody copies?

If you ever heard the term paradigm being used a thousand different ways, Kuhn is the one to blame; or at least, he’s the first to blame. He called more than a dozen different and overlapping things a “paradigm”.

That last one (concrete solved problem) is worth expanding on. Kuhn eventually decided it was the one he actually cared about, and later split his own paradigm term in two: the broad sense, which he renamed the disciplinary matrix, and the narrow one, which he called exemplars. These are the concrete solved problems that practitioners learn to imitate.

Nobody learns a science from its principles. You learn it by grinding through the worked problems at the end of the chapter until you can look at a new problem and see which of the old ones it resembles. That skill is a trained perception, not a rulebook. It’s tacit and acquired by imitation, not explicit rule-following. This is what makes a paradigm not merely unquestioned, but largely unstatable, and therefore mostly invisible from the inside.

For most of what follows I’ll use paradigm in the broad sense, meaning a theoretical framework of beliefs, world views and social practices, plus everything that this entails. This includes what entities there are; which questions are interesting or even considered answerable; which techniques count as legitimate; what a solution has to look like before it’s recognised as such; etc.

The picture Kuhn paints is extremely anti-Popperian, and at first glance seems somewhat anti-science, or even more broadly anti free-thought and progress. Kuhn however insists that this dogmatism is not a bug, but a feature. Progress necessitates solid foundations, and scientists can push absurdly deep on detail precisely because nobody questions the foundations every morning.

The unheroic, uncritical, taken-for-granted framework is the thing that makes cumulative results possible. This is true first and foremost in a practical sense, as there are only so many hours in a day. Spending most of them trying to refute the basic tenets of your theory would leave no time to gain actual new knowledge. But if you’d like a more philosophical justification, this is also true from a philosophy of explanation point of view, as an answer can only exist within the scope of accepted prior beliefs.

All of this plays nicely into another central Kuhn distinction - science (normal science, but we’ll get to that nuance in a bit) doesn’t answer open questions, but solves puzzles. A puzzle is something you have good reasons to believe is solvable, by known means, with agreed criteria for recognising the solution as such. Paradigms define what questions are puzzles (hence solvable), and denote other questions as unscientific. So for example, Descartes demanded a mechanism for how forces like gravity carry across reality. Newton accepted action at a distance as a brute fact of nature and moved on. Since Newton’s paradigm won, the question “what causes gravity” stopped being a legitimate question (until Einstein, but that just proves the point).

Puzzles can be assigned, funded, estimated, and completed by a competent person. Open questions can consume careers and produce a shrug. The former is science, the latter is philosophy (or crisis science, but we’ll get to that in a bit). A mature science is one that has manufactured a supply of puzzles out of a former landscape of open questions. That’s why science has progressed so much, while a modern philosophical debate may drum up an argument from a millennium ago.

It’s all fun and games until reality sneaks in

So, are we to salute our new paradigm overlords, solve our paint-by-number puzzles and consider ourselves lucky? Not so fast.

Popper kept us guessing if our theories were true or not. Not ideal, but that’s a fair price for working around Hume’s riddle. With Kuhn, however, truth doesn’t seem to play any significant role. Oh, there’s plenty of lower-case truth going around, i.e. truth as defined from within a paradigm; Truth with a capital T, i.e. reality - not so much. As a paradigm defines what is considered a fact, how to conduct a valid inquiry, how an answer is formulated and evaluated etc., any truth claim we can make seems to be scoped to the boundaries of the paradigm, and has little to do with anything outside it².

This has two implications. The first has to do with cross-paradigm communication, that the more radical readings of Kuhn would have you believe is flat out impossible. To borrow Kuhn’s language, there’s an inherent incommensurability between paradigms. Since paradigms determine the actual meaning of terms, when scientists from different paradigms talk about their disagreements, they may use the same words, but the words carry completely different meanings and entail different consequences (not surprising as the words gained their meaning through absorbing different exemplars). As a radical example, when a heliocentric person talks about a sunset, they mean something completely different than what a geocentric person means - one sees the sun moving and actually setting, the other sees the horizon raising to meet it. Kuhn thinks this cannot be resolved or avoided, and that people from different paradigms are destined to talk past each other; there’s no “objective” point of view they can use to bridge this gap.

The second implication deals with the fit between a paradigm and the actual real world. Paradigms are self-contained, self sustaining entities, but the real world is still out there, and has an annoying tendency to sneak in as anomalies. An anomaly is an observation that doesn’t agree with the ruling paradigm. That’s fine - part of doing science is exactly exploring and resolving these anomalies; and most of the time that’s exactly how things go (remember, Popper would have had us dropping the theory at the first hint of a discrepancy). In fact, even when a stubborn anomaly continues to resist explanations, usually it’ll just be chalked up as a quirk, while the paradigm carries on.

So far this is somewhat deflating, but not catastrophic. Kuhn paints a historically accurate³, recognisable picture of professional life: skilled people working productively inside a settled tradition, with truth assessed by the tradition’s own standards. It’s the next part of Kuhn’s analysis that not only deflates, but utterly destroys the normative depictions of science of his days.

Crisis

Anomalies keep on coming. Normal science, which is science according to the traditions, terms and ways of work of the paradigm, absorbs them through its well-developed mechanisms: measurement error, contamination, an auxiliary assumption in need of adjustment, a problem parked for future work.

But sometimes a concerning amount of anomalies cluster together and resist adjustment. They will simply not go away. More and more scientists shift their attention to them. Maybe one of them happens to have political pull and funding. And at some point the scientific community shifts into what Kuhn calls crisis. Note that this isn’t an epistemological or methodological crisis. There’s nothing philosophical about it; it’s a brute sociological fact about a community’s confidence in its practices and techniques.

Crisis science mode is the Wild West. The rules loosen. Practitioners entertain ideas that would usually have been dismissed as radical or ridiculous. Philosophical debates about fundamental entities and constructs come back with a vengeance, because foundations and axioms become fair game. People invent ad hoc techniques and run incompatible approaches side by side. Almost nobody is doing the tidy puzzle-solving work any more, because there is no longer a consensus about what counts as a valid solution. The ruling paradigm is in decline.

Crisis science gives rise to multiple sub-communities, each with its own home-brew paradigm. Most of them will resemble the declining paradigm, and a few will be radically different. Now, a power struggle ensues. Is this the point where normative, rational, evidence based science or philosophy of science shine? Far from it. According to Kuhn, this power struggle has nothing to do with any of that. If rational, rule based evaluation is always internal to a paradigm, then cross-paradigm evaluation is impossible. There is no “view from nowhere”, as Nagel would later call it, to rationalise from, and all that’s left is an actual power struggle.

To reiterate, in Kuhn’s world there’s no place for the normative or philosophical; it’s all just brute social facts about political power and luck. Kuhn’s account of how a new paradigm wins is not an account of proof. It’s persuasion; the promise of career building; aesthetic preference; institutional clout; and also the reality of generational turnover. Mob psychology, as it would later be derogatorily called.

When a new ruling paradigm rises, the mainstream scientific community adopts it (or more precisely, the adoption is what makes it a ruling paradigm). The practices of crisis science subside, the anomalies are assimilated and resolved, the textbooks are rewritten from the new paradigm’s point of view, and normal science detailed puzzle solving resumes.

Before we wrap our overview of Kuhn’s main ideas, there’s another piece of bad news. You might think that this Wild West brute force mentality of crisis science is worth it, since when the dust finally settles the anomalies that caused all this commotion are finally assimilated and resolved. Well, buckle up buddy, and let’s introduce the notion of Kuhn loss (not one of Kuhn’s own terms, but an apt one that stuck after the fact).

Kuhn notes that a paradigm shift is not straightforwardly cumulative. The new paradigm comes bearing gifts, yes. There are gains, obviously; that’s part of why it won. But it typically also loses: questions the old framework could pose and answer become unaskable, phenomena it explained get reclassified as not really needing explanation and techniques that worked stop making sense. Usually the textbooks are written around Kuhn loss and make it invisible from inside the winning paradigm, but puzzles gained come at the price of puzzles lost. One could try to defend the claim that over time, science manages to solve more and more puzzles, but Kuhn doesn’t seem to want to make that claim or argue for it.

The history lesson is over, now let’s see how it maps to our day to day

You feel good. Actually, this is the best you’ve felt in a while. 163 automated tests, all green; the pipeline badge gives the all clear; exit criteria met; sign-off in the ticket. You release. Time for that 5th cup of coffee you promised yourself you’d cut, but today you’ve earned it. Wait, what’s this? A support ticket arrives describing a behaviour that should not be possible. After lunch the support queue has 15 more. You already know what comes next.

There’s a meeting. Of course there’s a meeting. Someone asks how we missed it. The answer may change in phrasing, but its shape is always the same. It’s the easiest to prescribe and hardest to argue with. There was a gap in coverage, so we’ll add a test case. Maybe even 3, if we really want to show we mean business this time.

A Kuhnian reading of the situation would have us dig deeper, and see the proposed solution as a product of the very paradigm that caused the problem in the first place. The issue isn’t the specific gap, but the team’s conception of the state of the system - their shared mental model. The model wasn’t merely incomplete, it was wrong. Had been wrong for months. The test suite is the last place that wrongness could have surfaced, as it was built out of that very same model.

Before we delve deeper, a word of caution. There are as many ways to do a Kuhnian reading of a situation as there are to read tea leaves, and most of them about as precise. This is especially problematic because Kuhn used his terms so loosely - one can create almost any metaphoric mapping to support almost any conclusion they aim for. So, let’s proceed carefully and ask ourselves - what exactly is the paradigm here?

It’s tempting to say that the paradigm is the test strategy document. That’s not necessarily wrong, but it does make the rest of this trivial and somewhat naive. Instead, let’s take the paradigm to be the team’s shared model of the system and its risks - what and who is the system for; where in it do we expect to encounter issues and where not; what counts as a bug and what is considered a mere quirk, training issue or a feature request; what does “tested” mean (such that a person can say it and be believed); which questions are worth an afternoon. The strategy document mirrors the parts of this model that are made explicit.

The abstract paradigm is made manifest through exemplars. Nobody learns testing from first principles. You join a team, and as part of the onboarding, someone shows you the test cases for an existing feature. You read them, absorb their granularity, tone, how much context they assume, what’s considered worth stating in the expected result and what’s left to the tester’s judgment. Then you write similar test cases for the next feature, then the next one, and so on for the following three years (and probably for the following three employers as well). Same goes for defects. The first time someone told you “that’s not a bug, it’s a training issue”, you learned a boundary that isn’t written at any particular place, and you’ve been using it ever since.

If that’s our paradigm, what does Kuhn’s normal science map to? Well, obviously to the project’s normal testing. Normal testing is puzzle-solving embodied. The regression suite. The parameterised checks on the table structure. The tediously maintained fixtures. God help us, it’s the traceability matrix. What makes the team’s work possible is that the mission isn’t to question “is this system OK” every morning, but rather “does this one specific behaviour match our model’s expectations”. The former is an open question, unbounded and unassignable; the latter is a puzzle: finite, divisible among five people and has agreed criteria for verification. It’s reportable on a Thursday. The whole delivery apparatus of modern software runs on systematically transforming loose open questions into distinct puzzles.

That’s nice, but remember, these puzzles are internal to the paradigm. They are constructed out of the team’s shared model of the system, not out of what the system actually is and how it actually behaves. And by now you know that reality creeps into a paradigm in the form of anomalies. These accumulate, and our paradigm, like all paradigms, has its immune system to absorb, swallow and mitigate them.

We all know the drill: can’t reproduce, closed. Works as designed, converted to a CR that dies quietly. Environment issue - possibly true, unfalsifiable nine times out of ten. Deferred, real, agreed, forever prioritised for the release after next. When a test goes red, we audit the tester, the data, the environment, the flake; we don’t audit the model. The process tests the tester, not the other way around. Each move is individually defensible and usually the correct course of action. Meanwhile, the gap between reality and the paradigm widens.

In hindsight, the sharp break into crisis can be identified by the slowly increasing divergence of two metrics that very rarely appear on the same dashboard. A rise in escaped defects, that’s coupled with a steady (or even increasing) pass rate of formal test cases. When the divergence already presents itself, adding another test case (or 3, if we mean business) won’t do any good, because the shape of our test cases, the terms they use and ways of work they’re built out of and fit into, are part of the problem.

Other common heralds of the coming storm are customers reporting classes of problems that don’t have a 1-1 fit onto the team’s categories. The test suite becoming an empty ritual (ran for show, but with no effect on anyone’s confidence either way). And in true Kuhnian fashion, a brewing crisis is a social phenomenon, meaning that all of these symptoms cumulate into a crisis when everyone has privately stopped believing the test report (while continuing to produce it).

So, things have slowly deteriorated over time until the sudden descent into crisis. What does crisis testing look like? For starters, the test plan is effectively suspended. Not formally at first (that would require a meeting, approvals, reports) - it just stops being mentioned or tracked. A war room channel appears on the team slack with the obligatory themed “project lighthouse” or “tiger team” name. The rules regarding what even counts as testing loosen. People are pulled in from other teams and are given logins and permissions they never should have.

Somewhere along the way someone dares to come out and just say it: forget the test cases, just go and use it like a user for two days. A developer who has never even opened the test management tool finds three serious problems in an afternoon, one of them just by misclicking the back button. Some poor devops junior who has no idea what observability mechanisms are already in place vibe codes a production logs scraping script and it becomes the most valuable testing asset the company ever had.

This is the Wild West, with all the hallmarks of a Kuhnian crisis science: rules loosened; multiple, incompatible ad hoc approaches and techniques running side by side; arguments about the fundamental entities of the system are suddenly allowed and encouraged. It may look like the collapse of discipline, but it is in fact an apt response. A feature, not a bug.

Incommensurability and Kuhn loss - why the argument never ends

The crisis passes. The patch is deployed. And on Monday, the real power struggle begins. One faction of the team wants to keep the crisis exploratory-style work. The other wants the plan back, tightened, with the specific gaps closed and even more test cases than before. They try to hash it out, but according to Kuhn, this is just a theatre of discussion.

Faction A reports: 82% of 240 cases passing, 100% of requirements traced, 14 defects open, 3 high. Faction B reports: we ran 9 sessions across the user flow, and here are the top 5 things that worry us, 2 of which we can’t yet pin down. How do you translate the two reports, let alone judge which one is more valuable or accurate?

You can’t. The vocabulary is incommensurable (well, debatable, but let’s pretend it is for the sake of dramatic effect). Coverage means a proportion of requirements for one side, and something like a vague map of what we’ve learned and where we haven’t looked for the other. Defect means deviation from spec for one faction, vs. anything that threatens value to the other. Done means criteria met vs. questions answered. Risk is either a register entry or a hunch. Well, which is it? Both sides use the same words, but mean completely different things by most of them.

If communication is incommensurable, or at the very least difficult, and there’s no objective point of view to compare and judge the evidence from, who actually decides the new paradigm for the team? You already know the answer from your own experience - whoever’s in the room. Whoever’s last release burned them the most. Where the organisation sits in its audit cycle. Whether the new Head of Engineering came from a bank or a startup and what kind of pull they have with upper management. Kuhn’s answer for the software industry mirrors his answer for science. After all, Kuhn’s whole point was that science isn’t epistemically or normatively special⁴, it’s all just people and the sociology of power dynamics.

In any case, whichever faction wins and whatever paradigm becomes dominant, it always comes at a cost, a Kuhn loss. Swing to exploratory and you lose the ability to answer “which requirement does this verify” (minor gap right up until it becomes a disputed contractual question). You lose evidence that survives the departure of the tester who produced it. You lose a new tester being useful in a week rather than a quarter. And you lose coordination (an estimate everyone knows is soft still lets six teams plan around each other).

Double down on tightening the formal procedure and you lose the thing the war room had, which is harder to name and therefore never makes it onto a slide. A reality-driven broad examination of the system as it is, not as it’s imagined by your design docs or stale planning materials.

Nobody records either loss, because the winners write the retrospective.

Epilogue: Post Kuhnian thought and part 2

Kuhn’s point of view revolutionised philosophy of science, and was immediately followed by many thinkers who sought to detail, oppose or radicalise his work. Some philosophers tried to show that science was epistemically special in practice and theory, despite what Kuhn showed (or because of it, continuing the “it’s not a bug, it’s a feature” defence). Others doubled down on the invitation to self-contained relativism and moved toward a fully sociological account of knowledge.

We’ve seen the need for a post-Kuhnian detailing in our own rundown of the modern software team’s dynamic. Like Kuhn’s framing of scientific work, it paints a nice picture that seems to explain a lot at first glance, but remains somewhat raw and loose (naive at times). It leaves you with absolutely nothing to do on Monday morning except wait for the next collapse.

Actually, that’s not true. There is one actionable item we can already point out. There is someone on your team that can cut through all of the paradigm’s numbing immune responses for absorbing and mitigating anomalies. Surprise surprise, it’s your newly hired tester that hasn’t been trained into your exemplars yet, and they are the cheapest anomaly detector you will ever have. Their naive initial questions (why does this screen even exist?, and wait, was that even supposed to happen?) are being asked from outside the paradigm, and you get maybe two weeks of those.

Cherish them. Most teams spend those two weeks teaching the new person to be quiet.

So that’s one takeaway. We’ll use the next part to add some more actionable nuance on top of our Kuhnian analysis by examining two post-Kuhnian thinkers - Imre Lakatos and Paul Feyerabend. Friends, colleagues and on opposing sides of our discussion. We’ll track their arguments for and against method, and map them onto our day to day software industry setting.

See you next time space cowboy.