(meme image edited with Gemini AI)

The Rat in exploratory testing

Buckle up buddy, this is going to be a wild ride. In this post we’re going to walk the border of rationality, attempting to touch the reality beyond our mechanisms of perception, while trying not to get bit in the process. It’s going to be unorthodox to say the least, but I think we’ll come through the other side with an interesting point of view and some unique insights. And for our guide, who better than one of the least stable philosophers of the last 50 years - Nick Land.

As with most of these posts, we’ll spend the 1st half going through some philosophical foundations (or in Land’s case, try to wrap our minds around his somewhat irrational rationality); then use the 2nd half to map them onto a modern software setting, extracting some insights and illuminating metaphors. If you’ve heard of Land, maybe you know he wrote quite a bit about AI and cybernetics, but we won’t use much of those parts of his work at all. Instead, we’ll focus on his early and more general philosophy, which again, will be quite a wild ride to go through.

Prologue

A bit of myth building biography: Land got his PhD in 1987 (on a Heidegger poem analysis thingy), then taught philosophy at Warwick for a decade. In 1995 he joined a radical research group with the cyberfeminist philosopher Sadie Plant - the CCRU. There they combined copious amounts of drugs (what he himself referred to as worshipping the amphetamine god), thoughts on cybernetics, the occult, death as a means of discovery and more. Less glamorous form of the craziness surrounding some of the founders of JPL, but just as radical and esoteric. The CCRU got disowned by the university (not that it was ever fully owned in that sense anyway), continued anyway for a while, then the whole thing ended in 1998 with Land’s resignation and his mental health breakdown.

Land disappeared, resurfacing in Shanghai, spending the 2010s writing reactionary political theory around the Dark Enlightenment. Authoritarian states show up there as exemplars, as Land sees them as engines for his anti-human fatalism and a way to accelerate humanity’s impending doom, which he’s keen on doing. Well, maybe. It’s hard to give a definitive answer around this, as Land shifts between anti humanism and straight up being anti human.

BTW, I personally hold the brave, unique and controversial political stance of being pro humans 🤷‍♂️. I do not share, support, or strictly speaking understand much of Land’s political and fatalistic-teleological point of view. I do, however, find his early anti-rationalistic anti-philosophical ideas wonderfully unique and surprisingly productive; for sure worth exploring.

With that out of the way, LFG. If you got any self supplied amphetamines, now would be the time to take them (joking, of course [though maybe not entirely]).

Land’s response to Kant’s dichotomy

Kant’s legacy

One of Kant’s major contributions to the history of thought was providing a structural account for the age old concept of “a world behind the world”. Almost all cultures have some sort of framework in which there’s a pure, higher, more substantive reality that peeks through our perceived slice of “reality”. It usually centres around the spiritual (souls, ghosts, life energy etc.), and sometimes it’s even more fundamental or abstract than that (Plato’s true forms, for example).

Kant’s brilliance was detailing this general concept by providing an underlying mechanism - our mind’s faculties of perception, and exploring the many repercussions and consequences that flow out of this. Kant’s “real” reality is the noumenon, the thing-in-itself (Kant’s distinction between the two isn’t important for our scope).

What is it? We’ll never know, as we can only perceive things using our minds, and the very mechanisms of our perception are like tinted glasses that we can never take off. The world as it’s presented to us has already been coloured and shaped by our perception into a realm of phenomena. Kant gives specifics (categories of understanding, intuitions of space and time), but the important thing for us is the split between the real noumenon and our construction of phenomena to be perceived from it.

All the things we could ever know are those that comport to our mind’s mechanisms (read: that were produced by our mind’s mechanisms), and those are the domain of phenomena. We are forever destined to walk blind to the reality of noumenon, trapped in the cage of our own constructed phenomena. For our scope, the only thing Kant seems to allow about noumenon is to posit it, as from the existence of phenomena, we can reverse engineer the existence of noumenon.

For all of Kant’s positive claims, his central thesis robs us of any direct knowledge of reality. Quite depressing when you think about it (As usual, I may have overstretched the negative for dramatic effect).

Land’s response

Land’s reaction to the fundamentals of Kant’s philosophy is exactly what you’d expect from an amphetamine fuelled, occultist philosopher. Well, maybe not in the sense that it’s expected, but in the sense that it’s weird and provocative. Land keeps the architecture, but inverts and subverts the focus. He’s less interested in the caged human enlightenment and rationality, and much more obsessed with the anti-anthropocentric outside with all its horrors and gnashing of teeth. Land’s Outside is active. It bites, hence his idea of fanged noumenon.

Kant’s unknowable Outside is completely passive. It just sits there, beyond the fenced rationality of your mind, being unknowable all day. Our caged rationality is also passive when it comes to dealing with it, merely ignoring it as it’s out of reach. In contrast, Land’s unknowable is active and has agency, and human rationality is also active in countering and keeping it at bay.

Land’s Outside doesn’t wait to be approached, rather it arrives unannounced. It intrudes, disrupts, corrupts, infects. It reorganises and hijacks systems that we thought were our own. This is aptly described through the cosmic horror of Lovecraft, that Land invokes by name. The world behind the rational and the knowable, that violently brings madness and calamity to all who come into contact with it; without any comprehensible purpose or intention. It’s unthinkable in the literal sense.

Fun stuff.

Land’s view doesn’t cancel the status of rationality or makes Kant’s project wrong, per se. It does make it provincial, meaning local and contingent; a small (unimportant) slice of what’s real, that mistakes its own coverage for completeness. The rational map seems complete, but it doesn’t show where the map ends, because the whitespace outside it can’t be represented by it. Rationality can’t even draw the borders of its own domain.

As noted, under Land’s view, the Outside isn’t the only thing being active. Our inside is also more than just passively perceiving and organising our caged rationality. Land shines a light on an apparatus that keeps our image of the world manageable and comprehensible against this fanged noumenon and the cosmic horrors of the Outside. In Land’s vocabulary, this is the human security system. It’s not just our cognitive limits but the broader concepts, ways of thought, sense of personal identity and of being a coherent persistent agent with intentions, all of these and more. The water we swim in without noticing.

Land marks all of these as part of a defensive structure that helps us keep the Outside out. Not by refuting it or debating it, or coming into contact with it, but by making it unthinkable. By ensuring there’s no category for it to land in, so that when we do come into some form of contact with it, it registers as noise, error, or nothing at all. Note this has nothing to do with truth, but rather with mental safety (and a theatre of safety at that). The feeling of being in control.

On feedback and intention

Land is also obsessed with the concept of positive feedback, i.e. acceleration. Meaning that when a deviation occurs in a system, it will be amplified, sometimes at the expense of the system’s sustainability and even its continued existence. Viral trends, bank runs, a nuclear fission chain etc. Human engineered systems (technical or social) usually treat positive feedback as a fault state, or at the very least a threat.

No, the human systems prefer negative feedback, which stops deviation in its tracks through some sort of corrective measure. Think of thermostats, biological homeostasis, process controllers, a well functioning immune system etc. This is the “control” in quality control. Land sees all of this as staleness, incapable of producing anything new. A system built on negative feedback can (almost) never discover, but only confirm.

Land ties positive feedback with the death of human intentionality. Putting his theory in a more grounded manner than he would: given a complex enough system, positive feedback is destined to occur and hijack it. This means that any intentions or purposes the system was built for are merely starting inputs; the process outpaces them, and will discard them the moment a more effective path is formed.

Complex systems aren’t our systems. They’re their own; we’re just observing them, mistaking ourselves to be what the story is about. Land pushes this into full anti-anthropocentrism. Not “humans are less important than we think” as a moral position, but as a methodological one: human purpose is the wrong explanatory frame for what large, complex systems do (even when it’s a system of our own makings). This goes for economic, social, and most importantly for us, technological systems.

The Rat(s)

One last piece of Land’s thought that will allow us a better mapping in the next section. Besides Lovecraftian and old folklore horror, Land uses other forms of pathology as the metaphors for his fanged noumenon. Contagion, infection, vermin; things spreading through a population of a system without anyone’s consent. All those can be distilled into the shape of the Rat.

The Rat is explored by Deleuze and Guattari, from whom Land borrows heavily. They see the rat (and the wolf) as a multiplicity, i.e. a pack. You do not encounter a rat, you encounter rats; the singular is mistaking the perception for the reality. And rats form burrows: shelter, supply, movement, evasion, breakout. They propagate at the edges, in the whitespace of the chart. This makes the visible instance not the phenomenon at all, but a sample from a population we can’t see.

The Rat also plays a part in Lovecraft’s The Rats in the Walls, in which a man moves into an ancestral house and hears scurrying inside the walls. No other human can hear it, and he’s told repeatedly that there’s nothing there. He investigates anyway, gets down beneath the foundations, and discovers that the house was built on something considerably worse than a rodent problem (hey, Lovecraft gonna lovecraft 🤷‍♂️). Land cares most about the structure of this story. It’s not a story about rats, but about a person whose senses are working, whose institutional support is absent¹, and whose knowledge is inadmissible until it’s too late. The horror isn’t in the walls; it’s being right about something you can’t comprehend, categorise or make intelligible, and having no standing.

There’s a lot more to Land, obviously; even without venturing into the more outlandish territory of “AI fatalistically creates itself through the fictional narrative of AI on the backs of humans-as-a-substrate”. However, this has already been quite a wild ride, and we have enough of Land’s ideas to map onto the software industry, and extract some insights in the comfort of our caged rationality.

Mapping abstract thought to the (surprisingly) concrete

The last few paragraphs may have seemed like unnecessarily getting lost down a rabbit hole (or rat hole), but actually they’ll provide us with a solid anchor to start from. After all, we’re dealing with vermin infestations throughout all of our professional lives. We call them bugs. We talk about squashing them and exterminating them, and the process of removing them is debugging.

The etymology works whether you focus on its origins with Edison or on its spread into popularity with the moth-in-the-relay story. Either way, our vocabulary isn’t neutral, and it carries ideological baggage. Specifically, in this case, that bugs are a foreign contaminant. They do not belong here, and so the goal state is to have zero of them.

Which is wrong. Sure, we can all say why it’s trivially practically wrong (effort, too many testable states, etc.); but Land’s perspective, echoing Deleuze and Guattari, gives us an additional non-trivial point of view. If bugs are mapped to the rats in the human city, then they are not invaders, as rats are endogenous to cities.

Rats exist at urban densities for reasons that are entirely architectural and inherent to the system itself: the sewers, the waste stream, the cavity walls, the food surplus, the subfloor voids, the warmth. No city, no rat population. The city doesn’t have a rat problem imposed on it from outside; it produces its rats, continuously, as a byproduct of being the kind of thing it is. It’s a case of a complex system not adhering to human intent and purpose, but treating those merely as some of its initial inputs.

Defects are not foreign external matter in our architecture that we can somehow block or exclude. They’re the architecture’s own metabolic output that doesn’t have a category to land in. Complexity is the food supply, integration points are the sewers, and of course the defect count scales with system size, as in this sense we’ve built more habitat.

Let’s sprinkle some more of Land’s view. If defects are doing the biting, what plays the role of our security system, producing a sense of control and comprehensibility? Well, it’s our entire narrative. Calling the defect foreign locates the problem outside of our design entirely. It preserves the felt integrity of the thing we’ve built (a system that serves us). The security system can also be seen in our planned test suite, as a means to reassure us and give us a sense of control over an out of control complex reality.

Cool story bro, does it come in useful? Well, surprisingly, we can use this mapping to ask some very useful questions.

Where do rats live?

Rats live in the parts of a building that were never really designed as a named “thing”: wall cavities, subfloor voids, the space above a suspended ceiling, the run between where one contractor’s responsibility ended and the next one’s began. Nobody drew these on the diagram; they’re just the leftover volume between the things that were drawn. Our map of the rational never maps the nameless, uncategorised whitespace. It can’t even draw its borders.

Our system architecture diagram has exactly the same property, and this gives us a usable test-design heuristic: read the inverse of the diagram. The boxes are where the thinking happened. They have owners, tests, and someone who can explain them. You may find a rat there, but the pack is somewhere else. The pack’s habitat is the arrows and the whitespace surrounding them. Even more to the point, the habitat is especially in the implications of the arrows.

What happens when this call is retried. Who owns the timeout. Which side handles the partial write. What the contract says (and more importantly, doesn’t say) about ordering, and whether both teams read the same version of it. The gap between “the queue guarantees at least once” and “the consumer may perform many reads”, which is a sentence two different people take to be the other one’s job. Nobody specified those regions because they weren’t regions. They were the connective tissue that never gets drawn. So, read as negative space. The blank between the boxes is not empty, just unlit.

That’s the fanged noumenon in its most domesticated form. Your specification is the phenomenal domain. It’s the apparatus through which the system is permitted to appear to you, and it decides in advance what will even count as behaviour. What it can’t represent doesn’t stop existing, it just stops being visible until it shows up in production with its own agenda.

Can our tests find the rats?

Land’s metaphors also provide a useful lens to examine our test suites through. When a defect escapes the usual question is why didn’t the tests catch it. The Landian question is: what does this apparatus protect us from knowing? Our test suite is part of our security system. Its domain is the knowable, comprehensible world of phenomena. There are some bugs there, sure. But that’s not where the pack that bites lives.

From this point of view, the test suite isn’t primarily an instrument of discovery, but an instrument of reassurance, and the reassurance is its most reliable output. Coverage percentages, pass rates, the regression burndown - they are not predictions about reality, but the constructs of our manufactured model of reality. The perceivable output of our methodological thought processes.

The honest reading of a green pipeline is not “the system is fine”, but “the system did not do any of the specific things we had already thought of”. That’s not nothing, for sure, but it’s a statement about the detector, not about the thing being detected, and for sure not about what could not be described in our vocabulary. The assurance in quality assurance. You’ve heard this tune before, for sure (Dijkstra comes to mind), but now Land sets it as part of an even broader and deep context.

We can cash this out further. The (regression) test suite is a form of negative feedback: It has a set point, the specified behaviour; it detects deviation from that set point; its output triggers a correction that returns the system to the set point. It is a homeostatic mechanism, and it’s a good one. But it follows from the structure that a regression suite is incapable of discovery (intensified for dramatic effect etc.). Its purpose is to guarantee that the outcome is the one already specified. It cannot tell you about anything that wasn’t in the set point, because “not in the set point” is precisely what it’s built to filter out.

This immediately cast exploratory testing as a form of positive feedback. Think about it: something looks slightly off (“smells wrong”, i.e. we’ve touched a contamination we can’t describe). The correct response, and the one every good tester makes without being told, is not to correct back to the plan. It’s to increase the gain. Push on it, vary it, give in and see what else smells bad. Amplify the attention and vary the disruptive signals until something breaks that you can hold up as evidence.

This also reframes the argument about ratios. The question “should we be 70/30 scripted to exploratory” is malformed, because these aren’t two flavours of the same activity. They’re opposite control regimes, one buys you assurance, the other touches the reality of your system (even when it can’t describe it). Remember that a negative feedback system can (almost) never discover, only pretend to control.

Why pretend? Well, Ross Ashby’s law of requisite variety formalises this: only sufficient variety can destroy (read: absorb) variety. A controller needs at least as many states as the the disturbances in the system it’s trying to control, and as we all know very well, “we’ll enumerate the test cases” is an arithmetical impossibility. You do not fix this with more test cases.

You fix it, to the extent it can be fixed, by putting a variety-generating instrument into the loop; for example, a human, touching reality in a manner he cannot comprehend or explain, but with a license to deviate, amplify and destroy. Yes, there’s a rising tide of automated variety-generating solutions, and machines are for sure excellent at generating variety at scale, but they still need a stated oracle. They find crashes because “crashed” is pre-specified; they finds violations of invariants someone already wrote down. The human’s irreplaceable function is oracle generation. Meaning humans can notice that something is wrong when no one has said what “wrong” is, and when “wrong” is so undefined that describing it would be beyond the boundary of our rational mental vocabulary.

How do rats react to our tests?

Remember, Land’s outside is active. Well, how do rats react to being hunted, not to say Debugged? Rats develop resistance, biologically and behaviourally. For example, Warfarin resistance has been documented in wild rat populations, and it’s a predictable consequence of applying a selective pressure to a fast-breeding population. You kill the susceptible ones and those who wondered to the middle of a lit room. The survivors are, by definition, the ones your poison and catch strategy didn’t work on. Then they breed, and the pack learns to avoid the thing that killed the last cohort. This is how you get to perceive a rat inside your room, while the pack bites from the outside.

Our test suite is a selection pressure on our defect population, and it works in similar ways. The test suite kills the defects it can detect and categorise. That’s the job and it does it well. However, from our new point of view, this means the defects surviving in a mature codebase are not a random sample, but rather the subset that your detection methods don’t and can’t catch. A software version of Beizer pesticide paradox, that Land can give more structural foundations to: patterns that produce detectable defects get corrected by negative feedback and stop recurring; patterns that produce undetectable ones persist, get copied and built on.

So our test suite runs negative feedback on the system’s behaviour: it detects deviation, corrects back to spec, and makes sure the observed status stay put. But it also runs positive feedback on its own blind spots. Every suppressed symptom enlarges the unlit region; the enlarged unlit region improves the survival odds of exactly the defects we can’t see; more undetectable defects produce more symptoms that have to be suppressed, because by construction we can’t diagnose them.

This also ties very well to the other side of the equation: a severity-one incident in production is the rat that came out in daylight. Not because prod is special, but because daylight emergence indicates population pressure. Something that had been living quietly inside the walls has been pushed out into visible space and it bit you. The incident is information about density, not just about itself.

No city has ever exterminated its rats². If bugs are foreign contaminants, then killing the ones you find is unambiguous progress and “zero known defects” is a coherent goal. If bugs are an endogenous population under selection, “zero known defects” is a statement about your knowledge, not about the population. Every mitigation that suppresses a symptom is just another form of evolutionary pressure.

How do our tests react to the rats?

But wait, you say. If the rats bite, this means we and detect them! Just use that to comprehend where they are and get rid of them. Remember, Land also ascribes activity to the inside, specifically for the rational immune system that reassures and controls. This is exactly where this system priorities the theatre of control and safety over truth:

• The flaky test gets quarantined rather than diagnosed. The intermittent behaviour it was noisily detecting is now undetected, and still there.
• The assertion gets loosened because it fails “too often”. You have widened the set point until the defect fits inside it.
• The retry gets added to the call, so the underlying timing problem stops surfacing. It didn’t stop happening.
• The warning gets suppressed because there were too many to triage, and you cannot afford to get overwhelmed.
• The test that “keeps breaking whenever we change that module” gets deleted, on the grounds that it’s high maintenance. But it was high maintenance exactly because it was meaningful.

Every one of these is a rational local decision made by a competent person. Collectively they are both a breeding program and an anaesthetic. Whatever your team’s most common mitigation pattern is, that pattern is defining which defects get to survive in your system, as well as which blindspots it has.

So, what CAN be done?

Leave the comprehension to the rationally designed boxes in your diagram, be a tracker (read: exploratory tester) in the whitespace. A tracker follows an animal they cannot see, whose interior life they have no access to. They read droppings (corrupt data), bent grass (bad API call), a print with a particular gait to it (strange sporadic lag). There is no comprehension in the contact with the thing and the gut feeling it evokes, but rather the smell of things and a sensitivity to signs.

Exploratory testers don’t work according to what the system was intended for or the way the architect designed it; they recognise those were some of the system inputs, but that it has outgrown them. The system’s behaviour might very well be beyond human comprehension, and that’s fine, because we can still touch it in unstructured, indescribable, incomprehensible ways. The architect has a comprehension of the diagram. The exploratory tester has touched the thing in itself.

That’s all well and good, but in order for this to have any tangible effect, we need to make sure the exploratory tester, unlike his Lovecraftian counterpart, has institutional standing. A tester who says “something is wrong here, I can’t tell you what and I can’t tell you why, but I’ve been in this thing for two days and it’s wrong” usually has no standing whatsoever. Not because people are stupid or dismissive. It’s because comprehension is the currency of credibility.

So the exploratory tester needs to be met with a disciplined inference process taking those pre-rational signs and producing claims which can be checked. The tracker’s authority is empirical and entirely earned by contact with reality, even (and especially) when it’s indescribable. The process transforms all of that into something the institution can record and has some taxonomy for.

Again, the testing community has been saying exploratory testing should have more standing for quite a while, our Landian point of view just gives this recommendation more context. We can now say this isn’t a matter of ratios of efforts, but of allowing ourselves to be less caged in our own rationality. To borrow from Wittgenstein, who would have loathed every word of Land’s: what cannot be spoken of might still be shown (he would have also loathed this use of his saying 🤷‍♂️).

This can be cashed out into different practical mechanisms around session note taking, unformulated concerns registers, pair testing, density metrics and more. You can get those from many other test professionals wayyyyy smarter than me. This blog is more about crazy points of view, and Land for sure gives us that.

Epilogue

Here’s a concrete summary of what we’ve been through: your test suite is not a window into the reality of the system you’re testing. It’s an immune system, and like every immune system, it’s defined by what it can recognise. It kills what it recognises, it’s blind to what it doesn’t, and it produces confidence irrespective of whether there’s anything to be confident of. Also, it has been quietly selecting your surviving defect population for as long as it’s been running.

That doesn’t make it bad, only partial, in both senses: incomplete, and biased towards a very specific point of view. It’s up to you to find the time and resources to be deliberately biased towards the other point of view, to counter and complete that. I’m referring here to the irrational side that can’t be put into words. When someone on your team says they can’t explain it but there’s something just wrong over there, don’t treat their handwaving as an absence of evidence. That’s a tracker reporting on something raw. They just can’t put it in the ticket because your taxonomy has no box for it.

They’re the only person in the building who has had physical contact with the thing in itself. After you’ve given them the respect they deserve, make sure to also give them a tetanus shot (or some amphetamines, if you have those laying around).

Weird guy Land. Still, surprisingly insightful.