If GenAI knows anything at all, it knows you’ve already lost the lottery

In this piece, we’ll ask if Generative AI can be said to know anything. We’ll take the road usually not taken for these kind of questions, and do that from a philosophy heavy point of view, concentrating on the epistemological concept of sensitive knowledge. First we’ll assemble a philosophical toolbox to help us on our journey: an overview of the epistemological definition of knowledge and the recent revolution it underwent, combined with concrete ideas developed around the legal value of knowledge. These specifically will provide us with a template for cashing out the abstract philosophical principles into relevant details.

Armed with this toolbox we’ll unpack the fuzzy question “does AI know” into more defined terms, and examine them in a non-trivial manner. We won’t end our journey with anything practical (no “action items for Monday morning” LinkedIn type of thing here), but will have gained a (hopefully) insightful point of view, and a whole host of ideas to develop it further.

Prologue

Let’s say I flunked math, or fell into a momentary fugue state, and bought a lottery ticket. The chances of winning are a million to one, and the draw is this evening. When I wake up the next morning, before I’ve had any chance to check the results, do I already know I’m holding a losing ticket?

Most people instinctively say no. Even though there’s a 99.9999% chance I am in fact holding a losing ticket, this doesn’t rise to the level of knowledge. Mainstream epistemology agree: even before we reach for precise definitions, knowledge “feels like” it belongs in a different category than believing something that is nearly certainly the case. The upgrade seems unwarranted here. On the other hand, that’s odd when you think it through, because we routinely claim to know a great many things that are far less certain than our lottery ticket losing the draw. There’s a break in intuition somewhere.

That’s an interesting result. Let’s see if a definition can help us sort things out.

So, what actually is knowledge?

Knowledge is one of the fundamental terms epistemology is built up from and around, and it has remained relatively unchanged for most of the field’s existence. The everyday practical definition can be traced all the way back to Plato’s Theaetetus, where it has been temporary suggested to be justified true belief (originally true judgement with an account, which Socrates dismantles). Meaning you know p when you:

1. Believe p
2. p is indeed true
3. You have a (rational) justification for that belief

Thinkers, Plato very much included, raised concerns about the completeness of this definition, but it went mostly uncontested in mainstream epistemology, and all was well. Notice, though, that it does nothing for our lottery problem. At first glance the statistics look like an excellent justification for believing I hold a losing ticket.

Things were running mostly smoothly until the 1960s. As the story goes, Edmund Gettier was minding his own business teaching philosophy at Wayne State University. His colleagues, who recognised his abilities and smarts, urged him to publish something to satisfy the administration. Gettier sat down for a weekend, and wrote a concise 3 page analysis titled “Is Justified True Belief Knowledge?”. In this Gettier nails down the analysis of knowledge that philosophers had implicitly used for centuries, and shows how that can (easily) be refuted. Gettier became famous overnight, published no further papers, and simply continued to teach philosophy quietly until his retirement.

The jaw dropping “hit and run” effect Gettier’s paper had can’t be overstated¹. It forever changed epistemology. So much so that after decades of trying to patch and repatch the definition of Knowledge in its wake, a lot of philosophers somewhat gave up on nailing down a precise definition, and instead shifted their focus to exploring what follows from loosely agreed-upon understandings and concepts of what we’d like knowledge to usually be (notice the amount of hedging and mitigations infused in this mission statement).

To understand the genius and simplicity of what Gettier demonstrated, let’s consider the original definition. What amounts to knowing something? Well, knowledge is a belief I hold (makes sense at first glance), that needs to be true (A-OK so far), and that I have a justification for. Wait, why is that last part important? Well, part of the whole idea of having a separate category for knowledge is the intuition that being considered knowledge needs to be earned. If I’m about to flip a coin, and I call heads, the fact that the coin toss indeed turned out heads doesn’t mean my prior belief can be upgraded to being knowledge. It’s clear that calling the coin toss was an instance of epistemic luck, and if that’s all I have going for my beliefs, rationally speaking, then they haven’t earned the upgrade. So bottom line, the requirement for rational justification excludes cases of epistemic luck from being upgraded to knowledge.

Gettier’s move was to construct cases where the justification is good, structured and rational, but where the belief comes true anyway by luck. Let’s consider the canonical one:

Jones is up for a job. Smith has excellent evidence that Jones will get it: the company president told him so. He also met Jones in the break room, and noticed that Jones got 4 coins as change after buying soda. Smith concludes, justifiably, that the person who gets the job would have 4 coins on him. Oddly specific, but Smith is weird that way. Surprisingly, the company president had a change of heart and gave the promotion to Smith(!), who by sheer coincidence also happened to have 4 coins in his pocket that day.

Smith believed the person who gets the job would have 4 coins on him. That belief was justified, and it indeed came true. Our definition would crown it as knowledge. Epistemically, it’s obvious the whole affair is as lucky as the coin toss. Well, well, looks like we’re in somewhat of a bind.

The reaction to Gettier has been sharp and dramatic. Many philosophers attempted to patch the definition of knowledge in interesting and productive ways. Attempting to forbid justified-but-false steps; moving to causal theories of knowledge; dropping justification in favour of requiring beliefs to be produced by a reliable process, and many others. After decades of insightful but ultimately problematic exploration, some (most?) even dropped the entire project and moved to using contextual scoped detailing of a loose conception of knowledge in order to support more productive queries.

For our purposes, we’ll examine one specific development that rose (mainly) out of this revolution; it remains extremely useful whether or not you subscribe to there being an ultimate definition of knowledge. This development is the concept of truth tracking / sensitive beliefs, and it has the nice benefit of resolving our lottery paradox as well.

On sense and sensitivity

Gettier demonstrated our initial trifecta doesn’t cut it. We need to enforce additional or different requirements to keep epistemic luck from being crowned as knowledge. One neat mechanism is truth tracking / sensitive beliefs. Conceptually, this is one of the mechanisms that ensures a belief is connected to the factual world (and thus when it matches up with fact, that isn’t the product of epistemic luck). One way to phrase this is demanding that if the world were different, our beliefs would also turn out differently. This type of “had the world been different” formulation is what is known as a counterfactual.

Two conditions usually travel together here. Sensitivity: had p not been the case, we wouldn’t have believed p. And adherence: had p been the case, we would have believed p. Sensitivity keeps our beliefs from floating free of a world that changed; adherence keeps them from floating free of a world that didn’t.

Truth tracking turns knowledge into somewhat of a sensor, and that protects it from including merely lucky cases. It’s a wonderful idea, that’s unfortunately plagued with technical issues, caveats and pitfalls. But the concept survived the theories built on top of it, and sensitivity remains valuable. Mainstream epistemology uses it effectively without having to commit to a truth-tracking definition or a specific wholesale theory of knowledge.

The two lotteries

One of the benefits of sensitivity analysis of knowledge is that it neatly resolves our lottery paradox from the intro. Let’s imagine there are two lotteries:

• Lottery One: One in a million ticket. The draw is complete, but you’ve yet to look at the published results. You believe you’ve lost based on the statistics.
• Lottery Two: A different ticket with better odds. One in a thousand. The morning after the draw you open the newspaper to see the winning ticket number. It’s not yours. Now, newspapers are generally reliable but for sure aren’t perfect; suppose that when you fold in everything, the initial odds, the chance of a misprint, whatever else, the probability that you won despite what the paper published turns out to be (surprise surprise) exactly one in a million, just like in lottery #1.

OK. In which case do you know you’ve lost?

We’ve already noted the answer for the first: no. For the second, intuition says yes, and it’s hard to say otherwise without collapsing into a global skepticism where almost nothing is known. Mainstream epistemology agrees on both cases. So what’s the difference? We rigged the example so the probabilities match so it can’t be the math.

It’s the sensitivity counterfactual. In the 1st case, had our ticket won, the probability of a single ticket winning would have been exactly what it is irregardless, and so would our belief. Our belief was welded to the background odds, and the odds are indifferent to which ticket actually came up. Nothing sensitive about it.

In the 2nd case, had our ticket won, the paper would probably have printed our ticket number, we would have read it and would have believed differently. The newspaper is a sensor aimed at the actual fact our belief is about. Not a perfect sensor, granted, but still connected to the relevant facts. Sensitive.

That’s the resolution to the lottery paradox. No matter how dramatic the probabilities, they are disconnected from our beliefs about the actual events of the world. Indifferent to the actual facts we’re trying to align to. Justification couldn’t identify that as a form of epistemic luck to be excluded, but our sensitivity analysis does that with ease.

From lotteries to buses to gatecrashers

If sensitivity was only useful for resolving lottery paradoxes and abstract epistemology discussions, I wouldn’t have written about it (who am I kidding, of course I would have written about it, that’s my whole shtick 🤷‍♂️). Luckily, sensitivity analysis and its counterfactual components are relevant to many real-world, day-to-day cases. We’ll focus on a paper that explores specifically the connections between the abstracts of epistemology and the practice and principles of evidence law: “Statistical Evidence, Sensitivity, and the Legal Value of Knowledge” by David Enoch, Levi Spectre and Talia Fisher. I had the pleasure of writing a university seminar exploring this article in a course I took with Dr. Spectre, and some of our journey here will retrace the work I did there.

So, let’s jump from the canonical lottery case to the paper’s bus case. A bus was involved in an accident and caused some harm. Let’s explore two possible scenarios:

1st scenario: There’s an eyewitness to the accident, and they identified it as a Blue Bus Company bus. The eyewitness surprisingly has a well established track record about these sort of things, and it is agreed they are roughly 70% reliable in these cases (it’s a terrible city with lots of bus crashes - there’s actually enough to build a statistic out of). The bulk of human law finds this acceptable and would be fine with ruling for the plaintiff based on this evidence. Cool.

2nd scenario: There aren’t any direct witnesses to the crash, but there’s uncontested data showing that Blue Bus operates 70% of the buses in that area. The bulk of human law doesn’t find this acceptable as evidence (maybe even not admissible), and will not base a ruling on it.

So, unsurprisingly, this was built to match up with the lottery example. Had the bus been a Big Red Bus instead, the eyewitness account would’ve probably changed, while the ownership statistics would have stayed exactly the same. One is sensitive, the other isn’t.

Let’s consider another example. Take your favourite philosopher; he’s scheduled to give a lecture at your local town stadium. Because these kinds of events regularly draw tens of thousands of people (a fella can dream etc.), the stadium is overwhelmed with excited fans who didn’t buy tickets and gatecrash the event. For every person who bought a ticket, a thousand didn’t. One person who actually gatecrashed the event gets prosecuted (poor guy named Smith, loves to count how many coins people have in their pockets). The ratio of gatecrashers to ticket buyers is overwhelming, far better than any eyewitness reliability. Statistics-wise, it’s solid. Would you convict him based on that? Feels almost obscene, doesn’t it?

Once more, we have machinery to provide us with the rationale to explain our general intuition around this. We have the sensitivity counterfactual: accepting this as evidence, had Smith bought a ticket, we would have convicted him anyway. The evidence doesn’t point to Smith, it’s disconnected from the facts of his specific case; indifferent to Smith’s innocence, which is exactly what we want it to track.

Two indicators for the difference

Let’s drive this point home. The main difference in all these cases is about what the evidence is made of. Market share is a fact about the world that is causally upstream and independent of the specific event at issue. It was 70% before the accident and remained 70% after. It would be 70% still in the counterfactual where a different bus crashed. It is an immutable background statistic, so any belief built on it is welded to the background, not to a specific event.

Eyewitness testimony (from a person, a tape, physical evidence) is causally downstream from the specific event in question. Photons come off the bus, the tire leaves a mark on the road, soundwaves disperse, take your pick. Change the event and you change everything downstream from it. Any belief built downstream of it will change accordingly.

Cool. Another interestingly cheap indicator that hints at the difference is which mistakes we would consider to demand an explanation. Let’s suppose we ruled against the Blue Bus Company based on the market share evidence, and it later came out the accident was in fact due to a Big Red Bus. Our response would probably be a shrug and a compensation for the bad ruling. We knew in advance we were going to be wrong about 30% of the time; you win some, you lose some. Nothing really to explain about it.

Now imagine the ruling was based on the eyewitness report, and it was revealed that the culprit was a Red Bus. This demands an explanation. Why did the eyewitness say blue? Was it badly lit? A funky line of sight? Maybe personal grudge? Insurance fraud? Something went wrong in the mechanism, and while we always knew the mechanism wasn’t perfect, somewhere there should be a fact of the matter to explain what happened this time specifically.

So a good proxy indicator would be “does this failure call for an explanation or a shrug”. The nice thing about this indicator is that it’s useful in many other scenarios, some of which we’ll explore today, and others at a later time.

A word about the legal analogies

We’re borrowing legal analogies to make the epistemology concrete. All good and well, but before we take the toolbox somewhere else, let’s consider that this seems a one way street; the law doesn’t care about the abstract philosophy I (maybe even you?) love so much.

Enoch, Spectre and Fisher make this clear in a thought experiment. You must choose the criminal justice system your children will live under. System A was built by epistemology nerds: it convicts only on sensitive evidence. System B is epistemically shabbier, but more accurate: its chance of convicting an innocent is lower. Which do you pick?

Choosing System A is what they call epistemological fetishism: trading some of your children’s freedom for abstract philosophical tidiness. The law is overwhelmingly functional. It should care about not making mistakes. Insisting on sensitivity means tolerating more mistakes in exchange for the mistakes being of a more respectable kind. On this view, knowledge as such has no legal value; accuracy does.

Is epistemology just window decoration for the law? The paper offers a cool bridge, built off the work of Chris Sanchirico in the philosophy of law. While the philosophy of law makes little room for epistemology as such, it does care about incentives. Let’s return to Smith at the stadium. If our court system will convict on crowd statistics, Smith, looking around at all the gatecrashers, has no incentive to buy a ticket. He would get convicted either way, because the statistics are indifferent to his actions. Statistical evidence, relied on systematically, corrodes the very behavioural incentives criminal law exists to create.

Sanchirico’s argument runs on counterfactual conditionals Smith entertains in advance: if I were to crash the gates, they’ll punish me; if I won’t, they won’t. After the fact, those become our sensitivity counterfactuals: had he not crashed the gates, we would not have punished him. The epistemic story and the incentive story are not the same story, and neither depends on the other, but they run on the same counterfactuals. That’s a neat little convergence that grounds the paper.

So what does this mean for our toolbox? Sensitivity isn’t a settled requirement for knowledge; the field is still too rattled for anything that definitive. But it’s widely accepted as epistemically desirable, it tracks a real distinction, it explains our intuitions across lotteries and courtrooms, and it turns out to correlate with something we independently care about. Good enough.

Is GenAI sensitive?

One last technical point before we answer this question. Sensitivity is a property of beliefs, a belief presupposes a believer. Does a model believe anything? Buddy, we’re not touching that with a 10 foot pole (not today, at least); we don’t need it. Recall the 2nd lottery case. Nobody asked whether the newspaper knew the winning number. The newspaper was evidence, and the question was whether our belief, formed downstream of it, was sensitive. For now, we’ll treat the model as a newspaper. We proxy its sensitivity through the sensitivity of our beliefs from using it.

Right. We’ve done the heavy lifting. Let’s point the toolbox at the thing. You feed this article to GenAI, and it spits back that the Theaetetus ends not with a final conclusion, but in aporia. That’s factually true. Is it sensitive?

At first glance, there are excellent indicators it is. Let’s steelman this position to the best of our ability.

Well yes

So let’s start with the model’s training data. It’s statistics, yes, but more akin to statistics of testimony. When the model tells you the Theaetetus ends in aporia, it’s not extrapolating from base rates about how Greek dialogues tend to end. It’s echoing the multitude of humans who read the actual dialogue and reported what they found (or copied their friend’s homework, sure). Every one of those reports is casually downstream from the text. Had the Theaetetus ended in a successful definition of knowledge, most of those people would have written that, the training data would have reflected that, and the model would have (probably) said that.

That’s a counterfactual chain. Laggy, tangled, mediated by an enormous training run, and an aggregate rather than a named witness we can point to. But it does exist, and the bus market share has no equivalent at all. So maybe this isn’t an absent sensor. Maybe it’s a distant one with a lousy dashboard.

A relevant footnote from Statistical Evidence, Sensitivity, and the Legal Value of Knowledge can provide a proper point of view here. The authors note that DNA evidence looks like an obvious counterexample to their claims: the law loves DNA evidence, but it’s presented as a match probability. The paper resolve the discrepancy by noting that DNA evidence is actually sensitive: had Smith not been at the scene (and had he not fiddled with the coins in his pockets), in all likelihood his DNA would not have been there and we would not have convicted him. It looks statistical, and claiming a DNA match genuinely is a statistical claim, but it’s a sensor underneath. GenAI has that same shape. Statistical in the presentation, sensor in the loop.

So, that’s one way of looking at it. Now let’s steelman the argument from the other side.

But actually no

For starters, the model’s statistics are a bit like the bus market share. Weights are a compression of statistical regularities frozen from a past snapshot. The training data may be causally downstream of our fact, but the output distribution is a black box, and we can’t say how much of that causal structure survived the processing chain, let alone how it bears on the specific question we asked just now. Nowhere along the token generation pipeline does the Theaetetus, as a distinguishable piece of data, get a vote.

This can be easily demonstrated. Ask the same question in slightly different ways (trivial paraphrases, reordered clauses, a changed name) and you’ll get very different outputs. You want to run it as a counterfactual and ask had the Theaetetus ended differently, would the output have been different? What’s the point, when half the time we can’t even get had the Theaetetus ended the same, would the output have been the same to play nicely? That’s the adherence failing, and adherence was supposed to be the easy one.

Second, and worse: GenAI’s apparent insensitivity to the specific fact in question comes bundled with “sensitivity”, in the everyday sense of the word, to practically everything else. Change minor, irrelevant details in a benchmark input and success rates drop dramatically. This is usually read as brittleness. In the context of our discussion, it means that if GenAI is a sensor aimed at anything, it is for sure not aimed at the thing we asked about. These results have been affirmed and reaffirmed in many different cases, be it GSM-NoOp (adding an irrelevant clause to math problems) or CatAttack (appending a sentence on the sleeping habits of cats to questions).

You can see why for our discussion, this reads as an insensitive result rather than merely an unreliable one. Math problems have nothing to do with the sleeping habits of cats. A sensor whose reading of reality includes (massively, it would seem) inputs unrelated to the property it was supposed to measure, was never really measuring that property to begin with Yes, these results have nuances and mitigations², but the overall point stands. Ask GenAI about x, and the answer you get doesn’t come from a sensor aimed at x. It’s from a sensor aimed at a strange quasi-entangled statistical mishmash that includes x’s pale shadow, or something that happens to correlate to it in some strange fashion.

Not enough? Let’s use our cheap indicator as well. When GenAI gets something wrong, are we compelled to provide an explanation, or do we shrug?

Not only do we shrug, we even give its mistake an endearing nickname. The model wasn’t wrong, per se, it just had an hallucination. But don’t you worry one bit, see the model’s error rate history? Do you see how it’s better than last year’s? Nothing to worry about. The industry has an entire vocabulary, built at least partially for the PR purpose of shrugging elegantly. This is exactly the type of response structure Enoch, Spectre and Fisher identified as the fingerprint of insensitive, merely statistical evidence: we knew going in we’d be wrong some percent of the time and this was one of those times. Hey, you win some you lose some, right?³

Well yes but actually no

So: decent theoretical grounds for saying GenAI can be sensitive, and rather convincing evidence that in practice it often isn’t. Maybe it would be wise to reframe our question as a functional question. Forget “is GenAI sensitive” in general. Ask which of the two, the distant-but-real sensor, or insensitive statistical recall, won this time.

Can we tell? Deciding would require some sort of visibility into the processing chain, as sensitivity is a counterfactual claim, and counterfactuals need a model of the mechanism to be evaluated against. You need to know what to hold fixed when you’re asking “had the world been a little different, what else would have changed”. But the GenAI model is completely opaque and alien to us.

When a person makes an inferential mistake, we can usually reconstruct it. They confused two similar names. They anchored on the first number they saw. They were tired and dropped a carry. Those failures are legible, and their legibility is what “the mistake calls for an explanation” actually means: there’s a story, and we’re natively equipped to hear and understand it, because we run something like the same process ourselves. Legibility is what licenses the counterfactual.

When a model’s math answer flips because someone mentioned cats, no story of that kind is available to us. Something happened in a space of several hundred billion parameters to which we have no intuitive access at all. We can’t enumerate the relevant counterfactuals, since we don’t know which details this mechanism treats as relevant or in-scope. Even full access to the model’s weights wouldn’t give us the kind of understanding that licenses a counterfactual judgement. “Had this fact been otherwise, would this particular matrix have emitted a different token?” is not a question with an available answer. Not to us anyway.

All we can do is sample the counterfactual, which is what evals are. Sampling gives you a frequency, gathered by repetition, with no backward connection to the individual facts that produced each data point. This type of frequency is market share statistics. The only instrument we have for assessing sensitivity produces insensitive evidence about it 🤷‍♂️.

So: a grounded, retrieval-backed model output can be sensitive, and sometimes clearly is. But we are not in a position to tell when. Worse, the interfaces are built so that the sensitive case and the insensitive case arrive in the same voice, at the same fluency, in the same chat window, with no signal whatsoever about which one you just got.

Epilogue: Are we also destined to end in aporia?

Not so fast, we still have one last trick in our toolbox. Remember that Enoch, Spectre and Fisher didn’t conclude that the law should demand sensitivity. They worked around epistemic fetishism entirely, and reached for incentives as a functional equivalent. We can borrow their move for our own case.

Stop asking whether the thing knows. Let’s assume for the sake of argument that it does, but with the real possibility that this knowledge is insensitive. Let’s also grant, for argument, that the usual objections about accuracy and reliability have all been met. The question that remains isn’t epistemic but runs in parallel to the philosophy: what types of behavioural incentives are we’re creating the moment we systematically accept insensitivity?

Would Smith bother buying a ticket if he knew an insensitive model would be drafting the recommendation in his court case? Would you check the test report a model generated, when everyone downstream will feed it into their own insensitive model and skim the summary? When it goes straight into an insensitive CI/CD agent that decides whether the build ships? What’s your incentive to write your own homework when an insensitive GenAI is the one grading it?

Personally, I think that’s culturally corrosive and epistemically reprehensible. That’s only a value judgment, yes. YMMV, of course.

But if you take the deal, notice the blast radius of what you’re granting. If you’re willing to call this knowing, you’ve adopted a standard where you’ve already lost the lottery, along with every incentive that standard ever touched.

Have fun, and we’ll see you in court for gatecrashing your next concert.