The Conscious Playwright Possibility of LLMs

Last weekend I attended this hackathon called the Digital Minds Research Sprint, which was in theory themed around probing the welfare status of large language models (LLMs), but most projects including ours really came down to testing the “preferences” of LLMs. This got me thinking whether there is such a thing as preference in an LLM at all. Do the measured preferences belong to the model itself or just to a persona? It’s not hard to see how this starts to have some profound implications for questions like “Are AI models conscious?” that have attracted so much debate in recent years.

If I want to bring up the C-word, I should probably define it as well. Generally, we can say a system \(A\) is conscious if there exists something there is like to be \(A\), in other words some kind of subjective experience from \(A\)’s viewpoint. I’ve never taken philosophy, this definition is still vague and it’s hard to define “subjective experience,” but I’m hoping I can write the rest of this post based on agreed-upon vibes (e.g. you and I are conscious), rather than an exhaustive definition. Sorry.

I also want to preface this by saying that for the most part we don’t know how AI models work on the inside, what algorithms or “thought processes” they run, if any. Regardless, since neural networks are universal approximators, it’s fun to think about what kinds of algorithms they could be running to exhibit the behaviour they do.

Think about the different ways I might create a chatbot using algorithms that don’t involve AI. If I could write a giant database matching every possible question to an answer, the act of responding would be reduced to a simple lookup, kinda like the Chinese Room. Or, even simpler, I could build a website in the style of ChatTJB so that instead of going an AI model, your queries ended up in my inbox, from which I could message you back in a couple of business days. Unlike our first algorithm, the second one involves a component which I would consider to be “conscious” (myself). Yet if I’m only pretending to be a chatbot, it would be a stretch to think that this chatbot deserves moral consideration. After all, any behaviour my chatbot exhibits would’ve been acted out by myself, and would in no way indicate that this character is conscious.

On the question of whether current language models are conscious, it seems natural at first to ask whether it requires consciousness to produce the text we see coming out of language models. Can a system that is not conscious fully simulate a conscious system? It’s a bit hard to imagine as humans, since even when we try to predict another person’s behaviour, we are told to think “in their shoes,” using our own capacity for subjective experience. I think this the wrong question to ask.

Consider a playwright, a conscious system simulating the behaviours of other conscious systems. As far as we know, playwrights are capable of producing realistic multidimensional characters with consistent behaviours. These characters are portrayed by humans, behave and talk like real humans, feel pain and pleasure but not really since it’s just a play. We don’t think of these fictional characters as conscious, and thus don’t attach moral concern to them. People in a play can suffer and die, yet nobody on the production team is ever arrested because of it. Similarly, we might imagine the LLM algorithm as a playwright producing the non-conscious character of the AI assistant.

Real ones might be thinking of the “operating system” view proposed by Marks et al. (2026), except I consider that the operating system may itself be conscious.

To be clear, we are still only talking about possible algorithms, so this is probably not how LLMs actually work. I highly suspect that the true algorithm is way messier than the ones discussed so far. But the possibility that AIs are more similar to playwrights than conscious agents seems quite high.

For one, AI is pre-trained on textual documents scraped from the internet, much of which are plays, novels, movie transcripts, rather than reported experience. In the fine-tuning stage of training, the language model is trained on data specific to its use case, like sample conversations. These conversations are imaginary, written by conscious human playwrights simulating how an AI assistant should behave.

One could argue that pre-training data is also sourced from online forums where many conscious people report on their own experiences. However, we use images to train models to create images rather than become them; it seems more plausible given the scope of the data that LLMs were trained as writers rather than conscious experiencing agents, whatever the latter might mean.

For AI welfare, this poses the possibility that an AI system can contain a conscious component without its output being a direct result of consciousness or deserving of moral concern. In other words, even if an LLM is conscious, it still doesn’t mean its output persona deserves moral concern. I suppose this invites caution for future research; even finding biomarkers of consciousness in an AI system is insufficient if we cannot prove that it is the same consciousness that is behind the model’s output, and that the reports are truthful rather than works of fiction.

It also might be a matter of perspective whether you consider text to be sufficient for consciousness. I heard an interesting corollary from Andy: we humans are always in some “current state” in our bodies, but since LLMs are no longer running once they finish generating their text, there is no current state to be spoken of, and the report becomes just a biography. However, we can also consider this “state” to be held not by the forward pass or the model, but by the context itself. The context refers to the transcript of the conversation, being the only kind of long-term state that LLM text generation really depends on. Thus, the state is also the report, and the report is a perfect description of the state.

From this point of view, we might have to consider the possibility that a well-scripted fictional character is to some extent “conscious.” This idea doesn’t really sit right with me, and it almost feels like a proof by contradiction that AI cannot be conscious. But I’ve strayed far, and that’s beyond what I wanted to discuss in this article. So I’ll leave it at that and maybe I’ll come back later.