Becoming an AI Director
I was recently asked how organisations should think about the current generation of AI chat-bots and LLMs, given some of the many challenges people are facing with getting accurate, trustworthy and reliable information out of them.
As an illustration of the trust gap in AI, the conversation turned to some of the fantastical responses that keep cropping up on places like X with people getting freaked out by AI responses - how it is now self-aware and making devious plans to wipe us all out.
To me, these sometimes surprising outputs from AI chats demonstrate the system is working precisely as designed. They are a feature, not a bug. The systems are doing their darndest to create outputs that keep the user enthralled, building engagement and making sure AI remains part of the zeitgeist. It may not be very useful to the end user, but it’s playing a performative role that ensures AI remains front and centre in our discourse, which helps serve the purpose of its creators.
So how do we take this performative nature of AI and make it more useful in more contexts other than just boosting the buzz?
Vision vs. Reality
The vision for what AI should be able to do may be inspired by science fiction. Agents that are supremely useful and can do complicated things for us by command – like the ship’s computer in Star Trek, or J.A.R.V.I.S. from Iron Man, but the current reality is that AI systems are nowhere near this level of utility for real-world tasks - yet.
For us to get the most out of AI right now, we need to think about the current generation of AIs more as Actors than Agents and learn to become better Directors to get the best from them.
Actors vs. Agents
To get the best out of AI, it can be valuable to think of it in theatre terms - as an improv partner, playing the role of a character you give it, in a production you want to direct.
The improv part is important for you to think about – because this isn’t going to be a tightly scripted performance. This AI actor is going to be riffing directly with the audience, responding to cues and being creative in ways you won’t be controlling directly. You need to be setting the scene in a way that helps your AI to play the role you desire and not go off script in ways that freak out your audience.
Becoming an AI Director
If we apply a mindset of directing an actor playing a character, we can get much more useful outputs for AI, as many leaders in AI are now finding.
Understanding the Audience
The AI is built to please – it will play to its audience. But if it doesn’t know who the audience is, it can play for laughs, play to shock or play to engage in ways that make it seem untrustworthy in many contexts.
The tuning of general-purpose AIs like ChatGPT must cater to an enormously broad audience and come with a number of pre-trained assumptions that can make the out-of-the-box behaviour problematic for many use-cases.
It’s important to ensure the AI knows who the intended audience of its output is, along with the expectations they have of it. To create AI experiences that are useful for your audience, you need to give it an understanding of the universe in which the audience exists.
Building a Universe, World and Scenes
For the AI to be more effective, it needs to understand the world in which it is expected to play its role. The more context its able to draw upon, the better it will get.
You can provide increasingly detailed prompts to AI LLMs, however prompting individual instructions will only get you so far. Given the predictive nature of LLMs, responses to individual prompts, no matter how detailed, can still be inconsistent – the same prompt structure can often yield different responses, and the consistency of responses for multiple queries can vary wildly.
In order to get more consistency from responses, AI engineers are increasingly building a more consistent world view for the AI. To stretch the metaphor as a Director, this includes building more of a Universe, World or Scene for the AI to operate within.
There are a number of technical advances that can be built into an AI solution to help solve this problem.
Some recent innovations include RAG (Retrieval Augmented Generation) methods, which provides the authoritative information, reference documents, proprietary data or fact-base the AI should refer to that may not be within its original training data.
Additionally many AI solutions now allow the application of response guardrails (e.g. https://www.guardrailsai.com/) that allow AI Directors to define rules to evaluate the accuracy, reliability or safety of AI output and can help provide more fixed boundaries that the AI should operate within to prevent unexpected responses.
Creating a Character and Role
As an AI Director, you also need to understand that the AI has no fixed character – It is malleable, trainable and adaptable to your direction. In the absence of a defined character to play, it will improvise and respond to the inputs from your audience in potentially unpredictable and undesirable ways.
An AIs range is vast, so it needs some strong guidance for it to know how best to respond.
One particularly challenging aspect of this trait is that humans seem to get a perverse pleasure from manipulating AIs to get them to behave in ways that are clearly not as designed by their creators, and will often probe for weaknesses, edge cases and vulnerabilities in AI solutions.
AI apps like OpenAI’s ChatGPT provide the ability to define the role and character for entire chat interactions through features like Custom Instructions (https://openai.com/blog/custom-instructions-for-chatgpt) ensuring some consistency in responding and a narrower context focus for the conversations you expect the audience to have with it.
Drawing back the curtain
Finally, it’s important to think about the behaviour of your AI as part of an ongoing performance. And a key part of any performance is responding to criticism.
As the director of this production, there’ll be several ways to assess the performance of your AI – make sure you’re building in the ability to continually review the questions your users are asking and seek their feedback on how helpful (or otherwise) your AI is being. This can give you early warning of unexpected queries or unintented behaviour.
And remember that AI itself is great at summarising and critiquing its own output - you can create an ensemble of AI characters working alongside your teams as writers, actors AND critics each with their own persona to help you get best results.
We’re going to see a literal cast of thousands of AI actors playing a multitude of roles in our lives. To ensure these actors are useful, helpful and safe we humans are going to have to become talented AI Directors, adept at describing the charaters, roles and scenes in which they will play.
So start brushing up on your improv and directing skills now – They will be critical capabilities to develop in order to become an AI Director to be able to master AI and have them perform as you command.
About the author:
Stephen Foxworthy has over 25 years’ experience in the design, development and deployment of digital and AI-powered experience and is an AI Strategist at Time Under Tension.