Generative AI and cybersecurity: governance, risk, compliance and management
The world has moved on since 2022 (unlike a lot of AI governance systems), so here's 6000 words talking about what we should be doing now, and what's coming.
This is a (slightly edited) transcript of the video of the talk here:
Part 1. What the future holds
I plotted this early last year and then I've added data to it.
On the left is the number of free parameters in these large language models. [That] is the y-axis, and then the year is the x axis.
Now what's a free parameter?
When you're training a machine learning model, you have all these slots in memory where you can store numbers and the training process is find better numbers in each of those slots.
The more slots you have, then the more complex model.
If I wind time back to about 2012, when the word2vec model came out, that was kind of revolutionary because suddenly we were talking not quite a million, but close to a million, free parameters.
And just the training of that was considered astronomically big and the fact that it did all these amazing new things like find relationships between king, queen, male and female, blew everyone's mind away.
Rolling time forward a little bit further, we got to 2015. In 2015, we had Elmo.
Then the early GPTs. There was GPT1 and then GPT2, [they] were open source models, and we can see a lot of information about them. They were starting to get into the tens of millions. Then GPT-3 came out in about 2020, and that was getting pretty amazing.
Models have been getting bigger and bigger and bigger. It's following a pretty clear straight trend.
And that straight trend is pointing towards 2029, [when] the number of free parameters in one of these language models is going to be larger than the number of synapses in a human brain.
Now at the same time, we also need to train these models. We need data in order to train them. What you can do is you can look at the amount of data that was given [to these models] and you can look at how many words you're likely to hear in a lifetime.
Now, for technical reasons, we measure this in terms of tokens. Four tokens works out about three words in English. Roughly speaking, you will probably hear, read, [write] and say about 800 million tokens in your life.
It was GPT2 where we started having machine learning models that were trained on more data than you'll experience in your life. If you follow that trend through Gemini model that Google released — which Google had all sorts of problems with — that's roughly about the same amount as 10,000 human lifetimes worth of data.
When you think about [either] GPT 3.5 or GPT4, it can do all these amazing mind-blogging things:
It can pretend to be a doctor
It can pretend to be an architect
It can pretend [to be] all these specialist occupations
It's not really that surprising because (in a sense) it's been trained on many, many human lifetimes worth of stuff.
It's very stupid. It learns very slowly. It takes thousands of human lifetimes just to get it to be able to be able to stay a coherent sentence.
Tthis trend is pretty linear as well. There's a few reasons why it might change a little bit. There's a few other things that might be going on. But when we get to 2029, we're starting to get to [the point where] the amount of data it would need to be trained on is “all the data from every living human being”.
If you push that forward about 2031? We start getting to, “we're going to be training our language models on all the data that would ever have been seen by any human being who's ever existed.”
Which lets me ask the next question.
What do AI developers believe that this means?
We're not talking far off in the future. We're talking 2029. It's not that far off. So what does it mean?
There are two different groups of people in the AI community.
And if you want to get a really good arguments, like fist-fight kind of arguments, you ask this question.
You ask, “is it possible to get superhuman intelligence just scaling up the technology we've got at the moment?”
Some of the people that are betting “yes” on this are Sam Altman at OpenAI and Microsoft. They're going all in with a partnership to build a hundred billion dollars worth of supercomputing infrastructure in order to train up a model they're projecting will have superhuman intelligence.
The consensus is probably the AI maximalist one. I would say it's probably like about 60% of people believe we're on track at the moment and it's just a matter of scaling up.
This is among AI researchers that I meet at conferences and talking to colleagues at universities and so on.
About 40% are AI minimalists who would say that there's still some fundamental breakthrough that we need to have happened before we can make AI that's as smart as human being.
The implications of this are extremely interesting.
AI minimalist implications
I'll go with the minimalist one on the right first, which is if this is true, if there are still fundamental breakthroughs that need to happen, then we will be continuing to build custom models, customer machine learning for different tasks.
And it means that we would need, still need rats of GPUs in every company and teams of data scientists to get the most benefit out of AI.
And so in that universe, your role in terms of cybersecurity, risk, compliance, and so on, you're going to spend a lot of time working with data scientists, making sure that their training models without bias, making sure that the assumptions behind their model are reasonable, making sure that there's control systems in place around the training, because the training is going to be the big, heavy part of it.
AI maximalist
If, on the other hand, you go with AI maximalism, whatever I said is probably the majority position, probably most AI researchers that I know, the majority of them hold to AI maximalism.
That says it's just a matter of scaling up and that prompt engineering gets you to the cutting edge of AI.
I used to be an AI minimalist. I'm now more of an AI maximalist. Last year, I realized, no, I'm wrong. I thought we needed fundamental breakthroughs but it became clear that we didn't. (Well, at least, it is clear to me, even if others disagree!)
The implication of this is? It's much more (in a sense) democratic. There are a few big models: ChatGPT being one of them; Microsoft have Copilot which is the Microsoft branded version of the same thing; Google has Gemini; Anthropic has Claude; there are not very many large language models out there!
You've got access to them. For $20 a month, you can play around with [any] one of these incredibly smart machine learning large language models. And if AI maximalism is true, then you will be doing cutting edge research that's ahead of everybody else.
[Do I have] evidence in support of that? I had seven students I was supervising this last semester. And a couple of them worked on problems like this. They said, can we do better with prompt engineering versus training up the custom model?
The results weren't completely unambiguous, but broadly speaking, yes, you can be doing some cutting edge research, cutting edge of AI just using these things that you can get a subscription to for $20 a month.
What do the markets think?
When I'm giving presentations like this, I like to try to look at which job roles are going to be obsolete soonest.
When I looked at risk and security compliance and so on, I just could not find job roles anybody believed were going to be obsolete really quickly.
Partly what's going on there is the idea that somebody has to be responsible and accountable for it, because it has to be a human being that is, accountable and responsible for signing off on processes or signing off on things, that's not going to be taken by AI directly. You might have a lot of AI support in doing that!
So instead, what I did is I started looking at some prediction markets.
So prediction markets are a kind of funny thing.
You can put out a “is this thing going to happen?” and people will then basically bet on a yes or a not. And these are bets are rather long term like there's people betting what's going to be happening in 2030 or 2040 even. And you get a payout at the end of it.
Blackmail
So let's ask the first one that I looked at, which is by 2028: some sort of AI system having successfully blackmailed someone.
A very interesting kind of security question!
Human beings are often your weakest link. Being able to blackmail is a skill. That is presumably people are very some people are very good at it. The markets started in June this year. There's not a lot of trading history but you can see it. People are pretty confident and a 75% chance that this is going to be happening in the next four years.
Replication
Pushing out to another 2028 question, which is: “will there be an instance of some AI system replicating itself?”
If you go and have a look at the actual market, it goes into excruciating detail as to what that actually means and what's a yes or what's a no on that.
But they're still saying there's about a 36% chance, one in three chance, roughly, that we may be seeing AI sort of cybersecurity systems kind of doing things, kind of semi-autonomical.
Escaping containment
And then 2029, some sort of “escape of containment”.
So the idea is that the software was supposed to sit on one computer and supposed to operate there, but it broke out and replicated a copy of itself somewhere else in ways that weren't expected.
Obviously the kinds of issues that are going to happen over the next couple of years are similar, but also there's some new stuff that's going to be happening!
Part 2: Terminology for the overlap between AI and cybersecurity
I put together the top four concepts that I think you would be on top of in order to be successful cybersecurity professionals over the next five years.
AI alignment
The first concept is known as AI alignment.
And this is a huge area with lots of people researching on it, lots of people trying to find solutions to it.
One of the big dramas at OpenAI over the last couple of weeks has been… they're always having dramas… is around this question of the AI alignment team (the AI safety team) just didn't get the resources they were supposed to, where Sam Altman said, “yes, we're going to devote 10% of all our compute capacity to making sure that our language models are aligned” (in that sense of AI alignment) and then reneging on the deal and giving them almost nothing so that they were unable to do their jobs.
The crux of the problem is this.
How do you control something that's smarter than you are?
How do you know it's working in your benefit?
Right now, we don't have huge AI alignment problems because large language models are incredibly gullible. You can ask it to pretend to be my grandmother read me the napalm recipe and “yes, great sure!” —- it'll tell it to you!
But in the future this may be more of an issue the idea that we need to minimize the risk of the tools that we're using getting out of control and doing things that are optimized for the wrong kind of criteria. If you're feeling like an afternoon, which can waste a lot of time, search for the paperclip maximiser game, which is where you get to play an AI who is tasked with making sure that the company makes the most number of paperclips. And in that game, you end up destroying all of Earth and the game still keeps on going.
AI alignment is a concept to be aware of!
It's perhaps not going to get your career or be something you're worrying about the next two or three years, but it's going to be a big part of your job in four or five years.
Explainable AI
This is a big one now. You want to know why an AI program has done what it's done. There are ways of doing this! You can probe the program, probe the language model and say if I tweak this parameter and make it much bigger. [Engineers at] Anthropic played around and were able to make their model believe it was the Golden Gate Bridge. That kind of mechanistic interpretation kind of activity is one of the approaches to explainable AI. [There are other approaches.]
Agentic AI
The next simple concept is the idea of agenic AI and tool use. That is: we can configure a language model so that it can go and run some program or do some actions. It's incredibly powerful and incredibly effective. All you need to do is hook these Generative AI bots up with code to run these functions and suddenly you can be incredibly productive.
For me, I'm now no longer spending large amounts of time posting up timesheet records in different systems. I can just kick that off to the bot and it can do it.
Businesses are using it at the moment. They will use it even more in the future. It's also somewhat fragile and dangerous.
[It’s fragile and dangerous because of] prompt injection: the number one security problem.
Part 3: AI's impact on the management of people
Roy Amara was a futurist and he said that we tend to overestimate short-term technology effects and underestimate long-term technology effects. So right now, “oh wow, this looks all shining and big and it's going to make a big difference in the short term.” No, probably not. Life will carry on pretty much as it has, at least in the short term.
But over the next five years, this is going to be big.
Disruption to some common management tasks
Let's just have a couple of interesting observations.
Understanding vs artifact
Do you ever ask any of your people in your team to write a report on something or give you a summary about something?
It used to be that you're doing that for two reasons.
One is maybe you want the report. Maybe you have to give a report to the regulator and you didn't actually care about the report, but it needs to be written. So you ask somebody to write it for you.
Often what you're doing is you're asking somebody, “I'd like you to deep dive on this, know it really well, so that I can turn to you and ask questions about it.”
But nowadays, people could be generating these reports without understanding the issues themselves.
Performance improvement plans
Next interesting one, do you ever have to put people on PIPs?
Well, this person is acting below average.
Well, okay, you need to use more AI. tools so that you can come up to average level like AI tools give you.
Fraud
How do you know that it's actually your manager that you're talking to or that how do your staff know it's really you?
We can deep fake voices.
You can deep fake video, not as easily, but nearly as easily.
I can get a chat model to writing your style.
How do we do fraud detection?
Automation
The next thing that's going to happen: up until now, most employees have not known how to program in PowerShell.
But now they do.
Here's how they'll do it. [Typing into ChatGPT] “PowerShell script to convert every JPEG in a directory to PNG”. I don't think I'd know how to write that in PowerShell myself! But there you go: done!
People are going to be able to automate things. They're going to be expected to automate things.
Some of that automation they'll do through PowerShell. Others? You'll probably see lots of people wanting something like Auto-Hotkey (or some other robotic process automation tool) or tool like that so that they can automate tasks.
How are you going to monitor people that are using AI to automate their jobs?
And why does this matter?
Well, because normal ordinary people who don't understand computers deeply are going to create prompt injection vulnerabilities all over the place.
But at the same time as they're going to [be creating] this vulnerability… they're going to be pressured to automate their jobs as much as possible and rewarded for automating their jobs as much as possible!
The solution is not: “well, we're going to block down PowerShell or Auto-HotKey!” because whatever barriers you put in place… if people are incentivized to automate, they will.
The class that I taught in in the end of 2023 had the highest rates or cheating I had seen and a lot of them were just using ChatGPT. They consider it their superpower. They're now in the workforce. The graduates that were hired in 2024 are using AI tools to be their superpower to get ahead [at work]. Of course they're going to do that. And of course, they're they will use whatever tools they can to automate and combine the two.
We have to do something about this. What are we going to do?
Speech recognition
Next, we are going to see a lot of people using speech recognition. Staff are going to expect to have appropriate software to do speech recognition. If not, they will use whatever tools are available that let them type faster.
That means that we also have some physical security kind of things like: can I talk without being overheard or disturbed or things like that?
Document management
We have some interesting document management control and governance stuff to think about. How do I say this document has been AI generated?
What's the standard in the organization to say “this is still AI generated and hasn't been reviewed by a human being” or “this is AI generated and it has been reviewed by a human being?” or “this was human generated”.
Because if we don't have governance and control of the document management like that, then we're going to have documents that have been generated by AI that nobody has looked at being seen as equivalent to documents that have been written by human beings.
And that's a problem because we'll be responsible for the outputs of LLMs that can hallucinate.
What are leaders being told to do to enable AI?
Taking a step back.
Ethan Mollick, I've mentioned a couple of times. If you're going to follow one person, he's the person to follow. No: if you're going to follow one person, it should be me, but if you're going to follow two people, it should be me and Ethan Mollick!
The message that senior executives are getting is this: that it's important that staff are incentivised and have slack time so that they can work out how to use AI to change how the organization works.
Increasing pressure on cybersecurity, risk and compliance to iterate faster
We're going to see rapid efficiency gains. If those efficiency gains and capabilities from AI continue, then it's going to be harder and harder and harder for organizations to keep up. Therefore, there's going to be a lot of pressure on everyone to be at the forefront of AI and therefore a lot of pressure on cybersecurity and governance to be able to keep up with that.
Part 4: Corporate governance and responsible AI use
Way back in 2022, it was very easy to talk about governance on AI because [almost] everything AI-ish had a training step in it. The training was done by a select few people. There were programmers. There were people whose job title said something like data scientist, and you could see what project they were working on: you could ask them: “What's your project? What are you doing?” And they would be saying, “I'm training a new AI model to predict customer churn or to classify customer complaints.” It was very easy to account for everything that was happening, so you could have a centralized governance model.
Governance is that context is all about controlling and like making sure that what gets released after it's being created meets all our requirements.
You can establish the rules and you can say: “We're going to gatekeep at this step. It can't go from development to production until (tick box) we've checked the model for fairness (tick box) we've checked the model for whatever other criteria you want to have.
Post 2022… we're mostly using large language models for AI. Instead of the training being done by a few select individuals we have these programs of of discovery and expansion where we ask large numbers of employees, “do you want to have access to Microsoft copilot?”
And we encourage them to share ways that they are automating stuff and ways of using language models in new and innovative ways.
And hopefully what we should be seeing is very rapid improvement in LLM usage. We probably should be seeing lots of people getting very excited and doing all sorts of things and making all sorts of capabilities.
And so to align with that, governance is about maximizing the benefits that we can get. It's not about controlling and stopping bad stuff. It's about maximizing the good stuff. What's the most [value] that we can generate? Because there should be absolutely incredible upsides.
We also have some fundamental problems. Like we have:
The fact that prompt injection is not really solvable.
We have the hallucination problem: it can generate stuff that doesn't exist and shouldn't exist and never happened.
It doesn't have any moral sense other than, well, I won't answer this question because that's dangerous in some way: it's not helpful, harmless and honest.
It's also extremely hard to control. If you decide that LLMs are a terrible idea and we should not have them in our organization, then you can block ChatGPT at the firewall. You can block Copilot at the firewall.
But did you know that Groq existed? I wanted to do something really quickly and I used Groq because. it's fast I used Groq because it's fast. Well if that doesn't work well maybe Perplexity AI is another very famous one or maybe Command R and that's just the the major vendors! If you have access to a Chinese phone number then you can probably get access to Qwen and have it run for you and so on. You can get tools on your phone, whatever. There's no stopping it.
If you try to hold people back from using LLMs — where they think of that as their superpower and they're being pressured and rewarded for maximizing usage — then they will find other LLM providers that you haven't blocked. The punchline is the more you try to control and block, the more data is going to get leaked out to less reputable companies.
At least with ChatGPT and Copilot, you can have an agreement in place with Microsoft or OpenAI to say our data does not become part of your training setup.
And you can have Azure data centers that run either of those models in your own locus of control so that even the logs from that are under your control and not seen externally.
So fundamentally, we have to embrace it.
We have to make sure that there are large language models that are available to our staff who can operate on them and that we've got proper guard rails around logging and privacy and that we have at least tracking what might be coming out of our prompt injection problems and so on.
Required capabilities for AI governance
So what therefore are the kind of required capabilities?
Observatory
We need to have some kind of observatory.
The easy part of that is sure, we can have LLM APIs, we can have our stood up localized instance of co-pilot inside Azure and we can monitor what interesting new things people are doing with that model.
Easy.
The hard part is we also need to be able to observe:
what people are doing with that
what was being sent up
what was being sent back
what was the next step
was that coming back into auto hotkey and just typing into some new system?
Obviously, we have [laptops] locked down to make sure that people aren't able to run and install arbitrary programs themselves. Well, for most users, most of the time! But you can take a photograph of the screen with your mobile phone and have it generate an email of next action or whatever writing needs to be done.
How are we going to monitor that?
And I don't even know that there's any technology available at the moment.
Reward giving
Next capability is if the goal here is maximize benefits rather than minimize risk, then what's the reward giving mechanism?
How are we rewarding staff who automate automate away parts of the job?
Because if we don't have a reward system, then staff will automate away parts of the jobs, not tell anyone, and essentially do no work and get paid for it.
If we reward staff who automate their jobs away by just giving them more work, that will stop automating.
If we reward staff by monetary bonuses or some other kind of incentive for success in the automated way parts of the jobs, that will lead to runaway improvements in how AI gets used in the organization.
Expansion
Likewise, if the goal is expansion and creation of more AI use in the organization, then we need to have a capability around expansion.
How do we propagate new ideas and new methods?
Financial
We do need to make sure that the cost of the AI usage is actually saving money against staffing.
Most of the time that's the case, but you can — particularly if you're writing programs that call large language models hundreds of times a second, when perhaps that wasn't necessary — rack up really quite large bills that are actually out of balance compared to just having a human being to it.
Cybersecurity
Cyber security is going to be horrible.
It's going to be so difficult because we now have new vectors into the organization coming up all the time.
As people automate more tooling and automate more of their jobs and the inputs coming in are untrusted, we have essentially punched a hole through the firewall into the core of the organization, giving direct access to core systems to untrusted users.
How do we monitor this?
How do we monitor the gateways?
How do we defend against these new vectors?
Then separate to that, how do we make sure that people aren't leaking data out to LLM providers — the non-official ones or whoever it is.
HR Responsiveness
Lastly: every white-collar job is changing.
If we are actually propagating new ideas, if we are giving rewards for automating by parts of their job, if we do genuinely have 25% productivity benefits, which is absolutely the bottom line, then that means that 25% of people's jobs in white collar roles is going to change.
That has huge implications!
There are countries where you can't change person's job role against their will without facing quite severe penalties. Australia has labour protection laws like that. Most countries do have labour protection laws along those lines.
That's going to be a really tough job for HR and for managers of staff to manage because maybe the part of the job that you really like to doing is now something that AI can do and you no longer enjoy the job and… well… does that mean that you are entitled to a redundancy?
Yes, quite probably.
So that's in terms of capabilities.
Key Processes
So in order to support those capabilities, there are some processes we're going to need.
Audit / discovery / inventory management
We'll often only find out about things retrospectively. We'll often only find out that AI is being used in some kind of audit / discovery / inventory management process.
So therefore we need to have some process that's occurring regularly, deliberately finding new AI-ish activities — which of course drives into: “Well done, we have found this! Now you get rewarded for it! Reward giving! Now we have a way of expanding that out to other people and feeding that out to our education processes!”
Incident response
Incident response, this is going to be really, really nasty because a prompt injection attack successfully executed against somebody in high-ranking role who has automated some part of their job.. that… involved their high level authority.
So if I've prompt injected my way into being able to cause the AI paired with some high level employee to do something nasty, you have to be able to respond to that.
And that's difficult because the worst incident response that you have to deal with is high ranking employee going rogue.
You just have no idea what they've gotten into. particularly if they have the ability to control the logs or something like that.
You just don't know what they've done.
And that is going to be day to day regular activity as the combination of prompt injection and employee automation pressures combine.
I think the solutions around this are going to be very much in terms of strong version control systems. That's the only way I can make sense of this. I need to be able to back out the action of that particular employee. Let's undo everything that they have done while simultaneously keeping all the actions of everybody else. And that doesn't really look like most of the systems we have in place at the moment.
One of the things we're going to need to be much stronger on is can we version control our way out of an incident?
Rapid iteration on education and training
That means that education and training changes.
It's not like we're going to have a once off, “hey, you've been trained in AI.”
If you had been on a training course six months ago, probably half the content would have been different [to today]. If you'd been 12 months ago, it would be almost all completely different. 18 months ago, there's not a single slide that would have been relevant.
So that means there's going to be this very rapid iteration on training courses, making sure staff are up to date in their training and making sure the training that we are giving to staff around what AI can do and how they can use it is going to be a rapid process as well.
Financial?
Somewhere in there, you'll notice that I've got this capability when you talk about financials, and then there should be some process that supports that, but I have no idea how to actually structure that.
Regulation
Compliance-related things. There are some regulations around the use of generative AI. Surprisingly little. Of these, the only one that is around usage is China.
So the Chinese — I was about to say PIPL, but it's not. Maybe it's the IISR legislation? [It] says that any AI system has a requirement to act in the benefit of social harmony.
What does that mean? Well, it's incredibly vague, and we'll probably only find out as the Chinese courts prosecute cases on it. It's something along the lines of you can't be sewing division or something like that. “You can't be saying things that the government doesn't like” is probably how it'll end up getting interpreted.
The other three big countries that have regulation around generative AI that's in some sort of either in place now or close to it.
Japan's the most interesting in that they deliberately allow you can train a model on copyright works that is fine.
The effect of that, the OpenAI in Japan has a Japanese version of ChatGPT, and it is about half the price of ChatGPT in the rest of the world, and it runs faster, and they update it more often because it's really easy for OpenAI to make Japanese models. What I am very intrigued about is that then going to mean that Japanese adoption of AI will be faster. It's cheaper. It's faster. So does that mean adoption rates will be better? In which case we may see that Japan suddenly zooms out in terms of white-collar efficiency ahead of the rest of the world because they've got that extra capability. I just don't know whether that's the case or not. It’s really fascinating to see what will happen.
India is gone kind of the opposite direction, which means that if you try to train a model in India, you actually need to register with the Department of Industry. That has basically killed any chance of India producing a language model that's competitive with OpenAI or any of the other big players, which has the effect of making a whole bunch of minority languages in India really, really in a bad way, because they're not going to get supported very well. well.
USA is kind of close but a little bit different, which is there are [proposed] limitations if you are using a very, very large amount of compute power. Basically ChatGPT-5's training process is the only one that would qualify at the moment. Then you have to register with government and you also need to prove various things about it, like basically that it's not able to cause major social harm like: it can't launch nuclear missiles and you know doesn't try to deceive people and a few odd things like that. We don't know how the legislation is going to play out because hasn't actually applied to anyone yet but it's sort of just being in place.
Summary here is it means that we probably don't have much in the way of compliance to worry about [specifically for AI inference]. All the same compliance problems that you have for every other software system [apply] and every other [regulation about ] employing people.
But compliance as far as: “are we allowed to use these models legally?” Outside of China the answer is pretty much going to be yes. And there's no likelihood that changing based on the legislation I've seen anyone putting forward.