The AI black box problem: what you can’t see can hurt you

Most enterprise AI conversations today focus on authentication and authorisation. And that’s necessary – but it’s incomplete.

On the podcast, Dan Meacham (VP of Information Security at Legendary Entertainment) said: 

“What we don’t see is what’s inside the model.” 

And if you’re a CISO, that means you’re dealing with a critical system where traditional visibility simply doesn’t exist. 

No ediscovery, no audit trail

In most enterprise systems, investigation is relatively straightforward – you probably have logs, endpoints to manage, and audit trails. But that breaks down with AI models. 

“When I do my ediscovery I can’t do ediscovery inside Copilot…or inside Gemini,” Meacham pointed out. And that single limitation has far-reaching implications. If sensitive data is introduced into a model (intentionally or not), then it becomes very difficult to trace or remove it. 

Apart from the risk of leakage, a key issue here is non-repudiation – the ability to prove where data came from and how it was used.

“Let's say for example we were going to make a movie about a fish,” Meacham said, “but you came from another studio that was also making a movie about a fish. How do I know that what you did at the other studio isn't going to influence what we're doing with our movie here? And so if you were to put that into the AI model I need to be able to have the non-repudiation that determines when those elements came into my model.” 

Right now, that capability is largely missing. 

The contamination risk

One of Meacham’s more subtle points is also one of the most important: as well as storing data, AI can blend it.

“...how do I know that what you did at that other studio isn’t going to influence what we’re doing here?” 

This is the AI-era version of data contamination. Not a direct leak, but a bleed of ideas, patterns, or proprietary information across organisational boundaries.

In regulated industries, this is a legal headache. In creative industries, it’s a massive IP risk. And there’s currently no reliable way to inspect or validate it inside the model itself. 

Personal AI: the shadow data layer

It’s even more difficult to understand and manage how personal AI influences outputs at work. Devices are rapidly embedding on-device intelligence that observes user behaviour:

“That intelligence is looking at every single email, every single calendar…everything I’m looking for on the web browser.”

Now consider the boundary problem: corporate work and personal devices are increasingly intertwined.

“Even though I have my corporate system here, my personal device is looking at the same things.”

This creates a parallel data layer – one that sits outside corporate controls but still processes corporate context.

A new attack path

For threat actors, this opens up a different angle. Instead of targeting apps directly, they can target the intelligence layer itself.

Meacham sketches a plausible scenario: a seemingly harmless app that leverages AI context rather than explicit permissions.

“Let me create an app. It’s a card game I know that this CEO is going to be really interested in playing, and they just put it onto their phone – then I could access photos on their phone, their contacts and their calendar. Maybe I can't get to those because they’re locked down, but if I can access the AI on that phone then there's a back way in for me to get that same type of detail. So now I know when that next merger-acquisition is going to happen, or something else.” 

Even if direct access is restricted, the AI layer may still expose patterns or preferences – enough to infer sensitive activity. 

The control gap

The reality here is that traditional advice doesn’t quite work. You can’t inspect what’s happening inside most models, and you can’t realistically tell employees to disable AI on their personal devices. 

Which leaves organisations in an awkward middle ground:

• Partial visibility at the edges
• Limited control over the core
• Expanding exposure through everyday tools

“Those are the things that really keep me up at night,” Meacham added. 

What you can do: 

• Focus on data minimisation – assume anything entering a model may persist
• Strengthen endpoint and identity controls as compensating measures
• Build policies that account for personal AI use

AI adoption continues. But until organisations can see inside the systems they rely on, it’s critical they acknowledge that inputs and outputs aren’t the only risk – you have to consider what happens in between, too.