Managing third-party risks from GenAI and the use of 'Shadow AI'

We recently discussed both the "Shadow IT" challenge and the relationship between Generative AI (GenAI) and third-party cyber risk management. Now, it’s time to tie all of that together to understand the GenAI challenge in third parties more broadly, as well as the issue of Shadow AI.

To recap, AI risks (and GenAI specifically) have become a serious challenge for cybersecurity teams, as the adoption of AI technology often outpaces the efforts to craft and enforce corporate policy to govern its use. It's also quite common, especially when organizations decide not to deploy AI at all, for users to seek out AI tools on their own, introducing risks that were never expected and, thus, will not be mitigated or managed at all.

In addition, AI features are being continuously added to software products and SaaS platforms that are already deployed in the corporate environment. As users are updated to the environment where these tools become available, behavior can be difficult to predict and mitigate. While that has always been a problem with software updates, "hallucinations", prompt injection and other GenAI-specific attributes turn this into a novel challenge.

Deploying AI can be deceptively easy, because deploying AI safely, which is what we all would like, is not easy.

While this is a problem for most businesses today, Third-Party Cyber Risk Management programs (TPCRM) must consider that vendors, partners, or other third parties are facing the same challenges, sometimes with fewer resources to tackle them properly. This means that existing processes, including vendor contracts, may be outdated and failing to provide visibility or guarantees as to how that third party operates.

There's no single process or security measure that will solve all these problems. As already noted, even blocking AI or telling users to avoid it often does not solve the problem. If they deem that AI is the best tool for the job, they will use it, even if that requires saving corporate data on personal devices and uploading it to unsanctioned services.

With this in mind, it’s clear that the best way to achieve the outcomes we want must involve a combination of technology, processes, and policy working together.

Cloud service and AI security controls

As is the case with most novel business platforms and IT solutions today, AI usually relies on cloud computing infrastructure. With inside-out monitoring, we can leverage those platforms to collect data on how third parties are deploying their AI applications and securing the infrastructure that supports them.

This automated approach is more reliable and more useful than traditional compliance questionnaires, since point-in-time assessments do not accurately reflect a third party's cybersecurity posture over the lifetime of a business relationship. Furthermore, the insights are actionable, allowing the third party to fix the issues as they appear, including noncompliant deployments, lack of AI guard rails, inadequate permissions, or other development errors that could expose unprotected environments.

For instance, to help manage risks stemming from AI use in third parties, Zanshin also checks certain settings and controls related to AI applications. As an example, a first-party would use Zanshin to perform daily tests on the security of their application provider (such as a SaaS business application provider) to ensure they are using AI securely.

Here are three examples, one for each major cloud platform:

Amazon Web Services: AWS provides a service called Bedrock for building GenAI applications and agents. In some scenarios, Bedrock can help control costs and maintain data governance when other solutions would require data sharing with other third-party environments or services. There are some caveats, though.

Bedrock has guardrails to prevent prompt injection attacks. For third parties using Bedrock, Zanshin can verify that this feature is enabled, which can help prevent certain types of unintended behavior or even cyberattacks, especially against AI agents. This should be combined with robust system prompts and other features for better results, though the lack of a guardrail is a strong indicator of a broader AI risk management immaturity that should be investigated with your third party.

Google Cloud: Google's Gemini Enterprise Agent Platform, formerly called Vertex, is Google's offering for building AI agents. It has a different feature set from Bedrock, however.

By default, all content in the Gemini Enterprise Agent Platform is encrypted with keys managed by Google. Privacy regulations or contractual obligations may require the use of keys that are not under Google's control, or customer-managed encryption keys (CMEK). 

Zanshin can warn about the existence of endpoints without CMEK. In environments where such keys are required, endpoints that don't use them are the weakest link in the chain. It could also indicate a deployment or process failure that has allowed an endpoint to be configured without the required keys.

Azure: Microsoft's platform offers visibility over Defender settings and features that Zanshin can query for additional insights. One such control in Defender for Cloud can collect evidence and excerpts from suspicious prompts detected by Defender for AI Services, a security extension for Microsoft Foundry. Foundry is similar to both Amazon and Google's offerings for AI applications and platforms.

Model collapse and the accuracy problem

While AI can reduce costs, it comes with a quality variance that not every AI deployment is taking into account. There are now reports of businesses rehiring employees after finding AI systems to be performing below expectations.

In our previous article, we already mentioned that using AI for both filling out and analyzing questionnaire answers for TPCRM purposes could eventually lead to a model collapse, which is a situation where the AI has been fed so much of its own error-prone output that it becomes unable to provide useful results. More broadly, this indicates that AI output may degrade over time if there are no measures in place to keep it consistent or improve it.

This problem also exists for services provided by vendors integrating AI. Unless AI adoption is matched by new processes tailored for quality control and review, the results may contain errors that were never present in the former process. A quality control process that worked very well for human output is often inadequate for AI output, because humans and AI systems make different mistakes.

Vendors may also neglect to disclose that they are making changes that integrate AI, which can lead to failures that the existing quality processes will not detect. In a worst-case scenario, even the vendor may be unaware that AI is being used due to employees or contractors relying on unsanctioned AI services (Shadow AI).

This is where policy, contracts, quality metrics, and other measures need to be leveraged to ensure that AI does not create unforeseen operational issues down the line. Whenever possible, its use should be disclosed and understood by both parties, and any requirements should be extended to everyone in the supply chain. Procurement departments can be a key ally in this effort.

Awareness, training, and education

AI represents a shift in how computers work, moving applications from deterministic logic to probabilistic inference. Users have little experience with this kind of technology. Furthermore, chatbots are designed to be interacted with through a human-like dialogue interface, which immediately oversells their capabilities.

Cybersecurity awareness and training are often necessary to ensure that people are aware of how GenAI works, its limitations, and associated risks. In some environments, it may be possible to monitor whether users are signing into AI services using corporate accounts, and each business must decide the extent of data that can be shared with these services. Awareness efforts should be associated with an internal review of authentication tokens or calls to AI-related APIs to detect whether AI services are being used without authorization (Shadow AI).

This training should be extended to system administrators and tailored to the new features that they may have to configure or disable in the SaaS platforms and software products already deployed.

An example of a feature that can represent a security risk can be found in the SonicWall Cloud Backup incident. The company suffered a data breach, leaking encrypted credentials and sensitive configuration files of all customers who used the Cloud Backup feature, according to SonicWall's advisory from last year.

Around the same time, there was a surge of ransomware attacks using SonicWall as an entry point. One insurance company claimed that Akira, a ransomware group known for targeting SonicWall, was responsible for 40% of all insurance claims in 2025.

The links between the attacks are difficult to confirm. Nevertheless, at least one company sued SonicWall over this leak. While this was not an AI feature, similar data-sharing practices will often be necessary to make such functionality work, and system administrators should be aware of the risks.

The risk of underestimating risk

Some technologies have an inherent barrier to entry, whether due to complexity, inconvenience, or some other roadblock, that prevents most users from having access to them. The effort required for use provides a certain level of mitigation, sparing us the trouble of having to actively manage the risk.

For example, there is little need to tell users about the dangers of running a file-sharing server, because most of them are not going to be running one. Recent operating systems make it difficult to openly share anything by accident.

But AI is still in a phase where the barriers are few and far between. It's simple, convenient, and can apparently "run" on any device thanks to the cloud. This outward simplicity hides a web of technical limitations, legal uncertainties, geopolitical concerns, and privacy implications that are far from obvious.

With documented incidents like Vercel's, where an AI tool led to a data breach, there is no question that AI presents many risks. However, there's no way to know if this is where it ends.

The lesson here is that AI risk mitigation compels us to be deliberate. There is a push to ensure that using AI feels ordinary or even unintentional, so risk mitigation must be a counterbalance. In some ways, the friendliness of GenAI invites us to underestimate its risks, and avoiding that is a major step.

‍