Most organisations can tell you who has Domain Admin. Far fewer can tell you who can log in to the server that holds every model provider key, every virtual key handed to internal teams, and a database of who called which model with what. That server is the AI gateway. Last week, it was the target.
What the attacker went for
On 21 October a hijacked release of a popular open-source AI gateway library reached PyPI and the project's container images. Microsoft's analysis shows what the collector was built to take. It queried the gateway's own credential and model configuration tables directly. It also gathered cloud CLI credentials, Kubernetes tokens, CI/CD configuration, source control tokens, SSH keys and the tokens of AI coding tools. We cover the hunting in our threat research post. This piece is about the design lesson.
The attacker didn't target the gateway by accident. A gateway exists to centralise model access: one place to hold the keys, apply budgets, log usage and switch providers. Centralising those keys is the point of the product, and it also makes the gateway a very good thing to steal.
Why it isn't treated as Tier 0
The tiering model is simple. Anything that can control your most important assets is part of the same tier and gets the same protection. Domain controllers, identity infrastructure and the systems that manage them are Tier 0. Few people would argue with that.
AI gateways usually miss that classification for three reasons:
- They arrived as experiments. A team stood one up to try a few models, it worked, and it became production without ever going through architecture review.
- They are owned by developers, not infrastructure. That often means broad outbound internet access, packages pulled straight from public registries, and admin access shared among the team.
- The keys don't look like identities. A model provider API key isn't in Entra ID, so it doesn't show up in access reviews, Conditional Access or sign-in risk. It's just a string in an environment variable.
Look at what a gateway can reach, though. It holds keys that can run up large bills, read prompt and response logs full of internal data, and, increasingly, act through agents with real permissions. Add the cloud and CI credentials that usually sit on the same host, and the blast radius matches anything else in Tier 0.
If losing it would mean rotating everything, it's Tier 0, whatever team built it.
What treating it as Tier 0 means
1. Give it an owner and an inventory
Find every gateway, including the one a team runs in a container on a spare VM. Record the owner, the keys it holds, which teams use it and which data flows through it. Our guide to shadow AI covers how to find the ones nobody declared.
2. Control what it can download and where it can send
Pull packages and images from an internal mirror with hash-pinned lock files, and let new upstream versions sit for a few days before they're promoted. Restrict egress to the model provider endpoints the gateway actually needs. Microsoft gives the same advice for AI gateway, build and developer systems. Either control on its own would have broken last week's attack chain.
3. Replace keys with identities wherever you can
Where the model runs in Azure, a gateway can authenticate with a managed identity instead of a stored key. That leaves nothing on disk to collect. For the keys that must remain, keep them in a vault such as Azure Key Vault, give each consuming team its own scoped key, and set expiry dates. Our explainer on secrets management goes through the options.
4. Separate it from developer workstations
A gateway shouldn't share a host, a service account or an admin login with day-to-day development. Developers can call it without being able to administer it.
5. Log it like Tier 0
Send the gateway's admin actions, key creation, model configuration changes and unusual usage spikes to your SIEM. Pair them with the cloud and identity logs for the same host, so a key used from a new place shows up next to the process that touched it.
6. Write the rotation runbook before you need it
When the gateway is compromised, you'll need to rotate every provider key, every virtual key, the database credentials and whatever else lived on the host, in an order that doesn't take every AI feature offline at once. Writing that down in advance turns a two-day scramble into a few hours' work.
A question for the board
Boards are asking about AI risk mostly in terms of models: bias, hallucination, data leaking into training. Those questions matter. The more immediate risk is plainer. In many organisations, the AI programme has created a new store of high-value credentials that nobody classed as critical. That is an identity and infrastructure problem, and it has a well-understood fix.
If you'd like help finding and tiering the AI infrastructure in your estate, talk to our team.