Skip to content
Help guide the future of agentgateway — Take the survey

For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Azure

Verified : Automated tests passed for: Kubernetes resources: The Kubernetes resources on this page were checked against their API schemas.
Page as Markdown

Configuration and setup for Azure AI services provider

Configure Azure as an LLM provider in agentgateway.

Azure supports two endpoint types:

  • Azure OpenAI (OpenAI): Connect to Azure OpenAI Service deployments at {resourceName}.openai.azure.com.
  • Azure AI Foundry (Foundry): Connect to Azure AI Foundry project endpoints at {resourceName}.services.ai.azure.com.

Before you begin

Install and set up an agentgateway proxy.

Authentication

You can authenticate to Azure with an API key or with implicit Entra ID authentication through DefaultAzureCredential. On Kubernetes, implicit authentication can obtain a token from managed identity or workload identity. It does not require a Kubernetes secret or policies.auth.

You can also configure explicit managed identity or workload identity authentication in the AgentgatewayBackend. Managed identity requires access to an Azure managed identity endpoint. Workload identity exchanges the projected ServiceAccount token with Microsoft Entra ID.

Workload identity is not limited to AKS. Microsoft Entra ID can trust the ServiceAccount tokens of any Kubernetes cluster whose token issuer is publicly discoverable, such as a Google Kubernetes Engine (GKE) cluster. To set up the trust, you create a federated credential for the token issuer of the cluster. On AKS, the Azure workload identity webhook adds the token and the Azure settings to the proxy pod. On other clusters, you can install the open source webhook, or add the token and the settings yourself. To set up workload identity, follow the steps for your cluster:

Use Microsoft Entra Workload ID on AKS

To authenticate with Microsoft Entra Workload ID, prepare the AKS cluster and the proxy service account before you create the AgentgatewayBackend.

  1. Enable workload identity on the AKS cluster.

  2. Create a user-assigned managed identity. Then, grant that identity the least-privilege role that the Azure AI resource requires. For example, the Foundry User role grants access to Azure AI Foundry. This role was previously named Azure AI User. For more information, see Role-based access control for Microsoft Foundry.

  3. Create a federated credential that trusts the service account that the proxy uses. By default, the deployer names this service account after the Gateway. For the default gateway from the quickstart, use the following subject. The subject must match the namespace and the service account name exactly.

    system:serviceaccount:agentgateway-system:agentgateway-proxy
  4. Create an AgentgatewayParameters resource that annotates the service account with the managed identity client ID, and that labels the proxy pod for the Azure workload identity webhook. If the gateway already refers to an AgentgatewayParameters resource, add these overlays to that resource instead.

    kubectl apply --server-side -f- <<EOF
    apiVersion: agentgateway.dev/v1alpha1
    kind: AgentgatewayParameters
    metadata:
      name: azure-workload-identity
      namespace: agentgateway-system
    spec:
      serviceAccount:
        metadata:
          annotations:
            azure.workload.identity/client-id: <managed-identity-client-id>
      deployment:
        spec:
          template:
            metadata:
              labels:
                azure.workload.identity/use: "true"
    EOF
  5. Attach the AgentgatewayParameters resource to the gateway.

    kubectl patch gateway agentgateway-proxy \
      --namespace agentgateway-system \
      --type merge \
      --patch '{"spec":{"infrastructure":{"parametersRef":{"group":"agentgateway.dev","kind":"AgentgatewayParameters","name":"azure-workload-identity"}}}}'

Use Microsoft Entra Workload ID on GKE

GKE serves the token issuer of each cluster from a public Google endpoint, so Microsoft Entra ID can verify the ServiceAccount tokens of the proxy. GKE does not include the Azure workload identity webhook. The following steps add the federated token volume and the Azure environment variables to the proxy pod in an AgentgatewayParameters resource. You can also install the webhook on the cluster yourself. In that case, complete steps 1 to 4, and then configure the proxy as described in steps 4 and 5 of the AKS steps. You can use the same steps on other Kubernetes clusters that have a publicly discoverable token issuer.

Before you begin, install jq and the Azure CLI, and set your kubectl context to the GKE cluster.

  1. Get the token issuer URL from the discovery document of the cluster. Do not build the URL from the selfLink value of the cluster. For a zonal cluster, selfLink contains /zones/, but the issuer contains /locations/. Microsoft Entra ID rejects a token if the issuer does not match exactly.

    export ISSUER=$(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer)
    echo $ISSUER

    Example output:

    https://container.googleapis.com/v1/projects/my-project/locations/us-central1-b/clusters/my-cluster
  2. Confirm that the discovery document is publicly available. Microsoft Entra ID gets the document and the signing keys without credentials. The following command sends no credentials. The output is the same issuer URL.

    curl -s "$ISSUER/.well-known/openid-configuration" | jq -r .issuer
  3. Create a user-assigned managed identity. Then, grant that identity the least-privilege role that the Azure AI resource requires, such as Foundry User. Record the client ID of the identity and the ID of your Microsoft Entra tenant for a later step.

  4. Create a federated credential on the managed identity that trusts the service account of the proxy. The deployer names this service account after the Gateway. The following example uses the default gateway from the quickstart.

    az identity federated-credential create \
      --name agentgateway-gke \
      --identity-name <managed-identity-name> \
      --resource-group <resource-group> \
      --issuer "$ISSUER" \
      --subject "system:serviceaccount:agentgateway-system:agentgateway-proxy" \
      --audiences "api://AzureADTokenExchange"
    SettingDescription
    --issuerThe token issuer URL of the cluster from step 1.
    --subjectThe service account of the proxy, in the format system:serviceaccount:<namespace>:<name>. The namespace and the name must match exactly.
    --audiencesSet to api://AzureADTokenExchange. This value must match the audience of the projected token in the next step.
  5. Create an AgentgatewayParameters resource for the proxy pod. The resource projects a ServiceAccount token for Microsoft Entra ID, and sets the Azure environment variables that the proxy reads. If the gateway already refers to an AgentgatewayParameters resource, add these settings to that resource instead.

    kubectl apply --server-side -f- <<EOF
    apiVersion: agentgateway.dev/v1alpha1
    kind: AgentgatewayParameters
    metadata:
      name: azure-workload-identity
      namespace: agentgateway-system
    spec:
      env:
      - name: AZURE_CLIENT_ID
        value: <managed-identity-client-id>
      - name: AZURE_TENANT_ID
        value: <entra-tenant-id>
      - name: AZURE_FEDERATED_TOKEN_FILE
        value: /var/run/secrets/azure/tokens/azure-identity-token
      deployment:
        spec:
          template:
            spec:
              containers:
              - name: agentgateway
                volumeMounts:
                - name: azure-identity-token
                  mountPath: /var/run/secrets/azure/tokens
                  readOnly: true
              volumes:
              - name: azure-identity-token
                projected:
                  sources:
                  - serviceAccountToken:
                      path: azure-identity-token
                      audience: api://AzureADTokenExchange
                      expirationSeconds: 3600
    EOF
    SettingDescription
    env.AZURE_CLIENT_IDThe client ID of the managed identity.
    env.AZURE_TENANT_IDThe ID of your Microsoft Entra tenant. The proxy gets the token from the Microsoft Entra endpoint of the Azure public cloud, https://login.microsoftonline.com. To use a different endpoint, such as for a sovereign cloud, also set the AZURE_AUTHORITY_HOST environment variable.
    env.AZURE_FEDERATED_TOKEN_FILEThe path to the projected token in the proxy container. The path is the mountPath of the volume, followed by the path of the token.
    deployment.spec.template.spec.containers[].nameSet to agentgateway, the name of the proxy container. The controller merges the volume mount into the container that has this name.
    volumes[].projected.sources[].serviceAccountToken.audienceSet to api://AzureADTokenExchange, the audience that Microsoft Entra ID requires.
    volumes[].projected.sources[].serviceAccountToken.expirationSecondsThe lifetime of the projected token. The kubelet replaces the token before it expires, and the proxy reads the file again. You do not have to restart the proxy.
  6. Attach the AgentgatewayParameters resource to the gateway.

    kubectl patch gateway agentgateway-proxy \
      --namespace agentgateway-system \
      --type merge \
      --patch '{"spec":{"infrastructure":{"parametersRef":{"group":"agentgateway.dev","kind":"AgentgatewayParameters","name":"azure-workload-identity"}}}}'
  7. Confirm that the proxy deployment mounts the token with the Microsoft Entra audience.

    kubectl get deployment agentgateway-proxy \
      --namespace agentgateway-system \
      -o jsonpath='{.spec.template.spec.volumes[?(@.name=="azure-identity-token")].projected.sources[0].serviceAccountToken.audience}'

    Example output:

    api://AzureADTokenExchange

Set up access to Azure

  1. Retrieve the resource name and, if applicable, the project name from the Azure AI Foundry portal or the Azure portal. For example:

    • For an Azure OpenAI endpoint like https://{my-resource}.openai.azure.com, the resource name is my-resource.
    • For an Azure AI Foundry endpoint like https://{my-resource}.services.ai.azure.com and path /api/projects/{my-project}, the resource name is my-resource and the project name is my-project. If the resource name and the project name are the same, you can leave the projectName field empty.
  2. Store the API key to access your model deployment in an environment variable. If you are using implicit Entra ID authentication (such as managed identity or workload identity), you can skip this step.

    export AZURE_API_KEY=<insert your model deployment key>
  3. Create a Kubernetes secret to store your API key. If you are using implicit Entra ID authentication, skip this step.

    kubectl apply -f- <<EOF
    apiVersion: v1
    kind: Secret
    metadata:
      name: azure-secret
      namespace: agentgateway-system
    type: Opaque
    stringData:
      Authorization: $AZURE_API_KEY
    EOF
  4. Create an AgentgatewayBackend resource to configure the Azure LLM provider.

    kubectl apply -f- <<EOF
    apiVersion: agentgateway.dev/v1alpha1
    kind: AgentgatewayBackend
    metadata:
      name: azure
      namespace: agentgateway-system
    spec:
      ai:
        provider:
          azure:
            resourceName: my-resource
            resourceType: OpenAI
            model: gpt-4.1-mini
      policies:
        auth:
          secretRef:
            name: azure-secret
    EOF

    To use explicit managed identity or workload identity authentication instead of an API key, apply one of the following AgentgatewayBackend configurations.

    To use workload identity, first prepare the cluster as described in Use Microsoft Entra Workload ID on AKS or Use Microsoft Entra Workload ID on GKE. Then, select workload identity in the AgentgatewayBackend.

    kubectl apply -f- <<EOF
    apiVersion: agentgateway.dev/v1alpha1
    kind: AgentgatewayBackend
    metadata:
      name: azure
      namespace: agentgateway-system
    spec:
      ai:
        provider:
          azure:
            resourceName: my-resource
            resourceType: OpenAI
            model: gpt-4.1-mini
      policies:
        auth:
          azure:
            workloadIdentity: {}
    EOF

    Review the following table to understand this configuration. For more information, see the API reference.

    SettingDescription
    ai.provider.azureDefine the Azure provider.
    azure.resourceNameThe Azure resource name used to construct the endpoint hostname.
    azure.resourceTypeThe endpoint type: OpenAI for Azure OpenAI Service, or Foundry for Azure AI Foundry.
    azure.modelThe model to use for requests, such as gpt-4.1-mini.
    azure.projectNameThe Foundry project name. Required when resourceType is Foundry.
    azure.apiVersionOptional API version override. Defaults to v1. For legacy deployments, use a dated version like 2025-01-01-preview.
    policies.auth.azure.workloadIdentityUse Azure workload identity. Leave the object empty. The proxy uses the federated token and the Azure environment variables in the proxy pod. On AKS, the Azure workload identity webhook adds them. On other clusters, the AgentgatewayParameters resource adds them.
    policies.auth.azure.managedIdentityUse an Azure managed identity. Leave the object empty to use the system-assigned identity. To use a user-assigned identity, set one of clientId, objectId, or resourceId.
  5. Create an HTTPRoute resource that routes incoming traffic to the AgentgatewayBackend. The following example sets up a route. Note that agentgateway automatically rewrites the endpoint to the appropriate chat completion endpoint of the LLM provider for you, based on the LLM provider that you set up in the AgentgatewayBackend resource.

    kubectl apply -f- <<EOF
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
      name: azure
      namespace: agentgateway-system
    spec:
      parentRefs:
        - name: agentgateway-proxy
          namespace: agentgateway-system
      rules:
      - backendRefs:
        - name: azure
          namespace: agentgateway-system
          group: agentgateway.dev
          kind: AgentgatewayBackend
    EOF
  6. Send a request to the LLM provider API along the route that you previously created. Verify that the request succeeds and that you get back a response from the chat completion API.

    Cloud Provider LoadBalancer:

    curl "$INGRESS_GW_ADDRESS/v1/chat/completions" -H content-type:application/json  -d '{
       "model": "gpt-4.1-mini",
       "messages": [
         {
           "role": "system",
           "content": "You are a helpful assistant."
         },
         {
           "role": "user",
           "content": "Write a short haiku about cloud computing."
         }
       ]
     }' | jq

    Localhost:

    curl "localhost:8080/v1/chat/completions" -H content-type:application/json  -d '{
       "model": "gpt-4.1-mini",
       "messages": [
         {
           "role": "system",
           "content": "You are a helpful assistant."
         },
         {
           "role": "user",
           "content": "Write a short haiku about cloud computing."
         }
       ]
     }' | jq

    Example output:

    {
      "id": "chatcmpl-9A8B7C6D5E4F3G2H1",
      "object": "chat.completion",
      "created": 1727967462,
      "model": "gpt-4.1-mini",
      "choices": [
        {
          "index": 0,
          "message": {
            "role": "assistant",
            "content": "Floating servers bright,\nData streams through endless sky,\nClouds hold all we need."
          },
          "finish_reason": "stop"
        }
      ],
      "usage": {
        "prompt_tokens": 28,
        "completion_tokens": 19,
        "total_tokens": 47
      }
    }

Use Claude models on Azure AI Foundry

Azure AI Foundry hosts Anthropic Claude models at native Anthropic endpoints. When you set resourceType: Foundry and a model name that starts with claude-, agentgateway automatically routes requests to the Anthropic-native path (/anthropic/v1/messages) instead of the OpenAI-compatible path, and injects the required anthropic-version header. No extra configuration is needed beyond specifying a Claude model name.

Note

For more information about Claude models on Azure AI Foundry, see the Microsoft documentation.

  1. Create a Kubernetes secret to store your Azure AI Foundry API key.

    export AZURE_API_KEY=<insert your Azure AI Foundry API key>
    kubectl apply -f- <<EOF
    apiVersion: v1
    kind: Secret
    metadata:
      name: azure-claude-secret
      namespace: agentgateway-system
    type: Opaque
    stringData:
      Authorization: $AZURE_API_KEY
    EOF
  2. Create an AgentgatewayBackend resource that uses the azure provider with resourceType: Foundry and a Claude model name.

    kubectl apply -f- <<EOF
    apiVersion: agentgateway.dev/v1alpha1
    kind: AgentgatewayBackend
    metadata:
      name: azure-claude
      namespace: agentgateway-system
    spec:
      ai:
        provider:
          azure:
            resourceName: my-foundry-resource
            resourceType: Foundry
            projectName: my-project
            model: claude-3-5-haiku-20241022
      policies:
        auth:
          secretRef:
            name: azure-claude-secret
    EOF

    Review the following table to understand this configuration.

    SettingDescription
    azure.resourceNameThe Azure AI Foundry resource name used to construct the endpoint hostname.
    azure.resourceTypeSet to Foundry to use Azure AI Foundry endpoints.
    azure.projectNameThe Foundry project name.
    azure.modelThe Claude model to use, for example claude-3-5-haiku-20241022. The model name must start with claude- to trigger routing to the Anthropic-native endpoint.
    policies.auth.secretRefReferences the secret that holds the Azure AI Foundry API key. The key is automatically sent in the Authorization header.
  3. Create an HTTPRoute resource that routes incoming traffic to the AgentgatewayBackend.

    kubectl apply -f- <<EOF
    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
      name: azure-claude
      namespace: agentgateway-system
    spec:
      parentRefs:
        - name: agentgateway-proxy
          namespace: agentgateway-system
      rules:
      - backendRefs:
        - name: azure-claude
          namespace: agentgateway-system
          group: agentgateway.dev
          kind: AgentgatewayBackend
        matches:
           - path:
               type: PathPrefix
               value: /azure-claude #Path  example
    EOF
  4. Send a request to verify the setup.

    Cloud Provider LoadBalancer:

    curl "$INGRESS_GW_ADDRESS/azure-claude" -H content-type:application/json -d '{
      "max_tokens": 256,
      "messages": [
        {
          "role": "user",
          "content": "Hello!"
        }
      ]
    }' | jq

    Localhost:

    curl "localhost:8080/azure-claude" -H content-type:application/json -d '{
      "max_tokens": 256,
      "messages": [
        {
          "role": "user",
          "content": "Hello!"
        }
      ]
    }' | jq

Use Decisions on Azure

Azure models can answer classification questions about an input, such as which of several choices fits best or how likely a statement is true. agentgateway accepts two client request formats for these models:

  • Decisions API (/v1/decisions): The OpenAI-compatible format, with predicate, choice, and score question types.
  • SystemOne (/v1/systemone): The TypeSafe format, with corresponding noul, choice, and score question types.

Choose the Azure endpoint that hosts your model. Agentgateway converts requests and responses to the upstream endpoint’s native format.

  • Azure OpenAI: Deploy a model that supports the Azure OpenAI Decisions API, such as gpt-6-luna. Azure OpenAI receives both client formats at /openai/v1/decisions: Decisions requests are sent unchanged, and SystemOne requests are converted to Decisions.

  • Azure AI Foundry: Deploy Microsoft-Decision-1 in your Foundry resource. Foundry receives both client formats at https://{resourceName}.cognitiveservices.azure.com/providers/microsoft/v1/systemone: SystemOne requests are sent unchanged, and Decisions requests are converted to SystemOne.

Before you begin, set up workload identity for the agentgateway-proxy Gateway, either on AKS or on GKE.

Configure the Azure backend

Create the AgentgatewayBackend for the endpoint that hosts your model. Replace the resource, project, and deployment names with your own values.

Azure OpenAI

kubectl apply --server-side -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: azure-decisions
  namespace: agentgateway-system
spec:
  ai:
    provider:
      azure:
        model: my-decision-deployment
        resourceName: my-openai-resource
        resourceType: OpenAI
  policies:
    ai:
      routes:
        /v1/decisions: Decisions
        /v1/systemone: SystemOne
    auth:
      azure:
        workloadIdentity: {}
EOF

Azure AI Foundry

kubectl apply --server-side -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: azure-decisions
  namespace: agentgateway-system
spec:
  ai:
    provider:
      azure:
        model: my-decision-deployment
        projectName: my-project
        resourceName: my-foundry-resource
        resourceType: Foundry
  policies:
    ai:
      routes:
        /v1/decisions: Decisions
        /v1/systemone: SystemOne
    auth:
      azure:
        workloadIdentity: {}
EOF

Review the following table to understand this configuration.

SettingDescription
azure.resourceNameThe Azure resource name used to construct the endpoint hostname.
azure.resourceTypeSet to OpenAI for Azure OpenAI or Foundry for Azure AI Foundry.
azure.projectNameThe Foundry project name. The CRD requires this field for the Foundry resource type, even though Decisions and SystemOne requests go to the resource endpoint and do not use the project. Do not set this field for OpenAI.
azure.modelThe name of your Azure deployment. agentgateway sends this name upstream and replaces the model value in the client request.
policies.ai.routesMaps each request path to a request format, Decisions or SystemOne.
policies.auth.azure.workloadIdentityAuthenticate to Azure with the workload identity that you set up for the proxy. Leave the object empty.

Route Decisions and SystemOne traffic

Create one HTTPRoute that sends both client request formats to the backend.

kubectl apply --server-side -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: azure-decisions
  namespace: agentgateway-system
spec:
  parentRefs:
  - name: agentgateway-proxy
  rules:
  - backendRefs:
    - group: agentgateway.dev
      kind: AgentgatewayBackend
      name: azure-decisions
    matches:
    - method: POST
      path:
        type: Exact
        value: /v1/decisions
    - method: POST
      path:
        type: Exact
        value: /v1/systemone
EOF

Send requests

  1. Set the address of the gateway.

    Cloud Provider LoadBalancer:

    export GATEWAY_URL=http://$INGRESS_GW_ADDRESS

    Localhost: Forward the gateway Service in one terminal. Then set the address in another terminal.

    kubectl port-forward -n agentgateway-system \
      service/agentgateway-proxy 8080:80
    export GATEWAY_URL=http://localhost:8080
  2. Send a choice question through the Decisions API.

    curl --fail-with-body --max-time 120 "$GATEWAY_URL/v1/decisions" \
      -H 'Content-Type: application/json' \
      -d '{
        "input": "I was charged twice for my subscription.",
        "model": "my-decision-deployment",
        "questions": [{
          "choices": [
            {"description": "Payments and invoices", "value": "billing"},
            {"description": "Software problems", "value": "technical"}
          ],
          "instructions": "Which team should handle this support ticket?",
          "name": "department",
          "type": "choice"
        }]
      }'

    The response includes a department choice with a probability for each value, and token usage. The Decisions API also supports predicate and score questions.

  3. Send the same question in the SystemOne client format. SystemOne takes the text in state and the questions as a map keyed by name, and returns an answers map. Its question types are noul, choice, and score.

    curl --fail-with-body --max-time 120 "$GATEWAY_URL/v1/systemone" \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "my-decision-deployment",
        "questions": {
          "department": {
            "criteria": {
              "billing": "Payments and invoices",
              "technical": "Software problems"
            },
            "instructions": "Which team should handle this support ticket?",
            "type": "choice"
          }
        },
        "state": "I was charged twice for my subscription."
      }'

    Note

    Prompt guards are disabled for requests in the SystemOne format.

Clean up

You can remove the resources that you created in this guide.

kubectl delete httproute azure-decisions -n agentgateway-system
kubectl delete AgentgatewayBackend azure-decisions -n agentgateway-system

If you forwarded the Service, stop the port-forward with Ctrl+C.

Next steps

Was this page helpful?
Agentgateway assistant

Ask me anything about agentgateway configuration, features, or usage.

Note: AI-generated content might contain errors; please verify and test all returned information.

Tip: one topic per conversation gives the best results. Use the + button in the chat header to start a new conversation.

Switching topics? Starting a new conversation improves accuracy.
↑↓ navigate ↵ select esc dismiss

What could be improved?

Your feedback helps us improve assistant answers and identify docs gaps we should fix.

Need more help? Join us on Discord: https://discord.gg/y9efgEmppm

Want to use your own agent? Add the Solo MCP server to query our docs directly. Get started here: https://search.solo.io/.