Skip to main content

Build an AI controller with Crossplane

In this tutorial, you run a Kubernetes controller whose reconciliation logic is written in plain English. A Crossplane WatchOperation watches an nginx Deployment and calls a local LLM whenever it changes. The LLM reads the current state, applies the rule in its systemPrompt, and returns a corrected manifest. Crossplane applies it.

By the end of this tutorial, you can:

  • Deploy a WatchOperation that calls a local LLM on every resource change
  • Watch the controller detect and correct a policy violation automatically
  • Update the enforcement rule by editing a single field in YAML

The model running in this tutorial is qwen2.5:1.5b via Ollama — running entirely on your local machine. No cloud API key is required.

Prerequisites​

Install the following before starting:

Install the up CLI​

This tutorial requires up CLI v0.44.3.

curl -sL "https://cli.upbound.io" | VERSION=v0.44.3 sh

Move the binary into your PATH:

sudo mv up /usr/local/bin/

If you don't have sudo access:

mkdir -p ~/.local/bin && mv up ~/.local/bin/
export PATH="$HOME/.local/bin:$PATH"

Add the export line to your shell profile (~/.bashrc, ~/.zshrc, or equivalent) to make it permanent.

Verify the installation:

up version

Create the project​

Create the project directory​

mkdir english-controller
cd english-controller

All commands from this point run from inside the english-controller directory.

Create the project manifest​

cat > upbound.yaml <<'EOF'
apiVersion: meta.dev.upbound.io/v2alpha1
kind: Project
metadata:
  name: english-controller
spec:
  dependsOn:
  - apiVersion: pkg.crossplane.io/v1
    kind: Function
    package: xpkg.upbound.io/crossplane-contrib/function-auto-ready
    version: '>=v0.0.0'
  - apiVersion: pkg.crossplane.io/v1
    kind: Function
    package: xpkg.upbound.io/upbound/function-openai
    version: v0.3.0
  description: A Kubernetes controller whose enforcement logic is written in plain English.
EOF

Create the WatchOperation​

The WatchOperation is the controller. It watches the nginx Deployment and calls upbound-function-openai whenever it changes. The function sends the current resource state to the LLM along with the systemPrompt rule. The LLM returns a corrected manifest. Crossplane applies it.

mkdir -p operations/replicas
cat > operations/replicas/operation.yaml <<'EOF'
apiVersion: ops.crossplane.io/v1alpha1
kind: WatchOperation
metadata:
  name: replicas
spec:
  concurrencyPolicy: Forbid
  successfulHistoryLimit: 3
  failedHistoryLimit: 1
  operationTemplate:
    spec:
      mode: Pipeline
      pipeline:
      - functionRef:
          name: upbound-function-openai
        input:
          apiVersion: openai.fn.upbound.io/v1alpha1
          kind: Prompt
          systemPrompt: |-
            You are a Kubernetes controller. Output raw YAML only — no markdown, no code fences, no backticks, no explanations.

            Rule: if spec.replicas is less than 3, set it to 3. Otherwise keep it unchanged.
          userPrompt: |-
            Inspect the nginx Deployment and output the corrected manifest.
            Output only the Deployment manifest with the correct spec.replicas value.
            Include apiVersion, kind, metadata (name: nginx, namespace: default), and spec.
            Start your response with 'apiVersion:'
        step: deployment-analysis
        credentials:
        - name: gpt
          source: Secret
          secretRef:
            namespace: crossplane-system
            name: gpt
  watch:
    apiVersion: apps/v1
    kind: Deployment
    namespace: default
EOF
info

The explicit output instructions in userPrompt are needed for qwen2.5:1.5b. With a larger model like gpt-4o, the systemPrompt can be much simpler — just the rule itself, without format guidance.

Create the nginx deployment​

Create the starting state — 1 replica. The AI controller will correct this.

mkdir -p examples
cat > examples/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
EOF

Set up Ollama​

Ollama runs the LLM locally. Install it and pull the model before starting the cluster — the model is ~1 GB.

Install Ollama​

curl -fsSL https://ollama.com/install.sh | sh

If the install script doesn't work for your OS, download directly from ollama.com/download.

Start Ollama​

On Linux, the install script registers a systemd service that starts Ollama automatically. On macOS, start it manually in a separate terminal if ollama list returns "could not connect to ollama server":

ollama serve

Pull the model​

ollama pull qwen2.5:1.5b

Confirm the model downloaded:

ollama list

You should see qwen2.5:1.5b in the output.

Start the project​

Run from inside the english-controller directory:

up project run --local --control-plane-version=2.1.4-up.2

This creates a kind cluster, installs UXP, and deploys the function packages declared in upbound.yaml. It exits when the cluster is ready.

warning

If up project run --local exits non-zero and prints traces export: context deadline exceeded, check whether functions were installed:

kubectl get functions

If functions appear, provisioning succeeded despite the telemetry error. If the list is empty, delete the cluster and retry:

kind delete cluster --name up-english-controller

Verify your network allows outbound connections to xpkg.upbound.io on port 443.

Configure kubectl​

kind get kubeconfig --name up-english-controller > ~/.kube/config
warning

This overwrites your existing ~/.kube/config. To preserve existing contexts, merge instead:

kind get kubeconfig --name up-english-controller > ~/.kube/config-upbound
KUBECONFIG=~/.kube/config:~/.kube/config-upbound \
  kubectl config view --flatten > ~/.kube/config.merged
mv ~/.kube/config.merged ~/.kube/config

Verify the connection:

kubectl get nodes

Wire Ollama into the cluster​

The kind cluster's pods need to reach Ollama running on your host. Create a Kubernetes Service and Endpoints that route cluster traffic to your machine.

  1. Get the host IP on the kind bridge network:

    Linux:

    HOST_IP=$(docker network inspect kind -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}')
    echo "Host IP: $HOST_IP"

    macOS (Docker Desktop):

    HOST_IP=$(docker run --rm alpine sh -c 'getent hosts host.docker.internal' 2>/dev/null | awk '{print $1}')
    echo "Host IP: $HOST_IP"

  2. Create the ollama namespace and register Ollama as a cluster service:

    kubectl create namespace ollama --dry-run=client -o yaml | kubectl apply -f -

    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: Service
    metadata:
      name: ollama
      namespace: ollama
    spec:
      ports:
      - port: 11434
        targetPort: 11434
    ---
    apiVersion: v1
    kind: Endpoints
    metadata:
      name: ollama
      namespace: ollama
    subsets:
    - addresses:
      - ip: ${HOST_IP}
      ports:
      - port: 11434
    EOF

  3. Create the credentials secret that function-openai uses to reach Ollama:

    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: Secret
    metadata:
      name: gpt
      namespace: crossplane-system
    stringData:
      OPENAI_API_KEY: ollama
      OPENAI_BASE_URL: http://${HOST_IP}:11434/v1
      OPENAI_MODEL: qwen2.5:1.5b
    EOF

    The OPENAI_BASE_URL points to Ollama's OpenAI-compatible API. To switch to a cloud model, replace OPENAI_BASE_URL with https://api.openai.com/v1, set OPENAI_API_KEY to your API key, and update OPENAI_MODEL. The WatchOperation works identically regardless of which model runs.

Verify the setup​

Wait for function-openai to become healthy:

kubectl get functions

Wait until upbound-function-openai shows HEALTHY: True.

warning

If kubectl get functions returns No resources found, up project run --local did not complete successfully. Delete the cluster with kind delete cluster --name up-english-controller and restart from Start the project.

Apply the starting state​

Apply the nginx Deployment at 1 replica:

kubectl apply -f examples/deployment.yaml

Verify it's running:

kubectl get deployment nginx

You should see READY: 1/1.

Run the AI controller​

An nginx Deployment is running in the cluster with only 1 replica. Apply the WatchOperation and watch it fix that.

See the current state​

kubectl get deployment nginx

READY 1/1 is the starting point.

Apply the WatchOperation​

Crossplane Operations are Kubernetes objects that run logic against your cluster on a trigger. There are three kinds:

KindTrigger
WatchOperationEvery time a specific resource changes
CronOperationOn a schedule
OperationOnce, on demand

This tutorial uses a WatchOperation. It watches the nginx Deployment and calls an LLM every time it changes.

kubectl apply -f operations/replicas/operation.yaml

The WatchOperation fires immediately because the Deployment already exists.

Watch it act​

kubectl get deployment nginx -w

Within 60–90 seconds, replicas jump from 1 to 3. The LLM read the Deployment, decided it violated the rule, and patched it.

Press Ctrl+C when replicas reach 3.

Inspect the operation records​

Each Operation object is a record of a single invocation.

kubectl get watchoperations
kubectl get operations

Pick one of the operation names and describe it:

kubectl describe operation <name>

The Events section shows the exact YAML the model returned and what the controller applied.

Watch it heal​

The WatchOperation re-evaluates on every change. If anything modifies the Deployment — a human, a CI pipeline, a rollout — the rule re-applies. This is drift detection with reasoning: not just "was this field changed" but "does this still satisfy the intent?"

Scale down nginx​

kubectl scale deployment nginx --replicas=1

Watch the controller heal it​

kubectl get deployment nginx -w

Within 30–60 seconds, replicas climb back to 3. The WatchOperation fired because the Deployment changed. The LLM saw 1 replica, decided it violated the rule, and patched it.

Press Ctrl+C when replicas are back at 3.

See what fired​

kubectl get watchoperations
kubectl get operations

Each entry is a record of what fired, what the model decided, and what changed. The most recent one captured the scale-down event and the correction.

See where the model runs​

kubectl get secret gpt -n crossplane-system -o yaml

OPENAI_BASE_URL points to Ollama's OpenAI-compatible API running locally on your machine — no data leaves the machine. Change that URL to https://api.openai.com/v1 and update OPENAI_MODEL, and the WatchOperation works identically.

Change the rules​

The enforcement logic is a text field. To change the policy, edit systemPrompt and re-apply. No code change. No build pipeline. No rollout.

Update the minimum replicas to 5​

Open operations/replicas/operation.yaml. Find the systemPrompt and change the rule line from:

Rule: if spec.replicas is less than 3, set it to 3. Otherwise keep it unchanged.

To:

Rule: if spec.replicas is less than 5, set it to 5. Otherwise keep it unchanged.

Edit the file directly:

macOS:

sed -i '' 's/less than 3, set it to 3/less than 5, set it to 5/' \
  operations/replicas/operation.yaml

Linux:

sed -i 's/less than 3, set it to 3/less than 5, set it to 5/' \
  operations/replicas/operation.yaml
info

With qwen2.5:1.5b, keep the full userPrompt output instructions in place. The explicit YAML template keeps the small model's output reliable. With a larger model like gpt-4o, you can remove the userPrompt entirely and keep only the rule in systemPrompt.

Apply the updated operation​

kubectl apply -f operations/replicas/operation.yaml

Trigger and observe​

Scale nginx down to 1:

kubectl scale deployment nginx --replicas=1

Watch the updated rule enforce 5 replicas:

kubectl get deployment nginx -w

This takes 30–45 seconds. Press Ctrl+C when you see 5 ready replicas.

Verify​

kubectl get watchoperations
kubectl get operations

Same architecture, different policy — changed by editing a text field.

tip

Try adding a conditional rule to the systemPrompt:

If the deployment name contains 'prod', require at least 5 replicas.
Otherwise, require at least 2.

The model interprets natural language conditions the same way it interprets simple numeric rules. Any platform engineer can read the rule, change it, and version it in Git — without writing Go.

Clean up​

Delete the demo resources:

kubectl delete watchoperation replicas
kubectl delete operations --all
kubectl delete deployment nginx

Delete the cluster:

kind delete cluster --name up-english-controller

Next steps​

In this tutorial, you:

  • Created a Crossplane project with a WatchOperation and a KCL function
  • Deployed a controller that calls a local LLM on every Deployment change
  • Watched the controller detect and correct a replica count violation
  • Updated the enforcement policy by editing a single field in YAML

Continue with: