Helm Installation Guide

Installing the KFP Operator

This guide provides comprehensive instructions for installing the Kubeflow Pipelines Operator in your Kubernetes cluster. We recommend using Helm for a declarative approach to managing Kubernetes resources.

Overview

The KFP Operator installation consists of three main components:

1. Prerequisites: Required dependencies (Argo Workflows, Argo Events)
2. KFP Operator: The core operator and controllers
3. Providers: ML orchestration platform integrations (KFP, Vertex AI)

Prerequisites

Before installing the KFP Operator, ensure your cluster meets these requirements:

Cluster Requirements

• Kubernetes: v1.21 or later (tested up to v1.28)
• Cluster Admin Access: Required for installing CRDs and cluster-wide resources
• Storage: Persistent storage for pipeline artifacts and metadata
• Network: Outbound internet access for downloading container images

Required Dependencies

1. Argo Workflows

Version: 3.1.6 - 3.4.x Purpose: Workflow execution engine for pipeline orchestration

# Install Argo Workflows (cluster-wide)
kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.4.4/install.yaml

Verification:

kubectl get pods -n argo
# Should show argo-server and workflow-controller pods running

2. Argo Events (Optional but Recommended)

Version: 1.7.4 or later Purpose: Event-driven pipeline automation

# Install Argo Events (cluster-wide)
kubectl create namespace argo-events
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install.yaml

Verification:

kubectl get pods -n argo-events
# Should show eventbus-controller and eventsource-controller pods running

Optional Dependencies

Cert-Manager (Recommended for Production)

Purpose: Automatic TLS certificate management for webhooks

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

Prometheus Operator (For Monitoring)

Purpose: Metrics collection and monitoring

# Install Prometheus Operator
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack

Helm Installation

This guide assumes you have Helm 3.x installed:

# Verify Helm installation
helm version
# Should show version 3.x

Installing the KFP Operator

The KFP Operator installation consists of the core operator and at least one provider for ML orchestration.

Quick Installation

For a basic installation with default settings:

# Add the KFP Operator Helm repository
helm repo add kfp-operator https://sky-uk.github.io/kfp-operator/
helm repo update

# Install with default values
helm install kfp-operator kfp-operator/kfp-operator

Custom Installation

For production deployments, create a custom values.yaml file:

# values.yaml - Basic configuration
namespace:
  create: true
  name: kfp-operator-system

manager:
  replicas: 2  # High availability
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi

  # Monitoring configuration
  monitoring:
    create: true
    serviceMonitor:
      create: true  # For Prometheus Operator

# Enable event-driven workflows
statusFeedback:
  enabled: true

# Logging configuration
logging:
  verbosity: 1  # 0=error, 1=info, 2=debug

Install with custom configuration:

helm install kfp-operator kfp-operator/kfp-operator -f values.yaml

Installation Methods

Method 1: Helm Repository (Recommended)

# Add repository and install
helm repo add kfp-operator https://sky-uk.github.io/kfp-operator/
helm install kfp-operator kfp-operator/kfp-operator -f values.yaml

Method 2: Local Development (Requires local kubernetes cluster)

# Clone repository and install from source
git clone https://github.com/sky-uk/kfp-operator.git
cd kfp-operator
helm install kfp-operator ./helm/kfp-operator -f values.yaml

Verification

Verify the installation was successful:

# Check operator pods
kubectl get pods -n kfp-operator-system

# Check CRDs were installed
kubectl get crd | grep pipelines.kubeflow.org

# Check operator logs
kubectl logs -n kfp-operator-system deployment/kfp-operator-controller-manager

Expected output:

NAME                                           READY   STATUS    RESTARTS   AGE
kfp-operator-controller-manager-xxx-xxx        2/2     Running   0          2m

Namespace Configuration

The operator can be installed in any namespace. Common patterns:

Dedicated Namespace (Recommended)

namespace:
  create: true
  name: kfp-operator-system

Existing Namespace

namespace:
  create: false
  name: ml-platform

Multi-tenant Setup

# Install operator in system namespace
namespace:
  name: kfp-operator-system

# Configure RBAC for multiple tenant namespaces
manager:
  rbac:
    create: true
    # Additional cluster roles will be created

Configuration Values

Valid configuration options to override the Default values.yaml are:

Examples for these values can be found in the test configuration

Providers

Please refer to your chosen provider instructions before proceeding. Supported providers are:

• Kubeflow Pipelines V2
• Vertex AI

To install your chosen provider, create a Provider resource in a namespace that the operator can access (see the rbac setup below for reference). Once it is applied the Provider controller will reconcile and create the Provider Deployment and Provider Service within the same namespace that the Provider resource was applied.

Provider workflows

The operator runs resource-management Argo Workflows in the same namespace as the Provider they belong to, so that each provider is isolated from the others. Argo cannot reference a WorkflowTemplate across namespaces, therefore every provider namespace must contain its own copy of the WorkflowTemplates and the execution RBAC (ServiceAccount, Role and RoleBinding) the workflows run with.

These per-namespace resources are provisioned by the kfp-provider-workflows chart, which is installed once per provider namespace. Install one release for each namespace in which you create a Provider:

helm install <release-name> ./helm/kfp-operator/charts/kfp-provider-workflows \
  --namespace <provider-namespace> \
  --set namespace=<provider-namespace>

By default the chart also creates the Provider resource. Provide its required spec fields, or set provider.create=false to manage the Provider yourself:

helm install <release-name> ./helm/kfp-operator/charts/kfp-provider-workflows \
  --namespace <provider-namespace> \
  --set namespace=<provider-namespace> \
  --set provider.name=<provider-name> \
  --set provider.serviceImage=<provider-service-image> \
  --set provider.pipelineRootStorage=<pipeline-root-storage>

Valid configuration options to override the Default values.yaml are:

The argo.* execution values (argo.ttlStrategy, argo.stepTimeoutSeconds, argo.securityContext, argo.containerDefaults and argo.metadata) control how the provider’s Argo Workflows run. Override them per release if a provider namespace needs different workflow behaviour.

Note: because workflows now run in the provider namespace, the artifact-repository credentials Secret that your Argo workflow-controller references (the accessKeySecret/secretKeySecret in its artifactRepository configuration — for example the MinIO/S3 secret) must exist in each provider namespace. Argo resolves that Secret in the namespace the workflow pods run in, so a copy that only lives in the Argo or operator namespace is not sufficient. The kfp-provider-workflows chart does not manage this Secret — it holds credentials owned by the platform team — so you must create it in every provider namespace yourself.

What the chart provisions

The kfp-provider-workflows chart provisions only namespace-scoped resources — everything a provider needs to operate inside its own namespace. Cluster-scoped resources are deliberately left to the platform team (see Cluster-scoped RBAC below). A single release renders:

Role-based access control (RBAC) for providers

The namespace-scoped RBAC above is handled by the chart. The resources in this section are cluster-scoped and are therefore not created by the chart — you must create them once (they are shared across provider namespaces). If you are not using the chart at all, see Deploying without Helm for the complete list including the namespace-scoped resources.

In order for Event Source Servers and the Controller to read the Providers you must configure their service accounts to have read permissions of Provider resources. e.g:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kfp-operator-kfp-providers-viewer-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kfp-operator-providers-viewer-role
subjects:
- kind: ServiceAccount
  name: kfp-operator-kfp #Used by Event Source Server
  namespace: kfp-operator-system
- kind: ServiceAccount
  name: kfp-operator-controller-manager #Used by KFP Controller
  namespace: kfp-operator-system

The provider service account itself (kfp-provider-<provider-name> by default) is created by the chart in the provider namespace. It still needs cluster-scoped read access to RunConfiguration and Run resources, granted by binding it to the ClusterRoles the operator ships. Create one ClusterRoleBinding per provider service account (replace kfp-provider-example and kfp-namespace with your values):

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kfp-operator-example-runconfiguration-viewer-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kfp-operator-runconfiguration-viewer-role
subjects:
- kind: ServiceAccount
  name: kfp-provider-example
  namespace: kfp-namespace
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kfp-operator-example-run-viewer-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kfp-operator-run-viewer-role
subjects:
- kind: ServiceAccount
  name: kfp-provider-example
  namespace: kfp-namespace

Note: the workflow-executor Role/RoleBinding and the Argo workflow ServiceAccount are namespace-scoped and are created for you by the kfp-provider-workflows chart. You only need to author them manually when deploying without Helm.

Kubeflow completion eventing required RBACs

If using the Kubeflow Pipelines Provider you will also need a ClusterRole for permission to interact with argo workflows for the eventing system for run completion events.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kfp-operator-kfp-eventsource-server-role
rules:
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  verbs:
  - get
  - list
  - patch
  - update
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kfp-operator-kfp-eventsource-server-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kfp-operator-kfp-eventsource-server-role
subjects:
- kind: ServiceAccount
  name: kfp-provider-example
  namespace: kfp-namespace

Deploying without Helm

If you do not use the kfp-provider-workflows chart, you are responsible for creating every resource a provider needs yourself. Because Argo cannot reference a WorkflowTemplate across namespaces, the namespace-scoped resources must be recreated once per provider namespace, while the cluster-scoped resources are created once per cluster and shared.

Namespace-scoped resources (once per provider namespace)

These are the resources the chart would otherwise create. All of them live in the provider’s namespace.

Example of the namespace-scoped RBAC (replace argo-workflow-sa, kfp-provider-example and kfp-namespace with your values):

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo-workflow-sa
  namespace: kfp-namespace
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kfp-provider-example
  namespace: kfp-namespace
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: workflow-executor
  namespace: kfp-namespace
rules:
- apiGroups: [""]
  resources: [pods]
  verbs: [get, patch]
- apiGroups: [argoproj.io]
  resources: [workflowtaskresults]
  verbs: [create, patch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: workflow-executor
  namespace: kfp-namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: workflow-executor
subjects:
- kind: ServiceAccount
  name: argo-workflow-sa
  namespace: kfp-namespace

Cluster-scoped resources (once per cluster)

These are shared across all provider namespaces and are the same resources described in RBAC for providers and Kubeflow completion eventing above:

• The providers-viewer ClusterRoleBinding for the Event Source Server and Controller service accounts.
• A run-viewer and a runconfiguration-viewer ClusterRoleBinding per provider service account.
• For the Kubeflow Pipelines provider only, the eventing ClusterRole and ClusterRoleBinding.

The providers-viewer-role, run-viewer-role and runconfiguration-viewer-role ClusterRoles are installed by the operator chart; when deploying the operator without Helm you must create these too (see helm/kfp-operator/templates/rbac).