Provider

The Provider resource represents the provider specific configuration required to submit / update / delete ml resources with the given provider. e.g Kubeflow Pipelines or the Vertex AI Platform. Providers configuration can be set using this resource and permissions for access can be configured via service accounts.

Note: changing the provider of a resource that was previously managed by another provider will result in a resource error. Any referenced resources must always match the provider of the referencing resource (e.g. RunConfiguration to Pipeline) as updates are not propagated or checked and will result in runtime errors on the provider.

Common Fields

NameDescriptionExample
spec.serviceImageContainer image of the provider servicekfp-operator-kfp-provider-service:0.0.2
spec.executionModeKFP compiler execution modev1 (currently KFP) or v2 (Vertex AI)
spec.serviceAccountService Account name to be used for all provider-specific operations (see respective provider)kfp-operator-vertex-ai
spec.pipelineRootStorageThe storage location used by TFX (pipeline-root) to store pipeline artifacts and outputs - this should be a top-level directory and not specific to a single pipelinegcs://kubeflow-pipelines-bucket
spec.parametersParameters specific to each provider, i.e. KFP and VAIgcs://kubeflow-pipelines-bucket
spec.frameworksFrameworks supported by the provider. Currently only tfx is supported.
spec.frameworks[0].nameName of the framework.tfx
spec.frameworks[0].imageFramework image.ghcr.io/kfp-operator/kfp-operator-tfx-compiler:version-tag
spec.frameworks[0].patchesList of JSON patches that will be applied to every Pipeline resource that uses this Provider before it’s passed (as JSON) to the corresponding Argo Workflow for processing.
spec.frameworks[0].patches[0].typeThe type of patch to be applied to the pipeline resource definition JSON. Can be either json (RFC6902) or merge (RFC7396).json
spec.frameworks[0].patches[0].payloadThe patch to be applied to the pipeline resource definition JSON.[{ "op": "add", "path": "/framework/parameters/beamArgs/0", "value": { "name": "newArg", "value": "newValue" } }]

Kubeflow:

apiVersion: pipelines.kubeflow.org/v1beta1
kind: Provider
metadata:
  name: kfp
  namespace: kfp-operator
spec:
  serviceImage: kfp-operator-kfp-provider-service:<version>
  executionMode: v1
  pipelineRootStorage: gs://<storage_location>
  serviceAccount: kfp-operator-kfp
  parameters:
    grpcKfpApiAddress: ml-pipeline.kubeflow:8887
    grpcMetadataStoreAddress: metadata-grpc-service.kubeflow:8080
    kfpNamespace: kubeflow
    restKfpApiUrl: http://ml-pipeline.kubeflow:8888
  frameworks:
  - name: tfx
    image: ghcr.io/kfp-operator/kfp-operator-tfx-compiler:version-tag
    patches:
    - type: json
      patch: |
        [
          {
            "op": "add",
            "path": "/framework/parameters/beamArgs/0",
            "value": {
              "name": "project",
              "Value": "<project>"
            }
          }
        ]        

Kubeflow Specific Parameters

NameDescription
parameters.grpcKfpApiAddressThe exposed grpc endpoint used to interact with Kubeflow pipelines
parameters.grpcMetadataStoreAddressThe exposed grpc endpoint used for metadata store with Kubeflow pipelines
parameters.kfpNamespaceThe namespace where Kubeflow is deployed
parameters.restKfpApiUrlThe exposed restful endpoint used to interact with Kubeflow pipelines

Vertex AI:

apiVersion: pipelines.kubeflow.org/v1beta1
kind: Provider
metadata:
  name: vai
  namespace: kfp-operator
spec:
  serviceImage: kfp-operator-vai-provider-service:<version>
  executionMode: v2
  pipelineRootStorage: gs://<storage_location>
  serviceAccount: kfp-operator-vai
  parameters:
    eventsourcePipelineEventsSubscription: kfp-operator-vai-run-events-eventsource
    maxConcurrentRunCount: 1
    pipelineBucket: pipeline-storage-bucket
    vaiJobServiceAccount: kfp-operator-vai@<project>.iam.gserviceaccount.com
    vaiLocation: europe-west2
    vaiProject: <project>
  frameworks:
  - name: tfx
    image: ghcr.io/kfp-operator/kfp-operator-tfx-compiler:version-tag
    patches:
    - type: json
      patch: |
        [
          {
            "op": "add",
            "path": "/framework/parameters/beamArgs/0",
            "value": {
              "name": "project",
              "Value": "<project>"
            }
          }
        ]        

Vertex AI Specific Parameters

NameDescription
parameters.eventsourcePipelineEventsSubscriptionThe eventsource subscription used to capture run-completion events
parameters.maxConcurrentRunCountThe number of pipelines that may run concurrently
parameters.pipelineBucketThe output storage bucket for a trained pipeline model
parameters.vaiJobServiceAccountThe service account should be used by VAI when submitting a pipeline
parameters.vaiLocationThe region VAI should run a pipeline within
parameters.vaiProjectThe project VAI should run a pipeline within