Provider
The Provider resource represents the provider specific configuration required to submit / update / delete ml resources with the given provider.
e.g Kubeflow Pipelines or the Vertex AI Platform.
Providers configuration can be set using this resource and permissions for access can be configured via service accounts.
Note: changing the provider of a resource that was previously managed by another provider will result in a resource error.
Any referenced resources must always match the provider of the referencing resource (e.g. RunConfiguration to Pipeline) as updates are not propagated or checked and will result in runtime errors on the provider.
Common Fields
Name | Description | Example |
---|
spec.serviceImage | Container image of the provider service | kfp-operator-kfp-provider-service:0.0.2 |
spec.executionMode | KFP compiler execution mode | v1 (currently KFP) or v2 (Vertex AI) |
spec.serviceAccount | Service Account name to be used for all provider-specific operations (see respective provider) | kfp-operator-vertex-ai |
spec.pipelineRootStorage | The storage location used by TFX (pipeline-root ) to store pipeline artifacts and outputs - this should be a top-level directory and not specific to a single pipeline | gcs://kubeflow-pipelines-bucket |
spec.parameters | Parameters specific to each provider, i.e. KFP and VAI | gcs://kubeflow-pipelines-bucket |
spec.frameworks | Frameworks supported by the provider. Currently only tfx is supported. | |
spec.frameworks[0].name | Name of the framework. | tfx |
spec.frameworks[0].image | Framework image. | ghcr.io/kfp-operator/kfp-operator-tfx-compiler:version-tag |
spec.frameworks[0].patches | List of JSON patches that will be applied to every Pipeline resource that uses this Provider before it’s passed (as JSON) to the corresponding Argo Workflow for processing. | |
spec.frameworks[0].patches[0].type | The type of patch to be applied to the pipeline resource definition JSON. Can be either json (RFC6902) or merge (RFC7396). | json |
spec.frameworks[0].patches[0].payload | The patch to be applied to the pipeline resource definition JSON. | [{ "op": "add", "path": "/framework/parameters/beamArgs/0", "value": { "name": "newArg", "value": "newValue" } }] |
Kubeflow:
apiVersion: pipelines.kubeflow.org/v1beta1
kind: Provider
metadata:
name: kfp
namespace: kfp-operator
spec:
serviceImage: kfp-operator-kfp-provider-service:<version>
executionMode: v1
pipelineRootStorage: gs://<storage_location>
serviceAccount: kfp-operator-kfp
parameters:
grpcKfpApiAddress: ml-pipeline.kubeflow:8887
grpcMetadataStoreAddress: metadata-grpc-service.kubeflow:8080
kfpNamespace: kubeflow
restKfpApiUrl: http://ml-pipeline.kubeflow:8888
frameworks:
- name: tfx
image: ghcr.io/kfp-operator/kfp-operator-tfx-compiler:version-tag
patches:
- type: json
patch: |
[
{
"op": "add",
"path": "/framework/parameters/beamArgs/0",
"value": {
"name": "project",
"Value": "<project>"
}
}
]
Kubeflow Specific Parameters
Name | Description |
---|
parameters.grpcKfpApiAddress | The exposed grpc endpoint used to interact with Kubeflow pipelines |
parameters.grpcMetadataStoreAddress | The exposed grpc endpoint used for metadata store with Kubeflow pipelines |
parameters.kfpNamespace | The namespace where Kubeflow is deployed |
parameters.restKfpApiUrl | The exposed restful endpoint used to interact with Kubeflow pipelines |
Vertex AI:
apiVersion: pipelines.kubeflow.org/v1beta1
kind: Provider
metadata:
name: vai
namespace: kfp-operator
spec:
serviceImage: kfp-operator-vai-provider-service:<version>
executionMode: v2
pipelineRootStorage: gs://<storage_location>
serviceAccount: kfp-operator-vai
parameters:
eventsourcePipelineEventsSubscription: kfp-operator-vai-run-events-eventsource
maxConcurrentRunCount: 1
pipelineBucket: pipeline-storage-bucket
vaiJobServiceAccount: kfp-operator-vai@<project>.iam.gserviceaccount.com
vaiLocation: europe-west2
vaiProject: <project>
frameworks:
- name: tfx
image: ghcr.io/kfp-operator/kfp-operator-tfx-compiler:version-tag
patches:
- type: json
patch: |
[
{
"op": "add",
"path": "/framework/parameters/beamArgs/0",
"value": {
"name": "project",
"Value": "<project>"
}
}
]
Vertex AI Specific Parameters
Name | Description |
---|
parameters.eventsourcePipelineEventsSubscription | The eventsource subscription used to capture run-completion events |
parameters.maxConcurrentRunCount | The number of pipelines that may run concurrently |
parameters.pipelineBucket | The output storage bucket for a trained pipeline model |
parameters.vaiJobServiceAccount | The service account should be used by VAI when submitting a pipeline |
parameters.vaiLocation | The region VAI should run a pipeline within |
parameters.vaiProject | The project VAI should run a pipeline within |