Kubernetes

Note: Up-to-date Helm Chart Documentation

This page provides an overview of deploying Adaptive Engine on Kubernetes. For the most up-to-date and detailed instructions on using the Adaptive Helm chart, please refer to the official documentation in the adaptive-helm-chart repository.

Adaptive Engine can be easily configured and deployed on Kubernetes using our Helm chart. This guide will walk you through an overview of the deployment process and key configuration areas.

Requirements

Before deploying, please ensure your environment meets the following prerequisites as described in the helm chart’s README.

Getting Started

To get started, we recommend you follow the installation instructions in the helm chart’s README. This will guide you on how to:

Install the chart from the GitHub OCI Registry.
Get the default values.yaml configuration file.
Customize values.yaml for your environment.
Deploy the chart to your Kubernetes cluster.

Configuration Highlights

The values.yaml file contains all the configuration options for the Adaptive Engine Helm chart. Below are some of the key sections you’ll need to configure.

Container Registry Information

You will need to provide the details for the container registry where the Adaptive Engine images are stored.

containerRegistry: <aws_account_id>.dkr.ecr.<region>.amazonaws.com
harmony:
  image:
    repository: adaptive-repository # Adaptive Repository you have been granted access to
    tag: harmony:latest # Harmony image tag

controlPlane:
  image:
    repository: adaptive-repository # Adaptive Repository you have been granted access to
    tag: control-plane:latest # Control plane image tag

Resource Limits

Adjust the resource limits based on your cluster’s capabilities and workload/model requirements. harmony.gpusPerNode should match the available GPU resources for each node in the cluster where Adaptive Harmony will be deployed. For example:

harmony:
    replicaCount: 1
    # Should be equal to, or a divisor of the # of GPUs on each node
    gpusPerReplica: 8
    resources:
        limits:
            cpu: 8
            memory: 64Gi
        requests:
            cpu: 8
            memory: 60Gi

Configuration Secrets

The chart requires several secrets for configuration, such as S3 bucket URLs, database connection strings, and authentication provider details. You can set these directly in values.yaml or use an external secrets manager.

secrets:
  # S3 bucket for model registry
  modelRegistryUrl: "s3://bucket-name/model_registry"
  # Use same bucket as above and can use a different prefix
  sharedDirectoryUrl: "s3://bucket-name/shared"

  # Postgres database connection string
  dbUrl: "postgres://username:password@db_address:5432/db_name"
  # Secret used to sign cookies. Must be the same on all servers of a cluster and >= 64 chars
  cookiesSecret: "change-me-secret-db40431e-c2fd-48a6-acd6-854232c2ed94-01dd4d01-dr7b-4315" # Must be >= 64 chars

  auth:
    oidc:
      providers:
        # Name of your OpenId provider displayed in the ui
        - name: "Google"
          # Key of your provider, the callback url will be '<rootUrl>/api/v1/auth/login/<key>/callback'
          key: "google"
          issuer_url: "https://accounts.google.com" # openid connect issuer url
          client_id: "replace_client_id" # client id
          client_secret: "replace_client_secret" # client_secret, optional
          scopes: ["email", "profile"] # scopes required for auth, requires email and profile
          # true if your provider supports pkce (recommended)
          pkce: true
          # if true, user account will be created if it does not exist
          allow_sign_up: true

For information on using an external secrets manager, refer to the “Using External Secret Management” section in the helm chart’s README.

Considerations for deployment on shared clusters

When deploying Adaptive Engine in a shared cluster where other workloads are running, there are a few best practices you can implement to enforce resource isolation:

Deploy Adaptive in a separate namespace

When installing the Adaptive Helm chart, you can do so in a separate namespace by passing the --namespace option. Example:

helm install adaptive \
  adaptive/adaptive \
  --values ./values.yaml
  --namespace adaptive-engine

You can also pass the --create-namespace if the namespace does not exist yet.

Use Node Selectors to schedule Adaptive on specific GPU nodes

You can use the harmony.nodeSelector value in values.yaml to schedule Adaptive Harmony only on a specific node group. For example, if you are deploying Adaptive on an Amazon EKS cluster, you might add:

harmony:
  nodeSelector: 
    eks.amazonaws.com/nodegroup: p5-h100

Dedicated GPU node tenancy

Although the Adaptive control plane can run on any node where there are available CPU and memory resources, it is recommended that Harmony is scheduled to request and take ownership of all of the GPUs available on each GPU-enabled node. Although you might have already made sure Adaptive Harmony is only scheduled on a designated GPU node group using the instructions in the step above, you might want to guarantee no other workloads can be scheduled on those nodes. To dedicate a set of GPU nodes for Adaptive Harmony, you can use a combination of:

Adding a taint to the GPU nodes
Adding a corresponding toleration to Harmony in the values.yaml of the Adaptive Helm Chart

To add a taint to a node, you can first run kubectl get nodes -o name to see all the existing node names, and then taint them as exemplified below (replacing node_name):

kubectl taint nodes node_name dedicated=adaptive-engine:NoSchedule

You can then add a matching toleration to Harmony in the values.yaml file (harmony.tolerations) which will allow it to be scheduled on the tainted nodes:

harmony:
  tolerations:
  - key: dedicated
    operator: Equal
    value: adaptive-engine
    effect: NoSchedule

You can find more about taints and tolerations in the official Kubernetes documentation.

Advanced configuration

Database SSL/TLS configuration

Adaptive Engine supports secure TLS connections between the database and control plane.

Basic setting

If your PostgreSQL database supports TLS, you can enforce encrypted connections by adding the parameter sslmode=require to your PostgreSQL connection string dbUrl in the Helm chart’s values.yaml file:

  dbUrl: "postgres://<user>:<password>@<host>/<db>?sslmode=require"

Although sslmode=require encrypts the database connection, it does not verify the server’s identity.

Server certificate verification

In order for the application to be able to verify the server certificate, you must set sslmode to verify-ca or -verify-full.

verify-ca will verify the server certificate
verify-full will verify the server certificate and also that the server host name matches the name stored in the server certificate

verify-full is the recommended option for maximum security. You will need to provide the application with a root certificate to make server certification possible. You can do so by following these steps:

Download the db server certificate (if you’re using AWS RDS for example, refer to this page), for instance rds-ca-rsa2048-g1.pem
Upload the pem file to your k8s cluster. As the certificate is non-critical, public information, it can uploaded as a ConfigMap

kubectl create configmap -n <namespace> db-ca --from-file=rds-ca-rsa2048-g1.pem

Mount the file as a volume to the control plane deployment by editing values.yaml:

...

volumes:
  - name: db-ca
    configMap:
      name: db-ca

volumeMounts:
  - name: db-ca
    mountPath: /mnt/db-ca/
    readOnly: true

Use the sslrootcert parameter to refer to the certificate in the PostgresDB connection url, specifying mountPath + filename:

  dbUrl: "postgres://<user>:<password>@<host>/<db>?sslmode=verify-full&sslrootcert=/mnt/db-ca/rds-ca-rsa2048-g1.pem"

Refer to the official documentation for SSL support on PostgresSQL for more information.

Platform

Inference

Evaluation

Graders

Recipes & Runs

Datasets

Interactions

Integrations

Deployment

Note: Up-to-date Helm Chart Documentation

Requirements

Getting Started

Configuration Highlights

Container Registry Information

Resource Limits

Configuration Secrets

Considerations for deployment on shared clusters

Deploy Adaptive in a separate namespace

Use Node Selectors to schedule Adaptive on specific GPU nodes

Dedicated GPU node tenancy

Advanced configuration

Database SSL/TLS configuration

Basic setting

Server certificate verification

Note: Up-to-date Helm Chart Documentation

​Requirements

​Getting Started

​Configuration Highlights

​Container Registry Information

​Resource Limits

​Configuration Secrets

​Considerations for deployment on shared clusters

​Deploy Adaptive in a separate namespace

​Use Node Selectors to schedule Adaptive on specific GPU nodes

​Dedicated GPU node tenancy

​Advanced configuration

​Database SSL/TLS configuration

​Basic setting

​Server certificate verification

Requirements

Getting Started

Configuration Highlights

Container Registry Information

Resource Limits

Configuration Secrets

Considerations for deployment on shared clusters

Deploy Adaptive in a separate namespace

Use Node Selectors to schedule Adaptive on specific GPU nodes

Dedicated GPU node tenancy

Advanced configuration

Database SSL/TLS configuration

Basic setting

Server certificate verification