genai-toolbox/docs/en/how-to/deploy_gke.md

title, type, weight, description

title	type	weight	description
Deploy to Kubernetes	docs	4	How to set up and configure Toolbox to deploy on Kubernetes with Google Kubernetes Engine (GKE).

Before you begin

1. Set the PROJECT_ID environment variable:

   export PROJECT_ID="my-project-id"
   

2. Install the gcloud CLI.

3. Initialize gcloud CLI:

   gcloud init
   gcloud config set project $PROJECT_ID
   

4. You must have the following APIs enabled:

   gcloud services enable artifactregistry.googleapis.com \
                          cloudbuild.googleapis.com \
                          container.googleapis.com \
                          iam.googleapis.com
   

5. kubectl is used to manage Kubernetes, the cluster orchestration system used by GKE. Verify if you have kubectl installed:

   kubectl version --client
   

6. If needed, install kubectl component using the Google Cloud CLI:

   gcloud components install kubectl
   

Create a service account

1. Specify a name for your service account with an environment variable:

   export SA_NAME=toolbox
   

2. Create a backend service account:

   gcloud iam service-accounts create $SA_NAME
   

3. Grant any IAM roles necessary to the IAM service account. Each source has a list of necessary IAM permissions listed on its page. The example below is for cloud sql postgres source:

   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com \
       --role roles/cloudsql.client
   

   • AlloyDB IAM Identity
   • CloudSQL IAM Identity
   • Spanner IAM Identity

Deploy to Kubernetes

1. Set environment variables:

   export CLUSTER_NAME=toolbox-cluster
   export DEPLOYMENT_NAME=toolbox
   export SERVICE_NAME=toolbox-service
   export REGION=us-central1
   export NAMESPACE=toolbox-namespace
   export SECRET_NAME=toolbox-config
   export KSA_NAME=toolbox-service-account
   

2. Create a GKE cluster.

   gcloud container clusters create-auto $CLUSTER_NAME \
       --location=us-central1
   

3. Get authentication credentials to interact with the cluster. This also configures kubectl to use the cluster.

   gcloud container clusters get-credentials $CLUSTER_NAME \
       --region=$REGION \
       --project=$PROJECT_ID
   

4. View the current context for kubectl.

   kubectl config current-context
   

5. Create namespace for the deployment.

   kubectl create namespace $NAMESPACE
   

6. Create a Kubernetes Service Account (KSA).

   kubectl create serviceaccount $KSA_NAME --namespace $NAMESPACE
   

7. Enable the IAM binding between Google Service Account (GSA) and Kubernetes Service Account (KSA).

   gcloud iam service-accounts add-iam-policy-binding \
       --role="roles/iam.workloadIdentityUser" \
       --member="serviceAccount:$PROJECT_ID.svc.id.goog[$NAMESPACE/$KSA_NAME]" \
       $SA_NAME@$PROJECT_ID.iam.gserviceaccount.com
   

8. Add annotation to KSA to complete binding:

   kubectl annotate serviceaccount \
       $KSA_NAME \
       iam.gke.io/gcp-service-account=$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com \
       --namespace $NAMESPACE
   

9. Prepare the Kubernetes secret for your tools.yaml file.

   kubectl create secret generic $SECRET_NAME \
       --from-file=./tools.yaml \
       --namespace=$NAMESPACE
   

10. Create a Kubernetes manifest file (k8s_deployment.yaml) to build deployment.

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: toolbox
      namespace: toolbox-namespace
    spec:
      selector:
        matchLabels:
          app: toolbox
      template:
        metadata:
          labels:
            app: toolbox
        spec:
          serviceAccountName: toolbox-service-account
          containers:
            - name: toolbox
              # Recommend to use the latest version of toolbox
              image: us-central1-docker.pkg.dev/database-toolbox/toolbox/toolbox:latest
              args: ["--address", "0.0.0.0"]
              ports:
                - containerPort: 5000
              volumeMounts:
                - name: toolbox-config
                  mountPath: "/app/tools.yaml"
                  subPath: tools.yaml
                  readOnly: true
          volumes:
            - name: toolbox-config
              secret:
                secretName: toolbox-config
                items:
                - key: tools.yaml
                  path: tools.yaml
    

    {{< notice tip >}} To prevent DNS rebinding attack, use the --allowed-origins flag to specify a list of origins permitted to access the server. E.g. args: ["--address", "0.0.0.0", "--allowed-hosts", "foo.bar:5000"]

To implement CORs, use the --allowed-origins flag to specify a list of origins permitted to access the server. E.g. args: ["--address", "0.0.0.0", "--allowed-origins", "https://foo.bar"] {{< /notice >}}

1. Create the deployment.

   kubectl apply -f k8s_deployment.yaml --namespace $NAMESPACE
   

2. Check the status of deployment.

   kubectl get deployments --namespace $NAMESPACE
   

3. Create a Kubernetes manifest file (k8s_service.yaml) to build service.

   apiVersion: v1
   kind: Service
   metadata:
     name: toolbox-service
     namespace: toolbox-namespace
     annotations:
       cloud.google.com/l4-rbs: "enabled"
   spec:
     selector:
       app: toolbox
     ports:
       - port: 5000
         targetPort: 5000
     type: LoadBalancer
   

4. Create the service.

   kubectl apply -f k8s_service.yaml --namespace $NAMESPACE
   

5. You can find your IP address created for your service by getting the service information through the following.

   kubectl describe services $SERVICE_NAME --namespace $NAMESPACE
   

6. To look at logs, run the following.

   kubectl logs -f deploy/$DEPLOYMENT_NAME --namespace $NAMESPACE
   

7. You might have to wait a couple of minutes. It is ready when you can see EXTERNAL-IP with the following command:

   kubectl get svc -n $NAMESPACE
   

8. Access toolbox locally.

   curl <EXTERNAL-IP>:5000
   

Clean up resources

1. Delete secret.

   kubectl delete secret $SECRET_NAME --namespace $NAMESPACE
   

2. Delete deployment.

   kubectl delete deployment $DEPLOYMENT_NAME --namespace $NAMESPACE
   

3. Delete the application's service.

   kubectl delete service $SERVICE_NAME --namespace $NAMESPACE
   

4. Delete the Kubernetes cluster.

   gcloud container clusters delete $CLUSTER_NAME \
       --location=$REGION