genai-toolbox/docs/en/resources/sources/serverless-spark.md at 4d4f169e42576bf7ba69f7ca5f8f63881f851369

mirror of https://github.com/googleapis/genai-toolbox.git synced 2026-01-08 15:14:00 -05:00

Files

Dave Borowitz 17a979207d feat(serverless-spark): add create_spark_batch tool

This tool is almost identical to create_pyspark_batch, but for Java
Spark batches. There are some minor differences in how the main files
and args are provided.

2025-12-04 11:05:53 -08:00

2.7 KiB

Raw Blame History

title, type, weight, description

title	type	weight	description
Serverless for Apache Spark	docs	1	Google Cloud Serverless for Apache Spark lets you run Spark workloads without requiring you to provision and manage your own Spark cluster.

About

The Serverless for Apache Spark source allows Toolbox to interact with Spark batches hosted on Google Cloud Serverless for Apache Spark.

Available Tools

serverless-spark-list-batches List and filter Serverless Spark batches.
serverless-spark-get-batch Get a Serverless Spark batch.
serverless-spark-cancel-batch Cancel a running Serverless Spark batch operation.
serverless-spark-create-pyspark-batch Create a Serverless Spark PySpark batch operation.
serverless-spark-create-spark-batch Create a Serverless Spark Java batch operation.

Requirements

IAM Permissions

Serverless for Apache Spark uses Identity and Access Management (IAM) to control user and group access to serverless Spark resources like batches and sessions.

Toolbox will use your Application Default Credentials (ADC) to authorize and authenticate when interacting with Google Cloud Serverless for Apache Spark. When using this method, you need to ensure the IAM identity associated with your ADC has the correct permissions for the actions you intend to perform. Common roles include roles/dataproc.serverlessEditor (which includes permissions to run batches) or roles/dataproc.serverlessViewer. Follow this guide to set up your ADC.

Example

sources:
  my-serverless-spark-source:
    kind: serverless-spark
    project: my-project-id
    location: us-central1

Reference

field	type	required	description
kind	string	true	Must be "serverless-spark".
project	string	true	ID of the GCP project with Serverless for Apache Spark resources.
location	string	true	Location containing Serverless for Apache Spark resources.

2.7 KiB Raw Blame History