mirror of
https://github.com/googleapis/genai-toolbox.git
synced 2026-02-14 00:54:56 -05:00
Compare commits
3 Commits
v0.27.0
...
feat/add-g
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9a2092ffd6 | ||
|
|
5fcf3df882 | ||
|
|
195767bdcd |
3
.gitignore
vendored
3
.gitignore
vendored
@@ -18,6 +18,9 @@ node_modules
|
||||
# coverage
|
||||
.coverage
|
||||
|
||||
# python
|
||||
__pycache__/
|
||||
|
||||
# executable
|
||||
genai-toolbox
|
||||
toolbox
|
||||
108
GEMINI.md
Normal file
108
GEMINI.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# MCP Toolbox Context
|
||||
|
||||
This file (symlinked as `CLAUDE.md` and `AGENTS.md`) provides context and guidelines for AI agents working on the MCP Toolbox for Databases project. It summarizes key information from `CONTRIBUTING.md` and `DEVELOPER.md`.
|
||||
|
||||
## Project Overview
|
||||
|
||||
**MCP Toolbox for Databases** is a Go-based project designed to provide Model Context Protocol (MCP) tools for various data sources and services. It allows Large Language Models (LLMs) to interact with databases and other tools safely and efficiently.
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Language:** Go (1.23+)
|
||||
- **Documentation:** Hugo (Extended Edition v0.146.0+)
|
||||
- **Containerization:** Docker
|
||||
- **CI/CD:** GitHub Actions, Google Cloud Build
|
||||
- **Linting:** `golangci-lint`
|
||||
|
||||
## Key Directories
|
||||
|
||||
- `cmd/`: Application entry points.
|
||||
- `internal/sources/`: Implementations of database sources (e.g., Postgres, BigQuery).
|
||||
- `internal/tools/`: Implementations of specific tools for each source.
|
||||
- `tests/`: Integration tests.
|
||||
- `docs/`: Project documentation (Hugo site).
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Go 1.23 or later.
|
||||
- Docker (for building container images and running some tests).
|
||||
- Access to necessary Google Cloud resources for integration testing (if applicable).
|
||||
|
||||
### Building and Running
|
||||
|
||||
1. **Build Binary:** `go build -o toolbox`
|
||||
2. **Run Server:** `go run .` (Listens on port 5000 by default)
|
||||
3. **Run with Help:** `go run . --help`
|
||||
4. **Test Endpoint:** `curl http://127.0.0.1:5000`
|
||||
|
||||
### Testing
|
||||
|
||||
- **Unit Tests:** `go test -race -v ./cmd/... ./internal/...`
|
||||
- **Integration Tests:**
|
||||
- Run specific source tests: `go test -race -v ./tests/<source_dir>`
|
||||
- Example: `go test -race -v ./tests/alloydbpg`
|
||||
- Add new sources to `.ci/integration.cloudbuild.yaml`
|
||||
- **Linting:** `golangci-lint run --fix`
|
||||
|
||||
## Developing Documentation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Hugo (Extended Edition v0.146.0+)
|
||||
- Node.js (for `npm ci`)
|
||||
|
||||
### Running Local Server
|
||||
|
||||
1. Navigate to `.hugo` directory: `cd .hugo`
|
||||
2. Install dependencies: `npm ci`
|
||||
3. Start server: `hugo server`
|
||||
|
||||
### Versioning Workflows
|
||||
|
||||
1. **Deploy In-development docs**: Merges to main -> `/dev/`.
|
||||
2. **Deploy Versioned Docs**: New Release -> `/<version>/` and root.
|
||||
3. **Deploy Previous Version Docs**: Manual workflow for older versions.
|
||||
|
||||
## Coding Conventions
|
||||
|
||||
### Tool Naming
|
||||
|
||||
- **Tool Name:** `snake_case` (e.g., `list_collections`, `run_query`).
|
||||
- Do *not* include the product name (e.g., avoid `firestore_list_collections`).
|
||||
- **Tool Type:** `kebab-case` (e.g., `firestore-list-collections`).
|
||||
- *Must* include the product name.
|
||||
|
||||
### Branching and Commits
|
||||
|
||||
- **Branch Naming:** `feat/`, `fix/`, `docs/`, `chore/` (e.g., `feat/add-gemini-md`).
|
||||
- **Commit Messages:** [Conventional Commits](https://www.conventionalcommits.org/) format.
|
||||
- Format: `<type>(<scope>): <description>`
|
||||
- Example: `feat(source/postgres): add new connection option`
|
||||
- Types: `feat`, `fix`, `docs`, `chore`, `test`, `ci`, `refactor`, `revert`, `style`.
|
||||
|
||||
## Adding New Features
|
||||
|
||||
### Adding a New Data Source
|
||||
|
||||
1. Create a new directory: `internal/sources/<newdb>`.
|
||||
2. Define `Config` and `Source` structs in `internal/sources/<newdb>/<newdb>.go`.
|
||||
3. Implement `SourceConfig` interface (`SourceConfigType`, `Initialize`).
|
||||
4. Implement `Source` interface (`SourceType`).
|
||||
5. Implement `init()` to register the source.
|
||||
6. Add unit tests in `internal/sources/<newdb>/<newdb>_test.go`.
|
||||
|
||||
### Adding a New Tool
|
||||
|
||||
1. Create a new directory: `internal/tools/<newdb>/<toolname>`.
|
||||
2. Define `Config` and `Tool` structs.
|
||||
3. Implement `ToolConfig` interface (`ToolConfigType`, `Initialize`).
|
||||
4. Implement `Tool` interface (`Invoke`, `ParseParams`, `Manifest`, `McpManifest`, `Authorized`).
|
||||
5. Implement `init()` to register the tool.
|
||||
6. Add unit tests.
|
||||
|
||||
### Adding Documentation
|
||||
|
||||
- Add source documentation to `docs/en/resources/sources/`.
|
||||
- Add tool documentation to `docs/en/resources/tools/`.
|
||||
257
cmd/internal/imports.go
Normal file
257
cmd/internal/imports.go
Normal file
@@ -0,0 +1,257 @@
|
||||
// Copyright 2026 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package internal
|
||||
|
||||
import (
|
||||
// Import prompt packages for side effect of registration
|
||||
_ "github.com/googleapis/genai-toolbox/internal/prompts/custom"
|
||||
|
||||
// Import tool packages for side effect of registration
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/alloydb/alloydbcreatecluster"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/alloydb/alloydbcreateinstance"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/alloydb/alloydbcreateuser"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/alloydb/alloydbgetcluster"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/alloydb/alloydbgetinstance"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/alloydb/alloydbgetuser"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/alloydb/alloydblistclusters"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/alloydb/alloydblistinstances"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/alloydb/alloydblistusers"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/alloydb/alloydbwaitforoperation"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/alloydbainl"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/bigquery/bigqueryanalyzecontribution"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/bigquery/bigqueryconversationalanalytics"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/bigquery/bigqueryexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/bigquery/bigqueryforecast"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/bigquery/bigquerygetdatasetinfo"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/bigquery/bigquerygettableinfo"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/bigquery/bigquerylistdatasetids"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/bigquery/bigquerylisttableids"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/bigquery/bigquerysearchcatalog"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/bigquery/bigquerysql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/bigtable"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cassandra/cassandracql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/clickhouse/clickhouseexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/clickhouse/clickhouselistdatabases"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/clickhouse/clickhouselisttables"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/clickhouse/clickhousesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudgda"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcarefhirfetchpage"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcarefhirpatienteverything"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcarefhirpatientsearch"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcaregetdataset"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcaregetdicomstore"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcaregetdicomstoremetrics"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcaregetfhirresource"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcaregetfhirstore"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcaregetfhirstoremetrics"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcarelistdicomstores"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcarelistfhirstores"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcareretrieverendereddicominstance"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcaresearchdicominstances"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcaresearchdicomseries"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudhealthcare/cloudhealthcaresearchdicomstudies"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudloggingadmin/cloudloggingadminlistlognames"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudloggingadmin/cloudloggingadminlistresourcetypes"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudloggingadmin/cloudloggingadminquerylogs"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudmonitoring"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsql/cloudsqlcloneinstance"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsql/cloudsqlcreatebackup"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsql/cloudsqlcreatedatabase"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsql/cloudsqlcreateusers"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsql/cloudsqlgetinstances"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsql/cloudsqllistdatabases"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsql/cloudsqllistinstances"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsql/cloudsqlrestorebackup"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsql/cloudsqlwaitforoperation"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsqlmssql/cloudsqlmssqlcreateinstance"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsqlmysql/cloudsqlmysqlcreateinstance"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsqlpg/cloudsqlpgcreateinstances"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cloudsqlpg/cloudsqlpgupgradeprecheck"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cockroachdb/cockroachdbexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cockroachdb/cockroachdblistschemas"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cockroachdb/cockroachdblisttables"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/cockroachdb/cockroachdbsql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/couchbase"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/dataform/dataformcompilelocal"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/dataplex/dataplexlookupentry"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/dataplex/dataplexsearchaspecttypes"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/dataplex/dataplexsearchentries"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/dgraph"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/elasticsearch/elasticsearchesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/firebird/firebirdexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/firebird/firebirdsql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/firestore/firestoreadddocuments"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/firestore/firestoredeletedocuments"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/firestore/firestoregetdocuments"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/firestore/firestoregetrules"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/firestore/firestorelistcollections"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/firestore/firestorequery"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/firestore/firestorequerycollection"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/firestore/firestoreupdatedocument"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/firestore/firestorevalidaterules"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/http"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookeradddashboardelement"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookeradddashboardfilter"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookerconversationalanalytics"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookercreateprojectfile"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookerdeleteprojectfile"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookerdevmode"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergenerateembedurl"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetconnectiondatabases"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetconnections"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetconnectionschemas"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetconnectiontablecolumns"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetconnectiontables"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetdashboards"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetdimensions"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetexplores"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetfilters"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetlooks"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetmeasures"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetmodels"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetparameters"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetprojectfile"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetprojectfiles"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookergetprojects"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookerhealthanalyze"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookerhealthpulse"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookerhealthvacuum"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookermakedashboard"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookermakelook"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookerquery"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookerquerysql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookerqueryurl"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookerrundashboard"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookerrunlook"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookerupdateprojectfile"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/looker/lookervalidateproject"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mindsdb/mindsdbexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mindsdb/mindsdbsql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mongodb/mongodbaggregate"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mongodb/mongodbdeletemany"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mongodb/mongodbdeleteone"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mongodb/mongodbfind"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mongodb/mongodbfindone"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mongodb/mongodbinsertmany"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mongodb/mongodbinsertone"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mongodb/mongodbupdatemany"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mongodb/mongodbupdateone"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mssql/mssqlexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mssql/mssqllisttables"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mssql/mssqlsql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mysql/mysqlexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mysql/mysqlgetqueryplan"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mysql/mysqllistactivequeries"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mysql/mysqllisttablefragmentation"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mysql/mysqllisttables"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mysql/mysqllisttablesmissinguniqueindexes"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/mysql/mysqlsql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/neo4j/neo4jcypher"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/neo4j/neo4jexecutecypher"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/neo4j/neo4jschema"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/oceanbase/oceanbaseexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/oceanbase/oceanbasesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/oracle/oracleexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/oracle/oraclesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgresdatabaseoverview"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgresexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgresgetcolumncardinality"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistactivequeries"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistavailableextensions"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistdatabasestats"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistindexes"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistinstalledextensions"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistlocks"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistpgsettings"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistpublicationtables"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistquerystats"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistroles"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistschemas"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistsequences"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgresliststoredprocedure"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslisttables"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslisttablespaces"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslisttablestats"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslisttriggers"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslistviews"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgreslongrunningtransactions"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgresreplicationstats"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/postgres/postgressql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/redis"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/serverlessspark/serverlesssparkcancelbatch"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/serverlessspark/serverlesssparkcreatepysparkbatch"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/serverlessspark/serverlesssparkcreatesparkbatch"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/serverlessspark/serverlesssparkgetbatch"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/serverlessspark/serverlesssparklistbatches"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/singlestore/singlestoreexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/singlestore/singlestoresql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/snowflake/snowflakeexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/snowflake/snowflakesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/spanner/spannerexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/spanner/spannerlistgraphs"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/spanner/spannerlisttables"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/spanner/spannersql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/sqlite/sqliteexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/sqlite/sqlitesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/tidb/tidbexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/tidb/tidbsql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/trino/trinoexecutesql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/trino/trinosql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/utility/wait"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/valkey"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/yugabytedbsql"
|
||||
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/alloydbadmin"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/alloydbpg"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/bigquery"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/bigtable"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/cassandra"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/clickhouse"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/cloudgda"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/cloudhealthcare"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/cloudloggingadmin"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/cloudmonitoring"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/cloudsqladmin"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/cloudsqlmssql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/cloudsqlmysql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/cloudsqlpg"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/cockroachdb"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/couchbase"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/dataplex"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/dgraph"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/elasticsearch"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/firebird"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/firestore"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/http"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/looker"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/mindsdb"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/mongodb"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/mssql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/mysql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/neo4j"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/oceanbase"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/oracle"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/postgres"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/redis"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/serverlessspark"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/singlestore"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/snowflake"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/spanner"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/sqlite"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/tidb"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/trino"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/valkey"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/yugabytedb"
|
||||
)
|
||||
@@ -18,37 +18,15 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
"github.com/googleapis/genai-toolbox/internal/log"
|
||||
"github.com/googleapis/genai-toolbox/cmd/internal"
|
||||
"github.com/googleapis/genai-toolbox/internal/server"
|
||||
"github.com/googleapis/genai-toolbox/internal/server/resources"
|
||||
"github.com/googleapis/genai-toolbox/internal/util/parameters"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
// RootCommand defines the interface for required by invoke subcommand.
|
||||
// This allows subcommands to access shared resources and functionality without
|
||||
// direct coupling to the root command's implementation.
|
||||
type RootCommand interface {
|
||||
// Config returns a copy of the current server configuration.
|
||||
Config() server.ServerConfig
|
||||
|
||||
// Out returns the writer used for standard output.
|
||||
Out() io.Writer
|
||||
|
||||
// LoadConfig loads and merges the configuration from files, folders, and prebuilts.
|
||||
LoadConfig(ctx context.Context) error
|
||||
|
||||
// Setup initializes the runtime environment, including logging and telemetry.
|
||||
// It returns the updated context and a shutdown function to be called when finished.
|
||||
Setup(ctx context.Context) (context.Context, func(context.Context) error, error)
|
||||
|
||||
// Logger returns the logger instance.
|
||||
Logger() log.Logger
|
||||
}
|
||||
|
||||
func NewCommand(rootCmd RootCommand) *cobra.Command {
|
||||
func NewCommand(opts *internal.ToolboxOptions) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "invoke <tool-name> [params]",
|
||||
Short: "Execute a tool directly",
|
||||
@@ -58,17 +36,17 @@ Example:
|
||||
toolbox invoke my-tool '{"param1": "value1"}'`,
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
RunE: func(c *cobra.Command, args []string) error {
|
||||
return runInvoke(c, args, rootCmd)
|
||||
return runInvoke(c, args, opts)
|
||||
},
|
||||
}
|
||||
return cmd
|
||||
}
|
||||
|
||||
func runInvoke(cmd *cobra.Command, args []string, rootCmd RootCommand) error {
|
||||
func runInvoke(cmd *cobra.Command, args []string, opts *internal.ToolboxOptions) error {
|
||||
ctx, cancel := context.WithCancel(cmd.Context())
|
||||
defer cancel()
|
||||
|
||||
ctx, shutdown, err := rootCmd.Setup(ctx)
|
||||
ctx, shutdown, err := opts.Setup(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -76,16 +54,16 @@ func runInvoke(cmd *cobra.Command, args []string, rootCmd RootCommand) error {
|
||||
_ = shutdown(ctx)
|
||||
}()
|
||||
|
||||
// Load and merge tool configurations
|
||||
if err := rootCmd.LoadConfig(ctx); err != nil {
|
||||
_, err = opts.LoadConfig(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Initialize Resources
|
||||
sourcesMap, authServicesMap, embeddingModelsMap, toolsMap, toolsetsMap, promptsMap, promptsetsMap, err := server.InitializeConfigs(ctx, rootCmd.Config())
|
||||
sourcesMap, authServicesMap, embeddingModelsMap, toolsMap, toolsetsMap, promptsMap, promptsetsMap, err := server.InitializeConfigs(ctx, opts.Cfg)
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("failed to initialize resources: %w", err)
|
||||
rootCmd.Logger().ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
@@ -96,7 +74,7 @@ func runInvoke(cmd *cobra.Command, args []string, rootCmd RootCommand) error {
|
||||
tool, ok := resourceMgr.GetTool(toolName)
|
||||
if !ok {
|
||||
errMsg := fmt.Errorf("tool %q not found", toolName)
|
||||
rootCmd.Logger().ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
@@ -109,7 +87,7 @@ func runInvoke(cmd *cobra.Command, args []string, rootCmd RootCommand) error {
|
||||
if paramsInput != "" {
|
||||
if err := json.Unmarshal([]byte(paramsInput), ¶ms); err != nil {
|
||||
errMsg := fmt.Errorf("params must be a valid JSON string: %w", err)
|
||||
rootCmd.Logger().ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
}
|
||||
@@ -117,14 +95,14 @@ func runInvoke(cmd *cobra.Command, args []string, rootCmd RootCommand) error {
|
||||
parsedParams, err := parameters.ParseParams(tool.GetParameters(), params, nil)
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("invalid parameters: %w", err)
|
||||
rootCmd.Logger().ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
parsedParams, err = tool.EmbedParams(ctx, parsedParams, resourceMgr.GetEmbeddingModelMap())
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("error embedding parameters: %w", err)
|
||||
rootCmd.Logger().ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
@@ -132,19 +110,19 @@ func runInvoke(cmd *cobra.Command, args []string, rootCmd RootCommand) error {
|
||||
requiresAuth, err := tool.RequiresClientAuthorization(resourceMgr)
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("failed to check auth requirements: %w", err)
|
||||
rootCmd.Logger().ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
if requiresAuth {
|
||||
errMsg := fmt.Errorf("client authorization is not supported")
|
||||
rootCmd.Logger().ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
result, err := tool.Invoke(ctx, resourceMgr, parsedParams, "")
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("tool execution failed: %w", err)
|
||||
rootCmd.Logger().ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
@@ -152,10 +130,10 @@ func runInvoke(cmd *cobra.Command, args []string, rootCmd RootCommand) error {
|
||||
output, err := json.MarshalIndent(result, "", " ")
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("failed to marshal result: %w", err)
|
||||
rootCmd.Logger().ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
fmt.Fprintln(rootCmd.Out(), string(output))
|
||||
fmt.Fprintln(opts.IOStreams.Out, string(output))
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -12,16 +12,38 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package cmd
|
||||
package invoke
|
||||
|
||||
import (
|
||||
"context"
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/googleapis/genai-toolbox/cmd/internal"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/bigquery"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/sqlite"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/bigquery/bigquerysql"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/sqlite/sqlitesql"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func invokeCommand(args []string) (string, error) {
|
||||
parentCmd := &cobra.Command{Use: "toolbox"}
|
||||
|
||||
buf := new(bytes.Buffer)
|
||||
opts := internal.NewToolboxOptions(internal.WithIOStreams(buf, buf))
|
||||
internal.PersistentFlags(parentCmd, opts)
|
||||
|
||||
cmd := NewCommand(opts)
|
||||
parentCmd.AddCommand(cmd)
|
||||
parentCmd.SetArgs(args)
|
||||
|
||||
err := parentCmd.Execute()
|
||||
return buf.String(), err
|
||||
}
|
||||
|
||||
func TestInvokeTool(t *testing.T) {
|
||||
// Create a temporary tools file
|
||||
tmpDir := t.TempDir()
|
||||
@@ -86,7 +108,7 @@ tools:
|
||||
|
||||
for _, tc := range tcs {
|
||||
t.Run(tc.desc, func(t *testing.T) {
|
||||
_, got, err := invokeCommandWithContext(context.Background(), tc.args)
|
||||
got, err := invokeCommand(tc.args)
|
||||
if (err != nil) != tc.wantErr {
|
||||
t.Fatalf("got error %v, wantErr %v", err, tc.wantErr)
|
||||
}
|
||||
@@ -121,7 +143,7 @@ tools:
|
||||
}
|
||||
|
||||
args := []string{"invoke", "bq-tool", "--tools-file", toolsFilePath}
|
||||
_, _, err := invokeCommandWithContext(context.Background(), args)
|
||||
_, err := invokeCommand(args)
|
||||
if err == nil {
|
||||
t.Fatal("expected error for tool requiring client auth, but got nil")
|
||||
}
|
||||
251
cmd/internal/options.go
Normal file
251
cmd/internal/options.go
Normal file
@@ -0,0 +1,251 @@
|
||||
// Copyright 2026 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package internal
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"github.com/googleapis/genai-toolbox/internal/log"
|
||||
"github.com/googleapis/genai-toolbox/internal/prebuiltconfigs"
|
||||
"github.com/googleapis/genai-toolbox/internal/server"
|
||||
"github.com/googleapis/genai-toolbox/internal/telemetry"
|
||||
"github.com/googleapis/genai-toolbox/internal/util"
|
||||
)
|
||||
|
||||
type IOStreams struct {
|
||||
In io.Reader
|
||||
Out io.Writer
|
||||
ErrOut io.Writer
|
||||
}
|
||||
|
||||
// ToolboxOptions holds dependencies shared by all commands.
|
||||
type ToolboxOptions struct {
|
||||
IOStreams IOStreams
|
||||
Logger log.Logger
|
||||
Cfg server.ServerConfig
|
||||
ToolsFile string
|
||||
ToolsFiles []string
|
||||
ToolsFolder string
|
||||
PrebuiltConfigs []string
|
||||
}
|
||||
|
||||
// Option defines a function that modifies the ToolboxOptions struct.
|
||||
type Option func(*ToolboxOptions)
|
||||
|
||||
// NewToolboxOptions creates a new instance with defaults, then applies any
|
||||
// provided options.
|
||||
func NewToolboxOptions(opts ...Option) *ToolboxOptions {
|
||||
o := &ToolboxOptions{
|
||||
IOStreams: IOStreams{
|
||||
In: os.Stdin,
|
||||
Out: os.Stdout,
|
||||
ErrOut: os.Stderr,
|
||||
},
|
||||
}
|
||||
|
||||
for _, opt := range opts {
|
||||
opt(o)
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// Apply allows you to update an EXISTING ToolboxOptions instance.
|
||||
// This is useful for "late binding".
|
||||
func (o *ToolboxOptions) Apply(opts ...Option) {
|
||||
for _, opt := range opts {
|
||||
opt(o)
|
||||
}
|
||||
}
|
||||
|
||||
// WithIOStreams updates the IO streams.
|
||||
func WithIOStreams(out, err io.Writer) Option {
|
||||
return func(o *ToolboxOptions) {
|
||||
o.IOStreams.Out = out
|
||||
o.IOStreams.ErrOut = err
|
||||
}
|
||||
}
|
||||
|
||||
// Setup create logger and telemetry instrumentations.
|
||||
func (opts *ToolboxOptions) Setup(ctx context.Context) (context.Context, func(context.Context) error, error) {
|
||||
// If stdio, set logger's out stream (usually DEBUG and INFO logs) to
|
||||
// errStream
|
||||
loggerOut := opts.IOStreams.Out
|
||||
if opts.Cfg.Stdio {
|
||||
loggerOut = opts.IOStreams.ErrOut
|
||||
}
|
||||
|
||||
// Handle logger separately from config
|
||||
logger, err := log.NewLogger(opts.Cfg.LoggingFormat.String(), opts.Cfg.LogLevel.String(), loggerOut, opts.IOStreams.ErrOut)
|
||||
if err != nil {
|
||||
return ctx, nil, fmt.Errorf("unable to initialize logger: %w", err)
|
||||
}
|
||||
|
||||
ctx = util.WithLogger(ctx, logger)
|
||||
opts.Logger = logger
|
||||
|
||||
// Set up OpenTelemetry
|
||||
otelShutdown, err := telemetry.SetupOTel(ctx, opts.Cfg.Version, opts.Cfg.TelemetryOTLP, opts.Cfg.TelemetryGCP, opts.Cfg.TelemetryServiceName)
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("error setting up OpenTelemetry: %w", err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
return ctx, nil, errMsg
|
||||
}
|
||||
|
||||
shutdownFunc := func(ctx context.Context) error {
|
||||
err := otelShutdown(ctx)
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("error shutting down OpenTelemetry: %w", err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
instrumentation, err := telemetry.CreateTelemetryInstrumentation(opts.Cfg.Version)
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("unable to create telemetry instrumentation: %w", err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
return ctx, shutdownFunc, errMsg
|
||||
}
|
||||
|
||||
ctx = util.WithInstrumentation(ctx, instrumentation)
|
||||
|
||||
return ctx, shutdownFunc, nil
|
||||
}
|
||||
|
||||
// LoadConfig checks and merge files that should be loaded into the server
|
||||
func (opts *ToolboxOptions) LoadConfig(ctx context.Context) (bool, error) {
|
||||
// Determine if Custom Files should be loaded
|
||||
// Check for explicit custom flags
|
||||
isCustomConfigured := opts.ToolsFile != "" || len(opts.ToolsFiles) > 0 || opts.ToolsFolder != ""
|
||||
|
||||
// Determine if default 'tools.yaml' should be used (No prebuilt AND No custom flags)
|
||||
useDefaultToolsFile := len(opts.PrebuiltConfigs) == 0 && !isCustomConfigured
|
||||
|
||||
if useDefaultToolsFile {
|
||||
opts.ToolsFile = "tools.yaml"
|
||||
isCustomConfigured = true
|
||||
}
|
||||
|
||||
logger, err := util.LoggerFromContext(ctx)
|
||||
if err != nil {
|
||||
return isCustomConfigured, err
|
||||
}
|
||||
|
||||
var allToolsFiles []ToolsFile
|
||||
|
||||
// Load Prebuilt Configuration
|
||||
|
||||
if len(opts.PrebuiltConfigs) > 0 {
|
||||
slices.Sort(opts.PrebuiltConfigs)
|
||||
sourcesList := strings.Join(opts.PrebuiltConfigs, ", ")
|
||||
logMsg := fmt.Sprintf("Using prebuilt tool configurations for: %s", sourcesList)
|
||||
logger.InfoContext(ctx, logMsg)
|
||||
|
||||
for _, configName := range opts.PrebuiltConfigs {
|
||||
buf, err := prebuiltconfigs.Get(configName)
|
||||
if err != nil {
|
||||
logger.ErrorContext(ctx, err.Error())
|
||||
return isCustomConfigured, err
|
||||
}
|
||||
|
||||
// Parse into ToolsFile struct
|
||||
parsed, err := parseToolsFile(ctx, buf)
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("unable to parse prebuilt tool configuration for '%s': %w", configName, err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
return isCustomConfigured, errMsg
|
||||
}
|
||||
allToolsFiles = append(allToolsFiles, parsed)
|
||||
}
|
||||
}
|
||||
|
||||
// Load Custom Configurations
|
||||
if isCustomConfigured {
|
||||
// Enforce exclusivity among custom flags (tools-file vs tools-files vs tools-folder)
|
||||
if (opts.ToolsFile != "" && len(opts.ToolsFiles) > 0) ||
|
||||
(opts.ToolsFile != "" && opts.ToolsFolder != "") ||
|
||||
(len(opts.ToolsFiles) > 0 && opts.ToolsFolder != "") {
|
||||
errMsg := fmt.Errorf("--tools-file, --tools-files, and --tools-folder flags cannot be used simultaneously")
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
return isCustomConfigured, errMsg
|
||||
}
|
||||
|
||||
var customTools ToolsFile
|
||||
var err error
|
||||
|
||||
if len(opts.ToolsFiles) > 0 {
|
||||
// Use tools-files
|
||||
logger.InfoContext(ctx, fmt.Sprintf("Loading and merging %d tool configuration files", len(opts.ToolsFiles)))
|
||||
customTools, err = LoadAndMergeToolsFiles(ctx, opts.ToolsFiles)
|
||||
} else if opts.ToolsFolder != "" {
|
||||
// Use tools-folder
|
||||
logger.InfoContext(ctx, fmt.Sprintf("Loading and merging all YAML files from directory: %s", opts.ToolsFolder))
|
||||
customTools, err = LoadAndMergeToolsFolder(ctx, opts.ToolsFolder)
|
||||
} else {
|
||||
// Use single file (tools-file or default `tools.yaml`)
|
||||
buf, readFileErr := os.ReadFile(opts.ToolsFile)
|
||||
if readFileErr != nil {
|
||||
errMsg := fmt.Errorf("unable to read tool file at %q: %w", opts.ToolsFile, readFileErr)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
return isCustomConfigured, errMsg
|
||||
}
|
||||
customTools, err = parseToolsFile(ctx, buf)
|
||||
if err != nil {
|
||||
err = fmt.Errorf("unable to parse tool file at %q: %w", opts.ToolsFile, err)
|
||||
}
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
logger.ErrorContext(ctx, err.Error())
|
||||
return isCustomConfigured, err
|
||||
}
|
||||
allToolsFiles = append(allToolsFiles, customTools)
|
||||
}
|
||||
|
||||
// Modify version string based on loaded configurations
|
||||
if len(opts.PrebuiltConfigs) > 0 {
|
||||
tag := "prebuilt"
|
||||
if isCustomConfigured {
|
||||
tag = "custom"
|
||||
}
|
||||
// prebuiltConfigs is already sorted above
|
||||
for _, configName := range opts.PrebuiltConfigs {
|
||||
opts.Cfg.Version += fmt.Sprintf("+%s.%s", tag, configName)
|
||||
}
|
||||
}
|
||||
|
||||
// Merge Everything
|
||||
// This will error if custom tools collide with prebuilt tools
|
||||
finalToolsFile, err := mergeToolsFiles(allToolsFiles...)
|
||||
if err != nil {
|
||||
logger.ErrorContext(ctx, err.Error())
|
||||
return isCustomConfigured, err
|
||||
}
|
||||
|
||||
opts.Cfg.SourceConfigs = finalToolsFile.Sources
|
||||
opts.Cfg.AuthServiceConfigs = finalToolsFile.AuthServices
|
||||
opts.Cfg.EmbeddingModelConfigs = finalToolsFile.EmbeddingModels
|
||||
opts.Cfg.ToolConfigs = finalToolsFile.Tools
|
||||
opts.Cfg.ToolsetConfigs = finalToolsFile.Toolsets
|
||||
opts.Cfg.PromptConfigs = finalToolsFile.Prompts
|
||||
|
||||
return isCustomConfigured, nil
|
||||
}
|
||||
@@ -12,57 +12,38 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package cmd
|
||||
package internal
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io"
|
||||
"testing"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func TestCommandOptions(t *testing.T) {
|
||||
func TestToolboxOptions(t *testing.T) {
|
||||
w := io.Discard
|
||||
tcs := []struct {
|
||||
desc string
|
||||
isValid func(*Command) error
|
||||
isValid func(*ToolboxOptions) error
|
||||
option Option
|
||||
}{
|
||||
{
|
||||
desc: "with logger",
|
||||
isValid: func(c *Command) error {
|
||||
if c.outStream != w || c.errStream != w {
|
||||
isValid: func(o *ToolboxOptions) error {
|
||||
if o.IOStreams.Out != w || o.IOStreams.ErrOut != w {
|
||||
return errors.New("loggers do not match")
|
||||
}
|
||||
return nil
|
||||
},
|
||||
option: WithStreams(w, w),
|
||||
option: WithIOStreams(w, w),
|
||||
},
|
||||
}
|
||||
for _, tc := range tcs {
|
||||
t.Run(tc.desc, func(t *testing.T) {
|
||||
got, err := invokeProxyWithOption(tc.option)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := NewToolboxOptions(tc.option)
|
||||
if err := tc.isValid(got); err != nil {
|
||||
t.Errorf("option did not initialize command correctly: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func invokeProxyWithOption(o Option) (*Command, error) {
|
||||
c := NewCommand(o)
|
||||
// Keep the test output quiet
|
||||
c.SilenceUsage = true
|
||||
c.SilenceErrors = true
|
||||
// Disable execute behavior
|
||||
c.RunE = func(*cobra.Command, []string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
err := c.Execute()
|
||||
return c, err
|
||||
}
|
||||
46
cmd/internal/persistent_flags.go
Normal file
46
cmd/internal/persistent_flags.go
Normal file
@@ -0,0 +1,46 @@
|
||||
// Copyright 2026 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package internal
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/googleapis/genai-toolbox/internal/prebuiltconfigs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
// PersistentFlags sets up flags that are available for all commands and
|
||||
// subcommands
|
||||
// It is also used to set up persistent flags during subcommand unit tests
|
||||
func PersistentFlags(parentCmd *cobra.Command, opts *ToolboxOptions) {
|
||||
persistentFlags := parentCmd.PersistentFlags()
|
||||
|
||||
persistentFlags.StringVar(&opts.ToolsFile, "tools-file", "", "File path specifying the tool configuration. Cannot be used with --tools-files, or --tools-folder.")
|
||||
persistentFlags.StringSliceVar(&opts.ToolsFiles, "tools-files", []string{}, "Multiple file paths specifying tool configurations. Files will be merged. Cannot be used with --tools-file, or --tools-folder.")
|
||||
persistentFlags.StringVar(&opts.ToolsFolder, "tools-folder", "", "Directory path containing YAML tool configuration files. All .yaml and .yml files in the directory will be loaded and merged. Cannot be used with --tools-file, or --tools-files.")
|
||||
persistentFlags.Var(&opts.Cfg.LogLevel, "log-level", "Specify the minimum level logged. Allowed: 'DEBUG', 'INFO', 'WARN', 'ERROR'.")
|
||||
persistentFlags.Var(&opts.Cfg.LoggingFormat, "logging-format", "Specify logging format to use. Allowed: 'standard' or 'JSON'.")
|
||||
persistentFlags.BoolVar(&opts.Cfg.TelemetryGCP, "telemetry-gcp", false, "Enable exporting directly to Google Cloud Monitoring.")
|
||||
persistentFlags.StringVar(&opts.Cfg.TelemetryOTLP, "telemetry-otlp", "", "Enable exporting using OpenTelemetry Protocol (OTLP) to the specified endpoint (e.g. 'http://127.0.0.1:4318')")
|
||||
persistentFlags.StringVar(&opts.Cfg.TelemetryServiceName, "telemetry-service-name", "toolbox", "Sets the value of the service.name resource attribute for telemetry data.")
|
||||
// Fetch prebuilt tools sources to customize the help description
|
||||
prebuiltHelp := fmt.Sprintf(
|
||||
"Use a prebuilt tool configuration by source type. Allowed: '%s'. Can be specified multiple times.",
|
||||
strings.Join(prebuiltconfigs.GetPrebuiltSources(), "', '"),
|
||||
)
|
||||
persistentFlags.StringSliceVar(&opts.PrebuiltConfigs, "prebuilt", []string{}, prebuiltHelp)
|
||||
persistentFlags.StringSliceVar(&opts.Cfg.UserAgentMetadata, "user-agent-metadata", []string{}, "Appends additional metadata to the User-Agent.")
|
||||
}
|
||||
@@ -22,7 +22,7 @@ import (
|
||||
"path/filepath"
|
||||
"sort"
|
||||
|
||||
"github.com/googleapis/genai-toolbox/internal/log"
|
||||
"github.com/googleapis/genai-toolbox/cmd/internal"
|
||||
"github.com/googleapis/genai-toolbox/internal/server"
|
||||
"github.com/googleapis/genai-toolbox/internal/server/resources"
|
||||
"github.com/googleapis/genai-toolbox/internal/tools"
|
||||
@@ -30,28 +30,9 @@ import (
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
// RootCommand defines the interface for required by skills-generate subcommand.
|
||||
// This allows subcommands to access shared resources and functionality without
|
||||
// direct coupling to the root command's implementation.
|
||||
type RootCommand interface {
|
||||
// Config returns a copy of the current server configuration.
|
||||
Config() server.ServerConfig
|
||||
|
||||
// LoadConfig loads and merges the configuration from files, folders, and prebuilts.
|
||||
LoadConfig(ctx context.Context) error
|
||||
|
||||
// Setup initializes the runtime environment, including logging and telemetry.
|
||||
// It returns the updated context and a shutdown function to be called when finished.
|
||||
Setup(ctx context.Context) (context.Context, func(context.Context) error, error)
|
||||
|
||||
// Logger returns the logger instance.
|
||||
Logger() log.Logger
|
||||
}
|
||||
|
||||
// Command is the command for generating skills.
|
||||
type Command struct {
|
||||
// skillsCmd is the command for generating skills.
|
||||
type skillsCmd struct {
|
||||
*cobra.Command
|
||||
rootCmd RootCommand
|
||||
name string
|
||||
description string
|
||||
toolset string
|
||||
@@ -59,15 +40,13 @@ type Command struct {
|
||||
}
|
||||
|
||||
// NewCommand creates a new Command.
|
||||
func NewCommand(rootCmd RootCommand) *cobra.Command {
|
||||
cmd := &Command{
|
||||
rootCmd: rootCmd,
|
||||
}
|
||||
func NewCommand(opts *internal.ToolboxOptions) *cobra.Command {
|
||||
cmd := &skillsCmd{}
|
||||
cmd.Command = &cobra.Command{
|
||||
Use: "skills-generate",
|
||||
Short: "Generate skills from tool configurations",
|
||||
RunE: func(c *cobra.Command, args []string) error {
|
||||
return cmd.run(c)
|
||||
return run(cmd, opts)
|
||||
},
|
||||
}
|
||||
|
||||
@@ -81,11 +60,11 @@ func NewCommand(rootCmd RootCommand) *cobra.Command {
|
||||
return cmd.Command
|
||||
}
|
||||
|
||||
func (c *Command) run(cmd *cobra.Command) error {
|
||||
func run(cmd *skillsCmd, opts *internal.ToolboxOptions) error {
|
||||
ctx, cancel := context.WithCancel(cmd.Context())
|
||||
defer cancel()
|
||||
|
||||
ctx, shutdown, err := c.rootCmd.Setup(ctx)
|
||||
ctx, shutdown, err := opts.Setup(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -93,39 +72,37 @@ func (c *Command) run(cmd *cobra.Command) error {
|
||||
_ = shutdown(ctx)
|
||||
}()
|
||||
|
||||
logger := c.rootCmd.Logger()
|
||||
|
||||
// Load and merge tool configurations
|
||||
if err := c.rootCmd.LoadConfig(ctx); err != nil {
|
||||
_, err = opts.LoadConfig(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := os.MkdirAll(c.outputDir, 0755); err != nil {
|
||||
if err := os.MkdirAll(cmd.outputDir, 0755); err != nil {
|
||||
errMsg := fmt.Errorf("error creating output directory: %w", err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
logger.InfoContext(ctx, fmt.Sprintf("Generating skill '%s'...", c.name))
|
||||
opts.Logger.InfoContext(ctx, fmt.Sprintf("Generating skill '%s'...", cmd.name))
|
||||
|
||||
// Initialize toolbox and collect tools
|
||||
allTools, err := c.collectTools(ctx)
|
||||
allTools, err := cmd.collectTools(ctx, opts)
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("error collecting tools: %w", err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
if len(allTools) == 0 {
|
||||
logger.InfoContext(ctx, "No tools found to generate.")
|
||||
opts.Logger.InfoContext(ctx, "No tools found to generate.")
|
||||
return nil
|
||||
}
|
||||
|
||||
// Generate the combined skill directory
|
||||
skillPath := filepath.Join(c.outputDir, c.name)
|
||||
skillPath := filepath.Join(cmd.outputDir, cmd.name)
|
||||
if err := os.MkdirAll(skillPath, 0755); err != nil {
|
||||
errMsg := fmt.Errorf("error creating skill directory: %w", err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
@@ -133,7 +110,7 @@ func (c *Command) run(cmd *cobra.Command) error {
|
||||
assetsPath := filepath.Join(skillPath, "assets")
|
||||
if err := os.MkdirAll(assetsPath, 0755); err != nil {
|
||||
errMsg := fmt.Errorf("error creating assets dir: %w", err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
@@ -141,7 +118,7 @@ func (c *Command) run(cmd *cobra.Command) error {
|
||||
scriptsPath := filepath.Join(skillPath, "scripts")
|
||||
if err := os.MkdirAll(scriptsPath, 0755); err != nil {
|
||||
errMsg := fmt.Errorf("error creating scripts dir: %w", err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
@@ -154,10 +131,10 @@ func (c *Command) run(cmd *cobra.Command) error {
|
||||
|
||||
for _, toolName := range toolNames {
|
||||
// Generate YAML config in asset directory
|
||||
minimizedContent, err := generateToolConfigYAML(c.rootCmd.Config(), toolName)
|
||||
minimizedContent, err := generateToolConfigYAML(opts.Cfg, toolName)
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("error generating filtered config for %s: %w", toolName, err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
@@ -166,7 +143,7 @@ func (c *Command) run(cmd *cobra.Command) error {
|
||||
destPath := filepath.Join(assetsPath, specificToolsFileName)
|
||||
if err := os.WriteFile(destPath, minimizedContent, 0644); err != nil {
|
||||
errMsg := fmt.Errorf("error writing filtered config for %s: %w", toolName, err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
}
|
||||
@@ -175,40 +152,40 @@ func (c *Command) run(cmd *cobra.Command) error {
|
||||
scriptContent, err := generateScriptContent(toolName, specificToolsFileName)
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("error generating script content for %s: %w", toolName, err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
scriptFilename := filepath.Join(scriptsPath, fmt.Sprintf("%s.js", toolName))
|
||||
if err := os.WriteFile(scriptFilename, []byte(scriptContent), 0755); err != nil {
|
||||
errMsg := fmt.Errorf("error writing script %s: %w", scriptFilename, err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
}
|
||||
|
||||
// Generate SKILL.md
|
||||
skillContent, err := generateSkillMarkdown(c.name, c.description, allTools)
|
||||
skillContent, err := generateSkillMarkdown(cmd.name, cmd.description, allTools)
|
||||
if err != nil {
|
||||
errMsg := fmt.Errorf("error generating SKILL.md content: %w", err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
skillMdPath := filepath.Join(skillPath, "SKILL.md")
|
||||
if err := os.WriteFile(skillMdPath, []byte(skillContent), 0644); err != nil {
|
||||
errMsg := fmt.Errorf("error writing SKILL.md: %w", err)
|
||||
logger.ErrorContext(ctx, errMsg.Error())
|
||||
opts.Logger.ErrorContext(ctx, errMsg.Error())
|
||||
return errMsg
|
||||
}
|
||||
|
||||
logger.InfoContext(ctx, fmt.Sprintf("Successfully generated skill '%s' with %d tools.", c.name, len(allTools)))
|
||||
opts.Logger.InfoContext(ctx, fmt.Sprintf("Successfully generated skill '%s' with %d tools.", cmd.name, len(allTools)))
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *Command) collectTools(ctx context.Context) (map[string]tools.Tool, error) {
|
||||
func (c *skillsCmd) collectTools(ctx context.Context, opts *internal.ToolboxOptions) (map[string]tools.Tool, error) {
|
||||
// Initialize Resources
|
||||
sourcesMap, authServicesMap, embeddingModelsMap, toolsMap, toolsetsMap, promptsMap, promptsetsMap, err := server.InitializeConfigs(ctx, c.rootCmd.Config())
|
||||
sourcesMap, authServicesMap, embeddingModelsMap, toolsMap, toolsetsMap, promptsMap, promptsetsMap, err := server.InitializeConfigs(ctx, opts.Cfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to initialize resources: %w", err)
|
||||
}
|
||||
@@ -12,17 +12,36 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package cmd
|
||||
package skills
|
||||
|
||||
import (
|
||||
"context"
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/googleapis/genai-toolbox/cmd/internal"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/sources/sqlite"
|
||||
_ "github.com/googleapis/genai-toolbox/internal/tools/sqlite/sqlitesql"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func invokeCommand(args []string) (string, error) {
|
||||
parentCmd := &cobra.Command{Use: "toolbox"}
|
||||
|
||||
buf := new(bytes.Buffer)
|
||||
opts := internal.NewToolboxOptions(internal.WithIOStreams(buf, buf))
|
||||
internal.PersistentFlags(parentCmd, opts)
|
||||
|
||||
cmd := NewCommand(opts)
|
||||
parentCmd.AddCommand(cmd)
|
||||
parentCmd.SetArgs(args)
|
||||
|
||||
err := parentCmd.Execute()
|
||||
return buf.String(), err
|
||||
}
|
||||
|
||||
func TestGenerateSkill(t *testing.T) {
|
||||
// Create a temporary directory for tests
|
||||
tmpDir := t.TempDir()
|
||||
@@ -55,10 +74,7 @@ tools:
|
||||
"--description", "hello tool",
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
_, got, err := invokeCommandWithContext(ctx, args)
|
||||
got, err := invokeCommand(args)
|
||||
if err != nil {
|
||||
t.Fatalf("command failed: %v\nOutput: %s", err, got)
|
||||
}
|
||||
@@ -136,7 +152,7 @@ func TestGenerateSkill_NoConfig(t *testing.T) {
|
||||
"--description", "test",
|
||||
}
|
||||
|
||||
_, _, err := invokeCommandWithContext(context.Background(), args)
|
||||
_, err := invokeCommand(args)
|
||||
if err == nil {
|
||||
t.Fatal("expected command to fail when no configuration is provided and tools.yaml is missing")
|
||||
}
|
||||
@@ -170,7 +186,7 @@ func TestGenerateSkill_MissingArguments(t *testing.T) {
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
_, got, err := invokeCommandWithContext(context.Background(), tt.args)
|
||||
got, err := invokeCommand(tt.args)
|
||||
if err == nil {
|
||||
t.Fatalf("expected command to fail due to missing arguments, but it succeeded\nOutput: %s", got)
|
||||
}
|
||||
349
cmd/internal/tools_file.go
Normal file
349
cmd/internal/tools_file.go
Normal file
@@ -0,0 +1,349 @@
|
||||
// Copyright 2026 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package internal
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"github.com/goccy/go-yaml"
|
||||
"github.com/googleapis/genai-toolbox/internal/server"
|
||||
)
|
||||
|
||||
type ToolsFile struct {
|
||||
Sources server.SourceConfigs `yaml:"sources"`
|
||||
AuthServices server.AuthServiceConfigs `yaml:"authServices"`
|
||||
EmbeddingModels server.EmbeddingModelConfigs `yaml:"embeddingModels"`
|
||||
Tools server.ToolConfigs `yaml:"tools"`
|
||||
Toolsets server.ToolsetConfigs `yaml:"toolsets"`
|
||||
Prompts server.PromptConfigs `yaml:"prompts"`
|
||||
}
|
||||
|
||||
// parseEnv replaces environment variables ${ENV_NAME} with their values.
|
||||
// also support ${ENV_NAME:default_value}.
|
||||
func parseEnv(input string) (string, error) {
|
||||
re := regexp.MustCompile(`\$\{(\w+)(:([^}]*))?\}`)
|
||||
|
||||
var err error
|
||||
output := re.ReplaceAllStringFunc(input, func(match string) string {
|
||||
parts := re.FindStringSubmatch(match)
|
||||
|
||||
// extract the variable name
|
||||
variableName := parts[1]
|
||||
if value, found := os.LookupEnv(variableName); found {
|
||||
return value
|
||||
}
|
||||
if len(parts) >= 4 && parts[2] != "" {
|
||||
return parts[3]
|
||||
}
|
||||
err = fmt.Errorf("environment variable not found: %q", variableName)
|
||||
return ""
|
||||
})
|
||||
return output, err
|
||||
}
|
||||
|
||||
// parseToolsFile parses the provided yaml into appropriate configs.
|
||||
func parseToolsFile(ctx context.Context, raw []byte) (ToolsFile, error) {
|
||||
var toolsFile ToolsFile
|
||||
// Replace environment variables if found
|
||||
output, err := parseEnv(string(raw))
|
||||
if err != nil {
|
||||
return toolsFile, fmt.Errorf("error parsing environment variables: %s", err)
|
||||
}
|
||||
raw = []byte(output)
|
||||
|
||||
raw, err = convertToolsFile(raw)
|
||||
if err != nil {
|
||||
return toolsFile, fmt.Errorf("error converting tools file: %s", err)
|
||||
}
|
||||
|
||||
// Parse contents
|
||||
toolsFile.Sources, toolsFile.AuthServices, toolsFile.EmbeddingModels, toolsFile.Tools, toolsFile.Toolsets, toolsFile.Prompts, err = server.UnmarshalResourceConfig(ctx, raw)
|
||||
if err != nil {
|
||||
return toolsFile, err
|
||||
}
|
||||
return toolsFile, nil
|
||||
}
|
||||
|
||||
func convertToolsFile(raw []byte) ([]byte, error) {
|
||||
var input yaml.MapSlice
|
||||
decoder := yaml.NewDecoder(bytes.NewReader(raw), yaml.UseOrderedMap())
|
||||
|
||||
// convert to tools file v2
|
||||
var buf bytes.Buffer
|
||||
encoder := yaml.NewEncoder(&buf)
|
||||
|
||||
v1keys := []string{"sources", "authSources", "authServices", "embeddingModels", "tools", "toolsets", "prompts"}
|
||||
for {
|
||||
if err := decoder.Decode(&input); err != nil {
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
for _, item := range input {
|
||||
key, ok := item.Key.(string)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("unexpected non-string key in input: %v", item.Key)
|
||||
}
|
||||
// check if the key is config file v1's key
|
||||
if slices.Contains(v1keys, key) {
|
||||
// check if value conversion to yaml.MapSlice successfully
|
||||
// fields such as "tools" in toolsets might pass the first check but
|
||||
// fail to convert to MapSlice
|
||||
if slice, ok := item.Value.(yaml.MapSlice); ok {
|
||||
// Deprecated: convert authSources to authServices
|
||||
if key == "authSources" {
|
||||
key = "authServices"
|
||||
}
|
||||
transformed, err := transformDocs(key, slice)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// encode per-doc
|
||||
for _, doc := range transformed {
|
||||
if err := encoder.Encode(doc); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// invalid input will be ignored
|
||||
// we don't want to throw error here since the config could
|
||||
// be valid but with a different order such as:
|
||||
// ---
|
||||
// tools:
|
||||
// - tool_a
|
||||
// kind: toolsets
|
||||
// ---
|
||||
continue
|
||||
}
|
||||
} else {
|
||||
// this doc is already v2, encode to buf
|
||||
if err := encoder.Encode(input); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return buf.Bytes(), nil
|
||||
}
|
||||
|
||||
// transformDocs transforms the configuration file from v1 format to v2
|
||||
// yaml.MapSlice will preserve the order in a map
|
||||
func transformDocs(kind string, input yaml.MapSlice) ([]yaml.MapSlice, error) {
|
||||
var transformed []yaml.MapSlice
|
||||
for _, entry := range input {
|
||||
entryName, ok := entry.Key.(string)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("unexpected non-string key for entry in '%s': %v", kind, entry.Key)
|
||||
}
|
||||
entryBody := ProcessValue(entry.Value, kind == "toolsets")
|
||||
|
||||
currentTransformed := yaml.MapSlice{
|
||||
{Key: "kind", Value: kind},
|
||||
{Key: "name", Value: entryName},
|
||||
}
|
||||
|
||||
// Merge the transformed body into our result
|
||||
if bodySlice, ok := entryBody.(yaml.MapSlice); ok {
|
||||
currentTransformed = append(currentTransformed, bodySlice...)
|
||||
} else {
|
||||
return nil, fmt.Errorf("unable to convert entryBody to MapSlice")
|
||||
}
|
||||
transformed = append(transformed, currentTransformed)
|
||||
}
|
||||
return transformed, nil
|
||||
}
|
||||
|
||||
// ProcessValue recursively looks for MapSlices to rename 'kind' -> 'type'
|
||||
func ProcessValue(v any, isToolset bool) any {
|
||||
switch val := v.(type) {
|
||||
case yaml.MapSlice:
|
||||
// creating a new MapSlice is safer for recursive transformation
|
||||
newVal := make(yaml.MapSlice, len(val))
|
||||
for i, item := range val {
|
||||
// Perform renaming
|
||||
if item.Key == "kind" {
|
||||
item.Key = "type"
|
||||
}
|
||||
// Recursive call for nested values (e.g., nested objects or lists)
|
||||
item.Value = ProcessValue(item.Value, false)
|
||||
newVal[i] = item
|
||||
}
|
||||
return newVal
|
||||
case []any:
|
||||
// Process lists: If it's a toolset top-level list, wrap it.
|
||||
if isToolset {
|
||||
return yaml.MapSlice{{Key: "tools", Value: val}}
|
||||
}
|
||||
// Otherwise, recurse into list items (to catch nested objects)
|
||||
newVal := make([]any, len(val))
|
||||
for i := range val {
|
||||
newVal[i] = ProcessValue(val[i], false)
|
||||
}
|
||||
return newVal
|
||||
default:
|
||||
return val
|
||||
}
|
||||
}
|
||||
|
||||
// mergeToolsFiles merges multiple ToolsFile structs into one.
|
||||
// Detects and raises errors for resource conflicts in sources, authServices, tools, and toolsets.
|
||||
// All resource names (sources, authServices, tools, toolsets) must be unique across all files.
|
||||
func mergeToolsFiles(files ...ToolsFile) (ToolsFile, error) {
|
||||
merged := ToolsFile{
|
||||
Sources: make(server.SourceConfigs),
|
||||
AuthServices: make(server.AuthServiceConfigs),
|
||||
EmbeddingModels: make(server.EmbeddingModelConfigs),
|
||||
Tools: make(server.ToolConfigs),
|
||||
Toolsets: make(server.ToolsetConfigs),
|
||||
Prompts: make(server.PromptConfigs),
|
||||
}
|
||||
|
||||
var conflicts []string
|
||||
|
||||
for fileIndex, file := range files {
|
||||
// Check for conflicts and merge sources
|
||||
for name, source := range file.Sources {
|
||||
if _, exists := merged.Sources[name]; exists {
|
||||
conflicts = append(conflicts, fmt.Sprintf("source '%s' (file #%d)", name, fileIndex+1))
|
||||
} else {
|
||||
merged.Sources[name] = source
|
||||
}
|
||||
}
|
||||
|
||||
// Check for conflicts and merge authServices
|
||||
for name, authService := range file.AuthServices {
|
||||
if _, exists := merged.AuthServices[name]; exists {
|
||||
conflicts = append(conflicts, fmt.Sprintf("authService '%s' (file #%d)", name, fileIndex+1))
|
||||
} else {
|
||||
merged.AuthServices[name] = authService
|
||||
}
|
||||
}
|
||||
|
||||
// Check for conflicts and merge embeddingModels
|
||||
for name, em := range file.EmbeddingModels {
|
||||
if _, exists := merged.EmbeddingModels[name]; exists {
|
||||
conflicts = append(conflicts, fmt.Sprintf("embedding model '%s' (file #%d)", name, fileIndex+1))
|
||||
} else {
|
||||
merged.EmbeddingModels[name] = em
|
||||
}
|
||||
}
|
||||
|
||||
// Check for conflicts and merge tools
|
||||
for name, tool := range file.Tools {
|
||||
if _, exists := merged.Tools[name]; exists {
|
||||
conflicts = append(conflicts, fmt.Sprintf("tool '%s' (file #%d)", name, fileIndex+1))
|
||||
} else {
|
||||
merged.Tools[name] = tool
|
||||
}
|
||||
}
|
||||
|
||||
// Check for conflicts and merge toolsets
|
||||
for name, toolset := range file.Toolsets {
|
||||
if _, exists := merged.Toolsets[name]; exists {
|
||||
conflicts = append(conflicts, fmt.Sprintf("toolset '%s' (file #%d)", name, fileIndex+1))
|
||||
} else {
|
||||
merged.Toolsets[name] = toolset
|
||||
}
|
||||
}
|
||||
|
||||
// Check for conflicts and merge prompts
|
||||
for name, prompt := range file.Prompts {
|
||||
if _, exists := merged.Prompts[name]; exists {
|
||||
conflicts = append(conflicts, fmt.Sprintf("prompt '%s' (file #%d)", name, fileIndex+1))
|
||||
} else {
|
||||
merged.Prompts[name] = prompt
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// If conflicts were detected, return an error
|
||||
if len(conflicts) > 0 {
|
||||
return ToolsFile{}, fmt.Errorf("resource conflicts detected:\n - %s\n\nPlease ensure each source, authService, tool, toolset and prompt has a unique name across all files", strings.Join(conflicts, "\n - "))
|
||||
}
|
||||
|
||||
return merged, nil
|
||||
}
|
||||
|
||||
// LoadAndMergeToolsFiles loads multiple YAML files and merges them
|
||||
func LoadAndMergeToolsFiles(ctx context.Context, filePaths []string) (ToolsFile, error) {
|
||||
var toolsFiles []ToolsFile
|
||||
|
||||
for _, filePath := range filePaths {
|
||||
buf, err := os.ReadFile(filePath)
|
||||
if err != nil {
|
||||
return ToolsFile{}, fmt.Errorf("unable to read tool file at %q: %w", filePath, err)
|
||||
}
|
||||
|
||||
toolsFile, err := parseToolsFile(ctx, buf)
|
||||
if err != nil {
|
||||
return ToolsFile{}, fmt.Errorf("unable to parse tool file at %q: %w", filePath, err)
|
||||
}
|
||||
|
||||
toolsFiles = append(toolsFiles, toolsFile)
|
||||
}
|
||||
|
||||
mergedFile, err := mergeToolsFiles(toolsFiles...)
|
||||
if err != nil {
|
||||
return ToolsFile{}, fmt.Errorf("unable to merge tools files: %w", err)
|
||||
}
|
||||
|
||||
return mergedFile, nil
|
||||
}
|
||||
|
||||
// LoadAndMergeToolsFolder loads all YAML files from a directory and merges them
|
||||
func LoadAndMergeToolsFolder(ctx context.Context, folderPath string) (ToolsFile, error) {
|
||||
// Check if directory exists
|
||||
info, err := os.Stat(folderPath)
|
||||
if err != nil {
|
||||
return ToolsFile{}, fmt.Errorf("unable to access tools folder at %q: %w", folderPath, err)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return ToolsFile{}, fmt.Errorf("path %q is not a directory", folderPath)
|
||||
}
|
||||
|
||||
// Find all YAML files in the directory
|
||||
pattern := filepath.Join(folderPath, "*.yaml")
|
||||
yamlFiles, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
return ToolsFile{}, fmt.Errorf("error finding YAML files in %q: %w", folderPath, err)
|
||||
}
|
||||
|
||||
// Also find .yml files
|
||||
ymlPattern := filepath.Join(folderPath, "*.yml")
|
||||
ymlFiles, err := filepath.Glob(ymlPattern)
|
||||
if err != nil {
|
||||
return ToolsFile{}, fmt.Errorf("error finding YML files in %q: %w", folderPath, err)
|
||||
}
|
||||
|
||||
// Combine both file lists
|
||||
allFiles := append(yamlFiles, ymlFiles...)
|
||||
|
||||
if len(allFiles) == 0 {
|
||||
return ToolsFile{}, fmt.Errorf("no YAML files found in directory %q", folderPath)
|
||||
}
|
||||
|
||||
// Use existing LoadAndMergeToolsFiles function
|
||||
return LoadAndMergeToolsFiles(ctx, allFiles)
|
||||
}
|
||||
2141
cmd/internal/tools_file_test.go
Normal file
2141
cmd/internal/tools_file_test.go
Normal file
File diff suppressed because it is too large
Load Diff
@@ -1,30 +0,0 @@
|
||||
// Copyright 2024 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"io"
|
||||
)
|
||||
|
||||
// Option is a function that configures a Command.
|
||||
type Option func(*Command)
|
||||
|
||||
// WithStreams overrides the default writer.
|
||||
func WithStreams(out, err io.Writer) Option {
|
||||
return func(c *Command) {
|
||||
c.outStream = out
|
||||
c.errStream = err
|
||||
}
|
||||
}
|
||||
891
cmd/root.go
891
cmd/root.go
File diff suppressed because it is too large
Load Diff
2204
cmd/root_test.go
2204
cmd/root_test.go
File diff suppressed because it is too large
Load Diff
@@ -21,6 +21,7 @@ import (
|
||||
"os"
|
||||
|
||||
yaml "github.com/goccy/go-yaml"
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"github.com/googleapis/genai-toolbox/cmd"
|
||||
)
|
||||
@@ -50,7 +51,7 @@ func tmpFileWithCleanup(content []byte) (string, func(), error) {
|
||||
type CmdExec struct {
|
||||
Out io.ReadCloser
|
||||
|
||||
cmd *cmd.Command
|
||||
cmd *cobra.Command
|
||||
cancel context.CancelFunc
|
||||
closers []io.Closer
|
||||
done chan bool // closed once the cmd is completed
|
||||
@@ -77,7 +78,7 @@ func StartCmd(ctx context.Context, toolsFile map[string]any, args ...string) (*C
|
||||
return nil, nil, fmt.Errorf("unable to open stdout pipe: %w", err)
|
||||
}
|
||||
|
||||
c := cmd.NewCommand(cmd.WithStreams(pw, pw))
|
||||
c := cmd.GenerateCommand(pw, pw)
|
||||
c.SetArgs(args)
|
||||
|
||||
t := &CmdExec{
|
||||
|
||||
Reference in New Issue
Block a user