This PR introduces a significant update to the Toolbox configuration
file format, which is one of the primary **breaking changes** required
for the implementation of the Advanced Control Plane.
# Summary of Changes
The configuration schema has been updated to enforce resource isolation
and facilitate atomic, incremental updates.
* Resource Isolation: Resource definitions are now separated into
individual blocks, using a distinct structure for each resource type
(Source, Tool, Toolset, etc.). This improves readability, management,
and auditing of configuration files.
* Field Name Modification: Internal field names have been modified to
align with declarative methodologies. Specifically, the configuration
now separates kind (general resource type, e.g., Source) from type
(specific implementation, e.g., Postgres).
# User Impact
Existing tools.yaml configuration files are now in an outdated format.
Users must eventually update their files to the new YAML format.
# Mitigation & Compatibility
Backward compatibility is maintained during this transition to ensure no
immediate user action is required for existing files.
* Immediate Backward Compatibility: The source code includes a
pre-processing layer that automatically detects outdated configuration
files (v1 format) and converts them to the new v2 format under the hood.
* [COMING SOON] Migration Support: The new toolbox migrate subcommand
will be introduced to allow users to automatically convert their old
configuration files to the latest format.
# Example
Example for config file v2:
```
kind: sources
name: my-pg-instance
type: cloud-sql-postgres
project: my-project
region: my-region
instance: my-instance
database: my_db
user: my_user
password: my_pass
---
kind: authServices
name: my-google-auth
type: google
clientId: testing-id
---
kind: tools
name: example_tool
type: postgres-sql
source: my-pg-instance
description: some description
statement: SELECT * FROM SQL_STATEMENT;
parameters:
- name: country
type: string
description: some description
---
kind: tools
name: example_tool_2
type: postgres-sql
source: my-pg-instance
description: returning the number one
statement: SELECT 1;
---
kind: toolsets
name: example_toolset
tools:
- example_tool
```
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Averi Kitsch <akitsch@google.com>
To facilitate the transition of moving invocation implementation to
Source, we will have to move parameter to `internal/util`. This approach
is crucial because certain parameters may not be fully resolvable
pre-implementation. Since both `internal/sources` and `internal/tools`
will need access to `parameters`, it will be more relevant to move
parameters implementation to utils.
Add options for `RunToolInvokeTest()`, `RunMCPToolCallMethod()`,
`RunExecuteSqlToolInvokeTest()`.
When I was trying to update integration test for Firebird. They have a
different syntax from the usual `SELECT 1` that the other DBs are using.
If we want to update the `SELECT 1` statement in the test cases, we will
have to add a new argument and update all the other integration tests.
With `Options`, we can just add customization without having the need to
update all other integration tests. This allow the integration tests to
be more flexible. With this, we can also remove the
`GetNonSpannerWant()` since those can be set as default values for the
`Options`.
Update `tool.Invoke()` to return type `any` instead of `[]any`.
Toolbox return a map with the `results` key, and the SDK reads the
string from the key. So this won't break existing SDK implementation.
Fixes#870
- Spanner: convert array to typed before querying, as Spanner does not
accept non-typed array []any
- BigQuery: fix https://github.com/googleapis/genai-toolbox/issues/793
- Bigtable: add the required `ElemType` for array-type params
- Redis/Valkey: change indexing to append to avoid extra spaces
Add integration tests for array parameters, skipped for the sources not
supporting arrays:
- SQLite
- Cloud SQL MSSQL
- Cloud SQL MySQL
- MSSQL
- MySQL
Allow Toolbox server to automatically update when users modify their
tool configuration file(s), instead of requiring a restart.
This feature is automatically enabled, but can be turned off with the
flag `--disable-reload`.
Add `templateParameters` to support non-filter parameters and DDL
statements.
Added a new argument `ignoreDdl` at integration test. Admin client is
needed to execute ddl statement in spanner. Toolbox does not use admin
client.
Part of #535
Remove cleaning `\\` or `\` in `got` and `want` strings of test cases.
This is to prevent future the need to manually clean strings before
comparing. This also allows us to be aware of the amount of backspace to
prevent accidental escaping in src code.
Update integration test that is affected. This includes moving the
`want` of test cases for `my-param-tool` invocation to each sources
since spanner have a different format (return id as strings instead of
int).
Update http tools `Invoke` method.
* use `json.Unmarshal` for response body that are either list or json
map. This will allow us to return lists as is (e.g. `[{map}]` instead of
having to nest it in an additional list (e.g. `[]any{ [{map}] }`)
* for response body that are strings or fail to unmarshal, return it as
is.
