This PR introduces a significant update to the Toolbox configuration
file format, which is one of the primary **breaking changes** required
for the implementation of the Advanced Control Plane.
# Summary of Changes
The configuration schema has been updated to enforce resource isolation
and facilitate atomic, incremental updates.
* Resource Isolation: Resource definitions are now separated into
individual blocks, using a distinct structure for each resource type
(Source, Tool, Toolset, etc.). This improves readability, management,
and auditing of configuration files.
* Field Name Modification: Internal field names have been modified to
align with declarative methodologies. Specifically, the configuration
now separates kind (general resource type, e.g., Source) from type
(specific implementation, e.g., Postgres).
# User Impact
Existing tools.yaml configuration files are now in an outdated format.
Users must eventually update their files to the new YAML format.
# Mitigation & Compatibility
Backward compatibility is maintained during this transition to ensure no
immediate user action is required for existing files.
* Immediate Backward Compatibility: The source code includes a
pre-processing layer that automatically detects outdated configuration
files (v1 format) and converts them to the new v2 format under the hood.
* [COMING SOON] Migration Support: The new toolbox migrate subcommand
will be introduced to allow users to automatically convert their old
configuration files to the latest format.
# Example
Example for config file v2:
```
kind: sources
name: my-pg-instance
type: cloud-sql-postgres
project: my-project
region: my-region
instance: my-instance
database: my_db
user: my_user
password: my_pass
---
kind: authServices
name: my-google-auth
type: google
clientId: testing-id
---
kind: tools
name: example_tool
type: postgres-sql
source: my-pg-instance
description: some description
statement: SELECT * FROM SQL_STATEMENT;
parameters:
- name: country
type: string
description: some description
---
kind: tools
name: example_tool_2
type: postgres-sql
source: my-pg-instance
description: returning the number one
statement: SELECT 1;
---
kind: toolsets
name: example_toolset
tools:
- example_tool
```
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Averi Kitsch <akitsch@google.com>
Move source-related queries from `Invoke()` function into Source.
The following sources were updated in this PR:
* mssql
* cloudsql-mssql
* mysql
* cloudsql-mysql
This is an effort to generalizing tools to work with any Source that
implements a specific interface. This will provide a better segregation
of the roles for Tools vs Source.
Tool's role will be limited to the following:
* Resolve any pre-implementation steps or parameters (e.g. template
parameters)
* Retrieving Source
* Calling the source's implementation
To keep a persistent backend storage for configuration, we will have to
keep a single source of truth. This involves supporting bi-directional
conversion between Config and Source.
This PR make the following changes:
* Embed Config in Source
* Add `ToConfig()` to extract Config from Source.
## Description
---
Add auditable application name to mssql sources. Query the db with:
```
SELECT session_id, program_name
FROM sys.dm_exec_sessions;
```
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [ ] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [ ] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️ Fixes #<issue_number_goes_here>
## 1. Why do we need to support the `encrypt` parameter?
MSSQL databases that `genai-toolbox` attempts to connect to may have
their encryption levels set differently. For example, a testing/demo
purpose MSSQL database may not require encryption at all. However,
`genai-toolbox` currently uses the default encryption parameter
(`encrypt=false`) to connect to this type of database and will throw an
error:
```
ERROR "toolbox failed to initialize: unable to initialize configs: unable to initialize source "my-mssql-source": unable to connect successfully: TLS Handshake failed: cannot read handshake packet: EOF"
```
> In this case, the encryption parameter should be set to
`encrypt=disable`.
## 2. Is this a necessary feature?
`genai-toolbox` uses the `github.com/microsoft/go-mssqldb` package as a
dependency to connect to MSSQL databases. According to the
[README](https://github.com/microsoft/go-mssqldb?tab=readme-ov-file#common-parameters)
of the `github.com/microsoft/go-mssqldb` package, `encrypt` is one of
the common parameters. Therefore, I believe supporting the `encrypt`
parameter in `genai-toolbox` is necessary.
## 3. How to replicate the error mentioned above?
### 3.1 Use this `docker-compose.yaml` file to start a demo MSSQL
instance
```
services:
demo-mssql-database:
image: mcr.microsoft.com/mssql/server:2017-CU1-ubuntu
ports:
- "20256:1433"
environment:
ACCEPT_EULA: "Y"
MSSQL_SA_PASSWORD: "hellopassword!"
restart: unless-stopped
healthcheck:
test: ["CMD", "/opt/mssql-tools/bin/sqlcmd", "-S", "localhost", "-U", "sa", "-P", "hellopassword!", "-Q", "SELECT 1"]
interval: 5s
retries: 10
demo-mssql-database-init:
image: mcr.microsoft.com/mssql/server:2017-CU1-ubuntu
network_mode: service:demo-mssql-database
command: >
/bin/bash -c "/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P hellopassword! -d master -Q 'CREATE DATABASE DemoDatabase;'"
depends_on:
demo-mssql-database:
condition: service_healthy
```
### 3.2 Use `genai-toolbox` to connect to the above demo MSSQL database
with this `tools.yaml` configuration file:
```
sources:
my-mssql-source:
kind: mssql
host: localhost
port: 20256
database: master
user: sa
password: 'hellopassword!'
```
### 3.3 We shall see the error:
```
ERROR "toolbox failed to initialize: unable to initialize configs: unable to initialize source "my-mssql-source": unable to connect successfully: TLS Handshake failed: cannot read handshake packet: EOF"
```
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
This commit refactors the source configuration and loading mechanism to
use a dynamic registration pattern. Each source package now registers
itself with a central registry via its init() function.
The server configuration code uses this registry to decode and
initialize sources, decoupling it from specific source implementations
and simplifying the addition of new sources.
Key changes:
- Introduced `sources.Register()` and `newConfig()` constructor in each
source package.
- Moved source package imports to `cmd/root.go` as blank imports to
trigger `init()` functions for self-registration.
- Removed direct imports of specific source packages from
`internal/server/config.go`.
- Renamed `SourceKind` constants to `Kind` within each source package.
- Updated tests to use the new `Kind` constants and reflect registration
changes.
---------
Co-authored-by: Yuan Teoh <yuanteoh@google.com>
This only checks within `SourceConfig`, `ToolConfig`, and
`AuthSourceConfig`.
Error when an unknown field is provided:
`2025-01-27T22:43:46.988401-08:00 ERROR "unable to parse tool file at
\"tools.yaml\": unable to parse as \"cloud-sql-postgres\": [2:1] unknown
field \"extra\"\n 1 | database: test_database\n> 2 | extra: here\n ^\n 3
| instance: toolbox-cloudsql\n 4 | kind: cloud-sql-postgres\n 5 |
password: postgres\n 6 | "`
Error when a required field is not provided:
`2025-01-27T17:49:47.584846-08:00 ERROR "unable to parse tool file at
\"tools.yaml\": validation failed: Key: 'Config.Region' Error:Field
validation for 'Region' failed on the 'required' tag"`
---------
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
