- Users have the preference to choose their clients. Below example is
using Gemini CLI.
- Users can use the pre-built Dataplex tools by creating a settings.json
file under .gemini directory. The contents of settings.json would be as
follows:-
```
{
"mcpServers": {
"dataplex": {
"command": "./toolbox",
"args": ["--prebuilt","dataplex","--stdio"],
"env": {
"DATAPLEX_PROJECT": "test-project"
}
}
}
}
```
Fixes#831
---------
Co-authored-by: Wenxin Du <117315983+duwenxin99@users.noreply.github.com>
Co-authored-by: Mateusz Nowak <matnow@google.com>
Co-authored-by: Mateusz Nowak <kontakt@mateusznowak.pl>
## 1. Why do we need to support the `encrypt` parameter?
MSSQL databases that `genai-toolbox` attempts to connect to may have
their encryption levels set differently. For example, a testing/demo
purpose MSSQL database may not require encryption at all. However,
`genai-toolbox` currently uses the default encryption parameter
(`encrypt=false`) to connect to this type of database and will throw an
error:
```
ERROR "toolbox failed to initialize: unable to initialize configs: unable to initialize source "my-mssql-source": unable to connect successfully: TLS Handshake failed: cannot read handshake packet: EOF"
```
> In this case, the encryption parameter should be set to
`encrypt=disable`.
## 2. Is this a necessary feature?
`genai-toolbox` uses the `github.com/microsoft/go-mssqldb` package as a
dependency to connect to MSSQL databases. According to the
[README](https://github.com/microsoft/go-mssqldb?tab=readme-ov-file#common-parameters)
of the `github.com/microsoft/go-mssqldb` package, `encrypt` is one of
the common parameters. Therefore, I believe supporting the `encrypt`
parameter in `genai-toolbox` is necessary.
## 3. How to replicate the error mentioned above?
### 3.1 Use this `docker-compose.yaml` file to start a demo MSSQL
instance
```
services:
demo-mssql-database:
image: mcr.microsoft.com/mssql/server:2017-CU1-ubuntu
ports:
- "20256:1433"
environment:
ACCEPT_EULA: "Y"
MSSQL_SA_PASSWORD: "hellopassword!"
restart: unless-stopped
healthcheck:
test: ["CMD", "/opt/mssql-tools/bin/sqlcmd", "-S", "localhost", "-U", "sa", "-P", "hellopassword!", "-Q", "SELECT 1"]
interval: 5s
retries: 10
demo-mssql-database-init:
image: mcr.microsoft.com/mssql/server:2017-CU1-ubuntu
network_mode: service:demo-mssql-database
command: >
/bin/bash -c "/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P hellopassword! -d master -Q 'CREATE DATABASE DemoDatabase;'"
depends_on:
demo-mssql-database:
condition: service_healthy
```
### 3.2 Use `genai-toolbox` to connect to the above demo MSSQL database
with this `tools.yaml` configuration file:
```
sources:
my-mssql-source:
kind: mssql
host: localhost
port: 20256
database: master
user: sa
password: 'hellopassword!'
```
### 3.3 We shall see the error:
```
ERROR "toolbox failed to initialize: unable to initialize configs: unable to initialize source "my-mssql-source": unable to connect successfully: TLS Handshake failed: cannot read handshake packet: EOF"
```
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
- Added a dry run step to identify the query type (e.g., SELECT, DML),
which allows the tool to correctly handle the query's output.
- The recommended high-level client, cloud.google.com/go/bigquery, does
not expose the statement type from a dry run. To circumvent this
limitation, the low-level BigQuery REST API client
(google.golang.org/api/bigquery/v2) was added to gain access to these
necessary details.
fixes: #915
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
Firestore is a NoSQL document database built for automatic scaling, high
performance, and ease of application development. It's a fully managed,
serverless database that supports mobile, web, and server development.
This change adds Firestore as a source in toolbox
## Summary
- Added configurable query timeout to MySQL source configuration
- Updated connection DSN to include readTimeout parameter
- Added documentation and example usage
- Added test coverage for queryTimeout configuration
## Test plan
- [x] Added unit test for queryTimeout configuration parsing
- [x] Updated documentation with queryTimeout field description
- [x] Verified timeout parameter is correctly added to DSN when
specified
## Caveat
When queryTimeout is exceeded, we get an obscure error message
([screenshot](https://github.com/user-attachments/assets/fd292f91-328d-4ebc-9a87-2d92e9887300)):
```
unable to execute query: invalid connection
```
This seems to be a problem with the mysql-go library:
https://stackoverflow.com/q/65253798/10720618
I tried to use
[MAX_EXECUTION_TIME](https://dev.mysql.com/doc/refman/8.0/en/optimizer-hints.html#optimizer-hints-execution-time)
but it didn't work as expected (my `sleep(MAX_EXECUTION_TIME+3)` query
finished successfully after MAX_EXECUTION_TIME milliseconds)
Any ideas on what can be done here? The error message is very
misleading. My goal with adding timeouts is to communicate to the LLM
when it has issued a slow query and force it to adjust (e.g. query
indexes and write a more optimized query) but this defeats the purpose.
---------
Co-authored-by: Wenxin Du <117315983+duwenxin99@users.noreply.github.com>
This commit refactors the source configuration and loading mechanism to
use a dynamic registration pattern. Each source package now registers
itself with a central registry via its init() function.
The server configuration code uses this registry to decode and
initialize sources, decoupling it from specific source implementations
and simplifying the addition of new sources.
Key changes:
- Introduced `sources.Register()` and `newConfig()` constructor in each
source package.
- Moved source package imports to `cmd/root.go` as blank imports to
trigger `init()` functions for self-registration.
- Removed direct imports of specific source packages from
`internal/server/config.go`.
- Renamed `SourceKind` constants to `Kind` within each source package.
- Updated tests to use the new `Kind` constants and reflect registration
changes.
---------
Co-authored-by: Yuan Teoh <yuanteoh@google.com>
gosimple had been deprecated in favor of staticcheck:
https://github.com/golangci/golangci-lint/issues/357
Other requirements are all migrated.
`std-error-handling` exclusions is included because without that, it
will ask to check all error returns from (`Close()`, or `os.Setenv`s, or
`fmt.Fprint`s...
A `BigQuery` source can be added as the following example:
```yaml
sources:
my-bigquery-source:
kind: bigquery
project: bigframes-dev
location: us # This field is optional
```
A `BigQuery` tool can be added as below:
```yaml
tools:
search-hotels-by-name:
kind: bigquery-sql
source: my-bigquery-source
description: Search for hotels based on name.
parameters:
- name: name
type: string
description: The name of the hotel.
```
---------
Co-authored-by: Wenxin Du <117315983+duwenxin99@users.noreply.github.com>
# Add Bigtable support
A `bigtable` source can be added as the following example
```
sources:
test-bigtable-source:
kind: "bigtable"
project: "sample-project"
instance: "sample-instance"
```
A `bigtable` tool can be added as below
```
tools:
get-test-tool-data:
kind: bigtable-sql
source: test-bigtable-source
description: Some description
statement: SELECT * FROM `test-table` WHERE address['state'] = @state;
parameters:
- name: state
type: string
description: Filter by state
```
---------
Co-authored-by: Yuan <45984206+Yuan325@users.noreply.github.com>
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Co-authored-by: Wenxin Du <117315983+duwenxin99@users.noreply.github.com>
User agent will be set as "genai-toolbox/" + versionString.
Instead of a panic, Toolbox will throw an error if the source fail to
retrieve `user agent`.
This will also be consistent with setting and getting logger from
context.
This only checks within `SourceConfig`, `ToolConfig`, and
`AuthSourceConfig`.
Error when an unknown field is provided:
`2025-01-27T22:43:46.988401-08:00 ERROR "unable to parse tool file at
\"tools.yaml\": unable to parse as \"cloud-sql-postgres\": [2:1] unknown
field \"extra\"\n 1 | database: test_database\n> 2 | extra: here\n ^\n 3
| instance: toolbox-cloudsql\n 4 | kind: cloud-sql-postgres\n 5 |
password: postgres\n 6 | "`
Error when a required field is not provided:
`2025-01-27T17:49:47.584846-08:00 ERROR "unable to parse tool file at
\"tools.yaml\": validation failed: Key: 'Config.Region' Error:Field
validation for 'Region' failed on the 'required' tag"`
---------
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Add user agent to cloud databases that provides us anonymized data
request count, number of users, number of projects, and other
environment settings.
User agent is using the format: `genai-toolbox/$version+metadata`
Add integration test for CloudSQL for MySQL.
Added other integration tests' tag into `.golangci.yaml`, and fixing
lint errors.
Moved getCloudSQLDialOpts to `common_test.go` since it is used across
all three cloud sql integration tests.
1. `sql/database` provides a `Scan()`interface to scan query results
into typed variables. Therefore we have to create a slice of typed
variables (types retrieved from rows.ColumnTypes()) to pass them into
`Scan()`. Using []byte works but makes the printing result different
from other tools (e.g [1] instead of %!s(int32=1)]
2. MS SQL supports both named (e.g @name) and positional args (e.g @p2),
so we have to check if the name is contained in the original statement
before passing them into `db.Query()` as either named arg or as values.
Add CloudSQL for MySQL source and tool.
CloudSQLMySQL source is initialize with the following config:
```
sources:
my-cloudsqlmysql-source:
kind: cloud-sql-mysql
project: my-project-name
region: my-region
instance: my-instance-name
user: my_user
password: my_pass
database: my_db
# ipType: public # The default dialect is public.
```
MySQL tool is initialize with the following config.
```
tools:
test_tool:
kind: mysql
source: my-cloudsqlmysql-source
description: >
Testing tool.
statement: "SELECT 1;"
```
- configure neo4j source with url, username, password, database
- configure neo4j tools with cypher statement and paramters
- tests based on the postgres tests
- neo4j.yaml for integration tests
---------
Co-authored-by: duwenxin <duwenxin@google.com>
Add Spanner source and tool.
Spanner source is initialize with the following config:
```
sources:
my-spanner-source:
kind: spanner
project: my-project-name
instance: my-instance-name
database: my_db
# dialect: postgresql # The default dialect is google_standard_sql.
```
Spanner tool (with gsql dialect) is initialize with the following
config.
```
tools:
get_flight_by_id:
kind: spanner
source: my-cloud-sql-source
description: >
Use this tool to list all airports matching search criteria. Takes
at least one of country, city, name, or all and returns all matching
airports. The agent can decide to return the results directly to
the user.
statement: "SELECT * FROM flights WHERE id = @id"
parameters:
- name: id
type: int
description: 'id' represents the unique ID for each flight.
```
Spanner tool (with postgresql dialect) is initialize with the following
config.
```
tools:
get_flight_by_id:
kind: spanner
source: my-cloud-sql-source
description: >
Use this tool to list all airports matching search criteria. Takes
at least one of country, city, name, or all and returns all matching
airports. The agent can decide to return the results directly to
the user.
statement: "SELECT * FROM flights WHERE id = $1"
parameters:
- name: id
type: int
description: 'id' represents the unique ID for each flight.
```
Note: the only difference in config for both dialects is the sql
statement.
---------
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Allow user to set if their database uses private or public ip. The
reason we add this is because the dialer require different
initialization with private and public ip.
By default, toolbox will use public ip.
Allow user to set if their database uses private or public ip. The
reason we add this is because the dialer require different
initialization with private and public ip.
By default, toolbox will use public ip.
---------
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Fixes a bug introduced in #43 where postgres-tool didn't think any of
the sources were valid.
The root cause of the bug was inconsistent usage of pointer/value in the
receiver functions. The fix updates everything to consistently use a
pointer receiver function.
This PR introduces the following breaking change: The
`alloydb-pg-generic`, `cloud-sql-pg-generic`, and
`postgres-generic-tool` have been replaced by the `postgres-sql` tool,
which works with all 3 Postgres sources.
If you were using of the the previous tools, you will need to update it
as follows:
```diff
example_tool:
- kind: cloud-sql-pg-generic
+ kind: postgres-sql
source: my-cloud-sql-pg-instance
description: some description
statement: |
SELECT * FROM SQL_STATEMENT;
parameters:
- name: country
type: string
description: some description
```
I'm proposing this change for the following reasons:
1. It provides greater flexibility between postgres-compatible sources
-- you can change between "postgres" and "alloydb-postgres" without
issue
2. The name "postgres-sql" is more clear that "postgres-generic" -- it
indicates it's a tool that runs SQL on the source
3. It's easier for us to maintain feature compatibility across a single
"postgres-sql" tool
