Fixes#861
This PR adds support for DuckDB which is a free, open-source, embedded,
in-process, relational database management system (RDBMS) designed for
analytical processing (OLAP)
---------
Co-authored-by: Averi Kitsch <akitsch@google.com>
The relative links to the Looker source documentation page weren't
resolving correctly on the documentation website.
I updated the links in all of the Looker tool markdown files to use a
different relative path that should be correctly interpreted by the Hugo
static site generator. I changed the link from `../sources/looker.md` to
`../../sources/looker/`.
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
This pull request introduces a new tool, `neo4j-schema`, for extracting
and processing comprehensive schema information from Neo4j databases. It
includes updates to the documentation, implementation of caching
mechanisms, helper utilities for schema transformation, and
corresponding unit tests. The most important changes are grouped by
theme below:
### Tool Integration:
- **`cmd/root.go`**: Added import for the new `neo4j-schema` tool to
integrate it into the application.
### Documentation:
- **`docs/en/resources/tools/neo4j/neo4j-schema.md`**: Added detailed
documentation for the `neo4j-schema` tool, describing its functionality,
caching behavior, and usage examples.
### Caching Implementation:
- **`internal/tools/neo4j/neo4jschema/cache/cache.go`**: Implemented a
thread-safe, in-memory cache with expiration and optional janitor for
cleaning expired items.
### Unit Tests:
- **`internal/tools/neo4j/neo4jschema/cache/cache_test.go`**: Added
comprehensive tests for the caching system, including functionality for
setting, retrieving, expiration, janitor cleanup, and concurrent access.
### Helper Utilities:
- **`internal/tools/neo4j/neo4jschema/helpers/helpers.go`**: Added
utility functions for processing schema data, including support for APOC
and native Cypher queries, and converting raw query results into
structured formats.
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
- Users have the preference to choose their clients. Below example is
using Gemini CLI.
- Users can use the pre-built Dataplex tools by creating a settings.json
file under .gemini directory. The contents of settings.json would be as
follows:-
```
{
"mcpServers": {
"dataplex": {
"command": "./toolbox",
"args": ["--prebuilt","dataplex","--stdio"],
"env": {
"DATAPLEX_PROJECT": "test-project"
}
}
}
}
```
Fixes#831
---------
Co-authored-by: Wenxin Du <117315983+duwenxin99@users.noreply.github.com>
Co-authored-by: Mateusz Nowak <matnow@google.com>
Co-authored-by: Mateusz Nowak <kontakt@mateusznowak.pl>
## 1. Why do we need to support the `encrypt` parameter?
MSSQL databases that `genai-toolbox` attempts to connect to may have
their encryption levels set differently. For example, a testing/demo
purpose MSSQL database may not require encryption at all. However,
`genai-toolbox` currently uses the default encryption parameter
(`encrypt=false`) to connect to this type of database and will throw an
error:
```
ERROR "toolbox failed to initialize: unable to initialize configs: unable to initialize source "my-mssql-source": unable to connect successfully: TLS Handshake failed: cannot read handshake packet: EOF"
```
> In this case, the encryption parameter should be set to
`encrypt=disable`.
## 2. Is this a necessary feature?
`genai-toolbox` uses the `github.com/microsoft/go-mssqldb` package as a
dependency to connect to MSSQL databases. According to the
[README](https://github.com/microsoft/go-mssqldb?tab=readme-ov-file#common-parameters)
of the `github.com/microsoft/go-mssqldb` package, `encrypt` is one of
the common parameters. Therefore, I believe supporting the `encrypt`
parameter in `genai-toolbox` is necessary.
## 3. How to replicate the error mentioned above?
### 3.1 Use this `docker-compose.yaml` file to start a demo MSSQL
instance
```
services:
demo-mssql-database:
image: mcr.microsoft.com/mssql/server:2017-CU1-ubuntu
ports:
- "20256:1433"
environment:
ACCEPT_EULA: "Y"
MSSQL_SA_PASSWORD: "hellopassword!"
restart: unless-stopped
healthcheck:
test: ["CMD", "/opt/mssql-tools/bin/sqlcmd", "-S", "localhost", "-U", "sa", "-P", "hellopassword!", "-Q", "SELECT 1"]
interval: 5s
retries: 10
demo-mssql-database-init:
image: mcr.microsoft.com/mssql/server:2017-CU1-ubuntu
network_mode: service:demo-mssql-database
command: >
/bin/bash -c "/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P hellopassword! -d master -Q 'CREATE DATABASE DemoDatabase;'"
depends_on:
demo-mssql-database:
condition: service_healthy
```
### 3.2 Use `genai-toolbox` to connect to the above demo MSSQL database
with this `tools.yaml` configuration file:
```
sources:
my-mssql-source:
kind: mssql
host: localhost
port: 20256
database: master
user: sa
password: 'hellopassword!'
```
### 3.3 We shall see the error:
```
ERROR "toolbox failed to initialize: unable to initialize configs: unable to initialize source "my-mssql-source": unable to connect successfully: TLS Handshake failed: cannot read handshake packet: EOF"
```
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
Example:
```
alloydb-operations-get:
kind: wait-for-operation
source: alloydb-api-source
method: GET
path: /v1/projects/{{.projectId}}/locations/{{.locationId}}/operations/{{.operationId}}
description: "Makes API call to check whether operation is done or not. This tool is run first then wait for tool. if its still in create phase trigger it after 3 minutes. Print a message saying still not done. We will notify once its done."
pathParams:
- name: projectId
type: string
description: The dynamic path parameter
- name: locationId
type: string
description: The dynamic path parameter
default: us-central1
- name: operationId
type: string
description: Operation status check for previous task
```
This pull request introduces a new tool for executing arbitrary Cypher
queries against a Neo4j database (`neo4j-execute-cypher`) and implements
robust query classification functionality to distinguish between read
and write operations. The changes include updates to documentation, the
addition of a query classifier, and comprehensive test coverage for the
classifier.
### Addition of `neo4j-execute-cypher` tool:
- **Documentation**: Added a new markdown file `neo4j-execute-cypher.md`
that explains the tool's functionality, usage, and configuration
options, including the ability to enforce read-only mode for security.
- **Import statement**: Registered the new tool in the `cmd/root.go`
file to make it available in the toolbox.
### Query classification functionality:
- **Query classifier implementation**: Added `QueryClassifier` in
`classifier.go`, which classifies Cypher queries into read or write
operations based on keywords, procedures, and subquery analysis. It
supports handling edge cases like nested subqueries, multi-word
keywords, and invalid syntax.
- **Test coverage**: Created extensive tests in `classifier_test.go` to
validate the classifier's behavior across various query types, including
abuse cases, subqueries, and procedure calls. Tests ensure the
classifier is robust and does not panic on malformed queries.
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
support both generic and typed map. Config example:
```
parameters:
- name: user_scores
type: map
description: A map of user IDs to their scores. All scores must be integers.
valueType: integer # This enforces the value type for all entries. Leave it blank for generic map
```
Represented as `Object` with `additionalProperties` in manifests.
Added a util function to convert json.Number (string type) to int/float
types to address the problem where int/float values are converted to
strings for the generic map.
Firestore is a NoSQL document database built for automatic scaling, high
performance, and ease of application development. It's a fully managed,
serverless database that supports mobile, web, and server development.
This change adds Firestore as a source in toolbox
`
tools:
wait_for_tool:
kind: wait-for
description: "Tool puts chat to sleep for given timeout example 3m,10s,
5m etc. For tasks such as cluster creation no need to wait. It takes
longer time such as 8m for instance creation. Use timeout value as
default if nothing is provided"
timeout: 10s
`
Output:
`
prernakakkar@prernakakkar:~/senseai/genai-toolbox$ curl -X POST -H
"Content-Type: application/json" -d '{"duration": "5m"}'
http://127.0.0.1:5000/api/tool/wa
it_for_tool/invoke
{"result":"[\"Wait for 5m0s completed successfully.\"]"}
`
feat(docs): Enhance Bigtable GoogleSQL DML clarity
**Description:**
This PR updates the `bigtable-sql` tool documentation to provide a
clearer understanding of GoogleSQL's DML capabilities within Bigtable.
**Changes Made:**
- Added a note indicating that Bigtable's GoogleSQL dialect only
supports `SELECT` DML operations.
- Explicitly states that `INSERT`, `UPDATE`, and `DELETE` DML statements
are not supported.
- Updated the external link to directly reference the "Use cases"
section in the official Google Cloud Bigtable GoogleSQL overview, where
this limitation is detailed.
**Reasoning:**
The previous documentation, while linking to the overview, did not
explicitly highlight the DML limitations. This clarification will
prevent potential confusion for users expecting full DML support,
ensuring the documentation accurately reflects Bigtable's capabilities
with GoogleSQL.
---------
Co-authored-by: Twisha Bansal <58483338+twishabansal@users.noreply.github.com>
## Summary
- Added configurable query timeout to MySQL source configuration
- Updated connection DSN to include readTimeout parameter
- Added documentation and example usage
- Added test coverage for queryTimeout configuration
## Test plan
- [x] Added unit test for queryTimeout configuration parsing
- [x] Updated documentation with queryTimeout field description
- [x] Verified timeout parameter is correctly added to DSN when
specified
## Caveat
When queryTimeout is exceeded, we get an obscure error message
([screenshot](https://github.com/user-attachments/assets/fd292f91-328d-4ebc-9a87-2d92e9887300)):
```
unable to execute query: invalid connection
```
This seems to be a problem with the mysql-go library:
https://stackoverflow.com/q/65253798/10720618
I tried to use
[MAX_EXECUTION_TIME](https://dev.mysql.com/doc/refman/8.0/en/optimizer-hints.html#optimizer-hints-execution-time)
but it didn't work as expected (my `sleep(MAX_EXECUTION_TIME+3)` query
finished successfully after MAX_EXECUTION_TIME milliseconds)
Any ideas on what can be done here? The error message is very
misleading. My goal with adding timeouts is to communicate to the LLM
when it has issued a slow query and force it to adjust (e.g. query
indexes and write a more optimized query) but this defeats the purpose.
---------
Co-authored-by: Wenxin Du <117315983+duwenxin99@users.noreply.github.com>
Optional projectID parameter enables dynamic, cross-project resource
access in BigQuery tools.
This allows a single tool configuration to target different projects at
runtime, rather than being fixed to the project in its source
configuration.
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
Co-authored-by: Wenxin Du <117315983+duwenxin99@users.noreply.github.com>
## Problem
Users attempting to filter results in a SQL query based on an array
parameter from a tool may intuitively write a `statement` using the `IN`
clause, like so:
```sql
SELECT * FROM flights WHERE preferred_airlines IN ($1);
```
When this query is executed with an array argument (e.g., `["Delta",
"United"]`), it fails with a cryptic error message from the database
driver:
```
Exception: error while invoking tool: unable to execute query: failed to encode args[0]: unable to encode []interface {}{"Delta", "United"} into text format for text (OID 25): cannot find encode plan
```
This error occurs because the driver does not automatically expand the
single `$1` placeholder into a list of values `('Delta', 'United')`.
Instead, it tries to encode the entire Go slice `[]interface{}` as a
single text value, which fails. This creates a point of friction, as the
correct syntax is not immediately obvious and can lead to user
frustration and debugging time.
## Solution
This PR updates our documentation and example usage to demonstrate the
correct SQL syntax for handling array parameters. The proper way to
check for a value's existence in an array parameter is by using
PostgreSQL's `ANY()` operator:
```sql
SELECT * FROM flights WHERE preferred_airlines = ANY($1);
```
When this syntax is used, the database driver correctly interprets the
Go slice passed as `$1` as a PostgreSQL array, and the query executes as
intended.
## Impact
Saves developers time they would otherwise spend troubleshooting a
non-obvious database driver behavior.
Add a `default` field to parameters, that enables users to specify a
default value.
e.g.
```
parameters:
- name: name
type: string
default: "some-default-value"
description: The name of the hotel.
```
if this parameter is invoked without specifying `name`, the parameter
would default to "some-default-value"
For parameter manifest, there will be an additional `Required` field.
The default `Required` field is true. If a `default` value is presented,
`Required: false`. Array parameter's item's `Required` field will
inherit the array's `Required` field.
Fixes#475
## Description
This PR enhances our documentation by integrating `toolbox-core` code
snippets into the quickstart examples on the docsite.
## Context
Previously, our client-facing documentation primarily showcased agent
implementations using third-party frameworks like LangChain, LlamaIndex,
and ADK. This created a documentation gap, as there were no baseline
examples demonstrating how to use our foundational `toolbox-core` SDK
directly. Users who wanted to build a custom integration without a
specific framework had no direct reference in the guides.
## Changes
This PR introduces a new "Core" tab to all relevant code examples within
the auth services and local quickstart (with BigQuery) documentations.
---------
Co-authored-by: Yuan <45984206+Yuan325@users.noreply.github.com>
This tool can be used across spanner sources.
`spanner-execute-sql` config is as below:
```
tools:
spanner_execute_sql_tool:
kind: "spanner-execute-sql"
source: my-spanner-source
description: Use this tool to execute sql.
```
The `spanner-execute-sql` tool takes one parameter. Example request as
follow:
```
curl -X POST -H "Content-Type: application/json" -d '{"sql": "SELECT 1"}' http://127.0.0.1:5000/api/tool/spanner_execute_sql_tool/invoke
```
---------
Co-authored-by: Yuan <45984206+Yuan325@users.noreply.github.com>
This tool can be used across mysql sources.
`mysql-execute-sql` config is as below:
```
tools:
mysql_execute_sql_tool:
kind: "mysql-execute-sql"
source: my-mysql-source
description: Use this tool to execute sql.
```
The `mysql-execute-sql` tool takes one parameter. Example request as
follow:
```
curl -X POST -H "Content-Type: application/json" -d '{"sql": "SELECT 1"}' http://127.0.0.1:5000/api/tool/mysql_execute_sql_tool/invoke
```
