Fixes#861
This PR adds support for DuckDB which is a free, open-source, embedded,
in-process, relational database management system (RDBMS) designed for
analytical processing (OLAP)
---------
Co-authored-by: Averi Kitsch <akitsch@google.com>
This pull request introduces a new tool, `neo4j-schema`, for extracting
and processing comprehensive schema information from Neo4j databases. It
includes updates to the documentation, implementation of caching
mechanisms, helper utilities for schema transformation, and
corresponding unit tests. The most important changes are grouped by
theme below:
### Tool Integration:
- **`cmd/root.go`**: Added import for the new `neo4j-schema` tool to
integrate it into the application.
### Documentation:
- **`docs/en/resources/tools/neo4j/neo4j-schema.md`**: Added detailed
documentation for the `neo4j-schema` tool, describing its functionality,
caching behavior, and usage examples.
### Caching Implementation:
- **`internal/tools/neo4j/neo4jschema/cache/cache.go`**: Implemented a
thread-safe, in-memory cache with expiration and optional janitor for
cleaning expired items.
### Unit Tests:
- **`internal/tools/neo4j/neo4jschema/cache/cache_test.go`**: Added
comprehensive tests for the caching system, including functionality for
setting, retrieving, expiration, janitor cleanup, and concurrent access.
### Helper Utilities:
- **`internal/tools/neo4j/neo4jschema/helpers/helpers.go`**: Added
utility functions for processing schema data, including support for APOC
and native Cypher queries, and converting raw query results into
structured formats.
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
- Users have the preference to choose their clients. Below example is
using Gemini CLI.
- Users can use the pre-built Dataplex tools by creating a settings.json
file under .gemini directory. The contents of settings.json would be as
follows:-
```
{
"mcpServers": {
"dataplex": {
"command": "./toolbox",
"args": ["--prebuilt","dataplex","--stdio"],
"env": {
"DATAPLEX_PROJECT": "test-project"
}
}
}
}
```
Fixes#831
---------
Co-authored-by: Wenxin Du <117315983+duwenxin99@users.noreply.github.com>
Co-authored-by: Mateusz Nowak <matnow@google.com>
Co-authored-by: Mateusz Nowak <kontakt@mateusznowak.pl>
Example:
```
alloydb-operations-get:
kind: wait-for-operation
source: alloydb-api-source
method: GET
path: /v1/projects/{{.projectId}}/locations/{{.locationId}}/operations/{{.operationId}}
description: "Makes API call to check whether operation is done or not. This tool is run first then wait for tool. if its still in create phase trigger it after 3 minutes. Print a message saying still not done. We will notify once its done."
pathParams:
- name: projectId
type: string
description: The dynamic path parameter
- name: locationId
type: string
description: The dynamic path parameter
default: us-central1
- name: operationId
type: string
description: Operation status check for previous task
```
This pull request introduces a new tool for executing arbitrary Cypher
queries against a Neo4j database (`neo4j-execute-cypher`) and implements
robust query classification functionality to distinguish between read
and write operations. The changes include updates to documentation, the
addition of a query classifier, and comprehensive test coverage for the
classifier.
### Addition of `neo4j-execute-cypher` tool:
- **Documentation**: Added a new markdown file `neo4j-execute-cypher.md`
that explains the tool's functionality, usage, and configuration
options, including the ability to enforce read-only mode for security.
- **Import statement**: Registered the new tool in the `cmd/root.go`
file to make it available in the toolbox.
### Query classification functionality:
- **Query classifier implementation**: Added `QueryClassifier` in
`classifier.go`, which classifies Cypher queries into read or write
operations based on keywords, procedures, and subquery analysis. It
supports handling edge cases like nested subqueries, multi-word
keywords, and invalid syntax.
- **Test coverage**: Created extensive tests in `classifier_test.go` to
validate the classifier's behavior across various query types, including
abuse cases, subqueries, and procedure calls. Tests ensure the
classifier is robust and does not panic on malformed queries.
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
**Package Renaming**: The `neo4j` package has been renamed to
`neo4jcypher` to better reflect its functionality. This change affects
the tool's implementation
(`internal/tools/neo4j/neo4jcypher/neo4jcypher.go`) and its associated
tests (`internal/tools/neo4j/neo4jcypher/neo4jcypher_test.go`).
Firestore is a NoSQL document database built for automatic scaling, high
performance, and ease of application development. It's a fully managed,
serverless database that supports mobile, web, and server development.
This change adds Firestore as a source in toolbox
`
tools:
wait_for_tool:
kind: wait-for
description: "Tool puts chat to sleep for given timeout example 3m,10s,
5m etc. For tasks such as cluster creation no need to wait. It takes
longer time such as 8m for instance creation. Use timeout value as
default if nothing is provided"
timeout: 10s
`
Output:
`
prernakakkar@prernakakkar:~/senseai/genai-toolbox$ curl -X POST -H
"Content-Type: application/json" -d '{"duration": "5m"}'
http://127.0.0.1:5000/api/tool/wa
it_for_tool/invoke
{"result":"[\"Wait for 5m0s completed successfully.\"]"}
`
Allow fsnotify `CREATE` and `RENAME` events to trigger dynamic reload,
instead of requiring `WRITE`.
This is due to some cases with specific editors/OS, (ex: vim on Mac),
where file writes can potentially occur without ever triggering a
`WRITE` operation on the watched tools file, but solely through creation
and renaming of .swp files.
Allow Toolbox server to automatically update when users modify their
tool configuration file(s), instead of requiring a restart.
This feature is automatically enabled, but can be turned off with the
flag `--disable-reload`.
# Add Multiple YAML Configuration File Support
## 🎯 Overview
This PR introduces support for loading and merging multiple YAML
configuration files in Toolbox, addressing the need for modular
configuration management in complex deployments.
## ✨ New Features
### 1. Multiple Files Support (`--tools-files`)
- **Usage**: `--tools-files=file1.yaml,file2.yaml,file3.yaml`
- Load and intelligently merge multiple YAML configuration files
- Comma-separated file paths for maximum flexibility
### 2. Directory Support (`--tools-folder`)
- **Usage**: `--tools-folder=config-directory`
- Automatically discover and load all `.yaml` and `.yml` files from a
directory
- Simplifies configuration management for organized deployments
### 3. Smart Merging Logic
- **Sources/AuthServices/Tools**: Later files override earlier files
with same names
- **Toolsets**: Tools from same-named toolsets are combined without
duplicates
- Preserves all existing functionality while enabling composition
## 🔒 Safety & Validation
- **Mutual Exclusivity**: Prevents simultaneous use of `--tools-file`,
`--tools-files`, `--tools-folder`, and `--prebuilt`
- **Clear Error Messages**: Descriptive validation errors guide users to
correct usage
- **Comprehensive Error Handling**: Proper handling of missing files,
directories, and parsing errors
- **Full Backward Compatibility**: Existing configurations continue to
work unchanged
## 🏗️ Implementation Details
### Core Functions Added
- `mergeToolsFiles()` - Smart merging with configurable override rules
- `loadAndMergeToolsFiles()` - Multi-file loading and processing
- `loadAndMergeToolsFolder()` - Directory scanning and batch loading
### Command Structure Updates
- New `tools_files []string` field for multiple file paths
- New `tools_folder string` field for directory path
- Enhanced validation logic in `run()` function
- Updated flag definitions with proper descriptions
## 📋 Use Cases
### Organizational Benefits
- **Modular Configuration**: Separate database, API, and auth
configurations
- **Team Collaboration**: Multiple developers can work on different
config files
- **Environment Management**: Easy configuration swapping for different
environments
- **Scalability**: Large configurations can be broken into manageable
chunks
### Example Usage Patterns
```bash
# Multiple specific files
./toolbox --tools-files=database.yaml,apis.yaml,auth.yaml
# Directory-based loading
./toolbox --tools-folder=./production-configs
# Error case (properly handled)
./toolbox --tools-file=single.yaml --tools-folder=configs
# ERROR: --tools-file, --tools-files, and --tools-folder flags cannot be used simultaneously
---------
Co-authored-by: Wenxin Du <117315983+duwenxin99@users.noreply.github.com>
Add `templateParameters` to support non-filter parameters and DDL
statements.
Added a new argument `ignoreDdl` at integration test. Admin client is
needed to execute ddl statement in spanner. Toolbox does not use admin
client.
Part of #535
Reduces complexity of the version string calculation and fixes the ci/cd
pipeline to pass the correct parameters.
---------
Co-authored-by: Averi Kitsch <akitsch@google.com>
Co-authored-by: Yuan <45984206+Yuan325@users.noreply.github.com>
Append "+prebuilt.<source>" to the user agent for tracking the prebuilt
tools usage. This is set before the server is initialized and the user
agent is set on the client.
Using the period to stay consistent with our other appended metadata
like "binary.linux"
This commit refactors the source configuration and loading mechanism to
use a dynamic registration pattern. Each source package now registers
itself with a central registry via its init() function.
The server configuration code uses this registry to decode and
initialize sources, decoupling it from specific source implementations
and simplifying the addition of new sources.
Key changes:
- Introduced `sources.Register()` and `newConfig()` constructor in each
source package.
- Moved source package imports to `cmd/root.go` as blank imports to
trigger `init()` functions for self-registration.
- Removed direct imports of specific source packages from
`internal/server/config.go`.
- Renamed `SourceKind` constants to `Kind` within each source package.
- Updated tests to use the new `Kind` constants and reflect registration
changes.
---------
Co-authored-by: Yuan Teoh <yuanteoh@google.com>
This PR refactors the tool configuration and loading mechanism to use a
dynamic registration pattern. Each tool package now registers itself
with a central registry, and the server configuration code uses this
registry to decode and initialize tools.
Key changes:
- Introduced tools.Register and tools.DecodeToolConfig for dynamic tool
handling.
- Removed direct imports of specific tool packages from
internal/server/config.go.
- Updated individual tool packages to include init() functions for
self-registration.
- Modified ToolKind constants to be local kind constants within each
tool package.
- Adjusted test files to reflect the changes in tool kind identifiers.
This change simplifies adding new tools and decouples the server
configuration from specific tool implementations.
---------
Co-authored-by: Yuan Teoh <yuanteoh@google.com>
Co-authored-by: Yuan <45984206+Yuan325@users.noreply.github.com>
gosimple had been deprecated in favor of staticcheck:
https://github.com/golangci/golangci-lint/issues/357
Other requirements are all migrated.
`std-error-handling` exclusions is included because without that, it
will ask to check all error returns from (`Close()`, or `os.Setenv`s, or
`fmt.Fprint`s...
Introduces a new --prebuilt <source_type> flag. This flag is mutually
exclusive with the existing --tools-file flag.
Added a new directory cmd/prebuiltconfigs/ to store the prebuilt
tools.yaml files (e.g., alloydb.yaml, postgres.yaml, etc.).
These YAML files are embedded into the Go binary using the //go:embed
directive.
---------
Co-authored-by: Averi Kitsch <akitsch@google.com>
Environment variable replacement is needed so that users don't have to
hardcode their secrets in configuring `tools.yaml`.
Both formats `$ENV_NAME` and `${ENV_NAME}` are standard ways to declare
an environment variable.
However, some database statement placeholders that are already using the
`$ENV_NAME` format.
Therefore, we only support env var declaration using `${ENV_NAME}` to
disambiguate it from other usages.
Fixes issue: https://github.com/googleapis/genai-toolbox/issues/431
Add `tools-file` flag and deprecate `tools_file` flag. This is not a
breaking change. The old `tools_file` flag is still usable.
User will see the following message when using `tools_file` flag:
```
Flag --tools_file has been deprecated, please use --tools-file instead
2025-04-03T10:09:12.803165-07:00 INFO "Initialized 2 sources."
```
Help command will reveal the new flag:
```
...
--telemetry-service-name string Sets the value of the service.name resource attribute for telemetry data. (default "toolbox")
--tools-file string File path specifying the tool configuration. (default "tools.yaml")
-v, --version version for toolbox
...
```
Fixes#383
