Move source-related queries from `Invoke()` function into Source.
This is an effort to generalizing tools to work with any Source that
implements a specific interface. This will provide a better segregation
of the roles for Tools vs Source.
Tool's role will be limited to the following:
* Resolve any pre-implementation steps or parameters (e.g. template
parameters)
* Retrieving Source
* Calling the source's implementation
Update bigquery test to include column order for SELECT statement.
Update mindsdb tests to drop table before creating. The whole
integration test pause when there's a failure from any one of
integration tests. If the test pause after `CREATE` and before `DROP`,
the int test will fail when running it again.
## Description
> Should include a concise description of the changes (bug or feature),
it's
> impact, along with a summary of the solution
## PR Checklist
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [ ] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [ ] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️ Fixes #<issue_number_goes_here>
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
## Description
---
This introduces a breaking change. The bigquery-get-dataset-info tool
will now enforce the allowed datasets setting from its BigQuery source
configuration. Previously, this setting had no effect on the tool.
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [ ] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [ ] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️ Part of https://github.com/googleapis/genai-toolbox/issues/873
This introduces a breaking change. The bigquery-get-table-info tool will
now enforce the allowed datasets setting from its BigQuery source
configuration. Previously, this setting had no effect on the tool.
Part of https://github.com/googleapis/genai-toolbox/issues/873
---------
Co-authored-by: Nikunj Badjatya <nikunj.badjatya@harness.io>
Summary
Adds an optional write_mode configuration to the BigQuery source,
enhancing security by controlling the types of SQL statements that can
be executed to prevent unauthorized data modification.
Key Changes
Added writeMode Configuration: A new write_mode field is added to the
BigQuery source, supporting three modes:
allowed (Default): Permits all SQL statements.
blocked: Allows only SELECT queries.
protected: Enables session-based execution, restricting write operations
(like CREATE TABLE) to the session's temporary dataset, thus protecting
permanent datasets. Note: at the moment, this won't work with
useClientOAuth, will fix this in the future.
These restrictions primarily apply to the bigquery-execute-sql tool and
the session may be used in other tools.
## Description
This introduces a breaking change. The bigquery-list-dataset-ids tool
will now enforce the allowed datasets setting from its BigQuery source
configuration. Previously, this setting had no effect on the tool.
The tool's behavior regarding this parameter is influenced by the
`allowedDatasets` restriction on the `bigquery` source:
- **Without `allowedDatasets` restriction:** The tool can list datasets
from any project specified by the `project` parameter.
- **With `allowedDatasets` restriction:** The tool directly returns the
pre-configured list of dataset IDs from the source, and the `project`
parameter is ignored.
---
> Should include a concise description of the changes (bug or feature),
it's
> impact, along with a summary of the solution
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [ ] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [ ] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️ Part of https://github.com/googleapis/genai-toolbox/issues/873
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
## Description
This introduces a breaking change. The bigquery-execute-sql tool will
now enforce the allowed datasets setting from its BigQuery source
configuration. Previously, this setting had no effect on the tool.
---
> Should include a concise description of the changes (bug or feature),
it's
> impact, along with a summary of the solution
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [ ] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [ ] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️ Fixes https://github.com/googleapis/genai-toolbox/issues/873
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
## Description
---
This introduces a breaking change. The bigquery-forecast tool will now
enforce the allowed datasets setting from its BigQuery source
configuration. Previously, this setting had no effect on the tool.
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [ ] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [ ] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️ Fixes #<issue_number_goes_here>
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
## Description
---
Add support to allowed datasets for conversational-analytics tool in
bigquery.
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [ ] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [ ] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️ Fixes #<issue_number_goes_here>
## Description
---
The tool can be considered as a subset of the dataplex_search_entries
tool. It automatically appends system=bigquery to all of the requests
and outputs BigQuery resources.
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [x] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [x] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️Fixes#1376
---------
Co-authored-by: Averi Kitsch <akitsch@google.com>
## Description
Added retries to bigquery-conversational-analytics api tests for timeout
error(25s in test) and service unavailable error(code 503). Other errors
will not retry.
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [ ] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [ ] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️ Fixes #<issue_number_goes_here>
## Description
---
- bigquery Source: The source configuration now supports a new
allowedDatasets field, which defines the list of datasets the tools are
allowed to access.
- bigquery-list-table-ids: Now verifies that the requested dataset is in
the allowed datasets list before listing its tables. An error is
returned if access is not permitted.
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [ ] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [ ] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/langchain-google-alloydb-pg-python/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️ Fixes #<issue_number_goes_here>
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
Previously we propagate tokens directly to the BQ API. But MCP inspector
adds a "Bearer" prefix to all authorization header. We will need to
parse the token accordingly to make it work.
Existing `/mcp` endpoint of Toolbox does not support auth (authorized
invocation and authenticated parameters). This PR add support for
Toolbox auth to the `/mcp` endpoint.
Added integration test for MCP with auth.
Note that Toolbox auth is **NOT** supported in stdio transport protocol,
invocations of tools with auth will result in error.
1. Add ask-data-insights tool based on conversational analytic API.
2. Add tokenSource for ask-data-insights tool, it uses access token
instead of client or restService.
3. Add a max row count to source, currently fixed to 50 and used only
for ask-data-insights tool. Later we may make it available for user to
make change and apply to bigquery-execute-sql and bigquery-sql to avoid
return too many data by accident.
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
Co-authored-by: Averi Kitsch <akitsch@google.com>
Add options for `RunToolInvokeTest()`, `RunMCPToolCallMethod()`,
`RunExecuteSqlToolInvokeTest()`.
When I was trying to update integration test for Firebird. They have a
different syntax from the usual `SELECT 1` that the other DBs are using.
If we want to update the `SELECT 1` statement in the test cases, we will
have to add a new argument and update all the other integration tests.
With `Options`, we can just add customization without having the need to
update all other integration tests. This allow the integration tests to
be more flexible. With this, we can also remove the
`GetNonSpannerWant()` since those can be set as default values for the
`Options`.
Add optional `dry_run` parameter to bigquery-execute-sql, which defaults
to `false`.
When the `dry_run` parameter is set to `true`, the tool returns the
metadata from the dry run instead of executing the query.
Fixes#703
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
This is to make bigquery-sql consistent with bigquery-execute-sql. May
not be necessary to have.
- Added a dry run step to identify the query type (e.g., SELECT, DML),
which allows the tool to correctly handle the query's output.
- The recommended high-level client, cloud.google.com/go/bigquery, does
not expose the statement type from a dry run. To circumvent this
limitation, the low-level BigQuery REST API client
(google.golang.org/api/bigquery/v2) was added to gain access to these
necessary details.
---------
Co-authored-by: Averi Kitsch <akitsch@google.com>
- Added a dry run step to identify the query type (e.g., SELECT, DML),
which allows the tool to correctly handle the query's output.
- The recommended high-level client, cloud.google.com/go/bigquery, does
not expose the statement type from a dry run. To circumvent this
limitation, the low-level BigQuery REST API client
(google.golang.org/api/bigquery/v2) was added to gain access to these
necessary details.
fixes: #915
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
Update `tool.Invoke()` to return type `any` instead of `[]any`.
Toolbox return a map with the `results` key, and the SDK reads the
string from the key. So this won't break existing SDK implementation.
Fixes#870
- Spanner: convert array to typed before querying, as Spanner does not
accept non-typed array []any
- BigQuery: fix https://github.com/googleapis/genai-toolbox/issues/793
- Bigtable: add the required `ElemType` for array-type params
- Redis/Valkey: change indexing to append to avoid extra spaces
Add integration tests for array parameters, skipped for the sources not
supporting arrays:
- SQLite
- Cloud SQL MSSQL
- Cloud SQL MySQL
- MSSQL
- MySQL
Allow Toolbox server to automatically update when users modify their
tool configuration file(s), instead of requiring a restart.
This feature is automatically enabled, but can be turned off with the
flag `--disable-reload`.
Optional projectID parameter enables dynamic, cross-project resource
access in BigQuery tools.
This allows a single tool configuration to target different projects at
runtime, rather than being fixed to the project in its source
configuration.
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
Co-authored-by: Wenxin Du <117315983+duwenxin99@users.noreply.github.com>
Add `templateParameters` to support non-filter parameters and DDL
statements.
Added a new argument `ignoreDdl` at integration test. Admin client is
needed to execute ddl statement in spanner. Toolbox does not use admin
client.
Part of #535
gosimple had been deprecated in favor of staticcheck:
https://github.com/golangci/golangci-lint/issues/357
Other requirements are all migrated.
`std-error-handling` exclusions is included because without that, it
will ask to check all error returns from (`Close()`, or `os.Setenv`s, or
`fmt.Fprint`s...
Remove cleaning `\\` or `\` in `got` and `want` strings of test cases.
This is to prevent future the need to manually clean strings before
comparing. This also allows us to be aware of the amount of backspace to
prevent accidental escaping in src code.
Update integration test that is affected. This includes moving the
`want` of test cases for `my-param-tool` invocation to each sources
since spanner have a different format (return id as strings instead of
int).
