## Description
This PR adds cloud logging admin source, tools, integration test and
docs.
1. Source is implemented in a manner consistent with the BigQuery
source. Supports ADC, OAuth and impersonate Service Account.
2. Total of 3 tools have been implemented
- `cloud-logging-admin-list-log-names`
- `cloud-logging-admin-list-resource-types`
- `cloud-logging-admin-query-logs`
3. docs added for resource and tools.
4. Supporting integration test is added with updated ci
Note for reviewers:
1. Integration test runs on cloud, will require `LOGADMIN_PROJECT` env
variable, the test creates logs in the project using the `logging`
client and then verifies working of the tools using the `logadmin`
client.
2. Moved `cache.go` from the BigQuery source to `sources/cache.go` due
to shared utility.
Regarding Tools:
1. `cloud-logging-admin-list-log-names` uses `client.Logs()` instead of
`client.Entries()`, as the latter is resource heavy and the tradeoff was
not being able to apply any filters, tool has an optional parameter
`limit` which defaults to 200.
2. `cloud-logging-admin-list-resource-types` uses
`client.ResourceDescriptors(ctx)`, aim of the tool is to enable the
agent become aware of the the resources present and utilise this
information in writing filters.
3. `cloud-logging-admin-query-logs` tool enables search and read logs
from Google Cloud.
Parameters:
`filter` (optional): A text string to search for specific logs.
`newestFirst` (optional): A simple true/false switch for ordering.
`startTime ` (optional): The start date and time to search from (e.g.,
2025-12-09T00:00:00Z). Defaults to 30 days ago if not set.
`endTime` (optional): The end date and time to search up to. Defaults to
"now".
`verbose` (optional): If set to true, Shows all available details for
each log entry else shows only the main info (timestamp, message,
severity).
`limit` (optional): The maximum number of log entries to return (default
is 200).
Looking forward to the feedback here, as `verbose` is simply implemented
to save context tokens, any alternative suggestion here is also
welcomed.
Simple tools.yaml
```
sources:
my-logging-admin:
kind: cloud-logging-admin
project: <Add project>
useClientOAuth: false
tools:
list_resource_types:
kind: cloud-logging-admin-list-resource-types
source: my-logging-admin
description: List the types of resource that are indexed by Cloud Logging.
list_log_names:
kind: cloud-logging-admin-list-log-names
source: my-logging-admin
description: List log names matching a filter criteria.
query_logs:
kind: cloud-logging-admin-query-logs
source: my-logging-admin
description: query logs
```
## PR Checklist
- [x] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [x] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️Fixes#1772
@anubhav756 @averikitsch Thanks for the guidance and feedback on the
implementation plan.
---------
Co-authored-by: Yuan Teoh <yuanteoh@google.com>
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
This PR introduces a significant update to the Toolbox configuration
file format, which is one of the primary **breaking changes** required
for the implementation of the Advanced Control Plane.
# Summary of Changes
The configuration schema has been updated to enforce resource isolation
and facilitate atomic, incremental updates.
* Resource Isolation: Resource definitions are now separated into
individual blocks, using a distinct structure for each resource type
(Source, Tool, Toolset, etc.). This improves readability, management,
and auditing of configuration files.
* Field Name Modification: Internal field names have been modified to
align with declarative methodologies. Specifically, the configuration
now separates kind (general resource type, e.g., Source) from type
(specific implementation, e.g., Postgres).
# User Impact
Existing tools.yaml configuration files are now in an outdated format.
Users must eventually update their files to the new YAML format.
# Mitigation & Compatibility
Backward compatibility is maintained during this transition to ensure no
immediate user action is required for existing files.
* Immediate Backward Compatibility: The source code includes a
pre-processing layer that automatically detects outdated configuration
files (v1 format) and converts them to the new v2 format under the hood.
* [COMING SOON] Migration Support: The new toolbox migrate subcommand
will be introduced to allow users to automatically convert their old
configuration files to the latest format.
# Example
Example for config file v2:
```
kind: sources
name: my-pg-instance
type: cloud-sql-postgres
project: my-project
region: my-region
instance: my-instance
database: my_db
user: my_user
password: my_pass
---
kind: authServices
name: my-google-auth
type: google
clientId: testing-id
---
kind: tools
name: example_tool
type: postgres-sql
source: my-pg-instance
description: some description
statement: SELECT * FROM SQL_STATEMENT;
parameters:
- name: country
type: string
description: some description
---
kind: tools
name: example_tool_2
type: postgres-sql
source: my-pg-instance
description: returning the number one
statement: SELECT 1;
---
kind: toolsets
name: example_toolset
tools:
- example_tool
```
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Averi Kitsch <akitsch@google.com>
This change allows the agent developer to control the maxium number of
rows returned from tools running BigQuery SQL query. Using this feature
the agent developer could limit how large output is presented to LLM in
an agentic user journey.
## Description
> Should include a concise description of the changes (bug or feature),
it's
> impact, along with a summary of the solution
## PR Checklist
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [x] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [ ] Make sure to open an issue
https://github.com/googleapis/genai-toolbox/issues/2261
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️Fixes#2261 2261
## Description
This change addresses the ask where the user may want to use custom
scopes. For instance, the default scope (bigquery) falls short from
running sql that utilizes integration with other google products, such
as Drive, Vertex AI, Cloud Run etc. With this change the user would be
able to configure custom scopes depending on their use case.
The custom scopes can be configured in the tools.yaml file, e.g.:
```yaml
sources:
bigquery-source:
kind: "bigquery"
project: ${BIGQUERY_PROJECT}
location: ${BIGQUERY_LOCATION:}
useClientOAuth: ${BIGQUERY_USE_CLIENT_OAUTH:false}
scopes:
- "https://www.googleapis.com/auth/bigquery"
- "https://www.googleapis.com/auth/drive"
```
and if the [bigquery prebuilt
config](https://github.com/googleapis/genai-toolbox/blob/main/internal/prebuiltconfigs/tools/bigquery.yaml)
is being used, then it can be set in the environment variable as well:
```shell
$ export BIGQUERY_SCOPES="https://www.googleapis.com/auth/bigquery,https://www.googleapis.com/auth/drive"
```
## PR Checklist
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [ ] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [x] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️Fixes#1942
Move source-related queries from `Invoke()` function into Source.
This is an effort to generalizing tools to work with any Source that
implements a specific interface. This will provide a better segregation
of the roles for Tools vs Source.
Tool's role will be limited to the following:
* Resolve any pre-implementation steps or parameters (e.g. template
parameters)
* Retrieving Source
* Calling the source's implementation
To keep a persistent backend storage for configuration, we will have to
keep a single source of truth. This involves supporting bi-directional
conversion between Config and Source.
This PR make the following changes:
* Embed Config in Source
* Add `ToConfig()` to extract Config from Source.
Add client cache and automatic cache cleanup.
The cache is managed by a map with OAuth access token as the keys.
Upon user tool invocation, get client from existing cache or create a
new one.
## Description
This change adds service account impersonation support to Bigquery.
Users can now optionally supply an `impersonateServiceAccount` field in
their `bigquery-source` config to enable impersonation.
---
> Should include a concise description of the changes (bug or feature),
it's
> impact, along with a summary of the solution
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [x] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [x] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)
- [x] Make sure to add `!` if this involve a breaking change
🛠️Fixes#906
## Description
---
allowed datasets default project id is from client, which may not be
available if useClientOAuth=True, changed to use r.Project instead.
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [ ] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [ ] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️ Fixes #<issue_number_goes_here>
Summary
Adds an optional write_mode configuration to the BigQuery source,
enhancing security by controlling the types of SQL statements that can
be executed to prevent unauthorized data modification.
Key Changes
Added writeMode Configuration: A new write_mode field is added to the
BigQuery source, supporting three modes:
allowed (Default): Permits all SQL statements.
blocked: Allows only SELECT queries.
protected: Enables session-based execution, restricting write operations
(like CREATE TABLE) to the session's temporary dataset, thus protecting
permanent datasets. Note: at the moment, this won't work with
useClientOAuth, will fix this in the future.
These restrictions primarily apply to the bigquery-execute-sql tool and
the session may be used in other tools.
## Problem
Fixes#1378 - BigQuery Conversational Analytics tool fails with
`ACCESS_TOKEN_SCOPE_INSUFFICIENT` error in Cloud Run environments while
other BigQuery tools work fine.
## Root Cause
The conversational analytics tool was using `BigQueryTokenSource()` with
limited `bigqueryapi.Scope`, but the Gemini Data Analytics API
(`geminidataanalytics.googleapis.com`) requires broader `cloud-platform`
scope.
## Solution
- Replace ADC token acquisition to use `google.DefaultTokenSource` with
`cloud-platform` scope
- Remove dependency on limited BigQuery scope from source
- Maintain compatibility with existing OAuth flow
## Testing
- ✅ **Local testing confirmed**: Tool now works perfectly
- ✅ **Test script**:
https://github.com/johanesalxd/bq-agent-app/blob/main/setup/mcp_toolbox_ca_issue/test_detailed_error.py
- ✅ **Successful response**: Returns proper schema information and
conversational answers
- ✅ **All BigQuery tool tests pass**: No regression in other tools
- ✅ **Build successful**: No compilation errors
## Impact
- **Fixes**: Cloud Run deployment authentication errors
- **Maintains**: Local development functionality
- **Preserves**: All existing BigQuery tool functionality
---------
Co-authored-by: Huan Chen <142538604+Genesis929@users.noreply.github.com>
## Description
---
The tool can be considered as a subset of the dataplex_search_entries
tool. It automatically appends system=bigquery to all of the requests
and outputs BigQuery resources.
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [x] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [x] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/genai-toolbox/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️Fixes#1376
---------
Co-authored-by: Averi Kitsch <akitsch@google.com>
## Description
---
- bigquery Source: The source configuration now supports a new
allowedDatasets field, which defines the list of datasets the tools are
allowed to access.
- bigquery-list-table-ids: Now verifies that the requested dataset is in
the allowed datasets list before listing its tables. An error is
returned if access is not permitted.
## PR Checklist
---
> Thank you for opening a Pull Request! Before submitting your PR, there
are a
> few things you can do to make sure it goes smoothly:
- [ ] Make sure you reviewed
[CONTRIBUTING.md](https://github.com/googleapis/genai-toolbox/blob/main/CONTRIBUTING.md)
- [ ] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/langchain-google-alloydb-pg-python/issues/new/choose)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)
- [ ] Make sure to add `!` if this involve a breaking change
🛠️ Fixes #<issue_number_goes_here>
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
Previously we propagate tokens directly to the BQ API. But MCP inspector
adds a "Bearer" prefix to all authorization header. We will need to
parse the token accordingly to make it work.
1. Add ask-data-insights tool based on conversational analytic API.
2. Add tokenSource for ask-data-insights tool, it uses access token
instead of client or restService.
3. Add a max row count to source, currently fixed to 50 and used only
for ask-data-insights tool. Later we may make it available for user to
make change and apply to bigquery-execute-sql and bigquery-sql to avoid
return too many data by accident.
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
Co-authored-by: Averi Kitsch <akitsch@google.com>
- Added a dry run step to identify the query type (e.g., SELECT, DML),
which allows the tool to correctly handle the query's output.
- The recommended high-level client, cloud.google.com/go/bigquery, does
not expose the statement type from a dry run. To circumvent this
limitation, the low-level BigQuery REST API client
(google.golang.org/api/bigquery/v2) was added to gain access to these
necessary details.
fixes: #915
---------
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
This commit refactors the source configuration and loading mechanism to
use a dynamic registration pattern. Each source package now registers
itself with a central registry via its init() function.
The server configuration code uses this registry to decode and
initialize sources, decoupling it from specific source implementations
and simplifying the addition of new sources.
Key changes:
- Introduced `sources.Register()` and `newConfig()` constructor in each
source package.
- Moved source package imports to `cmd/root.go` as blank imports to
trigger `init()` functions for self-registration.
- Removed direct imports of specific source packages from
`internal/server/config.go`.
- Renamed `SourceKind` constants to `Kind` within each source package.
- Updated tests to use the new `Kind` constants and reflect registration
changes.
---------
Co-authored-by: Yuan Teoh <yuanteoh@google.com>
A `BigQuery` source can be added as the following example:
```yaml
sources:
my-bigquery-source:
kind: bigquery
project: bigframes-dev
location: us # This field is optional
```
A `BigQuery` tool can be added as below:
```yaml
tools:
search-hotels-by-name:
kind: bigquery-sql
source: my-bigquery-source
description: Search for hotels based on name.
parameters:
- name: name
type: string
description: The name of the hotel.
```
---------
Co-authored-by: Wenxin Du <117315983+duwenxin99@users.noreply.github.com>
