This PR introduces a significant update to the Toolbox configuration
file format, which is one of the primary **breaking changes** required
for the implementation of the Advanced Control Plane.
# Summary of Changes
The configuration schema has been updated to enforce resource isolation
and facilitate atomic, incremental updates.
* Resource Isolation: Resource definitions are now separated into
individual blocks, using a distinct structure for each resource type
(Source, Tool, Toolset, etc.). This improves readability, management,
and auditing of configuration files.
* Field Name Modification: Internal field names have been modified to
align with declarative methodologies. Specifically, the configuration
now separates kind (general resource type, e.g., Source) from type
(specific implementation, e.g., Postgres).
# User Impact
Existing tools.yaml configuration files are now in an outdated format.
Users must eventually update their files to the new YAML format.
# Mitigation & Compatibility
Backward compatibility is maintained during this transition to ensure no
immediate user action is required for existing files.
* Immediate Backward Compatibility: The source code includes a
pre-processing layer that automatically detects outdated configuration
files (v1 format) and converts them to the new v2 format under the hood.
* [COMING SOON] Migration Support: The new toolbox migrate subcommand
will be introduced to allow users to automatically convert their old
configuration files to the latest format.
# Example
Example for config file v2:
```
kind: sources
name: my-pg-instance
type: cloud-sql-postgres
project: my-project
region: my-region
instance: my-instance
database: my_db
user: my_user
password: my_pass
---
kind: authServices
name: my-google-auth
type: google
clientId: testing-id
---
kind: tools
name: example_tool
type: postgres-sql
source: my-pg-instance
description: some description
statement: SELECT * FROM SQL_STATEMENT;
parameters:
- name: country
type: string
description: some description
---
kind: tools
name: example_tool_2
type: postgres-sql
source: my-pg-instance
description: returning the number one
statement: SELECT 1;
---
kind: toolsets
name: example_toolset
tools:
- example_tool
```
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Averi Kitsch <akitsch@google.com>
Update bigquery test to include column order for SELECT statement.
Update mindsdb tests to drop table before creating. The whole
integration test pause when there's a failure from any one of
integration tests. If the test pause after `CREATE` and before `DROP`,
the int test will fail when running it again.
🚀 Add MindsDB Integration: Expand Toolbox to Hundreds of Datasources
Overview
This PR introduces comprehensive MindsDB integration to the Google GenAI
Toolbox, enabling SQL queries across hundreds of datasources through a
unified interface. MindsDB is the most widely adopted AI federated
database that automatically translates MySQL queries into REST APIs,
GraphQL, and native protocols.
🎯 Key Value for Google GenAI Toolbox Ecosystem
1. Massive Datasource Expansion
Before: Toolbox limited to ~15 traditional databases
After: Access to hundreds of datasources including Salesforce, Jira,
GitHub, MongoDB, Gmail, Slack, and more
Impact: Dramatically expands the toolbox's reach and utility for
enterprise users
2. Cross-Datasource Analytics
New Capability: Perform joins and analytics across different datasources
seamlessly
Example: Join Salesforce opportunities with GitHub activity to correlate
sales with development activity
Value: Enables comprehensive data analysis that was previously
impossible
3. API Abstraction Layer
Innovation: Write standard SQL queries that automatically translate to
any API
Benefit: Developers can query REST APIs, GraphQL, and native protocols
using familiar SQL syntax
Impact: Reduces complexity and learning curve for accessing diverse
datasources
4. ML Model Integration
Enhanced Capability: Use ML models as virtual tables for real-time
predictions
Example: Query customer churn predictions directly through SQL
Value: Brings AI/ML capabilities into the standard SQL workflow
🔧 Technical Implementation
Source Layer
✅ New MindsDB source implementation using MySQL wire protocol
✅ Comprehensive test coverage with integration tests
✅ Updated existing MySQL tools to support MindsDB sources
✅ Created dedicated MindsDB tools for enhanced functionality
Tools Layer
✅ mindsdb-execute-sql: Direct SQL execution across federated datasources
✅ mindsdb-sql: Parameterized SQL queries with template support
✅ Backward compatibility with existing MySQL tools
Documentation & Configuration
✅ Comprehensive documentation with real-world examples
✅ Prebuilt configuration for easy setup
✅ Updated CLI help text and command-line options
📊 Supported Datasources
Business Applications
Salesforce (leads, opportunities, accounts)
Jira (issues, projects, workflows)
GitHub (repositories, commits, PRs)
Slack (channels, messages, teams)
HubSpot (contacts, companies, deals)
Databases & Storage
MongoDB (NoSQL collections as structured tables)
Redis (key-value stores)
Elasticsearch (search and analytics)
S3, filesystems, etc (file storage)
Communication & Email
Gmail/Outlook (emails, attachments)
Microsoft Teams (communications, files)
Discord (server data, messages)
🎯 Example Use Cases
Cross-Datasource Analytics
-- Join Salesforce opportunities with GitHub activity
```
SELECT
s.opportunity_name,
s.amount,
g.repository_name,
COUNT(g.commits) as commit_count
FROM salesforce.opportunities s
JOIN github.repositories g ON s.account_id = g.owner_id
WHERE s.stage = 'Closed Won';
```
Email & Communication Analysis
```
-- Analyze email patterns with Slack activity
SELECT
e.sender,
e.subject,
s.channel_name,
COUNT(s.messages) as message_count
FROM gmail.emails e
JOIN slack.messages s ON e.sender = s.user_name
WHERE e.date >= '2024-01-01';
```
🚀 Benefits for Google GenAI Toolbox
Enterprise Adoption: Enables access to enterprise datasources
(Salesforce, Jira, etc.)
Developer Productivity: Familiar SQL interface for any datasource
AI/ML Integration: Seamless integration of ML models into SQL workflows
Scalability: Single interface for hundreds of datasources
Competitive Advantage: Unique federated database capabilities in the
toolbox ecosystem
📈 Impact Metrics
Datasource Coverage: +1000% increase in supported datasources
API Abstraction: Eliminates need to learn individual API syntaxes
Cross-Platform Analytics: Enables previously impossible data
correlations
ML Integration: Brings AI capabilities into standard SQL workflows
🔗 Resources
MindsDB Documentation
MindsDB GitHub
Updated Toolbox Documentation
✅ Testing
✅ Unit tests for MindsDB source implementation
✅ Integration tests with real datasource examples
✅ Backward compatibility with existing MySQL tools
✅ Documentation examples tested and verified
This integration transforms the Google GenAI Toolbox from a traditional
database tool into a comprehensive federated data platform, enabling
users to query and analyze data across their entire technology stack
through a unified SQL interface.
---------
Co-authored-by: duwenxin <duwenxin@google.com>
Co-authored-by: setohe0909 <setohe.09@gmail.com>
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Co-authored-by: Wenxin Du <117315983+duwenxin99@users.noreply.github.com>
Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
