End to end integration test for cloudsql postgres.
Include checks for one tool's get (manifest) and post (invoke) endpoint.
Integration tests are excluded from regular unit tests.
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from
0.28.0 to 0.31.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="b4f1988a35"><code>b4f1988</code></a>
ssh: make the public key cache a 1-entry FIFO cache</li>
<li><a
href="7042ebcbe0"><code>7042ebc</code></a>
openpgp/clearsign: just use rand.Reader in tests</li>
<li><a
href="3e90321ac7"><code>3e90321</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="8c4e668694"><code>8c4e668</code></a>
x509roots/fallback: update bundle</li>
<li><a
href="6018723c74"><code>6018723</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="71ed71b4fa"><code>71ed71b</code></a>
README: don't recommend go get</li>
<li><a
href="750a45fe5e"><code>750a45f</code></a>
sha3: add MarshalBinary, AppendBinary, and UnmarshalBinary</li>
<li><a
href="36b172546b"><code>36b1725</code></a>
sha3: avoid trailing permutation</li>
<li><a
href="80ea76eb17"><code>80ea76e</code></a>
sha3: fix padding for long cSHAKE parameters</li>
<li><a
href="c17aa50fbd"><code>c17aa50</code></a>
sha3: avoid buffer copy</li>
<li>Additional commits viewable in <a
href="https://github.com/golang/crypto/compare/v0.28.0...v0.31.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/googleapis/genai-toolbox/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Yuan <45984206+Yuan325@users.noreply.github.com>
Add Spanner source and tool.
Spanner source is initialize with the following config:
```
sources:
my-spanner-source:
kind: spanner
project: my-project-name
instance: my-instance-name
database: my_db
# dialect: postgresql # The default dialect is google_standard_sql.
```
Spanner tool (with gsql dialect) is initialize with the following
config.
```
tools:
get_flight_by_id:
kind: spanner
source: my-cloud-sql-source
description: >
Use this tool to list all airports matching search criteria. Takes
at least one of country, city, name, or all and returns all matching
airports. The agent can decide to return the results directly to
the user.
statement: "SELECT * FROM flights WHERE id = @id"
parameters:
- name: id
type: int
description: 'id' represents the unique ID for each flight.
```
Spanner tool (with postgresql dialect) is initialize with the following
config.
```
tools:
get_flight_by_id:
kind: spanner
source: my-cloud-sql-source
description: >
Use this tool to list all airports matching search criteria. Takes
at least one of country, city, name, or all and returns all matching
airports. The agent can decide to return the results directly to
the user.
statement: "SELECT * FROM flights WHERE id = $1"
parameters:
- name: id
type: int
description: 'id' represents the unique ID for each flight.
```
Note: the only difference in config for both dialects is the sql
statement.
---------
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Running lint type checker (mypy) on the sdk tests can raise errors like:
https://github.com/googleapis/genai-toolbox/actions/runs/12194679407/job/34018934437
because it's looking in the wrong place for type definitions.
To fix this, we'll tell the type checker (mypy) to specifically include
the toolbox_llamaindex_sdk module from our SDK source code when it's
checking the tests. This will ensure it uses the correct type
information and avoids these errors.
This PR adds CI workflows for linting and integration tests to the
Langchain SDK.
Linting is failing due to type checking errors in the SDK (to be fixed
separately).
Integration tests are failing and need updates to match the latest SDK
code (also to be fixed separately).
Logger might be created pre- or post- flag. The logger defined pre- and
post- flag should be using the same out and err writer.
This will ensure that the server uses the same out and err writer
defined with the `WithStreams` options.
Allow user to set if their database uses private or public ip. The
reason we add this is because the dialer require different
initialization with private and public ip.
By default, toolbox will use public ip.
Different databases require different types for `Params` field when
adding parameters to their statement. e.g. alloydb, cloudsql, and
postgres uses `pgxpool` to query and build sql statement, whereas
spanner uses `Spanner` library.
Added a new `ParamValue` struct. `ParseParams` helper function parses
arbitraryJSON object into `[]ParamValue`, and the tool's invoke will
convert `[]ParamValue` into it's required type.
---------
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Allow user to set if their database uses private or public ip. The
reason we add this is because the dialer require different
initialization with private and public ip.
By default, toolbox will use public ip.
---------
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Logging support 4 different types of logging (debug, info, warn, error).
The default logging level is Info.
User will be able to set flag for log level (allowed values: "debug",
"info", "warn", "error"), example:
`go run . --log-level debug`
User will be able to set flag for logging format (allowed values:
"standard", "JSON"), example:
`go run . --logging-format json`
**sample http request log - std:**
server
```
2024-11-12T15:08:11.451377-08:00 INFO "Initalized 0 sources.\n"
```
httplog
```
2024-11-26T15:15:53.947287-08:00 INFO Response: 200 OK service: "httplog" httpRequest: {url: "http://127.0.0.1:5000/" method: "GET" path: "/" remoteIP: "127.0.0.1:64216" proto: "HTTP/1.1" requestID: "macbookpro.roam.interna/..." scheme: "http" header: {user-agent: "curl/8.7.1" accept: "*/*"}} httpResponse: {status: 200 bytes: 22 elapsed: 0.012417}
```
**sample http request log - structured:**
server
```
{
"timestamp":"2024-11-04T16:45:11.987299-08:00",
"severity":"ERROR",
"logging.googleapis.com/sourceLocation":{
"function":"github.com/googleapis/genai-toolbox/internal/log.(*StructuredLogger).Errorf",
"file":"/Users/yuanteoh/github/genai-toolbox/internal/log/log.go","line":157
},
"message":"unable to parse tool file at \"tools.yaml\": \"cloud-sql-postgres1\" is not a valid kind of data source"
}
```
httplog
```
{
"timestamp":"2024-11-26T15:12:49.290974-08:00",
"severity":"INFO",
"logging.googleapis.com/sourceLocation":{
"function":"github.com/go-chi/httplog/v2.(*RequestLoggerEntry).Write",
"file":"/Users/yuanteoh/go/pkg/mod/github.com/go-chi/httplog/v2@v2.1.1/httplog.go","line":173
},
"message":"Response: 200 OK",
"service":"httplog",
"httpRequest":{
"url":"http://127.0.0.1:5000/",
"method":"GET",
"path":"/",
"remoteIP":"127.0.0.1:64140",
"proto":"HTTP/1.1",
"requestID":"yuanteoh-macbookpro.roam.internal/NBrtYBu3q9-000001",
"scheme":"http",
"header":{"user-agent":"curl/8.7.1","accept":"*/*"}
},
"httpResponse":{"status":200,"bytes":22,"elapsed":0.0115}
}
```
This PR adds CI workflows for linting and integration tests to the
Llamaindex SDK.
Linting is failing due to type checking errors in the SDK (to be fixed
separately).
Integration tests are failing since there are no existing tests.
Does a few things to improve releasing process:
- Moves file to .ci folder
- Updates release script to check for version
- Uses printf instead of "echo" to work better between linux/mac
- removes comments about non-mac (worked on linux for me)
I've also updated the tigger to run on new tags.
---------
Co-authored-by: Yuan Teoh <yuanteoh@google.com>
Logging support 4 different types of logging (debug, info, warn, error).
Example of structured logger:
```
{
"timestamp":"2024-11-04T16:45:11.987299-08:00",
"severity":"ERROR",
"logging.googleapis.com/sourceLocation":{
"function":"github.com/googleapis/genai-toolbox/internal/log.(*StructuredLogger).Errorf",
"file":"/Users/yuanteoh/github/genai-toolbox/internal/log/log.go","line":157
},
"message":"unable to parse tool file at \"tools.yaml\": \"cloud-sql-postgres1\" is not a valid kind of data source"
}
```
```
{
"timestamp":"2024-11-04T16:45:11.987562-08:00",
"severity":"INFO",
"logging.googleapis.com/sourceLocation":{
"function":"github.com/googleapis/genai-toolbox/internal/log.(*StructuredLogger).Infof",
"file":"/Users/yuanteoh/github/genai-toolbox/internal/log/log.go","line":147
},
"message":"Initalized 0 sources.\n"
}
```
Logging support 4 different types of logging (debug, info, warn, error).
Example of standard logger:
`2024-11-12T15:08:11.451147-08:00 ERROR "unable to parse tool file at
\"tools.yaml\": \"cloud-sql-postgres1\" is not a valid kind of data
source"`
`2024-11-12T15:08:11.451377-08:00 INFO "Initalized 0 sources.\n"`
