Add Spanner source and tool.
Spanner source is initialize with the following config:
```
sources:
my-spanner-source:
kind: spanner
project: my-project-name
instance: my-instance-name
database: my_db
# dialect: postgresql # The default dialect is google_standard_sql.
```
Spanner tool (with gsql dialect) is initialize with the following
config.
```
tools:
get_flight_by_id:
kind: spanner
source: my-cloud-sql-source
description: >
Use this tool to list all airports matching search criteria. Takes
at least one of country, city, name, or all and returns all matching
airports. The agent can decide to return the results directly to
the user.
statement: "SELECT * FROM flights WHERE id = @id"
parameters:
- name: id
type: int
description: 'id' represents the unique ID for each flight.
```
Spanner tool (with postgresql dialect) is initialize with the following
config.
```
tools:
get_flight_by_id:
kind: spanner
source: my-cloud-sql-source
description: >
Use this tool to list all airports matching search criteria. Takes
at least one of country, city, name, or all and returns all matching
airports. The agent can decide to return the results directly to
the user.
statement: "SELECT * FROM flights WHERE id = $1"
parameters:
- name: id
type: int
description: 'id' represents the unique ID for each flight.
```
Note: the only difference in config for both dialects is the sql
statement.
---------
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Running lint type checker (mypy) on the sdk tests can raise errors like:
https://github.com/googleapis/genai-toolbox/actions/runs/12194679407/job/34018934437
because it's looking in the wrong place for type definitions.
To fix this, we'll tell the type checker (mypy) to specifically include
the toolbox_llamaindex_sdk module from our SDK source code when it's
checking the tests. This will ensure it uses the correct type
information and avoids these errors.
This PR adds CI workflows for linting and integration tests to the
Langchain SDK.
Linting is failing due to type checking errors in the SDK (to be fixed
separately).
Integration tests are failing and need updates to match the latest SDK
code (also to be fixed separately).
Logger might be created pre- or post- flag. The logger defined pre- and
post- flag should be using the same out and err writer.
This will ensure that the server uses the same out and err writer
defined with the `WithStreams` options.
Allow user to set if their database uses private or public ip. The
reason we add this is because the dialer require different
initialization with private and public ip.
By default, toolbox will use public ip.
Different databases require different types for `Params` field when
adding parameters to their statement. e.g. alloydb, cloudsql, and
postgres uses `pgxpool` to query and build sql statement, whereas
spanner uses `Spanner` library.
Added a new `ParamValue` struct. `ParseParams` helper function parses
arbitraryJSON object into `[]ParamValue`, and the tool's invoke will
convert `[]ParamValue` into it's required type.
---------
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Allow user to set if their database uses private or public ip. The
reason we add this is because the dialer require different
initialization with private and public ip.
By default, toolbox will use public ip.
---------
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Logging support 4 different types of logging (debug, info, warn, error).
The default logging level is Info.
User will be able to set flag for log level (allowed values: "debug",
"info", "warn", "error"), example:
`go run . --log-level debug`
User will be able to set flag for logging format (allowed values:
"standard", "JSON"), example:
`go run . --logging-format json`
**sample http request log - std:**
server
```
2024-11-12T15:08:11.451377-08:00 INFO "Initalized 0 sources.\n"
```
httplog
```
2024-11-26T15:15:53.947287-08:00 INFO Response: 200 OK service: "httplog" httpRequest: {url: "http://127.0.0.1:5000/" method: "GET" path: "/" remoteIP: "127.0.0.1:64216" proto: "HTTP/1.1" requestID: "macbookpro.roam.interna/..." scheme: "http" header: {user-agent: "curl/8.7.1" accept: "*/*"}} httpResponse: {status: 200 bytes: 22 elapsed: 0.012417}
```
**sample http request log - structured:**
server
```
{
"timestamp":"2024-11-04T16:45:11.987299-08:00",
"severity":"ERROR",
"logging.googleapis.com/sourceLocation":{
"function":"github.com/googleapis/genai-toolbox/internal/log.(*StructuredLogger).Errorf",
"file":"/Users/yuanteoh/github/genai-toolbox/internal/log/log.go","line":157
},
"message":"unable to parse tool file at \"tools.yaml\": \"cloud-sql-postgres1\" is not a valid kind of data source"
}
```
httplog
```
{
"timestamp":"2024-11-26T15:12:49.290974-08:00",
"severity":"INFO",
"logging.googleapis.com/sourceLocation":{
"function":"github.com/go-chi/httplog/v2.(*RequestLoggerEntry).Write",
"file":"/Users/yuanteoh/go/pkg/mod/github.com/go-chi/httplog/v2@v2.1.1/httplog.go","line":173
},
"message":"Response: 200 OK",
"service":"httplog",
"httpRequest":{
"url":"http://127.0.0.1:5000/",
"method":"GET",
"path":"/",
"remoteIP":"127.0.0.1:64140",
"proto":"HTTP/1.1",
"requestID":"yuanteoh-macbookpro.roam.internal/NBrtYBu3q9-000001",
"scheme":"http",
"header":{"user-agent":"curl/8.7.1","accept":"*/*"}
},
"httpResponse":{"status":200,"bytes":22,"elapsed":0.0115}
}
```
This PR adds CI workflows for linting and integration tests to the
Llamaindex SDK.
Linting is failing due to type checking errors in the SDK (to be fixed
separately).
Integration tests are failing since there are no existing tests.
Does a few things to improve releasing process:
- Moves file to .ci folder
- Updates release script to check for version
- Uses printf instead of "echo" to work better between linux/mac
- removes comments about non-mac (worked on linux for me)
I've also updated the tigger to run on new tags.
---------
Co-authored-by: Yuan Teoh <yuanteoh@google.com>
Logging support 4 different types of logging (debug, info, warn, error).
Example of structured logger:
```
{
"timestamp":"2024-11-04T16:45:11.987299-08:00",
"severity":"ERROR",
"logging.googleapis.com/sourceLocation":{
"function":"github.com/googleapis/genai-toolbox/internal/log.(*StructuredLogger).Errorf",
"file":"/Users/yuanteoh/github/genai-toolbox/internal/log/log.go","line":157
},
"message":"unable to parse tool file at \"tools.yaml\": \"cloud-sql-postgres1\" is not a valid kind of data source"
}
```
```
{
"timestamp":"2024-11-04T16:45:11.987562-08:00",
"severity":"INFO",
"logging.googleapis.com/sourceLocation":{
"function":"github.com/googleapis/genai-toolbox/internal/log.(*StructuredLogger).Infof",
"file":"/Users/yuanteoh/github/genai-toolbox/internal/log/log.go","line":147
},
"message":"Initalized 0 sources.\n"
}
```
Logging support 4 different types of logging (debug, info, warn, error).
Example of standard logger:
`2024-11-12T15:08:11.451147-08:00 ERROR "unable to parse tool file at
\"tools.yaml\": \"cloud-sql-postgres1\" is not a valid kind of data
source"`
`2024-11-12T15:08:11.451377-08:00 INFO "Initalized 0 sources.\n"`
