mirror of
https://github.com/googleapis/genai-toolbox.git
synced 2026-02-14 09:05:04 -05:00
Compare commits
56 Commits
feat/add-g
...
processing
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b29bb21c75 | ||
|
|
2d6a2a76d6 | ||
|
|
33a98db375 | ||
|
|
22657c9dad | ||
|
|
d829ee7bdf | ||
|
|
f598543ea4 | ||
|
|
4885f6ebfb | ||
|
|
77af793066 | ||
|
|
fe01519f61 | ||
|
|
4d0a2ce038 | ||
|
|
126dbe0d90 | ||
|
|
1f79754866 | ||
|
|
2ec6dac8bd | ||
|
|
fad11de9ff | ||
|
|
35d5bd67e8 | ||
|
|
eb8a46c175 | ||
|
|
c4f924e6f8 | ||
|
|
0fcf690d42 | ||
|
|
2564058fd9 | ||
|
|
2a8d6af4d1 | ||
|
|
4dad163803 | ||
|
|
b5c55020c9 | ||
|
|
1dc3417557 | ||
|
|
8ea10c2caf | ||
|
|
632dd10180 | ||
|
|
9771aa47df | ||
|
|
a212aedd19 | ||
|
|
9210e5555c | ||
|
|
b43af71793 | ||
|
|
da1f463dd1 | ||
|
|
3265f7e3a6 | ||
|
|
336743f747 | ||
|
|
911069ae8d | ||
|
|
cee59d52c3 | ||
|
|
9517daba09 | ||
|
|
3c61ee0597 | ||
|
|
19271eb9ee | ||
|
|
3a150c77ca | ||
|
|
ca6f31a192 | ||
|
|
d7faf7700f | ||
|
|
37a60ea2a6 | ||
|
|
8de16976ae | ||
|
|
49cb2f39f7 | ||
|
|
f169874e53 | ||
|
|
db8c3a3c77 | ||
|
|
8b33b0c67f | ||
|
|
35fa73516b | ||
|
|
66df3bfd21 | ||
|
|
73e0edc3cd | ||
|
|
3f32a9aab6 | ||
|
|
28006fc9b2 | ||
|
|
56c69131b4 | ||
|
|
ad4a509340 | ||
|
|
d39acac96c | ||
|
|
6df2ad28a9 | ||
|
|
8416378613 |
@@ -50,5 +50,30 @@ It is helpful to understand how tool-level processing differs from other scopes:
|
|||||||
- **Model Level**: Intercepts individual calls to the LLM (prompts and responses). Unlike tool-level, this applies globally to all text sent/received, making it better for global PII redaction or token tracking.
|
- **Model Level**: Intercepts individual calls to the LLM (prompts and responses). Unlike tool-level, this applies globally to all text sent/received, making it better for global PII redaction or token tracking.
|
||||||
- **Agent Level**: Wraps the high-level execution loop (e.g., a "turn" in the conversation). Unlike tool-level, this envelopes the entire turn (user input to final response), making it suitable for session management or end-to-end auditing.
|
- **Agent Level**: Wraps the high-level execution loop (e.g., a "turn" in the conversation). Unlike tool-level, this envelopes the entire turn (user input to final response), making it suitable for session management or end-to-end auditing.
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
### Security & Guardrails
|
||||||
|
|
||||||
|
- **Principle of Least Privilege**: Ensure that tools run with the minimum necessary permissions. Middleware is an excellent place to enforce "read-only" modes or verify user identity before executing sensitive actions.
|
||||||
|
- **Input Sanitization**: Actively strip potential PII (like credit card numbers or raw emails) from tool arguments before logging them.
|
||||||
|
- **Prompt Injection Defense**: Use pre-processing hooks to scan user inputs for known jailbreak patterns or malicious directives before they reach the model or tools.
|
||||||
|
|
||||||
|
### Observability & Debugging
|
||||||
|
|
||||||
|
- **Structured Logging**: Instead of simple print statements, use structured JSON logging with correlation IDs. This allows you to trace a single user request through multiple agent turns and tool calls.
|
||||||
|
- **Logging for Testability**: LLM responses are non-deterministic and may summarize away key details.
|
||||||
|
- **Pattern**: Add explicit logging markers in your post-processing middleware (e.g., `logger.info("ACTION_SUCCESS: <id>")`).
|
||||||
|
- **Benefit**: Your integration tests can grep logs for these stable markers to verify tool success, rather than painfully parsing variable natural language responses.
|
||||||
|
|
||||||
|
### Performance & Cost Optimization
|
||||||
|
|
||||||
|
- **Token Economy**: Tools often return verbose JSON. Use post-processing to strip unnecessary fields or summarize large datasets *before* returning the result to the LLM's context window. This saves tokens and reduces latency.
|
||||||
|
- **Caching**: For read-heavy tools (like "search_knowledge_base"), implement caching middleware to return previous results for identical queries, saving both time and API costs.
|
||||||
|
|
||||||
|
### Error Handling
|
||||||
|
|
||||||
|
- **Graceful Degradation**: If a tool fails (e.g., API timeout), catch the exception in middleware and return a structured error message to the LLM (e.g., `Error: Database timeout, please try again`).
|
||||||
|
- **Self-Correction**: Well-formatted error messages often allow the LLM to understand *why* a call failed and retry it with corrected parameters automatically.
|
||||||
|
|
||||||
|
|
||||||
## Samples
|
## Samples
|
||||||
|
|||||||
3
docs/en/samples/pre_post_processing/golden.txt
Normal file
3
docs/en/samples/pre_post_processing/golden.txt
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
AI:
|
||||||
|
Loyalty Points
|
||||||
|
POLICY CHECK: Intercepting 'update-hotel'
|
||||||
Reference in New Issue
Block a user