github/OpenHands
1
1
Fork 0
mirror of https://github.com/All-Hands-AI/OpenHands.git synced 2026-01-10 07:18:10 -05:00
Code Issues Packages Projects Releases Wiki Activity

Add ArcticInference doc (#9492)

Browse Source
This commit is contained in:
Graham Neubig
2025-07-01 14:15:13 -04:00
committed by GitHub
parent 6da7e051be
commit e05e627957
1 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/usage/llms/local-llms.mdx
View File

@@ -175,6 +175,10 @@ vllm serve mistralai/Devstral-Small-2505 \
--enable-prefix-caching
```
If you are interested in further improved inference speed, you can also try Snowflake's version
of vLLM, [ArcticInference](https://www.snowflake.com/en/engineering-blog/fast-speculative-decoding-vllm-arctic/),
which can achieve up to 2x speedup in some cases.
### Run OpenHands (Alternative Backends)
#### Using Docker