github/home-llm
1
1
Fork 0
mirror of https://github.com/acon96/home-llm.git synced 2026-01-09 13:48:05 -05:00
Code Issues Packages Projects Releases Wiki Activity

Add Llama3/LM Studio path to setup docs

Browse Source
This commit is contained in:
Alex O'Connell
2024-06-08 13:45:25 -04:00
parent 4165b66c82
commit 9e5f58ae1c
1 changed files with 35 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
docs/Setup.md
View File

@@ -103,6 +103,41 @@ Once the desired API has been selected, scroll to the bottom and click `Submit`.
> NOTE: The key settings in this case are that our prompt references the `{{ response_examples }}` variable and the `Enable in context learning (ICL) examples` option is turned on.
## Path 3: Using Llama-3-8B with LM Studio
### Overview
Another model you can use if you have a GPU is Meta's Llama-3-8B Model. This path assumes you have a machine with a GPU that already has [LM Studio](https://lmstudio.ai/) installed on it. This path utilizes in-context learning examples, to prompt the model to produce the output that we expect.
### Step 1: Downloading and serving the Model
Llama 3 8B can be set up and downloaded on the serving machine using LM Studio by:
1. Search for `lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF` in the main interface.
2. Select and download the version of the model that is recommended for your VRAM configuration.
3. Select the 'Local Server' tab on the left side of the application.
4. Load the model by selecting it from the bar in the top middle of the screen. The server should start automatically when the model finishes loading.
5. Take note of the port that the server is running on.
### Step 2: Connect to the LM Studio API
1. In Home Assistant: navigate to `Settings > Devices and Services`
2. Select the `+ Add Integration` button in the bottom right corner
3. Search for, and select `Local LLM Conversation`
4. Select `Generic OpenAI Compatible API` from the dropdown and click `Submit`
5. Set up the connection to the API:
- **IP Address**: Fill out IP Address for the machine hosting LM Studio
- **Port**: enter the port that was listed in LM Studio
- **Use HTTPS**: unchecked
- **Model Name**: This can be any value, as LM Studio uses the currently loaded model for all incoming requests.
- **API Key**: leave blank
6. Click `Submit`
### Step 3: Model Configuration
This step allows you to configure how the model is "prompted". See [here](./Model%20Prompting.md) for more information on how that works.
For now, defaults for the model should have been populated. If you would like the model to be able to control devices then you must select the `Assist` API.
Once the desired API has been selected, scroll to the bottom and click `Submit`.
> NOTE: The key settings in this case are that our prompt references the `{{ response_examples }}` variable and the `Enable in context learning (ICL) examples` option is turned on.
## Configuring the Integration as a Conversation Agent
Now that the integration is configured and providing the conversation agent, we need to configure Home Assistant to use our conversation agent instead of the built in intent recognition system.