mmarch/text-generation-webui
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Don't require llama.cpp models to be placed in subfolders

Browse source
This commit is contained in:
oobabooga 2023-04-22 14:56:48 -03:00
parent 06b6ff6c2e
commit fcb594b90e
4 changed files with 41 additions and 39 deletions
Download patch file Download diff file Expand all files Collapse all files

28
docs/llama.cpp-models.md
View file

@ -1,23 +1,12 @@
## Using llama.cpp in the web UI
1. Re-install the requirements.txt:
#### Pre-converted models
```
pip install -r requirements.txt -U
```
Simply place the model in the `models` folder, making sure that its name contains `ggml` somewhere and ends in `.bin`.
2. Follow the instructions in the llama.cpp README to generate the `ggml-model-q4_0.bin` file: https://github.com/ggerganov/llama.cpp#usage
#### Convert LLaMA yourself
3. Create a folder inside `models/` for your model and put `ggml-model-q4_0.bin` in it. For instance, `models/llamacpp-7b/ggml-model-q4_0.bin`.
4. Start the web UI normally:
```
python server.py --model llamacpp-7b
```
* This procedure should work for any `ggml*.bin` file. Just put it in a folder, and use the name of this folder as the argument after `--model` or as the model loaded inside the interface.
* You can change the number of threads with `--threads N`.
Follow the instructions in the llama.cpp README to generate the `ggml-model-q4_0.bin` file: https://github.com/ggerganov/llama.cpp#usage
## Performance
@ -25,11 +14,4 @@ This was the performance of llama-7b int4 on my i5-12400F:
> Output generated in 33.07 seconds (6.05 tokens/s, 200 tokens, context 17)
## Limitations
~* The parameter sliders in the interface (temperature, top_p, top_k, etc) are completely ignored. So only the default parameters in llama.cpp can be used.~
~* Only 512 tokens of context can be used.~
~Both of these should be improved soon when llamacpp-python receives an update.~
You can change the number of threads with `--threads N`.