mmarch/text-generation-webui
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Add flash-attention 2 for windows (#4235)

Browse source
This commit is contained in:
Brian Dashore 2023-10-21 02:46:23 -04:00 • committed by GitHub
parent 258d046218
commit 3345da2ea4
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
10 changed files with 139 additions and 110 deletions
Download patch file Download diff file Expand all files Collapse all files

24
README.md
View file

@ -58,7 +58,7 @@ To define persistent command-line flags like `--listen` or `--api`, edit the `CM
* There is no need to run any of those scripts as admin/root.
* For additional instructions about AMD setup, WSL setup, and nvcc installation, consult [this page](https://github.com/oobabooga/text-generation-webui/blob/main/docs/One-Click-Installers.md).
* The installer has been tested mostly on NVIDIA GPUs. If you can find a way to improve it for your AMD/Intel Arc/Mac Metal GPU, you are highly encouraged to submit a PR to this repository. The main file to be edited is `one_click.py`.
* For automated installation, you can use the `GPU_CHOICE`, `LAUNCH_AFTER_INSTALL`, and `INSTALL_EXTENSIONS` environment variables. For instance: `GPU_CHOICE=A LAUNCH_AFTER_INSTALL=False INSTALL_EXTENSIONS=False ./start_linux.sh`.
* For automated installation, you can use the `GPU_CHOICE`, `USE_CUDA118`, `LAUNCH_AFTER_INSTALL`, and `INSTALL_EXTENSIONS` environment variables. For instance: `GPU_CHOICE=A USE_CUDA118=FALSE LAUNCH_AFTER_INSTALL=FALSE INSTALL_EXTENSIONS=FALSE ./start_linux.sh`.
### Manual installation using Conda
@ -78,7 +78,7 @@ bash Miniconda3.sh
#### 1. Create a new conda environment
```
conda create -n textgen python=3.10
conda create -n textgen python=3.11
conda activate textgen
```
@ -86,11 +86,11 @@ conda activate textgen
| System | GPU | Command |
|--------|---------|---------|
| Linux/WSL | NVIDIA | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118` |
| Linux/WSL | NVIDIA | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121` |
| Linux/WSL | CPU only | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu` |
| Linux | AMD | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.6` |
| MacOS + MPS | Any | `pip3 install torch torchvision torchaudio` |
| Windows | NVIDIA | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118` |
| Windows | NVIDIA | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121` |
| Windows | CPU only | `pip3 install torch torchvision torchaudio` |
The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.
@ -98,7 +98,7 @@ The up-to-date commands can be found here: https://pytorch.org/get-started/local
For NVIDIA, you may also need to manually install the CUDA runtime libraries:
```
conda install -y -c "nvidia/label/cuda-11.8.0" cuda-runtime
conda install -y -c "nvidia/label/cuda-12.1.0" cuda-runtime
```
#### 3. Install the web UI
@ -140,12 +140,18 @@ cd text-generation-webui
git clone https://github.com/turboderp/exllama repositories/exllama
```
##### bitsandbytes on older NVIDIA GPUs
##### Older NVIDIA GPUs
bitsandbytes >= 0.39 may not work. In that case, to use `--load-in-8bit`, you may have to downgrade like this:
1) For Kepler GPUs and older, you will need to install CUDA 11.8 instead of 12:
* Linux: `pip install bitsandbytes==0.38.1`
* Windows: `pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl`
```
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
conda install -y -c "nvidia/label/cuda-11.8.0" cuda-runtime
```
2) bitsandbytes >= 0.39 may not work. In that case, to use `--load-in-8bit`, you may have to downgrade like this:
* Linux: `pip install bitsandbytes==0.38.1`
* Windows: `pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl`
##### Manual install