代码拉取完成，页面将自动刷新
A simple text generator for ComfyUI utilizing ExLlamaV2.
Navigate to the root ComfyUI directory, clone the repository to custom_nodes and install dependencies:
git clone https://github.com/Zuellni/ComfyUI-ExLlama-Nodes custom_nodes/ComfyUI-ExLlama-Nodes
pip install -r custom_nodes/ComfyUI-ExLlama-Nodes/requirements.txt
Optionally, you can install flash-attention by uncommenting the relevant lines in the requirements file. It should lower VRAM usage but your mileage may vary.
[!IMPORTANT] The wheels included in the requirements file should match the latest portable ComfyUI build. If you see any ExLlama-related errors while loading the nodes, try to install it manually following the official instructions.
Only EXL2 and 4-bit GPTQ models are supported. You can find a lot of them on Hugging Face. Refer to the model card in each repository for details about quant differences and instruction formats.
To use a model with the nodes, you should clone its repository with git or manually download all the files and place them in models/llm. For example, if you'd like to download the 4-bit 32g version of Zephyr 7B Beta, use the following command:
git clone https://huggingface.co/TheBloke/zephyr-7B-beta-GPTQ -b gptq-4bit-32g-actorder_True models/llm/zephyr-7b-gptq-32g
[!TIP] You can add your own
llmpath to the extra_model_paths.yaml file and place the models there instead.
| Loader | Loads models from the llm directory. |
|
| gpu_split | Comma-separated VRAM in GB per GPU, eg 6.9, 8. |
|
| cache_8bit | Lower VRAM usage but also lower speed. | |
| max_seq_len | Max context, higher number equals higher VRAM usage. 0 will default to config. |
|
| Generator | Generates text based on the given prompt. Refer to text-generation-webui for parameters. | |
| unload | Unloads the model after each generation. | |
| single_line | Stops the generation on newline. | |
| max_tokens | Max new tokens, 0 will use available context. |
|
| Preview | Displays generated text in the UI. | |
| Replace | Replaces variable names enclosed in brackets, eg [a], with their values. |
|
The example workflow is embedded in the image below and can be opened in ComfyUI.
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
马建仓 AI 助手