Instructions to use Gryphe/Various-Quants with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Gryphe/Various-Quants with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Gryphe/Various-Quants:Q6_K # Run inference directly in the terminal: llama cli -hf Gryphe/Various-Quants:Q6_K
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Gryphe/Various-Quants:Q6_K # Run inference directly in the terminal: llama cli -hf Gryphe/Various-Quants:Q6_K
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf Gryphe/Various-Quants:Q6_K # Run inference directly in the terminal: ./llama-cli -hf Gryphe/Various-Quants:Q6_K
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf Gryphe/Various-Quants:Q6_K # Run inference directly in the terminal: ./build/bin/llama-cli -hf Gryphe/Various-Quants:Q6_K
Use Docker
docker model run hf.co/Gryphe/Various-Quants:Q6_K
- LM Studio
- Jan
- Ollama
How to use Gryphe/Various-Quants with Ollama:
ollama run hf.co/Gryphe/Various-Quants:Q6_K
- Unsloth Desktop
- Docker Model Runner
How to use Gryphe/Various-Quants with Docker Model Runner:
docker model run hf.co/Gryphe/Various-Quants:Q6_K
- Lemonade
How to use Gryphe/Various-Quants with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Gryphe/Various-Quants:Q6_K
Run and chat with the model
lemonade run user.Various-Quants-Q6_K
List all available models
lemonade list
- Atomic Chat
Commit History
-
Update README.md
Verified signature7de2e16
-
Delete pantheon-roleplay-13b.q4_K_S.gguf
Verified signature1d5e896 -
Delete nyaa-7b.q6_K.gguf
Verified signaturea848d66 -
Upload pantheon-rp-l3-8b.q6_K.gguf
Verified signatureaf5098b
-
Upload pantheon-instruct-l3-8b.q6_K.gguf
Verified signature30a4f25
-
Upload pantheon-roleplay-13b.q4_K_S.gguf
Verified signatureac38e2b
-
Delete mythomist-7b.q6_K.gguf
Verified signature908e7b1 -
Delete tiamat-7b.q6_K.gguf
Verified signature9702b7a -
Delete tiamat-7b.q4_K_M.gguf
Verified signature79aa977 -
Delete mythomist-7b.q4_K_M.gguf
Verified signatureaba5f8e -
Update README.md
Verified signature223b783 -
Upload nyaa-7b.q6_K.gguf
Verified signaturee921f95
-
Upload 2 files
9679034
-
Delete MergeMonster-7b-20231124.q6_K.gguf
c18d064 -
Delete MergeMonster-13b-20231124.q6_K.gguf
41517dd
-
Upload MergeMonster-13b-20231124.q6_K.gguf
f3bace7 -
Upload MergeMonster-7b-20231124.q6_K.gguf
589560a
-
Update README.md
771fc82 -
Update README.md
39b208a
-
Upload mythomist-7b.q6_K.gguf
145ab84 -
Delete mythoboros-13b.ggmlv3.q5_0.bin
7cb7217 -
Delete mythomix-L2-13b.ggmlv3.q5_K_M.bin
ddc4fd8 -
Delete mythomix-variant-L2-13b.ggmlv3.q5_K_M.bin
d719c83 -
Delete Oniichat_hermes-limarp-13b-merged.ggmlv3.q4_K_M.bin
5fedbdb -
Delete Mikael110_llama-2-13b-guanaco.ggmlv3.q4_1.bin
7f7b19c -
Upload mythomist-7b.q4_K_M.gguf
2eb4ee5 -
Upload mythomist-7b.q4_K_M.gguf
e0fd2aa
-
Upload mythomist-7b.q4_K_M.gguf
027e098 -
Upload mythomist-7b.q4_K_M.gguf
21debe7
-
Upload mythomist-7b.q4_K_M.gguf
b2233ba
-
Upload tiny-aesir-1.1b.Q6_K.gguf
f3c3183
-
Upload mythomist-7b.q4_K_M.gguf
496da8e -
Upload mythomist-7b.q4_K_M.gguf
34af948
-
Upload mythomist-7b.q4_K_M.gguf
6d128d8
-
Rename mythomist-7b.gguf to mythomist-7b.q4_K_M.gguf
db91ce7 -
Upload mythomist-7b.gguf
e8a1baf
-
Upload mythomix-variant-L2-13b.ggmlv3.q5_K_M.bin
95100c4 -
Delete mythomax-L2-13b.ggmlv3.q5_K_M.bin
e6c59f4 -
Upload mythomax-L2-13b.ggmlv3.q5_K_M.bin
33097cc -
Delete mythomax-L2-13b.ggmlv3.q5_K_M.bin
d8d5f06 -
Upload mythomax-L2-13b.ggmlv3.q5_K_M.bin
48de299
-
Update README.md
a9710fc -
Upload mythomix-L2-13b.ggmlv3.q5_K_M.bin
da05ed6
-
Update README.md
52c9495
-
Rename Oniichat_hermes-limarp-13b-merged.q4_K_M.bin to Oniichat_hermes-limarp-13b-merged.ggmlv3.q4_K_M.bin (#1)
- Rename Oniichat_hermes-limarp-13b-merged.q4_K_M.bin to Oniichat_hermes-limarp-13b-merged.ggmlv3.q4_K_M.bin (510b8a3fd4a5f40897d6e499c62ea70a4fe081aa) Co-authored-by: IkariDev <IkariDev@users.noreply.huggingface.co>b689a3a -
Upload Oniichat_hermes-limarp-13b-merged.q4_K_M.bin
abea29d
-
Upload mythoboros-13b.ggmlv3.q5_0.bin
83075c3
-
Upload Mikael110_llama-2-13b-guanaco.ggmlv3.q4_1.bin
94d9cb7 -
Update README.md
69016f0 -
initial commit
a9f7bbc