Running on <4gb Vram
#93
by Jit2024 - opened
I found out a way of running the Minimax h3 model under 4gb Vram.
https://github.com/Jit-Roy/WeeLLM
It uses disk streaming and layer by layer swapping on gpu.
The speed is gonna be slow.
For me it works without any quantization n <4gb Vram RTX 3050 card. I also have <4gb Ram.
If anyone wanna check out go for it.