Hello . thanks for your effort . the model is too big to fit on consumer gpu . can be a FP8 quant ?
you can try gguf first， I can quantize a fp8 model，but need to test it and add codes
· Sign up or log in to comment
