Running MiniMax-H3 on a Single RTX 3060 12GB GPU: My Experiment with GGUF, LoRA and Dynamic Loading

I wanted to run MiniMax-H3 FL2VA on my RTX 3060 12GB. The model is large, and keeping all its weights in VRAM wasn't an option for my setup.

I already had dynamic weight loading working. The model weights stayed in system RAM and were loaded into VRAM when needed.

Then I tried adding the official 8-step Turbo LoRA to speed up generation. That's where things broke.

1. The Problem

Dynamic loading worked fine with the base model. The problem appeared only when I tried using the Turbo LoRA.

If the weights stayed loaded in VRAM, it worked. But when I tried to load and unload weights dynamically from RAM with the LoRA enabled, I couldn't get it working.

I didn't want to give up on dynamic loading just to use the LoRA, so I started looking for another way.

The solution I explored was to merge the LoRA into the GGUF model beforehand. This way, I wouldn't need to load a separate adapter during inference.

2. Merging the LoRA into GGUF

A LoRA adds a small weight update to the original model. The update is calculated using two matrices:

The weight update is calculated as:

ΔW = (α / r) × (B × A)

The updated weight is:

W_new = W_base + ΔW

The problem is that GGUF weights are quantized. I couldn't just add the LoRA update directly to the stored values.

So I dequantized the required weights to FP32, applied the LoRA update, and quantized them back into the original GGUF format.

After this, the LoRA changes were part of the model itself. There was no separate adapter to load during inference.

3. Processing Large Models Without Loading Everything into RAM

I didn't want to load the entire model into memory just to merge the LoRA.

Instead, I used gguf.GGUFReader to process the model one tensor at a time. Each tensor was read, updated if needed, quantized, and written to the output file.

For Q6_K, I used GGML's native quantize_row_q6_K function through ctypes and libggml-cpu.so. This let me use optimized native CPU code for quantization.

In my tests, the quantization throughput exceeded 40 million floats per second.

Tensors that didn't need LoRA updates, such as embeddings and normalization parameters, were left unchanged.

4. Loading a Model Larger Than VRAM

The next step was to run the merged model using dynamic loading.

The model weights stay in system RAM, and the inference engine loads the required weights into VRAM as needed. Once they're no longer needed, they can be unloaded to make room for other layers.

This means the GPU doesn't have to hold the entire model at once. It still needs memory for the active weights, video latents, and temporary buffers.

The actual VRAM usage depends on how the inference engine handles offloading and how much of the model it keeps on the GPU.

5. Why 8 Steps Matter

The original model uses around 50 diffusion steps. The Turbo LoRA reduces this to 8.

That's a big reduction, especially when weights have to move between system RAM and VRAM during inference.

For example, if a model has 16GB of weights and the entire set is transferred once per step:

That's an 84% reduction in total transfers under this assumption.

These are cumulative transfers, not the amount of VRAM required. Actual traffic depends on how the inference engine loads, unloads, and caches weights.

6. Q5_0 vs Q6_K

Here are the results from my setup.

Q5_0 is smaller and requires less data to be transferred. Q6_K uses more storage but retains more weight precision.

The speed difference and VRAM usage are specific to my setup. The visual quality also needs to be compared using the same prompts and generation settings.

7. Conclusion

I started with dynamic loading working on the base model. The issue came when I tried to add the Turbo LoRA.

Instead of changing the entire setup, I tried merging the LoRA into the GGUF model beforehand. That removed the need to load a separate adapter during inference.

The main takeaway is that a model being larger than your VRAM doesn't automatically mean you can't run it. Dynamic loading can help, but performance depends on how efficiently weights move between RAM and VRAM.

Reducing the number of diffusion steps also helps when those weights have to be transferred repeatedly.

This was my attempt to make a large video-generation model work on the hardware I already had, rather than upgrading the GPU just to run it.

8. Models on Hugging Face

I have published all four baked GGUF variants on Hugging Face.

Repository: udero/minimax-h3-turbo-gguf

All four models have the Turbo LoRA merged in, so you don't need a separate adapter during inference.

Q4_K is the smallest option, while Q8_0 is aimed at setups with more system RAM and storage. The Q8_0 version is recommended for systems with 64GB or more RAM, although actual memory requirements depend on the inference setup.

Download: MiniMax-H3 Turbo GGUF models on Hugging Face