Running Qwen3.5-9B on Dual RTX 3060s with vLLM

I wanted to run Qwen3.5-9B-AWQ on my dual RTX 3060 12GB GPUs with a 256K context window and improve inference speed without upgrading my hardware.

My Setup

• GPUs: 2 × NVIDIA RTX 3060 12GB
• OS: Ubuntu Linux
• Inference engine: vLLM
• Model: cyankiwi/Qwen3.5-9B-AWQ-4bit
• Frontend: Open WebUI
• Deployment: k3s and systemd

vLLM Configuration

These are the main arguments from my deployment:

args:
  - --served-model-name
  - Qwen3.5-9B-AWQ
  - --port
  - "8080"
  - --tensor-parallel-size
  - "2"
  - --gpu-memory-utilization
  - "0.94"
  - --max-model-len
  - "262144"
  - --max-num-seqs
  - "2"
  - --trust-remote-code
  - --enforce-eager
  - --language-model-only
  - --enable-auto-tool-choice
  - --tool-call-parser
  - hermes
  - --speculative-config
  - '{"method":"mtp","num_speculative_tokens":3}'

Key Arguments

• --tensor-parallel-size 2: Distributes model computation across both GPUs.
• --gpu-memory-utilization 0.94: Allows vLLM to use most of the available VRAM while retaining some headroom.
• --max-model-len 262144: Sets the maximum configured sequence length to 256K tokens.
• --max-num-seqs 2: Limits the number of concurrently scheduled sequences to two.
• --language-model-only: Disables the vision component because I only needed text inference.
• --enforce-eager: Disables CUDA Graphs to reduce their memory overhead.
• --enable-auto-tool-choice and --tool-call-parser hermes: Enable automatic tool selection and tool-call parsing for Open WebUI.

Disabling the vision component reduced startup time from approximately 85 seconds to 6 seconds and freed around 1.5GB of VRAM across both GPUs in my tests.

The 256K setting is the maximum configured sequence length, not a guarantee that two 256K requests can run simultaneously. Actual concurrency depends on KV-cache capacity and runtime memory requirements.

Enabling Native MTP

I tested speculative decoding using the model's native Multi-Token Prediction (MTP) capability.

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

MTP proposes multiple future tokens for verification, potentially reducing sequential decoding steps. I tested different speculative-token counts and compared the results against standard decoding.

Benchmark Results

These are results from my tests on the specified workloads. Performance varies with the prompt and the number of speculative tokens accepted.

MTP provided the largest gains on code refactoring and JSON extraction. Creative generation also improved, but by a smaller margin.

Multi-GPU Environment Settings

I used the following environment variables in the Kubernetes deployment:

env:
  - name: NCCL_P2P_DISABLE
    value: "1"
  - name: NCCL_IB_DISABLE
    value: "1"
  - name: VLLM_WORKER_MULTIPROC_METHOD
    value: "spawn"
  - name: PYTORCH_CUDA_ALLOC_CONF
    value: "expandable_segments:True"

• NCCL_P2P_DISABLE=1: Disables direct GPU peer-to-peer communication.
• NCCL_IB_DISABLE=1: Disables InfiniBand communication.
• VLLM_WORKER_MULTIPROC_METHOD=spawn: Uses spawn for worker initialization.
• PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True: Helps reduce memory fragmentation.

These settings were used for my setup. Disabling peer-to-peer communication can increase transfer overhead, so it is worth testing against the defaults on other systems.

Deployment

I used k3s to manage the vLLM deployment and systemd to control when the model runs.

systemctl --user start vllm
systemctl --user stop vllm

This lets me start the model when required and release GPU memory when I need the GPUs for other workloads.

Summary

The main improvements came from three changes:

1. Disabling the unused vision component reduced startup time and VRAM usage.
2. Using eager execution reduced CUDA Graph memory overhead.
3. Enabling native MTP improved generation speed, reaching 86.24 tokens/s in my JSON extraction benchmark.

The results show how vLLM configuration and speculative decoding can improve inference on dual RTX 3060s without changing the hardware. Performance is workload-dependent, and a 256K context limit does not guarantee that multiple maximum-length requests can run concurrently.