Qwen2.5-7B
Less memory.
Same quality.
Qwen2.5-7B used 43% less memory to load the model, while maintaining answer accuracy. That leaves more GPU memory available for running requests.
What changed
We compressed the model’s weights to an 8-bit format while keeping its request cache at BF16 precision. The smaller model footprint leaves more GPU memory available for serving requests.
Answer accuracy
Both versions answered the same 200 questions covering algebra, computer security, computer science and logical reasoning.
93% of answers were unchanged. Six incorrect answers became correct, while three correct answers became incorrect.
| Measure | Before | With GAISSA |
|---|---|---|
| Correct answers | 158 / 200 | 161 / 200 |
| Accuracy | 79.0% | 80.5% |
Memory use across four language models
| Model | Before | With GAISSA | Reduction |
|---|---|---|---|
| Qwen2.5-7B | 14.29 | 8.18 | 43% |
| Mistral-7B v0.3 | 13.51 | 7.01 | 48% |
| Phi-3.5 Mini | 7.14 | 3.80 | 47% |
| OLMo 2 7B | 13.60 | 7.60 | 44% |
Test setup
- Model
- Qwen2.5-7B Instruct
- Before
- vLLM 0.24.0, BF16 weights and BF16 KV cache.
- With GAISSA
- Offline FP8 weights and dynamic FP8 activations, with the BF16 KV cache retained.
- Hardware
- NVIDIA GeForce RTX 4090, 24 GB.
- Workload
- Runtime-reported model memory at loading. Quality was checked separately on 200 MMLU questions.
- Runs and reporting
- Recorded model-loading footprint for each setup. No repeated-run aggregate is available for this memory comparison.
The memory comparison measures the model’s footprint at loading, not total GPU memory during use. Accuracy was checked using the same task and answer checks before and after optimization.
What could improve in your AI?
Tell us what you run and what you want to improve.