SmolLM2
Generate text
with less energy.
SmolLM2 used 33–66% less GPU energy across four request loads. Responses took 8–21% longer, while answer accuracy stayed within the quality target.
What changed
We changed how SmolLM2 generates text, using patterns in the prompt to propose several tokens for the model to check together. The weights stayed unchanged. This reduced GPU energy use, but individual responses took longer.
How the saving changes with demand
Each load ran five times. The figures show the median change at that load.
| Simultaneous requests | Less GPU energy | Longer response time |
|---|---|---|
| 1 | 66% | 8% |
| 8 | 55% | 11% |
| 16 | 43% | 20% |
| 32 | 33% | 21% |
Performance and resource use
| Measure | Before | With GAISSA |
|---|---|---|
| GPU energykJ | 21.15 | 7.16 |
| Response time95th percentile, ms | 550.78 | 596.14 |
| Text generatedoutput tokens | 16,384 | 16,384 |
Answer checks
The quality check covered 1,060 questions across 53 MMLU subjects. One previously correct answer became incorrect, within the quality target.
Two of eight additional free-form responses changed. The optimized model does not produce identical text.
Test setup
- Model
- HuggingFaceTB SmolLM2 1.7B Instruct
- Before
- vLLM 0.24.0, BF16 weights, asynchronous scheduling and no speculative decoding.
- With GAISSA
- Prompt-lookup speculative decoding with up to five proposed tokens and synchronous scheduling. The BF16 weights were retained.
- Hardware
- NVIDIA GeForce RTX 4090, 24 GB.
- Workload
- 128 requests per setup and load, with 256 input tokens and 128 output tokens. Four loads: 1, 8, 16 and 32 simultaneous requests.
- Runs and reporting
- Five paired repetitions at each load, with the run order alternated. Savings are the median paired percentage reduction at each load.
The before-and-after chart shows the pair producing the median energy reduction at one simultaneous request. Both setups generated 16,384 output tokens. Energy measures the GPU board, not whole-system electricity use.
Response times increased by 8–21% across the tested loads. The selected configuration met the comparison’s response-time limits.
What could improve in your AI?
Tell us what you run and what you want to improve.