Phi-3.5 Mini
Faster answers.
Lower cost.
Phi-3.5 Mini delivered faster answers with an estimated 60% lower compute cost per response, at the same GPU hourly rate. Answer accuracy was maintained.
What changed
Compressing the model freed memory for handling requests. We increased the space available for request processing and adjusted scheduling, allowing all eight requests to run together. Previously, four ran while four waited.
The same GPU completed 2.5× as many requests per second, spreading its hourly compute cost across more responses.
Performance and resource use
| Measure | Before | With GAISSA |
|---|---|---|
| Requests completedper second | 2.69 | 6.80 |
| Text generatedtokens per second | 343.9 | 870.4 |
| First token95th percentile, ms | 1,627.1 | 183.9 |
| Model memoryat loading, GiB | 7.14 | 3.80 |
| GPU energyjoules per output token | 0.878 | 0.328 |
Answer accuracy
Both versions answered the same 200 questions covering algebra, computer security, computer science and logical reasoning.
| Measure | Before | With GAISSA |
|---|---|---|
| Correct answers | 141 / 200 | 144 / 200 |
| Accuracy | 70.5% | 72.0% |
91% of answers were unchanged. Eight incorrect answers became correct, while five correct answers became incorrect.
Compute-cost calculation and assumptions
At the same GPU hourly rate, completing more requests per second reduces the compute cost of each response. Using the measured request rates gives a 60.5% reduction.
| Measure | Before | With GAISSA |
|---|---|---|
| Relative cost per responsebefore = 100 | 100 | 39.5 |
This is an estimate for a busy GPU, not a measured reduction in a cloud bill. Idle time, other infrastructure and pricing commitments affect the actual bill.
Test setup
- Model
- Microsoft Phi-3.5 Mini Instruct
- Before
- vLLM 0.24.0, BF16 weights, 1 GiB BF16 KV cache and a 2,048-token scheduler budget.
- With GAISSA
- FP8 weights, 2 GiB BF16 KV cache and a 4,096-token scheduler budget, on the same runtime.
- Hardware
- NVIDIA GeForce RTX 4090, 24 GB.
- Workload
- 128 requests per setup, each with 512 input tokens and 128 output tokens. The featured comparison uses eight simultaneous requests.
- Runs and reporting
- One recorded paired performance comparison at each of three request loads (1, 4 and 8). These figures are not an average of repeated benchmark runs.
Quality was checked separately on the same 200 MMLU questions. Model memory is the footprint at loading, not total GPU allocation. GPU energy is board-level energy, not whole-system electricity use.
What could improve in your AI?
Tell us what you run and what you want to improve.