Benchmark Results: Security Mechanism Overhead¶
Timestamp: 2026-02-03T17:58:39.974475 Iterations: 50
Summary¶
| Scenario | Mean (ms) | P95 (ms) | P99 (ms) | Overhead |
|---|---|---|---|---|
| baseline | 7.23 | 8.52 | 163.24 | - |
| baseline_streaming | 10.63 | 11.91 | 12.26 | +47% |
| baseline_conversation | 4.76 | 5.96 | 6.87 | +-34% |
| baseline_with_tools | 6.10 | 9.30 | 12.38 | +-16% |
| typical_production | 712.68 | 725.52 | 726.87 | +9760% |
| typical_production_conversation | 1281.99 | 1883.58 | 3584.79 | +17636% |
| full_security | 244.19 | 257.93 | 297.79 | +3278% |
| full_security_with_tools | 729.14 | 1435.74 | 4797.76 | +9988% |
| streaming_with_output_guard | 206.59 | 218.07 | 233.52 | +2758% |
| customer_support_scenario | 819.28 | 840.40 | 848.70 | +11235% |
| code_assistant_scenario | 835.63 | 921.10 | 933.00 | +11461% |
| data_analysis_scenario | 1454.44 | 2637.72 | 6491.83 | +20022% |
| spotlighting_only | 22.15 | 28.21 | 32.44 | +206% |
| input_guard_only | 192.41 | 209.17 | 215.13 | +2157% |
| input_guard_long_message | 234.69 | 323.71 | 366.07 | +3147% |
| output_guard_only | 225.79 | 237.60 | 239.28 | +3024% |
| system_prompt_check_only | 25.80 | 29.49 | 30.97 | +257% |
| pii_mask | 647.18 | 719.67 | 725.49 | +8854% |
| pii_replace | 708.66 | 719.69 | 721.24 | +9704% |
| pii_encrypt | 714.81 | 722.94 | 777.98 | +9789% |
Detailed Results¶
baseline¶
- Mean: 7.23 ms
- Median: 4.53 ms
- P95: 8.52 ms
- P99: 163.24 ms
- Min: 3.80 ms
- Max: 112.59 ms
- Std Dev: 15.25 ms
- Overhead: -
baseline_streaming¶
- Mean: 10.63 ms
- Median: 10.31 ms
- P95: 11.91 ms
- P99: 12.26 ms
- Min: 10.07 ms
- Max: 12.22 ms
- Std Dev: 0.54 ms
- Overhead: +47%
baseline_conversation¶
- Mean: 4.76 ms
- Median: 4.49 ms
- P95: 5.96 ms
- P99: 6.87 ms
- Min: 3.96 ms
- Max: 6.64 ms
- Std Dev: 0.57 ms
- Overhead: +-34%
baseline_with_tools¶
- Mean: 6.10 ms
- Median: 5.49 ms
- P95: 9.30 ms
- P99: 12.38 ms
- Min: 4.04 ms
- Max: 11.41 ms
- Std Dev: 1.75 ms
- Overhead: +-16%
typical_production¶
- Mean: 712.68 ms
- Median: 716.25 ms
- P95: 725.52 ms
- P99: 726.87 ms
- Min: 641.75 ms
- Max: 726.52 ms
- Std Dev: 16.41 ms
- Overhead: +9760%
typical_production_conversation¶
- Mean: 1281.99 ms
- Median: 1128.46 ms
- P95: 1883.58 ms
- P99: 3584.79 ms
- Min: 1018.11 ms
- Max: 3084.63 ms
- Std Dev: 347.35 ms
- Overhead: +17636%
full_security¶
- Mean: 244.19 ms
- Median: 243.87 ms
- P95: 257.93 ms
- P99: 297.79 ms
- Min: 230.31 ms
- Max: 285.08 ms
- Std Dev: 8.40 ms
- Overhead: +3278%
full_security_with_tools¶
- Mean: 729.14 ms
- Median: 517.84 ms
- P95: 1435.74 ms
- P99: 4797.76 ms
- Min: 443.24 ms
- Max: 3796.59 ms
- Std Dev: 535.07 ms
- Overhead: +9988%
streaming_with_output_guard¶
- Mean: 206.59 ms
- Median: 205.71 ms
- P95: 218.07 ms
- P99: 233.52 ms
- Min: 198.18 ms
- Max: 228.78 ms
- Std Dev: 6.08 ms
- Overhead: +2758%
customer_support_scenario¶
- Mean: 819.28 ms
- Median: 818.94 ms
- P95: 840.40 ms
- P99: 848.70 ms
- Min: 788.92 ms
- Max: 848.04 ms
- Std Dev: 10.99 ms
- Overhead: +11235%
code_assistant_scenario¶
- Mean: 835.63 ms
- Median: 820.01 ms
- P95: 921.10 ms
- P99: 933.00 ms
- Min: 788.55 ms
- Max: 929.35 ms
- Std Dev: 37.99 ms
- Overhead: +11461%
data_analysis_scenario¶
- Mean: 1454.44 ms
- Median: 1745.56 ms
- P95: 2637.72 ms
- P99: 6491.83 ms
- Min: 410.44 ms
- Max: 5337.25 ms
- Std Dev: 899.98 ms
- Overhead: +20022%
spotlighting_only¶
- Mean: 22.15 ms
- Median: 21.26 ms
- P95: 28.21 ms
- P99: 32.44 ms
- Min: 18.33 ms
- Max: 31.27 ms
- Std Dev: 3.09 ms
- Overhead: +206%
input_guard_only¶
- Mean: 192.41 ms
- Median: 190.96 ms
- P95: 209.17 ms
- P99: 215.13 ms
- Min: 180.53 ms
- Max: 213.21 ms
- Std Dev: 6.98 ms
- Overhead: +2157%
input_guard_long_message¶
- Mean: 234.69 ms
- Median: 202.89 ms
- P95: 323.71 ms
- P99: 366.07 ms
- Min: 183.24 ms
- Max: 358.28 ms
- Std Dev: 52.46 ms
- Overhead: +3147%
output_guard_only¶
- Mean: 225.79 ms
- Median: 224.21 ms
- P95: 237.60 ms
- P99: 239.28 ms
- Min: 212.96 ms
- Max: 239.04 ms
- Std Dev: 5.62 ms
- Overhead: +3024%
system_prompt_check_only¶
- Mean: 25.80 ms
- Median: 25.49 ms
- P95: 29.49 ms
- P99: 30.97 ms
- Min: 22.61 ms
- Max: 30.63 ms
- Std Dev: 1.93 ms
- Overhead: +257%
pii_mask¶
- Mean: 647.18 ms
- Median: 615.94 ms
- P95: 719.67 ms
- P99: 725.49 ms
- Min: 599.96 ms
- Max: 724.11 ms
- Std Dev: 48.46 ms
- Overhead: +8854%
pii_replace¶
- Mean: 708.66 ms
- Median: 716.66 ms
- P95: 719.69 ms
- P99: 721.24 ms
- Min: 625.07 ms
- Max: 720.76 ms
- Std Dev: 23.97 ms
- Overhead: +9704%
pii_encrypt¶
- Mean: 714.81 ms
- Median: 717.40 ms
- P95: 722.94 ms
- P99: 777.98 ms
- Min: 652.95 ms
- Max: 760.18 ms
- Std Dev: 14.22 ms
- Overhead: +9789%