Optimized for Engagement

Seamless coaching, measurable change

What the Research Says

A landmark 2025 IEEE study at UC San Diego Health followed 19,500 employees over 8 months to measure the real-world effectiveness of phishing awareness training.

The findings challenge everything the industry assumes—and the engagement problem applies to security training broadly.

← Back to Overview

Key Findings

The numbers that should concern every organization

0%
Impact from annual training
(P = 0.06, not significant)
1.7%
Absolute improvement
from embedded training
37-51%
Training sessions with
zero engagement
24%
Users who actually
complete training

📄 Primary Research Citation

Ho, G., Mirian, A., Luo, E., Savage, S., Voelker, G. M., & Politz, J. (2025). Understanding the Efficacy of Phishing Training in Practice. Proceedings of the 46th IEEE Symposium on Security and Privacy (S&P Oakland 2025), San Francisco, CA, USA.

Read the full paper →

Key Research Findings

Why the status quo doesn't work

  • 🚫
    Annual security awareness training shows NO significant correlation with reduced phishing susceptibility. Employees who completed annual training performed no better than those who hadn't—regardless of how recently they took it.
  • 📉
    Embedded "teachable moment" training provides only marginal improvement (9.5% relative, 1.7% absolute). Even when training is delivered immediately after a phishing click, the improvement is dwarfed by the effectiveness of good phishing lures.
  • ⏱️
    37-51% of training sessions show ZERO seconds of engagement. Users click "acknowledge" or close the training instantly without reading any content.
  • ⚠️
    Static training may actually INCREASE failure rates by 18.5%. Users who received multiple static (non-interactive) training sessions performed worse, not better.
  • ✅
    Interactive training completed in full shows meaningful improvement (19% reduction). The only modality that worked—but only 24% of users actually complete it.

The Core Problem: Zero Engagement

Why 37-51% of phishing training sessions have literally zero impact

❌ Traditional Training Flow

1. User clicks phishing link
2. Redirect to training page
3. Click "Acknowledge" button
4. Done. (Zero learning)

The "Acknowledge" button is the root cause. Users skip through instantly—37-51% spend zero seconds on content.

✓ Catpilot Flow

1. User clicks phishing link
2. Slack DM arrives in 60 seconds
3. "What made that email suspicious?"
4. User responds → AI grades → follow-up
5. Comprehension verified before completion

Zero friction, 3 minutes total. No portal, no videos—just a quick Slack conversation that proves understanding before marking complete.

Platform Comparison

See where Catpilot leads on engagement—the root cause of training failure

Capability KnowBe4 Proofpoint Catpilot
⚠️ Engagement — The Root Cause of Training Failure
Requires real responses (no skip button) ❌ ❌ ✅
Forces multi-turn dialogue ❌ ❌ ✅
Verifies comprehension before completion ❌ Quiz only ✅
AI-graded free-text responses ❌ ❌ ✅
Adapts follow-up based on responses ❌ ❌ ✅
Tracks time-on-page ✅ ✅ ✅
Training Delivery
Annual security awareness modules ✅ ✅ ❌
Embedded post-click training ✅ ✅ ✅
Delivers via Slack/Teams (no portal) ❌ ❌ ✅
Phishing Simulation
Phishing simulation campaigns ✅ ✅ Integrates
Template library ✅ ✅ Integrates
Coaching within 60 seconds of click Redirect Redirect ✅
Developer Security (AppSec)
Secret detection coaching ❌ ❌ ✅
SAST/DAST finding coaching ❌ ❌ ✅
AI Guardrails (Copilot/Cursor rules) ❌ ❌ ✅
GitHub/GitLab integration ❌ ❌ ✅
Integration
Works alongside existing tools ❌ ❌ ✅
Vanta/Drata compliance sync ✅ ✅ ✅

Highlighted rows indicate capabilities directly addressing engagement failures identified in the research.

Catpilot works alongside your existing security tools. See how it works →

Product Overview Contact Us

References

[1] Ho, G., et al. (2025). "Understanding the Efficacy of Phishing Training in Practice." IEEE S&P Oakland 2025. PDF →

[2] Lain, D., et al. (2022). "Phishing in Organizations: Findings from a Large-Scale and Long-Term Study." IEEE S&P 2022. PDF →

[3] Caputo, D. D., et al. (2014). "Going Spear Phishing: Exploring Embedded Training and Awareness." IEEE Security & Privacy. PDF →

[4] Franz, A., et al. (2021). "SoK: Still Plenty of Phish in the Sea." SOUPS 2021. PDF →