What is Targeted Email Simulation? Steps, Costs, Measuring Effectiveness, and Preparing for CEO Fraud

Updated: September 12, 2026. Added and reorganized content regarding training procedures, cost checklists, measurement metrics, and preparation for CEO scams.

What is phishing simulation training?

Phishing simulation training (targeted email simulation) is an initiative where employees practice recognizing, verifying, and reporting suspicious emails using mock messages that mimic business communications or partner requests. It goes beyond measuring actions like link clicks or attachment opens to reinforce reporting channels and correct post-incident behaviors.

The goal is not to find and blame individuals who fall for simulations. Rather, it is to identify where employees struggle with decision-making, verify if they know where to report, and test verification procedures to improve education and business workflows. Sending emails and collecting metrics alone is not enough to achieve this.

Differences between Awareness, Security Education, and Training

• Security Awareness: Initiatives that cultivate an ongoing state of risk recognition and security-conscious behaviors in daily operations.

• Security Education: Learning corporate rules, information handling, and reporting procedures using videos, e-learning, and interactive training.

• Security Training: Putting learning into practice within simulated scenarios, including email simulations and tabletop exercises to practice response coordination.

For example, learning "verify suspicious requests" is education, while practicing whether employees actually verify an urgent request from the CEO is training. NIST SP 800-50 Rev.1 also addresses learning programs aimed at behavioral change and continuous improvement through evaluation. For more on designing education programs, see Approaches to Reviewing Security E-learning.

Steps for Conducting Phishing Simulations

1. Define target actions and audience

Establish specific target actions beyond "do not click," such as "report to internal helpdesk" or "verify request via pre-registered contact methods." Separate general contents from specialized scenarios for finance, HR, or executive assistants, while accounting for user endpoints, language, and working hours.

2. Align with stakeholders and scope

Define the simulation owner, audience, timeframe, support channels, reporting access, data retention periods, and abort conditions. Communicate the training policy and how results will be used. Avoid public ranking or blaming, which discourages reporting.

3. Conduct small-scale validation

Run tests with a limited audience to verify delivery, landing page access, and reporting channels. Ensure your metrics can distinguish automated email gateway inspection from actual user actions. Coordinate required allowlisting settings minimally to avoid opening broad vulnerabilities in production.

4. Monitor delivery and handle inquiries

Monitor delivery issues, unexpected external forwards, and operational impact. There is no need to harvest real passwords or ask for actual fund transfers in simulations. Ensure the helpdesk is prepared to handle user reports during the simulation.

5. Run post-simulation education and plan improvements

Explain the intent of the simulation, indicators of compromise, and proper reporting steps. Provide learning opportunities for both those who clicked and those who noticed but did not report. If the reporting path is unclear, adjust the workflow, not just the learning materials. Record scope and difficulty to compare progress in future simulations.

Measuring Effectiveness: Beyond Click Rates

Below are example definitions for designing your metrics. Since measurement logic varies by vendor, confirm the denominator and exclusion rules before deployment.

• Reporting rate: Percentage of recipients who reported the email using designated channels. Ensure duplicate reports from the same user are deduplicated.

• Time-to-report: Time elapsed from the defined start point (delivery/receipt) to the first report. Standardize handling of non-business hours and do not exclude non-reporters from aggregate metrics.

• Action rate (click/open): Percentage of recipients who interacted with mock links. If automated security scanning cannot be filtered, document this technical limitation.

• Completion and comprehension: Separately monitor training completion rates and quiz scores. Do not treat training completion as proof of secure behavior.

• Verification execution: Use exercises to verify whether secondary validation steps or standard internal approvals were followed.

Email opens depend on image rendering and mail client settings, while link clicks can be skewed by automated link crawlers. A decrease in click rate does not guarantee improved capability if email difficulty or target audiences differ. Ensure undelivered or unmeasurable outcomes are tracked separately from 0% rates.

Preparing for CEO Scams and Executive Spoofing

A CEO scam is a type of Business Email Compromise (BEC) that impersonates executives to demand wire transfers or confidential data. They often omit attachments or malicious links, starting with natural conversational exchanges. IPA's "Be Aware of Scam Emails Impersonating Presidents and Executives!" highlights methods where attackers spoof executives, demand LINE group creation and QR code replies, and follow up with wire transfer requests.

The following examples outline scenarios designed for authorized internal testing. They do not represent actual security incidents or Yagura client cases.

• Accounting/Finance: Use urgent wire transfer or bank detail modification requests to train staff to follow standard verification and approval steps without exception.

• Executive Assistants / Management: Practice validating identities when facing "urgent" or "confidential" demands, without relying on contact details provided in the suspicious email.

• All Employees: Train staff to consult internal security teams before migrating business conversations to external chat applications or LINE.

If a suspicious request is received, contact internal support before taking action. Verify the request using pre-registered contact methods rather than numbers listed in the email. If you have already responded, clicked, or initiated a transfer, report it immediately to your incident response team. For a detailed analysis of these attack vectors, see Attacks Steering Users from Email to LINE and Their Defenses.

Evaluating Cost and Service Selection

A comparison based solely on user count and delivery frequency is insufficient. Pricing models and deliverables vary by provider, so request quotes with standardized requirements.

• Audience metrics: Determine if licensing is based on registered users, active targets, or duration. Confirm handling of additions and employee offboarding.

• Scenarios and delivery: Assess options for simulation frequency, department-specific templates, multi-language support, and pre-delivery checks.

• Education: Check if post-simulation materials, comprehension tests, reminder paths, and content updates are included.

• Analytics and reporting: Verify definitions for reporting rates, handling of automated security scanners, CSV export options, and role-based access control for department managers.

• Operational support: Define who handles deliverability tuning, user inquiries, post-mortem reviews, and strategic planning.

• Data governance: Review data hosting locations, retention, deletion processes, admin privileges, and sub-processor agreements.

During a proof of concept (PoC), operators should run through the entire workflow: preparation, handling user reports, analyzing results, and deploying post-simulation training. Evaluate whether your team can maintain these operations long-term given your headcount.

Frequently Asked Questions

How many simulations should we run?

There is no single baseline for all organizations. Plan schedules around business risk, onboarding cycles, past performance, and your capacity to update training content. Ensure you can act on results from one simulation before increasing frequency.

Does announcing simulations in advance ruin their value?

Communicating simulation policies and objectives is separate from revealing specific schedules or email templates. Design scenarios to match your goals, whether testing baseline awareness or assessing reactions during normal daily operations.

Can email simulations prevent CEO scams?

They cannot guarantee prevention. Simulations must be combined with business procedures like secondary validation, multi-party approvals for wire transfers, and clear reporting paths. Practice confirming the authenticity of requests, including those migrating to phone calls or chat apps.

Finding the Right Training for Your Organization

First, map out your target departments, desired behaviors, current reporting structures, and simulation owners. Yagura Awareness: Phishing Simulations and Security Education introduces our training and education capabilities. To assess viability in your email environment, schedule a session via Training & Education Consultation.