Why Click-Rate Focused Training Has Lost Its Value

Targeted email simulation is a widely adopted initiative among Japanese enterprises. Public sector guidelines and industry practices treat its implementation as a baseline prerequisite. However, many organizations still rely primarily on click-through rates (CTR) and open rates as key performance indicators to measure simulation effectiveness. Meanwhile, modern cyber attacks have evolved from triggering a single human error to exploiting a chain of multiple decisions and actions. Email is merely one of many entry points; attackers now routinely use multiple channels—including chat, cloud services, and phone calls—to deceive targets. In this environment, a decrease in click-through rates does not necessarily translate to an improvement in practical defense capabilities. This white paper does not dismiss targeted email simulations, but aims to re-evaluate their evaluation metrics and positioning. By analyzing the structural limitations of click-through rate-centric evaluation models, we propose a new evaluation perspective focused on the chain of decisions and actions, along with a design framework integrated with SOC and operational security workflows. Ultimately, while CTR will continue to serve as a reference metric, it will no longer remain the core success indicator. Organizations must transition to simulations that verify whether employees can make appropriate decisions under uncertain conditions and take actions that function cohesively at the organizational level. This document provides the framework to guide this redesign.

Targeted Email Simulation as a "Prerequisite"

In Japanese enterprises, targeted email simulation is no longer a measure reserved only for a few forward-thinking organizations. The larger the organization, the higher the adoption rate. Surveys show that approximately 76% of organizations with 1,000 or more employees conduct targeted email simulations at least once a year [1]. This figure clearly demonstrates that email simulation has become established as a standard baseline measure rather than an exception.
This trend is supported by continuous guidance and recommendations from public agencies. IPA positions targeted email simulation as a tool to "verify whether appropriate actions can be taken" and provides guidelines on implementation design, including organizational workflow improvements [2]. These guidelines emphasize that simulations are not merely awareness or educational events, but practical means to validate security operations.

In the financial sector, the Financial Services Agency's "Guidelines for Cybersecurity in the Financial Sector" cite "reporting rates" in targeted email simulations as an example of a KPI [3]. This indicates that evaluating how employees behave and how organizational coordination functions based on simulation results—rather than simply whether a simulation was conducted—is already integrated into regulatory frameworks.
A common thread in these official documents is the absence of debate over whether targeted email simulations should be conducted. They assume implementation as a prerequisite, focusing instead on design methodologies, evaluation criteria, and operational considerations. In other words, targeted email simulations are no longer a subject of debate, but are treated as a foundational control that organizations must possess.

Why Click-Through Rate-Centric Simulations Have Lost Their Value

Based on this premise, failing to conduct simulations or lacking a systematic approach to running them means an organization likely falls short of the standards expected by current guidelines and industry practices. Of course, the presence or absence of simulation alone does not define an organization's defensive posture. However, in an environment where simulations are treated as a baseline prerequisite, not conducting them suggests that the organization remains at a stage prior to discussing broader security controls and operational design.
However, conducting simulations does not guarantee sufficient defensive capability on its own. In fact, many organizations have historically relied on open and click-through rates as KPIs, but these metrics require careful scrutiny regarding how accurately they reflect actual risk reduction and decision quality. The next chapter outlines why these evaluation metrics are being questioned, in light of recent shifts in attack structures.

Key Findings
01 — Standardized Baseline: Targeted email simulation is established as a standard practice among Japanese enterprises, implemented by approximately 76% of organizations with 1,000 or more employees. Public guidelines treat its implementation as a baseline prerequisite.
02 — Multi-Stage Attack Evolution: Modern attacks have shifted from isolated emails to exploiting a "chain of decisions" across multiple channels, including chat and phone calls. Click-through rates alone cannot measure practical defense capabilities.
03 — Decision and Action Chain Evaluation: Real defense relies on "how employees act after detection and how the organization responds." Evaluation must shift from simple opens or clicks to measuring whether employees notice anomalies before compromise and report them internally.

Related Articles

For details on linking evaluation metrics to simulation design, see Targeted Email Simulation Procedures and Effectiveness Measurement.

To learn about dividing roles to connect employee reports to organizational response, see Initial Incident Response and SOC Integration.

For incorporating non-email channels into your simulations, see How to Run Simulations Targeting Email, SMS, and Phone.