An email security tool is removing phishing tests from inboxes

Phishing tests are disappearing from a client’s inboxes. They arrived, then they were gone, often from several users at once and often right after someone reported one. The client runs an email security tool that checks mail after it reaches the mailbox, such as Ironscales, and that tool is removing our tests. For a Microsoft 365 client whose phishing tests go straight into the mailbox, one setting in INFIMA and one allowlist entry in the tool stop it.

Why it happens

These tools work inside the mailbox, after delivery. Ironscales and similar tools connect to the client’s Microsoft 365 or Google Workspace tenant and scan mail that has already arrived. That’s different from a mail gateway, which filters mail on its way in. Our phishing tests for sync’d clients go straight into the mailbox without passing through a gateway (the one exception is users we can’t reach through the sync; see the FAQ below), so a tool that scans the mailbox can still find them and take them out.

One report can remove many tests. When a user reports a phishing test with the tool’s own report button, the tool looks for similar messages across the client’s mailboxes and removes those too. So one report can pull older tests out of every inbox that received them.

Allowlisting the From domains of our tests doesn’t help. Each phishing template sends from its own lookalike domain, and the templates in the client’s rotation change over time. There’s no domain list you could keep up to date, and allowlisting them isn’t safe (see If the tool can only exclude senders). The fix is to allowlist our IP address instead, or the client’s own Return-Path for a tool that only checks senders. The setting below makes both possible.

Confirm the tool is the cause

Open the client’s Sent Emails page, find a phishing test that went missing and click it. If the detail says This email is no longer in the user’s mailbox, the test was delivered and then removed. When that happens to several users around the same time, or right after a report, check the email security tool’s incident or remediation history. The removed tests will be listed there.

For any other missing email, start with A user isn’t getting training emails.

You’ll need

The Administrator role in INFIMA.
Access to the client’s email security tool, or someone at the client who has it.
A Microsoft 365 client. The setting below only changes phishing tests we deliver straight into Microsoft 365 mailboxes.

Stop the tool removing new tests

Open the client → Tailor. On the Phishing card, click Edit.
Turn on Synthetic Received Header. The card is further down the page, below Simulation Frequency. Switch it to On.
Click Save Changes. From now on, every phishing test we deliver straight into this client’s Microsoft 365 mailboxes carries a header with our phishing-test IP address.
Allowlist 34.232.212.184 in the email security tool. Add it wherever the tool lets you exclude phishing simulation traffic by IP address. Menu names differ between tools and versions; the tool’s own documentation calls this something like a phishing simulation allowlist or a trusted IP list. If the client manages the tool, send them the IP address and ask them to add it. If the tool has no IP address option, see If the tool can only exclude senders.

Do both. The setting on its own changes nothing in the tool, and the allowlist entry on its own has nothing to match.

If the tool can only exclude senders

Some tools, such as Barracuda Impersonation Protection, have no IP address option and only let you exclude mail by sender address or domain. For those, allowlist this client’s own Return-Path instead of the IP address:

Turn on Synthetic Received Header and save, as in steps 1 to 3 above.
Copy the client’s Return-Path. Once the setting is saved, the card shows This client’s Return-Path with an address and a Copy button. Click Copy.
Add that exact address to the tool’s sender allowlist. The full address, never its domain. If the client manages the tool, send them the address and ask them to add it.

The address belongs to this client alone, and only phishing tests we deliver straight into the client’s Microsoft 365 mailboxes carry it. Users whose tests show InfimaMail on Sent Emails don’t get it. If the card says the client’s simulations don’t carry a Return-Path to allowlist, contact our support team with the client’s name and the name of the tool.

Don’t add the From addresses of our phishing tests, or their domains, to that list, even if the tool keeps removing tests:

We don’t own those domains. Anyone could use them to send real phishing, and your allowlist would let it through.
Some tests send from personal email services such as Gmail, Yahoo and Proton Mail. Allowlisting those lets in mail from anyone with an account there.
Allowlisted senders skip the tool’s protection. The tool stops checking their links and stops removing their mail, so a real attack from one of those senders would reach users untouched.

The client’s Return-Path is not one of those: it’s on a domain we own and made for this client alone. Add only that exact address.

Clean up tests the tool already removed

The setting only applies to phishing tests sent after you save it. Tests the tool has already removed stay removed until someone puts them back:

Release the removed tests from the tool’s quarantine or remediation view, so they return to users’ mailboxes.
Mark those incidents as a false positive or as a phishing simulation. This keeps the tool from treating the same tests as a threat again.
Confirm they’re back in users’ inboxes, from the tool’s release status or by asking a user to look.

You’ll know it worked when

The Synthetic Received Header card on the client’s Tailor → Phishing page reads On.
New phishing tests stay in users’ inboxes, and the email security tool stops opening incidents for them.
When a user reports a test with the tool’s report button, other users’ copies of it stay put.

Frequently asked

Should I turn this on for every client? No. Turn it on only for clients whose email security tool removes phishing tests after delivery. Other clients don’t need it, so leave it off.

The client is on Google Workspace. Does this work for them? No. The setting only changes phishing tests we deliver straight into Microsoft 365 mailboxes. If a tool is removing tests from a Google Workspace client, contact our support team with the client’s name and the name of the tool.

We added the IP address but tests are still being removed. Check that Synthetic Received Header is On for this client, and that you clicked Save Changes. On the client’s Sent Emails page, check that recent tests show Microsoft API as the method. Tests with any other method didn’t go straight into the mailbox and don’t carry the header. If that’s every user, the client’s sync connection needs fixing first (see A user isn’t getting training emails). If it’s only some users, showing InfimaMail, their mailboxes are ones we can’t reach through the sync, such as on-premises Exchange in a hybrid setup; see Mail gateways. Then check the date of the test that was removed. Tests sent before you saved don’t carry the header, and the tool can still remove them, especially when it clusters older tests after a report. If a test sent after you saved is removed, contact our support team with the client’s name and the test’s subject and date.

Why don’t we need a gateway allowlist for this client? For sync’d clients, our phishing tests go straight into the mailbox, so a gateway never sees them and there’s nothing for it to block. The allowlist entry here goes in the tool that scans the mailboxes, not in a gateway. The one exception is users we can’t reach through the sync (they show InfimaMail on Sent Emails); see Mail gateways.

Users reported the test with the tool’s button. Does that count as a report in INFIMA? No. We only see reports made with our Report Phishing button. To give a user credit for reporting with the tool’s button, open their result and click Mark as Reported. If the tool also opened links in the test while scanning it, a user may have been marked as clicking when they didn’t. A user got a click reminder but says they didn’t click shows how to check and correct it.

Allowlist phishing tests in a client’s email security: what gateways and link scanners need, and why we never allowlist our tests’ From addresses.
A user isn’t getting training emails: use the Sent Email Log to find any missing email.
A user got a click reminder but says they didn’t click: correct results a security tool caused.
Report Phishing button: so users’ reports reach us and count.