One misconfigured Conditional Access policy locked a Fortune 500 company’s 10,000 employees out for four hours. Another organisation spent eight days and over 60 support calls fighting Microsoft to regain admin access after a phishing-resistant MFA policy went sideways. With enterprise downtime costing $300,000 to $1 million per hour, deploying untested Conditional Access (CA) policies is a gamble no platform team should take. The solution: a PowerShell-based testing framework that simulates every sign-in scenario before a single policy hits production.
Microsoft’s What If Evaluation API, now available on the Graph v1.0 endpoint, lets you programmatically test CA policies against any combination of user, device, location, and risk conditions. Combined with report-only mode, the Maester testing framework, and a CI/CD pipeline, you can build a zero-downtime policy deployment workflow that satisfies SOC 2 and ISO 27001 auditors while keeping your users productive.
The What If API changes everything about policy testing
The old What If tool in the Entra portal was useful but manual. The Graph API version (POST /v1.0/identity/conditionalAccess/evaluate) turns policy testing into an automatable, scriptable operation. The PowerShell cmdlet Test-MgIdentityConditionalAccess from the Microsoft.Graph.Identity.SignIns module wraps this API cleanly.
Here’s a production-ready test that simulates a high-risk Android sign-in to SharePoint:
# Requires: Microsoft.Graph.Identity.SignIns v2.6+
# Permission: Policy.Read.All (application or delegated)
Connect-MgGraph -Scopes 'Policy.Read.All'
$params = @{
signInIdentity = @{
"@odata.type" = "#microsoft.graph.userSignIn"
userId = "15dc174b-f34c-4588-ac45-61d6e05dce93"
}
signInContext = @{
"@odata.type" = "#microsoft.graph.applicationContext"
includeApplications = @("00000003-0000-0ff1-ce00-000000000000")
}
signInConditions = @{
devicePlatform = "android"
clientAppType = "browser"
signInRiskLevel = "high"
country = "US"
ipAddress = "40.77.182.32"
deviceInfo = @{ isCompliant = $false }
}
appliedPoliciesOnly = $false # Return ALL policies with evaluation status
}
$results = Test-MgIdentityConditionalAccess -BodyParameter $params
$results | Where-Object { $_.policyApplies -eq $true } |
Format-Table displayName, state, @{
N='Controls'; E={$_.grantControls.builtInControls -join ', '}
}The API evaluates every enabled and report-only policy in your tenant. Each result includes a policyApplies boolean and analysisReasons explaining exactly why a policy matched or didn’t, with values like users, application, platform, or notEnoughInformation. Set appliedPoliciesOnly to $false to see your full 195-policy landscape (the per-tenant limit) in a single call.
One critical gotcha: the documented minimum permission Policy.Read.ConditionalAccess may not work for app-only authentication. Use Policy.Read.All as your baseline. And unlike the legacy portal tool, you cannot use bundle targets like “Office 365”, you must specify individual application IDs.
Build regression tests with Maester in under an hour
Raw API calls verify individual scenarios. The open-source Maester framework (maester.dev) transforms those scenarios into repeatable Pester-based regression tests that run in CI/CD. Built by Microsoft PM Merill Fernando and security MVPs Fabian Bader and Thomas Naunheim, Maester ships with 20+ Conditional Access tests out of the box and supports the What If API natively through Test-MtConditionalAccessWhatIf.
Install-Module Maester
Describe "Contoso CA Regression Suite" {
It "SharePoint access requires MFA for all users" {
$userId = (Get-MgUser -UserId '[email protected]').Id
$result = Test-MtConditionalAccessWhatIf -UserId $userId `
-IncludeApplications '67ad5377-2d78-4ac2-a867-6300cda00e85'
$result.grantControls.builtInControls | Should -Contain "mfa"
}
It "Azure portal blocked for non-admin users" {
$userId = (Get-MgUser -UserId '[email protected]').Id
$result = Test-MtConditionalAccessWhatIf -UserId $userId `
-IncludeApplications 'c44b4083-3bb0-49c1-b47d-974e53cbdf3c'
$result.grantControls.builtInControls | Should -Contain "block"
}
}Wire this into our Azure DevOps pipeline optimisation patterns that run nightly with workload identity federation, no stored secrets, and you catch policy drift before it becomes a security gap. The Conditional Access Validator plugin by Jasper Baes takes this further by auto-generating Maester tests from your current policy configuration, producing flow charts and persona reports that make audit conversations painless.
The deployment pipeline that eliminates lockouts
The organisations that get locked out share a pattern: they skip report-only mode, forget break-glass exclusions, or deploy directly in the portal without version control. Here’s the pipeline architecture that prevents all three.
Store policies as JSON in Git. Export every CA policy via Get-MgIdentityConditionalAccessPolicy -All, serialise to JSON, and commit. Every change now has a diff, a reviewer, and a rollback path via git revert.
Deploy through CI/CD, never the portal. A GitHub Actions or Azure DevOps pipeline validates naming conventions, runs Maester tests against the What If API, then deploys via New-MgIdentityConditionalAccessPolicy or Update-MgIdentityConditionalAccessPolicy. The pipeline always deploys new policies in enabledForReportingButNotEnforced state first.
Monitor report-only impact for 1-2 weeks minimum. Microsoft’s own managed policies stay in report-only for 45 days before auto-enabling. Use this KQL query in Log Analytics to find users who would be blocked:
SigninLogs
| mv-expand ConditionalAccessPolicies
| where ConditionalAccessPolicies["result"] == "reportOnlyFailure"
| project TimeGenerated, UserPrincipalName, AppDisplayName,
PolicyName = ConditionalAccessPolicies["displayName"]
| summarize BlockCount = count() by UserPrincipalName, PolicyName
| sort by BlockCount descPromote to enforced only after validation. Update the JSON state to enabled, commit, and let the pipeline handle enforcement. If something goes wrong, Microsoft’s new soft-delete feature retains deleted policies for 30 days, and git revert restores any previous configuration in minutes.
Guard rails every enterprise needs
Break-glass accounts are non-negotiable. Maintain at least two cloud-only emergency access accounts on your onmicrosoft.com domain, excluded from every CA policy, with FIDO2 keys stored in a fireproof safe. Test them quarterly. After Microsoft’s mandatory MFA enforcement for admin portals, these accounts need registered FIDO2 or certificate-based authentication, CA policy exclusion alone is no longer sufficient.
Alert on every policy change using this KQL query in Azure Monitor to catch modifications in near-real-time:
AuditLogs
| where OperationName in (
"Add conditional access policy",
"Update conditional access policy",
"Delete conditional access policy")
| extend Actor = tostring(parse_json(
tostring(InitiatedBy.user)).userPrincipalName)
| extend PolicyName = tostring(TargetResources[0].displayName)Don’t forget the compliance angle. SOC 2 control CC8.1 requires documented change management with testing before deployment. ISO 27001:2022 control A.8.9 explicitly mandates configuration management policies for security-affecting changes. A Git-based CA policy pipeline with Maester tests and report-only monitoring satisfies both frameworks with an auditable trail that maps directly to control objectives, following the same zero-trust security principles we’ve outlined for enterprise Azure environments.
What to build this week
Start small. Export your current policies to JSON and commit them to a private repo, that alone gives you backup and audit trail. Add the What If API calls for your three highest-risk scenarios: admin portal access, legacy authentication attempts, and non-compliant device sign-ins. Wire Maester into a scheduled GitHub Action. Within a sprint, you’ll have a testing framework that turns CA policy changes from “hope it works” into “prove it works.”
Microsoft blocks 4,000 password attacks every second. Your Conditional Access policies are the front line. Test them like it matters, because a policy that works in theory but locks out your CFO on earnings day is worse than no policy at all.
Useful Links
Microsoft Graph What If API Reference – Official documentation for the conditionalAccess/evaluate endpoint with request/response examples
Maester Framework Documentation – Open-source PowerShell testing framework for Microsoft 365 security with built-in Conditional Access tests
Plan Your Conditional Access Deployment – Microsoft’s comprehensive planning guide including report-only mode, break-glass accounts, and deployment best practices
Emergency Access Account Management – Official guidance on configuring and securing break-glass accounts
Microsoft.Graph.Identity.SignIns Module – PowerShell module documentation for managing Conditional Access policies via Graph API
Conditional Access Report-Only Mode – Microsoft’s documentation on testing policies without enforcement
Conditional Access Policy Templates – Microsoft’s library of pre-built policy templates for common scenarios
Azure Monitor KQL Query Language – Reference documentation for writing queries to monitor Conditional Access events
ISO 27001:2022 Control A.8.9 – Configuration management requirements for security-critical changes
Managing Conditional Access with PowerShell – Practical365 guide on policy automation and CI/CD integration
