Keeping track of Conditional Access changes

Conditional Access is an amazing feature within Azure-AD and is more or less the zero trust engine in the Microsoft 365 platform.
It lets us gather a lot of signals from the end-users sign-in process to decide how they should access the company data.
We can for example take decision based on location, device type, device os, device compliance status, application, sign-in risk etc!

One of the drawback with Conditional Access is the “Governance” capabilities. We simply do not have any good built-in feature to track down changes made to the existing policies.
I would love to see the same feature as we have on a security group, where we have the possibility to just check the audit logs related to the specific object.
Microsoft is currently working on this, please check out the user voice here

Why is it important to be able to track down changes?

Lets face it, we are all humans, and in an large enterprise environment, there might be a couple of administrators that have access to the Conditional Access policies.
All it takes is a simple miss-click when you are editing an policy, and you might lock-out a specific subset of users from the environment or simply disable an protection for a large portrait of your end-users.
This happened to an organization that reached out to me, this small miss-configuration enabled a small portion of their users to sign-in using legacy authentication, just a couple of days after the miss-configuration they had a breach due to a password spray attack.

How?

We will create a couple of Azure Monitoring alerts, based on KQL queries and the Azure-AD Audit logs that will alert us when an change have been detected to an Conditional Access policy.
This will hopefully help you to keep track of all your administrators changes, and hopefully you can detect an miss-configuration quite fast.
Please be aware that this requires that you have configured your Azure-AD logs to stream to Azure Monitor (Log analytics).
I have a blog post about it here

Creating alert within Azure Monitor

We will create a really simple alert, that will generate an alert on the following actions:
Creation of a new Conditional Access policy
Deletion of an Conditional Access policy
Changes to any current Conditional Access policy

1.Sign-in to the Azure-Portal
2.Open Monitor
3.Go to Logs
4.Enter the query and run it, to see that you get the correct data from the query

AuditLogs
| where LoggedByService == "Conditional Access"
| project
    ActivityDateTime,
    InitiatedBy.user.userPrincipalName,
    TargetResources[0].displayName,
    ActivityDisplayName

Example output:

Now we need to configure an alert, so we can get notifications when this kind of actions have been detected!
5. Click on “New alert rule”

6. Now we need to adjust the alert condition, click on the condition as the picture below

7. Configure the Alert logic, Number of results Greater Than 0 and configure the Period and Frequency to your needs

8. Now we will configure the Actions, add a new action group

9. Click on “Create action group”
10. Configure your Action group name and Display Name, then go to notifications

11. Configure your notification type, in this case email

12. We will not configure any actions on this action group, go ahead and click review and create

13. Now configure your alert details and then click on create alert

The alert should now be live, and you should get alerts when anyone creates, deletes och change any Conditional Access Policy

Okey, we are tracking policy changes now, how about exclusions of end-users?

To make it easier for administrators, I use Identity Governance, and more precise the Access review functionality for analyzing end-users exclusions from Conditional Access policies.
You can learn about it in one of my recent blog posts “Tracking excluded Conditional Access users with Identity Governance”