Introduction

With a good team, shift work can be enjoyable. However, if you take away the social aspect of working shifts (which remote work largely did), you’re left with all the health consequences and none of the fun. Add to it the overall cost of having a SOC working in shifts and the management overhead it brings, and you’ll have more than enough reason to automate out-of-business-hours security monitoring.

In this article I describe a way of automating alert escalations in Microsoft Sentinel that worked for me in multiple production environments and is both inexpensive and reliable. I will try to make it as simple as possible so that even if you have no prior Azure experience you should be able to follow the outlined steps.

Let’s dive in.

Requirements

You’ll need:

• An existing Microsoft Sentinel instance
• An API key from a mobile marketing platform of your choice (I suggest Infobip)
• A phone number from the same platform
• Two analytics rules in Microsoft Sentinel
• One automation rule in Microsoft Sentinel
• One Logic App
• A list of alerts that you always want to be escalated

High level overview

At a high level, this automation is simple:

• A Microsoft Sentinel incident is generated.
• An automation rule in Microsoft Sentinel triggers a playbook (Logic App).
• The Logic App retrieves the alert details from Microsoft Sentinel and calls the Infobip API.
• The Infobip API makes a phone call to the on-call analyst and reads the alert details aloud.
• At the same time, the Infobip API sends a text message to the SOC manager to inform them that an alert has been escalated.

This solution ensures you have a reliable way to trigger an outbound phone call whenever a Microsoft Sentinel incident is generated. The more challenging part is determining which alerts should be escalated.

What to escalate

As a T1 SOC analyst there are dozens of Standard Operating Procedures you follow but if an alert that isn’t an obvious false positive is triggered in your SIEM after office hours, those SOPs largely boil down to two options:

1. Escalate to a senior analyst: usually by making a phone call or using apps like PagerDuty.
2. Put on-hold until the next working day: T1 analyst adds context to the alert and either resolves it or puts the alert on hold until the next working day when it can be picked up by a more senior analyst.

If we can clearly define what should be escalated and what shouldn’t to our SOC analysts, we should also be able to create similar instructions for our automation.

Let’s with listing all alerts that require an immediate escalation. We will call it a list of ‘known bad’ alerts.

We still need to account for the following:

• our SOPs may not be great
• we may get a previously unseen alert for which there is no SOP yet can
• we may receive a high number of alerts that in isolation aren’t a cause for escalation, but the sheer number of them is suspicious

We can package those concerns into logic, that could look like this:

An alert comes to your SIEM queue after office hours:

This logic can potentially still miss alerts, but so can people. A good sign that your automation is ready for production is when you let it run for a month next to your SOC (without the phone call component) and it misses fewer escalations than your analysts.

A reliable way of programmatically making a phone call

To make a phone call we will be using a mobile marketing platform. It’s more reliable that Microsoft Teams and has two advantages:

• the typical volume of marketing calls is huge so the per-call price is extremely low
• those apps often integrate with Azure Logic Apps

After testing some of the platforms I chose Infobip as, to me, it is the most intuitive and transparent when it comes to pricing.

Creating the automation

We will need to created one Azure Logic App to handle phone calls and two analytics rules in Microsoft Sentinel. That’s it.

1. Get an API key from Infobip.

Go to portal.infobip.com and in Developer Tools select ‘API Keys’ and then ‘CREATE API KEY’. In the form give your API key a name and select ‘Public API’.

You will be taken to a different screen, where you can copy your API key:

2. Buy a phone number from Infobip

Now you need to buy a phone number. Go to Channels and Numbers and select ‘BUY NUMBER’. In the next UI you will be able to select a country where your phone number is based and capabilities such as VOICE, MMS or SMS. Select both VOICE and SMS:

Perfect. We have both the API key and phone number.

3. Create a Logic App:

Go to portal.azure.com and then type Logic apps in the search bar.

Pick a Subscription and Resource Group for the Logic App, name it in line with your naming policy (if you don’t have one I suggest this: Define your naming convention — Cloud Adoption Framework | Microsoft Learn) and select the ‘Consumption’ tier:

Select ‘Create’ and after a while your Logic app will be created.

4. Define connectors and actions

a) Microsoft Sentinel incident connector

Go to the Logic app designer and select ‘Blank logic app’. You should see a blank screen with a search bar for connectors and triggers. Type Microsoft Sentinel and from the list select ‘Microsoft Sentinel incident’

You will be asked to authenticate the connection. Choose to authenticate with a sign-in (there are other, better ways to authenticate but we’re keeping things simple).

b) Infobip connector

As the next step in the workflow select ‘Infobip’ as the connector and then ‘Make a voice call’ as the action. You will be asked to authenticate with the Infobip connector.

Use the credentials to the Infobip portal, and select ‘Create’.

c) Make a voice call action

From the Infobip connector select ‘Make a voice call’ action.

You will see a form with 4 options:

• language: choose what works for you, for me its ‘en’
• message: ‘An alert has been escalated to you. The name of the alert is triggerBody()?[‘object’]?[‘properties’]?[‘title’]’
• Recipient’s phone number: phone number of the person doing on-call duty
• Caller’s phone number: the number you bought via Infobip

Instead of pasting the ‘triggerBody()~’ string, you can simply click on the ‘Message’ field and select ‘Incident Title’ from the ‘Dynamic content’ tab like in the screenshot above.

d) Send a text message (SMS) action

Create a new step, select Infobip connector and then pick ‘Send a text message (SMS)’ action.

You will see a form with 3 options:

• Message: ‘An alert has been escalated to the on-call analyst. The name of the alert is triggerBody()?[‘object’]?[‘properties’]?[‘title’]’
• Recipient’s phone number: phone number of the person you want to send the text message to
• Sender’s phone number: the number you bought via Infobip

Before moving on, it makes sense to test if the app works as intended. Let’s now jump to Microsoft Sentinel, go to ‘Incidents’ and create an incident manually. Right-click on Incident and select ‘Run playbook’:

Select your newly created Logic app from the list, click ‘Run’, wait a moment and see if you’re getting a phone call and a text message.

Perfect. Now that we have a simple, 3-step Logic app capable of handling outbound phone calls and text messages we need to create analytics rules to trigger the app.

4. Create analytics rules

Go to Microsoft Sentinel and then ‘Analytics’. Click ‘Create’ and then ‘Scheduled query rule’.

a) Escalations for high severity + known bad alerts

Logic: high severity alerts or alerts from a list triggered outside of business hours or on a weekend

let Saturday = time(6.00:00:00);
let Sunday = time(0.00:00:00);
let knownBadAlerts = dynamic(['Serious alert 1', 'Serious alert 2']); //list alerts you always want escalated
SecurityIncident
| summarize arg_max(IncidentName, *) by IncidentNumber // remove duplicates
| where hourofday(TimeGenerated) !between (9..17) or //outside of business hours
dayofweek(TimeGenerated) in (Saturday, Sunday) //or on weekends
| where Title !startswith '[ON-CALL]' //avoiding alert loops
| where Title in(knownBadAlerts) or Severity == 'High'

Analytics rule creation settings: in the ‘General’ tab name the analytics rule whatever you want, I suggest this:

In the ‘Set rule logic’ portion paste the KQL from the above snippet into the ‘Rule query’ section. Then select the rule to run every 5 minutes and lookup data from the last 5 minutes. In ‘Alert details’ specify the following:

• Alert Name Format: [ON-CALL] {{Title}}
• Alert Description Format: {{Description}}

Scroll down to ‘Event grouping’ and select ‘Trigger an alert for each event’

You can now create the rule.

Test: our new rule should trigger out-of-business-hours if there is a new High severity incident generated in Microsoft Sentinel or if the incident’s title is either ‘Serious alert 1’ or ‘Serious alert 2’. Let’s then manually create two incidents — one with title ‘Random name’ and a High severity, one with an Informational severity and title ‘Serious alert 1’:

Perfect. Our analytics rule works as intended.

b) Escalations for multiple medium or low severity alerts

Logic: more than 10 medium or low severity alerts over a period of 30 minutes triggered outside of business hours or on a weekend

let Saturday = time(6.00:00:00);
let Sunday = time(0.00:00:00);
SecurityIncident
| summarize arg_max(IncidentName, *) by IncidentNumber // remove duplicates
| where hourofday(TimeGenerated) !between (9..17) or //outside of business hours
dayofweek(TimeGenerated) in (Saturday, Sunday) //or on weekends
| where Severity in ('Low', 'Medium')
| where Title !startswith "[ON-CALL]"
| summarize count()
| where count_ > 10
| project strcat("the number of incidents exceeded "
, count_, ' in the last 30 minutes')

Analytics rule creation settings: in the ‘General’ tab call the analytics rule whatever you want, I suggest this:

In the ‘Set rule logic’ part paste the KQL from the above snippet into the ‘Rule query’ section. Select the rule to run every 30 minutes and lookup data from the last 30 minutes. In ‘Alert details’ specify the following:

• Alert Name Format: [ON-CALL] {{newTitle}}
• Alert Description Format: This escalation was caused by multiple low or medium security alerts being triggered within 30 minutes.

Having done that feel free to create the rule.

Test: this rule should trigger out-of-business-hours if we create at least 10 medium or low severity alerts in the span of 30 minutes. Let’s then manually create more than 10 Medium severity alerts and verify if the rule works as intended.

Perfect. Both of our rules work. The only thing that’s left is to hook them up to our Logic app and we’re set.

5. Create an automation rule

We have a Logic app that handles phone calls, we also have two analytics rules that trigger when an alert needs to be escalated. We will now build a bridge between them so that the Logic app can be executed automatically when an alert starting with the keyword [ON-CALL] is created. To do so, go to Microsoft Sentinel and then to ‘Automation’. Select ‘Create’ and from the drop down pick ‘Automation rule’. You will have to fill in the following fields:

• Automation rule name: Escalation
• Trigger: When incident is created
• Conditions: incident provider = all, analytics rules name contains all, title starts with [ON-CALL]
• Actions: Run playbook — our playbook name

Done. Our automation is now functional.

Conclusion

By following this article you can create a simple Logic App that handles out of business hours alert escalations. It may lack features compared to paid alternatives, but its cheap and reliable. I hope you can make use of this automation as well and if you do use is in a production environment — let me know on LinkedIn.