Hands-on 1. Data Discovery using Azure REST API.ipynb

Kernel: Python 3.8 - AzureML

Hands on - Data Discovery using Azure REST API

Notebook Version: 1.0
Python Version: Python 3.8 - AzureML
Required Packages: No
Platforms Supported: Azure Machine Learning Notebooks

Data Source Required: Log Analytics tables

Description

This notebook will provide step-by-step instructions and sample code to guide you through Azure authentication, Microsoft Sentinel data discovery by using Azure REST API.
*** No need to download and install any other Python modules. ***
*** Please run the cells sequentially to avoid errors. ***

Table of Contents

1. Warm-up

2. Azure Authentication

3. List Microsoft Sentinel Watchlists Using API

4. List Microsoft Sentinel Incidents Using API

1. Warm-up

# If you need to know what Python modules are available, you may run this:
# help("modules")

# Loading Python libraries
from azure.identity import AzureCliCredential

import requests
import json
import pandas
from IPython.display import display, HTML, Markdown

# Functions will be used in this notebook
def read_config_values(file_path):
    "This loads pre-generated parameters for Microsoft Sentinel Workspace"
    with open(file_path) as json_file:
        if json_file:
            json_config = json.load(json_file)
            return (json_config["tenant_id"],
                    json_config["subscription_id"],
                    json_config["resource_group"],
                    json_config["workspace_id"],
                    json_config["workspace_name"],
                    json_config["user_alias"],
                    json_config["user_object_id"])
    return None

def has_valid_token():
    "Check to see if there is a valid AAD token"
    try:
        error = "Please run 'az login'"
        expired = "AADSTS70043: The refresh token has expired or is invalid"
        failed = "failed"
        validator = !az account get-access-token
        
        if any(expired in item for item in validator.get_list()):
            return '**The refresh token has expired. <br> Please continue your login process. Then: <br> 1. If you plan to run multiple notebooks on the same compute instance today, you may restart the compute instance by clicking "Compute" on left menu, then select the instance, clicking "Restart"; <br> 2. Otherwise, you may just restart the kernel from top menu. <br> Finally, close and re-load the notebook, then re-run cells one by one from the top.**'
        elif any(error in item for item in validator.get_list()) or any(failed in item for item in validator.get_list()):
            return "Please run 'az login' to setup account"
        else:
            return None
    except:
        return "Please login"
        
# Calling Microsoft Sentinel API for List, the same template can be used for calling other Azure REST APIs with different parameters.
# For different environments, such as national clouds, you may need to use different root_url, please contact with your admins.
# It can be ---.azure.us, ---.azure.microsoft.scloud, ---.azure.eaglex.ic.gov, etc.
def call_azure_rest_api_for_list(token, resource_name, api_version):
    "Calling Microsoft Sentinel REST API"
    headers = {"Authorization": token, "content-type":"application/json" }
    provider_name = "Microsoft.OperationalInsights"
    provider2_name = "Microsoft.SecurityInsights"
    target_resource_name = resource_name
    api_version = api_version
    root_url = "https://management.azure.com"
    arm_rest_url_template_for_list = "{0}/subscriptions/{1}/resourceGroups/{2}/providers/{3}/workspaces/{4}/providers/{5}/{6}?api-version={7}"
    arm_rest_url = arm_rest_url_template_for_list.format(root_url, subscription_id, resource_group, provider_name, workspace_name, provider2_name, target_resource_name, api_version)
    response = requests.get(arm_rest_url, headers=headers, verify=True)
    return response

def display_result_name(response):
    "Default to display column - name, you may change it to other columns"
    column_name = "name"
    if response != None:
        entries = [item[column_name] for item in response.json()["value"]] 
        display(entries)

def display_result(response):
    "Display the result set as pandas.DataFrame"
    if response != None:
        df = pandas.DataFrame(response.json()["value"])
        display(df)

# Calling the above function to populate Microsoft Sentinel workspace parameters
# The file, config.json, was generated by the system, however, you may modify the values, or manually set the variables
tenant_id, subscription_id, resource_group, workspace_id, workspace_name, user_alias, user_object_id = read_config_values('config.json');

2. Azure Authentication

# Azure CLI is used to get device code to login into Azure, you need to copy the code and open the DeviceLogin site.
# You may add [--tenant $tenant_id] to the command
if has_valid_token() != None:
    message = '**The refresh token has expired. <br> Please continue your login process. Then: <br> 1. If you plan to run multiple notebooks on the same compute instance today, you may restart the compute instance by clicking "Compute" on left menu, then select the instance, clicking "Restart"; <br> 2. Otherwise, you may just restart the kernel from top menu. <br> Finally, close and re-load the notebook, then re-run cells one by one from the top.**'
    display(Markdown(message))
    !echo -e '\e[42m'
    !az login --tenant $tenant_id --use-device-code

# Extract access token, which will be used to access Microsoft Sentinel Watchlist API for your Watchlist data.  
token = AzureCliCredential().get_token('https://management.azure.com')
access_token = token.token
header_token_value = "Bearer {}".format(access_token)

3. List Microsoft Sentinel Watchlists Using API

# Calling Microsoft Sentinel Watchlist API
# If you don't have Watchlist, you may create one, or try to access different features, such as Bookmarks.
response_watchlist = call_azure_rest_api_for_list(header_token_value, "watchlists", "2023-02-01")

# Display the result
display_result_name(response_watchlist)

4. List Microsoft Sentinel Incidents Using API

# Calling Microsoft Sentinel Incident API
# If you don't have incidents, you may create one through Azure Portal.
response_incident = call_azure_rest_api_for_list(header_token_value, "incidents", "2020-01-01")

# Display the result
display_result(response_incident)

Thanks for coming along all the way to the end. In the next Hands-on notebook, I will show you how to access data using Azure SDK for Python. And keep one of the watchlist name, it will be used in the next notebook. A la prochaine.