Guided Hunting - Anomalous Process Network Connections.ipynb

Kernel: Python 3.8 - AzureML

Guided Hunting - Anomalous Process Network Connections

Details... **Python Version:** Python 3.8 (including Python 3.8 - AzureML)
**Required Packages**: msticpy, pandas, numpy, matplotlib, plotly, ipywidgets, ipython, sklearn

Data Sources Required:

• Log Analytics - DeviceNetworkEvents

Brings together a series of queries and visualizations to help you investigate anomalous processes in your network. There are then guided hunting steps to investigate these occurences in further dept. This notebook authenticates with environment variables and requires the following:

• msticpyconfig.yaml has been properly configured

• Registered application has been created with API permissions given to Log Analytics API

• Key vault set up with a secret to the Registered Application

Setup Environment Variables

Please set the following environment variables in the code block below:

• AZURE_TENANT_ID

• AZURE_CLIENT_ID

• key_vault_name

• key_vault_url

• secret_client

import os
import msticpy as mp
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
from azure.mgmt.resource import ResourceManagementClient

# Set environment variables for tenant ID and client ID
os.environ['AZURE_TENANT_ID'] = ''
os.environ['AZURE_CLIENT_ID'] = ''

# Initialize DefaultAzureCredential
credential = DefaultAzureCredential()

# Create a SecretClient to interact with the Key Vault
key_vault_name = ""
key_vault_url = f""
secret_client = SecretClient(vault_url=key_vault_url, credential=credential)

# Retrieve the secret from Key Vault
secret_name = ""
retrieved_secret = secret_client.get_secret(secret_name)
os.environ['AZURE_CLIENT_SECRET'] = retrieved_secret.value

# Now you can use DefaultAzureCredential or other credential classes
print(credential)

Verify Environment Variables are Set

You should see the values of the following:

• AZURE_TENANT_ID

• AZURE_CLIENT_ID

• AZURE_CLIENT_SECRET

# Verify that the environment variables have been set
def verify_env_vars():
    tenant_id = os.getenv('AZURE_TENANT_ID')
    client_id = os.getenv('AZURE_CLIENT_ID')
    client_secret = os.getenv('AZURE_CLIENT_SECRET')
    
    if tenant_id and client_id and client_secret:
        print("Environment variables have been set successfully:")
        print(f"AZURE_TENANT_ID: {tenant_id}")
        print(f"AZURE_CLIENT_ID: {client_id}")
        print(f"AZURE_CLIENT_SECRET: {client_secret[:4]}... (hidden for security)")
    else:
        print("Failed to set environment variables.")

# Call the verification function
verify_env_vars()

Setup msticpyconfig.yaml

Ensure your msticpyconfig.yaml has been set up and saved in the current directory you are running this notebook.

import msticpy
from msticpy.config import MpConfigFile, MpConfigEdit
import os
import json
from pathlib import Path

mp_conf = "msticpyconfig.yaml"

# check if MSTICPYCONFIG is already an env variable
mp_env = os.environ.get("MSTICPYCONFIG")
mp_conf = mp_env if mp_env and Path(mp_env).is_file() else mp_conf

if not Path(mp_conf).is_file():
    print(
        "No msticpyconfig.yaml was found!",
        "Please check that there is a config.json file in your workspace folder.",
        "If this is not there, go back to the Microsoft Sentinel portal and launch",
        "this notebook from there.",
        sep="\n"
    )
else:
    mpedit = MpConfigEdit(mp_conf)
    mpconfig = MpConfigFile(mp_conf)
    
    # Convert SettingsDict to a regular dictionary
    settings_dict = {k: v for k, v in mpconfig.settings.items()}
    print(f"Configured Sentinel workspaces: {json.dumps(settings_dict, indent=4)}")

msticpy.settings.refresh_config()

Setup QueryProvider

# Refresh any config items that might have been saved
# to the msticpyconfig in the previous steps.
msticpy.settings.refresh_config()

# Initialize a QueryProvider for Microsoft Sentinel
qry_prov = mp.QueryProvider("AzureSentinel")

Connect to Sentinel

You should see "connected" output after running this code block. Once you are connected, you can continue on with the notebook.

# Get the default Microsoft Sentinel workspace details from msticpyconfig.yaml

ws_config = mp.WorkspaceConfig()

# Connect to Microsoft Sentinel with our QueryProvider and config details
qry_prov.connect(ws_config)

Run Anomaly Detection Script - Anomalous Processes

Change your KQL to reduce your data. Enter the field name you want to run the IsolationForest algorithm on to identify anomalies. This script is set to search for anomalous processes on the network. It is recommended to change the contamination rate to fit your environment. The bigger the environment, the smaller the contamination rate will likely need to be. After you select the "Analyze" button, you can search the data frame with the "Column" and "Value" text widgets. There is an option to graph the top ten most significant anomalies based on "Anomaly Score" with the "Graph Results" button.

from azure.monitor.query import LogsQueryClient
import msticpy as mp
from msticpy.config import MpConfigFile, MpConfigEdit
from azure.identity import ClientSecretCredential
from azure.identity import DefaultAzureCredential
from datetime import timedelta
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import LabelEncoder
import ipywidgets as widgets
from IPython.display import display
import re
import plotly.express as px

# Ensure inline plotting
%matplotlib inline



query_text = widgets.Textarea(
    value="""
    DeviceNetworkEvents
| where TimeGenerated >= ago(1d)
| where isnotempty(InitiatingProcessFileName)
| where ActionType == "ConnectionSuccess"
| where RemoteIPType == "Public"
| where RemoteIP matches regex @"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"
| project TimeGenerated, DeviceName, InitiatingProcessAccountName, InitiatingProcessFileName, LocalIP, RemoteIP, RemotePort
    """,
    placeholder='Enter your KQL query here',
    description='Query:',
    disabled=False
)

# Create a text widget for the field name
field_name_text = widgets.Text(
    value='InitiatingProcessFileName',
    placeholder='Enter the field name for Isolation Forest',
    description='Field:',
    disabled=False
)

# Create a text widget for column search
column_name_text = widgets.Text(
    value='',
    placeholder='Enter column name to search',
    description='Column:',
    disabled=False
)

# Create a text widget for value search
value_text = widgets.Text(
    value='',
    placeholder='Enter value to search in the column',
    description='Value:',
    disabled=False
)

# Create an "Analyze" button
analyze_button = widgets.Button(
    description='Analyze',
    disabled=False,
    button_style='',
    tooltip='Click to run the query',
    icon='search'
)

# Create a "Graph Results" button
graph_button = widgets.Button(
    description='Graph Results',
    disabled=True,  # Initially disabled until data is analyzed
    button_style='',
    tooltip='Click to display the scatterplot',
    icon='bar-chart'
)

# Create a "Search" button
search_button = widgets.Button(
    description='Search',
    disabled=True,  # Initially disabled until data is analyzed
    button_style='',
    tooltip='Click to search the DataFrame',
    icon='search'
)

# Display the text boxes and buttons
display(query_text, field_name_text, column_name_text, value_text, analyze_button, graph_button, search_button)

# Function to extract timespan from KQL query
def extract_timespan(query):
    match = re.search(r'ago\((\d+)([dhms])\)', query)
    if match:
        value, unit = int(match.group(1)), match.group(2)
        if unit == 'd':
            return timedelta(days=value)
        elif unit == 'h':
            return timedelta(hours=value)
        elif unit == 'm':
            return timedelta(minutes=value)
        elif unit == 's':
            return timedelta(seconds=value)
    return None

# Function to run the query
def run_query(query):
    timespan = extract_timespan(query)
    response = qry_prov.exec_query(query=query)
    
    # Convert the response to a Pandas DataFrame
    data = response.to_dict(orient='records')
    df = pd.DataFrame(data)
    
    # Set Pandas option to display all columns
    pd.set_option('display.max_columns', None)

    # Set the maximum column width to None (no truncation)
    pd.set_option('display.max_colwidth', None)
    
    # Get the field name from the text widget
    field_name = field_name_text.value
    
    # Encode the selected field
    le = LabelEncoder()
    df['Outlier'] = le.fit_transform(df[field_name])
    
    
    # Apply Isolation Forest for anomaly detection
    iso_forest = IsolationForest(n_estimators=100, contamination=0.01, random_state=42)  # Adjust contamination as needed
    df['Anomaly'] = iso_forest.fit_predict(df[['Outlier']])

    # Get anomaly scores
    df['Anomaly_Score'] = iso_forest.decision_function(df[['Outlier']])
    
    # Store the DataFrame for later use
    global analyzed_df
    analyzed_df = df
    
    # Display the DataFrame with anomalies
    display(df.head(len(df)))
    
    # Enable the "Graph Results" and "Search" buttons
    graph_button.disabled = False
    search_button.disabled = False

# Bind the run_query function to the analyze button
analyze_button.on_click(lambda x: run_query(query_text.value))

import plotly.express as px

# Function to plot results
import plotly.express as px

# Function to plot results
import plotly.express as px

# Function to plot results
import plotly.express as px

# Function to plot results
def plot_results():
    # Filter anomalies
    anomalies = analyzed_df[analyzed_df['Anomaly'] == -1]
    
    # Sort by Anomaly_Score and select the top 10 most negative scores
    top_anomalies = anomalies.sort_values(by='Anomaly_Score').head(10)
    
    # Create scatter plot
    fig = px.scatter(
        top_anomalies,
        x='TimeGenerated',
        y=field_name_text.value,
        title='Top 10 Most Significant Anomalies Detected',
        hover_data={'LocalIP': True, 'RemoteIP': True, 'RemotePort': True, 'Anomaly_Score': True}
    )
    
    # Update hover template
    fig.update_traces(
        hovertemplate=''.join([
            'TimeGenerated: %{x}<br>',
            'Process: %{y}<br>',  # Correctly reference the y-axis value
            'More Information: %{customdata}<br>',
            
        ])
    )
    
    # Show plot
    fig.show()


# Bind the plot_results function to the graph button
graph_button.on_click(lambda x: plot_results())

# Function to search the DataFrame
def search_dataframe():
    column_name = column_name_text.value
    search_value = value_text.value
    if column_name and search_value:
        search_results = analyzed_df[analyzed_df[column_name].astype(str).str.contains(search_value, na=False)]
        display(search_results)
    else:
        print("Please enter both column name and value to search.")

# Bind the search_dataframe function to the search button
search_button.on_click(lambda x: search_dataframe())

What to do with this Information

Take note of the any of the anomalies that were generated. You can focus on the Top Anomalies from the graph or all of the anomalies from the data frame. A reminder that anything with a field value of "Anomaly = -1" was deemed to be anomalous process generating a successful network connection. You can follow some of the techniques below to investigate these anomalous processes further. In each of the following queries, it ends with "df.head(10)". This displays 10 results. If you want to change that number, just change the number 10 to the desired amount of results you would like to see.

Verify Parent Process

It is common to see a malicious process spawn from normal process. You can check the anomalous processes that were identified to see if there is anything unusual with the parent process of the original anomalous process. Replace process1.exe, process2.exe, and process3.exe with the names of the anomalous processes.

DeviceNetworkEvents
| where InitiatingProcessFileName in ("process1.exe", "process2.exe", "process3.exe")
| where TimeGenerated >= ago(7d)
| where isnotempty(InitiatingProcessFileName)
| where ActionType == "ConnectionSuccess"
| where RemoteIPType == "Public"
| where RemoteIP matches regex @"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"
| where InitiatingProcessParentFileName != InitiatingProcessFileName

query="""
DeviceNetworkEvents
| where InitiatingProcessFileName in ("process1.exe", "process2.exe", "process3.exe")
| where TimeGenerated >= ago(7d)
| where isnotempty(InitiatingProcessFileName)
| where ActionType == "ConnectionSuccess"
| where RemoteIPType == "Public"
| where RemoteIP matches regex @"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"
| where InitiatingProcessParentFileName != InitiatingProcessFileName
| project TimeGenerated, DeviceName, InitiatingProcessAccountName, InitiatingProcessParentFileName, InitiatingProcessFileName, InitiatingProcessSHA1
    """
# Set the maximum column width to None (no truncation)
pd.set_option('display.max_colwidth', None)
df = qry_prov.exec_query(query)
df.head(10)

Check if Process Spawned out of Temp File Path

Attackers commonly use a TEMP folder to spawn malicious processes. Ensure the anomalous process did not spawn out of this direction. Replace process1.exe, process2.exe, and process3.exe with the names of the anomalous processes.

DeviceNetworkEvents
| where InitiatingProcessFileName in ("process1.exe", "process2.exe", "process3.exe")
| where TimeGenerated >= ago(7d)
| where isnotempty(InitiatingProcessFileName)
| where ActionType == "ConnectionSuccess"
| where RemoteIPType == "Public"
| where RemoteIP matches regex @"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"
| where InitiatingProcessFolderPath contains_cs "temp"

query="""
DeviceNetworkEvents
| where InitiatingProcessFileName in ("process1.exe", "process2.exe", "process3.exe")
| where TimeGenerated >= ago(7d)
| where isnotempty(InitiatingProcessFileName)
| where ActionType == "ConnectionSuccess"
| where RemoteIPType == "Public"
| where RemoteIP matches regex @"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"
| where InitiatingProcessFolderPath contains_cs "temp"
| project TimeGenerated, DeviceName, InitiatingProcessAccountName, InitiatingProcessFolderPath, InitiatingProcessFileName, LocalIP, RemoteIP, RemotePort
  """
# Set the maximum column width to None (no truncation)
pd.set_option('display.max_colwidth', None)
df = qry_prov.exec_query(query)
df.head(10)

Check if cmd.exe or Powershell was Used

Actors will sometimes use remote code execution with cmd.exe or powershell in coordination with other processes. The following KQL will verify this. Replace process1.exe, process2.exe, and process3.exe with the names of the anomalous processes.

DeviceNetworkEvents
| where InitiatingProcessFileName in ("process1.exe", "process2.exe", "process3.exe")
| where TimeGenerated >= ago(7d)
| where isnotempty(InitiatingProcessFileName)
| where ActionType == "ConnectionSuccess"
| where RemoteIPType == "Public"
| where RemoteIP matches regex @"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"
| where InitiatingProcessCommandLine has_any ("cmd", "powershell", "ps.exe", "cmd.exe")

query="""
DeviceNetworkEvents
| where InitiatingProcessFileName in ("process1.exe", "process2.exe", "process3.exe")
| where TimeGenerated >= ago(7d)
| where isnotempty(InitiatingProcessFileName)
| where ActionType == "ConnectionSuccess"
| where RemoteIPType == "Public"
| where RemoteIP matches regex @"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"
| where InitiatingProcessCommandLine has_any ("cmd", "powershell", "ps.exe", "cmd.exe")
| project TimeGenerated, DeviceName, InitiatingProcessAccountName, InitiatingProcessFileName, LocalIP, RemoteIP, RemotePort
    """
# Set the maximum column width to None (no truncation)
pd.set_option('display.max_colwidth', None)
df = qry_prov.exec_query(query)
df.head(10)