Entity Explorer - Domain and URL.ipynb

Kernel: Python 3.10 - SDK v2

Entity Explorer - Domain and URL

Details...

Notebook Version: 2.0
Python Version: Python 3.10 (including Python 3.10 - SDK v2 - AzureML)
Required Packages: msticpy, msticnb

Data Sources Required:

• Log Analytics - SecurityAlerts, Bookmarks, DnsEvents, CommonSecurityLog, DeviceNetworkEvents
  TI Proviers Used

• VirusTotal, Open Page Rank, BrowShot(all required for certain elements), AlienVault OTX, IBM XForce (optional) - all providers require accounts and API keys

This Notebooks brings together a series of tools and techniques to enable threat hunting within the context of a domain name or URL that has been identified as of interest. It provides a series of techniques to assist in determining whether a domain or URL is malicious. Once this has been established it provides an overview of the scope of the domain or URL across an environment, along with indicators of areas for further investigation such as hosts of interest.

Table of Contents

• 1  Hunting Hypothesis:

• 2  Notebook initialization
• 2.1  Authentication
• 2.2  Import & Initalize Notebooklets
• 2.3  Select the domain or URL you wish to investigate

3  URL Overview

4  The URL in the Environmnet

• 4.1  Related Alerts
• 4.2  Related Bookmakrs
• 4.3  Hosts Observed Communicating with the URL
• 4.4  Additional Environment Data

Hunting Hypothesis:

Our broad initial hunting hypothesis is that a particular URL or domain might be malicious.
This notebook is designed to help you explore the data and identify if the URL is malicious and where the URL appears within the environment.

---

Notebook initialization

The next cell:

• Checks for the correct Python version

• Checks versions and optionally installs required packages

• Imports the required packages into the notebook

• Sets a number of configuration options.

More details...

This should complete without errors. If you encounter errors or warnings look at the following two notebooks:

• TroubleShootingNotebooks

• ConfiguringNotebookEnvironment

If you are running in the Microsoft Sentinel Notebooks environment (Azure Notebooks or Azure ML) you can run live versions of these notebooks:

• Run TroubleShootingNotebooks

• Run ConfiguringNotebookEnvironment

You may also need to do some additional configuration to successfully use functions such as Threat Intelligence service lookup and Geo IP lookup. There are more details about this in the ConfiguringNotebookEnvironment notebook and in these documents:

• msticpy configuration

• Threat intelligence provider configuration

from IPython.display import display, HTML, Image
from datetime import datetime, timedelta, timezone


display(HTML("<h3>Starting Notebook setup...</h3>"))

# If not using Azure Notebooks, install msticpy and msticnb with
# %pip install msticpy --upgrade
# %pip install msticnb --upgrade

import msticpy as mp

extra_imports = [
    "msticpy.nbtools, observationlist",
    "msticpy.context, domain_utils",
    "msticpy, nbwidgets",
    # "pyvis.network, Network",
]
mp.init_notebook(
    extra_imports=extra_imports,
    additional_packages=["msticnb>=1.0"],
);

# papermill default parameters
ws_name = "Default"
url = ""
end = datetime.now(timezone.utc)
start = end - timedelta(days=2)

Authentication

Details... If you are using user/device authentication, run the following cell. - Click the 'Copy code to clipboard and authenticate' button. - This will pop up an Azure Active Directory authentication dialog (in a new tab or browser window). The device code will have been copied to the clipboard. - Select the text box and paste (Ctrl-V/Cmd-V) the copied value. - You should then be redirected to a user authentication page where you should authenticate with a user account that has permission to query your Log Analytics workspace.

Use the following syntax if you are authenticating using an Azure Active Directory AppId and Secret:

%kql loganalytics://tenant(aad_tenant).workspace(WORKSPACE_ID).clientid(client_id).clientsecret(client_secret)

instead of

%kql loganalytics://code().workspace(WORKSPACE_ID)

Note: you may occasionally see a JavaScript error displayed at the end of the authentication - you can safely ignore this.
On successful authentication you should see a popup schema button. To find your Workspace Id go to Log Analytics. Look at the workspace properties to find the ID.

print(
    "Configured workspaces: ",
    ", ".join(mp.settings.get_config("AzureSentinel.Workspaces").keys()),
)
import ipywidgets as widgets

ws_param = widgets.Combobox(
    description="Workspace Name",
    value=ws_name,
    options=list(mp.settings.get_config("AzureSentinel.Workspaces").keys()),
)
ws_param

from msticpy.common.timespan import TimeSpan

# Authentication
qry_prov = QueryProvider(data_environment="MSSentinel")
qry_prov.connect(WorkspaceConfig(workspace=ws_param.value))

nb_timespan = TimeSpan(start, end)
qry_prov.query_time.timespan = nb_timespan
md("<hr>")
md("Confirm time range to search", "bold")
qry_prov.query_time

Authentication and Configuration Problems

Click for details about configuring your authentication parameters

The notebook is expecting your Microsoft Sentinel Tenant ID and Workspace ID to be configured in one of the following places:

• config.json in the current folder

• msticpyconfig.yaml in the current folder or location specified by MSTICPYCONFIG environment variable.

For help with setting up your config.json file (if this hasn't been done automatically) see the ConfiguringNotebookEnvironment notebook in the root folder of your Azure-Sentinel-Notebooks project. This shows you how to obtain your Workspace and Subscription IDs from the Microsoft Sentinel Portal. You can use the SubscriptionID to find your Tenant ID). To view the current config.json run the following in a code cell.

%pfile config.json

For help with setting up your msticpyconfig.yaml see the Setup section at the end of this notebook and the ConfigureNotebookEnvironment notebook

Import and initialize notebooklets

This imports the msticnb package and the notebooklets classes.

These are needed for the notebook's operation.

import msticnb as nb

nb.init(query_provider=qry_prov)

Select the domain or URL to investigate

Enter the domain or URL you wish to investigate. e.g. www.microsoft.com/index.html

url_txt = nbwidgets.GetText(
    prompt="Enter the URL to investigate:", value=url
)
display(url_txt)

URL Overview

The following cell runs the URL Summary notebooklet the collects relevant information about the URL, its domain, and associated IPs.
Use the output to understand the context of the URL and where it appears in the environment. From here you can identify further areas and scopes to hunt on.

url_nb = nb.nblts.azsent.url.URLSummary()
url_result = url_nb.run(value=url_txt.value, timespan=qry_prov.query_time.timespan, options=["+screenshot"])

The URL in the Environment

Once we have determined the nature of the domain or URL under investigation we want to see what the scope of impact is in our environment but identifying any presence of the domain or URL in our datasets.
If the domain has a high page rank score it is likely that it will be highly prevalent in a large environment, therefore you may wish to consider whether to run these cells for such a domain due to the data volumes involved.

if not url_result.dns_results.empty:
    url_result.dns_results.mp_plot.timeline(title=f"DNS lookup events for {url_txt.value}")
else:
    md(f"No dns events found relating to {url_txt.value}")

Related Alerts

Understanding where a URL has appeared in alerts can help provide context on when a URL was first seen in an environment and its link to malicious activity.

url_result.display_alert_timeline()
url_result.browse_alerts()

Related Bookmarks

As with alerts, understanding where a URL has appeared in bookmarks can help provide context on when a URL has previously been investigated within an environment.

if not url_result.bookmarks.empty:
    url_result.bookmarks.mp_plot.timeline()
    display(url_result.bookmarks)
else:
    md(f"No bookmarks found relating to {url_txt.value}")

Hosts Observed Communicating with the URL

During the cells executed above we have identified hosts communicating with the URL in question. This cell provides a summary of these hosts.
These hosts are potential candidates for further investigation using Microsoft Sentinel or via the host entity explorer Notebook.

if url_result.hosts:
    md(f"{len(url_result.hosts)} hosts observed connecting with {url_txt.value}")
    for host in url_result.hosts:
        md(host)

Additional environment data

To dig further into data regarding the URL in the environment there are a number of data sources returned by the URL Summary notebooklet.
There are:

• dns_results : shows DNS lookup events for the domain.

• flows : network flow logs for connections to the URL.

They can be accessed by using url_result.[data_source] in the cells below.

---

Use other notebooklets and pivots functions to drill down on other entities

You may want to drill down on other entities in the Host data. You can use methods of the IpAddress or Host entities, for example, to look at these in more detail.

Run the ip_address_summary notebooklet pivot

IpAddress = entities.IpAddress
ip_result = IpAddress.nblt.ip_address_summary("157.56.162.53")

View the TI results

ip_result.browse_ti_results()

---

More information:

Notebooklets and Pivots

Notebooklets

Pivot functions

Notebook/MSTICPy configuration

Getting Started
MSTICPy Configuration guide

ConfigureNotebookEnvironment notebook