Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
Azure
GitHub Repository: Azure/Azure-Sentinel-Notebooks
 Path: blob/master/Entity Explorer - IP Address.ipynb
3249 views
Kernel: Python 3.10 - SDK v2

Entity Explorer: IP Address

Notebook details...

Notebook Version: 2.0
 Python Version: Python 3.10
 Required Packages: msticpy, msticnb

Data Sources Required:

  • MS Sentinel (mostly optional)

    • Heartbeat

    • SecurityAlert

    • CommonSecurityLog

    • Azure AD Signin Logs

    • Azure Activity Logs

    • Office 356 Activity

    • AzureNetworkAnalytics_CL

    • MS Defender

    • VMComputer

  • External

    • VirusTotal (with API key)

    • Alienvault OTX (with API key)

    • IBM Xforce (with API key)

This notebook brings together a series of queries and visualizations to help you assess the security state of an IP address.

It works with both internal addresses and public addresses.

  • For internal addresses it focuses on traffic patterns and activity of the host using that IP address.

  • For public IPs it lets you perform threat intelligence lookups, passive dns, whois and other checks.
     It also allows you to examine Azure/Office activity, network traffic, host behavior
     and other data about the IP Address (subject to data availability).

The notebook uses the MSTIC notebooklets package to run most of the functionality. Summarized data is returned when it is run and more detailed information is contained in the returned result class.

Table of Contents

  • Hunting Hypothesis

  • Notebook Initialization

    • Get WorkspaceId and Authenticate to MS Sentinel

    • Import and initialize notebooklets

  • Enter the IP Address and query time window

  • Run the main ip_summary notebooklet

    • Browse alerts

    • View Threat Intel results

    • Map view of IP Address

  • Additional properties of the the result object

    • Viewing the Results class

    • Using Pivots to get more context information

  • User other notebooks and pivot functions to drill down on other entities

Contents

Hunting Hypothesis

Our broad initial hunting hypothesis is that a we have received IP address entity which is suspected to be compromized internal host or external public address to whom internal hosts are communicating in malicious manner, we will need to hunt from a range of different positions to validate or disprove this hypothesis.

Notebook initialization

This should complete without errors. If you encounter errors or warnings look at the following notebooks:

 Details... The next cell: - Checks for the correct Python version - Checks versions and optionally installs required packages - Imports the required packages into the notebook - Sets a number of configuration options.

If you are running in the Azure Sentinel Notebooks environment (Azure Notebooks or Azure ML) you can run live versions of these notebooks:

You may also need to do some additional configuration to successfully use functions such as Threat Intelligence service lookup and Geo IP lookup. There are more details about this in the ConfiguringNotebookEnvironment notebook and in these documents:

from datetime import datetime, timedelta, timezone

REQ_PYTHON_VER = "3.10"
REQ_MSTICPY_VER = "2.12.0"

# You may need to manually install msticpy with
# %pip install msticpy[azsentinel]

import msticpy as mp
from msticpy import nbwidgets
mp.init_notebook(
    namespace=globals(),
    additional_packages=["msticnb>=1.0"],
    verbosity=0,
);


# papermill default parameters
ws_name = "Default"
ip_address = ""
end = datetime.now(timezone.utc)
start = end - timedelta(days=2)

Contents

Get WorkspaceId and Authenticate to MS Sentinel

 Authentication help... If you want to use a workspace other than one you have defined in your
msticpyconfig.yaml create a connection string with your AAD TENANT_ID and
your WORKSPACE_ID (these should both be quoted UUID strings).
  workspace_cs = "loganalytics://code().tenant('TENANT_ID').workspace('WORKSPACE_ID')"

e.g.

  workspace_cs = "loganalytics://code().tenant('c3de0f06-dcb8-40fb-9d1a-b62faea29d9d').workspace('c62d3dc5-11e6-4e29-aa67-eac88d5e6cf6')"

Then in the Authentication cell replace the call to qry_prov.connect with the following:

  qry_prov.connect(connect_str=workspace_cs)

The cell should now look like this:

...
  # Authentication
  qry_prov = QueryProvider(data_environment="MSSentinel")
  qry_prov.connect(connect_str=workspace_cs)
...

On successful authentication you should see a popup schema button. To find your Workspace Id go to Log Analytics. Look at the workspace properties to find the ID.


Note: in VSCode (bug) the options may display in the VSCode status bar (bottom left)

print("Configured workspaces: ", ", ".join(mp.settings.get_config("AzureSentinel.Workspaces").keys()))
import ipywidgets as widgets
ws_param = widgets.Combobox(
    description="Workspace Name",
    value=ws_name,
    options=list(mp.settings.get_config("AzureSentinel.Workspaces").keys())
)
ws_param

from msticpy.common.timespan import TimeSpan

# Authentication
qry_prov = QueryProvider(data_environment="MSSentinel")
qry_prov.connect(WorkspaceConfig(workspace=ws_param.value))

nb_timespan = TimeSpan(start, end)
qry_prov.query_time.timespan = nb_timespan
md("<hr>")
md("Confirm time range to search", "bold")
qry_prov.query_time

Authentication and Configuration problems?

If you are having problems, expand the details section below

Click for details about configuring your authentication parameters

The notebook is expecting your Azure Sentinel Tenant ID and Workspace ID to be configured in one of the following places:

  • msticpyconfig.yaml in the current folder or location specified by MSTICPYCONFIG environment variable.

  • config.json in the current folder

For help with setting up your configuration (if this hasn't been done automatically) see the [Getting Started](./A Getting Started Guide For Azure Sentinel ML Notebooks.ipynb) notebook in the root folder of your Azure-Sentinel-Notebooks project.

Import and initialize notebooklets

This imports the msticnb package and the notebooklets classes.

These are needed for the notebook operations

import msticnb as nb

nb.init(query_provider=qry_prov)
pivot.timespan = qry_prov.query_time.timespan

Enter the IP Address and query time window

Type the IP address you want to search for and the time bounds over which search.

You can specify the IP address value in the widget e.g. 192.168.1.1

ipaddr_text = nbwidgets.GetText(prompt='Enter the IP Address to search for:', value=ip_address)

display(ipaddr_text)
md("<hr>")

Run the main ip_summary notebooklet run method to retrieve IP information

The notebooklet will query a variety of Sentinel tables and external data sources including:

  • MS Defender device information

  • Active Directory, Office and Azure Activity

  • Geo Location

  • WhoIs

  • Threat Intelligence providers

A summary of the data is displayed as it runs. You can run view the notebooklet result class to view the whole content or use individual data properties (DataFrames) to view detailed information.

See later in the notebook for how to access these.

Options you can use in the run method

Default options for the ip_address_summary function:

  • geoip: Get geo location information for IP address.

  • alerts: Get any alerts listing the IP address.

  • host_logons: Find any hosts with logons using this IP address as a source.

  • related_accounts: Find any accounts using this IP address in AAD or host logs.

  • device_info: Find any devices associated with this IP address.

  • device_network: Find any devices communicating with this IP address.

Other Options

  • bookmarks: Get any hunting bookmarks listing the IP address.

  • heartbeat: Get the latest heartbeat record for for this IP address.

  • az_net_if: Get the latest Azure network analytics interface data for this IP address.

  • vmcomputer: Get the latest VMComputer record for this IP address.

  • az_netflow: Get netflow information from AzureNetworkAnalytics table.

  • passive_dns: Force fetching passive DNS data from a TI Provider even if IP is internal.

  • az_activity: AAD sign-ins and Azure Activity logs.

  • office_365: Office 365 activity.

  • common_security: Get records from common security log.

  • ti: Force get threat intelligence reports even for internal public IPs.

Include additional options by prefixing the option with a "+"
 See the code in the next cell for an example. To remove an option, include in the list prefixed with a "-"

Running the notebooklet as a Pivot Function This can also be run as a pivot function from the Account entity.
The pivot function `account_summary` is in the `nblt` container of the `Account` entity.

Note: this can also be run as a pivot function from the IpAddress entity.
 The pivot function ip_summary is in the nblt container of the Account entity.

    IpAddress = entities.IpAddress
    ip_address = ipaddr_text.value.strip()
    ip_result = IpAddress.nblt.ip_address_summary(
        value=ip_address,
        options=["+az_activity", "+office_365", "+common_security"]
    )

ip_addr_nb = nb.nblts.azsent.network.IpAddressSummary()
md(
    "Note: Different result properties are populated depending on the IP type",
    "large, bold"
    )
ip_result = ip_addr_nb.run(
    value=ipaddr_text.value.strip(),
    timespan=qry_prov.query_time.timespan,
    options=["+az_activity", "+office_365", "+common_security"],
)

Browse alerts

View any recent alerts featuring the IP Address

ip_result.browse_alerts()

View Threat Intel results

View the full TI reports, if any, for the IP Address.

ip_result.browse_ti_results()

Map view of IP Address

Show the location of the IP address on a map

folium_map = FoliumMap(zoom_start=8)

icon_props = {"color": "green"}
folium_map.add_ip_cluster(ip_entities=[ip_result.ip_entity], **icon_props)
folium_map.center_map()
display(folium_map)

Other data and methods available for the IP notebooklet result

The notebooklet result has a number of data properties

ip_result.data_properties()

ip_result.list_methods()

Appendix - Additional properties from the Notebooklet result

These are static properties - usually DataFrames or visualizations. You can access each of these to see or manipulate the retrieved data.

To see help on the available attributes type:

>>> help(ip_result)

To see the available methods type:

>>> ip_result.list_methods()

Note, for the IP Summary notebooklet, the main data retrieval method is:

  • run
     There are several other methods that allow you to view individual plots or subsets of the data (such as alerts).

To view help on a specific method type:

>>> help(ip_result.method_name())

ip_result.passive_dns.T

Viewing the Result class

You can view all of the data in the results class by "running" it in a cell

Note: This produces a lot of output.
 Due to the way Jupyter display Javascript objects the plots may appear out of order.

  ip_result

Most of the properties of the results class are pandas DataFrames - you can use these directly for further analysis. Other property types include entities and visualizations.

The DataFrames displayed by running the result object are truncated to the first five rows.

You can also access individual data properties of the result as follows:

  result.data_property

ip_result

Using Pivots to get more context information

You can run a pivot function on the summary results to get additional context on the data.

Here is an example of looking up Whois information for Azure IPAddress requests.

whois_df = (
    ip_result                      # the results object
    .azure_activity_summary[["IPAddress"]]  # the property and the column we want
    .drop_duplicates()              # drop duplicates
    .mp_pivot.run(                  # run the pivot function IpAddress 'whois' function
        IpAddress.util.whois, column="IPAddress" 
    )
)
whois_df

Use other notebooklets and pivots functions to drill down on other entities

You may want to drill down on other entities in the data. You can use methods of the IpAddress or Host entities, for example, to look at these in more detail.

Run the ip_address_summary notebooklet pivot


acc_result = Account.nblt.account_summary("[email protected]")