Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
Azure
GitHub Repository: Azure/Azure-Sentinel-Notebooks
 Path: blob/master/tutorials-and-examples/feature-tutorials/MordorData.ipynb
3253 views
Kernel: Python (condadev)

MSTICpy - Mordor data provider and browser

Description

This notebook provides a guided example of using the Mordor data provider and browser included with MSTICpy.

For more information on the Mordor data sets see the Open Threat Research Forge Mordor GitHub repo

You must have msticpy installed to run this notebook:

%pip install --upgrade msticpy

MSTICpy versions >= 0.8.5

Contents:

  • Using the Mordor data provider to retrieve data sets

    • Listing queries

    • Running a query to retrieve data

    • Optional parameters

    • Searching for queries by Mordor property

  • Mordor Browser

Using the Data Provider to download datasets

Using the data provider you can download and render event data as a pandas DataFrame.

Note - Mordor includes both host event data and network capture data.
 Although Capture files can be downloaded and unpacked
 they currently cannot be populated into a pandas DataFrame. This is the case for most network datasets.
 Host event data is retrieved and populated into DataFrames.

from msticpy.data import QueryProvider
mdr_data = QueryProvider("Mordor", save_folder="./mordor")
mdr_data.connect()
Retrieving Mitre data... Retrieving Mordor data...
HBox(children=(HTML(value='Downloading Mordor metadata'), FloatProgress(value=0.0, max=83.0), HTML(value='')))

List Queries

Note: Many Mordor data entries have multiple data sets, so we see more queries than Mordor entries.

(Only first 15 shown)

mdr_data.list_queries()[:15]
['small.aws.collection.ec2_proxy_s3_exfiltration', 'small.windows.collection.host.msf_record_mic', 'small.windows.credential_access.host.cmd_lsass_memory_dumpert_syscalls', 'small.windows.credential_access.host.cmd_psexec_lsa_secrets_dump', 'small.windows.credential_access.host.cmd_sam_copy_esentutl', 'small.windows.credential_access.host.covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges', 'small.windows.credential_access.host.empire_dcsync_dcerpc_drsuapi_DsGetNCChanges', 'small.windows.credential_access.host.empire_mimikatz_backupkeys_dcerpc_smb_lsarpc', 'small.windows.credential_access.host.empire_mimikatz_extract_keys', 'small.windows.credential_access.host.empire_mimikatz_logonpasswords', 'small.windows.credential_access.host.empire_mimikatz_lsadump_patch', 'small.windows.credential_access.host.empire_mimikatz_sam_access', 'small.windows.credential_access.host.empire_over_pth_patch_lsass', 'small.windows.credential_access.host.empire_powerdump_sam_access', 'small.windows.credential_access.host.empire_shell_reg_dump_sam']

Retrieving/querying a data set

mdr_data.small.windows.credential_access.host.covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges().head(3)
https://raw.githubusercontent.com/OTRF/mordor/master/datasets/small/windows/credential_access/host/covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges.zip Extracting covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges_2020-08-05020926.json

Optional parameters

The data provider and the query functions support some parameters to control aspects of the query operation.

  • use_cached : bool, optional
     Try to use locally saved file first, by default True. If you’ve previously downloaded a file, it will use this rather than downloading a new copy.

  • save_folder : str, optional
     Path to output folder, by default ".". The path that downloaded and extracted files are saved to.

  • silent : bool
     If True, suppress feedback. By default, False.

If you specify these when you initialize the data provider, the settings will apply to all queries.

mdr_data = QueryProvider("Mordor", save_folder="./mordor")
mdr_data.connect()
Retrieving Mitre data... Retrieving Mordor data...

Using these parameters in the query will override the provider settings and defaults for that query.

mdr_data.small.windows.credential_access.host.covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges(silent=True, save_folder="./mordor").head(2)

Getting summary data about a query

Call the query function with a single "?" parameter.

mdr_data.small.windows.credential_access.host.covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges("?")
Query: covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges Data source: Mordor Covenant DCSync Notes ----- Mordor ID: SDWIN-200805020926 This dataset represents adversaries abusing Active Directory Replication services to retrieve secret domain data (i.e. NTLM hashes) from domain accounts. Mitre Techniques: T1003: OS Credential Dumping Mitre Tactics: TA0006: Credential Access Parameters ---------- Query: https://raw.githubusercontent.com/OTRF/mordor/master/datasets/small/windows/credential_access/host/covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges.zip

Searching for Queries with QueryProvider.search_queries()

Search queries for matching attributes.

Parameters

search : str Search string.

Substrings separated by commas will be treated as OR terms - e.g. "a, b" == "a" or "b".
 Substrings separated by "+" will be treated as AND terms - e.g. "a + b" == "a" and "b"

Returns

List of matching query names.

mdr_data.search_queries("AWS")
['small.aws.collection.ec2_proxy_s3_exfiltration (AWS Cloud Bank Breach S3)']
mdr_data.search_queries("Empire + T1222")
['small.windows.defense_evasion.host.empire_powerview_ldap_ntsecuritydescriptor (Empire Powerview Add-DomainObjectAcl)', 'small.windows.defense_evasion.network.empire_powerview_ldap_ntsecuritydescriptor (Empire Powerview Add-DomainObjectAcl)']
mdr_data.search_queries("Empire + Credential")
['small.windows.credential_access.host.empire_mimikatz_sam_access (Empire Mimikatz SAM Extract Hashes)', 'small.windows.credential_access.host.empire_mimikatz_logonpasswords (Empire Mimikatz LogonPasswords)', 'small.windows.credential_access.host.empire_dcsync_dcerpc_drsuapi_DsGetNCChanges (Empire DCSync)', 'small.windows.credential_access.network.empire_dcsync_dcerpc_drsuapi_DsGetNCChanges (Empire DCSync)', 'small.windows.credential_access.host.empire_mimikatz_lsadump_patch (Empire Mimikatz Lsadump LSA Patch)', 'small.windows.defense_evasion.host.empire_wdigest_downgrade.tar (Empire WDigest Downgrade)']

Mordor Browser

We've also built a more specialized browser for Mordor data. This uses the metadata in the repository to let you view full details of the dataset.

You can also preview the dataset (if it is convertible to a DataFrame).

For details of the data shown please see the Mordor GitHub repo
 and the Threat Hunter Playbook

from msticpy.data.browsers.mordor_browser import MordorBrowser

mdr_browser = MordorBrowser()
Retrieving Mitre data... Retrieving Mordor data...
VBox(children=(VBox(children=(HTML(value='<h2>Mordor dataset browser</h2>'), Select(description='Data sets', l…

Mordor Browser Details

The top scrollable list is a list of the Mordor datasets. Selecting one of these updates the data in the lower half of the browser.

Filter Drop-down

To narrow your search you can filter using a text search or filter by Mitre Attack Techniques or Tactics.

  • The Filter text box uses the same syntax as the provider search_queries() function.

    • Simple text string will find matches for datasets that contain this string

    • Strings separated by "," are treated as OR terms - i.e. it will match items that contain ANY of the substrings

    • Strings separated by "+" are treated as AND terms - i.e. it will match items that contain ALL of the substrings

  • The Mitre Techniques and Tactics lists are multi-select lists. Only items that have techniques and tactics matching the selected items will be show.

  • Reset Filter button will clear any filtering.

Main Details Window

  • title, ID, author, creation date, modification date and description are self-explanatory.

  • tags can be used for searching

  • file_paths (see below)

  • attacks - lists related Mitre Technique and Tactics. The item title is a link to the Mitre page describing the technique or tactic.

  • notebooks - if there is a notebook in the Threat Hunter Playbook site, a link to it is shown here. (multiple notebooks might be shown)

  • simulation - raw data listing the steps in the attack (and useful for replaying the attack in a demo environment).

  • references - links to any external data about the attack.

File_paths

This section allows you to select, download and (in most cases) display the event data relating to the attack.

Select a file and click on the Download button.

The zipped file is downloaded and extracted. If it is event data, this is converted to a pandas DataFrame and displayed below the rest of the data.

The current dataset is available as an attribute of the browser:

    mdr_browser.current_dataset

Datasets that you've downloaded and displayed in this session are also cached in the browser and available in the mdr_browser.datasets attribute.

Downloaded files

By default files are downloaded and extracted to the current folder. You can change this with the save_folder parameter when creating the MordorBrowser object.

You can also specify the use_cached parameter. By default, this is True, which causes downloaded files not to be deleted after extraction. These local copies are used if you try to view the same data set again. This also works across sessions.

If use_cache is set to False, files are deleted immediately after downloading, extracting and populating the DataFrame.

Using the standard query browser

Note - In the Example section, ignore the examples of parameters
 passed to the query - these are not needed and ignored.

mdr_data.browse_queries()
VBox(children=(Text(value='', description='Filter:', style=DescriptionStyle(description_width='initial')), Sel…

Remove cached files

from pathlib import Path
for file in Path("./mordor").glob("*"):
    file.unlink()
Path("./mordor").rmdir()