Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
Azure
GitHub Repository: Azure/Azure-Sentinel-Notebooks
 Path: blob/master/tutorials-and-examples/feature-tutorials/TIProviders.ipynb
3253 views
Kernel: Python 3.8 - AzureML

msticpy Threat Intel Lookup

This notebook describes the use of the Threat Intelligence lookup class in msticpy. The class allows lookup of individual or multiple IoCs from one or more TI providers.

TILookup is also extensible - you can subclass TIProvider to implement your own custom lookups. You can also subclass the HTTPProvider or KqlProvider classes, which provide support for querying a REST endpoint or Log Analytics table respectively.

You must have msticpy installed to run this notebook:

%pip install --upgrade msticpy

To use the Microsoft Sentinel Threat Intel provider you will also need the "azsentinel" components:

%pip install --upgrade msticpy[azsentinel]

# Imports
import sys
import warnings

from msticpy.common.utility import check_py_version
MIN_REQ_PYTHON = (3,6)
check_py_version(MIN_REQ_PYTHON)

from msticpy import init_notebook
init_notebook(namespace=globals());

TILookup class

Input can be a single IoC observable or a pandas DataFrame containing multiple observables. Processing may require a an API key and processing performance may be limited to a specific number of requests per minute for the account type that you have.

# TILookup class
display(Markdown("### Constructor\n"))
print(TILookup.__init__.__doc__)
display(Markdown("### Attributes\n"))
for name in [att for att in dir(TILookup) if not att.startswith("_")]:
    display(Markdown(f"#### _{name}()_"))
    print(getattr(TILookup, name).__doc__)
    print()

Constructor

Initialize TILookup instance. Parameters ---------- primary_providers : Optional[List[TIProvider]], optional Primary TI Providers, by default None secondary_providers : Optional[List[TIProvider]], optional Secondary TI Providers, by default None providers: Optional[List[str]], optional List of provider names to load, by default all available providers are loaded. To see the list of available providers call `TILookup.list_available_providers()`. Note: if primary_provides or secondary_providers is specified This will override the providers list.

Attributes

add_provider()

Add a TI provider to the current collection. Parameters ---------- provider : TIProvider Provider instance name : str, optional The name to use for the provider (overrides the class name of `provider`) primary : bool, optional "primary" or "secondary" if False, by default "primary"

available_providers()

Return a list of builtin providers. Returns ------- List[str] List of TI Provider classes.

browse_results()

Return TI Results list browser. Parameters ---------- data : pd.DataFrame TI Results data from TIProviders severities : Optional[List[str]], optional A list of the severity classes to show. By default these are ['warning', 'high']. Pass ['information', 'warning', 'high'] to see all results. Other Parameters ---------------- kwargs : passed to SelectItem constuctor. Returns ------- SelectItem SelectItem browser for TI Data.

configured_providers()

Return a list of avaliable providers that have configuration details present. Returns ------- List[str] List of TI Provider classes.

list_available_providers()

Print a list of builtin providers with optional usage. Parameters ---------- show_query_types : bool, optional Show query types supported by providers, by default False as_list : bool, optional Return list of providers instead of printing to stdout. Note: if you specify `show_query_types` this will be printed irrespective of this parameter setting. Returns ------- Optional[List[str]] A list of provider names (if `return_list=True`)

loaded_providers()

Return dictionary of loaded providers. Returns ------- Dict[str, TIProvider] [description]

lookup_ioc()

Lookup single IoC in active providers. Parameters ---------- observable : str IoC observable (`ioc` is also an alias for observable) ioc_type : str, optional One of IoCExtract.IoCType, by default None If none, the IoC type will be inferred ioc_query_type: str, optional The ioc query type (e.g. rep, info, malware) providers: List[str] Explicit list of providers to use prov_scope : str, optional Use "primary", "secondary" or "all" providers, by default "primary" kwargs : Additional arguments passed to the underlying provider(s) Returns ------- Tuple[bool, List[Tuple[str, LookupResult]]] The result returned as a tuple(bool, list): bool indicates whether a TI record was found in any provider list has an entry for each provider result

lookup_iocs()

Lookup a collection of IoCs. Parameters ---------- data : Union[pd.DataFrame, Mapping[str, str], Iterable[str]] Data input in one of three formats: 1. Pandas dataframe (you must supply the column name in `obs_col` parameter) 2. Mapping (e.g. a dict) of [observable, IoCType] 3. Iterable of observables - IoCTypes will be inferred obs_col : str, optional DataFrame column to use for observables, by default None ioc_type_col : str, optional DataFrame column to use for IoCTypes, by default None ioc_query_type: str, optional The ioc query type (e.g. rep, info, malware) providers: List[str] Explicit list of providers to use prov_scope : str, optional Use "primary", "secondary" or "all" providers, by default "primary" kwargs : Additional arguments passed to the underlying provider(s) Returns ------- pd.DataFrame DataFrame of results

provider_status()

Return loaded provider status. Returns ------- Iterable[str] List of providers and descriptions.

provider_usage()

Print usage of loaded providers.

reload_provider_settings()

Reload provider settings from config.

reload_providers()

Reload providers based on currrent settings in config. Parameters ---------- clear_keyring : bool, optional Clears any secrets cached in keyring, by default False

result_to_df()

Return DataFrame representation of IoC Lookup response. Parameters ---------- ioc_lookup : Tuple[bool, List[Tuple[str, LookupResult]]] Output from `lookup_ioc` Returns ------- pd.DataFrame The response as a DataFrame with a row for each provider response.

Available Providers

The msticpy TI Provider library can lookup IoCs in multiple providers.

The list below shows the current set of providers.

TILookup.list_available_providers()
AzSTI GreyNoise OPR OTX Tor VirusTotal XForce

You can view the list of supported query types for each provider with the show_query_types=True parameter

TILookup.list_available_providers(show_query_types=True)
AzSTI Microsoft Sentinel TI provider class. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv6 ioc_type=linux_path ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=windows_path GreyNoise GreyNoise Lookup. Supported query types: ioc_type=ipv4 ioc_type=ipv4, ioc_query_type=full ioc_type=ipv4, ioc_query_type=quick OPR Open PageRank Lookup. Supported query types: ioc_type=dns OTX AlientVault OTX Lookup. Supported query types: ioc_type=dns ioc_type=dns, ioc_query_type=geo ioc_type=dns, ioc_query_type=passivedns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv4, ioc_query_type=geo ioc_type=ipv4, ioc_query_type=passivedns ioc_type=ipv6 ioc_type=ipv6, ioc_query_type=geo ioc_type=ipv6, ioc_query_type=passivedns ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url Tor Tor Exit Nodes Lookup. Supported query types: ioc_type=ipv4 VirusTotal VirusTotal Lookup. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=ipv4 ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url XForce IBM XForce Lookup. Supported query types: ioc_type=dns ioc_type=dns, ioc_query_type=malware ioc_type=dns, ioc_query_type=passivedns ioc_type=dns, ioc_query_type=whois ioc_type=file_hash ioc_type=hostname, ioc_query_type=whois ioc_type=ipv4 ioc_type=ipv4, ioc_query_type=malware ioc_type=ipv4, ioc_query_type=passivedns ioc_type=ipv4, ioc_query_type=rep ioc_type=ipv4, ioc_query_type=whois ioc_type=ipv6 ioc_type=ipv6, ioc_query_type=malware ioc_type=ipv6, ioc_query_type=passivedns ioc_type=ipv6, ioc_query_type=rep ioc_type=ipv6, ioc_query_type=whois ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=url, ioc_query_type=malware

Loading TIProviders

Calling TILookup with no parameters will load all of the available providers that have a configuration entry in msticpyconfig.yaml (see next section)

# load all configured providers
ti_lookup = TILookup()
ti_lookup.provider_status

# Restricting which providers get loaded
#ti_lookup = TILookup(providers=["VirusTotal", "XForce"])
Attempting to sign-in with environment variable credentials... Using Open PageRank. See https://www.domcop.com/openpagerank/what-is-openpagerank Please wait. Loading Kqlmagic extension...
['OTX - AlientVault OTX Lookup. (primary)', 'VirusTotal - VirusTotal Lookup. (primary)', 'XForce - IBM XForce Lookup. (primary)', 'GreyNoise - GreyNoise Lookup. (primary)', 'AzSTI - Microsoft Sentinel TI provider class. (primary)', 'OPR - Open PageRank Lookup. (secondary)']

Configuration File

You can configure primary and secondary providers. Primary providers are used by default.

You may need to supply an authorization (API) key and in some cases a user ID for each provider.

For LogAnalytics/Microsoft Sentinel providers, you will need the workspace ID and tenant ID and will need to authenticate in order to access the data (although if you have an existing authenticated connection with the same workspace/tenant, this connection will be re-used).

The configuration file is read from the current directory.

Alternatively, you can specify a location for this file in an environment variable MSTICPYCONFIG.

If you need to create a config file, uncomment the lines in the following cell.

Warning - this will overwrite a file of the same name in the current directory

Delete any provider entries that you do not want to use and add the missing parameters for your providers. 

# %%writefile msticpyconfig.yaml
# QueryDefinitions:

# TIProviders:
#   OTX:
#     Args:
#       AuthKey: "your-otx-key"
#     Primary: True
#     Provider: "OTX" # Explicitly name provider to override
#   VirusTotal:
#     Args:
#       AuthKey: "your-vt-key"
#     Primary: True
#     Provider: "VirusTotal"
#   XForce:
#     Args:
#       ApiID: "your-xforce-id"
#       AuthKey: "your-xforce-key"
#     Primary: True
#     Provider: "XForce"
#   GreyNoise:
#     Args:
#       AuthKey: ""
#     Primary: True
#     Provider: "GreyNoise"
#   AzureSentinel:
#     # Note if you do not specify any settings in the Args key for the AzureSentinel
#     # provider, it will default to using your default Microsoft Sentinel workspace.
#     Args:
#       WorkspaceID: "your-azure-sentinel-workspace-id"
#       TenantID: "your-azure-sentinel-tenant-id"
#     Primary: True
#     Provider: "AzSTI"

Reload providers to pick up new settings

ti_lookup.reload_providers()
ti_lookup.provider_status
Settings reloaded. Use reload_providers to update settings for loaded providers. Using Open PageRank. See https://www.domcop.com/openpagerank/what-is-openpagerank
['OTX - AlientVault OTX Lookup. (primary)', 'VirusTotal - VirusTotal Lookup. (primary)', 'XForce - IBM XForce Lookup. (primary)', 'GreyNoise - GreyNoise Lookup. (primary)', 'AzSTI - Microsoft Sentinel TI provider class. (primary)', 'OPR - Open PageRank Lookup. (secondary)']

Looking up IoCs

lookup_ioc

To lookup a single IoC.

ti_lookup.lookup_ioc(
    observable: str = None,
    ioc_type: str = None,
    ioc_query_type: str = None,
    providers: List[str] = None,
    prov_scope: str = 'primary',
    **kwargs,
) -> Tuple[bool, List[Tuple[str, msticpy.sectools.tiproviders.ti_provider_base.LookupResult]]]

Lookup single IoC in active providers.

Parameters
----------
observable : str
    IoC observable
    (`ioc` is also an alias for observable)
ioc_type : str, optional
    One of IoCExtract.IoCType, by default None
    If none, the IoC type will be inferred
ioc_query_type: str, optional
    The ioc query type (e.g. rep, info, malware)
providers: List[str]
    Explicit list of providers to use
prov_scope : str, optional
    Use primary, secondary or all providers, by default "primary"
kwargs :
    Additional arguments passed to the underlying provider(s)

Returns
-------
Tuple[bool, List[Tuple[str, LookupResult]]]
    The result returned as a tuple(bool, list):
    bool indicates whether a TI record was found in any provider
    list has an entry for each provider result

# Uncomment this and run to see the document string
# ti_lookup.lookup_ioc?

Lookup an IoC from a single provider

And show the output

result = ti_lookup.lookup_ioc(observable="52.183.120.194", providers=["AzSTI", "XForce"])
ti_lookup.result_to_df(result)

result = ti_lookup.lookup_ioc(observable="52.183.120.194")
ti_lookup.result_to_df(result).T

import pprint
pp = pprint.PrettyPrinter(indent=2)

result, details = ti_lookup.lookup_ioc(observable="38.75.137.9", providers=["OTX"])

# the details is a list (since there could be multiple responses for an IoC)
for provider, detail in details:
    print(provider)
    detail.summary
# Un-comment to view raw response
#     print("\nRaw Results")
#     pp.pprint(detail.raw_result)
OTX ioc: 38.75.137.9 ( ipv4 ) result: True { 'names': [ 'Underminer.EK - Exploit Kit IOC Feed', '<script ' 'src="https://otx.alienvault.com/pulse/5d4d8ccdbe24622d01f9ce9f.js"></script>', 'Underminer.EK - Exploit Kit IOC Feed', 'Underminer EK'], 'pulse_count': 4, 'references': [ [], [], [], [ 'https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/']], 'tags': [['Underminer.EK'], ['Underminer.EK'], ['Underminer.EK'], []]} reference: https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/general

Or convert result to a DataFrame and let pandas do the display work...

result = ti_lookup.lookup_ioc(observable="38.75.137.9", providers=["OTX"])
ti_lookup.result_to_df(result).T

# Extract a single field (RawResult) from the dataframe (.iloc[0] is to select the row)
ti_lookup.result_to_df(result)["RawResult"].iloc[0]
{'whois': 'http://whois.domaintools.com/38.75.137.9', 'reputation': 0, 'indicator': '38.75.137.9', 'type': 'IPv4', 'type_title': 'IPv4', 'base_indicator': {'id': 2127020821, 'indicator': '38.75.137.9', 'type': 'IPv4', 'title': '', 'description': '', 'content': '', 'access_type': 'public', 'access_reason': ''}, 'pulse_info': {'count': 4, 'pulses': [{'id': '5d4d8ccdbe24622d01f9ce9f', 'name': 'Underminer.EK - Exploit Kit IOC Feed', 'description': 'IPs and hostnames for the Exploit Kit known as Underminer.EK.', 'modified': '2021-04-14T19:27:12.332000', 'created': '2019-08-09T15:10:05.503000', 'tags': ['Underminer.EK'], 'references': [], 'public': 1, 'adversary': '', 'targeted_countries': [], 'malware_families': [], 'attack_ids': [], 'industries': [], 'TLP': 'white', 'cloned_from': None, 'export_count': 770, 'upvotes_count': 0, 'downvotes_count': 0, 'votes_count': 0, 'locked': False, 'pulse_source': 'api', 'validator_count': 0, 'comment_count': 0, 'follower_count': 0, 'vote': 0, 'author': {'username': 'otxrobottwo_testing', 'id': '83138', 'avatar_url': 'https://otx.alienvault.com/assets/images/default-avatar.png', 'is_subscribed': False, 'is_following': False}, 'indicator_type_counts': {}, 'indicator_count': 0, 'is_author': False, 'is_subscribing': None, 'subscriber_count': 445, 'modified_text': '6 days ago ', 'is_modified': True, 'groups': [], 'in_group': False, 'threat_hunter_scannable': False, 'threat_hunter_has_agents': 1, 'related_indicator_type': 'IPv4', 'related_indicator_is_active': 0}, {'id': '5f78ad73cb75802b9c81068f', 'name': '<script src="https://otx.alienvault.com/pulse/5d4d8ccdbe24622d01f9ce9f.js"></script>', 'description': '', 'modified': '2020-10-03T16:57:55.377000', 'created': '2020-10-03T16:57:23.187000', 'tags': ['Underminer.EK'], 'references': [], 'public': 1, 'adversary': '', 'targeted_countries': [], 'malware_families': [], 'attack_ids': [], 'industries': [], 'TLP': 'white', 'cloned_from': '5d4d8ccdbe24622d01f9ce9f', 'export_count': 3, 'upvotes_count': 0, 'downvotes_count': 0, 'votes_count': 0, 'locked': False, 'pulse_source': 'web', 'validator_count': 0, 'comment_count': 0, 'follower_count': 0, 'vote': 0, 'author': {'username': 'fisher7801', 'id': '125860', 'avatar_url': 'https://otx.alienvault.com/assets/images/default-avatar.png', 'is_subscribed': False, 'is_following': False}, 'indicator_type_counts': {}, 'indicator_count': 0, 'is_author': False, 'is_subscribing': None, 'subscriber_count': 8, 'modified_text': '200 days ago ', 'is_modified': False, 'groups': [], 'in_group': False, 'threat_hunter_scannable': False, 'threat_hunter_has_agents': 1, 'related_indicator_type': 'IPv4', 'related_indicator_is_active': 0}, {'id': '5db816cba3e59aeced1fad16', 'name': 'Underminer.EK - Exploit Kit IOC Feed', 'description': 'IPs and hostnames for the Exploit Kit known as Underminer.EK.', 'modified': '2019-11-04T13:21:54.514000', 'created': '2019-10-29T10:39:07.558000', 'tags': ['Underminer.EK'], 'references': [], 'public': 1, 'adversary': '', 'targeted_countries': [], 'malware_families': [], 'attack_ids': [], 'industries': [], 'TLP': 'white', 'cloned_from': None, 'export_count': 2, 'upvotes_count': 0, 'downvotes_count': 0, 'votes_count': 0, 'locked': False, 'pulse_source': 'api', 'validator_count': 0, 'comment_count': 0, 'follower_count': 0, 'vote': 0, 'author': {'username': 'otxrobottwo', 'id': '78495', 'avatar_url': 'https://otx20-web-media.s3.amazonaws.com/media/avatars/user_78495/resized/80/avatar_ba5a8acdbd.png', 'is_subscribed': False, 'is_following': False}, 'indicator_type_counts': {'hostname': 1, 'domain': 1, 'IPv4': 7}, 'indicator_count': 9, 'is_author': False, 'is_subscribing': None, 'subscriber_count': 397, 'modified_text': '534 days ago ', 'is_modified': True, 'groups': [], 'in_group': False, 'threat_hunter_scannable': True, 'threat_hunter_has_agents': 1, 'related_indicator_type': 'IPv4', 'related_indicator_is_active': 1}, {'id': '5d41d77901a2f8c6e9b650e9', 'name': 'Underminer EK', 'description': '', 'modified': '2019-07-31T18:01:29.744000', 'created': '2019-07-31T18:01:29.744000', 'tags': [], 'references': ['https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/'], 'public': 1, 'adversary': '', 'targeted_countries': [], 'malware_families': [], 'attack_ids': [], 'industries': [], 'TLP': 'white', 'cloned_from': None, 'export_count': 1, 'upvotes_count': 0, 'downvotes_count': 0, 'votes_count': 0, 'locked': False, 'pulse_source': 'web', 'validator_count': 0, 'comment_count': 0, 'follower_count': 0, 'vote': 0, 'author': {'username': 'mattvittitoe', 'id': '79520', 'avatar_url': 'https://otx.alienvault.com/assets/images/default-avatar.png', 'is_subscribed': False, 'is_following': False}, 'indicator_type_counts': {'URL': 16, 'FileHash-MD5': 5, 'IPv4': 3}, 'indicator_count': 24, 'is_author': False, 'is_subscribing': None, 'subscriber_count': 33, 'modified_text': '630 days ago ', 'is_modified': False, 'groups': [], 'in_group': False, 'threat_hunter_scannable': True, 'threat_hunter_has_agents': 1, 'related_indicator_type': 'IPv4', 'related_indicator_is_active': 1}], 'references': ['https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/'], 'related': {'alienvault': {'adversary': [], 'malware_families': [], 'industries': []}, 'other': {'adversary': [], 'malware_families': [], 'industries': []}}}, 'false_positive': [], 'validation': [], 'asn': 'AS63023 AS-GLOBALTELEHOST', 'city_data': True, 'city': 'Los Angeles', 'region': 'CA', 'continent_code': 'NA', 'country_code3': 'USA', 'country_code2': 'US', 'subdivision': 'CA', 'latitude': 34.0544, 'postal_code': '90009', 'longitude': -118.244, 'accuracy_radius': 1000, 'country_code': 'US', 'country_name': 'United States of America', 'dma_code': 803, 'charset': 0, 'area_code': 0, 'flag_url': '/assets/images/flags/us.png', 'flag_title': 'United States of America', 'sections': ['general', 'geo', 'reputation', 'url_list', 'passive_dns', 'malware', 'nids_list', 'http_scans']}

Lookup using all primary providers

result = ti_lookup.lookup_ioc(observable="188.127.231.124")
ti_lookup.result_to_df(result)

Provider Usage

This shows the supported IoC Types.

In some cases an IoC type will also support special types of sub-query such as geo-ip and passive-dns

display(ti_lookup.provider_status)
ti_lookup.loaded_providers["AzSTI"].usage()
['OTX - AlientVault OTX Lookup. (primary)', 'VirusTotal - VirusTotal Lookup. (primary)', 'XForce - IBM XForce Lookup. (primary)', 'GreyNoise - GreyNoise Lookup. (primary)', 'AzSTI - Microsoft Sentinel TI provider class. (primary)', 'OPR - Open PageRank Lookup. (secondary)']
Microsoft Sentinel TI provider class. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv6 ioc_type=linux_path ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=windows_path 
ti_lookup.provider_usage()
Primary providers ----------------- Provider class: OTX AlientVault OTX Lookup. Supported query types: ioc_type=dns ioc_type=dns, ioc_query_type=geo ioc_type=dns, ioc_query_type=passivedns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv4, ioc_query_type=geo ioc_type=ipv4, ioc_query_type=passivedns ioc_type=ipv6 ioc_type=ipv6, ioc_query_type=geo ioc_type=ipv6, ioc_query_type=passivedns ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url Provider class: VirusTotal VirusTotal Lookup. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=ipv4 ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url Provider class: XForce IBM XForce Lookup. Supported query types: ioc_type=dns ioc_type=dns, ioc_query_type=malware ioc_type=dns, ioc_query_type=passivedns ioc_type=dns, ioc_query_type=whois ioc_type=file_hash ioc_type=hostname, ioc_query_type=whois ioc_type=ipv4 ioc_type=ipv4, ioc_query_type=malware ioc_type=ipv4, ioc_query_type=passivedns ioc_type=ipv4, ioc_query_type=rep ioc_type=ipv4, ioc_query_type=whois ioc_type=ipv6 ioc_type=ipv6, ioc_query_type=malware ioc_type=ipv6, ioc_query_type=passivedns ioc_type=ipv6, ioc_query_type=rep ioc_type=ipv6, ioc_query_type=whois ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=url, ioc_query_type=malware Provider class: GreyNoise GreyNoise Lookup. Supported query types: ioc_type=ipv4 ioc_type=ipv4, ioc_query_type=full ioc_type=ipv4, ioc_query_type=quick Provider class: AzSTI Microsoft Sentinel TI provider class. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv6 ioc_type=linux_path ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=windows_path Secondary providers ------------------- Provider class: OPR Open PageRank Lookup. Supported query types: ioc_type=dns

Use to do a passive DNS lookup

result = ti_lookup.lookup_ioc(observable="38.75.137.9", ico_type="ipv4", ioc_query_type="passivedns", providers=["XForce"])
print(result)
print("\nProvider result:")
result[1][0][1].raw_result
(True, [('XForce', LookupResult(ioc='38.75.137.9', ioc_type='ipv4', safe_ioc='38.75.137.9', query_subtype='passivedns', provider='XForce', result=True, severity='information', details={'records': 1}, raw_result={'Passive': {'query': '0x00000000000000000000ffff264b8909', 'records': []}, 'RDNS': ['9-137-75-38.clients.gthost.com'], 'total_rows': 1}, reference='https://api.xforce.ibmcloud.com/resolve/38.75.137.9', status=0))]) Provider result:
{'Passive': {'query': '0x00000000000000000000ffff264b8909', 'records': []}, 'RDNS': ['9-137-75-38.clients.gthost.com'], 'total_rows': 1}

Use to do a GeoIP lookup

result = ti_lookup.lookup_ioc(observable="38.75.137.9", ico_type="ipv4", ioc_query_type="geo", providers=["OTX"])
print(result)
print("\nProvider result:")
result[1][0][1].raw_result
(True, [('OTX', LookupResult(ioc='38.75.137.9', ioc_type='ipv4', safe_ioc='38.75.137.9', query_subtype='geo', provider='OTX', result=True, severity='information', details={}, raw_result={'asn': 'AS63023 AS-GLOBALTELEHOST', 'city_data': True, 'city': 'Los Angeles', 'region': 'CA', 'continent_code': 'NA', 'country_code3': 'USA', 'country_code2': 'US', 'subdivision': 'CA', 'latitude': 34.0544, 'postal_code': '90009', 'longitude': -118.244, 'accuracy_radius': 1000, 'country_code': 'US', 'country_name': 'United States of America', 'dma_code': 803, 'charset': 0, 'area_code': 0, 'flag_url': '/assets/images/flags/us.png', 'flag_title': 'United States of America'}, reference='https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/geo', status=0))]) Provider result:
{'asn': 'AS63023 AS-GLOBALTELEHOST', 'city_data': True, 'city': 'Los Angeles', 'region': 'CA', 'continent_code': 'NA', 'country_code3': 'USA', 'country_code2': 'US', 'subdivision': 'CA', 'latitude': 34.0544, 'postal_code': '90009', 'longitude': -118.244, 'accuracy_radius': 1000, 'country_code': 'US', 'country_name': 'United States of America', 'dma_code': 803, 'charset': 0, 'area_code': 0, 'flag_url': '/assets/images/flags/us.png', 'flag_title': 'United States of America'}

Inferring IoC Type vs. Specifying explicity

If you do a lookup without specifying a type, TILookup will try to infer the type by matching regexes. There are patterns for all supported types but there are some caveats:

  • The match is not 100% foolproof - e.g. some URLs and hash types may be misidentified.

  • The inference adds an overhead to each lookup.

If you know the type that you want to look up, it is always better to explicitly include it.

  • For single IoC lookup, use the ioc_type parameter.

  • For multiple IoC lookups (see below), supply either:

    • a DataFrame with a column that specifies the type for each entry

    • a dictionary of the form {ioc_observable: ioc_type}

Looking up Multiple IoCs

lookup_iocs

Signature:
ti_lookup.lookup_iocs(
    data: Union[pandas.core.frame.DataFrame, Mapping[str, str], Iterable[str]],
    obs_col: str = None,
    ioc_type_col: str = None,
    ioc_query_type: str = None,
    providers: List[str] = None,
    prov_scope: str = 'primary',
    **kwargs,
) -> pandas.core.frame.DataFrame

Lookup a collection of IoCs.

Parameters
----------
data : Union[pd.DataFrame, Mapping[str, str], Iterable[str]]
    Data input in one of three formats:
    1. Pandas dataframe (you must supply the column name in
    `obs_col` parameter)
    2. Mapping (e.g. a dict) of [observable, IoCType]
    3. Iterable of observables - IoCTypes will be inferred
obs_col : str, optional
    DataFrame column to use for observables, by default None
ioc_type_col : str, optional
    DataFrame column to use for IoCTypes, by default None
ioc_query_type: str, optional
    The ioc query type (e.g. rep, info, malware)
providers: List[str]
    Explicit list of providers to use
prov_scope : str, optional
    Use primary, secondary or all providers, by default "primary"
kwargs :
    Additional arguments passed to the underlying provider(s)

Returns
-------
pd.DataFrame
    DataFrame of results

# Uncomment this and run to see the document string
# ti_lookup.lookup_iocs?

Multiple IP Lookup from single provider


ioc_ips = [
    "51.75.29.61",
    "190.2.144.45"
    "52.183.120.194",
    "146.56.231.70",
    "1.2.3.4",
    "109.236.89.61",
    "1.2.3.5",
]

ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI")

Multiple IoCs using all providers

Output sorted by IoC

Note that these URLs were picked randomly from the TI databases of the three providers used. In most cases the IoC is found by only that provider, which 

ioc_urls = [
    "http://cheapshirts.us/zVnMrG.php",
    "http://chinasymbolic.com/i9jnrc",
    "https://hotel-bristol.lu/dlry/MAnJIPnY/",
    "http://businesstobuy.net",
    "http://append.pl/srh9xsz",
    "http://104.248.196.145/apache2",
    "http://ajaraheritage.ge/g7cberv",
    "http://cic-integration.com/hjy93JNBasdas",
    "https://google.com",  # benign
    "https://microsoft.com",  # benign
    "https://python.org",  # benign
]
results = ti_lookup.lookup_iocs(data=ioc_urls)
results.sort_values("Ioc")

Multiple Mixed IoC Types

ioc_mixed = [
    "http://104.248.196.145/apache2",
    "http://ajaraheritage.ge/g7cberv",
    "http://cic-integration.com/hjy93JNBasdas",
    "51.75.29.61",
    "33.44.55.66",
    "52.183.120.194",
    "f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5cf281cdf3145865092",
    "cc2db822f652ca67038ba7cca8a8bde3",
    "ajaraheritage.ge",
]
results = ti_lookup.lookup_iocs(data=ioc_mixed)
results

Browsing TI Results

To make it easier to walk through the returned results there is a browser. This shows you results aggregated by the IoC value (e.g. an individual IP Address or URL) for all providers.

For each provider that returns a result for an IoC, the summarized details will be shown in a table below the browse list. Click on Raw results from provider... to see all returned data.

Note: the reference URL may not work if you have not authenticated to the service

The value of the selected IoC entry is available as ti_selector.value

You can match this back to the original results DataFrame as follows:

results[results["Ioc"] == ti_selector.value[0]]

from msticpy.nbtools.ti_browser import browse_results
ti_selector = browse_results(data=results, height="200px")
ti_selector
VBox(children=(Text(value='', description='Filter:', style=DescriptionStyle(description_width='initial')), Sel…

Specifying Time Ranges

Some providers (currently only AzSTI) support time ranges so that you can specify specific periods to search for.

If a provider does not support time ranges, the parameters will be ignored

from datetime import datetime
search_origin = datetime.now()
q_times = nbwidgets.QueryTime(units="hour", auto_display=True, origin_time=search_origin, max_after=24, max_before=24)
VBox(children=(HTML(value='<h4>Set query time boundaries</h4>'), HBox(children=(DatePicker(value=datetime.date…
# Using this data range returned no results
ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI", start=q_times.start, end=q_times.end).head()

from datetime import datetime
search_origin = datetime.now()
q_times = nbwidgets.QueryTime(units="day", auto_display=True, origin_time=search_origin, max_after=24, max_before=24)
VBox(children=(HTML(value='<h4>Set query time boundaries</h4>'), HBox(children=(DatePicker(value=datetime.date…
# Using a wider ranges produces results
ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI", start=q_times.start, end=q_times.end)