VTLookupV3.ipynb

Kernel: Python (condadev)

VT Graphs in Jupyter Notebook

In this notebook we will explore how to obtain attributes and relationship for different entities using VirusTotal API v3. Finally we can render all the relationships we have obtained using VTGraph.

Import libraries

from msticpy.sectools.vtlookupv3 import VTLookupV3, VTEntityType

import networkx as nx
import matplotlib.pyplot as plt
import os
import pandas as pd

pd.set_option('max_colwidth', 200)

try:
    import nest_asyncio
except ImportError as err:
    print("nest_asyncio is required for running VTLookup3 in notebooks.")
    resp = input("Install now? (y/n)")
    if resp.strip().lower().startswith("y"):
        !pip install nest_asyncio
        import nest_asyncio
    else:
        raise err
nest_asyncio.apply()

Create Lookup instance

from msticpy.common.provider_settings import get_provider_settings
# Try to obtain key from env varaible
vt_key = os.environ.get("VT_API_KEY")
if not vt_key:
    # if not try provider settings to get from msticpyconfig.yaml
    vt_key = get_provider_settings("TIProviders")["VirusTotal"].args["AuthKey"]

# Instantiate vt_lookup object
vt_lookup = VTLookupV3(vt_key)

# Utility function to get full details of a VT Object
def get_object(vt_id: str, vt_type: str) -> pd.DataFrame:
    """
    Return the full VT object as a DataFrame.

    This function will be added to a future version of VTLookup3
    """
    if VTEntityType(vt_type) not in vt_lookup._SUPPORTED_VT_TYPES:
        raise KeyError(f"Property type {vt_type} not supported")

    endpoint_name = vt_lookup._get_endpoint_name(vt_type)
    try:
        response = vt_lookup._vt_client.get_object(
            f"/{endpoint_name}/{vt_id}"
        )
        return pd.DataFrame(data=response.to_dict()).drop(columns=["id", "type"])
    finally:
        vt_lookup._vt_client.close()

# The ID (SHA256 hash) of the file to lookup
FILE = 'ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa'

example_attribute_df = vt_lookup.lookup_ioc(observable=FILE, vt_type='file')
example_attribute_df

Example showing all details for this ID

We can use get_object to retrieve all details or just look it up directly at https://www.virustotal.com/gui/home/search

get_object(FILE, "file")

example_relationship_df = vt_lookup.lookup_ioc_relationships(
    observable=FILE, 
    vt_type='file', 
    relationship='execution_parents')
example_relationship_df

Obtaining result for multiple entities

The function lookup_iocs is able to obtain attributes for all the rows in a DataFrame. If no observable_column and observable_type parameters are specified, the function will obtain the attributes of all the entities that are in the column target, and will obtain their types from the target_type column.

This function is especially useful when a user has obtained a set of relationships, and would like to obtain their attributes.

Note: it can take some time to fetch results, depending on the number of nodes and relationships.

example_multiple_attribute_df = vt_lookup.lookup_iocs(example_relationship_df)
example_multiple_attribute_df

# Expand dates
example_multiple_attribute_df.assign(
    first_submission=pd.to_datetime(example_multiple_attribute_df.first_submission_date, unit="s", utc=True),
    last_submission=pd.to_datetime(example_multiple_attribute_df.last_submission_date, unit="s", utc=True)
)

Also, if we would like to obtain the relationships for a set of entities, we have the function lookup_iocs_relationships. Here also, if no observable_column and observable_type parameters are specified, the function will obtain the relationships of all the entities that are in the column target, and will obtain their types from the target_type column.

Note: it can take some time to fetch results

example_multiple_relationship_df = vt_lookup.lookup_iocs_relationships(example_relationship_df, 'contacted_domains')
example_multiple_relationship_df

Simple plot of the relationships

We can display a simple plot of the relataionships locally but it doesn't tell us much about what the nodes are and they types of relationships between them.

from bokeh.io import output_notebook, show
from bokeh.plotting import figure, from_networkx
from bokeh.models import HoverTool

graph = nx.from_pandas_edgelist(
    example_multiple_relationship_df.reset_index(),
    source="source",
    target="target",
    edge_attr="relationship_type",
)

plot = figure(
    title="Simple graph plot", x_range=(-1.1, 1.1), y_range=(-1.1, 1.1), tools="hover"
)
g_plot = from_networkx(graph, nx.spring_layout, scale=2, center=(0, 0))
plot.renderers.append(g_plot)

output_notebook()
show(plot)

MIME type unknown not supported

MIME type unknown not supported

Integration with VTGraph

Once we have some DataFrames with the relationships, we are able to generate and visualize a VT Graph in our notebook. The function create_vt_graph accepts as input a list of Relationship DataFrames.

Note: it can take some time to generate the graph, depending on the number of nodes and relationships.

Unlike our local graph, this displays rich information about the nodes and relationship and allows us to expand our investigation with further searches or ad hoc nodes.

Note: - the inline graph displays node attributes but doesn't allow you edit or to add to the graph with further searches.
Click on the link in the frame to go to the VirusTotal site to view.

graph_id = vt_lookup.create_vt_graph(
    relationship_dfs=[example_relationship_df, example_multiple_relationship_df],
    name="My first Jupyter Notebook Graph",
    private=False,
)
graph_id

'g20091e04457e441ab3d061480caf5e3c626208e1da5a41e08522f78b4e31b574'

vt_lookup.render_vt_graph(
    graph_id = graph_id,
    width = 900,
    height = 600
)