GitHub Repository: Azure/Azure-Sentinel-Notebooks
Path: blob/master/tutorials-and-examples/feature-tutorials/FoliumMap.ipynb
³²⁵³ views

Kernel: Python (condadev)

Title: msticpy - Folium Map Plotting

Introduction

This module contains a class that wraps the folium package to plot geo-location data.

Read the Folium documentation

You must have msticpy installed to run this notebook:

%pip install --upgrade msticpy

In [1]:

# Imports
import sys
MIN_REQ_PYTHON = (3,6)
if sys.version_info < MIN_REQ_PYTHON:
    print('Check the Kernel->Change Kernel menu and ensure that Python 3.6')
    print('or later is selected as the active kernel.')
    sys.exit("Python %s.%s or later is required.\n" % MIN_REQ_PYTHON)


from IPython.display import display
import pandas as pd

import msticpy.sectools as sectools
from msticpy.nbtools import *
from msticpy.nbtools.entityschema import IpAddress, GeoLocation
from msticpy.nbtools.foliummap import FoliumMap

FoliumMap class

FoliumMap(
    title: str = 'layer1',
    zoom_start: float = 2.5,
    tiles=None,
    width: str = '100%',
    height: str = '100%',
    location: list = None,
)
Wrapper class for Folium/Leaflet mapping.

Parameters
----------
title : str, optional
    Name of the layer (the default is 'layer1')
zoom_start : int, optional
    The zoom level of the map (the default is 7)
tiles : [type], optional
    Custom set of tiles or tile URL (the default is None)
width : str, optional
    Map display width (the default is '100%')
height : str, optional
    Map display height (the default is '100%')
location : list, optional
    Location to center map on

Attributes
----------
folium_map : folium.Map

In [2]:

folium_map = FoliumMap(width="50%", height="50%", location=(47.5982328,-122.331), zoom_start=14)
folium_map

Out[2]:

The underlying folium map object is accessible as the folium_map attribute

In [3]:

type(folium_map.folium_map)

Out[3]:

folium.folium.Map

Adding IP Entities to the map

fol_map.add_ip_cluster(
    ip_entities: Iterable[msticpy.nbtools.entityschema.IpAddress],
    **kwargs,
)

Add a collection of IP Entities to the map.

Parameters
----------
ip_entities : Iterable[IpAddress]
    a iterable of IpAddress Entities

Other Parameters
----------------
    kwargs: icon properties to use for displaying this cluster

In [4]:

import pickle
with open(b"data/ip_entities.pkl", "rb") as fh:
    ip_entities = pickle.load(fh)
ip_entities = [ip for ip in ip_entities if ip.Location and ip.Location.Latitude]

folium_map = FoliumMap(zoom_start=9)
folium_map.add_ip_cluster(ip_entities=ip_entities, color='orange')
folium_map.center_map()
folium_map

Out[4]:

In [13]:

# Read in some data
geo_loc_df = pd.read_csv("data/ip_locs.csv", index_col=0)
geo_loc_df.head()

Out[13]:

In [14]:

# Create IP and GeoLocation Entities from the dataframe
def create_ip_entity(row):
    ip_ent = IpAddress(Address=row["AllExtIPs"])
    geo_loc = create_geo_entity(row)
    ip_ent.Location = geo_loc
    return ip_ent

def create_geo_entity(row):
    # get subset of fields for GeoLocation
    loc_props = row[["CountryCode", "CountryName","State", "City", "Longitude", "Latitude"]]
    geo_loc = GeoLocation(**loc_props.to_dict())
    return geo_loc

geo_locs = list(geo_loc_df.apply(create_geo_entity, axis=1).values)
ip_ents = list(geo_loc_df.apply(create_ip_entity, axis=1).values)
ip_ents[:5]

Out[14]:

[IpAddress(Address=65.55.44.109, Location={ 'AdditionalData': {},
   'City': 'Boydton',
   'C...),
 IpAddress(Address=13.71.172.128, Location={ 'AdditionalData': {},
   'City': 'Toronto',
   '...),
 IpAddress(Address=13.71.172.130, Location={ 'AdditionalData': {},
   'City': 'Toronto',
   '...),
 IpAddress(Address=40.124.45.19, Location={ 'AdditionalData': {},
   'City': 'San Antonio',
 ...),
 IpAddress(Address=104.43.212.12, Location={ 'AdditionalData': {},
   'City': 'Des Moines',
 ...)]

In [15]:

geo_loc_df.apply(lambda x: (x.Latitude, x.Longitude), axis=1).values

Out[15]:

array([(36.6534, -78.375), (43.6644, -79.4195), (43.6644, -79.4195),
       (29.4247, -98.4935), (41.6006, -93.6112), (41.1399, -104.8193),
       (36.6534, -78.375), (43.6644, -79.4195), (38.7095, -78.1539),
       (38.7095, -78.1539), (36.6534, -78.375), (47.6742, -122.1243),
       (38.7095, -78.1539), (53.3338, -6.2488), (29.4247, -98.4935),
       (41.6006, -93.6112), (41.6006, -93.6112), (41.6006, -93.6112),
       (38.7095, -78.1539), (36.6534, -78.375), (36.6534, -78.375),
       (37.3388, -121.8914), (37.3388, -121.8914), (38.7095, -78.1539),
       (38.7095, -78.1539), (38.7095, -78.1539), (38.7095, -78.1539),
       (38.7095, -78.1539), (38.7095, -78.1539), (38.7095, -78.1539),
       (53.3338, -6.2488), (41.6006, -93.6112), (38.7095, -78.1539),
       (38.7095, -78.1539), (38.7095, -78.1539), (36.6534, -78.375),
       (38.7095, -78.1539), (41.6006, -93.6112), (41.6006, -93.6112),
       (41.6006, -93.6112), (38.7095, -78.1539), (37.3388, -121.8914),
       (37.3388, -121.8914), (38.7095, -78.1539), (41.6006, -93.6112),
       (40.4953, -111.9439), (41.6006, -93.6112), (41.6006, -93.6112),
       (36.6534, -78.375), (38.7095, -78.1539), (38.7095, -78.1539)],
      dtype=object)

Plot IPAddress entities with location data

In [16]:

fmap_ips = FoliumMap()
fmap_ips.add_ip_cluster(ip_entities=ip_ents[:20], color='blue')
fmap_ips.center_map()
fmap_ips

Out[16]:

Use different colors and icons

In [17]:

fmap_ips.add_ip_cluster(ip_entities=ip_ents[30:40], color='red', icon="flash")
fmap_ips.center_map()
fmap_ips

Out[17]:

Custom Icons

By default folium uses the information icon (i). Icons can be taken from the default Bootstrap set. See the default list here glyphicons

Alternatively you can use icons from the Font Awesome collection by adding prefx="fa" and icon="icon_name" to the call to add_ip_cluster or add_geo_cluster.

In [18]:

fmap_ips.add_geoloc_cluster(geo_locations=geo_locs[40:50], color='darkblue', icon="desktop", prefix="fa")
fmap_ips.center_map()
fmap_ips

Out[18]:

Utility Functions

Calculate center point of entity locations

In [19]:

from msticpy.nbtools.foliummap import get_map_center, get_center_ip_entities, get_center_geo_locs

print(get_center_geo_locs(geo_locs))
print(get_center_geo_locs(geo_locs, mode="mean"))

# get_map_center Will accept iterable of any entity type that is either
# an IpAddress entity or an entity that has properties of type IpAddress
print(get_map_center(ip_ents[30:40]))
print(get_map_center(ip_ents[:20]))

print(get_center_ip_entities(ip_ents[:20]))

Out[19]:

(38.7095, -78.375)
(39.70266078431372, -85.0484019607843)
(40.15505, -78.26445)
(39.9247, -79.4195)
(39.9247, -79.4195)

Calculate distance between entity locations

In [27]:

from msticpy.sectools.geoip import entity_distance
print("Distance between")
print(ip_ents[0], ip_ents[1])
print("\n", entity_distance(ip_ents[0], ip_ents[1]), "km")

print("Distance between")
print(ip_ents[0], ip_ents[13])
print("\n", entity_distance(ip_ents[0], ip_ents[13]), "km")

Out[27]:

Distance between
{ 'AdditionalData': {},
  'Address': '65.55.44.109',
  'Location': { 'AdditionalData': {},
                'City': 'Boydton',
                'CountryCode': 'US',
                'CountryName': 'United States',
                'Latitude': 36.6534,
                'Longitude': -78.375,
                'State': 'Virginia',
                'Type': 'geolocation',
                'edges': set()},
  'ThreatIntelligence': [],
  'Type': 'ipaddress',
  'edges': set()} { 'AdditionalData': {},
  'Address': '13.71.172.128',
  'Location': { 'AdditionalData': {},
                'City': 'Toronto',
                'CountryCode': 'CA',
                'CountryName': 'Canada',
                'Latitude': 43.6644,
                'Longitude': -79.4195,
                'State': 'Ontario',
                'Type': 'geolocation',
                'edges': set()},
  'ThreatIntelligence': [],
  'Type': 'ipaddress',
  'edges': set()}

 784.604908273247 km
Distance between
{ 'AdditionalData': {},
  'Address': '65.55.44.109',
  'Location': { 'AdditionalData': {},
                'City': 'Boydton',
                'CountryCode': 'US',
                'CountryName': 'United States',
                'Latitude': 36.6534,
                'Longitude': -78.375,
                'State': 'Virginia',
                'Type': 'geolocation',
                'edges': set()},
  'ThreatIntelligence': [],
  'Type': 'ipaddress',
  'edges': set()} { 'AdditionalData': {},
  'Address': '40.77.226.250',
  'Location': { 'AdditionalData': {},
                'City': 'Dublin',
                'CountryCode': 'IE',
                'CountryName': 'Ireland',
                'Latitude': 53.3338,
                'Longitude': -6.2488,
                'State': 'Leinster',
                'Type': 'geolocation',
                'edges': set()},
  'ThreatIntelligence': [],
  'Type': 'ipaddress',
  'edges': set()}

 5699.044784950343 km