Path: blob/master/tutorials-and-examples/how-tos/TroubleShootingNotebooks.ipynb
3253 views
Troubleshooting Microsoft Sentinel Notebooks
If you are having trouble with Jupyter notebooks run this notebook to help identify where the problem might be.
Select the notebook menu item Cell->Run All - check for any warnings or errors.
Read the text above the cell(s) that produce errors - the text contains links to resources that describe how to fix the error.
Important: you can run the cells individually but please run the Python Version Check cell first, since this contains some function definitions used by the other cells.
Python Version Check
Note You can set the default Python version in Azure Notebooks project settings. For details on how to do this see AzureNotebooks-ConfigurePythonVersion
If you are using a Data Science Virtual Machine as your Azure Notebooks compute you should read Provisioning a DSVM
Checking Python version...
Python version 3.7.10 - OK
Package Import Check
This section checks the import of msticpy and its dependent packages.
Note If you are repeatedly seeing packages going missing when working in Azure Notebooks this may be because the docker containers running the Python kernel are recycled after a few hours when not in use. This causes the environments to reset to defaults. To prevent this you should configure you Azure Notebooks project with a requirements.txt file that is automatically run (and packages installed) when the contain is initialized.
For details on how to do this see AzureNotebooks-ConfigurePythonVersion
Checking msticpy...
msticpy version 1.1.0 - OK
Pandas Version Check
Many of the notebooks and msticpy features require a mininum pandas version of 0.25.0.
Checking pandas...
Pandas version 1.1.3 - OK
Workspace Configuration Check
This section checks for presence of configuration files config.json and msticpyconfig.yaml
The msticpyconfig.yaml can store the workspace and tenant information for your Microsoft Sentinel workspace. It can also store values for multiple workspaces. If you have the values configured in this file you do not need to worry about the values in config.json.
You can specify the location of your msticpyconfig.yaml in the environment variable MSTICPYCONFIG. This will make the file accessible to all notebooks running on the system. For more information on configuring msticpyconfig.yaml see the next cell mstipcy Configuration
If you want to transfer your workspace settings to msticpyconfig.yaml from config.json, simply copy the value of the tenant_id and workspace_id settings to the relevant section.
Note the value names in msticpyconfig.yaml use slightly different naming conventions:
Workspace Configuration - config.json
Creating a Microsoft Notebooks project from Microsoft Sentinel will automatically create a config.json file in the root of your Azure Notebooks project and populate values for your Microsoft Sentinel workspace.
If you have copied the notebooks elsewhere (e.g. to run them locally, or you are running them on a Data Science Virtual machine) you should copy this original config.json to the folder from which you are running notebooks.
Note if you are using a
msticpyconfig.yamlto store your workspace settings, most notebooks will take values from that. As withconfig.json- you must have a locally accessible copy of this file, so you will need to copy it to other systems if you are running notebooks from there.
If you are using the config.json (default config for Microsoft Sentinel with Azure Notebooks), your config.json should look something like this
The tenant_id and workspace_id values must be configured, other values are optional but recommended.
Checking Microsoft Sentinel Workspace config...
Workspace configuration found in 'e:\src\microsoft\msticpyconfig.yaml' - OK
msticpy Initialization
This section duplicates the setup cells of most of the notebooks. It may duplicate warnings seen in the previous cell (since it runs some of the same checks).
For more information on msticpy configuration file settings, please refer to the following items:
MSTICPy configuration ran without critical errors. - OK
msticpy general troubleshooting
To help determine the cause of a problem you can turn on the following settings:
MSTICPY debug logging
Verbose reporting of Exceptions in Jupyter notebooks
How to do these is covered in the code below
From the logging you can see more detail of where an operation in the code is failing.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[6], line 13
10 my_param = "test"
11 func_with_error(param=my_param)
---> 13 func_calling_error()
Cell In[6], line 11, in func_calling_error()
9 def func_calling_error():
10 my_param = "test"
---> 11 func_with_error(param=my_param)
my_param = 'test'
Cell In[6], line 7, in func_with_error(*args=(), **kwargs={'param': 'test'})
5 def func_with_error(*args, **kwargs):
6 param = kwargs.get("param")
----> 7 raise ValueError("This is a deliberate error")
ValueError: This is a deliberate error
Verbose Exception tracing
Notice that in verbose mode you can see the values passed as parameters to functions, which helps a lot when diagnosing what has gone wrong
Example code with a failure
Resultant traceback
Summary
Info/Success:
Python version 3.7.10
MSTICPy configuration ran with errors.