GitHub Repository: YStrano/DataScience_GA
Path: blob/master/lessons/lesson_17/Oil_residuals.ipynb
¹⁹⁰⁴ views

Kernel: Python 3

Analyzing Oil Markets using Residuals

Today we will be analyzing oil markets using the us dollar, industrial production (as a proxy for demand) and the vix

In [1]:

%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np

In [2]:

ip = pd.read_csv('data/oil/ip_oecd.csv') #industrial production is the predcitor of oil demand
vix = pd.read_csv('data/oil/vix.csv') #financial indicator for uncertainty or fear on the market
oil_usd = pd.read_csv('data/oil/oil_usd.csv')

Notice I have different frequencies throughout these datasets

In [4]:

## quarterly
ip.head()

Out[4]:

In [5]:

## monthly
oil_usd.head()

Out[5]:

In [6]:

## Daily
vix.head()

Out[6]:

Define a little function to convert the "DATE" column to datetime, resample to quarterly, and make it the index

In [7]:

def make_date_index(df):
    df['DATE'] = pd.to_datetime(df['DATE'])
    df.set_index('DATE', inplace=True)
    df = df.resample('Q').mean()
    return df

oil_usd = make_date_index(oil_usd)
vix = make_date_index(vix)
ip = make_date_index(ip)

In [8]:

ip.tail()

Out[8]:

In [9]:

## Merge into our final data frame
df = oil_usd.join(ip)
df = df.join(vix)
df.head()

Out[9]:

In [10]:

df.tail()

Out[10]:

In [11]:

df.plot()

Out[11]:

<matplotlib.axes._subplots.AxesSubplot at 0x10983d048>

issue with scale, to understand the data in graph

In [12]:

## Lets drop nulls
df.dropna(inplace=True)

In [13]:

## Lets take logs
df = df.apply(lambda x: np.log(x), axis=1)

In [14]:

df.plot()

Out[14]:

<matplotlib.axes._subplots.AxesSubplot at 0x10f1c1eb8>

In [15]:

## Lets take differences (now we are in log differences!)
df = df.apply(lambda x: x.diff())
df.plot()

Out[15]:

<matplotlib.axes._subplots.AxesSubplot at 0x10f2c40f0>

In [18]:

df.corr()

Out[18]:

In [19]:

import statsmodels.formula.api as smf
lm = smf.ols('oil ~ usd + ip_oecd + vix', data= df).fit()
lm.summary()

Out[19]:

Lets look at the Residuals

In [20]:

residuals = lm.resid

In [21]:

residuals.plot()

Out[21]:

<matplotlib.axes._subplots.AxesSubplot at 0x1a1a4b1fd0>

In [22]:

from statsmodels.graphics.tsaplots import plot_acf
plot_acf(residuals)

Out[22]:

/anaconda3/lib/python3.6/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
  from pandas.core import datetools

Key events in the oil market

March 1999: OPEC Production Cut
2010-2011: Arab Spring/ Libyan civil war
2015 - Present: reduction in oil investment due to shale boom

In [23]:

residuals.rolling(4).mean().plot()

Out[23]:

<matplotlib.axes._subplots.AxesSubplot at 0x1c1a94e5c0>

this shows that log diff is very close to % change

In [24]:

np.log(100)-np.log(110)

Out[24]:

-0.09531017980432477

In [28]:

(np.log(100)-np.log(110)) == (np.log(110)-np.log(100))

Out[28]:

False

In [27]:

print((np.log(100)-np.log(110)), (np.log(110)-np.log(100)))

Out[27]:

-0.09531017980432477 0.09531017980432477

Analyzing Oil Markets using Residuals

Notice I have different frequencies throughout these datasets

Define a little function to convert the "DATE" column to datetime, resample to quarterly, and make it the index

Lets look at the Residuals

Key events in the oil market

Product

Resources

Company