Jupyter notebook Linear Regression.ipynb

¹⁸²¹ views

Kernel: Python 2 (SageMath)

NDL Training #5: Linear Regression

To motivate our example of Linear Regression, we will be looking at the problem of predicitng housing prices based on some given features. This notebook will help to give you a bit of a flavor of what it's like to perform predictive analytics on a given dataset. By the end of this, you should have a good enough understanding of the process behind machine learning and being able to leverage scikit-learn to perform regression analysis.

Let's start off by importing some of the required libraries we will need for today...

In [303]:

import numpy as np
import pandas as pd
from numpy import genfromtxt
from sklearn import linear_model
from sklearn.cross_validation import train_test_split, cross_val_score

Importing the Data

First we will have to import the data. We will use pandas again to import our CSV file.

Let's take a look at the first few records to see what we are working with...

In [304]:

data = pd.read_csv('Sold Tenafly Houses.csv')
data.head(10)

Out[304]:

Exploratory Data Analysis

Before we start diving into the dataset and building our models, let's take a look at the data and explore some statistical properties of the dataset. In this process, we will also look at what data we might want to potentially utilize for our input into our regression model.

Notice when we saw the first few rows of the data, there are many values with NaN or empty values. We will have to find some way to mitigate this issue by performing some preprocessing methods.

Last Sold Price

Let's start by looking at the last sold price, our target values for our regression problem.

It would probably be best if we removed any NaN values in this column, since we will need all samples to have a price listed or else it we would not have the corresponding values to learn about for every features. So let's start by selecting only the rows where the data is not null (or NaN in this case)...

In [305]:

data = data[pd.notnull(data['LAST SOLD PRICE'])]
data.head(5)

Out[305]:

Now for some descriptive stats regarding the Last Sold Price...

In [306]:

print data.shape # Our new dataset dimension
data['LAST SOLD PRICE'].describe()

Out[306]:

(184, 13)

count    1.840000e+02
mean     1.085071e+06
std      7.176260e+05
min      2.950000e+05
25%      6.187500e+05
50%      8.465000e+05
75%      1.277500e+06
max      4.450000e+06
Name: LAST SOLD PRICE, dtype: float64

Bed, Bath and Beyond!

Now let's check out some of the input features we have in the dataset, primarily to see how many NaNs they might have and if we can somehow augment the dataset a bit to make it easier for us to work with.

There are certainly many different ways we can approach this, but we will only try few techniques on how to handle missing data.

In [307]:

# Beds
print 'NaN Count: ' + str(data['BEDS'].isnull().sum())    # Great! We have no NaNs!
data['BEDS'].describe()

Out[307]:

NaN Count: 0

count    184.000000
mean       4.206522
std        1.192307
min        2.000000
25%        3.000000
50%        4.000000
75%        5.000000
max        8.000000
Name: BEDS, dtype: float64

In [308]:

# Baths
print 'NaN Count: ' + str(data['BATHS'].isnull().sum())    # Great! We have no NaNs!
data['BATHS'].describe()

Out[308]:

NaN Count: 0

count    184.000000
mean       3.326087
std        1.514835
min        1.000000
25%        2.375000
50%        3.000000
75%        4.000000
max        9.500000
Name: BATHS, dtype: float64

Now let's look at square foot and lot size. For those that are not really familiar with housing terminology:

Sq Foot refers to how much does your house take up floor area wise.
Lot Size refers to the size of the land or property that you own.

In [309]:

# Sq Feet
print 'NaN Count: ' + str(data['SQFT'].isnull().sum())    # Hmm, pretty much SQFT is useless in this dataset. Let's throw this out!
del data['SQFT']  # Drop the SQFT column from our dataset

Out[309]:

NaN Count: 184

In [310]:

# Lot Size
print 'NaN Count: ' + str(data['LOT SIZE'].isnull().sum())  # Okay, good enough... let's see if we can try some imputation!

data['LOT SIZE'] = data['LOT SIZE'].fillna(data['LOT SIZE'].mean())  # Perform imputation and fill it by the mean value of the lot size column.
data.head(5)

Out[310]:

NaN Count: 16

In [311]:

# Year Built - Also useless, throwing away...
print 'NaN Count: ' + str(data['YEAR BUILT'].isnull().sum())
del data['YEAR BUILT']

Out[311]:

NaN Count: 184

In [312]:

# Days on Market
print 'NaN Count: ' + str(data['DAYS ON MARKET'].isnull().sum())  # Perfect, no NaNs!
data['DAYS ON MARKET'].describe()

Out[312]:

NaN Count: 0

count    184.000000
mean     216.201087
std      146.833565
min        1.000000
25%       85.250000
50%      194.500000
75%      370.750000
max      444.000000
Name: DAYS ON MARKET, dtype: float64

Data Set Final Form

Here is the dataset in it's final form. For our first regression analysis, we will use Lot Size as our input into the model.

As for other interesting types of exploratory data analysis, consider some of the following analysis as an exercise:

House Numbers
Counts of Street Names ending with St, Ave, Ter, Rd, Dr, etc.
Analysis on the street names themselves
Longitude and Latitude

In [313]:

data.head(10)

Out[313]:

Simple Linear Regression

Now that we have understood the scope of our data, analyzed some of the features and preprocessed some of the NaN values, we can now start building our models.

Recall that the linear regression model is in the form of $y = \theta_0 + \theta_1 x$ and we want to find the parameters for the corresponding target function. To keep it simple, we will first start out with a simple regression model - a mapping between one value to another. For this example, we will use our lot size to predict the last sold price of the home.

First we must setup our variables to feed into the model, we need our input parameters $X$ and the target values $y$ and also reshape the dataset to make sure the dimensions are proper. Let's also print the shape just to make sure everything is consistent.

In [314]:

X = data['LOT SIZE'].values.reshape(data['LOT SIZE'].count(),1)
y = data['LAST SOLD PRICE'].values.reshape(data['LAST SOLD PRICE'].count(),1)

print X.shape
print y.shape

Out[314]:

(184, 1)
(184, 1)

Now, before we train our data set, we must split our data set into training and validation, as we need to evaluate our model to see how well it performs on accuracy of predicting housing prices.

For that, we use a special function provided by scikit learn to help us partition the dataset. This function also shuffles the dataset before hand so that we can get a random sample for each subsample.

In [315]:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=9892)

Now that we have our training and validation (which we will call it testing in our program), we will now start to build our Linear Regression model.

In [316]:

simp_reg = linear_model.LinearRegression(fit_intercept=True)  # Creates a Scikit Linear Regression Model Object
simp_reg.fit(X_train, y_train)  # Train the model on the training dataset.

Out[316]:

LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

Now that we have trained our first model, let's see how well it did and test it against our validation set.

In [317]:

print 'Coefficients: \n' + str(simp_reg.coef_)
# The mean squared error
print 'Mean squared error: ' + str(np.mean((simp_reg.predict(X_test) - y_test) ** 2))
print 'Variance score: ' + str(simp_reg.score(X_test, y_test))

Out[317]:

Coefficients: 
[[ 43.07059549]]
Mean squared error: 295412627132.0
Variance score: 0.435433041364

Congratulations! You trained your first machine learning model!!

But unfortunately it's probably not the best, (whew, that MSE...), because of the features we chose towards our model. As you can see, we have scored around 0.43 variance score (which is pretty low)... 😦

By the way, variance score according to scikit learn's documentation is defined as follows:

expvar(y, \hat{y}) = 1 - \dfrac{Var(y - \hat{y})}{Var(y)}

The higher the variance score (meaning, close to 1.0), the better our model is.

But worry not, we can do better than this!

In order to better imporve our model, one way to handle this issue is to use better or more features. Another way to improve this would be to find a dataset with a larger sample size. (But since we can't get any more data, we can only rely on what we are given, for now...)

Let's try to use some more features from our dataset, such as:

Bed Count
Bath Count
Days on Market

In [318]:

X = data[['LOT SIZE', 'BEDS', 'BATHS', 'DAYS ON MARKET']].values
y = data['LAST SOLD PRICE'].values.reshape(data['LAST SOLD PRICE'].count(),1)

print X.shape
print y.shape

Out[318]:

(184, 4)
(184, 1)

And let's split our data set and train the model!

In [319]:

# Split Dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=9892)

# Train Linear Model
mult_reg = linear_model.LinearRegression(fit_intercept=True)  # Creates a Scikit Linear Regression Model Object
mult_reg.fit(X_train, y_train)  # Train the model on the training dataset.

# Evaluation
print 'Coefficients: \n' + str(mult_reg.coef_)
print 'Mean squared error: ' + str(np.mean((mult_reg.predict(X_test) - y_test) ** 2))
print 'Variance score: ' + str(mult_reg.score(X_test, y_test))

Out[319]:

Coefficients: 
[[  1.66243719e+01   7.81617117e+04   2.68432243e+05   3.48409084e+02]]
Mean squared error: 68339305680.0
Variance score: 0.869395853733

Wow! With just adding three more features to our training data, we got a huge boost in our variance score! As you can see, the features that you put into the model holds a great significance as to how well your model performs.

Remember, it's not always about the algorithms... it's all in the data.

Now to wrap this up, let's try predicting how much I should sell my house for according to the model we just made...

In [340]:

myHome = np.array([[9374.0, 4, 3, 0]])
print 'Predicted Price: $' + str(mult_reg.predict(myHome)[0][0])

Out[340]:

Predicted Price: $784748.595245