tutorial_10.ipynb

Kernel: R

Tutorial 10 - Clustering

Lecture and Tutorial Learning Goals:

After completing this week's lecture and tutorial work, you will be able to:

• Describe a case where clustering would be an appropriate tool, and what insight it would bring from the data.

• Explain the k-means clustering algorithm.

• Interpret the output of a k-means cluster analysis.

• Perform k-means clustering in R using k-means

• Visualize the output of k-means clustering in R using a scatter plot facetted across each access

• Identify when it is necessary to scale variables before clustering and do this using R

• Use the elbow method to choose the number of clusters for k-means

• Describe advantages, limitations and assumptions of the kmeans clustering algorithm.

### Run this cell before continuing.
library(tidyverse)
library(repr)
library(GGally)
library(broom)
source('tests_tutorial_10.R')

1. Pokemon

We will be working with the Pokemon dataset from Kaggle, which can be found here. This dataset compiles the statistics on 721 Pokemon. The information in this dataset includes Pokemon name, type, health points, attack strength, defensive strength, speed points etc. We are interested in seeing if there are any sub-groups/clusters of pokemon based on these statistics. And if so, how many sub-groups/clusters there are.

Source: https://media.giphy.com/media/3oEduV4SOS9mmmIOkw/giphy.gif

Question 1.0
{points: 1}

Use read_csv to load pokemon.csv from the data/ folder.

Assign your answer to an object called pm_data.

# your code here
fail() # No Answer - remove if you provide an answer
head(pm_data)

test_1.0()

Question 1.1
{points: 1}

Create a matrix of plots using ggpairs, choosing columns 5 to 11 (or equivalently, Total to Speed) from pm_data. There are several ways to do this, the most familar way would be using the select function to give a range of column names:

data %>% select(start_column_name:end_column_name)

Another is to pass in the column numbers to the ggpairs function as so:

ggpairs(name_of_dataset, columns = c(number:number))

Assign your answer to an object called pm_pairs.

# your code here
fail() # No Answer - remove if you provide an answer
pm_pairs

test_1.1()

Question 1.2
{points: 1}

Make a scatterplot to visualize the relationship between Speed and Defense of the Pokemon. Put the Speed variable on the x-axis, and the Defense variable on the y-axis.

Assign your plot to an object called pm_scatter. Don't forget to do everything needed to make an effective visualization.

# your code here
fail() # No Answer - remove if you provide an answer
pm_scatter

test_1.2()

Question 1.3
{points: 1}

Select the columns Speed and Defense, creating a new dataframe with only those columns.

Assign your answer to an object named km_data.

# your code here
fail() # No Answer - remove if you provide an answer
head(km_data)

test_1.3()

Question 1.4.1
{points: 3}

We are going to cluster the Pokemon based on their Speed and Defense. Do we need to scale our variables before clustering? Explain why or why not.

YOUR ANSWER HERE

Question 1.4.2
{points: 1}

Now, let's use the kmeans function to cluster the Pokemon based on their Speed and Defense variables. For this question, use K = 4.

Assign your answer to an object called pokemon_clusters.

Note: We set the random seed here because kmeans initializes observations to random clusters.

#DON'T CHANGE THE SEED VALUE BELOW!
set.seed(2019)

# your code here
fail() # No Answer - remove if you provide an answer
pokemon_clusters

test_1.4.2()

Question 1.5
{points: 1}

Let's visualize the clusters we built in pokemon_clusters. For this we can use the broom package.

"The broom package takes the messy output of built-in functions in R, such as lm, nls, or t.test, and turns them into tidy data frames." - Broom Package

Your tasks:

1. Use the augment function create a data frame with the cluster assignments for each data point from Kmeans (should have the columns Speed and Defense and .cluster).

2. Create a scatter plot of Speed (x-axis) vs Defense (y-axis) with the points coloured by their cluster assignment. N

Name this plot answer1.5.

# your code here
fail() # No Answer - remove if you provide an answer
answer1.5

test_1.5()

Question 1.6
{points: 3}

Below you can see multiple initializations of k-means with different seeds for K = 4. Can you explain what is happening and how we can mitigate this in the kmeans function?

YOUR ANSWER HERE

Question 1.7
{points: 1}

We know that choosing a K is an important step of the process. We can do this by examining how the total within-cluster sum of squares changes as we change K on a plot (which we call an elbow plot).

For this exercise, from K = 1 to K = 10, calculate the total within-cluster sum of squares. Set nstart to be 10.

Assign your answer to a data frame object named elbow_stats that has the columns k, totss, tot.withinss, betweenss, and iter.

Remember, to acess the total within-cluster sum of squares, you can use the glance function also from the broom package:

glance(pokemon_clusters)

# your code here
fail() # No Answer - remove if you provide an answer
head(elbow_stats)

test_1.7()

Question 1.8
{points: 3}

Create the elbow plot. Put the within-cluster sum of squares on the y-axis, and the number of clusters on the x-axis.

Assign your plot to an object called elbow_plot

# your code here
fail() # No Answer - remove if you provide an answer
elbow_plot

Question 1.9
{points: 3}

Based on the elbow plot above, what value of k do you choose? Explain why.

YOUR ANSWER HERE

Question 1.10
{points: 3}

Using the value that you chose for k, perform the k-means algorithm and create a plot to visualize the clusters. Again, set nstart to be 10, and set the seed to be 2019.

# your code here
fail() # No Answer - remove if you provide an answer

Question 1.11
{points: 3}

Using Speed and Defense, we find 3 clusters of pokemon. However, we have more information in our dataset that might be useful for clustering. Let's incorporate all of the numeric values to our kmeans model. Again use nstart = 10.

Your tasks:

1. Select the numeric values only. For example, do not include the # or Generation columns (they are not pokemon statistics).

2. Use the elbow plot method to determine the number of clusters.

3. Train a k-means model with the number of clusters determined in (2).

4. Print the cluster means for the trained model.

#DON'T CHANGE THIS SEED VALUE
set.seed(2019)

# your code here
fail() # No Answer - remove if you provide an answer

Question 1.12
{points: 3}

Visualizing these clusters is not a simple task given the high-dimensionality of the model. But does the cluster means output help? Justify your reasoning.

YOUR ANSWER HERE

2. Tourism Reviews

Source: https://media.giphy.com/media/xUNd9IsOQ4BSZPfnLG/giphy.gif

The Ministry of Land, Infrastructure, Transport and Tourism of Japan is interested in knowing the type of tourists that visit East Asia. They know the majority of their visitors come from this region and would like to stay competitive in the region to keep growing the tourism industry. For this, they have hired us to perform segmentation of the tourists. A dataset from TripAdvisor has been scraped and it's provided to you.

This dataset contains the following variables:

• User ID : Unique user id

• Category 1 : Average user feedback on art galleries

• Category 2 : Average user feedback on dance clubs

• Category 3 : Average user feedback on juice bars

• Category 4 : Average user feedback on restaurants

• Category 5 : Average user feedback on museums

• Category 6 : Average user feedback on resorts

• Category 7 : Average user feedback on parks/picnic spots

• Category 8 : Average user feedback on beaches

• Category 9 : Average user feedback on theaters

• Category 10 : Average user feedback on religious institutions

Question 2.0
{points: 3}

Load the data set from https://archive.ics.uci.edu/ml/machine-learning-databases/00484/tripadvisor_review.csv and clean it so that only the Category # columns are in the data frame (i.e., remove the User ID column).

# your code here
fail() # No Answer - remove if you provide an answer

Question 2.1
{points: 3}

Perform k-means and vary K from 1 to 10 to identify the optimal number of clusters. Create an elbow plot to help you choose K. At all steps use nstart = 100 and do not forget to set a seed.

#DON'T CHANGE THIS SEED VALUE
set.seed(2019)

# your code here
fail() # No Answer - remove if you provide an answer

Question 2.2
{points: 3}

From the elbow plot above, which K should you choose? Explain why you chose that K.

YOUR ANSWER HERE

Question 2.3
{points: 3}

Run kmeans (don't forget nstart) again, with the optimal K, and then use the augment function to get the cluster assignments for each point. Name the data frame cluster_assignments.

#DONT CHANGE THIS SEED VALUE
set.seed(2019)

# your code here
fail() # No Answer - remove if you provide an answer
head(cluster_assignments)

For the following 2 questions use the following plot as reference.

options(repr.plot.height = 6, repr.plot.width = 8)
cluster_assignments %>%
    gather(key = 'category', value = 'value', -.cluster) %>%
    ggplot(aes(value, fill = .cluster)) +
        geom_density(alpha = 0.4, colour = 'white') +
        facet_wrap(~ category, scales = 'free') +
        theme_minimal()

Question 2.4
{points: 3}

From the plots above, which categories might we hypothesize are driving the clustering? (i.e., are useful to distinguish between the type of tourists?) And explain why you think so for each category? We list the table of the categories below.

• Category 1 : Average user feedback on art galleries

• Category 2 : Average user feedback on dance clubs

• Category 3 : Average user feedback on juice bars

• Category 4 : Average user feedback on restaurants

• Category 5 : Average user feedback on museums

• Category 6 : Average user feedback on resorts

• Category 7 : Average user feedback on parks/picnic spots

• Category 8 : Average user feedback on beaches

• Category 9 : Average user feedback on theaters

• Category 10 : Average user feedback on religious institutions

YOUR ANSWER HERE

Question 2.5
{points: 3}

Discuss one disadvantage of not being able to visualize the clusters when dealing with multidimensional data.

YOUR ANSWER HERE