Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
dsc-courses
GitHub Repository: dsc-courses/dsc10-2022-fa
 Path: blob/main/lectures/lec07/lec07.ipynb
3058 views
Kernel: Python 3 (ipykernel)
# Set up packages for lecture. Don't worry about understanding this code, but
# make sure to run it if you're following along.
import numpy as np
import babypandas as bpd
import pandas as pd

import matplotlib.pyplot as plt
from matplotlib_inline.backend_inline import set_matplotlib_formats
set_matplotlib_formats("svg")
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = (10, 5)

np.set_printoptions(threshold=20, precision=2, suppress=True)
pd.set_option("display.max_rows", 7)
pd.set_option("display.max_columns", 8)
pd.set_option("display.precision", 2)

from IPython.display import HTML, display, IFrame

Lecture 7 – Data Visualization 📈

DSC 10, Fall 2022

Announcements

Agenda

  • Why visualize?

  • Terminology.

  • Scatter plots.

  • Line plots.

  • Bar charts.

Don't forget about the DSC 10 Reference Sheet and the Resources tab of the course website!

Aside: keyboard shortcuts

There are several keyboard shortcuts built into Jupyter Notebooks designed to help you save time. To see them, either click the keyboard button in the toolbar above or hit the H key on your keyboard (as long as you're not actively editing a cell).

Particularly useful shortcuts:

ActionKeyboard shortcut
Run cell + jump to next cellSHIFT + ENTER
Save the notebookCTRL/CMD + S
Create new cell above/belowA/B
Delete cellDD
Convert cell to MarkdownM
Convert cell to codeY

Note: the latter three only work if you're not actively editing a cell (to exit "edit mode", click somewhere outside of a cell).

Why visualize?

Run these cells to load the Little Women data from Lecture 1.

chapters = open('data/lw.txt').read().split('CHAPTER ')[1:]

# Counts of names in the chapters of Little Women

counts = bpd.DataFrame().assign(
    Amy=np.char.count(chapters, 'Amy'),
    Beth=np.char.count(chapters, 'Beth'),
    Jo=np.char.count(chapters, 'Jo'),
    Meg=np.char.count(chapters, 'Meg'),
    Laurie=np.char.count(chapters, 'Laurie'),
)

# cumulative number of times each name appears

lw_counts = bpd.DataFrame().assign(
    Amy=np.cumsum(counts.get('Amy')),
    Beth=np.cumsum(counts.get('Beth')),
    Jo=np.cumsum(counts.get('Jo')),
    Meg=np.cumsum(counts.get('Meg')),
    Laurie=np.cumsum(counts.get('Laurie')),
    Chapter=np.arange(1, 48, 1)
)

lw_counts

Little Women

In Lecture 1, we were able to answer questions about the plot of Little Women without having to read the novel. Some of those questions included:

  • Who is the main character?

  • Which pair of characters gets married in Chapter 35?

lw_counts.plot(x='Chapter');

Napoleon's March

John Snow

Why visualize?

  • Computers are better than humans at crunching numbers, but humans are better at identifying visual patterns.

  • Visualizations allow us to understand lots of data quickly – they make it easier to spot trends and communicate our results with others.

  • There are many types of visualizations; in this class, we'll look at scatter plots, line plots, bar charts, and histograms, but there are many others.

    • The right choice depends on the type of data.

Terminology

Individuals and variables

  • Individual (row): Person/place/thing for which data is recorded. Also called an observation.

  • Variable (column): Something that is recorded for each individual. Also called a feature.

Types of variables

There are two main types of variables:

  • Numerical: It makes sense to do arithmetic with the values.

  • Categorical: Values fall into categories, that may or may not have some order to them.

Examples of numerical variables

  • Salaries of NBA players 🏀.

    • Individual: an NBA player.

    • Variable: their salary.

  • Movie gross earnings 💰.

    • Individual: a movie.

    • Variable: its gross earnings.

  • Booster doses administered per day 💉.

    • Individual: date.

    • Variable: number of booster doses administered on that date.

Examples of categorical variables

  • Movie genres 🎬.

    • Individual: a movie.

    • Variable: its genre.

  • Zip codes 🏠.

    • Individual: US resident.

    • Variable: zip code.

      • Even though they look like numbers, zip codes are categorical (arithmetic doesn't make sense).

  • Level of prior programming experience for students in DSC 10 🧑‍🎓.

    • Individual: student in DSC 10.

    • Variable: their level of prior programming experience, e.g. none, low, medium, or high.

      • There is an order to these categories!

Concept Check ✅ – Answer at cc.dsc10.com

Which of these is not a numerical variable?

A. Fuel economy in miles per gallon.

B. Number of quarters at UCSD.

C. College at UCSD (Sixth, Seventh, etc).

D. Bank account number.

E. More than one of these are not numerical variables.

Types of visualizations

The type of visualization we create depends on the kinds of variables we're visualizing.

  • Scatter plot: numerical vs. numerical.

  • Line plot: sequential numerical (time) vs. numerical.

  • Bar chart: categorical vs. numerical.

  • Histogram: numerical.

    • Will cover next time.

Note: We may interchange the words "plot", "chart", and "graph"; they all mean the same thing.

Scatter plots

Dataset of 50 top-grossing actors

ColumnContents
'Actor'Name of actor
'Total Gross'Total gross domestic box office receipt, in millions of dollars, of all of the actor’s movies
'Number of Movies'The number of movies the actor has been in
'Average per Movie'Total gross divided by number of movies
'#1 Movie'The highest grossing movie the actor has been in
'Gross'Gross domestic box office receipt, in millions of dollars, of the actor’s #1 Movie
actors = bpd.read_csv('data/actors.csv').set_index('Actor')
actors

Scatter plots

What is the relationship between 'Number of Movies' and 'Total Gross'?

actors.plot(kind='scatter', x='Number of Movies', y='Total Gross');

Scatter plots

  • Scatter plots visualize the relationship between two numerical variables.

  • To create one from a DataFrame df, use

df.plot(
    kind='scatter', 
    x=x_column_for_horizontal, 
    y=y_column_for_vertical
)

  • The resulting scatter plot has one point per row of df.

  • If you put a semicolon after a call to .plot, it will hide the weird text output that displays.

Scatter plots

What is the relationship between 'Number of Movies' and 'Average per Movie'?

actors.plot(kind='scatter', x='Number of Movies', y='Average per Movie');

Note that in the above plot, there's a negative association and an outlier.

Who was in 60 or more movies?

actors[actors.get('Number of Movies') >= 60]

Who is the outlier?

Whoever they are, they made very few, high grossing movies.

actors[actors.get('Number of Movies') < 10]

Anthony Daniels

Line plots 📉

Dataset aggregating movies by year

ColumnContent
'Year'Year
'Total Gross in Billions'Total domestic box office gross, in billions of dollars, of all movies released
'Number of Movies'Number of movies released
'#1 Movie'Highest grossing movie
movies_by_year = bpd.read_csv('data/movies_by_year.csv').set_index('Year')
movies_by_year

Line plots

How has the number of movies changed over time? 🤔

movies_by_year.plot(kind='line', y='Number of Movies');

Line plots

  • Line plots show trends in numerical variables over time.

  • To create one from a DataFrame df, use

df.plot(
    kind='line', 
    x=x_column_for_horizontal, 
    y=y_column_for_vertical
)

Plotting tip

  • Tip: if you want the x-axis to be the index, omit the x= argument!

  • Doesn't work for scatter plots, but works for most other plot types.

movies_by_year.plot(kind='line', y='Number of Movies');

Since the year 2000

We can create a line plot of just 2000 onwards by querying movies_by_year before calling .plot.

movies_by_year[movies_by_year.index >= 2000].plot(kind='line', y='Number of Movies');

What do you think explains the declines around 2008 and 2020?

How did this affect total gross?

movies_by_year[movies_by_year.index >= 2000].plot(kind='line', y='Total Gross in Billions');

What was the top grossing movie of 2016? 🐟

...

Bar charts 📊

Dataset of the global top 200 songs on Spotify as of Tuesday (10/4/22)

Downloaded from here – check it out!

charts = (bpd.read_csv('data/regional-global-daily-2022-10-04.csv')
          .set_index('rank')
          .get(['track_name', 'artist_names', 'streams', 'uri'])
         )
charts

Bar charts

How many streams do the top 10 songs have?

charts

charts.take(np.arange(10))

charts.take(np.arange(10)).plot(kind='barh', x='track_name', y='streams');

Bar charts

  • Bar charts visualize the relationship between a categorical variable and a numerical variable.

  • In a bar chart...

    • The thickness and spacing of bars is arbitrary.

    • The order of the categorical labels doesn't matter.

  • To create one from a DataFrame df, use

df.plot(
    kind='barh', 
    x=categorical_column_name, 
    y=numerical_column_name
)

  • The "h" in 'barh' stands for "horizontal".

    • It's easier to read labels this way.

  • In the previous chart, we set y='Streams' even though streams are measured by x-axis length.

# The bars appear in the opposite order relative to the DataFrame
(charts
 .take(np.arange(10))
 .sort_values(by='streams')
 .plot(kind='barh', x='track_name', y='streams')
);

How many songs do the top 15 artists have in the top 200?

First, let's create a DataFrame with a single column that describes the number of songs in the top 200 per artist. This involves using .groupby with .count(). Since we want one row per artist, we will group by 'artist_names'.

charts

songs_per_artist = charts.groupby('artist_names').count()
songs_per_artist

Using .sort_values and .take, we'll keep just the top 15 artists. Note that all columns in songs_per_artist contain the same information (this is a consequence of using .count()).

top_15_artists = (songs_per_artist
                  .sort_values('streams', ascending=False)
                  .take(np.arange(15)))
top_15_artists

Using .assign and .drop, we'll create a column named 'count' that contains the same information that the other 3 columns contain, and then .get only that column (or equivalently, drop the other 3 columns).

# If we give .get a list, it will return a DataFrame instead of a Series!
top_15_artists = (top_15_artists
                  .assign(count=top_15_artists.get('streams'))
                  .get(['count']))
top_15_artists

Before calling .plot(kind='barh', y='count'), we'll sort top_15_artists by 'count' in increasing order. This is because, weirdly, Python reverses the order of rows when creating bars in horizontal bar charts.

top_15_artists.sort_values(by='count').plot(kind='barh', y='count');

Vertical bar charts

To create a vertical bar chart, use kind='bar' instead of kind='barh'. These are typically harder to read, though.

top_15_artists.plot(kind='bar', y='count');

Aside: How many streams did Justin Bieber's songs on the chart receive?

(charts
 [charts.get('artist_names') == 'Justin Bieber']
 .sort_values('streams')
 .plot(kind='barh', x='track_name', y='streams')
);

It seems like we're missing a popular song...

Answer: Using .str.contains.

bieber = charts[charts.get('artist_names').str.contains('Justin Bieber')]
bieber

bieber.sort_values('streams').plot(kind='barh', x='track_name', y='streams');

Fun demo 🎵

# Run this cell, don't worry about what it does.
def show_spotify(uri):
    code = uri[uri.rfind(':')+1:]
    src = f"https://open.spotify.com/embed/track/{code}"
    width = 400
    height = 75
    display(IFrame(src, width, height))

Let's find the URI of a song we care about.

charts

favorite_song = 'Mr. Brightside'

song_uri = (charts
            [charts.get('track_name') == favorite_song]
            .get('uri')
            .iloc[0])
song_uri

Watch what happens! 🎶

show_spotify(song_uri)

Try it out yourself!

Summary

Summary

  • Visualizations make it easy to extract patterns from datasets.

  • There are two main types of variables: categorical and numerical.

  • The types of the variables we're visualizing inform our choice of which type of visualization to use.

  • Today, we looked at scatter plots, line plots, and bar charts.

  • Next time: Histograms and overlaid plots.

Let's discuss!

  • As mentioned earlier, visualizations allow us to easily spot trends and communicate our results with others.

  • Some visualizations make it more difficult to see the trend in data, by

    • Adding "chart junk."

    • Using misleading axes and sizes.

  • In this thread on EdStem, post some examples of particularly misleading or interesting visualizations! We'll share our favorites in class on Monday.