CoCalc -- Mastery Day 1.ipynb

Unit 2 / Mastery Days / Mastery Day 1.ipynb

¹¹²⁴ views
ubuntu2004

Kernel: Python 3 (system-wide)

In [0]:

#Using the attached csv we are going to pracitce some default dataframe methods and functions as a refresher and then we'll take a look at the actual data set and clustering algorithm

#Useeful video: https://www.youtube.com/watch?v=F6kmIpWWEdU&list=PLeo1K3hjS3uuASpe-1LjfG5f14Bnozjwy&index=2
#https://www.geeksforgeeks.org/create-a-new-column-in-pandas-dataframe-based-on-the-existing-columns/

#Using K Means Clustering, determine any insights from the 2012 arrest data.

In [2]:

import pandas as pd
#allows us simple data access features

In [3]:

import numpy as np
#allows us to use numpy specific attributes

In [4]:

import matplotlib.pyplot as plt
#allows us to create plotting and visualizations for our program

In [5]:

import sklearn
#imports the machine learning algorithm we desire

In [6]:

from sklearn.cluster import KMeans
#imports the KMeans clustering algorithm specifically

In [7]:

nypd_data = pd.read_csv("nypd_arrest_2012.csv")
#searches for a local csv file and converts into a pandas dataframe. 
#A dataframe is almost like an excel/sheets data structure that exists in your program.
#A dataframe behaves like an object.

In [8]:

nypd_data.head()
#prints the first 5 rows of data in our dataframe.

Out[8]:

In [9]:

nypd_data.tail()
#prints the last five data entries

Out[9]:

In [10]:

nypd_data.shape
#(rows,col)

Out[10]:

(8296, 19)

In [11]:

nypd_data.dtypes
#Tells us the type of data that each of our entries are.

Out[11]:

ARREST_KEY                    int64
ARREST_DATE                  object
PD_CD                       float64
PD_DESC                      object
KY_CD                       float64
OFNS_DESC                    object
LAW_CODE                     object
LAW_CAT_CD                   object
ARREST_BORO                  object
ARREST_PRECINCT               int64
JURISDICTION_CODE             int64
AGE_GROUP                    object
PERP_SEX                     object
PERP_RACE                    object
X_COORD_CD                    int64
Y_COORD_CD                    int64
Latitude                    float64
Longitude                   float64
New Georeferenced Column     object
dtype: object

In [12]:

nypd_data.describe()
#gives us the mean, median, mode and standard deviation of our data

Out[12]:

In [13]:

nypd_data.columns

Out[13]:

Index(['ARREST_KEY', 'ARREST_DATE', 'PD_CD', 'PD_DESC', 'KY_CD', 'OFNS_DESC',
       'LAW_CODE', 'LAW_CAT_CD', 'ARREST_BORO', 'ARREST_PRECINCT',
       'JURISDICTION_CODE', 'AGE_GROUP', 'PERP_SEX', 'PERP_RACE', 'X_COORD_CD',
       'Y_COORD_CD', 'Latitude', 'Longitude', 'New Georeferenced Column'],
      dtype='object')

In [14]:

nypd_data[0:2]
#Data Slicing here to access the first two rows of my dataframe

Out[14]:

In [15]:

nypd_data["OFNS_DESC"][0:5]
#I can call data by passing in the column name and I can even slice that data as well to get a piece of it

Out[15]:

               DANGEROUS DRUGS
             DANGEROUS WEAPONS
  ASSAULT 3 & RELATED OFFENSES
                FELONY ASSAULT
                FELONY ASSAULT
Name: OFNS_DESC, dtype: object

In [16]:

#Grab the first 10 entries of the agegroup column

In [17]:

nypd_data["AGE_GROUP"][0:10]

Out[17]:

  25-44
  45-64
  25-44
  45-64
  18-24
    <18
  25-44
  18-24
  25-44
  25-44
Name: AGE_GROUP, dtype: object

In [18]:

nypd_data[["AGE_GROUP", "PERP_SEX"]][0:10]
#I can pass in multiple columns and slice them as well.

Out[18]:

In [19]:

nypd_data.columns

Out[19]:

Index(['ARREST_KEY', 'ARREST_DATE', 'PD_CD', 'PD_DESC', 'KY_CD', 'OFNS_DESC',
       'LAW_CODE', 'LAW_CAT_CD', 'ARREST_BORO', 'ARREST_PRECINCT',
       'JURISDICTION_CODE', 'AGE_GROUP', 'PERP_SEX', 'PERP_RACE', 'X_COORD_CD',
       'Y_COORD_CD', 'Latitude', 'Longitude', 'New Georeferenced Column'],
      dtype='object')

In [20]:

#SLice the first ten rows of arrest precicnt and perp race
nypd_data[["ARREST_PRECINCT", "PERP_RACE"]][0:10]

Out[20]:

In [21]:

#SLice the first ten rows of arrest precicnt and arrest boro
nypd_data[["ARREST_PRECINCT","ARREST_BORO"]][0:10]

Out[21]:

In [22]:

#SLice the first ten rows of arrest precicnt and arrest boro and perp race
nypd_data[["ARREST_PRECINCT","ARREST_BORO", "PERP_RACE"]][0:10]

Out[22]:

In [23]:

#SLice the first ten rows of arrest precicnt and arrest boro and perp sex
nypd_data[["ARREST_PRECINCT","ARREST_BORO", "PERP_SEX"]][0:10]

Out[23]:

In [24]:

#SLice the LAST ten rows of arrest precicnt and perp sex
nypd_data[["ARREST_PRECINCT","PERP_SEX"]][8285:]

Out[24]:

In [27]:

#Values method allows us to see all the different types of data for our column
nypd_data["PERP_SEX"].values

Out[27]:

array(['F', 'M', 'M', ..., 'M', 'M', 'M'], dtype=object)

In [28]:

nypd_data["PERP_SEX"].value_counts()

Out[28]:

M    6998
F    1298
Name: PERP_SEX, dtype: int64

In [0]:

#Use value counts on the following individual categories: OFNS DESC, ARRRST BORO & PERP RACE

In [29]:

nypd_data["OFNS_DESC"].value_counts()

Out[29]:

ASSAULT 3 & RELATED OFFENSES      1516
FELONY ASSAULT                     804
PETIT LARCENY                      786
MISCELLANEOUS PENAL LAW            609
CRIMINAL MISCHIEF & RELATED OF     558
BURGLARY                           520
DANGEROUS DRUGS                    519
ROBBERY                            492
DANGEROUS WEAPONS                  427
OFFENSES AGAINST PUBLIC ADMINI     314
GRAND LARCENY                      311
SEX CRIMES                         161
POSSESSION OF STOLEN PROPERTY      133
VEHICLE AND TRAFFIC LAWS           129
OFF. AGNST PUB ORD SENSBLTY &      123
CRIMINAL TRESPASS                  123
GRAND LARCENY OF MOTOR VEHICLE      82
MURDER & NON-NEGL. MANSLAUGHTE      71
INTOXICATED & IMPAIRED DRIVING      70
FORGERY                             66
BURGLAR'S TOOLS                     66
OFFENSES AGAINST THE PERSON         53
UNAUTHORIZED USE OF A VEHICLE       45
FOR OTHER AUTHORITIES               43
RAPE                                37
OFFENSES INVOLVING FRAUD            33
OTHER STATE LAWS (NON PENAL LA      32
OTHER OFFENSES RELATED TO THEF      27
FRAUDS                              25
GAMBLING                            22
NYS LAWS-UNCLASSIFIED FELONY        22
OTHER TRAFFIC INFRACTION            19
INTOXICATED/IMPAIRED DRIVING        10
OTHER STATE LAWS                     7
ARSON                                6
THEFT-FRAUD                          3
ADMINISTRATIVE CODE                  3
ALCOHOLIC BEVERAGE CONTROL LAW       3
KIDNAPPING & RELATED OFFENSES        2
OFFENSES RELATED TO CHILDREN         2
ENDAN WELFARE INCOMP                 2
HARRASSMENT 2                        2
OFFENSES AGAINST PUBLIC SAFETY       2
MOVING INFRACTIONS                   2
DISORDERLY CONDUCT                   2
JOSTLING                             1
CHILD ABANDONMENT/NON SUPPORT        1
HOMICIDE-NEGLIGENT,UNCLASSIFIE       1
Name: OFNS_DESC, dtype: int64

In [30]:

nypd_data["ARREST_BORO"].value_counts()

Out[30]:

K    2291
B    1964
Q    1843
M    1811
S     387
Name: ARREST_BORO, dtype: int64

In [31]:

nypd_data["PERP_RACE"].value_counts()

Out[31]:

BLACK                             4046
WHITE HISPANIC                    2210
WHITE                              911
BLACK HISPANIC                     703
ASIAN / PACIFIC ISLANDER           388
UNKNOWN                             21
AMERICAN INDIAN/ALASKAN NATIVE      17
Name: PERP_RACE, dtype: int64

In [0]:

#Conditional Formatting

In [37]:

females = nypd_data[nypd_data["PERP_SEX"] == "F"]

In [38]:

females["OFNS_DESC"].value_counts()

Out[38]:

ASSAULT 3 & RELATED OFFENSES      363
FELONY ASSAULT                    211
PETIT LARCENY                     128
CRIMINAL MISCHIEF & RELATED OF    103
MISCELLANEOUS PENAL LAW            72
ROBBERY                            61
DANGEROUS DRUGS                    42
OFFENSES AGAINST PUBLIC ADMINI     40
BURGLARY                           40
DANGEROUS WEAPONS                  37
GRAND LARCENY                      35
SEX CRIMES                         26
CRIMINAL TRESPASS                  24
OFF. AGNST PUB ORD SENSBLTY &      18
INTOXICATED & IMPAIRED DRIVING     15
GRAND LARCENY OF MOTOR VEHICLE     12
VEHICLE AND TRAFFIC LAWS           11
POSSESSION OF STOLEN PROPERTY       9
OTHER OFFENSES RELATED TO THEF      9
MURDER & NON-NEGL. MANSLAUGHTE      6
OFFENSES AGAINST THE PERSON         6
UNAUTHORIZED USE OF A VEHICLE       5
FOR OTHER AUTHORITIES               3
FORGERY                             3
BURGLAR'S TOOLS                     3
OFFENSES INVOLVING FRAUD            2
OFFENSES RELATED TO CHILDREN        2
FRAUDS                              2
OTHER STATE LAWS (NON PENAL LA      2
ALCOHOLIC BEVERAGE CONTROL LAW      1
INTOXICATED/IMPAIRED DRIVING        1
OTHER STATE LAWS                    1
GAMBLING                            1
NYS LAWS-UNCLASSIFIED FELONY        1
CHILD ABANDONMENT/NON SUPPORT       1
ARSON                               1
THEFT-FRAUD                         1
Name: OFNS_DESC, dtype: int64

In [0]:

#Use conditional formatting to create groups of data based on borough.

In [42]:

#bronx filtering
bronx = nypd_data[nypd_data["ARREST_BORO"] == "B"]

In [43]:

#brooklyn filtering
brooklyn = nypd_data[nypd_data["ARREST_BORO"] == "K"]

In [46]:

#staten island filtering
staten = nypd_data[nypd_data["ARREST_BORO"] == "S"]

In [47]:

#queens filtering
queens = nypd_data[nypd_data["ARREST_BORO"] == "Q"]

In [48]:

#manhattan filtering
manhattan = nypd_data[nypd_data["ARREST_BORO"] == "M"]

In [0]:

#Use value counts to find the ofns desc by borough

In [45]:

#bronx value counts
bronx["OFNS_DESC"].value_counts()

Out[45]:

ASSAULT 3 & RELATED OFFENSES      436
FELONY ASSAULT                    227
DANGEROUS DRUGS                   172
CRIMINAL MISCHIEF & RELATED OF    136
ROBBERY                           131
PETIT LARCENY                     131
OFFENSES AGAINST PUBLIC ADMINI    108
DANGEROUS WEAPONS                 103
BURGLARY                           76
MISCELLANEOUS PENAL LAW            64
GRAND LARCENY                      49
VEHICLE AND TRAFFIC LAWS           39
OFF. AGNST PUB ORD SENSBLTY &      38
SEX CRIMES                         32
POSSESSION OF STOLEN PROPERTY      24
MURDER & NON-NEGL. MANSLAUGHTE     22
CRIMINAL TRESPASS                  18
FOR OTHER AUTHORITIES              16
INTOXICATED & IMPAIRED DRIVING     14
OFFENSES AGAINST THE PERSON        14
RAPE                               13
FORGERY                            12
OFFENSES INVOLVING FRAUD           11
BURGLAR'S TOOLS                    11
OTHER OFFENSES RELATED TO THEF     10
GAMBLING                            9
GRAND LARCENY OF MOTOR VEHICLE      9
UNAUTHORIZED USE OF A VEHICLE       9
FRAUDS                              6
OTHER TRAFFIC INFRACTION            5
NYS LAWS-UNCLASSIFIED FELONY        4
OTHER STATE LAWS (NON PENAL LA      4
ALCOHOLIC BEVERAGE CONTROL LAW      2
OFFENSES AGAINST PUBLIC SAFETY      2
INTOXICATED/IMPAIRED DRIVING        1
HOMICIDE-NEGLIGENT,UNCLASSIFIE      1
ENDAN WELFARE INCOMP                1
JOSTLING                            1
DISORDERLY CONDUCT                  1
KIDNAPPING & RELATED OFFENSES       1
ARSON                               1
Name: OFNS_DESC, dtype: int64

In [49]:

#brooklyn value counts
brooklyn["OFNS_DESC"].value_counts()

Out[49]:

ASSAULT 3 & RELATED OFFENSES      371
FELONY ASSAULT                    231
PETIT LARCENY                     205
MISCELLANEOUS PENAL LAW           203
DANGEROUS WEAPONS                 163
BURGLARY                          144
CRIMINAL MISCHIEF & RELATED OF    138
DANGEROUS DRUGS                   135
ROBBERY                           109
GRAND LARCENY                      88
OFFENSES AGAINST PUBLIC ADMINI     79
SEX CRIMES                         46
POSSESSION OF STOLEN PROPERTY      43
GRAND LARCENY OF MOTOR VEHICLE     38
CRIMINAL TRESPASS                  36
VEHICLE AND TRAFFIC LAWS           33
FORGERY                            33
OFF. AGNST PUB ORD SENSBLTY &      28
BURGLAR'S TOOLS                    27
INTOXICATED & IMPAIRED DRIVING     21
OFFENSES AGAINST THE PERSON        17
RAPE                               13
GAMBLING                           10
OTHER OFFENSES RELATED TO THEF      9
OFFENSES INVOLVING FRAUD            9
MURDER & NON-NEGL. MANSLAUGHTE      9
OTHER STATE LAWS (NON PENAL LA      8
FOR OTHER AUTHORITIES               8
NYS LAWS-UNCLASSIFIED FELONY        6
UNAUTHORIZED USE OF A VEHICLE       6
INTOXICATED/IMPAIRED DRIVING        5
OTHER TRAFFIC INFRACTION            4
FRAUDS                              4
OTHER STATE LAWS                    2
ADMINISTRATIVE CODE                 2
HARRASSMENT 2                       1
ALCOHOLIC BEVERAGE CONTROL LAW      1
DISORDERLY CONDUCT                  1
ARSON                               1
MOVING INFRACTIONS                  1
Name: OFNS_DESC, dtype: int64

In [50]:

#staten value counts
staten["OFNS_DESC"].value_counts()

Out[50]:

ASSAULT 3 & RELATED OFFENSES      55
MISCELLANEOUS PENAL LAW           52
PETIT LARCENY                     49
CRIMINAL MISCHIEF & RELATED OF    32
FELONY ASSAULT                    29
BURGLARY                          24
GRAND LARCENY                     23
DANGEROUS DRUGS                   17
DANGEROUS WEAPONS                 16
OFFENSES AGAINST PUBLIC ADMINI    13
OFF. AGNST PUB ORD SENSBLTY &     10
POSSESSION OF STOLEN PROPERTY     10
VEHICLE AND TRAFFIC LAWS           9
ROBBERY                            7
SEX CRIMES                         6
GRAND LARCENY OF MOTOR VEHICLE     6
CRIMINAL TRESPASS                  5
MURDER & NON-NEGL. MANSLAUGHTE     4
UNAUTHORIZED USE OF A VEHICLE      3
OTHER STATE LAWS (NON PENAL LA     3
INTOXICATED & IMPAIRED DRIVING     3
NYS LAWS-UNCLASSIFIED FELONY       2
OFFENSES AGAINST THE PERSON        2
ARSON                              1
OTHER TRAFFIC INFRACTION           1
OFFENSES INVOLVING FRAUD           1
FRAUDS                             1
OTHER OFFENSES RELATED TO THEF     1
RAPE                               1
Name: OFNS_DESC, dtype: int64

In [51]:

#queens value counts
queens["OFNS_DESC"].value_counts()

Out[51]:

ASSAULT 3 & RELATED OFFENSES      386
MISCELLANEOUS PENAL LAW           213
FELONY ASSAULT                    191
CRIMINAL MISCHIEF & RELATED OF    146
PETIT LARCENY                     133
ROBBERY                           120
BURGLARY                          107
DANGEROUS WEAPONS                  69
SEX CRIMES                         60
GRAND LARCENY                      59
DANGEROUS DRUGS                    53
OFFENSES AGAINST PUBLIC ADMINI     39
POSSESSION OF STOLEN PROPERTY      28
INTOXICATED & IMPAIRED DRIVING     26
GRAND LARCENY OF MOTOR VEHICLE     21
VEHICLE AND TRAFFIC LAWS           19
UNAUTHORIZED USE OF A VEHICLE      18
OTHER STATE LAWS (NON PENAL LA     17
OFF. AGNST PUB ORD SENSBLTY &      17
CRIMINAL TRESPASS                  15
OFFENSES AGAINST THE PERSON        13
MURDER & NON-NEGL. MANSLAUGHTE     11
FRAUDS                             10
BURGLAR'S TOOLS                     8
FOR OTHER AUTHORITIES               7
FORGERY                             6
RAPE                                6
OTHER TRAFFIC INFRACTION            6
OFFENSES INVOLVING FRAUD            5
NYS LAWS-UNCLASSIFIED FELONY        5
OTHER OFFENSES RELATED TO THEF      5
INTOXICATED/IMPAIRED DRIVING        4
OTHER STATE LAWS                    4
THEFT-FRAUD                         3
ARSON                               3
OFFENSES RELATED TO CHILDREN        2
ENDAN WELFARE INCOMP                1
CHILD ABANDONMENT/NON SUPPORT       1
MOVING INFRACTIONS                  1
ADMINISTRATIVE CODE                 1
Name: OFNS_DESC, dtype: int64

In [52]:

#manhattan value counts
manhattan["OFNS_DESC"].value_counts()

Out[52]:

PETIT LARCENY                     268
ASSAULT 3 & RELATED OFFENSES      268
BURGLARY                          169
DANGEROUS DRUGS                   142
FELONY ASSAULT                    126
ROBBERY                           125
CRIMINAL MISCHIEF & RELATED OF    106
GRAND LARCENY                      92
MISCELLANEOUS PENAL LAW            77
DANGEROUS WEAPONS                  76
OFFENSES AGAINST PUBLIC ADMINI     75
CRIMINAL TRESPASS                  49
OFF. AGNST PUB ORD SENSBLTY &      30
VEHICLE AND TRAFFIC LAWS           29
POSSESSION OF STOLEN PROPERTY      28
MURDER & NON-NEGL. MANSLAUGHTE     25
BURGLAR'S TOOLS                    20
SEX CRIMES                         17
FORGERY                            15
FOR OTHER AUTHORITIES              12
UNAUTHORIZED USE OF A VEHICLE       9
GRAND LARCENY OF MOTOR VEHICLE      8
OFFENSES INVOLVING FRAUD            7
OFFENSES AGAINST THE PERSON         7
INTOXICATED & IMPAIRED DRIVING      6
NYS LAWS-UNCLASSIFIED FELONY        5
RAPE                                4
FRAUDS                              4
OTHER TRAFFIC INFRACTION            3
GAMBLING                            3
OTHER OFFENSES RELATED TO THEF      2
KIDNAPPING & RELATED OFFENSES       1
HARRASSMENT 2                       1
OTHER STATE LAWS                    1
Name: OFNS_DESC, dtype: int64

In [53]:

nypd_data["PERP_SEX"]

Out[53]:

     F
     M
     M
     M
     M
       ..
  M
  M
  M
  M
  M
Name: PERP_SEX, Length: 8296, dtype: object

In [54]:

#Often times we have data that might not be suitable for our algorithm. 
#In this case our data is not numerical and so we need to convert it to be numerical.
#Let's create a function that will assign a 1 to F and a 0 to M. 
def perp_gender(row):
    if(row == "F"):
        return 1
    elif(row  == "M"):
        return 0

In [55]:

nypd_data["GENDER_NUM"] = nypd_data["PERP_SEX"].apply(perp_gender)

In [56]:

nypd_data[["PERP_SEX","GENDER_NUM"]]

Out[56]:

In [0]:

# For HW TONIGHT create a column for the different races groups. 
# Using a function and the apply method assign a numerical value to each different race group.

In [0]:

In [0]:

In [0]:

Product

Resources

Company