Exploratory Data Analysis

This notebook is a supplement to the lesson on exploratory data analysis. It will demonstrate some functionalities of the Pandas library that can be used in analysis tasks.

The Pandas library is a Python programming language library used for working with tabular data. It is characterized by a large number of functions that can facilitate and speed up work and provide useful information about the dataset. To be able to use the Pandas library, we first need to load it with the command import pandas as pd.

#%pip install pandas #uncomment if you need
import pandas as pd

We will also load the remaining libraries that we will need for our work.

import numpy as np
from matplotlib import pyplot as plt

Series and DataFrame Structures

The basic data structures used by the Pandas library are Series and DataFrame. We can think of the DataFrame structure as a tabular structure with features associated with rows and columns, while the Series structure can be imagined as a single column of this table, with features associated with rows.

The Series structure is created using the function of the same name. The following structure points_color contains information about the colors of the points.

points_color = pd.Series(['red', 'blue', 'red', 'green'])
points_color

As we can see, each entry is determined by its row index.

The DataFrame structure is also created using the function of the same name. The following code block creates a table tacke with points in the plane that contains the names of the points and their coordinates.

points = pd.DataFrame({
    'point_name': ['A', 'B', 'C', 'D'],
    'x': [20, 11, 3, 27],
    'y': [18, -4, 10, 2]
})
points

Entries in such structures are determined by row indices and column indices. This allows us to access and read their values. One of the functions we can use for this purpose is the loc function, which expects the row index and column index separated by commas.

points.loc[1, 'x']

If it is necessary to read the values of a single column, for example, the one with the names of the points, we can do this by specifying the column name in square brackets as in the following example. If it is necessary to read the values of multiple columns, a list with the column names is specified.

points['point_name']

If it is necessary to read the value of a row, for example, the one with index 2, we can do this using the iloc function.

points.iloc[2]

Loading the Dataset

The Pandas library offers support for working with many formats such as CSV, JSON, HTML, Excel, and others. For each of these formats, there is a corresponding function for loading data: read_csv, read_json, read_html, read_excel. As we can notice, they all start with the prefix read_.

The dataset we will use in this exercise is known as Titanic and contains information about the passengers of the Titanic ship that sank in the Atlantic Ocean in 1912 after hitting an iceberg. This dataset can be downloaded from the Kaggle platform at https://www.kaggle.com/competitions/titanic/. The platform provides two parts of this dataset, titanic_train.csv and titanic_test.csv, the first for training and the second for testing models. You need to download the first dataset and transfer it to the Google Drive platform according to the guidelines in the Google Drive usage instructions. It should be found under the same name in the sample_data directory.

Since the data is in CSV format, we will use the read_csv function to load it.

data_path = '/titanic/train.csv'
data = pd.read_csv(data_path)

Information about the dimensions of the dataset, i.e., the number of rows and columns, can be obtained using the shape attribute. The number of rows corresponds to the number of instances, while the number of columns corresponds to the number of attributes.

data.shape

The first few rows of the loaded dataset can be obtained by calling the head function. Similarly, the last few rows of the loaded dataset can be obtained by calling the tail function. Both functions can take the number of rows to display as an argument - by default, five rows are displayed.

data.head()

Attribute Analysis

A quick overview of the attribute names and types can be obtained using the info function. This function will also print the number of non-missing values for each attribute.

data.info()

The attributes present in the dataset are: passenger identifier (PassengerId), indicator of whether the passenger survived the shipwreck or not (Survived), passenger class (Pclass), passenger name (Name), passenger gender (Sex), age (Age), number of family members on board (SibSp), number of children or parents on board (Parch), ticket code (Ticket), ticket price (Fare), cabin number (Cabin), and the station where the passenger boarded the ship (Embarked). You can read more about the meaning of these attributes on the Kaggle platform page.

Among the numerical attributes, we will single out one attribute and analyze it separately. Let it be the Fare attribute which represents the ticket price. Additional information about this attribute can be obtained by calling the describe function - it will inform us about the minimum and maximum value of the attribute, the average value, and the percentile values.

data['Fare'].describe()

In the output, count indicates the number of values in the column, mean the average value, std the standard deviation, and min and max the minimum and maximum values, respectively. The outputs 25%, 50%, and 75% indicate the percentile values. Thus, the 50th percentile is the value below which half of all values lie. This value is also known as the median. Similarly, the 25th percentile indicates the value below which one-quarter of all values lie, and the 75th percentile the value below which three-quarters of all values lie. These values are also called the first and third quartiles, while the median is also referred to as the second quartile. The purpose of all these calculations is to give us insight into the distribution of attribute values.

An overview of attribute values is also obtained by displaying histograms.

data['Fare'].hist()

From these analyses, it may be interesting to subsequently check entries with a price of 0.

data[data['Fare']==0].shape

data[data['Fare']==0].head()

Similarly, it may be interesting to subsequently check entries with the highest prices.

data[data['Fare']>200].shape

data[data['Fare']>200].head()

The average ticket price can be calculated using the mean function.

 data['Fare'].mean()

We will also single out one categorical attribute. Let it be the gender of the passengers Sex. For attributes of this type, we are usually interested in the set of possible values and their frequency. The value_counts function can help us with this task, as it calculates the number of occurrences for each possible value.

data['Sex'].value_counts()

It is convenient to display statistics related to categorical variables graphically.

 statistics_by_gender = data['Sex'].value_counts()

plt.title('Passenger Statistics by Gender')
plt.bar(statistics_by_gender.index, statistics_by_gender.values, color='orange')
plt.show()

Missing Values

As we can notice from the output of the info function, there are attributes with missing values.

One such attribute is the Embarked attribute, which marks the station where the passenger boarded the ship. Since we only have two missing values, it is most natural to delete the rows in which they appear. The isna function helps us check if a value is missing, while the row deletion itself can be implemented using the drop function.

data[data['Embarked'].isna()]

data.drop(index=[61, 829], inplace=True)

data.shape

As we can notice, we now have two rows less in the dataset. By providing the index argument to the drop function, we specified the indices of the rows to be deleted, while with the inplace argument, we indicated that we want these instances to be deleted "in place". The default behavior when deleting is to generate a modified copy of the data.

In the Cabin column, we see that a lot of values are missing - only 204 values are known. Therefore, it is most convenient to delete this column entirely. We will achieve this using the drop function, but this time specifying the column name instead of the index.

data.drop(columns=['Cabin'], inplace=True)

data.shape

data.head()

The last column with missing values is the Age column. Deleting rows with missing values for this attribute would result in losing about one-fifth of the dataset, and the column itself has more known values than missing ones. Therefore, it is convenient to replace the missing values, for example, with the average age of the passengers. The fillna function, which expects the value to replace the missing values as an argument, can help us with this task. We also performed this replacement in place by using the inplace argument.

average_age = data['Age'].mean()

data['Age'].fillna(average_age, inplace=True)

data['Age'].hist()

Duplicates

The presence of duplicates in the dataset can be checked using the duplicated function. This function will return True for each repeated row and False otherwise.

duplicate = data.duplicated()

duplicate.value_counts()

As we can see, all values are False, so we can conclude that there are no duplicates.

Detecting Outliers

Detecting outliers involves examining individual attribute values and further analyzing those that seem suspicious or domain-irrelevant. Let's further analyze the Age attribute.

By examining the values of the Age attribute using the describe function and graphically plotting with the boxplot function, we notice that there are some unusually small values and some larger values. These need to be further investigated.

data['Age'].describe()

data.boxplot(column=['Age'])

data[data['Age']<1]

It appears that there were children on the ship who were younger than one year old (fortunately, they all survived!). For instances where the passenger identifier is 470 and 645, it seems that all data overlaps except for the name. This might be a potential duplicate or they could be sisters; it is unusual that the ticket number is the same.

data[data['Age']>70]

It seems that among the selected instances where the age is greater than 70, there are no unusual values.

Similarly, the remaining attributes should be examined further in the same manner.

Attribute Correlation

To evaluate the correlation of attributes in the dataset, we can use the corr function. By default, it calculates the Pearson correlation coefficient between each pair of attributes. Before using this function, let's select the attributes whose correlation we want to examine. Let them be Survived, Pclass, Sex, Age, SibSp, Parch, and Fare.

data.head()

data_restriction = data[['Survived', 'Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare']].copy()

Since the attribute Sex is categorical, we will map the value male to 0 and the value female to 1. The map function will help us with this task.

data_restriction['Sex'] = data_restriction['Sex'].map({'male': 0, 'female':1})

The corr function calculates the correlation coefficient values between attributes.

corelation_matrix = data_restriction.corr()

corelation_matrix

As we discussed, for easier analysis of the obtained values, the correlation matrix is also displayed in the form of a heatmap. The function show_corelations enables such a display.

def show_corelations(podaci):

  # calculate the correlation matrix
  corelation_matrix = data.corr()

  # prepare axis labels
  atributes_number = podaci.columns.shape[0]
  plt.xticks(np.arange(0, atributes_number), data.columns, rotation=45)
  plt.yticks(np.arange(0, atributes_number), data.columns)

  # prepare the heatmap
  plt.imshow(corelation_matrix)

  # display the color bar with paired colors and values it represents
  plt.colorbar()

  # display the prepared graph
  plt.show()

show_corelations(data_restriction)

From this matrix, conclusions can be drawn such as that the lower the passenger class, the lower the ticket price (dark purple square at the intersection of the Fare row and Pclass column), that the gender of the passenger positively correlates with the information of whether the passenger survived (it is useful to know that the evacuation of women and children was carried out), and other information.

Completion requirements:

Make a submission