Pandas indexing reprise

This page is a reminder on indexing for Pandas data frames and Series.

You have already seen the basics of Pandas indexing; this page is just a reminder of the later parts of the basic indexing page.

import pandas as pd

We use the familiar dataset on student ratings of professors. It is a table where the rows are course subjects and the columns include average ratings for all University professors / lecturers teaching that subject. See the dataset page for more detail.

# Load the dataset as a data frame
ratings = pd.read_csv('rate_my_course.csv')
# Reorder by Easiness
ratings_by_easy = ratings.sort_values('Easiness', ascending=False)
# Make a smaller data frame with the first six rows
top_by_easy = ratings_by_easy.head(6)
# Show the smaller data frame.
top_by_easy

Here is an example Boolean Series that has True for rows where the “Clarity” rating was greater than 4.1, and False otherwise.

is_clear = top_by_easy['Clarity'] > 4.1
is_clear

64     True
36    False
49     True
61    False
31    False
19    False
Name: Clarity, dtype: bool

We will use that in the examples below.

Direct indexing

Direct indexing is where the indexing bracket [ goes right after the data frame. Examples are:

# Direct indexing with a column name.
top_by_easy['Discipline']

64               Reading
36    Physical Education
49                Speech
61     Child Development
31               Theater
19                 Music
Name: Discipline, dtype: object

# Direct indexing with a Boolean sequence.
top_by_easy[is_clear]

As you have seen in the Pandas indexing page, the examples above are the two types of safe direct indexing into Pandas data frames:

1. Direct indexing with a column name.
2. Direct indexing with a Boolean sequence.

Indirect indexing by position with iloc

Indirect indexing is where we use the special .iloc and .loc attributes of data frames and Series. The data frame or series goes first, followed by .iloc or .loc, followed by the opening square bracket [, the specifiers for the values we want, and the closing square bracket ].

.iloc selects rows and columns by position. For example, here we ask for the first three rows:

top_by_easy.iloc[:3]

If we send .iloc two arguments, separated by commas, then the first argument refers to the rows, and the second to the columns. Here we ask for the first three rows and the first three columns:

top_by_easy.iloc[:3, :3]

We can use : to select everything. For example, this selects all rows, and the last column:

clarity_with_iloc = top_by_easy.iloc[:, -1]
clarity_with_iloc

64    3.882635
36    3.832250
49    3.674701
61    3.606082
31    3.584508
19    3.542273
Name: Easiness, dtype: float64

Indirect indexing by label with .loc

We can also select items by their row and column labels. In this case, the row labels are also counting numbers (integers), so they are easily mistaken for positions if you are not careful.

row_labeled_64 = top_by_easy.loc[64]
row_labeled_64

Discipline              Reading
Number of Professors        148
Clarity                 4.15939
Helpfulness             4.18892
Overall Quality          4.1775
Easiness                3.88264
Name: 64, dtype: object

This is a different result than the one we get from iloc, which does look at position rather than label:

row_position_0 = top_by_easy.iloc[0]
row_position_0

Discipline              Reading
Number of Professors        148
Clarity                 4.15939
Helpfulness             4.18892
Overall Quality          4.1775
Easiness                3.88264
Name: 64, dtype: object

We can ask for multiple rows by label:

ratings_by_label = top_by_easy.loc[[64, 49, 31]]
ratings_by_label

If we send .loc two arguments, separated by commas, then the first argument refers to the rows, and the second to the columns. The column labels are the column names. Here we ask for the rows labeled 64, 49, 31, and the column labeled “Discipline”:

ratings_by_row_col_label = top_by_easy.loc[[64, 49, 31], 'Discipline']
ratings_by_row_col_label

64    Reading
49     Speech
31    Theater
Name: Discipline, dtype: object

If we want multiple columns we can pass a list of column names:

ratings_by_row_col_label = top_by_easy.loc[[64, 49, 31], ['Discipline', 'Clarity']]
ratings_by_row_col_label

This is a good way of selecting a subset of the columns from the data frame, using : to select all the rows:

some_columns = top_by_easy.loc[:, ['Discipline', 'Easiness']]
some_columns

You can use Boolean sequences to select rows with .loc.

clear_clarity = top_by_easy.loc[is_clear, 'Clarity']
clear_clarity

64    4.159392
49    4.133191
Name: Clarity, dtype: float64

You can also use some Boolean sequences for .iloc, but it’s a bit more complicated. See Booleans and labels for more detail.