Introduction to data frames

Pandas is a Python package that implements data frames, and functions that operate on data frames.

# Load the Pands data science library, call it 'pd'
import pandas as pd

We will also use the usual Numpy array library:

# Load the Numpy array library, call it 'np'
import numpy as np

Data frames and series

We start by loading data from a Comma Separated Value file (CSV file). If you are running on your laptop, you should download the gender_stats.csv file to the same directory as this notebook.

See the gender statistics description page for more detail on the dataset.

# Load the data file
gender_data = pd.read_csv('gender_stats.csv')

This is our usual assignment statement. The LHS is gender_data, the variable name. The RHS is an expression, that returns a value.

What type of value does it return?

type(gender_data)

pandas.core.frame.DataFrame

Pandas integrates with the Notebook, so, if you display a data frame in the notebook, it does a nice display of rows and columns.

gender_data

The data frame has rows and columns. Like other Python objects, it has attributes. These are pieces of data associated with the data frame. You have already seen methods, which are functions associated with the data frame. You can access attributes in the same way as you access methods, by typing the variable name, followed by a dot ., followed by the attribute name.

For example, one attribute of the data frame, is the shape:

gender_data.shape

(216, 9)

Another attribute is columns. This has the column names. For example, here is a good way of quickly seeing the column names for a data frame:

gender_data.columns

Index(['country_name', 'country_code', 'fert_rate', 'gdp_us_billion',
       'health_exp_per_cap', 'health_exp_pub', 'prim_ed_girls',
       'mat_mort_ratio', 'population'],
      dtype='object')

You need more information about what these column names refer to. Here are the longer descriptions from the original data source (link above):

• fert_rate: Fertility rate, total (births per woman).
• gdp_us_billion: GDP (in current US \$ billions).
• health_exp_per_cap: Health expenditure per capita, PPP (constant 2011 international \$).
• health_exp_pub: Health expenditure, public (% of GDP).
• prim_ed_girls: Primary education, pupils (% female).
• mat_mort_ratio: Maternal mortality ratio (modeled estimate, per 100,000 live births).
• population: Population, total.

You have just seen array slicing (in Selecting with arrays. You remember that array slicing uses square brackets. Data frames also allow slicing. For example, we often want to get all the data for a single column of the data frame. To do this, we use the same square bracket notation as we use for array slicing, with the name of the column inside the square brackets.

gdp = gender_data['gdp_us_billion']

What type of thing is this column of data?

type(gdp)

pandas.core.series.Series

Here are the values for gdp. You will notice that these are the same values you saw in the “gdp_us_billion” column when you displayed the whole data frame.

gdp

0               NaN
1         19.961015
2        111.936542
3         12.327586
4          3.197538
5        375.027082
6        550.980968
7         10.885362
8          0.640500
9          1.298213
10      1422.994116
11       407.494276
12        62.003001
13         2.876978
14       494.221836
15         8.778151
16        11.753054
17       174.545099
18        53.797612
19        32.004011
20         8.688000
21        17.323327
22        64.782942
23         1.680325
24         5.555624
25        31.509324
26      2198.765606
27         4.413080
28        15.719223
29         1.975145
           ...     
186             NaN
187       11.945942
188        4.183610
189      406.136904
190        8.036228
191       37.973096
192        1.361430
193        0.439179
194       24.570947
195       44.824374
196      895.175577
197        0.036470
198       44.935542
199       25.941461
200      135.379275
201       54.345132
202    17369.124600
203       61.340649
204        0.730107
205      376.146268
206             NaN
207        3.812000
208      181.820736
209        0.782876
210        0.799887
211        6.804620
212       36.819337
213      345.209888
214       24.280990
215       15.495514
Name: gdp_us_billion, Length: 216, dtype: float64

Missing values and NaN

Looking at the values of gdp (and therefore, the values of the gdp_us_billion column of gender_data, we see that some of the values are NaN, which means Not a Number. Pandas uses this marker to indicate values that are not available, or missing data.

Numpy does not like to calculate with NaN values. Here is Numpy trying to calculate the median of the gdp values.

np.median(gdp)

/Users/mb312/Library/Python/3.7/lib/python/site-packages/numpy/lib/function_base.py:3250: RuntimeWarning: Invalid value encountered in median
  r = func(a, **kwargs)

nan

Notice the warning about an invalid value.

Numpy recognizes that one or more values are NaN and refuses to guess what to do, when calculating the median.

You saw from the shape above that gender_data has 263 rows. We can use the general Python len function, to see how many elements there are in gdp.

len(gdp)

216

As expected, it has the same number of elements as there are rows in gender_data.

The count method of the series gives the number of values that are not missing - that is - not NaN.

gdp.count()

200

Plotting with methods

We start with the magic incantation to load the plotting library.

# Load the library for plotting, name it 'plt'
import matplotlib.pyplot as plt
# Display plots inside the notebook.
%matplotlib inline
# Make plots look a little more fancy
plt.style.use('fivethirtyeight')

The gdp variable is a sequence of values, so we can do a histogram on these values, as we have done histograms on arrays.

plt.hist(gdp);

/Users/mb312/Library/Python/3.7/lib/python/site-packages/numpy/lib/histograms.py:754: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= first_edge)
/Users/mb312/Library/Python/3.7/lib/python/site-packages/numpy/lib/histograms.py:755: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= last_edge)

Notice the multiple warnings as Matplotlib tried to calculate the bin widths for the histogram. These warnings result from Matplotlib struggline with NaN values.

Another way to do the histogram, is to use the hist method of the series.

A method is a function attached to a value. In this case hist is a function attached to a value of type Series.

Using the hist method instead of the plt.hist function can make the code a bit easier to read. The method also has the advantage that it discards the NaN values, by default, so it does not generate the same warnings.

gdp.hist();

Now we have had a look at the GDP values, we will look at the values for the mat_mort_ratio column. These are the numbers of women who die in childbirth for every 100,000 births.

mmr = gender_data['mat_mort_ratio']
mmr

0         NaN
1      444.00
2      501.25
3       29.25
4         NaN
5        6.00
6       53.75
7       27.25
8         NaN
9         NaN
10       6.00
11       4.00
12      25.25
13     747.25
14       7.00
15     417.50
16     384.25
17     194.75
18      10.75
19      15.25
20      81.50
21      11.75
22       4.00
23      29.25
24        NaN
25     218.25
26      49.50
27      28.00
28      23.75
29     161.75
        ...  
186       NaN
187    892.25
188    380.75
189     21.00
190     33.25
191     43.50
192    240.25
193    129.25
194     63.25
195     63.25
196     17.50
197       NaN
198    429.50
199    366.50
200     24.25
201     15.50
202     14.00
203     37.00
204     45.75
205     97.00
206       NaN
207       NaN
208     54.75
209     82.50
210     54.75
211       NaN
212    399.75
213    143.75
214    233.75
215    398.00
Name: mat_mort_ratio, Length: 216, dtype: float64

mmr.hist();

We are interested in the relationship of gpp and mmr. Maybe richer countries have better health care, and fewer maternal deaths.

Here is a plot, using the standard Matplotlib scatter function.

plt.scatter(gdp, mmr);

We can do the same plot using the plot.scatter method on the data frame. In that case, we specify the column names that should go on the x and the y axes.

gender_data.plot.scatter('gdp_us_billion', 'mat_mort_ratio');

An advantage of doing it this way is that we get the column names on the x and y axes by default.

Showing the top 5 values with the head method

We have already seen that Pandas will display the data frame with nice formatting. If the data frame is long, it will display only the first few and the last few rows:

gender_data

Notice the ... in the center of this listing, to show that it has not printed some rows.

Sometimes we do not want to see all these rows, but only - say - the top five rows. The head method of the data frame is a useful way to do this:

gender_data.head()

The Series also has a head method, that does the same thing:

gdp.head()

0           NaN
1     19.961015
2    111.936542
3     12.327586
4      3.197538
Name: gdp_us_billion, dtype: float64

Selecting rows

We often want to select rows from the data frame that match some criterion.

Say we want to select the rows corresponding the countries with a high GDP.

Looking at the histogram of gdp above, we could try this as a threshold to identify high GDP countries.

high_gdp = gdp > 1000
high_gdp.head()

0    False
1    False
2    False
3    False
4    False
Name: gdp_us_billion, dtype: bool

type(high_gdp)

pandas.core.series.Series

Notice that high_gdp is a Boolean series, like the Boolean arrays you have already seen. It has True for elements corresponding to countries with gdp value greater than 1000 and False otherwise.

We can use this Boolean series to select rows from the data frame.

Remember indexing. When we follow a name of something, like an array or series or data frame, with an open square bracket, this means we are trying to get data from the array or Series. The stuff inside the square brackets says what we want.

When we put our Boolean series inside the square brackets, it works like this:

rich_gender_data = gender_data[high_gdp]
rich_gender_data

type(rich_gender_data)

pandas.core.frame.DataFrame

rich_gender_data is a new data frame, that is a subset of the original gender_data frame. It contains only the rows where the GDP value is greater than 1000 billion dollars. Check the display of rich_gender_data above to confirm that the values in the gdp_us_billion column are all greater than 1000.

We can do a scatter plot of GDP values against maternal mortality rate, and we find, oddly, that for rich countries, there is little relationship between GDP and maternal mortality.

rich_gender_data.plot.scatter('gdp_us_billion', 'mat_mort_ratio')

<matplotlib.axes._subplots.AxesSubplot at 0x1150beb90>

One thing that stands out is the very high value at around 180. Which country does this refer to? We can use sorting to find out.

Sorting data frames

Data frames have a method sort_value. This returns a new data frame with the rows sorted by the values in the column we specify.

Here are the first five rows of the data frame of the rich countries:

rich_gender_data.head()

We are interested to find which of these richer countries has a high maternal mortality ratio. To do this, we can make a new data frame where the rows are sorted by the values in the mat_mort_ratio column:

rich_by_mmr = rich_gender_data.sort_values('mat_mort_ratio')
rich_by_mmr.head()

Notice that the rows are in ascending order of mat_mort_ratio. To find the countries with high maternal mortality, we might prefer to sort in descending order. As usual you can explore how you might do this by looking at the help for the sort_values method with:

rich_by_mmr.sort_values?

in a new cell. If you do that, you discover the ascending argument, that you can use like this:

rich_by_descending_mmr = rich_gender_data.sort_values('mat_mort_ratio', ascending=False)
rich_by_descending_mmr.head()

As you might have guessed by now, Series also have a sort_values method. For a Series, you don’t have to specify the column to sort from, because you are using the Series values.

rich_mmr = rich_gender_data['mat_mort_ratio']
type(rich_mmr)

pandas.core.series.Series

rich_mmr.sort_values(ascending=False)

88     185.25
26      49.50
124     40.00
35      28.75
164     25.25
202     14.00
104     12.00
67       9.25
63       8.75
32       7.25
49       6.25
10       6.00
97       5.75
58       5.00
94       4.00
Name: mat_mort_ratio, dtype: float64

Calculation on data frames

We can calculate with Pandas Series, just as we can with arrays.

For example, now we know that India has both a high GDP, and a high maternal mortality ratio, we wonder whether this is because India also has a large population, and therefore, relatively little money per person to spend on health care.

So, we would like know the GDP per capita. Luckily the data frame as a column “population”:

rich_population = rich_by_descending_mmr["population"]
rich_population

88     1293.742537
26      204.159544
124     124.203450
35     1364.446000
164     143.793504
202     318.558175
104      50.727212
67       64.641557
63       66.302099
32       35.517119
49       81.281645
10       23.444560
97      127.297102
58       46.553128
94       60.378795
Name: population, dtype: float64

We can divide the GDP by the population in millions to get US billion dollars per million population.

This works exactly as it does for arrays:

rich_gdp = rich_by_descending_mmr["gdp_us_billion"]
rich_gdp

88      2019.005411
26      2198.765606
124     1188.802780
35     10182.790479
164     1822.691700
202    17369.124600
104     1346.751162
67      2768.864417
63      2647.649725
32      1708.473627
49      3601.226158
10      1422.994116
97      5106.024760
58      1299.724261
94      2005.983980
Name: gdp_us_billion, dtype: float64

gdp_per_million = rich_gdp / rich_population
gdp_per_million

88      1.560593
26     10.769840
124     9.571415
35      7.462949
164    12.675758
202    54.524184
104    26.548890
67     42.834123
63     39.933121
32     48.102821
49     44.305528
10     60.696133
97     40.111084
58     27.919161
94     33.223319
dtype: float64

Notice that the result is elementwise division, that is Python divides each element in rich_gdp by the corresponding element in rich_population.

Remember that India is the first country in the rich_by_descending_mmr data frame. It also has by far the lowest GDP per million population of any of this selection of rich countries. Here’s a plot of gdp_per_million against the corresponding values in mat_mort_ratio:

plt.scatter(gdp_per_million, rich_by_descending_mmr['mat_mort_ratio'])

<matplotlib.collections.PathCollection at 0x114251dd0>

It does look as if low income per person predisposes to high maternal mortality.