Introduction to data frames

Start by loading the usual plotting libraries.

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
# Make plots look a little bit more fancy
plt.style.use('fivethirtyeight')

Pandas is a Python package that implements data frames, and functions that operate on data frames.

import pandas as pd

Data frames and series

We start by loading data from a Comma Separated Value file (CSV file). If you are running on your laptop, you should download the gender_stats.csv file to the same directory as this notebook.

# Load the data file
gender_data = pd.read_csv('gender_stats.csv')

This is our usual assignment statement. The LHS is gender_data, the variable name. The RHS is an expression, that returns a value.

What type of value does it return?

type(gender_data)

pandas.core.frame.DataFrame

Pandas integrates with the Notebook, so, if you display a data frame in the notebook, it does a nice display of rows and columns.

gender_data

The data frame has rows and columns. Like other Python objects, it has attributes. These are pieces of data associated with the data frame. You have already seen methods, which are functions associated with the data frame. You can access attributes in the same way as you access methods, by typing the variable name, followed by a dot ., followed by the attribute name.

For example, one attribute of the data frame, is the shape:

gender_data.shape

(263, 8)

Another attribute is columns. This has the column names. For example, here is a good way of quickly seeing the column names for a data frame:

gender_data.columns

Index(['country', 'fert_rate', 'gdp', 'health_exp_per_cap', 'health_exp_pub',
       'prim_ed_girls', 'mat_mort_ratio', 'population'],
      dtype='object')

You need more information about what these column names refer to. Here are the longer descriptions from the original data source (link above):

• fert_rate: Fertility rate, total (births per woman).
• gdp: GDP (current US$).
• health_exp_per_cap: Health expenditure per capita, PPP (constant 2011 international \$).
• health_exp_pub: Health expenditure, public (% of GDP).
• prim_ed_girls: Primary education, pupils (% female).
• mat_mort_ratio: Maternal mortality ratio (modeled estimate, per 100,000 live births).
• population: Population, total.

You have just seen array slicing (in Selecting with arrays. You remember that array slicing uses square brackets. Data frames also allow slicing. For example, we often want to get all the data for a single column of the data frame. To do this, we use the same square bracket notation as we use for array slicing, with the name of the column inside the square brackets.

gdp = gender_data['gdp']

What type of thing is this column of data?

type(gdp)

pandas.core.series.Series

Here are the values for gdp. You will notice that these are the same values you saw in the “gdp” column when you displayed the whole data frame.

gdp

0      1.996102e+10
1      1.232759e+10
2      1.907346e+11
3      6.405000e+08
4      3.197538e+09
5      1.119365e+11
6      1.298213e+09
7      2.709059e+12
8      5.509810e+11
9      1.088536e+10
10              NaN
11     1.422994e+12
12     4.074943e+11
13     6.200300e+10
14     8.688000e+09
15     3.200401e+10
16     1.745451e+11
17     4.413080e+09
18     6.478294e+10
19     4.942218e+11
20     1.680325e+09
21     8.778151e+09
22     5.555624e+09
23     1.975145e+09
24     3.150932e+10
25     1.732333e+10
26     1.511339e+10
27     2.198766e+12
28              NaN
29     1.571922e+10
           ...     
233             NaN
234    8.036228e+09
235    4.493554e+10
236    4.061369e+11
237    1.361430e+09
238    4.183610e+09
239    4.391789e+08
240    2.457095e+10
241    4.482437e+10
242    8.951756e+11
243    3.797310e+10
244             NaN
245    3.646999e+07
246    2.594146e+10
247    1.353793e+11
248    3.750271e+11
249    2.768864e+12
250    1.736912e+13
251    2.097441e+13
252    5.434513e+10
253    6.134065e+10
254    7.828760e+08
255    3.761463e+11
256    1.818207e+11
257    3.812000e+09
258    1.250822e+10
259    7.613006e+13
260    3.681934e+10
261    2.428099e+10
262    1.549551e+10
Name: gdp, Length: 263, dtype: float64

What are these values like 6.405000e+08? These are numbers, in scientific notation. Scientific notation is a compact way of writing very large or very small numbers. The value after e above is the exponent, in this case 08. The number above means $6.405

• 10^8$. For example, here is $2 * 10^7$:

2e7

20000000.0

Missing values and NaN

Looking at the values of gdp (and therefore, the values of the gdp column of gender_data, we see that some of the values are NaN, which means Not a Number. Pandas uses this marker to indicate values that are not available, or missing data.

Numpy does not like to calculate with NaN values. Here is Numpy trying to calculate the median of the gdp values.

np.median(gdp)

nan

Notice the warning about an invalid value.

Numpy recognizes that one or more values are NaN and refuses to guess what to do, when calculating the median.

You saw from the shape above that gender_data has 263 rows. We can use the general Python len function, to see how many elements there are in gdp.

len(gdp)

263

As expected, it has the same number of elements as there are rows in gender_data.

The count method of the series gives the number of values that are not missing - that is - not NaN.

gdp.count()

246

Plotting with methods

The gdp variable is a sequence of values, so we can do a histogram on these values, as we have done histograms on arrays.

plt.hist(gdp);

Notice the multiple warnings as Matplotlib tried to calculate the bin widths for the histogram. These are from the NaN values.

Another way to do the histogram, is to use the hist method of the series.

A method is a function attached to a value. In this case hist is a function attached to a value of type Series.

Using the hist method instead of the plt.hist function can make the code a bit easier to read. The method also has the advantage that it discards the NaN values, by default, so it does not generate the same warnings.

gdp.hist();

Now we have had a look at the GDP values, we will look at the values for the mat_mort_ratio column. These are the numbers of women who die in childbirth for every 100,000 births.

mmr = gender_data['mat_mort_ratio']
mmr

0      444.00
1       29.25
2      142.50
3         NaN
4         NaN
5      501.25
6         NaN
7      161.00
8       53.75
9       27.25
10        NaN
11       6.00
12       4.00
13      25.25
14      81.50
15      15.25
16     194.75
17      28.00
18       4.00
19       7.00
20      29.25
21     417.50
22        NaN
23     161.75
24     218.25
25      11.75
26     138.75
27      49.50
28        NaN
29      23.75
        ...  
233     62.00
234     33.25
235    429.50
236     21.00
237    240.25
238    380.75
239    129.25
240     63.25
241     63.25
242     17.50
243     43.50
244       NaN
245       NaN
246    366.50
247     24.25
248      6.00
249      9.25
250     14.00
251     43.25
252     15.50
253     37.00
254     82.50
255     97.00
256     54.75
257       NaN
258     47.50
259    223.75
260    399.75
261    233.75
262    398.00
Name: mat_mort_ratio, Length: 263, dtype: float64

mmr.hist();

We are interested in the relationship of gpp and mmr. Maybe richer countries have better health care, and fewer maternal deaths.

Here is a plot, using the standard Matplotlib scatter function.

plt.scatter(gdp, mmr);

We can do the same plot using the plot.scatter method on the data frame. In that case, we specify the column names that should go on the x and the y axes.

gender_data.plot.scatter('gdp', 'mat_mort_ratio');

An advantage of doing it this way is that we get the column names on the x and y axes by default.

Showing the top 5 values with the head method

We have already seen that Pandas will display the data frame with nice formatting. If the data frame is long, it will display only the first few and the last few rows:

gender_data

Notice the ... in the center of this listing, to show that it has not printed some rows.

Sometimes we do not want to see all these rows, but only - say - the top five rows. The head method of the data frame is a useful way to do this:

gender_data.head()

The Series also has a head method, that does the same thing:

gdp.head()

0    1.996102e+10
1    1.232759e+10
2    1.907346e+11
3    6.405000e+08
4    3.197538e+09
Name: gdp, dtype: float64

Selecting rows

We often want to select rows from the data frame that match some criterion.

Say we want to select the rows corresponding the countries with a high GDP.

Looking at the histogram of gdp above, we could try this as a threshold to identify high GDP countries.

high_gdp = gdp > 1e13
high_gdp

0      False
1      False
2      False
3      False
4      False
5      False
6      False
7      False
8      False
9      False
10     False
11     False
12     False
13     False
14     False
15     False
16     False
17     False
18     False
19     False
20     False
21     False
22     False
23     False
24     False
25     False
26     False
27     False
28     False
29     False
       ...  
233    False
234    False
235    False
236    False
237    False
238    False
239    False
240    False
241    False
242    False
243    False
244    False
245    False
246    False
247    False
248    False
249    False
250     True
251     True
252    False
253    False
254    False
255    False
256    False
257    False
258    False
259     True
260    False
261    False
262    False
Name: gdp, Length: 263, dtype: bool

type(high_gdp)

pandas.core.series.Series

Notice that high_gdp is a Boolean series, like the Boolean arrays you have already seen. It has True for elements corresponding to countries with gdp value greater than 1e13 and False otherwise.

We can use this Boolean series to select rows from the data frame. The loc attribute of the data frame allows us to LOCate values in the data frame. For our Boolean series, it works like this:

rich_gender_data = gender_data.loc[high_gdp]
rich_gender_data

type(rich_gender_data)

pandas.core.frame.DataFrame

rich_gender_data is a new data frame, that is a subset of the original gender_data frame. It contains only the rows where the GDP value is greater than 1e13 dollars. Check the display of rich_gender_data above to confirm that the values in the gdp column are all greater than 1e13.

We can do a scatter plot of GDP values against maternal mortality rate, and we find, oddly, that for rich countries, there is little relationship between GDP and maternal mortality.

rich_gender_data.plot.scatter('gdp', 'mat_mort_ratio')

<matplotlib.axes._subplots.AxesSubplot at 0x1178bb7b8>

Sorting data frames

Data frames have a method sort_value. This returns a new data frame with the rows sorted by the values in the column we specify.

Here are the first five rows of the original data frame:

gender_data.head()

We can make a new data frame where the rows are sorted by the values in the gdp column:

gender_data_by_gdp = gender_data.sort_values('gdp')
gender_data_by_gdp.head()

Notice that the rows are in ascending order of gdp. You can imagine, that we often want descending order. As usual you can explore how you might do this by looking at the help for the sort_values method with:

gender_data.sort_values?

in a new cell. If you do that, you discover the ascending argument, that you can use like this:

gender_data.sort_values('gdp', ascending=False)

As you might have guessed by now, Series also have a sort_values method. For a Series, you don’t have to specify the column to sort from, because you are using the Series values.

gdp.sort_values(ascending=False)

259    7.613006e+13
98     4.884635e+13
180    4.787743e+13
193    4.541806e+13
103    2.803020e+13
140    2.724634e+13
160    2.691014e+13
102    2.607726e+13
72     2.191519e+13
61     2.168128e+13
251    2.097441e+13
177    1.908336e+13
128    1.862014e+13
250    1.736912e+13
75     1.731910e+13
71     1.255692e+13
63     1.242383e+13
62     1.239991e+13
60     1.019283e+13
44     1.018279e+13
142    5.929275e+12
129    5.845138e+12
130    5.636800e+12
131    5.377437e+12
117    5.106025e+12
73     4.213635e+12
74     3.710324e+12
85     3.601226e+12
157    3.380647e+12
249    2.768864e+12
           ...     
254    7.828760e+08
224    7.301068e+08
3      6.405000e+08
46     6.039190e+08
58     5.130350e+08
239    4.391789e+08
156    3.193208e+08
202    3.145400e+08
185    2.548400e+08
152    1.843189e+08
121    1.774306e+08
169    1.063908e+08
245    3.646999e+07
10              NaN
28              NaN
38              NaN
42              NaN
53              NaN
68              NaN
81              NaN
87              NaN
122             NaN
137             NaN
162             NaN
172             NaN
201             NaN
209             NaN
223             NaN
233             NaN
244             NaN
Name: gdp, Length: 263, dtype: float64