---
jupyter:
  jupytext:
    formats: ''
    split_at_heading: true
    text_representation:
      extension: .Rmd
      format_name: rmarkdown
      format_version: '1.1'
      jupytext_version: 1.2.4
  kernelspec:
    display_name: Python 3
    language: python
    name: python3
  jupyterbook:
    show_stderr: true
---

# Introduction to data frames


[Pandas](https://pandas.pydata.org) is a Python package that
implements data frames, and functions that operate on data
frames.

```{python}
# Load the Pandas data science library, call it 'pd'
import pandas as pd
# Turn on a setting to use Pandas more safely.
# We will discuss this setting later.
pd.set_option('mode.chained_assignment', 'raise')
```

We will also use the usual Numpy array library:

```{python}
# Load the Numpy array library, call it 'np'
import numpy as np
```

## Data frames and series

We start by loading data from a Comma Separated Value file (CSV
file).  If you are running on your laptop, you should download
the {download}`gender_stats.csv <../data/gender_stats.csv>`
file to the same directory as this notebook.

See the [gender statistics description page](../data/gender_stats) for more detail on the dataset.

```{python}
# Load the data file
gender_data = pd.read_csv('gender_stats.csv')
```

This is our usual assignment statement. The LHS is `gender_data`, the variable name.  The RHS is an expression, that returns a value.

What type of value does it return?

```{python}
type(gender_data)
```

Pandas integrates with the Notebook, so, if you display a data
frame in the notebook, it does a nice display of rows and
columns.

```{python}
gender_data
```

Notice the `NaN` at the top of the GDP column.  This is a *missing value*.  We
will come to these in [missing values](missing_values).

For the moment, we will do something quick and dirty, which is to drop all the
missing values from the data frame.   *Be careful* - this is rarely the right
thing to do, without a lot of investigation as to why the values are missing.

```{python}
# Drop all missing values.  Be careful, this is rarely the right thing to do.
gender_data_no_na = gender_data.dropna()
gender_data_no_na
```

The data frame has rows and columns.  Like other Python objects, it has
*attributes*.  These are pieces of data associated with the data frame.  You
have already seen *methods*, which are functions associated with the data
frame.  You can access attributes in the same way as you access methods, by
typing the variable name, followed by a dot `.`, followed by the attribute
name.

For example, one attribute of the data frame, is the `shape`:

```{python}
gender_data_no_na.shape
```

Another attribute is `columns`.  This has the column names. For
example, here is a good way of quickly seeing the column names
for a data frame:

```{python}
gender_data_no_na.columns
```

You need more information about what these column names refer to.  Here are the longer descriptions from the original data source (link above):

* `fert_rate`: Fertility rate, total (births per woman).
* `gdp_us_billion`: GDP (in current US \$ billions).
* `health_exp_per_cap`: Health expenditure per capita, PPP (constant 2011
  international \$).
* `health_exp_pub`: Health expenditure, public (% of GDP).
* `prim_ed_girls`: Primary education, pupils (% female).
* `mat_mort_ratio`: Maternal mortality ratio (modeled estimate, per 100,000
  live births).
* `population`: Population, total.

You have seen array slicing (in [Selecting with
arrays](../arrays/array_indexing).  You remember that array slicing uses
square brackets.   Data frames also allow slicing.  For example, we often want
to get all the data for a single column of the data frame.  To do this, we use
the same square bracket notation as we use for array slicing, with the name of
the column inside the square brackets.

```{python}
gdp = gender_data_no_na['gdp_us_billion']
```

What `type` of thing is this column of data?

```{python}
type(gdp)
```

Here are the values for `gdp`.  You will notice that these are the same values
you saw in the "gdp_us_billion" column when you displayed the whole data
frame.

```{python}
gdp
```


## Plotting, the basic way

There are two ways of getting plots from data in data frames.   Here we use
the most basic method, that you have already seen.   Soon, we will get onto [a
more elegant plotting method](df_plotting).

We start with the magic incantation to load the plotting library.

```{python}
# Load the library for plotting, name it 'plt'
import matplotlib.pyplot as plt
# Display plots inside the notebook.
# %matplotlib inline
# Make plots look a little more fancy
plt.style.use('fivethirtyeight')
```

The `gdp` variable is a sequence of values, so we can do a histogram on these
values, as we have done histograms on arrays.

```{python}
plt.hist(gdp);
```

Now we have had a look at the GDP values, we will look at the
values for the `mat_mort_ratio` column.  These are the numbers
of women who die in childbirth for every 100,000 births.

```{python}
mmr = gender_data_no_na['mat_mort_ratio']
mmr
```

```{python}
plt.hist(mmr);
```

We are interested in the relationship of `gpp` and `mmr`.  Maybe richer
countries have better health care, and fewer maternal deaths.

Here is a plot, using the standard Matplotlib `scatter`
function.

```{python}
plt.scatter(gdp, mmr);
```

## Showing the top 5 values with the `head` method


We have already seen that Pandas will display the data frame with nice formatting.  If the data frame is long, it will display only the first few and the last few rows:

```{python}
gender_data_no_na
```

Notice the `...` in the center of this listing, to show that it has not printed some rows.


Sometimes we do not want to see all these rows, but only - say - the top five rows.   The `head` method of the data frame is a useful way to do this:

```{python}
gender_data_no_na.head()
```

The `Series` also has a `head` method, that does the same thing:

```{python}
gdp.head()
```

## Selecting rows


We often want to select rows from the data frame that match some criterion.

Say we want to select the rows corresponding the countries with a high GDP.

Looking at the histogram of `gdp` above, we could try this as a threshold to
identify high GDP countries.

```{python}
high_gdp = gdp > 1000
high_gdp.head()
```

```{python}
type(high_gdp)
```

Notice that `high_gdp` is a Boolean series, like the Boolean arrays you have
already seen.  It has `True` for elements corresponding to countries with
`gdp` value greater than `1000` and `False` otherwise.

We can use this Boolean series to select rows from the data frame.

Remember *indexing*.  When we follow a name of something, like an array or
series or data frame, with an open square bracket, this means we are trying to
get data from the array or Series.  The stuff inside the square brackets says
what we want.

When we put our Boolean series inside the square brackets, it works like this:

```{python}
rich_gender_data = gender_data_no_na[high_gdp]
rich_gender_data
```

```{python}
type(rich_gender_data)
```

`rich_gender_data` is a new data frame, that is a subset of the original
`gender_data_no_na` frame.  It contains only the rows where the GDP value is
greater than `1000` billion dollars.  Check the display of `rich_gender_data`
above to confirm that the values in the `gdp_us_billion` column are all
greater than `1000`.

We can do a scatter plot of GDP values against maternal
mortality rate, and we find, oddly, that for rich countries,
there is little relationship between GDP and maternal mortality.

```{python}
plt.scatter(rich_gender_data['gdp_us_billion'],
            rich_gender_data['mat_mort_ratio'])
```

One thing that stands out is the very high value at around 180.   Which country does this refer to?  We can use sorting to find out.

## Sorting data frames


Data frames have a method `sort_value`.  This returns a new data frame with the
rows sorted by the values in the column we specify.

Here are the first five rows of the data frame of the rich countries:

```{python}
rich_gender_data.head()
```

We are interested to find which of these richer countries has a high maternal mortality ratio.  To do this, we can make a new data frame where the rows are sorted by the values in the
`mat_mort_ratio` column:

```{python}
rich_by_mmr = rich_gender_data.sort_values('mat_mort_ratio')
rich_by_mmr.head()
```

Notice that the rows are in *ascending* order of `mat_mort_ratio`.  To find the countries with high maternal mortality, we might prefer to sort in *descending* order.  As usual you can explore how
you might do this by looking at the help for the `sort_values` method with:

```
rich_by_mmr.sort_values?
```

in a new cell.   If you do that, you discover the `ascending` argument, that
you can use like this:

```{python}
rich_by_descending_mmr = rich_gender_data.sort_values('mat_mort_ratio', ascending=False)
rich_by_descending_mmr.head()
```

As you might have guessed by now, `Series` also have a `sort_values` method.
For a Series, you don't have to specify the column to sort from, because you
are using the Series values.

```{python}
rich_mmr = rich_gender_data['mat_mort_ratio']
type(rich_mmr)
```

```{python}
rich_mmr.sort_values(ascending=False)
```

## Calculation on data frames


We can calculate with Pandas Series, just as we can with arrays.

For example, now we know that India has both a high GDP, and a high maternal mortality ratio, we wonder whether this is because India also has a large population, and therefore, relatively little money per person to spend on health care.

So, we would like know the GDP per capita.  Luckily the data frame as a column "population":

```{python}
rich_population = rich_by_descending_mmr["population"]
rich_population
```

We can divide the GDP by the population in millions to get US billion dollars per million population.

This works exactly as it does for arrays:

```{python}
rich_gdp = rich_by_descending_mmr["gdp_us_billion"]
rich_gdp
```

```{python}
gdp_per_million = rich_gdp / rich_population
gdp_per_million
```

Notice that the result is *elementwise* division.  Python divides each element
in `rich_gdp` by the *corresponding* element in `rich_population`.


Remember that India is the first country in the `rich_by_descending_mmr`
data frame.  It also has by far the lowest GDP per million population of
any of this selection of rich countries.   Here's a plot of
`gdp_per_million` against the corresponding values in `mat_mort_ratio`:

```{python}
plt.scatter(gdp_per_million, rich_by_descending_mmr['mat_mort_ratio'])
```

It does look as if low income per person predisposes to high maternal mortality.
