---
jupyter:
  jupytext:
    notebook_metadata_filter: all,-language_info
    split_at_heading: true
    text_representation:
      extension: .Rmd
      format_name: rmarkdown
      format_version: '1.2'
      jupytext_version: 1.6.0
  kernelspec:
    display_name: Python 3
    language: python
    name: python3
---

# The magic of 2 by 2 tables

```{python}
# Our usual imports
import numpy as np
import pandas as pd
# Safe settings for Pandas.
pd.set_option('mode.chained_assignment', 'raise')
```

```{python tags=c("hide-cell")}
# HIDDEN
# An extra tweak to make sure we always get the same random numbers.
# Do not use this in your own code; you nearly always want an unpredictable
# stream of random numbers.  Making them predictable in this way only makes
# sense for a very limited range of things, like tutorials and tests.
np.random.seed(1938)
```

At the end of the [noble politics](noble_politics) page, you came across an
enigma, from analyzing two by two tables of counts.

This page shows you another version of that enigma, and gives a few
explanations for these strange effects.

To start with, we make a simple data frame, where each row corresponds
to a person.  Each person, and therefore, each row, has two labels.  If you
have read the Noble Politics page, you will see what inspired this example.

The first label for each person, called "Party" can be either "Yes", or "No".

The second label for each person, called "Respondent", can be either "R" or
"B".

Here is our data frame from scratch:

```{python tags=c("hide-input")}
# Create a data frame from scratch with some data.
people = pd.DataFrame([["Yes", "R"],
                       ["Yes", "R"],
                       ["Yes","R"],
                       ["No", "R"],
                       ["No", "R"],
                       ["Yes","B"],
                       ["Yes","B"],
                       ["No","B"]],
                       columns = ['Respondent', 'Party'])
# Show the result
people
```

We can ask Pandas to make a 2 by 2 table of counts, for these labels.  Each
entry in the table gives the number of times the combination of labels occurs.
For example, the cell in the "Yes" row, "R" column gives the number of rows in
the original table that have a "Yes" label for "Party" *and* a "R" label for
"Respondent".  Call this the *Yes-with-R* cell.

```{python}
two_by_two = pd.crosstab(people['Party'], people['Respondent'])
two_by_two
```

You might want to confirm these counts.  You should see that there are in fact
3 rows which have both the "Yes" label and the "R" label, and this matches the
bottom right entry in the `pd.crosstab` table.

We can add the row and column totals to this table, with some fancy Pandas
code (unhide the code in this page, or see the notebook version for details):

```{python tags=c("hide-cell")}
# Fancy code for adding row / column totals, giving them colors.
from matplotlib.colors import ListedColormap

PALE_GREY = ListedColormap([0.8, 0.8, 0.8])
MID_GREY = ListedColormap([0.6, 0.6, 0.6])

def with_margins(df):
    """ Data frame displayed with row, column totals added, colored.
    """
    # Add row totals to copy of data frame.
    row_margin = df.sum().astype(int).rename('Row totals')
    df_margins = df.append(row_margin)
    # Add column totals.
    df_o = df_margins.assign(**{'Col totals': df_margins.sum(axis='columns')})
    # Set colors for row, column totals.
    styler = df_o.style.background_gradient(subset='Col totals', cmap=PALE_GREY)
    styler = styler.background_gradient(subset=('Row totals',), cmap=PALE_GREY)
    styler = styler.background_gradient(subset=('Row totals', 'Col totals'),
                                                cmap=MID_GREY)
    return styler
```

```{python}
# Display the table with colors, using fancy plotting function.
with_margins(two_by_two)
```

Now let's say we want to create a random association between the "Party" (Yes/No) column and the "Respondent" (R/B) column.   We can do that by shuffling (permuting) the Yes/No values, and recreating the table, like this:

```{python}
parties = people['Party']
respondents = people['Respondent']
permuted_tab_1 = pd.crosstab(np.random.permutation(parties), respondents)
with_margins(permuted_tab_1)
```

Note that I could have shuffled the "Respondent" labels instead, or both sets
of labels - any of these operations creates what we want - which is a random
association between the "Party" and "Respondent" labels.

Let's make another few of those random tables:

```{python}
permuted_tab_2 = pd.crosstab(np.random.permutation(parties), respondents)
with_margins(permuted_tab_2)
```

```{python}
permuted_tab_3 = pd.crosstab(np.random.permutation(parties), respondents)
with_margins(permuted_tab_3)
```

You will see that, if the top left value goes up, or down, compared to the
original table, then the bottom right value goes up, or down, by the same
amount.  Call the change in the top-right value, from the original `c`.  The
original value for that element was 1, so if the new value is 2, $c = 1$.  If
the new value was 0, then $c = -1$.

You will also see that the bottom left value goes *down* by the same amount as
the top left value goes up.  If the top left value goes up by $c$, then the
bottom left value does down by $c$.

All the values in the table appear to be linked, and move in lock-step.

Notice too that the column and row totals do not --- and in fact cannot ---
change, whatever the association of the R/B and Yes/No labels, for reasons we go into below.

This is another version of the enigma you saw at the end of the [Noble
Politics](noble_politics) page.

## Why are the table values linked?

### Explanation 1: the row and column totals cannot change

The first explanation for the relationship between the values is --- the total
counts over the rows and columns are fixed.

They are fixed because they depend only on the labels, not on the relationship of the labels.   So, we cannot change the counts over rows or columns by shuffling the order of either or both set of labels.

For example, consider the sum of counts over columns.  In our example, the
columns are for "B" and "R", and the column sums of the counts are the total
number of "B" and "R" labels.

By shuffling the order of the labels, we can change the number in the
B-with-Yes cell, but we can't change the total number of "B" labels, so, if
the B-with-Yes number increases, the B-with-No number has to decrease by the
same amount.

The same argument applies for the row totals.

You may now see that, if you give me the value for any one of the four
elements in the table, I can fill in the rest, because they follow from the
fact that I know what the rows and columns have to add up to.

### Explanation 2: what goes up, must come down

Here we think about what happens as we permute the Yes/No labels.  The same
argument applies to permuting the R/B labels, and in fact, to permuting both
the labels.

Consider the original table.

```{python}
with_margins(two_by_two)
```

Now consider permutations in the order of the Yes/No column of the original
table.  We will just permute the first "Yes", for now.

Imagine the first "Yes" gets swapped with the "Yes" in one of the other "R",
"Yes" rows.  We still end up with the same two by two counts table.

Next swap the first "Yes" with the "No" in one of the "R", "No" rows.  The
first row is "R", "No" but the other row has now become "R", "Yes", we still
have the same number of "R", "Yes" and "R", "No", and the counts table doesn't
change.

Now swap the first "Yes" with the "Yes" in one of the "B", "Yes" rows.  We
still have the same number of "R", "Yes" and "B", "Yes" rows, just in a
different row order, and the counts table doesn't change.

Last, we swap the first "Yes" with the "No" in one of the "B", "No" rows. Now,
for the first time, the counts table does change, like this:

```{python tags=c("hide-input")}
# The data frame with first "Yes" permuted to last row.
perm_people = pd.DataFrame([["No", "R"],
                            ["Yes", "R"],
                            ["Yes","R"],
                            ["No", "R"],
                            ["No", "R"],
                            ["Yes","B"],
                            ["Yes","B"],
                            ["Yes","B"]],
                            columns = ['Respondent', 'Party'])
perm_people
```

Here is the cross-tabulation for that case:

```{python tags=c("hide-input")}
perm_crosstab = pd.crosstab(perm_people['Party'], perm_people['Respondent'])
with_margins(perm_crosstab)
```

When we do this single change, we have:

* One less "R", "Yes" row (bottom-right count goes down by 1)
* One less "B", "No" row (top-left count down by 1)
* One more "R", "No" row (bottom-left count up by 1)
* One more "B", "Yes" row (top-right count up by 1)

As we permute the "Yes", "No" labels, we either leave the total number of
unique "R/B", "Yes/No" pairings unchanged, or we change the number of all the
pairings at once.
