---
jupytext:
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.16.6
kernelspec:
  display_name: Bash
  language: bash
  name: bash
---

# Git-Annex from scratch

I found I missed an under-the-hood explanation of what Git-Annex added on top
of an ordinary Git repository.

I'm partly starting from [Joey Hess' talk](https://www.youtube.com/watch?v=zQTAeX4prZo).

Here is a set of fundamental objects for Git-Annex, from that talk:

## Git-Annex objects (GAOs)

GAOs are objects in the same sense that Git stores *objects* in its object
store.

For *Git* (not Git-Annex), when we do `git add myfile`, Git takes the contents
of `myfile` (well, in fact, a compressed version of `myfile`) and copies it
into the `.git/objects` directory, with a filename generated from the SHA1 hash
of the object.

However, Git-Annex is designed to work with very large files that should not be
compressed or copied.  So when you create a Git-Annex object (with `git annex
add mybigfile`):

* The file is *moved*, not copied.
* Within the working tree, Git-Annex leaves behind a pointer (usually
  a symbolic link) to the now-moved file.  See below.
* The moved file is not compressed.
* The contents go into `.git/annex/objects` instead of `.git/objects`.
* Git-Annex removes write permission on the moved file, to prevent you
  overwriting the file accidentally (see
  [lock](https://git-annex.branchable.com/git-annex-lock/)/[unlock](https://git-annex.branchable.com/git-annex-unlock/)).
* There are different rules for making the hash filename that we don't need to
  worry about for now.

## Git-Annex pointers

Above, we said that when we make a file into a Git-Annex object, we leave
a *pointer* to the now-moved file, in the working tree.  The pointer is (on a Unix system) a symlink to the now-moved file in the `.git/annex` directory.

(Some systems, such as Windows by default, can't use symbolic links; in this
case Git-Annex can use pointer files).

It is these pointers (usually symbolic links) that we check into the Git
repository.

## Git-Annex remotes

Git-Annex has *remotes*.  Remotes are stores for GAOs.

This can be confusing, because:

* Git-Annex can re-use ordinary Git repositories for this purpose.  In this
  case the remote in the Git-Annex sense can also be a remote in the Git sense.
* But Git-Annex also has the concept of a *special remote*, that is not a Git
  repository, and therefore cannot be a remote in the Git sense. Special
  remotes have various types of storage backing and interfaces, such as
  `rclone` pointing to Google Drive and so on.

I'll call both Git re-used repositories, and special remotes — Git-Annex
Remotes (GARs) — because they have the specific purpose of storing the GAOs.
GARs are pointers to particular data stores that can store GAOs.

Each GAR has a Universally Unique Identifier (UUID), so that Git-Annex can
record which of its remotes has each of the potential GAOs.

As this implies, any one individual GAR need not store all possible GAOs - it
might have a subset.  Git-Annex has to keep track of this information, so it
can `git annex get` any files that the user asks for, that they do not already
have in their `.git/annex` GAO object store.

Here is the confusion — your local Git repository also functions as a GAR — and
has its own UUID.  That is because, on your machine, after you `git annex add`
(or, as you see later `git annex get`) a large file, your own local `.git`
directory will have a copy of the GAO — in `.git/annex/objects`.  So Git-Annex
has to know that you also have a copy of the file, in your `.git/annex/objects`
directory, so you or your collaborators can ask for a copy of the file from
you, if they need to (using `git annex get` etc).

To say it again - a GAR is a different thing from a standard Git remote.  It is
specifically a place that Git-Annex can store or retrieve GAOs. Your local
`.git` repository store is one such place, so it also functions as a GAR, from
which you, or other people connected to your network, can get GAOs.

## Git-Annex metadata

Git-Annex makes and uses its own branch named `git-annex` to store metadata
about the location of GAOs within the known remotes.  We'll come across that
branch in the walkthrough below.

## Walkthrough

Make a new directory for the Git repository, that will soon also be
Git-Annex-enhanced.

```{code-cell}
:tags: [remove-cell]

# Clean up any prior runs before we start.
if [ -d annex-repos ]; then
    # Work round git annex permission guards.
    chmod -R u+rwX annex-repos
    rm -rf annex-repos
fi

# Start in the annex-repos directory (we'll make several
# directories there).
mkdir annex-repos
cd annex-repos
```

```{code-cell}
mkdir my-repo
cd my-repo
```

When we do `git init` we have the usual `.git` subdirectories — this is, so
far, a normal Git repository.

```{code-cell}
git init
ls .git
```

```{code-cell}
tree .git/objects
```

```{code-cell}
echo "Some text" > my_small_file
# Makes my_small_file into a Git object, stored in .git/objects
git add my_small_file
tree .git/objects
```

Make an ordinary Git commit.  This generates a new directory-listing object, and a new commit object — the usual Git behavior.[^git-objects]

[^git-objects]: This is nothing to do with Git-Annex, but if you review the
  hash filenames of two new Git objects,  you'll see that one of them starts
  with the same short (7 character) hash given for the commit — that is the
  commit object. The other is the directory listing.

```{code-cell}
git commit -m "Added first file"
tree .git/objects
```

We only have one branch - the default `main` branch:

```{code-cell}
git branch -a
```

Now we overlay the Git-Annex stuff on the normal Git repository:

```{code-cell}
git annex init
```

Notice that we now have a new `annex` directory in the `.git` directory.

```{code-cell}
ls .git
```

`.git/annex` doesn't have any objects yet, just some housekeeping files.

```{code-cell}
tree .git/annex
```

We also have a new `git-annex` branch:

```{code-cell}
git branch -a
```

Next we look at `git annex info` to show the Git-Annex-Remotes:

```{code-cell}
git annex info
```

For the moment, notice the "semitrusted repositories".  These are GARs - places
that Git-Annex knows can be sources or destinations for the Git-Annex-Object
files.  The first two (`web` and `bittorrent`) are generic sources
corresponding to URLs, and bittorrent files, but we'll ignore these for now.
The third points to the local `.git` directory.  This is a "remote" in a rather
confusing sense - that is - it's a place that can serve as a source for the
files that Git-Annex will store (the GAOs).   It's a remote in the sense that
other Git-Annex overlaid repositories may be able to use to get those files.
You can think of it as a remote to *other repositories*, and a local store to
this one.

+++

Now we'll add some potentially large file, using `git annex add`.  That is going to make a Git-Annex Object (GAO), and leave a pointer (symlink) behind in the working directory.

First we make a file that we will pretend is large.  We'll also calculate its SHA1 sum - you'll see why later.

```{code-cell}
echo "Something really large" > my_large_file
shasum my_large_file
```

Now we make that large file into a Git-Annex Object with `git annex add`:

```{code-cell}
git annex add my_large_file
```

Notice now that:

* The file has moved to `.git/annex/objects`.
* The moved file is now read-only (to prevent you accidentally overwriting it
  — see
  [lock](https://git-annex.branchable.com/git-annex-lock/)/[unlock](https://git-annex.branchable.com/git-annex-unlock/)).
* There's a symlink to that file in the working directory.
* Git-Annex added *the symlink* (and not the file itself) to the ordinary Git
  staging area.

In order:

```{code-cell}
# It's the same file exactly as the one previously in the working directory.
shasum .git/annex/objects/*/*/*/*
```

```{code-cell}
# The moved file is read-only to prevent accidental overwrites.
ls -al .git/annex/objects/*/*/*/*
```

```{code-cell}
# The file in the working tree has become a symlink.
ls -al my_large_file
```

```{code-cell}
# The symlink has been added to the Git staging area.
git status
```

`git annex list` tells us which GAR has each GAO.  `here` is the name for the local GAR, in `.git/annex`:

```{code-cell}
git annex list
```

We can now do a commit - but notice - the thing that gets added to the standard Git history, and stays in `.git/objects`, is nothing but the symlink:

```{code-cell}
git commit -m "Add link to my_large_file"
```

`git show` below identifies the symlink with the file mode 120000:

```{code-cell}
# Show contents of last commit.
git show main
```

To recap - the Git `.git/objects` directory never got a copy of
`my_large_file`.  The content of `my_large_file` belongs to Git-Annex, and is
stored in the `.git/annex/objects` directory, that does not get transferred
when you `git push`, or you `git clone` the repository.  Git itself only has
the symbolic link to the file.  But as we'll see soon, it does know where
copies of `my_large_file` live, through the information in the `git-annex`
branch.

To illustrate, if we do a typical clone of this `my-repo` repository, we do not
have the contents of `my_large_file`; we only have the symlink:

```{code-cell}
cd ..
# Simulate pushing to some upstream service such as Github.
git clone --bare my-repo my-repo-upstream.git
# Simulate cloning from there.
git clone my-repo-upstream.git my-repo-clone
cd my-repo-clone
```

We have put ourselves into the usual situation, where we've done a `git push`
to a remote service, and then done a `git clone` from the remote service, to
another computer.  After doing that:

```{code-cell}
# No .git/annex directory in the clone.
ls .git
```

```{code-cell}
# But we do have access to the remote git-annex branch, which maps
# repositories to GAOs.
git branch -a
```

```{code-cell}
# my_large_file is a broken symlink.
file my_large_file
```

If we try and fetch the file to which the symlink points, with `git annex get`,
Git-Annex can't do it, because it has no record (in the `git-annex` remote
branch) that the `origin` (upstream) repository has that file in its
`.git/annex` filesystem.

```{code-cell}
git annex list
```

However, the `git-annex` remote branch has told it where we might find the file
(see [Git-Annex internals](https://git-annex.branchable.com/internals)).
Notice the suggestion generated during the failure:

```{code-cell}
:tags: [raises-exception]

git annex get my_large_file
```

This message is telling us that we need to point directly to the original
repository as a source for the GAO files (in its `.git/annex` folder).

```{code-cell}
# Add the original repository as an ordinary Git remote.
git remote add original-repo ../my-repo --fetch
```

```{code-cell}
# Git-Annex now knows it can get the file.
git annex list
```

We ask Git-Annex to fetch the file.

```{code-cell}
git annex get my_large_file
```

```{code-cell}
# There are two repositories that Git-Annex knows have
# the file (in `.git/annex`) - this repo, and the original.
git annex list
```

```{code-cell}
# After git annex get, the symlink is fixed, because the file exists.
file my_large_file
```

## The special remotes

As you have seen, Git-Annex can use Git repositories on filesystems as storage
for GAOs — in the repository `.git/annex` directory.

But Git-Annex can also have *special remotes*, that have no necessary relationship to Git repositories.  For example, they can work with directories on file-sharing systems such as Dropbox and Google Drive.

You create special remotes with the `git annex initremote` command. I don't
cover those here, but see the [Git-Annex page on special
remotes](https://git-annex.branchable.com/walkthrough/#index12h2), and the [doc
page on special remotes](https://git-annex.branchable.com/special_remotes).

## Which files go where

Consider setting `annex.largefiles` entries with `git annex config`, or in your
`.gitattributes` file, to tell Git-Annex which files it should handle (as GAOs)
and which Git should handle.  See the [annex.largefiles
page](https://git-annex.branchable.com/tips/largefiles) for details.  Note that
`annex.largefiles` just identifies files that Git-Annex should handle.  You can
use values for `annex.largefiles` to make Git-Annex always operate on files
larger than a particular size, but you can also use that setting to configure
Git / Git-Annex to select files by path name.

Consider using `git annex wanted` commands to tell Git-Annex which remotes
should house which files.   These rules get stored in the `git-annex` branch.
See the [Git-Annex wanted
page](https://git-annex.branchable.com/git-annex-wanted) for more.

## Whither Git-Annex

That was a tour of the basics.  You might now want to have a look at the
primary Git-Annex pages.

* The [main Git-Annex
  walkthrough](https://git-annex.branchable.com/walkthrough)
* [Git-Annex internals](https://git-annex.branchable.com/internals)
* [Scientific computing Git-Annex
  tutorial](https://scicomp.aalto.fi/scicomp/git-annex/)