Git-Annex from scratch

Git-Annex from scratch#

I found I missed an under-the-hood explanation of what Git-Annex added on top of an ordinary Git repository.

I’m partly starting from Joey Hess’ talk.

Here is a set of fundamental objects for Git-Annex, from that talk:

Git-Annex objects (GAOs)#

GAOs are objects in the same sense that Git stores objects in its object store.

For Git (not Git-Annex), when we do git add myfile, Git takes the contents of myfile (well, in fact, a compressed version of myfile) and copies it into the .git/objects directory, with a filename generated from the SHA1 hash of the object.

However, Git-Annex is designed to work with very large files that should not be compressed or copied. So when you create a Git-Annex object (with git annex add mybigfile):

  • The file is moved, not copied.

  • Within the working tree, Git-Annex leaves behind a pointer (usually a symbolic link) to the now-moved file. See below.

  • The moved file is not compressed.

  • The contents go into .git/annex/objects instead of .git/objects.

  • Git-Annex removes write permission on the moved file, to prevent you overwriting the file accidentally (see lock/unlock).

  • There are different rules for making the hash filename that we don’t need to worry about for now.

Git-Annex pointers#

Above, we said that when we make a file into a Git-Annex object, we leave a pointer to the now-moved file, in the working tree. The pointer is (on a Unix system) a symlink to the now-moved file in the .git/annex directory.

(Some systems, such as Windows by default, can’t use symbolic links; in this case Git-Annex can use pointer files).

It is these pointers (usually symbolic links) that we check into the Git repository.

Git-Annex remotes#

Git-Annex has remotes. Remotes are stores for GAOs.

This can be confusing, because:

  • Git-Annex can re-use ordinary Git repositories for this purpose. In this case the remote in the Git-Annex sense can also be a remote in the Git sense.

  • But Git-Annex also has the concept of a special remote, that is not a Git repository, and therefore cannot be a remote in the Git sense. Special remotes have various types of storage backing and interfaces, such as rclone pointing to Google Drive and so on.

I’ll call both Git re-used repositories, and special remotes — Git-Annex Remotes (GARs) — because they have the specific purpose of storing the GAOs. GARs are pointers to particular data stores that can store GAOs.

Each GAR has a Universally Unique Identifier (UUID), so that Git-Annex can record which of its remotes has each of the potential GAOs.

As this implies, any one individual GAR need not store all possible GAOs - it might have a subset. Git-Annex has to keep track of this information, so it can git annex get any files that the user asks for, that they do not already have in their .git/annex GAO object store.

Here is the confusion — your local Git repository also functions as a GAR — and has its own UUID. That is because, on your machine, after you git annex add (or, as you see later git annex get) a large file, your own local .git directory will have a copy of the GAO — in .git/annex/objects. So Git-Annex has to know that you also have a copy of the file, in your .git/annex/objects directory, so you or your collaborators can ask for a copy of the file from you, if they need to (using git annex get etc).

To say it again - a GAR is a different thing from a standard Git remote. It is specifically a place that Git-Annex can store or retrieve GAOs. Your local .git repository store is one such place, so it also functions as a GAR, from which you, or other people connected to your network, can get GAOs.

Git-Annex metadata#

Git-Annex makes and uses its own branch named git-annex to store metadata about the location of GAOs within the known remotes. We’ll come across that branch in the walkthrough below.

Walkthrough#

Make a new directory for the Git repository, that will soon also be Git-Annex-enhanced.

mkdir my-repo
cd my-repo

When we do git init we have the usual .git subdirectories — this is, so far, a normal Git repository.

git init
ls .git

Initialized empty Git repository in /tmp/tmpzhf2kc9x/annex-repos/my-repo/.git/
HEAD  config  description  hooks  info	objects  refs

tree .git/objects

.git/objects
├── info
└── pack
3 directories, 0 files

echo "Some text" > my_small_file
# Makes my_small_file into a Git object, stored in .git/objects
git add my_small_file
tree .git/objects

.git/objects
├── 01
│   └── b51e6ac5e39d1eb2fec3143437c6f117a58f03
├── info
└── pack
4 directories, 1 file

Make an ordinary Git commit. This generates a new directory-listing object, and a new commit object — the usual Git behavior.[1]

git commit -m "Added first file"
tree .git/objects

[main (root-commit) d28516c] Added first file
 1 file changed, 1 insertion(+)
 create mode 100644 my_small_file
.git/objects
├── 01
│   └── b51e6ac5e39d1eb2fec3143437c6f117a58f03
├── 7b
│   └── 527389ed9d0c64db733e80fadb582e211b5b7c
├── d2
│   └── 8516ce6e1514bed089804b10be3817e2f92076
├── info
└── pack
6 directories, 3 files

We only have one branch - the default main branch:

git branch -a

* main

Now we overlay the Git-Annex stuff on the normal Git repository:

git annex init

init  ok
(recording state in git...)

Notice that we now have a new annex directory in the .git directory.

ls .git

COMMIT_EDITMSG	annex	description  index  logs     refs
HEAD		config	hooks	     info   objects

.git/annex doesn’t have any objects yet, just some housekeeping files.

tree .git/annex

.git/annex
├── fsck
│   ├── fsck.lck
│   └── fsckdb
│       └── db
├── index
├── index.lck
├── journal
├── journal.lck
├── mergedrefs
├── othertmp
├── othertmp.lck
├── sentinal
└── sentinal.cache
5 directories, 9 files

We also have a new git-annex branch:

git branch -a

  git-annex
* main

Next we look at git annex info to show the Git-Annex-Remotes:

git annex info

trusted repositories: 0
semitrusted repositories: 3
	00000000-0000-0000-0000-000000000001 -- web
 	00000000-0000-0000-0000-000000000002 -- bittorrent
 	ae3098b4-633a-4130-bf1c-bbe3ffa9dd80 -- runner@runnervmgx7h7:/tmp/tmpzhf2kc9x/annex-repos/my-repo [here]
untrusted repositories: 0
transfers in progress: none
available local disk space: 90.81 gigabytes (+100 megabytes reserved)
local annex keys: 0
local annex size: 0 bytes
annexed files in working tree: 0
size of annexed files in working tree: 0 bytes
combined annex size of all repositories: 0 bytes
annex sizes of repositories:
backend usage:
bloom filter size: 32 mebibytes (0% full)

For the moment, notice the “semitrusted repositories”. These are GARs - places that Git-Annex knows can be sources or destinations for the Git-Annex-Object files. The first two (web and bittorrent) are generic sources corresponding to URLs, and bittorrent files, but we’ll ignore these for now. The third points to the local .git directory. This is a “remote” in a rather confusing sense - that is - it’s a place that can serve as a source for the files that Git-Annex will store (the GAOs). It’s a remote in the sense that other Git-Annex overlaid repositories may be able to use to get those files. You can think of it as a remote to other repositories, and a local store to this one.

Now we’ll add some potentially large file, using git annex add. That is going to make a Git-Annex Object (GAO), and leave a pointer (symlink) behind in the working directory.

First we make a file that we will pretend is large. We’ll also calculate its SHA1 sum - you’ll see why later.

echo "Something really large" > my_large_file
shasum my_large_file

eee0481dabe361a026281b3260a37732fa79881d  my_large_file

Now we make that large file into a Git-Annex Object with git annex add:

git annex add my_large_file

add my_large_file
100%  23 B             77 KiB/s 0s
ok
(recording state in git...)

Notice now that:

  • The file has moved to .git/annex/objects.

  • The moved file is now read-only (to prevent you accidentally overwriting it — see lock/unlock).

  • There’s a symlink to that file in the working directory.

  • Git-Annex added the symlink (and not the file itself) to the ordinary Git staging area.

In order:

# It's the same file exactly as the one previously in the working directory.
shasum .git/annex/objects/*/*/*/*

eee0481dabe361a026281b3260a37732fa79881d  .git/annex/objects/4k/p7/SHA256E-s23--5533fe49b4ab945dc8c4b9d878a8c6e74232c8e00fe9a2b2e5f8d88a70406720/SHA256E-s23--5533fe49b4ab945dc8c4b9d878a8c6e74232c8e00fe9a2b2e5f8d88a70406720

# The moved file is read-only to prevent accidental overwrites.
ls -al .git/annex/objects/*/*/*/*

-r--r--r-- 1 runner runner 23 Aug 30 09:36 .git/annex/objects/4k/p7/SHA256E-s23--5533fe49b4ab945dc8c4b9d878a8c6e74232c8e00fe9a2b2e5f8d88a70406720/SHA256E-s23--5533fe49b4ab945dc8c4b9d878a8c6e74232c8e00fe9a2b2e5f8d88a70406720

# The file in the working tree has become a symlink.
ls -al my_large_file

lrwxrwxrwx 1 runner runner 180 Aug 30 09:36 my_large_file -> .git/annex/objects/4k/p7/SHA256E-s23--5533fe49b4ab945dc8c4b9d878a8c6e74232c8e00fe9a2b2e5f8d88a70406720/SHA256E-s23--5533fe49b4ab945dc8c4b9d878a8c6e74232c8e00fe9a2b2e5f8d88a70406720

# The symlink has been added to the Git staging area.
git status

On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	new file:   my_large_file

git annex list tells us which GAR has each GAO. here is the name for the local GAR, in .git/annex:

git annex list

here
|web
||bittorrent
|||
X__ my_large_file

We can now do a commit - but notice - the thing that gets added to the standard Git history, and stays in .git/objects, is nothing but the symlink:

git commit -m "Add link to my_large_file"

[main d302fb2] Add link to my_large_file
 1 file changed, 1 insertion(+)
 create mode 120000 my_large_file

git show below identifies the symlink with the file mode 120000:

# Show contents of last commit.
git show main

commit d302fb2a1aa1285acc4707b523370cd00d890fac (HEAD -> main)
Author: ci <ci@example.com>
Date:   Sun Aug 30 09:36:48 2026 +0000
    Add link to my_large_file
diff --git a/my_large_file b/my_large_file
new file mode 120000
index 0000000..aa09c8c
--- /dev/null
+++ b/my_large_file
@@ -0,0 +1 @@
+.git/annex/objects/4k/p7/SHA256E-s23--5533fe49b4ab945dc8c4b9d878a8c6e74232c8e00fe9a2b2e5f8d88a70406720/SHA256E-s23--5533fe49b4ab945dc8c4b9d878a8c6e74232c8e00fe9a2b2e5f8d88a70406720
\ No newline at end of file

To recap - the Git .git/objects directory never got a copy of my_large_file. The content of my_large_file belongs to Git-Annex, and is stored in the .git/annex/objects directory, that does not get transferred when you git push, or you git clone the repository. Git itself only has the symbolic link to the file. But as we’ll see soon, it does know where copies of my_large_file live, through the information in the git-annex branch.

To illustrate, if we do a typical clone of this my-repo repository, we do not have the contents of my_large_file; we only have the symlink:

cd ..
# Simulate pushing to some upstream service such as Github.
git clone --bare my-repo my-repo-upstream.git
# Simulate cloning from there.
git clone my-repo-upstream.git my-repo-clone
cd my-repo-clone

Cloning into bare repository 'my-repo-upstream.git'...
done.
Cloning into 'my-repo-clone'...
done.

We have put ourselves into the usual situation, where we’ve done a git push to a remote service, and then done a git clone from the remote service, to another computer. After doing that:

# No .git/annex directory in the clone.
ls .git

HEAD  config  description  hooks  index  info  logs  objects  packed-refs  refs

# But we do have access to the remote git-annex branch, which maps
# repositories to GAOs.
git branch -a

* main
  remotes/origin/HEAD -> origin/main
  remotes/origin/git-annex
  remotes/origin/main

# my_large_file is a broken symlink.
file my_large_file

my_large_file: broken symbolic link to .git/annex/objects/4k/p7/SHA256E-s23--5533fe49b4ab945dc8c4b9d878a8c6e74232c8e00fe9a2b2e5f8d88a70406720/SHA256E-s23--5533fe49b4ab945dc8c4b9d878a8c6e74232c8e00fe9a2b2e5f8d88a70406720

If we try and fetch the file to which the symlink points, with git annex get, Git-Annex can’t do it, because it has no record (in the git-annex remote branch) that the origin (upstream) repository has that file in its .git/annex filesystem.

git annex list

here
|origin
||web
|||bittorrent
||||
____ my_large_file

However, the git-annex remote branch has told it where we might find the file (see Git-Annex internals). Notice the suggestion generated during the failure:

git annex get my_large_file

get my_large_file (not available)
  Maybe add some of these git remotes (git remote add ...):
  	ae3098b4-633a-4130-bf1c-bbe3ffa9dd80 -- runner@runnervmgx7h7:/tmp/tmpzhf2kc9x/annex-repos/my-repo
failed
get: 1 failed

This message is telling us that we need to point directly to the original repository as a source for the GAO files (in its .git/annex folder).

# Add the original repository as an ordinary Git remote.
git remote add original-repo ../my-repo --fetch

Updating original-repo
From ../my-repo
 * [new branch]      git-annex  -> original-repo/git-annex
 * [new branch]      main       -> original-repo/main

# Git-Annex now knows it can get the file.
git annex list

here
|origin
||original-repo
|||web
||||bittorrent
|||||
__X__ my_large_file

We ask Git-Annex to fetch the file.

git annex get my_large_file

get my_large_file (from original-repo...)
100%  23 B             10 KiB/s 0s
ok
(recording state in git...)

# There are two repositories that Git-Annex knows have
# the file (in `.git/annex`) - this repo, and the original.
git annex list

here
|origin
||original-repo
|||web
||||bittorrent
|||||
X_X__ my_large_file

# After git annex get, the symlink is fixed, because the file exists.
file my_large_file

my_large_file: symbolic link to .git/annex/objects/4k/p7/SHA256E-s23--5533fe49b4ab945dc8c4b9d878a8c6e74232c8e00fe9a2b2e5f8d88a70406720/SHA256E-s23--5533fe49b4ab945dc8c4b9d878a8c6e74232c8e00fe9a2b2e5f8d88a70406720

The special remotes#

As you have seen, Git-Annex can use Git repositories on filesystems as storage for GAOs — in the repository .git/annex directory.

But Git-Annex can also have special remotes, that have no necessary relationship to Git repositories. For example, they can work with directories on file-sharing systems such as Dropbox and Google Drive.

You create special remotes with the git annex initremote command. I don’t cover those here, but see the Git-Annex page on special remotes, and the doc page on special remotes.

Which files go where#

Consider setting annex.largefiles entries with git annex config, or in your .gitattributes file, to tell Git-Annex which files it should handle (as GAOs) and which Git should handle. See the annex.largefiles page for details. Note that annex.largefiles just identifies files that Git-Annex should handle. You can use values for annex.largefiles to make Git-Annex always operate on files larger than a particular size, but you can also use that setting to configure Git / Git-Annex to select files by path name.

Consider using git annex wanted commands to tell Git-Annex which remotes should house which files. These rules get stored in the git-annex branch. See the Git-Annex wanted page for more.

Whither Git-Annex#

That was a tour of the basics. You might now want to have a look at the primary Git-Annex pages.