How to install lfs git?

• Download the windows installer from here.
• Run the windows installer.
• start a command prompt /or git bash for windows prompt and run git lfs install.

DM uses Git LFS to manage test datasets within our normal Git workflow. Git LFS is developed by GitHub, though DM uses its own backend storage infrastructure (see SQR-001: The Git LFS Architecture for background).

All DM repositories should use Git LFS to store binary data, such as FITS files, for CI. Examples of LFS-backed repositories are lsst/afw, lsst/hsc_ci, lsst/testdata_decam and lsst/testdata_cfht.

This page describes how to use Git LFS for DM development.

Open the terminal

Click the link and download the archive.

Go to the directory with archive and unpack it

To install, type

and you will get message that the extension initialized

In your repository direcotry, run

You’ll need to run this in your repository directory, once per repository.

Git LFS can be used when you want to version large files, usually, valuable output data, which is larger than Github limit (100Mb). These files can be plain text or binaries.

ARFC Git Large File Storage is quite large, but be respectful to your colleagues and make sure you store only valuable data of reasonable (100Mb-5Gb) size. Consider using alternative storage options such as Box if version control is not necessary or if files of a larger size need to be stored.

Git is a distributed version control system, meaning the entire history of the repository is transferred to the client during the cloning process. For projects containing large files, particularly large files that are modified regularly, this initial clone can take a huge amount of time, as every version of every file has to be downloaded by the client. Git LFS (Large File Storage) is a Git extension developed by Atlassian, GitHub, and a few other open source contributors, that reduces the impact of large files in your repository by downloading the relevant versions of them lazily. Specifically, large files are downloaded during the checkout process rather than during cloning or fetching.

Git LFS does this by replacing large files in your repository with tiny pointer files. During normal usage, you'll never see these pointer files as they are handled automatically by Git LFS:

Git LFS is seamless: in your working copy you'll only see your actual file content. This means you can use Git LFS without changing your existing Git workflow; you simply git checkout, edit, git add, and git commit as normal. git clone and git pull operations will be significantly faster as you only download the versions of large files referenced by commits that you actually check out, rather than every version of the file that ever existed.

To use Git LFS, you will need a Git LFS aware host such as Bitbucket Cloud or Bitbucket Data Center. Repository users will need to have the Git LFS command-line client installed, or a Git LFS aware GUI client such as Sourcetree. Fun fact: Steve Streeting, the Atlassian developer who invented Sourcetree, is also a major contributor to the Git LFS project, so Sourcetree and Git LFS work together rather well.

What is Git LFS?

To create a new Git LFS aware repository, you'll need to run git lfs install after you create the repository:

This installs a special pre-push Git hook in your repository that will transfer Git LFS files to the server when you git push.

Git LFS is automatically enabled for all Bitbucket Cloud repositories. For Bitbucket Data Center, you'll need to enable Git LFS in your repository settings:

Once Git LFS is initialized for your repository, you can specify which files to track using git lfs track.

Once Git LFS is installed, you can clone a Git LFS repository as normal using git clone. At the end of the cloning process Git will check out the default branch (usually main), and any Git LFS files needed to complete the checkout process will be automatically downloaded for you. For example:

There are four PNGs in this repository being tracked by Git LFS. When running git clone, Git LFS files are downloaded one at a time as pointer files are checked out of your repository.

If you're cloning a repository with a large number of LFS files, the explicit git lfs clone command offers far better performance:

Rather than downloading Git LFS files one at a time, the git lfs clone command waits until the checkout is complete, and then downloads any required Git LFS files as a batch. This takes advantage of parallelized downloads, and dramatically reduces the number of HTTP requests and processes spawned (which is especially important for improving performance on Windows).

Just like cloning, you can pull from a Git LFS repository using a normal git pull. Any needed Git LFS files will be downloaded as part of the automatic checkout process once the pull completes:

No explicit commands are needed to retrieve Git LFS content. However, if the checkout fails for an unexpected reason, you can download any missing Git LFS content for the current commit with git lfs pull:

Like git lfs clone, git lfs pull downloads your Git LFS files as a batch. If you know a large number of files have changed since the last time you pulled, you may wish to disable the automatic Git LFS download during checkout, and then batch download your Git LFS content with an explicit git lfs pull. This can be done by overriding your Git config with the -c option when you invoke git pull:

Since that's rather a lot of typing, you may wish to create a simple Git alias to perform a batched Git and Git LFS pull for you:

This will greatly improve performance when a large number of Git LFS files need to be downloaded (again, especially on Windows).

When you add a new type of large file to your repository, you'll need to tell Git LFS to track it by specifying a pattern using the git lfs track command:

Note that the quotes around "*.ogg" are important. Omitting them will cause the wildcard to be expanded by your shell, and individual entries will be created for each .ogg file in your current directory:

The patterns supported by Git LFS are the same as those supported by .gitignore, for example:

These patterns are relative to the directory in which you ran the git lfs track command. To keep things simple, it is best to run git lfs track from the root of your repository. Note that Git LFS does not support negative patterns like .gitignore does.

After running git lfs track, you'll notice a new file named .gitattributes in the directory you ran the command from. .gitattributes is a Git mechanism for binding special behaviors to certain file patterns. Git LFS automatically creates or updates .gitattributes files to bind tracked file patterns to the Git LFS filter. However, you will need to commit any changes to the .gitattributes file to your repository yourself:

For ease of maintenance, it is simplest to keep all Git LFS patterns in a single .gitattributes file by always running git lfs track from the root of your repository. However, you can display a list of all patterns that are currently tracked by Git LFS (and the .gitattributes files they are defined in) by invoking git lfs track with no arguments:

You can stop tracking a particular pattern with Git LFS by simply removing the appropriate line from your .gitattributes file, or by running the git lfs untrack command:

After running git lfs untrack you will again have to commit the changes to .gitattributes yourself.

You can commit and push as normal to a repository that contains Git LFS content. If you have committed changes to files tracked by Git LFS, you will see some additional output from git push as the Git LFS content is transferred to the server:

If transferring the LFS files fails for some reason, the push will be aborted and you can safely try again. Like Git, Git LFS storage is content addressable: content is stored against a key which is a SHA-256 hash of the content itself. This means it is always safe to re-attempt transferring Git LFS files to the server; you can't accidentally overwrite a Git LFS file's contents with the wrong version.

To migrate a Git LFS repository from one hosting provider to another, you can use a combination of git lfs fetch and git lfs push with the --all option specified.

For example, to move all Git and Git LFS repository from a remote named github to a remote named bitbucket 😉 :

Git LFS typically only downloads the files needed for commits that you actually checkout locally. However, you can force Git LFS to download extra content for other recently modified branches using git lfs fetch --recent:

This is useful for batch downloading new Git LFS content while you're out at lunch, or if you're planning on reviewing work from your teammates and will not be able to download content later on due to limited internet connectivity. For example, you may wish to run git lfs fetch --recent before jumping on a plane!

Git LFS considers any branch or tag containing a commit newer than seven days as recent. You can configure the number of days considered as recent by setting the lfs.fetchrecentrefsdays property:

By default, git lfs fetch --recent will only download Git LFS content for the commit at the tip of a recent branch or tag.

However you can configure Git LFS to download content for earlier commits on recent branches and tags by configuring the lfs.fetchrecentcommitsdays property:

Use this setting with care: if you have fast moving branches, this can result in a huge amount of data being downloaded. However it can be useful if you need to review interstitial changes on a branch, cherry picking commits across branches, or rewrite history.

As discussed in Moving a Git LFS repository between hosts, you can also elect to fetch all Git LFS content for your repository with git lfs fetch --all:

You can delete files from your local Git LFS cache with the git lfs prune command:

This will delete any local Git LFS files that are considered old. An old file is any file not referenced by:

By default, a recent commit is any commit created in the last ten days. This is calculated by adding:

You can configure the prune offset to retain Git LFS content for a longer period:

Unlike Git's built-in garbage collection, Git LFS content is not pruned automatically, so running git lfs prune on a regular basis is a good idea to keep your local repository size down.

You can test out what effect a prune operation will have with git lfs prune --dry-run:

And exactly which Git LFS objects will be pruned with git lfs prune --verbose --dry-run:

The long hexadecimal strings output by --verbose mode are SHA-256 hashes (also known as Object IDs, or OIDs) of the Git LFS objects to be pruned. You can use the techniques described in Finding paths or commits that reference a Git LFS object to find our more about the objects that will be pruned.

As an additional safety check, you can use the --verify-remote option to check whether the remote Git LFS store has a copy of your Git LFS objects before they are pruned:

This makes the pruning process significantly slower, but gives you peace of mind knowing that any pruned objects are recoverable from the server. You can enable the --verify-remote option permanently for your system by configuring the lfs.pruneverifyremotealways property globally:

Or you can enable remote verification for just the context repository by omitting the --global option from the command above.

The Git LFS command-line client doesn't support pruning files from the server, so how you delete them depends on your hosting provider.

In Bitbucket Cloud, you can view and delete Git LFS files via Repository Settings > Git LFS:

Note that each Git LFS file is indexed by its SHA-256 OID; the paths that reference each file are not visible via the UI. This is because there could be many different paths at many different commits that may refer to a given object, so looking them up would be a very slow process.

To determine what a given Git LFS file actually contains, you have three options:

If you have a Git LFS SHA-256 OID, you can determine which commits reference it with git log --all -p -S :

This git log incantation generates a patch (-p) from commits on any branch (--all) that add or remove a line (-S) containing the specified string (a Git LFS SHA-256 OID).

The patch shows you the commit and the path to the LFS object, as well as who added it, and when it was committed. You can simply checkout the commit, and Git LFS will download the file if needed and place it in your working copy.

If you suspect that a particular Git LFS object is in your current HEAD, or on a particular branch, you can use git grep to find the file path that references it:

You can replace HEAD or power-ups with any ref, commit, or tree that contains the Git LFS object.

In some situations you may want to only download a subset of the available Git LFS content for a particular commit. For example, when configuring a CI build to run unit tests, you may only need your source code, so may want to exclude heavyweight files that aren't necessary to build your code.

You can exclude a pattern or subdirectory using git lfs fetch -X (or --exclude):

Alternatively, you may want to only include a particular pattern or subdirectory. For example, an audio engineer could fetch just ogg and wav files with git lfs fetch -I (or --include):

If you combine includes and excludes, only files that match an include pattern and do not match an exclude pattern will be fetched. For example, you can fetch everything in your Assets directory except gifs with:

Excludes and includes support the same patterns as git lfs track and .gitignore. You can make these patterns permanent for a particular repository by setting the lfs.fetchinclude and lfs.fetchexclude config properties:

These settings can also be applied to every repository on your system by appending the --global option.

Unfortunately, there is no easy way of resolving binary merge conflicts. With Git LFS file locking, you can lock files by extension or by file name and prevent binary files from being overwritten during a merge.

In order to take advantage of LFS' file locking feature, you first need to tell Git which type of files are lockable. In the example below, the `--lockable` flag is appended to the `git lfs track` command which both stores PSD files in LFS and marks them as lockable.

Then add the following to your .gitattributes file:

When preparing to make changes to an LFS file, you'll use the lock command in order to register the file as locked on your Git server.

Once you no longer need the file lock, you can remove it using Git LFS' unlock command.

Git LFS file locks can be overridden, similar to git push, using a --force flag. Do not use the --force flag unless you’re absolutely sure you know what you’re doing.

If you're interested in learning more about clean and smudge filters, pre-push hooks, and the other interesting computer science behind Git LFS, check out this presentation from Atlassian on Git LFS at LinuxCon 2016:

Managing large files such as audio, video and graphics files has always been one of the shortcomings of Git. The general recommendation is to not have Git repositories larger than 1 GB to preserve performance.

Files tracked by Git LFS display an icon to indicate if the file is stored as a blob or an LFS pointer.

Git LFS client communicates with the GitLab server over HTTPS. It uses HTTP Basic Authentication to authorize client requests. After the request is authorized, Git LFS client receives instructions from where to fetch or where to push the large file.

Documentation for GitLab instance administrators is under LFS administration doc.

When you add an LFS object to a repository, GitLab:

When your repository is forked, LFS objects from the upstream project are associated with the fork. When the fork is created, the LFS object storage for the fork is equal to the storage used by the upstream project. If new LFS objects are added to the fork, the total object storage changes for the fork, but not the upstream project.

If you create a merge request from the fork back to the upstream project, any new LFS objects in the fork become associated with the upstream project.

Let’s take a look at the workflow for checking large files into your Git repository with Git LFS. For example, if you want to upload a very large file and check it into your Git repository:

After you mark a file extension for tracking as a LFS object you can use Git as usual without redoing the command to track a file with the same extension:

Make sure that .gitattributes is tracked by Git. Otherwise Git LFS doesn’t work properly for people cloning the project:

Cloning the repository works the same as before. Git automatically detects the LFS-tracked files and clones them via HTTP. If you performed the git clone command with a SSH URL, you have to enter your GitLab credentials for HTTP authentication.

If you already cloned the repository and you want to get the latest LFS object that are on the remote repository, such as for a branch from origin:

Make sure your files aren’t listed in .gitignore, otherwise, they are ignored by Git and are not pushed to the remote repository.

Read the documentation on how to migrate an existing Git repository with Git LFS.

To remove objects from LFS:

See the documentation on File Locking.

Prior to GitLab 13.5, project source downloads would include Git LFS pointers instead of the actual objects. For example, LFS pointers look like the following:

In GitLab version 13.5 and later, these pointers are converted to the uploaded LFS object.

Technical details about how this works can be found in the development documentation for LFS.

This error indicates the files are expected to be tracked by LFS, but the repository is not tracking them as LFS. This issue can be one potential reason for this error: Files not tracked with LFS when uploaded through the web interface

To resolve the problem, migrate the affected file (or files) and push back to the repository:

This error can occur for a few reasons, including:

Check if you have permissions to push to the project or fetch from the project.

LFS object you are trying to push to the project or fetch from the project is not available to the project anymore. Probably the object was removed from the server.

Git LFS logs the failures into a log file. To view this log file, while in project directory:

If the status error 501 is shown, it is because:

If you push an LFS object to a project and receive an error like this, the LFS client is trying to reach GitLab through HTTPS. However, your GitLab instance is being served on HTTP:

This behavior is caused by Git LFS using HTTPS connections by default when a lfsurl is not set in the Git configuration.

To prevent this from happening, set the LFS URL in project Git configuration:

Git LFS authenticates the user with HTTP Basic Authentication on every push for every object, so user HTTPS credentials are required.

By default, Git has support for remembering the credentials for each repository you use. For more information, see the official Git documentation.

For example, you can tell Git to remember the password for a period of time in which you expect to push the objects:

This remembers the credentials for an hour, after which Git operations require re-authentication.

If you are using OS X you can use osxkeychain to store and encrypt your credentials. For Windows, you can use wincred or Microsoft’s Git Credential Manager for Windows.

More details about various methods of storing the user credentials can be found on Git Credential Storage documentation.

GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab.

Verify that LFS is installed locally and consider a manual push with git lfs push --all.

If you are storing LFS files outside of GitLab you can disable LFS on the project by setting lfs_enabled: false with the projects API.