Remove Secrets with the BFG Repo Cleaner

Photo by Ujesh Krishnan on Unsplash

This time I want to show you how to remove secrets from a repository. Sometimes you are developing a feature which contains sensitive information like a password, a token or some other information. This kind of information shouldn’t be in your repository. It could be that someone with access to your repository could misuse it. This has happened to big companies like Disney, Adobe and Microsoft already (see Source Code Leaked Online).

Of course it’s the best is to avoid pushes to the repository, which contains sensitive information in the first place. But what to do if you have already such kind of information in your repository? Unfortunately, its not enough to delete the secret locally and push it to the repository. Due to the nature of Git the secret is still in the history. BFG Repo Cleaner is a tool, which can help you out. You can download the binary from the homepage or check out the github repository. There you can also find additional information about the tool.

First, you have to identify your secrets and remove them from the repository. Don’t forget to push your commit. The BFG Repo Cleaner doesn’t touch your latest commit. So if you don’t remove the secrets beforehand you wouldn’t get rid of them.

The next step is to create a new clone from the repository but with the -mirror flag. By doing this your repository will end with the suffix .git and it doesn’t check out the branch. Its intention is to be the remote origin where no one is working on. When you take a look into the directory you will see only files and folders necessary for git. You can double check it by taking a look into the .git folder of your daily work repository. The mirrored repository will be cleaned up and we will override the current remote origin with this one. So let’s clone the repository with this command:

git clone --mirror git://example.com/my-repo.git

To be on the save site you should create a backup. If something goes wrong you can push the backup to the remote and you are back at the starting point.

git clone --mirror git://example.com/my-repo.git my-repo-backup.git

Remove passwords from repository

BFG has a --replace-text flag which is able to read a file. The easiest way to remove the secrets from your repository is to put all your secrets in a file. Every secret needs to be in a separate line. BFG goes through all files in the history and replaces the given strings from the file with the text ***REMOVED*** by default. Here’s the help output from BFG for replace-text:

--replace-text <expressions-file>
        filter content of files, replacing matched text. Match expressions should be
        listed in the file, one expression per line - by default, each expression is
        treated as a literal, but 'regex:' & 'glob:' prefixes are supported, with '==>'

The snippet gives you an incomplete overview of the format:

admin123            # Replace admin123 with ***REMOVED*** (default) admin123==>password # Replace with ‘password’ 
insteadadmin123==>  # Replace with the empty string regex:password=\w+==>password=  #Replace, using a regex regex:\r(\n)==>$1   # Replace Windows newlines with Unix newlines

Save your secrets in the separate file with the name password.txt and type in the following command:

java -jar bfg-version.jar --replace-text passwords.txt  my-repo.git

Depending on your password.txt BFG replaces all matching secrets in your history with the text ***REMOVED***. During this process, you will see statistic information in the terminal and it should finish with the line:

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive

Change into the directory of your repository and follow the advice to run the git commands.

Remove sensitive files from repository

Let’s say you pushed your private RSA key to the repository by coincidence. This is definitely a case to create a new one and don’t use the old one anymore. Nevertheless you can also remove this file from your history.

java -jar bfg-version.jar --delete-files id_rsa  my-repo.git

Instead to give one specific file you can also make use of a wildcard and add multiple file names:

java -jar bfg-version.jar --delete-files *.{class,log}  my-repo.git

In the above example all files which ends with the filename class or log will be deleted.

Push your changes to origin

Of course, your branch has diverged commits and you have to push your cleaned repository with the force flag:

git push -f

If you work in a team, this command has an impact on them, too. So everyone has to run the commands

git fetch origin
git reset --hard origin/master

Issues

I had the issue that the git push -f command rejected a pull request made by GitHub. I got the following error message:

! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref)

What you can do is to merge or close the open pull request beforehand. So you don’t run into this issue.

If you get the following message it means probably that you haven’t removed the secret/file from the repository yet.

These are your protected commits, and so their contents will NOT be altered:
 * commit 5c04950a (protected by 'HEAD') - contains 1 dirty file : 
	- id_rsa (3.2 KB)
WARNING: The dirty content above may be removed from other commits, but as
the *protected* commits still use it, it will STILL exist in your repository.

I hope this tutorial can help you to cleanse your secrets from the repository and make it more secure. BFG has also other nice features like to remove folders, remove big files or exclude specific files from removing. For more detailed information check out the BFG Homepage. Let a comment and stay tuned.