1GIT-FILTER-REPO(1) Git Manual GIT-FILTER-REPO(1)
2
6 git-filter-repo - Rewrite repository history
7
9 git filter-repo --analyze
10 git filter-repo [<path_filtering_options>] [<content_filtering_options>]
11 [<ref_renaming_options>] [<commit_message_filtering_options>]
12 [<name_or_email_filtering_options>] [<parent_rewriting_options>]
13 [<generic_callback_options>] [<miscellaneous_options>]
14
17 Rapidly rewrite entire repository history using user-specified filters.
18 This is a destructive operation which should not be used lightly; it
19 writes new commits, trees, tags, and blobs corresponding to (but
20 filtered from) the original objects in the repository, then deletes the
21 original history and leaves only the new. See the section called
22 “DISCUSSION” for more details on the ramifications of using this tool.
23 Several different types of history rewrites are possible; examples
24 include (but are not limited to):
25 · stripping large files (or large directories or large extensions)
27 · stripping unwanted files by path
29 · extracting wanted paths and their history (stripping everything
31 else)
32 · restructuring the file layout (such as moving all files into a
34 subdirectory in preparation for merging with another repo, making a
35 subdirectory become the new toplevel directory, or merging two
36 directories with independent filenames into one directory)
37 · renaming tags (also often in preparation for merging with another
39 repo)
40 · replacing or removing sensitive text such as passwords
42 · making mailmap rewriting of user names or emails permanent
44 · making grafts or replacement refs permanent
46 · rewriting commit messages
48 Additionally, several concerns are handled automatically (many of these
50 can be overridden, but they are all on by default):
51 · rewriting (possibly abbreviated) hashes in commit messages to refer
53 to the new post-rewrite commit hashes
54 · pruning commits which become empty due to the above filters (also
56 handles edge cases like pruning of merge commits which become
57 degenerate and empty)
58 · creating replace-refs (see git-replace(1)) for old commit hashes,
60 which if pushed and fetched will allow users to continue to refer
61 to new commits using (unabbreviated) old commit IDs
62 · stripping of original history to avoid mixing old and new history
64 · repacking the repository post-rewrite to shrink the repo for the
66 user
67 Also, it’s worth noting that there is an important safety mechanism:
69 · abort if run from a repo that is not a fresh clone (to prevent
71 accidental data loss from rewriting local history that doesn’t
72 exist anywhere else). See the section called “FRESH CLONE SAFETY
73 CHECK AND --FORCE”.
74 For those who know that there is large unwanted stuff in their history
76 and want help finding it, this command also
77 · provides an option to analyze a repository and generate reports
79 that can be useful in determining what to filter (or in determining
80 whether a separate filtering command was successful).
81 See also the section called “VERSATILITY”, the section called
83 “DISCUSSION”, the section called “EXAMPLES”, and the section called
84 “INTERNALS”.
85
87 Analysis Options
88 --analyze
89 Analyze repository history and create a report that may be useful
90 in determining what to filter in a subsequent run (or in
91 determining if a previous filtering command did what you wanted).
92 Will not modify your repo.
93 Filtering based on paths (see also --filename-callback)
95 --invert-paths
96 Invert the selection of files from the specified
97 --path-{match,glob,regex} options below, i.e. only select files
98 matching none of those options.
99 --path-match <dir_or_file>, --path <dir_or_file>
101 Exact paths (files or directories) to include in filtered history.
102 Multiple --path options can be specified to get a union of paths.
103 --path-glob <glob>
105 Glob of paths to include in filtered history. Multiple --path-glob
106 options can be specified to get a union of paths.
107 --path-regex <regex>
109 Regex of paths to include in filtered history. Multiple
110 --path-regex options can be specified to get a union of paths.
111 --use-base-name
113 Match on file base name instead of full path from the top of the
114 repo. Incompatible with --path-rename, and incompatible with
115 matching against directory names.
116 Renaming based on paths (see also --filename-callback)
118 Note: if you combine path filtering with path renaming, be aware that a
119 rename directive does not select paths, it only says how to rename
120 paths that are selected with the filters.
121 --path-rename <old_name:new_name>, --path-rename-match
123 <old_name:new_name>
124 Path to rename; if filename or directory matches <old_name> rename
125 to <new_name>. Multiple --path-rename options can be specified.
126 Path shortcuts
128 --paths-from-file <filename>
129 Specify several path filtering and renaming directives, one per
130 line. Lines with ==> in them specify path renames, and lines can
131 begin with literal: (the default), glob:, or regex: to specify
132 different matching styles. Blank lines and lines starting with a #
133 are ignored (if you have a filename that you want to filter on that
134 starts with literal:, #, glob:, or regex:, then prefix the line
135 with literal:).
136 --subdirectory-filter <directory>
138 Only look at history that touches the given subdirectory and treat
139 that directory as the project root. Equivalent to using --path
140 <directory>/ --path-rename <directory>/:
141 --to-subdirectory-filter <directory>
143 Treat the project root as instead being under <directory>.
144 Equivalent to using --path-rename :<directory>/
145 Content editing filters (see also --blob-callback)
147 --replace-text <expressions_file>
148 A file with expressions that, if found, will be replaced. By
149 default, each expression is treated as literal text, but regex: and
150 glob: prefixes are supported. You can end the line with ==> and
151 some replacement text to choose a replacement choice other than the
152 default of ***REMOVED***.
153 --strip-blobs-bigger-than <size>
155 Strip blobs (files) bigger than specified size (e.g. 5M, 2G, etc)
156 --strip-blobs-with-ids <blob_id_filename>
158 Read git object ids from each line of the given file, and strip all
159 of them from history
160 Renaming of refs (see also --refname-callback)
162 --tag-rename <old:new>
163 Rename tags starting with <old> to start with <new>. For example,
164 --tag-rename foo:bar will rename tag foo-1.2.3 to bar-1.2.3; either
165 <old> or <new> can be empty.
166 Filtering of commit messages (see also --message-callback)
168 --preserve-commit-hashes
169 By default, since commits are rewritten and thus gain new hashes,
170 references to old commit hashes in commit messages are replaced
171 with new commit hashes (abbreviated to the same length as the old
172 reference). Use this flag to turn off updating commit hashes in
173 commit messages.
174 --preserve-commit-encoding
176 Do not reencode commit messages into UTF-8. By default, if the
177 commit object specifies an encoding for the commit message, the
178 message is re-encoded into UTF-8.
179 Filtering of names & emails (see also --name-callback and --email-callback)
181 --mailmap <filename>
182 Use specified mailmap file (see git-shortlog(1) for details on the
183 format) when rewriting author, committer, and tagger names and
184 emails. If the specified file is part of git history, historical
185 versions of the file will be ignored; only the current contents are
186 consulted.
187 --use-mailmap
189 Same as: --mailmap .mailmap
190 Parent rewriting
192 --replace-refs {delete-no-add, delete-and-add, update-no-add,
193 update-or-add, update-and-add}
194 Replace refs (see git-replace(1)) are used to rewrite parents
195 (unless turned off by the usual git mechanism); this flag specifies
196 what do do with those refs afterward. Replace refs can either be
197 deleted or updated to point at new commit hashes. Also, new replace
198 refs can be added for each commit rewrite. With update-or-add, new
199 replace refs are only added for commit rewrites that aren’t used to
200 update an existing replace ref. default is update-and-add if
201 $GIT_DIR/filter-repo/already_ran does not exist; update-or-add
202 otherwise.
203 --prune-empty {always, auto, never}
205 Whether to prune empty commits. auto (the default) means only
206 prune commits which become empty (not commits which were empty in
207 the original repo, unless their parent was pruned). When the parent
208 of a commit is pruned, the first non-pruned ancestor becomes the
209 new parent.
210 --prune-degenerate {always, auto, never}
212 Since merge commits are needed for history topology, they are
213 typically exempt from pruning. However, they can become degenerate
214 with the pruning of other commits (having fewer than two parents,
215 having one commit serve as both parents, or having one parent as
216 the ancestor of the other.) If such merge commits have no file
217 changes, they can be pruned. The default (auto) is to only prune
218 empty merge commits which become degenerate (not which started as
219 such).
220 --no-ff
222 Even if the first parent is or becomes an ancestor of another
223 parent, do not prune it. This modifies how --prune-degenerate
224 behaves, and may be useful in projects who always use merge
225 --no-ff.
226 Generic callback code snippets
228 --filename-callback <function_body>
229 Python code body for processing filenames; see the section called
230 “CALLBACKS”.
231 --message-callback <function_body>
233 Python code body for processing messages (both commit messages and
234 tag messages); see the section called “CALLBACKS”.
235 --name-callback <function_body>
237 Python code body for processing names of people; see the section
238 called “CALLBACKS”.
239 --email-callback <function_body>
241 Python code body for processing emails addresses; see the section
242 called “CALLBACKS”.
243 --refname-callback <function_body>
245 Python code body for processing refnames; see the section called
246 “CALLBACKS”.
247 --blob-callback <function_body>
249 Python code body for processing blob objects; see the section
250 called “CALLBACKS”.
251 --commit-callback <function_body>
253 Python code body for processing commit objects; see the section
254 called “CALLBACKS”.
255 --tag-callback <function_body>
257 Python code body for processing tag objects; see the section called
258 “CALLBACKS”.
259 --reset-callback <function_body>
261 Python code body for processing reset objects; see the section
262 called “CALLBACKS”.
263 Location to filter from/to
265 Note
266 Specifying alternate source or target locations implies --partial
267 except that the normal default for --replace-refs is used. However,
268 unlike normal uses of --partial, this doesn’t risk mixing old and
269 new history since the old and new histories are in different
270 repositories.
271 --source <source>
273 Git repository to read from
274 --target <target>
276 Git repository to overwrite with filtered history
277 Miscellaneous options
279 --help, -h
280 Show a help message and exit.
281 --force, -f
283 Ignore fresh clone checks and rewrite history (an irreversible
284 operation, especially since it by default ends with an immediate
285 pruning of reflogs and old objects). See the section called “FRESH
286 CLONE SAFETY CHECK AND --FORCE”. Note that when cloning repos on a
287 local filesystem, it is better to pass --no-local to git clone than
288 passing --force to git-filter-repo.
289 --partial
291 Do a partial history rewrite, resulting in the mixture of old and
292 new history. This implies a default of update-no-add for
293 --replace-refs, disables rewriting refs/remotes/origin/* to
294 refs/heads/*, disables removing of the origin remote, disables
295 removing unexported refs, disables expiring the reflog, and
296 disables the automatic post-filter gc. Also, this modifies
297 --tag-rename and --refname-callback options such that instead of
298 replacing old refs with new refnames, it will instead create new
299 refs and keep the old ones around. Use with caution.
300 --refs <refs+>
302 Limit history rewriting to the specified refs. Implies --partial.
303 In addition to the normal caveats of --partial (mixing old and new
304 history, no automatic remapping of refs/remotes/origin/* to
305 refs/heads/*, etc.), this also may cause problems for pruning of
306 degenerate empty merge commits when negative revisions are
307 specified.
308 --dry-run
310 Do not change the repository. Run git fast-export and filter its
311 output, and save both the original and the filtered version for
312 comparison. This also disables rewriting commit messages due to not
313 knowing new commit IDs and disables filtering of some empty commits
314 due to inability to query the fast-import backend.
315 --debug
317 Print additional information about operations being performed and
318 commands being run. (If used together with --dry-run, shows extra
319 information about what would be run).
320 --stdin
322 Instead of running git fast-export and filtering its output, filter
323 the fast-export stream from stdin. The stdin must be in the
324 expected input format (e.g. it needs to include original-oid
325 directives).
326 --quiet
328 Pass --quiet to other git commands called.
329
331 Every time filter-repo is run, files are created in the
332 .git/filter-repo/ directory. These files overwritten unconditionally on
333 every run.
334 Commit map
336 The .git/filter-repo/commit-map file contains a mapping of how all
337 commits were (or were not) changed.
338 · A header is the first line with the text "old" and "new"
340 · Commit mappings are in no particular order
342 · All commits in range of the rewrite will be listed, even commits
344 that are unchanged (e.g. because the commit pre-dated when the
345 large file(s) were introduced to the repo).
346 · An all-zeros hash, or null SHA, represents a non-existant object.
348 When in the "new" column, this means the commit was removed
349 entirely.
350 Reference map
352 The .git/filter-repo/ref-map file contains a mapping of which local
353 references were changed.
354 · A header is the first line with the text "old" and "new"
356 · Reference mappings are in no particular order
358 · An all-zeros hash, or null SHA, represents a non-existant object.
360 When in the "new" column, this means the ref was removed entirely.
361
363 Since filter-repo does irreversible rewriting of history, it is
364 important to avoid making changes to a repo for which the user doesn’t
365 have a good backup. The primary defense mechanism is to simply educate
366 users and rely on them to be good stewards of their data; thus there
367 are several warnings in the documentation about how filter repo
368 rewrites history.
369 However, as a service to users, we would like to provide an additional
371 safety check beyond the documentation. There isn’t a good way to check
372 if the user has a good backup, but we can ask a related question that
373 is an imperfect but quite reasonable proxy: "Is this repository a fresh
374 clone?" Unfortunately, that is also a question we can’t get a perfect
375 answer to; git provides no way to answer that question. However, there
376 are approximately a dozen things that I found that seem to always be
377 true of brand new clones (assuming they are either clones of remote
378 repositories or are made with the --no-local flag), and I check for all
379 of those.
380 These checks can have both false positives and false negatives. Someone
382 might have a perfectly good backup of their repo without it actually
383 being a fresh clone — but there’s no way for filter-repo to know that.
384 Conversely, someone could look at all things that filter-repo checks
385 for in its safety checks and then just tweak their non-backed-up
386 repository to satisfy those conditions (though it would take a fair
387 amount of effort, and it’s astronomically unlikely that a repo that
388 isn’t a fresh clone randomly happens to match all the criteria). In
389 practice, the safety checks filter-repo uses seem to be really good at
390 avoiding people accidentally running filter-repo on a repository that
391 they shouldn’t be running it on. It even caught me once when I did mean
392 to run filter-repo but was in a different directory than I thought I
393 was.
394 In short, it’s perfectly fine to use ‘--force` to override the safety
396 checks as long as you’re okay with filter-repo irreversibly rewriting
397 the contents of the current repository. It is a really bad idea to get
398 in the habit of always specifying --force; if you do, one day you will
399 run one of your commands in the wrong directory like I did, and you
400 won’t have the safety check anymore to bail you out. Also, it is
401 definitely NOT okay to recommend --force on forums, Q&A sites, or in
402 emails to other users without first carefully explaining that --force
403 means putting your repositories’ data at risk. I am especially bothered
404 by people who suggest the flag when it clearly is NOT needed; they are
405 needlessly putting other peoples' data at risk.
406
408 filter-repo has a hierarchy of capabilities on the spectrum from easy
409 to use convenience flags that perform pre-defined types of filtering,
410 to choices that provide lots of flexibility in controlling how
411 filtering occurs. This spectrum includes the following:
412 · Convenience flags making common types of history rewriting simple
414 (e.g. --path, --strip-blobs-bigger-than, --replace-text, --mailmap)
415 · Options which are shorthand for others or which provide greater
417 control than others (e.g. --subdirectory-filter could just be
418 written using both a path selection (--path) and a path rename
419 (--path-rename) filter; --paths-from-file can handle all other
420 --path* options and more such as regex renaming of paths)
421 · Generic python callbacks for handling a certain type of data (the
423 filename, message, name, email, and refname callbacks)
424 · Generic python callbacks for handling fundamental git objects,
426 allowing greater control over the combination of data types the
427 object holds (the commit, tag, blob, and reset callbacks)
428 · The ability to import filter-repo as a module in a python program
430 and use its classes and functions for even greater control and
431 flexibility while still leveraging lots of basic capabilities. One
432 can even use this to write new tools with a completely different
433 interface.
434 For more information about callbacks, see the section called
436 “CALLBACKS”. For examples on writing python programs that import
437 filter-repo as a module to create new history rewriting tools, look at
438 the contrib/filter-repo-demos/ directory. That directory includes,
439 among other examples, a reimplementation of git-filter-branch which is
440 faster than git-filter-branch, and a reimplementation of BFG Repo
441 Cleaner with several bug fixes and new features.
442
444 Using filter-repo is relatively simple, but rewriting history is part
445 of a larger discussion in terms of collaboration. When you rewrite
446 history, the old and new histories are no longer compatible; if you
447 push this history somewhere for others to view, it will look as though
448 you’ve done a rebase of all branches and tags. Make sure you are
449 familiar with the "RECOVERING FROM UPSTREAM REBASE" section of git-
450 rebase(1) (and in particular, "The hard case") before proceeding, in
451 addition to this section.
452 Steps to use git-filter-repo as part of the bigger picture of doing a
454 history rewrite are roughly as follows:
455 1. Create a clone of your repository (if you created special refs
457 outside of refs/heads/ or refs/tags/, make sure to fetch those
458 too). You may pass --bare or --mirror to git clone, if you prefer.
459 You should pass --no-local if the repository you are cloning from
460 is on the local filesystem. Avoid other flags; some might confuse
461 the fresh clone check, and others could cause parts of the data to
462 be missing that are needed for the rewrite.
463 2. (Optional) Run git filter-repo --analyze. This will create a
465 directory of reports mentioning renames that have occurred in your
466 repo and also listing sizes of objects aggregated by
467 path/directory/extension/blob-id; this information may be useful in
468 choosing how to filter your repo. It can also be useful to re-run
469 --analyze after filtering to verify the changes look correct.
470 3. Run filter-repo with your desired filtering options. Many examples
472 are given below. For more complex cases, note that doing the
473 filtering in multiple steps (by running multiple filter-repo
474 invocations in a sequence) is supported. If anything goes wrong
475 here, simply delete your clone and restart.
476 4. Push your new repository to its new home (note that
478 refs/remotes/origin/* will have been moved to refs/heads/* as the
479 first part of filter-repo, so you can just deal with normal
480 branches instead of remote tracking branches). While you can force
481 push this to the same URL you cloned from, there are good reasons
482 to consider pushing to a different location instead:
483 · People who cloned from the original repo will have old history.
485 When they fetch the new history you force pushed up, unless
486 they do a git reset --hard @{u} on their branches or rebase
487 their local work, git will think they have hundreds or
488 thousands of commits with very similar commit messages as what
489 exist upstream (but which include files you wanted excised from
490 history), and allow the user to merge the two histories,
491 resulting in what looks like two copies of each commit. If they
492 then push this history back up, then everyone now has history
493 with two copies of each commit and the bad files have returned.
494 You’re more likely to succeed in forcing people to get rid of
495 the old history if they have to clone a new URL.
496 · Rewriting history will rewrite tags; those who have already
498 downloaded tags will not get the updated tags by default (see
499 the "On Re-tagging" section of git-tag(1)). Every user trying
500 to use an existing clone will have to forcibly delete all tags
501 and re-fetch them; it may be easier for them to just re-clone,
502 which they are more likely to do with a new clone URL.
503 · Rewriting history may delete some refs (e.g. branches that only
505 had files that you wanted excised from history); unless you run
506 git push with the --mirror or --prune options, those refs will
507 continue to exist on the server. If folks then merge these
508 branches into others, then people have started mixing old and
509 new history. If users had already cloned these branches,
510 removing them from the server isn’t enough; you need all users
511 to delete any local branches based on these refs and run fetch
512 with the --prune option as well. Simply re-cloning from a new
513 URL is easier.
514 · The server may not allow you to force push over some refs. For
516 example, code review systems may have special ref namespaces
517 (e.g. refs/changes/, refs/pull/, refs/merge-requests/) that
518 they have locked down.
519 5. If you still want to push your rewritten history back to the
521 original url despite my warnings above, you’ll have to manage it
522 very carefully:
523 · git-filter-repo deletes the "origin" remote to help avoid
525 people accidentally repushing to the same repository, so you’ll
526 need to remind git what origin’s url was. You’ll have to look
527 up the command for that.
528 · You’ll need to carefully synchronize with everyone who has
530 cloned the repository, and will also need to carefully
531 synchronize with everything (e.g. CI systems) that has cloned
532 it. Every single clone will either need to be thrown away and
533 re-cloned, or need to take all the steps outlined in item 4 as
534 well as follow the necessary steps from "RECOVERING FROM
535 UPSTREAM REBASE" section of git-rebase(1). If you miss fixing
536 any clones, you’ll risk mixing old and new history and end up
537 with an even worse mess to clean up.
538 · Finally, you’ll need to consult any documentation from your
540 hosting provider about how to remove any server-side references
541 to the old commits (example: GitLab’s docs on reducing
542 repository size[1]).
543 6. (Optional) Some additional considerations
545 · filter-repo by default creates replace refs (see git-
547 replace(1)) for each rewritten commit ID, allowing you to use
548 old (unabbreviated) commit hashes to refer to the newly
549 rewritten commits. If you want to use these replace refs, push
550 them to the relevant clone URL and tell users to adjust their
551 fetch refspec (e.g. git config --add remote.origin.fetch
552 +refs/replace/*:refs/replace/*) Sadly, some existing git
553 servers (e.g. Gerrit, GitHub) do not yet understand replace
554 refs, and thus one can’t use old commit hashes within their UI;
555 this may change in the future. But replace refs at least help
556 users locally within the git CLI.
557 · If you have a central repo, you may want to prevent people from
559 pushing old commit IDs, in order to avoid mixing old and new
560 history. Every repository manager does this differently, some
561 provide specialized commands (e.g.
562 https://gerrit-review.googlesource.com/Documentation/cmd-ban-commit.html),
563 others require you to write hooks.
564
566 Path based filtering
567 To only keep the README.md file plus the directories guides and
568 tools/releases/:
569 git filter-repo --path README.md --path guides/ --path tools/releases
571 Directory names can be given with or without a trailing slash, and all
574 filenames are relative to the toplevel of the repo. To keep all files
575 except these paths, just add --invert-paths:
576 git filter-repo --path README.md --path guides/ --path tools/releases --invert-paths
578 If you want to have both an inclusion filter and an exclusion filter,
581 just run filter-repo multiple times. For example, to keep the src/main
582 subdirectory but exclude files under src/main named data, run:
583 git filter-repo --path src/main/
585 git filter-repo --path-glob 'src/*/data' --invert-paths
586 Note that the asterisk (*) will match across multiple directories, so
589 the second command would remove e.g. src/main/org/whatever/data. Also,
590 the second command by itself would also remove e.g.
591 src/not-main/foo/data, but since src/not-main/ was removed by the first
592 command, that’s not an issue. Also, the use of quotes around the
593 asterisk is sometimes important to avoid glob expansion by the shell.
594 You can also select paths by regular expression (see
596 https://docs.python.org/3/library/re.html#regular-expression-syntax).
597 For example, to only include files from the repo whose name is in the
598 format YYYY-MM-DD.txt and is found at least two subdirectories deep:
599 git filter-repo --path-regex '^.*/.*/[0-9]{4}-[0-9]{2}-[0-9]{2}.txt$'
601 If you want two directories to be renamed (and maybe merged if both are
604 renamed to the same location), use --path-rename; for example, to
605 rename both cmds/ and src/scripts/ to tools/:
606 git filter-repo --path-rename cmds:tools --path-rename src/scripts/:tools/
608 As with --path, directories can be specified with or without a trailing
611 slash for --path-rename.
612 If you do a --path-rename to something that was already in use, it will
614 be silently overwritten. However, if you try to rename multiple files
615 to the same location (e.g. src/scripts/run_release.sh and
616 cmds/run_release.sh both existed and had different content with the
617 renames above), then you will be given an error. If you have such a
618 case, you may want to add another rename command to move one of the
619 paths somewhere else where it won’t collide:
620 git filter-repo --path-rename cmds/run_release.sh:tools/do_release.sh \
622 --path-rename cmds/:tools/ \
623 --path-rename src/scripts/:tools/
624 Also, --path-rename brings up ordering issues; all path arguments are
627 applied in order. Thus, a command like
628 git filter-repo --path-rename sources/:src/main/ --path src/main/
630 would make sense but reversing the two arguments would not (src/main/
633 is created by the rename so reversing the two would give you an empty
634 repo). Also, note that the rename of cmds/run_release.sh a couple
635 examples ago was done before the other renames.
636 Note that path renaming does not do path filtering, thus the following
638 command
639 git filter-repo --path src/main/ --path-rename tools/:scripts/
641 would not result in the tools or scripts directories being present,
644 because the single filter selected only src/main/. It’s likely that you
645 would instead want to run:
646 git filter-repo --path src/main/ --path tools/ --path-rename tools/:scripts/
648 If you prefer to filter based solely on basename, use the
651 --use-base-name flag (though this is incompatible with --path-rename).
652 For example, to only include README.md and Makefile files from any
653 directory:
654 git filter-repo --use-base-name --path README.md --path Makefile
656 If you wanted to delete all .DS_Store files in any directory, you could
659 either use:
660 git filter-repo --invert-paths --path '.DS_Store' --use-base-name
662 or
665 git filter-repo --invert-paths --path-glob '*/.DS_Store' --path '.DS_Store'
667 (the --path-glob isn’t sufficient by itself as it might miss a toplevel
670 .DS_Store file; further while something like --path-glob '*.DS_Store'
671 would workaround that problem it would also grab files named
672 foo.DS_Store or bar/baz.DS_Store)
673 Finally, see also the --filename-callback from the section called
675 “CALLBACKS”.
676 Filtering based on many paths
678 If you have a long list of files, directories, globs, or regular
679 expressions to filter on, you can stick them in a file and use
680 --paths-from-file; for example, with a file named stuff-i-want.txt with
681 contents of
682 # Blank lines and comment lines are ignored.
684 # Examples similar to --path:
685 README.md
686 guides/
687 tools/releases
688 # An example that is like --path-glob:
690 glob:*.py
691 # An example that is like --path-regex:
693 regex:^.*/.*/[0-9]{4}-[0-9]{2}-[0-9]{2}.txt$
694 # An example of renaming a path
696 tools/==>scripts/
697 # An example of using a regex to rename a path
699 regex:(.*)/([^/]*)/([^/]*)\.text$==>\2/\1/\3.txt
700 then you could run
703 git filter-repo --paths-from-file stuff-i-want.txt
705 to get a repo containing only the toplevel README.md file, the guides/
708 and tools/releases/ directories, all python files, files whose name was
709 of the form YYYY.MM-DD.txt at least two subdirectories deep, and would
710 rename tools/ to scripts/ and rename files like foo/bar/baz.text to
711 bar/foo/baz.txt. Note the special line prefixes of glob: and regex: and
712 the special string ==> denoting renames.
713 Sometimes you have a way of easily generating all the files you want.
715 For example, if you know that none of the currently tracked files have
716 any newlines or special characters in them (see core.quotePath from git
717 config --help) so that git ls-files would print all files literally one
718 per line, and you knew that you wanted to keep only the files that are
719 currently tracked (thus deleting from all commits in history any files
720 that only appear on other branches or that only appear in older
721 commits), then you could use a pair of commands such as
722 git ls-files >../paths-i-want.txt
724 git filter-repo --paths-from-file ../paths-i-want.txt
725 Similarly, you could use --paths-from-file to delete many files. For
728 example, you could run git filter-repo --analyze to get reports, look
729 in one such as .git/filter-repo/analysis/path-deleted-sizes.txt and
730 copy all the filenames into a file such as
731 /tmp/files-i-dont-want-anymore.txt and then run
732 git filter-repo --invert-paths --paths-from-file /tmp/files-i-dont-want-anymore.txt
734 to delete them all.
737 Directory based shortcuts
739 Let’s say you had a directory structure like the following:
740 module/
742 foo.c
743 bar.c
744 otherDir/
745 blah.config
746 stuff.txt
747 zebra.jpg
748 If you wanted just the module/ directory and you wanted it to become
750 the new root so that your new directory structure looked like
751 foo.c
753 bar.c
754 then you could run:
756 git filter-repo --subdirectory-filter module/
758 If you wanted all the files from the original repo, but wanted to move
761 everything under a subdirectory named my-module/, so that your new
762 directory structure looked like
763 my-module/
765 module/
766 foo.c
767 bar.c
768 otherDir/
769 blah.config
770 stuff.txt
771 zebra.jpg
772 then you would instead run run
774 git filter-repo --to-subdirectory-filter my-module/
776 Content based filtering
779 If you want to filter out all files bigger than a certain size, you can
780 use --strip-blobs-bigger-than with some size (K, M, and G suffixes are
781 recognized), e.g.:
782 git filter-repo --strip-blobs-bigger-than 10M
784 If you want to strip out all files with specified git object ids
787 (hashes), list the hashes in a file and run
788 git filter-repo --strip-blobs-with-ids FILE_WITH_GIT_BLOB_IDS
790 If you want to modify file contents, you can do so based on a list of
793 expressions in a file, one per line. For example, with a file named
794 expressions.txt containing
795 p455w0rd
797 foo==>bar
798 glob:*666*==>
799 regex:\bdriver\b==>pilot
800 literal:MM/DD/YYYY==>YYYY-MM-DD
801 regex:([0-9]{2})/([0-9]{2})/([0-9]{4})==>\3-\1-\2
802 then running
805 git filter-repo --replace-text expressions.txt
807 will go through and replace p455w0rd with ***REMOVED***, foo with bar,
810 any line containing 666 with a blank line, the word driver with pilot
811 (but not if it has letters before or after; e.g. drivers will be
812 unmodified), replace the exact text MM/DD/YYYY with YYYY-MM-DD and
813 replace date strings of the form MM/DD/YYYY with ones of the form
814 YYYY-MM-DD. In the expressions file, there are a few things to note:
815 · Every line has a replacement, given by whatever is on the right of
817 ==>. If ==> does not appear on the line, the default replacement is
818 ***REMOVED***.
819 · Lines can start with literal:, glob:, or regex: to specify whether
821 to do literal string matches, globs (see
822 https://docs.python.org/3/library/fnmatch.html), or regular
823 expressions (see
824 https://docs.python.org/3/library/re.html#regular-expression-syntax).
825 If none of these are specified, literal: is assumed.
826 · If multiple matches are found, all are replaced.
828 · globs and regexes are applied to the entire file, but without any
830 special flags turned on. Some folks may be interested in adding
831 (?m) to the regex to turn on MULTILINE mode, so that ^ and $ match
832 the beginning and ends of lines rather than the beginning and end
833 of file. See https://docs.python.org/3/library/re.html for details.
834 See also the --blob-callback from the section called “CALLBACKS”.
836 Refname based filtering
838 To rename tags, use --tag-rename, e.g.:
839 git filter-repo --tag-rename foo:bar
841 This will rename any tags starting with foo to now start with bar.
844 Either side of the colon could be blank, e.g.
845 git filter-repo --tag-rename '':'my-module-'
847 For more general refname modification, see --refname-callback from the
850 section called “CALLBACKS”.
851 User and email based filtering
853 To modify username and emails of commits, you can create a mailmap file
854 in the format accepted by git-shortlog(1). For example, if you have a
855 file named my-mailmap you can run
856 git filter-repo --mailmap my-mailmap
858 and if the current contents of that file are as follows (if the
861 specified mailmap file is version controlled, historical versions of
862 the file are ignored):
863 Name For User <email@addre.ss>
865 <new@ema.il> <old1@ema.il>
866 New Name And <new@ema.il> <old2@ema.il>
867 New Name And <new@ema.il> Old Name And <old3@ema.il>
868 then we can update username and/or emails based on the specified
871 mapping.
872 See also the --name-callback and --email-callback from the section
874 called “CALLBACKS”.
875 Parent rewriting
877 To replace $commit_A with $commit_B (e.g. make all commits which had
878 $commit_A as a parent instead have $commit_B for that parent), and
879 rewrite history to make it permanent:
880 git replace $commit_A $commit_B
882 git filter-repo --force
883 To create a new commit with the same contents as $commit_A except with
886 different parent(s) and then replace $commit_A with the new commit, and
887 rewrite history to make it permanent:
888 git replace --graft $commit_A $new_parent_or_parents
890 git filter-repo --force
891 The reason to specify --force is two-fold: filter-repo will error out
894 if no arguments are specified, and the new graft commit would otherwise
895 trigger the not-a-fresh-clone check.
896 Partial history rewrites
898 To rewrite the history on just one branch (which may cause it to no
899 longer share any common history with other branches), use --refs. For
900 example, to remove a file named extraneous.txt from the master branch:
901 git filter-repo --invert-paths --path extraneous.txt --refs master
903 To rewrite just some recent commits:
906 git filter-repo --invert-paths --path extraneous.txt --refs master~3..master
908
911 For flexibility, filter-repo allows you to specify functions on the
912 command line to further filter all changes. Please note that there are
913 some API compatibility caveats associated with these callbacks that you
914 should be aware of before using them; see the "API BACKWARD
915 COMPATIBILITY CAVEAT" comment near the top of git-filter-repo source
916 code.
917 All callback functions are of the same general format. For a command
919 line argument like
920 --foo-callback 'BODY'
922 the following code will be compiled and called:
925 def foo_callback(foo):
927 BODY
928 Thus, you just need to make sure your BODY modifies and returns foo
931 appropriately. One important thing to note for all callbacks is that
932 filter-repo uses bytestrings (see
933 https://docs.python.org/3/library/stdtypes.html#bytes) everywhere
934 instead of strings.
935 There are four callbacks that allow you to operate directly on raw
937 objects that contain data that’s easy to write in fast-import(1)
938 format:
939 --blob-callback
941 --commit-callback
942 --tag-callback
943 --reset-callback
944 We’ll come back to these later because it is often the case that the
947 other callbacks are more convenient. The other callbacks operate on a
948 small piece of the raw objects or operate on pieces across multiple
949 types of raw object (e.g. author names and committer names and tagger
950 names across commits and tags, or refnames across commits, tags, and
951 resets, or messages across commits and tags). The convenience callbacks
952 are:
953 --filename-callback
955 --message-callback
956 --name-callback
957 --email-callback
958 --refname-callback
959 in each you are expected to simply return a new value based on the one
962 passed in. For example,
963 git-filter-repo --name-callback 'return name.replace(b"Wiliam", b"William")'
965 would result in the following function being called:
968 def name_callback(name):
970 return name.replace(b"Wiliam", b"William")
971 The email callback is quite similar:
974 git-filter-repo --email-callback 'return email.replace(b".cm", b".com")'
976 The refname callback is also similar, but note that the refname passed
979 in and returned are expected to be fully qualified (e.g.
980 b"refs/heads/master" instead of just b"master" and b"refs/tags/v1.0.7"
981 instead of b"1.0.7"):
982 git-filter-repo --refname-callback '
984 # Change e.g. refs/heads/master to refs/heads/prefix-master
985 rdir,rpath = os.path.split(refname)
986 return rdir + b"/prefix-" + rpath'
987 The message callback is quite similar to the previous three callbacks,
990 though it operates on a bytestring that is likely more than one line:
991 git-filter-repo --message-callback '
993 if b"Signed-off-by:" not in message:
994 message += b"\nSigned-off-by: Me My <self@and.eye>"
995 return re.sub(b"[Ee]-?[Mm][Aa][Ii][Ll]", b"email", message)'
996 The filename callback is slightly more interesting. Returning None
999 means the file should be removed from all commits, returning the
1000 filename unmodified marks the file to be kept, and returning a
1001 different name means the file should be renamed. An example:
1002 git-filter-repo --filename-callback '
1004 if b"/src/" in filename:
1005 # Remove all files with a directory named "src" in their path
1006 # (except when "src" appears at the toplevel).
1007 return None
1008 elif filename.startswith(b"tools/"):
1009 # Rename tools/ -> scripts/misc/
1010 return b"scripts/misc/" + filename[6:]
1011 else:
1012 # Keep the filename and do not rename it
1013 return filename
1014 '
1015 In contrast, the blob, reset, tag, and commit callbacks are not
1018 expected to return a value, but are instead expected to modify the
1019 object passed in. Major fields for these objects are (subject to API
1020 backward compatibility caveats mentioned previously):
1021 · Blob: original_id (original hash) and data
1023 · Reset: ref (name of reference) and from_ref (hash or integer mark)
1025 · Tag: ref, from_ref, original_id, tagger_name, tagger_email,
1027 tagger_date, message
1028 · Commit: branch, original_id, author_name, author_email,
1030 author_date, committer_name, committer_email, committer_date,
1031 message, file_changes (list of FileChange objects, each containing
1032 a type, filename, mode, and blob_id), parents (list of hashes or
1033 integer marks)
1034 An example of each:
1036 git filter-repo --blob-callback '
1038 if len(blob.data) > 25:
1039 # Mark this blob for removal from all commits
1040 blob.skip()
1041 else:
1042 blob.data = blob.data.replace(b"Hello", b"Goodbye")
1043 '
1044 git filter-repo --reset-callback 'reset.ref = reset.ref.replace(b"master", b"dev")'
1048 git filter-repo --tag-callback '
1052 if tag.tagger_name == b"Jim Williams":
1053 # Omit this tag
1054 tag.skip()
1055 else:
1056 tag.message = tag.message + b"\n\nTag of %s by %s on %s" % (tag.ref, tag.tagger_email, tag.tagger_date)'
1057 git filter-repo --commit-callback '
1061 # Remove executable files with three 6s in their name (including
1062 # from leading directories).
1063 # Also, undo deletion of sources/foo/bar.txt (change types are
1064 # either b"D" (deletion) or b"M" (add or modify); renames are
1065 # handled by deleting the old file and adding a new one)
1066 commit.file_changes = [
1067 change for change in commit.file_changes
1068 if not (change.mode == b"100755" and
1069 change.filename.count(b"6") == 3) and
1070 not (change.type == b"D" and
1071 change.filename == b"sources/foo/bar.txt")]
1072 # Mark all .sh files as executable; modes in git are always one of
1073 # 100644 (normal file), 100755 (executable), 120000 (symlink), or
1074 # 160000 (submodule)
1075 for change in commit.file_changes:
1076 if change.filename.endswith(b".sh"):
1077 change.mode = b"100755"
1078 '
1079
1082 You probably don’t need to read this section unless you are just very
1083 curious or you are trying to do a very complex history rewrite.
1084 How filter-repo works
1086 Roughly, filter-repo works by running
1087 git fast-export <options> | filter | git fast-import <options>
1089 where filter-repo not only launches the whole pipeline but also serves
1092 as the filter in the middle. However, filter-repo does a few additional
1093 things on top in order to make it into a well-rounded filtering tool. A
1094 sequence that more accurately reflects what filter-repo runs is:
1095 1. Verify we’re in a fresh clone
1097 2. git fetch -u . refs/remotes/origin/*:refs/heads/*
1099 3. git remote rm origin
1101 4. git fast-export --show-original-ids --reference-excluded-parents
1103 --fake-missing-tagger --signed-tags=strip
1104 --tag-of-filtered-object=rewrite --use-done-feature --no-data
1105 --reencode=yes --mark-tags --all | filter | git -c
1106 core.ignorecase=false fast-import --date-format=raw-permissive
1107 --force --quiet
1108 5. git update-ref --no-deref --stdin, fed with a list of refs to nuke,
1110 and a list of replace refs to delete, create, or update.
1111 6. git reset --hard
1113 7. git reflog expire --expire=now --all
1115 8. git gc --prune=now
1117 Some notes or exceptions on each of the above:
1119 1. If we’re not in a fresh clone, users will not be able to recover if
1121 they used the wrong command or ran in the wrong repo. (Though
1122 --force overrides this check, and it’s also off if you’ve already
1123 ran filter-repo once in this repo.)
1124 2. Technically, we actually use a git update-ref command fed with a
1126 lot of input due to the fact that users can use --force when local
1127 branches might not match remote branches. But this fetch command
1128 catches the intent rather succinctly.
1129 3. We don’t want users accidentally pushing back to the original repo,
1131 as discussed in the section called “DISCUSSION”. It also reminds
1132 users that since history has been rewritten, this repo is no longer
1133 compatible with the original. Finally, another minor benefit is
1134 this allows users to push with the --mirror option to their new
1135 home without accidentally sending remote tracking branches.
1136 4. Some of these flags are always used but others are actually
1138 conditional. For example, filter-repo’s --replace-text and
1139 --blob-callback options need to work on blobs so --no-data cannot
1140 be passed to fast-export. But when we don’t need to work on blobs,
1141 passing --no-data speeds things up. Also, other flags may change
1142 the structure of the pipeline as well (e.g. --dry-run and --debug)
1143 5. We use this step to write replace refs for accessing the newly
1145 written commit hashes using their previous names. Also, if refs
1146 were renamed by various steps, we need to delete the old refnames
1147 in order to avoid mixing old and new history.
1148 6. Users also have old versions of files in their working tree and
1150 index; we want those cleaned up to match the rewritten history as
1151 well. Note that this step is skipped in bare repos.
1152 7. Reflogs will hold on to old history, so we need to expire them.
1154 8. We need to gc to avoid mixing new and old history. Also, it shrinks
1156 the repository for users, so they don’t have to do extra work.
1157 (Odds are that they’ve only rewritten trees and commits and maybe a
1158 few blobs, so --aggressive isn’t needed and would be too slow.)
1159 Information about these steps is printed out when --debug is passed to
1161 filter-repo. When doing a --partial history rewrite, steps 2, 3, 7, and
1162 8 are unconditionally skipped, step 5 is skipped if --replace-refs is
1163 update-no-add, and just the nuke-unused-refs portion of step 5 is
1164 skipped if --replace-refs is something else.
1165 Limitations
1167 Inherited limitations
1168 Since git filter-repo calls fast-export and fast-import to do a lot
1169 of the heavy lifting, it inherits limitations from those systems:
1170 · extended commit headers, if any, are stripped
1172 · commits get rewritten meaning they will have new hashes;
1174 therefore, signatures on commits and tags cannot continue to
1175 work and instead are just removed (thus signed tags become
1176 annotated tags)
1177 · tags of commits are supported. Prior to git-2.24.0, tags of
1179 blobs and tags of tags are not supported (fast-export would die
1180 on such tags). tags of trees are not supported in any git
1181 version (since fast-export ignores tags of trees with a warning
1182 and fast-import provides no way to import them).
1183 · annotated and signed tags outside of the refs/tags/ namespace
1185 are not supported (their location will be mangled in weird
1186 ways)
1187 · fast-import will die on various forms of invalid input, such as
1189 a timezone with more than four digits
1190 · fast-export cannot reencode commit messages into UTF-8 if the
1192 commit message is not valid in its specified encoding (in such
1193 cases, it’ll leave the commit message and the encoding header
1194 alone).
1195 · commits without an author will be given one matching the
1197 committer
1198 · tags without a tagger will be given a fake tagger
1200 · references that include commit cycles in their history (which
1202 can be created with git-replace(1)) will not be flagged to the
1203 user as an error but will be silently deleted by fast-export as
1204 though the branch or tag contained no interesting files
1205 There are also some limitations due to the design of these systems:
1207 · Trying to insert additional files into the stream can be
1209 tricky; since fast-export only lists file changes in a merge
1210 relative to its first parent, if you insert additional files
1211 into a commit that is in the second (or third or fourth) parent
1212 history of a merge, then you also need to add it to the merge
1213 manually. (Similarly, if you change which parent is the first
1214 parent in a merge commit, you need to manually update the list
1215 of file changes to be relative to the new first parent.)
1216 · fast-export and fast-import work with exact file contents, not
1218 patches. (e.g. "Whatever the current contents of this file,
1219 update them to now have these contents") Because of this,
1220 removing the changes made in a single commit or inserting
1221 additional changes to a file in some commit and expecting them
1222 to propagate forward is not something that can be done with
1223 these tools. Use git-rebase(1) for that.
1224 Intrinsic limitations
1226 Some types of filtering have limitations that would affect any tool
1227 attempting to perform them; the most any tool can do is attempt to
1228 notify the user when it detects an issue:
1229 · When rewriting commit hashes in commit messages, there are a
1231 variety of cases when the hash will not be updated (whenever
1232 this happens, a note is written to
1233 .git/filter-repo/suboptimal-issues):
1234 · if a commit hash does not correspond to a commit in the old
1236 repo
1237 · if a commit hash corresponds to a commit that gets pruned
1239 · if an abbreviated hash is not unique
1241 · Pruning of empty commits can cause a merge commit to lose an
1243 entire ancestry line and become a non-merge. If the merge
1244 commit had no changes then it can be pruned too, but if it
1245 still has changes it needs to be kept. This might cause minor
1246 confusion since the commit will likely have a commit message
1247 that makes it sound like a merge commit even though it’s not.
1248 (Whenever a merge commit becomes a non-merge commit, a note is
1249 written to .git/filter-repo/suboptimal-issues)
1250 Issues specific to filter-repo
1252 · Multiple repositories in the wild have been observed which use
1253 a bogus timezone (+051800); google will find you some reports.
1254 The intended timezone wasn’t clear or wasn’t always the same.
1255 Replace with a different bogus timezone that fast-import will
1256 accept (+0261).
1257 · --path-rename can result in pathname collisions; to avoid
1259 excessive memory requirements of tracking which files are in
1260 all commits or looking up what files exist with either every
1261 commit or every usage of --path-rename, we just tell the user
1262 that they might clobber other changes if they aren’t careful.
1263 We can check if the clobbering comes from another --path-rename
1264 without much overhead. (Perhaps in the future it’s worth adding
1265 a slow mode to --path-rename that will do the more exhaustive
1266 checks?)
1267 · There is no mechanism for directly controlling which flags are
1269 passed to fast-export (or fast-import); only pre-defined flags
1270 can be turned on or off as a side-effect of other options.
1271 Direct control would make little sense because some options
1272 like --full-tree would require additional code in filter-repo
1273 (to parse new directives), and others such as -M or -C would
1274 break assumptions used in other places of filter-repo.
1275 · Partial-repo filtering, while supported, runs counter to
1277 filter-repo’s "avoid mixing old and new history" design. This
1278 support has required improvements to core git as well (e.g. it
1279 depends upon the --reference-excluded-parents option to
1280 fast-export that was added specifically for this usage within
1281 filter-repo). The --partial and --refs options will continue to
1282 be supported since there are people with usecases for them;
1283 however, I am concerned that this inconsistency about mixing
1284 old and new history seems likely to lead to user mistakes. For
1285 now, I just hope that long explanations of caveats in the
1286 documentation of these options suffice to curtail any such
1287 problems.
1288 Comments on reversibility
1290 Some people are interested in reversibility of of a rewrite; e.g.
1291 rewrite history, possibly add some commits, then unrewrite and get
1292 the original history back plus a few new "unrewritten" commits.
1293 Obviously this is impossible if your rewrite involves throwing away
1294 information (e.g. filtering out files or replacing several
1295 different strings with ***REMOVED***), but may be possible with
1296 some rewrites. filter-repo is likely to be a poor fit for this type
1297 of workflow for a few reasons:
1298 · most of the limitations inherited from fast-export and
1300 fast-import are of a type that cause reversibility issues
1301 · grafts and replace refs, if present, are used in the rewrite
1303 and made permanent
1304 · rewriting of commit hashes will probably be reversible, but it
1306 is possible for rewritten abbreviated hashes to not be unique
1307 even if the original abbreviated hashes were.
1308 · filter-repo defaults to several forms of unreversible rewriting
1310 that you may need to turn off (e.g. the last two bullet points
1311 above or reencoding commit messages into UTF-8); it’s possible
1312 that additional forms of unreversible rewrites will be added in
1313 the future.
1314 · I assume that people use filter-repo for one-shot conversions,
1316 not ongoing data transfers. I explicitly reserve the right to
1317 change any API in filter-repo based on this presumption (and a
1318 comment to this effect is found in multiple places in the code
1319 and examples). You have been warned.
1320
1322 git-rebase(1), git-filter-branch(1)
1323
1325 Part of the git(1) suite
1326
1328 1. GitLab’s docs on reducing repository size
1329 https://docs.gitlab.com/ee/user/project/repository/reducing_the_repo_size_using_git.html
1330Git 2.29.0.dirty 10/19/2020 GIT-FILTER-REPO(1)