1GIT-FAST-IMPORT(1)                Git Manual                GIT-FAST-IMPORT(1)
2
3
4

NAME

6       git-fast-import - Backend for fast Git data importers
7

SYNOPSIS

9       frontend | git fast-import [<options>]
10

DESCRIPTION

12       This program is usually not what the end user wants to run directly.
13       Most end users want to use one of the existing frontend programs, which
14       parses a specific type of foreign source and feeds the contents stored
15       there to git fast-import.
16
17       fast-import reads a mixed command/data stream from standard input and
18       writes one or more packfiles directly into the current repository. When
19       EOF is received on standard input, fast import writes out updated
20       branch and tag refs, fully updating the current repository with the
21       newly imported data.
22
23       The fast-import backend itself can import into an empty repository (one
24       that has already been initialized by git init) or incrementally update
25       an existing populated repository. Whether or not incremental imports
26       are supported from a particular foreign source depends on the frontend
27       program in use.
28

OPTIONS

30       --force
31           Force updating modified existing branches, even if doing so would
32           cause commits to be lost (as the new commit does not contain the
33           old commit).
34
35       --quiet
36           Disable the output shown by --stats, making fast-import usually be
37           silent when it is successful. However, if the import stream has
38           directives intended to show user output (e.g.  progress
39           directives), the corresponding messages will still be shown.
40
41       --stats
42           Display some basic statistics about the objects fast-import has
43           created, the packfiles they were stored into, and the memory used
44           by fast-import during this run. Showing this output is currently
45           the default, but can be disabled with --quiet.
46
47       --allow-unsafe-features
48           Many command-line options can be provided as part of the
49           fast-import stream itself by using the feature or option commands.
50           However, some of these options are unsafe (e.g., allowing
51           fast-import to access the filesystem outside of the repository).
52           These options are disabled by default, but can be allowed by
53           providing this option on the command line. This currently impacts
54           only the export-marks, import-marks, and import-marks-if-exists
55           feature commands.
56
57               Only enable this option if you trust the program generating the
58               fast-import stream! This option is enabled automatically for
59               remote-helpers that use the `import` capability, as they are
60               already trusted to run their own code.
61
62   Options for Frontends
63       --cat-blob-fd=<fd>
64           Write responses to get-mark, cat-blob, and ls queries to the file
65           descriptor <fd> instead of stdout. Allows progress output intended
66           for the end-user to be separated from other output.
67
68       --date-format=<fmt>
69           Specify the type of dates the frontend will supply to fast-import
70           within author, committer and tagger commands. See “Date Formats”
71           below for details about which formats are supported, and their
72           syntax.
73
74       --done
75           Terminate with error if there is no done command at the end of the
76           stream. This option might be useful for detecting errors that cause
77           the frontend to terminate before it has started to write a stream.
78
79   Locations of Marks Files
80       --export-marks=<file>
81           Dumps the internal marks table to <file> when complete. Marks are
82           written one per line as :markid SHA-1. Frontends can use this file
83           to validate imports after they have been completed, or to save the
84           marks table across incremental runs. As <file> is only opened and
85           truncated at checkpoint (or completion) the same path can also be
86           safely given to --import-marks.
87
88       --import-marks=<file>
89           Before processing any input, load the marks specified in <file>.
90           The input file must exist, must be readable, and must use the same
91           format as produced by --export-marks. Multiple options may be
92           supplied to import more than one set of marks. If a mark is defined
93           to different values, the last file wins.
94
95       --import-marks-if-exists=<file>
96           Like --import-marks but instead of erroring out, silently skips the
97           file if it does not exist.
98
99       --[no-]relative-marks
100           After specifying --relative-marks the paths specified with
101           --import-marks= and --export-marks= are relative to an internal
102           directory in the current repository. In git-fast-import this means
103           that the paths are relative to the .git/info/fast-import directory.
104           However, other importers may use a different location.
105
106           Relative and non-relative marks may be combined by interweaving
107           --(no-)-relative-marks with the --(import|export)-marks= options.
108
109   Submodule Rewriting
110       --rewrite-submodules-from=<name>:<file>,
111       --rewrite-submodules-to=<name>:<file>
112           Rewrite the object IDs for the submodule specified by <name> from
113           the values used in the from <file> to those used in the to <file>.
114           The from marks should have been created by git fast-export, and the
115           to marks should have been created by git fast-import when importing
116           that same submodule.
117
118           <name> may be any arbitrary string not containing a colon
119           character, but the same value must be used with both options when
120           specifying corresponding marks. Multiple submodules may be
121           specified with different values for <name>. It is an error not to
122           use these options in corresponding pairs.
123
124           These options are primarily useful when converting a repository
125           from one hash algorithm to another; without them, fast-import will
126           fail if it encounters a submodule because it has no way of writing
127           the object ID into the new hash algorithm.
128
129   Performance and Compression Tuning
130       --active-branches=<n>
131           Maximum number of branches to maintain active at once. See “Memory
132           Utilization” below for details. Default is 5.
133
134       --big-file-threshold=<n>
135           Maximum size of a blob that fast-import will attempt to create a
136           delta for, expressed in bytes. The default is 512m (512 MiB). Some
137           importers may wish to lower this on systems with constrained
138           memory.
139
140       --depth=<n>
141           Maximum delta depth, for blob and tree deltification. Default is
142           50.
143
144       --export-pack-edges=<file>
145           After creating a packfile, print a line of data to <file> listing
146           the filename of the packfile and the last commit on each branch
147           that was written to that packfile. This information may be useful
148           after importing projects whose total object set exceeds the 4 GiB
149           packfile limit, as these commits can be used as edge points during
150           calls to git pack-objects.
151
152       --max-pack-size=<n>
153           Maximum size of each output packfile. The default is unlimited.
154
155       fastimport.unpackLimit
156           See git-config(1)
157

PERFORMANCE

159       The design of fast-import allows it to import large projects in a
160       minimum amount of memory usage and processing time. Assuming the
161       frontend is able to keep up with fast-import and feed it a constant
162       stream of data, import times for projects holding 10+ years of history
163       and containing 100,000+ individual commits are generally completed in
164       just 1-2 hours on quite modest (~$2,000 USD) hardware.
165
166       Most bottlenecks appear to be in foreign source data access (the source
167       just cannot extract revisions fast enough) or disk IO (fast-import
168       writes as fast as the disk will take the data). Imports will run faster
169       if the source data is stored on a different drive than the destination
170       Git repository (due to less IO contention).
171

DEVELOPMENT COST

173       A typical frontend for fast-import tends to weigh in at approximately
174       200 lines of Perl/Python/Ruby code. Most developers have been able to
175       create working importers in just a couple of hours, even though it is
176       their first exposure to fast-import, and sometimes even to Git. This is
177       an ideal situation, given that most conversion tools are throw-away
178       (use once, and never look back).
179

PARALLEL OPERATION

181       Like git push or git fetch, imports handled by fast-import are safe to
182       run alongside parallel git repack -a -d or git gc invocations, or any
183       other Git operation (including git prune, as loose objects are never
184       used by fast-import).
185
186       fast-import does not lock the branch or tag refs it is actively
187       importing. After the import, during its ref update phase, fast-import
188       tests each existing branch ref to verify the update will be a
189       fast-forward update (the commit stored in the ref is contained in the
190       new history of the commit to be written). If the update is not a
191       fast-forward update, fast-import will skip updating that ref and
192       instead prints a warning message. fast-import will always attempt to
193       update all branch refs, and does not stop on the first failure.
194
195       Branch updates can be forced with --force, but it’s recommended that
196       this only be used on an otherwise quiet repository. Using --force is
197       not necessary for an initial import into an empty repository.
198

TECHNICAL DISCUSSION

200       fast-import tracks a set of branches in memory. Any branch can be
201       created or modified at any point during the import process by sending a
202       commit command on the input stream. This design allows a frontend
203       program to process an unlimited number of branches simultaneously,
204       generating commits in the order they are available from the source
205       data. It also simplifies the frontend programs considerably.
206
207       fast-import does not use or alter the current working directory, or any
208       file within it. (It does however update the current Git repository, as
209       referenced by GIT_DIR.) Therefore an import frontend may use the
210       working directory for its own purposes, such as extracting file
211       revisions from the foreign source. This ignorance of the working
212       directory also allows fast-import to run very quickly, as it does not
213       need to perform any costly file update operations when switching
214       between branches.
215

INPUT FORMAT

217       With the exception of raw file data (which Git does not interpret) the
218       fast-import input format is text (ASCII) based. This text based format
219       simplifies development and debugging of frontend programs, especially
220       when a higher level language such as Perl, Python or Ruby is being
221       used.
222
223       fast-import is very strict about its input. Where we say SP below we
224       mean exactly one space. Likewise LF means one (and only one) linefeed
225       and HT one (and only one) horizontal tab. Supplying additional
226       whitespace characters will cause unexpected results, such as branch
227       names or file names with leading or trailing spaces in their name, or
228       early termination of fast-import when it encounters unexpected input.
229
230   Stream Comments
231       To aid in debugging frontends fast-import ignores any line that begins
232       with # (ASCII pound/hash) up to and including the line ending LF. A
233       comment line may contain any sequence of bytes that does not contain an
234       LF and therefore may be used to include any detailed debugging
235       information that might be specific to the frontend and useful when
236       inspecting a fast-import data stream.
237
238   Date Formats
239       The following date formats are supported. A frontend should select the
240       format it will use for this import by passing the format name in the
241       --date-format=<fmt> command-line option.
242
243       raw
244           This is the Git native format and is <time> SP <offutc>. It is also
245           fast-import’s default format, if --date-format was not specified.
246
247           The time of the event is specified by <time> as the number of
248           seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is
249           written as an ASCII decimal integer.
250
251           The local offset is specified by <offutc> as a positive or negative
252           offset from UTC. For example EST (which is 5 hours behind UTC)
253           would be expressed in <tz> by “-0500” while UTC is “+0000”. The
254           local offset does not affect <time>; it is used only as an
255           advisement to help formatting routines display the timestamp.
256
257           If the local offset is not available in the source material, use
258           “+0000”, or the most common local offset. For example many
259           organizations have a CVS repository which has only ever been
260           accessed by users who are located in the same location and time
261           zone. In this case a reasonable offset from UTC could be assumed.
262
263           Unlike the rfc2822 format, this format is very strict. Any
264           variation in formatting will cause fast-import to reject the value,
265           and some sanity checks on the numeric values may also be performed.
266
267       raw-permissive
268           This is the same as raw except that no sanity checks on the numeric
269           epoch and local offset are performed. This can be useful when
270           trying to filter or import an existing history with e.g. bogus
271           timezone values.
272
273       rfc2822
274           This is the standard email format as described by RFC 2822.
275
276           An example value is “Tue Feb 6 11:22:18 2007 -0500”. The Git parser
277           is accurate, but a little on the lenient side. It is the same
278           parser used by git am when applying patches received from email.
279
280           Some malformed strings may be accepted as valid dates. In some of
281           these cases Git will still be able to obtain the correct date from
282           the malformed string. There are also some types of malformed
283           strings which Git will parse wrong, and yet consider valid.
284           Seriously malformed strings will be rejected.
285
286           Unlike the raw format above, the time zone/UTC offset information
287           contained in an RFC 2822 date string is used to adjust the date
288           value to UTC prior to storage. Therefore it is important that this
289           information be as accurate as possible.
290
291           If the source material uses RFC 2822 style dates, the frontend
292           should let fast-import handle the parsing and conversion (rather
293           than attempting to do it itself) as the Git parser has been well
294           tested in the wild.
295
296           Frontends should prefer the raw format if the source material
297           already uses UNIX-epoch format, can be coaxed to give dates in that
298           format, or its format is easily convertible to it, as there is no
299           ambiguity in parsing.
300
301       now
302           Always use the current time and time zone. The literal now must
303           always be supplied for <when>.
304
305           This is a toy format. The current time and time zone of this system
306           is always copied into the identity string at the time it is being
307           created by fast-import. There is no way to specify a different time
308           or time zone.
309
310           This particular format is supplied as it’s short to implement and
311           may be useful to a process that wants to create a new commit right
312           now, without needing to use a working directory or git
313           update-index.
314
315           If separate author and committer commands are used in a commit the
316           timestamps may not match, as the system clock will be polled twice
317           (once for each command). The only way to ensure that both author
318           and committer identity information has the same timestamp is to
319           omit author (thus copying from committer) or to use a date format
320           other than now.
321
322   Commands
323       fast-import accepts several commands to update the current repository
324       and control the current import process. More detailed discussion (with
325       examples) of each command follows later.
326
327       commit
328           Creates a new branch or updates an existing branch by creating a
329           new commit and updating the branch to point at the newly created
330           commit.
331
332       tag
333           Creates an annotated tag object from an existing commit or branch.
334           Lightweight tags are not supported by this command, as they are not
335           recommended for recording meaningful points in time.
336
337       reset
338           Reset an existing branch (or a new branch) to a specific revision.
339           This command must be used to change a branch to a specific revision
340           without making a commit on it.
341
342       blob
343           Convert raw file data into a blob, for future use in a commit
344           command. This command is optional and is not needed to perform an
345           import.
346
347       alias
348           Record that a mark refers to a given object without first creating
349           any new object. Using --import-marks and referring to missing marks
350           will cause fast-import to fail, so aliases can provide a way to set
351           otherwise pruned commits to a valid value (e.g. the nearest
352           non-pruned ancestor).
353
354       checkpoint
355           Forces fast-import to close the current packfile, generate its
356           unique SHA-1 checksum and index, and start a new packfile. This
357           command is optional and is not needed to perform an import.
358
359       progress
360           Causes fast-import to echo the entire line to its own standard
361           output. This command is optional and is not needed to perform an
362           import.
363
364       done
365           Marks the end of the stream. This command is optional unless the
366           done feature was requested using the --done command-line option or
367           feature done command.
368
369       get-mark
370           Causes fast-import to print the SHA-1 corresponding to a mark to
371           the file descriptor set with --cat-blob-fd, or stdout if
372           unspecified.
373
374       cat-blob
375           Causes fast-import to print a blob in cat-file --batch format to
376           the file descriptor set with --cat-blob-fd or stdout if
377           unspecified.
378
379       ls
380           Causes fast-import to print a line describing a directory entry in
381           ls-tree format to the file descriptor set with --cat-blob-fd or
382           stdout if unspecified.
383
384       feature
385           Enable the specified feature. This requires that fast-import
386           supports the specified feature, and aborts if it does not.
387
388       option
389           Specify any of the options listed under OPTIONS that do not change
390           stream semantic to suit the frontend’s needs. This command is
391           optional and is not needed to perform an import.
392
393   commit
394       Create or update a branch with a new commit, recording one logical
395       change to the project.
396
397                   'commit' SP <ref> LF
398                   mark?
399                   original-oid?
400                   ('author' (SP <name>)? SP LT <email> GT SP <when> LF)?
401                   'committer' (SP <name>)? SP LT <email> GT SP <when> LF
402                   ('encoding' SP <encoding>)?
403                   data
404                   ('from' SP <commit-ish> LF)?
405                   ('merge' SP <commit-ish> LF)*
406                   (filemodify | filedelete | filecopy | filerename | filedeleteall | notemodify)*
407                   LF?
408
409       where <ref> is the name of the branch to make the commit on. Typically
410       branch names are prefixed with refs/heads/ in Git, so importing the CVS
411       branch symbol RELENG-1_0 would use refs/heads/RELENG-1_0 for the value
412       of <ref>. The value of <ref> must be a valid refname in Git. As LF is
413       not valid in a Git refname, no quoting or escaping syntax is supported
414       here.
415
416       A mark command may optionally appear, requesting fast-import to save a
417       reference to the newly created commit for future use by the frontend
418       (see below for format). It is very common for frontends to mark every
419       commit they create, thereby allowing future branch creation from any
420       imported commit.
421
422       The data command following committer must supply the commit message
423       (see below for data command syntax). To import an empty commit message
424       use a 0 length data. Commit messages are free-form and are not
425       interpreted by Git. Currently they must be encoded in UTF-8, as
426       fast-import does not permit other encodings to be specified.
427
428       Zero or more filemodify, filedelete, filecopy, filerename,
429       filedeleteall and notemodify commands may be included to update the
430       contents of the branch prior to creating the commit. These commands may
431       be supplied in any order. However it is recommended that a
432       filedeleteall command precede all filemodify, filecopy, filerename and
433       notemodify commands in the same commit, as filedeleteall wipes the
434       branch clean (see below).
435
436       The LF after the command is optional (it used to be required). Note
437       that for reasons of backward compatibility, if the commit ends with a
438       data command (i.e. it has no from, merge, filemodify, filedelete,
439       filecopy, filerename, filedeleteall or notemodify commands) then two LF
440       commands may appear at the end of the command instead of just one.
441
442       author
443           An author command may optionally appear, if the author information
444           might differ from the committer information. If author is omitted
445           then fast-import will automatically use the committer’s information
446           for the author portion of the commit. See below for a description
447           of the fields in author, as they are identical to committer.
448
449       committer
450           The committer command indicates who made this commit, and when they
451           made it.
452
453           Here <name> is the person’s display name (for example “Com M
454           Itter”) and <email> is the person’s email address
455           (“cm@example.com”). LT and GT are the literal less-than (\x3c) and
456           greater-than (\x3e) symbols. These are required to delimit the
457           email address from the other fields in the line. Note that <name>
458           and <email> are free-form and may contain any sequence of bytes,
459           except LT, GT and LF. <name> is typically UTF-8 encoded.
460
461           The time of the change is specified by <when> using the date format
462           that was selected by the --date-format=<fmt> command-line option.
463           See “Date Formats” above for the set of supported formats, and
464           their syntax.
465
466       encoding
467           The optional encoding command indicates the encoding of the commit
468           message. Most commits are UTF-8 and the encoding is omitted, but
469           this allows importing commit messages into git without first
470           reencoding them.
471
472       from
473           The from command is used to specify the commit to initialize this
474           branch from. This revision will be the first ancestor of the new
475           commit. The state of the tree built at this commit will begin with
476           the state at the from commit, and be altered by the content
477           modifications in this commit.
478
479           Omitting the from command in the first commit of a new branch will
480           cause fast-import to create that commit with no ancestor. This
481           tends to be desired only for the initial commit of a project. If
482           the frontend creates all files from scratch when making a new
483           branch, a merge command may be used instead of from to start the
484           commit with an empty tree. Omitting the from command on existing
485           branches is usually desired, as the current commit on that branch
486           is automatically assumed to be the first ancestor of the new
487           commit.
488
489           As LF is not valid in a Git refname or SHA-1 expression, no quoting
490           or escaping syntax is supported within <commit-ish>.
491
492           Here <commit-ish> is any of the following:
493
494           •   The name of an existing branch already in fast-import’s
495               internal branch table. If fast-import doesn’t know the name,
496               it’s treated as a SHA-1 expression.
497
498           •   A mark reference, :<idnum>, where <idnum> is the mark number.
499
500               The reason fast-import uses : to denote a mark reference is
501               this character is not legal in a Git branch name. The leading :
502               makes it easy to distinguish between the mark 42 (:42) and the
503               branch 42 (42 or refs/heads/42), or an abbreviated SHA-1 which
504               happened to consist only of base-10 digits.
505
506               Marks must be declared (via mark) before they can be used.
507
508           •   A complete 40 byte or abbreviated commit SHA-1 in hex.
509
510           •   Any valid Git SHA-1 expression that resolves to a commit. See
511               “SPECIFYING REVISIONS” in gitrevisions(7) for details.
512
513           •   The special null SHA-1 (40 zeros) specifies that the branch is
514               to be removed.
515
516           The special case of restarting an incremental import from the
517           current branch value should be written as:
518
519                       from refs/heads/branch^0
520
521           The ^0 suffix is necessary as fast-import does not permit a branch
522           to start from itself, and the branch is created in memory before
523           the from command is even read from the input. Adding ^0 will force
524           fast-import to resolve the commit through Git’s revision parsing
525           library, rather than its internal branch table, thereby loading in
526           the existing value of the branch.
527
528       merge
529           Includes one additional ancestor commit. The additional ancestry
530           link does not change the way the tree state is built at this
531           commit. If the from command is omitted when creating a new branch,
532           the first merge commit will be the first ancestor of the current
533           commit, and the branch will start out with no files. An unlimited
534           number of merge commands per commit are permitted by fast-import,
535           thereby establishing an n-way merge.
536
537           Here <commit-ish> is any of the commit specification expressions
538           also accepted by from (see above).
539
540       filemodify
541           Included in a commit command to add a new file or change the
542           content of an existing file. This command has two different means
543           of specifying the content of the file.
544
545           External data format
546               The data content for the file was already supplied by a prior
547               blob command. The frontend just needs to connect it.
548
549                           'M' SP <mode> SP <dataref> SP <path> LF
550
551               Here usually <dataref> must be either a mark reference
552               (:<idnum>) set by a prior blob command, or a full 40-byte SHA-1
553               of an existing Git blob object. If <mode> is 040000` then
554               <dataref> must be the full 40-byte SHA-1 of an existing Git
555               tree object or a mark reference set with --import-marks.
556
557           Inline data format
558               The data content for the file has not been supplied yet. The
559               frontend wants to supply it as part of this modify command.
560
561                           'M' SP <mode> SP 'inline' SP <path> LF
562                           data
563
564               See below for a detailed description of the data command.
565
566           In both formats <mode> is the type of file entry, specified in
567           octal. Git only supports the following modes:
568
569100644 or 644: A normal (not-executable) file. The majority of
570               files in most projects use this mode. If in doubt, this is what
571               you want.
572
573100755 or 755: A normal, but executable, file.
574
575120000: A symlink, the content of the file will be the link
576               target.
577
578160000: A gitlink, SHA-1 of the object refers to a commit in
579               another repository. Git links can only be specified by SHA or
580               through a commit mark. They are used to implement submodules.
581
582040000: A subdirectory. Subdirectories can only be specified by
583               SHA or through a tree mark set with --import-marks.
584
585           In both formats <path> is the complete path of the file to be added
586           (if not already existing) or modified (if already existing).
587
588           A <path> string must use UNIX-style directory separators (forward
589           slash /), may contain any byte other than LF, and must not start
590           with double quote (").
591
592           A path can use C-style string quoting; this is accepted in all
593           cases and mandatory if the filename starts with double quote or
594           contains LF. In C-style quoting, the complete name should be
595           surrounded with double quotes, and any LF, backslash, or double
596           quote characters must be escaped by preceding them with a backslash
597           (e.g., "path/with\n, \\ and \" in it").
598
599           The value of <path> must be in canonical form. That is it must not:
600
601           •   contain an empty directory component (e.g.  foo//bar is
602               invalid),
603
604           •   end with a directory separator (e.g.  foo/ is invalid),
605
606           •   start with a directory separator (e.g.  /foo is invalid),
607
608           •   contain the special component .  or ..  (e.g.  foo/./bar and
609               foo/../bar are invalid).
610
611           The root of the tree can be represented by an empty string as
612           <path>.
613
614           It is recommended that <path> always be encoded using UTF-8.
615
616       filedelete
617           Included in a commit command to remove a file or recursively delete
618           an entire directory from the branch. If the file or directory
619           removal makes its parent directory empty, the parent directory will
620           be automatically removed too. This cascades up the tree until the
621           first non-empty directory or the root is reached.
622
623                       'D' SP <path> LF
624
625           here <path> is the complete path of the file or subdirectory to be
626           removed from the branch. See filemodify above for a detailed
627           description of <path>.
628
629       filecopy
630           Recursively copies an existing file or subdirectory to a different
631           location within the branch. The existing file or directory must
632           exist. If the destination exists it will be completely replaced by
633           the content copied from the source.
634
635                       'C' SP <path> SP <path> LF
636
637           here the first <path> is the source location and the second <path>
638           is the destination. See filemodify above for a detailed description
639           of what <path> may look like. To use a source path that contains SP
640           the path must be quoted.
641
642           A filecopy command takes effect immediately. Once the source
643           location has been copied to the destination any future commands
644           applied to the source location will not impact the destination of
645           the copy.
646
647       filerename
648           Renames an existing file or subdirectory to a different location
649           within the branch. The existing file or directory must exist. If
650           the destination exists it will be replaced by the source directory.
651
652                       'R' SP <path> SP <path> LF
653
654           here the first <path> is the source location and the second <path>
655           is the destination. See filemodify above for a detailed description
656           of what <path> may look like. To use a source path that contains SP
657           the path must be quoted.
658
659           A filerename command takes effect immediately. Once the source
660           location has been renamed to the destination any future commands
661           applied to the source location will create new files there and not
662           impact the destination of the rename.
663
664           Note that a filerename is the same as a filecopy followed by a
665           filedelete of the source location. There is a slight performance
666           advantage to using filerename, but the advantage is so small that
667           it is never worth trying to convert a delete/add pair in source
668           material into a rename for fast-import. This filerename command is
669           provided just to simplify frontends that already have rename
670           information and don’t want bother with decomposing it into a
671           filecopy followed by a filedelete.
672
673       filedeleteall
674           Included in a commit command to remove all files (and also all
675           directories) from the branch. This command resets the internal
676           branch structure to have no files in it, allowing the frontend to
677           subsequently add all interesting files from scratch.
678
679                       'deleteall' LF
680
681           This command is extremely useful if the frontend does not know (or
682           does not care to know) what files are currently on the branch, and
683           therefore cannot generate the proper filedelete commands to update
684           the content.
685
686           Issuing a filedeleteall followed by the needed filemodify commands
687           to set the correct content will produce the same results as sending
688           only the needed filemodify and filedelete commands. The
689           filedeleteall approach may however require fast-import to use
690           slightly more memory per active branch (less than 1 MiB for even
691           most large projects); so frontends that can easily obtain only the
692           affected paths for a commit are encouraged to do so.
693
694       notemodify
695           Included in a commit <notes_ref> command to add a new note
696           annotating a <commit-ish> or change this annotation contents.
697           Internally it is similar to filemodify 100644 on <commit-ish> path
698           (maybe split into subdirectories). It’s not advised to use any
699           other commands to write to the <notes_ref> tree except
700           filedeleteall to delete all existing notes in this tree. This
701           command has two different means of specifying the content of the
702           note.
703
704           External data format
705               The data content for the note was already supplied by a prior
706               blob command. The frontend just needs to connect it to the
707               commit that is to be annotated.
708
709                           'N' SP <dataref> SP <commit-ish> LF
710
711               Here <dataref> can be either a mark reference (:<idnum>) set by
712               a prior blob command, or a full 40-byte SHA-1 of an existing
713               Git blob object.
714
715           Inline data format
716               The data content for the note has not been supplied yet. The
717               frontend wants to supply it as part of this modify command.
718
719                           'N' SP 'inline' SP <commit-ish> LF
720                           data
721
722               See below for a detailed description of the data command.
723
724           In both formats <commit-ish> is any of the commit specification
725           expressions also accepted by from (see above).
726
727   mark
728       Arranges for fast-import to save a reference to the current object,
729       allowing the frontend to recall this object at a future point in time,
730       without knowing its SHA-1. Here the current object is the object
731       creation command the mark command appears within. This can be commit,
732       tag, and blob, but commit is the most common usage.
733
734                   'mark' SP ':' <idnum> LF
735
736       where <idnum> is the number assigned by the frontend to this mark. The
737       value of <idnum> is expressed as an ASCII decimal integer. The value 0
738       is reserved and cannot be used as a mark. Only values greater than or
739       equal to 1 may be used as marks.
740
741       New marks are created automatically. Existing marks can be moved to
742       another object simply by reusing the same <idnum> in another mark
743       command.
744
745   original-oid
746       Provides the name of the object in the original source control system.
747       fast-import will simply ignore this directive, but filter processes
748       which operate on and modify the stream before feeding to fast-import
749       may have uses for this information
750
751                   'original-oid' SP <object-identifier> LF
752
753       where <object-identifier> is any string not containing LF.
754
755   tag
756       Creates an annotated tag referring to a specific commit. To create
757       lightweight (non-annotated) tags see the reset command below.
758
759                   'tag' SP <name> LF
760                   mark?
761                   'from' SP <commit-ish> LF
762                   original-oid?
763                   'tagger' (SP <name>)? SP LT <email> GT SP <when> LF
764                   data
765
766       where <name> is the name of the tag to create.
767
768       Tag names are automatically prefixed with refs/tags/ when stored in
769       Git, so importing the CVS branch symbol RELENG-1_0-FINAL would use just
770       RELENG-1_0-FINAL for <name>, and fast-import will write the
771       corresponding ref as refs/tags/RELENG-1_0-FINAL.
772
773       The value of <name> must be a valid refname in Git and therefore may
774       contain forward slashes. As LF is not valid in a Git refname, no
775       quoting or escaping syntax is supported here.
776
777       The from command is the same as in the commit command; see above for
778       details.
779
780       The tagger command uses the same format as committer within commit;
781       again see above for details.
782
783       The data command following tagger must supply the annotated tag message
784       (see below for data command syntax). To import an empty tag message use
785       a 0 length data. Tag messages are free-form and are not interpreted by
786       Git. Currently they must be encoded in UTF-8, as fast-import does not
787       permit other encodings to be specified.
788
789       Signing annotated tags during import from within fast-import is not
790       supported. Trying to include your own PGP/GPG signature is not
791       recommended, as the frontend does not (easily) have access to the
792       complete set of bytes which normally goes into such a signature. If
793       signing is required, create lightweight tags from within fast-import
794       with reset, then create the annotated versions of those tags offline
795       with the standard git tag process.
796
797   reset
798       Creates (or recreates) the named branch, optionally starting from a
799       specific revision. The reset command allows a frontend to issue a new
800       from command for an existing branch, or to create a new branch from an
801       existing commit without creating a new commit.
802
803                   'reset' SP <ref> LF
804                   ('from' SP <commit-ish> LF)?
805                   LF?
806
807       For a detailed description of <ref> and <commit-ish> see above under
808       commit and from.
809
810       The LF after the command is optional (it used to be required).
811
812       The reset command can also be used to create lightweight
813       (non-annotated) tags. For example:
814
815           reset refs/tags/938
816           from :938
817
818       would create the lightweight tag refs/tags/938 referring to whatever
819       commit mark :938 references.
820
821   blob
822       Requests writing one file revision to the packfile. The revision is not
823       connected to any commit; this connection must be formed in a subsequent
824       commit command by referencing the blob through an assigned mark.
825
826                   'blob' LF
827                   mark?
828                   original-oid?
829                   data
830
831       The mark command is optional here as some frontends have chosen to
832       generate the Git SHA-1 for the blob on their own, and feed that
833       directly to commit. This is typically more work than it’s worth
834       however, as marks are inexpensive to store and easy to use.
835
836   data
837       Supplies raw data (for use as blob/file content, commit messages, or
838       annotated tag messages) to fast-import. Data can be supplied using an
839       exact byte count or delimited with a terminating line. Real frontends
840       intended for production-quality conversions should always use the exact
841       byte count format, as it is more robust and performs better. The
842       delimited format is intended primarily for testing fast-import.
843
844       Comment lines appearing within the <raw> part of data commands are
845       always taken to be part of the body of the data and are therefore never
846       ignored by fast-import. This makes it safe to import any file/message
847       content whose lines might start with #.
848
849       Exact byte count format
850           The frontend must specify the number of bytes of data.
851
852                       'data' SP <count> LF
853                       <raw> LF?
854
855           where <count> is the exact number of bytes appearing within <raw>.
856           The value of <count> is expressed as an ASCII decimal integer. The
857           LF on either side of <raw> is not included in <count> and will not
858           be included in the imported data.
859
860           The LF after <raw> is optional (it used to be required) but
861           recommended. Always including it makes debugging a fast-import
862           stream easier as the next command always starts in column 0 of the
863           next line, even if <raw> did not end with an LF.
864
865       Delimited format
866           A delimiter string is used to mark the end of the data. fast-import
867           will compute the length by searching for the delimiter. This format
868           is primarily useful for testing and is not recommended for real
869           data.
870
871                       'data' SP '<<' <delim> LF
872                       <raw> LF
873                       <delim> LF
874                       LF?
875
876           where <delim> is the chosen delimiter string. The string <delim>
877           must not appear on a line by itself within <raw>, as otherwise
878           fast-import will think the data ends earlier than it really does.
879           The LF immediately trailing <raw> is part of <raw>. This is one of
880           the limitations of the delimited format, it is impossible to supply
881           a data chunk which does not have an LF as its last byte.
882
883           The LF after <delim> LF is optional (it used to be required).
884
885   alias
886       Record that a mark refers to a given object without first creating any
887       new object.
888
889                   'alias' LF
890                   mark
891                   'to' SP <commit-ish> LF
892                   LF?
893
894       For a detailed description of <commit-ish> see above under from.
895
896   checkpoint
897       Forces fast-import to close the current packfile, start a new one, and
898       to save out all current branch refs, tags and marks.
899
900                   'checkpoint' LF
901                   LF?
902
903       Note that fast-import automatically switches packfiles when the current
904       packfile reaches --max-pack-size, or 4 GiB, whichever limit is smaller.
905       During an automatic packfile switch fast-import does not update the
906       branch refs, tags or marks.
907
908       As a checkpoint can require a significant amount of CPU time and disk
909       IO (to compute the overall pack SHA-1 checksum, generate the
910       corresponding index file, and update the refs) it can easily take
911       several minutes for a single checkpoint command to complete.
912
913       Frontends may choose to issue checkpoints during extremely large and
914       long running imports, or when they need to allow another Git process
915       access to a branch. However given that a 30 GiB Subversion repository
916       can be loaded into Git through fast-import in about 3 hours, explicit
917       checkpointing may not be necessary.
918
919       The LF after the command is optional (it used to be required).
920
921   progress
922       Causes fast-import to print the entire progress line unmodified to its
923       standard output channel (file descriptor 1) when the command is
924       processed from the input stream. The command otherwise has no impact on
925       the current import, or on any of fast-import’s internal state.
926
927                   'progress' SP <any> LF
928                   LF?
929
930       The <any> part of the command may contain any sequence of bytes that
931       does not contain LF. The LF after the command is optional. Callers may
932       wish to process the output through a tool such as sed to remove the
933       leading part of the line, for example:
934
935           frontend | git fast-import | sed 's/^progress //'
936
937       Placing a progress command immediately after a checkpoint will inform
938       the reader when the checkpoint has been completed and it can safely
939       access the refs that fast-import updated.
940
941   get-mark
942       Causes fast-import to print the SHA-1 corresponding to a mark to stdout
943       or to the file descriptor previously arranged with the --cat-blob-fd
944       argument. The command otherwise has no impact on the current import;
945       its purpose is to retrieve SHA-1s that later commits might want to
946       refer to in their commit messages.
947
948                   'get-mark' SP ':' <idnum> LF
949
950       See “Responses To Commands” below for details about how to read this
951       output safely.
952
953   cat-blob
954       Causes fast-import to print a blob to a file descriptor previously
955       arranged with the --cat-blob-fd argument. The command otherwise has no
956       impact on the current import; its main purpose is to retrieve blobs
957       that may be in fast-import’s memory but not accessible from the target
958       repository.
959
960                   'cat-blob' SP <dataref> LF
961
962       The <dataref> can be either a mark reference (:<idnum>) set previously
963       or a full 40-byte SHA-1 of a Git blob, preexisting or ready to be
964       written.
965
966       Output uses the same format as git cat-file --batch:
967
968           <sha1> SP 'blob' SP <size> LF
969           <contents> LF
970
971       This command can be used where a filemodify directive can appear,
972       allowing it to be used in the middle of a commit. For a filemodify
973       using an inline directive, it can also appear right before the data
974       directive.
975
976       See “Responses To Commands” below for details about how to read this
977       output safely.
978
979   ls
980       Prints information about the object at a path to a file descriptor
981       previously arranged with the --cat-blob-fd argument. This allows
982       printing a blob from the active commit (with cat-blob) or copying a
983       blob or tree from a previous commit for use in the current one (with
984       filemodify).
985
986       The ls command can also be used where a filemodify directive can
987       appear, allowing it to be used in the middle of a commit.
988
989       Reading from the active commit
990           This form can only be used in the middle of a commit. The path
991           names a directory entry within fast-import’s active commit. The
992           path must be quoted in this case.
993
994                       'ls' SP <path> LF
995
996       Reading from a named tree
997           The <dataref> can be a mark reference (:<idnum>) or the full
998           40-byte SHA-1 of a Git tag, commit, or tree object, preexisting or
999           waiting to be written. The path is relative to the top level of the
1000           tree named by <dataref>.
1001
1002                       'ls' SP <dataref> SP <path> LF
1003
1004       See filemodify above for a detailed description of <path>.
1005
1006       Output uses the same format as git ls-tree <tree> -- <path>:
1007
1008           <mode> SP ('blob' | 'tree' | 'commit') SP <dataref> HT <path> LF
1009
1010       The <dataref> represents the blob, tree, or commit object at <path> and
1011       can be used in later get-mark, cat-blob, filemodify, or ls commands.
1012
1013       If there is no file or subtree at that path, git fast-import will
1014       instead report
1015
1016           missing SP <path> LF
1017
1018       See “Responses To Commands” below for details about how to read this
1019       output safely.
1020
1021   feature
1022       Require that fast-import supports the specified feature, or abort if it
1023       does not.
1024
1025                   'feature' SP <feature> ('=' <argument>)? LF
1026
1027       The <feature> part of the command may be any one of the following:
1028
1029       date-format, export-marks, relative-marks, no-relative-marks, force
1030           Act as though the corresponding command-line option with a leading
1031           -- was passed on the command line (see OPTIONS, above).
1032
1033       import-marks, import-marks-if-exists
1034           Like --import-marks except in two respects: first, only one
1035           "feature import-marks" or "feature import-marks-if-exists" command
1036           is allowed per stream; second, an --import-marks= or
1037           --import-marks-if-exists command-line option overrides any of these
1038           "feature" commands in the stream; third, "feature
1039           import-marks-if-exists" like a corresponding command-line option
1040           silently skips a nonexistent file.
1041
1042       get-mark, cat-blob, ls
1043           Require that the backend support the get-mark, cat-blob, or ls
1044           command respectively. Versions of fast-import not supporting the
1045           specified command will exit with a message indicating so. This lets
1046           the import error out early with a clear message, rather than
1047           wasting time on the early part of an import before the unsupported
1048           command is detected.
1049
1050       notes
1051           Require that the backend support the notemodify (N) subcommand to
1052           the commit command. Versions of fast-import not supporting notes
1053           will exit with a message indicating so.
1054
1055       done
1056           Error out if the stream ends without a done command. Without this
1057           feature, errors causing the frontend to end abruptly at a
1058           convenient point in the stream can go undetected. This may occur,
1059           for example, if an import front end dies in mid-operation without
1060           emitting SIGTERM or SIGKILL at its subordinate git fast-import
1061           instance.
1062
1063   option
1064       Processes the specified option so that git fast-import behaves in a way
1065       that suits the frontend’s needs. Note that options specified by the
1066       frontend are overridden by any options the user may specify to git
1067       fast-import itself.
1068
1069               'option' SP <option> LF
1070
1071       The <option> part of the command may contain any of the options listed
1072       in the OPTIONS section that do not change import semantics, without the
1073       leading -- and is treated in the same way.
1074
1075       Option commands must be the first commands on the input (not counting
1076       feature commands), to give an option command after any non-option
1077       command is an error.
1078
1079       The following command-line options change import semantics and may
1080       therefore not be passed as option:
1081
1082       •   date-format
1083
1084       •   import-marks
1085
1086       •   export-marks
1087
1088       •   cat-blob-fd
1089
1090       •   force
1091
1092   done
1093       If the done feature is not in use, treated as if EOF was read. This can
1094       be used to tell fast-import to finish early.
1095
1096       If the --done command-line option or feature done command is in use,
1097       the done command is mandatory and marks the end of the stream.
1098

RESPONSES TO COMMANDS

1100       New objects written by fast-import are not available immediately. Most
1101       fast-import commands have no visible effect until the next checkpoint
1102       (or completion). The frontend can send commands to fill fast-import’s
1103       input pipe without worrying about how quickly they will take effect,
1104       which improves performance by simplifying scheduling.
1105
1106       For some frontends, though, it is useful to be able to read back data
1107       from the current repository as it is being updated (for example when
1108       the source material describes objects in terms of patches to be applied
1109       to previously imported objects). This can be accomplished by connecting
1110       the frontend and fast-import via bidirectional pipes:
1111
1112           mkfifo fast-import-output
1113           frontend <fast-import-output |
1114           git fast-import >fast-import-output
1115
1116       A frontend set up this way can use progress, get-mark, ls, and cat-blob
1117       commands to read information from the import in progress.
1118
1119       To avoid deadlock, such frontends must completely consume any pending
1120       output from progress, ls, get-mark, and cat-blob before performing
1121       writes to fast-import that might block.
1122

CRASH REPORTS

1124       If fast-import is supplied invalid input it will terminate with a
1125       non-zero exit status and create a crash report in the top level of the
1126       Git repository it was importing into. Crash reports contain a snapshot
1127       of the internal fast-import state as well as the most recent commands
1128       that lead up to the crash.
1129
1130       All recent commands (including stream comments, file changes and
1131       progress commands) are shown in the command history within the crash
1132       report, but raw file data and commit messages are excluded from the
1133       crash report. This exclusion saves space within the report file and
1134       reduces the amount of buffering that fast-import must perform during
1135       execution.
1136
1137       After writing a crash report fast-import will close the current
1138       packfile and export the marks table. This allows the frontend developer
1139       to inspect the repository state and resume the import from the point
1140       where it crashed. The modified branches and tags are not updated during
1141       a crash, as the import did not complete successfully. Branch and tag
1142       information can be found in the crash report and must be applied
1143       manually if the update is needed.
1144
1145       An example crash:
1146
1147           $ cat >in <<END_OF_INPUT
1148           # my very first test commit
1149           commit refs/heads/master
1150           committer Shawn O. Pearce <spearce> 19283 -0400
1151           # who is that guy anyway?
1152           data <<EOF
1153           this is my commit
1154           EOF
1155           M 644 inline .gitignore
1156           data <<EOF
1157           .gitignore
1158           EOF
1159           M 777 inline bob
1160           END_OF_INPUT
1161
1162           $ git fast-import <in
1163           fatal: Corrupt mode: M 777 inline bob
1164           fast-import: dumping crash report to .git/fast_import_crash_8434
1165
1166           $ cat .git/fast_import_crash_8434
1167           fast-import crash report:
1168               fast-import process: 8434
1169               parent process     : 1391
1170               at Sat Sep 1 00:58:12 2007
1171
1172           fatal: Corrupt mode: M 777 inline bob
1173
1174           Most Recent Commands Before Crash
1175           ---------------------------------
1176             # my very first test commit
1177             commit refs/heads/master
1178             committer Shawn O. Pearce <spearce> 19283 -0400
1179             # who is that guy anyway?
1180             data <<EOF
1181             M 644 inline .gitignore
1182             data <<EOF
1183           * M 777 inline bob
1184
1185           Active Branch LRU
1186           -----------------
1187               active_branches = 1 cur, 5 max
1188
1189           pos  clock name
1190           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1191            1)      0 refs/heads/master
1192
1193           Inactive Branches
1194           -----------------
1195           refs/heads/master:
1196             status      : active loaded dirty
1197             tip commit  : 0000000000000000000000000000000000000000
1198             old tree    : 0000000000000000000000000000000000000000
1199             cur tree    : 0000000000000000000000000000000000000000
1200             commit clock: 0
1201             last pack   :
1202
1203           -------------------
1204           END OF CRASH REPORT
1205

TIPS AND TRICKS

1207       The following tips and tricks have been collected from various users of
1208       fast-import, and are offered here as suggestions.
1209
1210   Use One Mark Per Commit
1211       When doing a repository conversion, use a unique mark per commit (mark
1212       :<n>) and supply the --export-marks option on the command line.
1213       fast-import will dump a file which lists every mark and the Git object
1214       SHA-1 that corresponds to it. If the frontend can tie the marks back to
1215       the source repository, it is easy to verify the accuracy and
1216       completeness of the import by comparing each Git commit to the
1217       corresponding source revision.
1218
1219       Coming from a system such as Perforce or Subversion this should be
1220       quite simple, as the fast-import mark can also be the Perforce
1221       changeset number or the Subversion revision number.
1222
1223   Freely Skip Around Branches
1224       Don’t bother trying to optimize the frontend to stick to one branch at
1225       a time during an import. Although doing so might be slightly faster for
1226       fast-import, it tends to increase the complexity of the frontend code
1227       considerably.
1228
1229       The branch LRU builtin to fast-import tends to behave very well, and
1230       the cost of activating an inactive branch is so low that bouncing
1231       around between branches has virtually no impact on import performance.
1232
1233   Handling Renames
1234       When importing a renamed file or directory, simply delete the old
1235       name(s) and modify the new name(s) during the corresponding commit. Git
1236       performs rename detection after-the-fact, rather than explicitly during
1237       a commit.
1238
1239   Use Tag Fixup Branches
1240       Some other SCM systems let the user create a tag from multiple files
1241       which are not from the same commit/changeset. Or to create tags which
1242       are a subset of the files available in the repository.
1243
1244       Importing these tags as-is in Git is impossible without making at least
1245       one commit which “fixes up” the files to match the content of the tag.
1246       Use fast-import’s reset command to reset a dummy branch outside of your
1247       normal branch space to the base commit for the tag, then commit one or
1248       more file fixup commits, and finally tag the dummy branch.
1249
1250       For example since all normal branches are stored under refs/heads/ name
1251       the tag fixup branch TAG_FIXUP. This way it is impossible for the fixup
1252       branch used by the importer to have namespace conflicts with real
1253       branches imported from the source (the name TAG_FIXUP is not
1254       refs/heads/TAG_FIXUP).
1255
1256       When committing fixups, consider using merge to connect the commit(s)
1257       which are supplying file revisions to the fixup branch. Doing so will
1258       allow tools such as git blame to track through the real commit history
1259       and properly annotate the source files.
1260
1261       After fast-import terminates the frontend will need to do rm
1262       .git/TAG_FIXUP to remove the dummy branch.
1263
1264   Import Now, Repack Later
1265       As soon as fast-import completes the Git repository is completely valid
1266       and ready for use. Typically this takes only a very short time, even
1267       for considerably large projects (100,000+ commits).
1268
1269       However repacking the repository is necessary to improve data locality
1270       and access performance. It can also take hours on extremely large
1271       projects (especially if -f and a large --window parameter is used).
1272       Since repacking is safe to run alongside readers and writers, run the
1273       repack in the background and let it finish when it finishes. There is
1274       no reason to wait to explore your new Git project!
1275
1276       If you choose to wait for the repack, don’t try to run benchmarks or
1277       performance tests until repacking is completed. fast-import outputs
1278       suboptimal packfiles that are simply never seen in real use situations.
1279
1280   Repacking Historical Data
1281       If you are repacking very old imported data (e.g. older than the last
1282       year), consider expending some extra CPU time and supplying --window=50
1283       (or higher) when you run git repack. This will take longer, but will
1284       also produce a smaller packfile. You only need to expend the effort
1285       once, and everyone using your project will benefit from the smaller
1286       repository.
1287
1288   Include Some Progress Messages
1289       Every once in a while have your frontend emit a progress message to
1290       fast-import. The contents of the messages are entirely free-form, so
1291       one suggestion would be to output the current month and year each time
1292       the current commit date moves into the next month. Your users will feel
1293       better knowing how much of the data stream has been processed.
1294

PACKFILE OPTIMIZATION

1296       When packing a blob fast-import always attempts to deltify against the
1297       last blob written. Unless specifically arranged for by the frontend,
1298       this will probably not be a prior version of the same file, so the
1299       generated delta will not be the smallest possible. The resulting
1300       packfile will be compressed, but will not be optimal.
1301
1302       Frontends which have efficient access to all revisions of a single file
1303       (for example reading an RCS/CVS ,v file) can choose to supply all
1304       revisions of that file as a sequence of consecutive blob commands. This
1305       allows fast-import to deltify the different file revisions against each
1306       other, saving space in the final packfile. Marks can be used to later
1307       identify individual file revisions during a sequence of commit
1308       commands.
1309
1310       The packfile(s) created by fast-import do not encourage good disk
1311       access patterns. This is caused by fast-import writing the data in the
1312       order it is received on standard input, while Git typically organizes
1313       data within packfiles to make the most recent (current tip) data appear
1314       before historical data. Git also clusters commits together, speeding up
1315       revision traversal through better cache locality.
1316
1317       For this reason it is strongly recommended that users repack the
1318       repository with git repack -a -d after fast-import completes, allowing
1319       Git to reorganize the packfiles for faster data access. If blob deltas
1320       are suboptimal (see above) then also adding the -f option to force
1321       recomputation of all deltas can significantly reduce the final packfile
1322       size (30-50% smaller can be quite typical).
1323
1324       Instead of running git repack you can also run git gc --aggressive,
1325       which will also optimize other things after an import (e.g. pack loose
1326       refs). As noted in the "AGGRESSIVE" section in git-gc(1) the
1327       --aggressive option will find new deltas with the -f option to git-
1328       repack(1). For the reasons elaborated on above using --aggressive after
1329       a fast-import is one of the few cases where it’s known to be
1330       worthwhile.
1331

MEMORY UTILIZATION

1333       There are a number of factors which affect how much memory fast-import
1334       requires to perform an import. Like critical sections of core Git,
1335       fast-import uses its own memory allocators to amortize any overheads
1336       associated with malloc. In practice fast-import tends to amortize any
1337       malloc overheads to 0, due to its use of large block allocations.
1338
1339   per object
1340       fast-import maintains an in-memory structure for every object written
1341       in this execution. On a 32 bit system the structure is 32 bytes, on a
1342       64 bit system the structure is 40 bytes (due to the larger pointer
1343       sizes). Objects in the table are not deallocated until fast-import
1344       terminates. Importing 2 million objects on a 32 bit system will require
1345       approximately 64 MiB of memory.
1346
1347       The object table is actually a hashtable keyed on the object name (the
1348       unique SHA-1). This storage configuration allows fast-import to reuse
1349       an existing or already written object and avoid writing duplicates to
1350       the output packfile. Duplicate blobs are surprisingly common in an
1351       import, typically due to branch merges in the source.
1352
1353   per mark
1354       Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
1355       bytes, depending on pointer size) per mark. Although the array is
1356       sparse, frontends are still strongly encouraged to use marks between 1
1357       and n, where n is the total number of marks required for this import.
1358
1359   per branch
1360       Branches are classified as active and inactive. The memory usage of the
1361       two classes is significantly different.
1362
1363       Inactive branches are stored in a structure which uses 96 or 120 bytes
1364       (32 bit or 64 bit systems, respectively), plus the length of the branch
1365       name (typically under 200 bytes), per branch. fast-import will easily
1366       handle as many as 10,000 inactive branches in under 2 MiB of memory.
1367
1368       Active branches have the same overhead as inactive branches, but also
1369       contain copies of every tree that has been recently modified on that
1370       branch. If subtree include has not been modified since the branch
1371       became active, its contents will not be loaded into memory, but if
1372       subtree src has been modified by a commit since the branch became
1373       active, then its contents will be loaded in memory.
1374
1375       As active branches store metadata about the files contained on that
1376       branch, their in-memory storage size can grow to a considerable size
1377       (see below).
1378
1379       fast-import automatically moves active branches to inactive status
1380       based on a simple least-recently-used algorithm. The LRU chain is
1381       updated on each commit command. The maximum number of active branches
1382       can be increased or decreased on the command line with
1383       --active-branches=.
1384
1385   per active tree
1386       Trees (aka directories) use just 12 bytes of memory on top of the
1387       memory required for their entries (see “per active file” below). The
1388       cost of a tree is virtually 0, as its overhead amortizes out over the
1389       individual file entries.
1390
1391   per active file entry
1392       Files (and pointers to subtrees) within active trees require 52 or 64
1393       bytes (32/64 bit platforms) per entry. To conserve space, file and tree
1394       names are pooled in a common string table, allowing the filename
1395       “Makefile” to use just 16 bytes (after including the string header
1396       overhead) no matter how many times it occurs within the project.
1397
1398       The active branch LRU, when coupled with the filename string pool and
1399       lazy loading of subtrees, allows fast-import to efficiently import
1400       projects with 2,000+ branches and 45,114+ files in a very limited
1401       memory footprint (less than 2.7 MiB per active branch).
1402

SIGNALS

1404       Sending SIGUSR1 to the git fast-import process ends the current
1405       packfile early, simulating a checkpoint command. The impatient operator
1406       can use this facility to peek at the objects and refs from an import in
1407       progress, at the cost of some added running time and worse compression.
1408

SEE ALSO

1410       git-fast-export(1)
1411

GIT

1413       Part of the git(1) suite
1414
1415
1416
1417Git 2.33.1                        2021-10-12                GIT-FAST-IMPORT(1)
Impressum