1GIT-FAST-IMPORT(1) Git Manual GIT-FAST-IMPORT(1)
2
6 git-fast-import - Backend for fast Git data importers
7
9 frontend | git-fast-import [options]
10
12 This program is usually not what the end user wants to run directly.
13 Most end users want to use one of the existing frontend programs, which
14 parses a specific type of foreign source and feeds the contents stored
15 there to git-fast-import.
16 fast-import reads a mixed command/data stream from standard input and
18 writes one or more packfiles directly into the current repository. When
19 EOF is received on standard input, fast import writes out updated
20 branch and tag refs, fully updating the current repository with the
21 newly imported data.
22 The fast-import backend itself can import into an empty repository (one
24 that has already been initialized by git-init(1)) or incrementally
25 update an existing populated repository. Whether or not incremental
26 imports are supported from a particular foreign source depends on the
27 frontend program in use.
28
30 --date-format=<fmt>
31 Specify the type of dates the frontend will supply to fast-import
32 within author, committer and tagger commands. See “Date Formats”
33 below for details about which formats are supported, and their
34 syntax.
35 --force
37 Force updating modified existing branches, even if doing so would
38 cause commits to be lost (as the new commit does not contain the
39 old commit).
40 --max-pack-size=<n>
42 Maximum size of each output packfile, expressed in MiB. The default
43 is 4096 (4 GiB) as that is the maximum allowed packfile size (due
44 to file format limitations). Some importers may wish to lower this,
45 such as to ensure the resulting packfiles fit on CDs.
46 --depth=<n>
48 Maximum delta depth, for blob and tree deltification. Default is
49 10.
50 --active-branches=<n>
52 Maximum number of branches to maintain active at once. See “Memory
53 Utilization” below for details. Default is 5.
54 --export-marks=<file>
56 Dumps the internal marks table to <file> when complete. Marks are
57 written one per line as :markid SHA-1. Frontends can use this file
58 to validate imports after they have been completed, or to save the
59 marks table across incremental runs. As <file> is only opened and
60 truncated at checkpoint (or completion) the same path can also be
61 safely given to --import-marks.
62 --import-marks=<file>
64 Before processing any input, load the marks specified in <file>.
65 The input file must exist, must be readable, and must use the same
66 format as produced by --export-marks. Multiple options may be
67 supplied to import more than one set of marks. If a mark is defined
68 to different values, the last file wins.
69 --export-pack-edges=<file>
71 After creating a packfile, print a line of data to <file> listing
72 the filename of the packfile and the last commit on each branch
73 that was written to that packfile. This information may be useful
74 after importing projects whose total object set exceeds the 4 GiB
75 packfile limit, as these commits can be used as edge points during
76 calls to git-pack-objects(1).
77 --quiet
79 Disable all non-fatal output, making fast-import silent when it is
80 successful. This option disables the output shown by --stats.
81 --stats
83 Display some basic statistics about the objects fast-import has
84 created, the packfiles they were stored into, and the memory used
85 by fast-import during this run. Showing this output is currently
86 the default, but can be disabled with --quiet.
87
89 The design of fast-import allows it to import large projects in a
90 minimum amount of memory usage and processing time. Assuming the
91 frontend is able to keep up with fast-import and feed it a constant
92 stream of data, import times for projects holding 10+ years of history
93 and containing 100,000+ individual commits are generally completed in
94 just 1-2 hours on quite modest (~$2,000 USD) hardware.
95 Most bottlenecks appear to be in foreign source data access (the source
97 just cannot extract revisions fast enough) or disk IO (fast-import
98 writes as fast as the disk will take the data). Imports will run faster
99 if the source data is stored on a different drive than the destination
100 Git repository (due to less IO contention).
101
103 A typical frontend for fast-import tends to weigh in at approximately
104 200 lines of Perl/Python/Ruby code. Most developers have been able to
105 create working importers in just a couple of hours, even though it is
106 their first exposure to fast-import, and sometimes even to Git. This is
107 an ideal situation, given that most conversion tools are throw-away
108 (use once, and never look back).
109
111 Like git-push or git-fetch, imports handled by fast-import are safe to
112 run alongside parallel git repack -a -d or git gc invocations, or any
113 other Git operation (including git prune, as loose objects are never
114 used by fast-import).
115 fast-import does not lock the branch or tag refs it is actively
117 importing. After the import, during its ref update phase, fast-import
118 tests each existing branch ref to verify the update will be a
119 fast-forward update (the commit stored in the ref is contained in the
120 new history of the commit to be written). If the update is not a
121 fast-forward update, fast-import will skip updating that ref and
122 instead prints a warning message. fast-import will always attempt to
123 update all branch refs, and does not stop on the first failure.
124 Branch updates can be forced with --force, but its recommended that
126 this only be used on an otherwise quiet repository. Using --force is
127 not necessary for an initial import into an empty repository.
128
130 fast-import tracks a set of branches in memory. Any branch can be
131 created or modified at any point during the import process by sending a
132 commit command on the input stream. This design allows a frontend
133 program to process an unlimited number of branches simultaneously,
134 generating commits in the order they are available from the source
135 data. It also simplifies the frontend programs considerably.
136 fast-import does not use or alter the current working directory, or any
138 file within it. (It does however update the current Git repository, as
139 referenced by GIT_DIR.) Therefore an import frontend may use the
140 working directory for its own purposes, such as extracting file
141 revisions from the foreign source. This ignorance of the working
142 directory also allows fast-import to run very quickly, as it does not
143 need to perform any costly file update operations when switching
144 between branches.
145
147 With the exception of raw file data (which Git does not interpret) the
148 fast-import input format is text (ASCII) based. This text based format
149 simplifies development and debugging of frontend programs, especially
150 when a higher level language such as Perl, Python or Ruby is being
151 used.
152 fast-import is very strict about its input. Where we say SP below we
154 mean exactly one space. Likewise LF means one (and only one) linefeed.
155 Supplying additional whitespace characters will cause unexpected
156 results, such as branch names or file names with leading or trailing
157 spaces in their name, or early termination of fast-import when it
158 encounters unexpected input.
159 Stream Comments
161 To aid in debugging frontends fast-import ignores any line that begins
162 with # (ASCII pound/hash) up to and including the line ending LF. A
163 comment line may contain any sequence of bytes that does not contain an
164 LF and therefore may be used to include any detailed debugging
165 information that might be specific to the frontend and useful when
166 inspecting a fast-import data stream.
167 Date Formats
169 The following date formats are supported. A frontend should select the
170 format it will use for this import by passing the format name in the
171 --date-format=<fmt> command line option.
172 raw
174 This is the Git native format and is <time> SP <offutc>. It is also
175 fast-import´s default format, if --date-format was not specified.
176 The time of the event is specified by <time> as the number of
178 seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is
179 written as an ASCII decimal integer.
180 The local offset is specified by <offutc> as a positive or negative
182 offset from UTC. For example EST (which is 5 hours behind UTC)
183 would be expressed in <tz> by “-0500” while UTC is “+0000”. The
184 local offset does not affect <time>; it is used only as an
185 advisement to help formatting routines display the timestamp.
186 If the local offset is not available in the source material, use
188 “+0000”, or the most common local offset. For example many
189 organizations have a CVS repository which has only ever been
190 accessed by users who are located in the same location and
191 timezone. In this case a reasonable offset from UTC could be
192 assumed.
193 Unlike the rfc2822 format, this format is very strict. Any
195 variation in formatting will cause fast-import to reject the value.
196 rfc2822
198 This is the standard email format as described by RFC 2822.
199 An example value is “Tue Feb 6 11:22:18 2007 -0500”. The Git parser
201 is accurate, but a little on the lenient side. It is the same
202 parser used by git-am(1) when applying patches received from email.
203 Some malformed strings may be accepted as valid dates. In some of
205 these cases Git will still be able to obtain the correct date from
206 the malformed string. There are also some types of malformed
207 strings which Git will parse wrong, and yet consider valid.
208 Seriously malformed strings will be rejected.
209 Unlike the raw format above, the timezone/UTC offset information
211 contained in an RFC 2822 date string is used to adjust the date
212 value to UTC prior to storage. Therefore it is important that this
213 information be as accurate as possible.
214 If the source material uses RFC 2822 style dates, the frontend
216 should let fast-import handle the parsing and conversion (rather
217 than attempting to do it itself) as the Git parser has been well
218 tested in the wild.
219 Frontends should prefer the raw format if the source material
221 already uses UNIX-epoch format, can be coaxed to give dates in that
222 format, or its format is easily convertible to it, as there is no
223 ambiguity in parsing.
224 now
226 Always use the current time and timezone. The literal now must
227 always be supplied for <when>.
228 This is a toy format. The current time and timezone of this system
230 is always copied into the identity string at the time it is being
231 created by fast-import. There is no way to specify a different time
232 or timezone.
233 This particular format is supplied as its short to implement and
235 may be useful to a process that wants to create a new commit right
236 now, without needing to use a working directory or git-update-
237 index(1).
238 If separate author and committer commands are used in a commit the
240 timestamps may not match, as the system clock will be polled twice
241 (once for each command). The only way to ensure that both author
242 and committer identity information has the same timestamp is to
243 omit author (thus copying from committer) or to use a date format
244 other than now.
245 Commands
247 fast-import accepts several commands to update the current repository
248 and control the current import process. More detailed discussion (with
249 examples) of each command follows later.
250 commit
252 Creates a new branch or updates an existing branch by creating a
253 new commit and updating the branch to point at the newly created
254 commit.
255 tag
257 Creates an annotated tag object from an existing commit or branch.
258 Lightweight tags are not supported by this command, as they are not
259 recommended for recording meaningful points in time.
260 reset
262 Reset an existing branch (or a new branch) to a specific revision.
263 This command must be used to change a branch to a specific revision
264 without making a commit on it.
265 blob
267 Convert raw file data into a blob, for future use in a commit
268 command. This command is optional and is not needed to perform an
269 import.
270 checkpoint
272 Forces fast-import to close the current packfile, generate its
273 unique SHA-1 checksum and index, and start a new packfile. This
274 command is optional and is not needed to perform an import.
275 progress
277 Causes fast-import to echo the entire line to its own standard
278 output. This command is optional and is not needed to perform an
279 import.
280 commit
282 Create or update a branch with a new commit, recording one logical
283 change to the project.
284 ´commit´ SP <ref> LF
287 mark?
288 (´author´ SP <name> SP LT <email> GT SP <when> LF)?
289 ´committer´ SP <name> SP LT <email> GT SP <when> LF
290 data
291 (´from´ SP <committish> LF)?
292 (´merge´ SP <committish> LF)?
293 (filemodify | filedelete | filecopy | filerename | filedeleteall)*
294 LF?
295 where <ref> is the name of the branch to make the commit on. Typically
296 branch names are prefixed with refs/heads/ in Git, so importing the CVS
297 branch symbol RELENG-1_0 would use refs/heads/RELENG-1_0 for the value
298 of <ref>. The value of <ref> must be a valid refname in Git. As LF is
299 not valid in a Git refname, no quoting or escaping syntax is supported
300 here.
301 A mark command may optionally appear, requesting fast-import to save a
303 reference to the newly created commit for future use by the frontend
304 (see below for format). It is very common for frontends to mark every
305 commit they create, thereby allowing future branch creation from any
306 imported commit.
307 The data command following committer must supply the commit message
309 (see below for data command syntax). To import an empty commit message
310 use a 0 length data. Commit messages are free-form and are not
311 interpreted by Git. Currently they must be encoded in UTF-8, as
312 fast-import does not permit other encodings to be specified.
313 Zero or more filemodify, filedelete, filecopy, filerename and
315 filedeleteall commands may be included to update the contents of the
316 branch prior to creating the commit. These commands may be supplied in
317 any order. However it is recommended that a filedeleteall command
318 precede all filemodify, filecopy and filerename commands in the same
319 commit, as filedeleteall wipes the branch clean (see below).
320 The LF after the command is optional (it used to be required).
322 author
324 An author command may optionally appear, if the author
325 information might differ from the committer information. If
326 author is omitted then fast-import will automatically use the
327 committer´s information for the author portion of the commit.
328 See below for a description of the fields in author, as they are
329 identical to committer.
330 committer
332 The committer command indicates who made this commit, and when
333 they made it.
334 Here <name> is the person´s display name (for example “Com M
336 Itter”) and <email> is the person´s email address
337 (“cm@example.com”). LT and GT are the literal less-than (\x3c)
338 and greater-than (\x3e) symbols. These are required to delimit
339 the email address from the other fields in the line. Note that
340 <name> is free-form and may contain any sequence of bytes,
341 except LT and LF. It is typically UTF-8 encoded.
342 The time of the change is specified by <when> using the date
344 format that was selected by the --date-format=<fmt> command line
345 option. See “Date Formats” above for the set of supported
346 formats, and their syntax.
347 from
349 The from command is used to specify the commit to initialize
350 this branch from. This revision will be the first ancestor of
351 the new commit.
352 Omitting the from command in the first commit of a new branch
354 will cause fast-import to create that commit with no ancestor.
355 This tends to be desired only for the initial commit of a
356 project. Omitting the from command on existing branches is
357 usually desired, as the current commit on that branch is
358 automatically assumed to be the first ancestor of the new
359 commit.
360 As LF is not valid in a Git refname or SHA-1 expression, no
362 quoting or escaping syntax is supported within <committish>.
363 Here <committish> is any of the following:
365 · The name of an existing branch already in fast-import´s
368 internal branch table. If fast-import doesn´t know the name,
369 its treated as a SHA-1 expression.
370 · A mark reference, :<idnum>, where <idnum> is the mark
372 number.
373 The reason fast-import uses : to denote a mark reference is
375 this character is not legal in a Git branch name. The
376 leading : makes it easy to distinguish between the mark 42
377 (:42) and the branch 42 (42 or refs/heads/42), or an
378 abbreviated SHA-1 which happened to consist only of base-10
379 digits.
380 Marks must be declared (via mark) before they can be used.
382 · A complete 40 byte or abbreviated commit SHA-1 in hex.
384 · Any valid Git SHA-1 expression that resolves to a commit.
386 See “SPECIFYING REVISIONS” in git-rev-parse(1) for details.
387 The special case of restarting an incremental import from the
388 current branch value should be written as:
389 from refs/heads/branch^0
393 The ^0 suffix is necessary as fast-import does not permit a
395 branch to start from itself, and the branch is created in memory
396 before the from command is even read from the input. Adding ^0
397 will force fast-import to resolve the commit through Git´s
398 revision parsing library, rather than its internal branch table,
399 thereby loading in the existing value of the branch.
400 merge
402 Includes one additional ancestor commit, and makes the current
403 commit a merge commit. An unlimited number of merge commands per
404 commit are permitted by fast-import, thereby establishing an
405 n-way merge. However Git´s other tools never create commits with
406 more than 15 additional ancestors (forming a 16-way merge). For
407 this reason it is suggested that frontends do not use more than
408 15 merge commands per commit.
409 Here <committish> is any of the commit specification expressions
411 also accepted by from (see above).
412 filemodify
414 Included in a commit command to add a new file or change the
415 content of an existing file. This command has two different
416 means of specifying the content of the file.
417 External data format
419 The data content for the file was already supplied by a
420 prior blob command. The frontend just needs to connect it.
421 ´M´ SP <mode> SP <dataref> SP <path> LF
424 Here <dataref> can be either a mark reference (:<idnum>) set
425 by a prior blob command, or a full 40-byte SHA-1 of an
426 existing Git blob object.
427 Inline data format
429 The data content for the file has not been supplied yet. The
430 frontend wants to supply it as part of this modify command.
431 ´M´ SP <mode> SP ´inline´ SP <path> LF
434 data
435 See below for a detailed description of the data command.
436 In both formats <mode> is the type of file entry, specified in
437 octal. Git only supports the following modes:
438 · 100644 or 644: A normal (not-executable) file. The majority
441 of files in most projects use this mode. If in doubt, this
442 is what you want.
443 · 100755 or 755: A normal, but executable, file.
445 · 120000: A symlink, the content of the file will be the link
447 target.
448 In both formats <path> is the complete path of the file to be
449 added (if not already existing) or modified (if already
450 existing).
451 A <path> string must use UNIX-style directory separators
453 (forward slash /), may contain any byte other than LF, and must
454 not start with double quote (").
455 If an LF or double quote must be encoded into <path> shell-style
457 quoting should be used, e.g. "path/with\n and \" in it".
458 The value of <path> must be in canonical form. That is it must
460 not:
461 · contain an empty directory component (e.g. foo//bar is
464 invalid),
465 · end with a directory separator (e.g. foo/ is invalid),
467 · start with a directory separator (e.g. /foo is invalid),
469 · contain the special component . or .. (e.g. foo/./bar and
471 foo/../bar are invalid).
472 It is recommended that <path> always be encoded using UTF-8.
473 filedelete
475 Included in a commit command to remove a file or recursively
476 delete an entire directory from the branch. If the file or
477 directory removal makes its parent directory empty, the parent
478 directory will be automatically removed too. This cascades up
479 the tree until the first non-empty directory or the root is
480 reached.
481 ´D´ SP <path> LF
484 here <path> is the complete path of the file or subdirectory to
485 be removed from the branch. See filemodify above for a detailed
486 description of <path>.
487 filecopy
489 Recursively copies an existing file or subdirectory to a
490 different location within the branch. The existing file or
491 directory must exist. If the destination exists it will be
492 completely replaced by the content copied from the source.
493 ´C´ SP <path> SP <path> LF
496 here the first <path> is the source location and the second
497 <path> is the destination. See filemodify above for a detailed
498 description of what <path> may look like. To use a source path
499 that contains SP the path must be quoted.
500 A filecopy command takes effect immediately. Once the source
502 location has been copied to the destination any future commands
503 applied to the source location will not impact the destination
504 of the copy.
505 filerename
507 Renames an existing file or subdirectory to a different location
508 within the branch. The existing file or directory must exist. If
509 the destination exists it will be replaced by the source
510 directory.
511 ´R´ SP <path> SP <path> LF
514 here the first <path> is the source location and the second
515 <path> is the destination. See filemodify above for a detailed
516 description of what <path> may look like. To use a source path
517 that contains SP the path must be quoted.
518 A filerename command takes effect immediately. Once the source
520 location has been renamed to the destination any future commands
521 applied to the source location will create new files there and
522 not impact the destination of the rename.
523 Note that a filerename is the same as a filecopy followed by a
525 filedelete of the source location. There is a slight performance
526 advantage to using filerename, but the advantage is so small
527 that it is never worth trying to convert a delete/add pair in
528 source material into a rename for fast-import. This filerename
529 command is provided just to simplify frontends that already have
530 rename information and don´t want bother with decomposing it
531 into a filecopy followed by a filedelete.
532 filedeleteall
534 Included in a commit command to remove all files (and also all
535 directories) from the branch. This command resets the internal
536 branch structure to have no files in it, allowing the frontend
537 to subsequently add all interesting files from scratch.
538 ´deleteall´ LF
541 This command is extremely useful if the frontend does not know
542 (or does not care to know) what files are currently on the
543 branch, and therefore cannot generate the proper filedelete
544 commands to update the content.
545 Issuing a filedeleteall followed by the needed filemodify
547 commands to set the correct content will produce the same
548 results as sending only the needed filemodify and filedelete
549 commands. The filedeleteall approach may however require
550 fast-import to use slightly more memory per active branch (less
551 than 1 MiB for even most large projects); so frontends that can
552 easily obtain only the affected paths for a commit are
553 encouraged to do so.
554 mark
556 Arranges for fast-import to save a reference to the current object,
557 allowing the frontend to recall this object at a future point in time,
558 without knowing its SHA-1. Here the current object is the object
559 creation command the mark command appears within. This can be commit,
560 tag, and blob, but commit is the most common usage.
561 ´mark´ SP ´:´ <idnum> LF
564 where <idnum> is the number assigned by the frontend to this mark. The
565 value of <idnum> is expressed as an ASCII decimal integer. The value 0
566 is reserved and cannot be used as a mark. Only values greater than or
567 equal to 1 may be used as marks.
568 New marks are created automatically. Existing marks can be moved to
570 another object simply by reusing the same <idnum> in another mark
571 command.
572 tag
574 Creates an annotated tag referring to a specific commit. To create
575 lightweight (non-annotated) tags see the reset command below.
576 ´tag´ SP <name> LF
579 ´from´ SP <committish> LF
580 ´tagger´ SP <name> SP LT <email> GT SP <when> LF
581 data
582 where <name> is the name of the tag to create.
583 Tag names are automatically prefixed with refs/tags/ when stored in
585 Git, so importing the CVS branch symbol RELENG-1_0-FINAL would use just
586 RELENG-1_0-FINAL for <name>, and fast-import will write the
587 corresponding ref as refs/tags/RELENG-1_0-FINAL.
588 The value of <name> must be a valid refname in Git and therefore may
590 contain forward slashes. As LF is not valid in a Git refname, no
591 quoting or escaping syntax is supported here.
592 The from command is the same as in the commit command; see above for
594 details.
595 The tagger command uses the same format as committer within commit;
597 again see above for details.
598 The data command following tagger must supply the annotated tag message
600 (see below for data command syntax). To import an empty tag message use
601 a 0 length data. Tag messages are free-form and are not interpreted by
602 Git. Currently they must be encoded in UTF-8, as fast-import does not
603 permit other encodings to be specified.
604 Signing annotated tags during import from within fast-import is not
606 supported. Trying to include your own PGP/GPG signature is not
607 recommended, as the frontend does not (easily) have access to the
608 complete set of bytes which normally goes into such a signature. If
609 signing is required, create lightweight tags from within fast-import
610 with reset, then create the annotated versions of those tags offline
611 with the standard git-tag(1) process.
612 reset
614 Creates (or recreates) the named branch, optionally starting from a
615 specific revision. The reset command allows a frontend to issue a new
616 from command for an existing branch, or to create a new branch from an
617 existing commit without creating a new commit.
618 ´reset´ SP <ref> LF
621 (´from´ SP <committish> LF)?
622 LF?
623 For a detailed description of <ref> and <committish> see above under
624 commit and from.
625 The LF after the command is optional (it used to be required).
627 The reset command can also be used to create lightweight
629 (non-annotated) tags. For example:
630 reset refs/tags/938
633 from :938
634 would create the lightweight tag refs/tags/938 referring to whatever
635 commit mark :938 references.
636 blob
638 Requests writing one file revision to the packfile. The revision is not
639 connected to any commit; this connection must be formed in a subsequent
640 commit command by referencing the blob through an assigned mark.
641 ´blob´ LF
644 mark?
645 data
646 The mark command is optional here as some frontends have chosen to
647 generate the Git SHA-1 for the blob on their own, and feed that
648 directly to commit. This is typically more work than its worth however,
649 as marks are inexpensive to store and easy to use.
650 data
652 Supplies raw data (for use as blob/file content, commit messages, or
653 annotated tag messages) to fast-import. Data can be supplied using an
654 exact byte count or delimited with a terminating line. Real frontends
655 intended for production-quality conversions should always use the exact
656 byte count format, as it is more robust and performs better. The
657 delimited format is intended primarily for testing fast-import.
658 Comment lines appearing within the <raw> part of data commands are
660 always taken to be part of the body of the data and are therefore never
661 ignored by fast-import. This makes it safe to import any file/message
662 content whose lines might start with #.
663 Exact byte count format
665 The frontend must specify the number of bytes of data.
666 ´data´ SP <count> LF
669 <raw> LF?
670 where <count> is the exact number of bytes appearing within <raw>.
671 The value of <count> is expressed as an ASCII decimal integer. The
672 LF on either side of <raw> is not included in <count> and will not
673 be included in the imported data.
674 The LF after <raw> is optional (it used to be required) but
676 recommended. Always including it makes debugging a fast-import
677 stream easier as the next command always starts in column 0 of the
678 next line, even if <raw> did not end with an LF.
679 Delimited format
681 A delimiter string is used to mark the end of the data. fast-import
682 will compute the length by searching for the delimiter. This format
683 is primarily useful for testing and is not recommended for real
684 data.
685 ´data´ SP ´<<´ <delim> LF
688 <raw> LF
689 <delim> LF
690 LF?
691 where <delim> is the chosen delimiter string. The string <delim>
692 must not appear on a line by itself within <raw>, as otherwise
693 fast-import will think the data ends earlier than it really does.
694 The LF immediately trailing <raw> is part of <raw>. This is one of
695 the limitations of the delimited format, it is impossible to supply
696 a data chunk which does not have an LF as its last byte.
697 The LF after <delim> LF is optional (it used to be required).
699 checkpoint
701 Forces fast-import to close the current packfile, start a new one, and
702 to save out all current branch refs, tags and marks.
703 ´checkpoint´ LF
706 LF?
707 Note that fast-import automatically switches packfiles when the current
708 packfile reaches --max-pack-size, or 4 GiB, whichever limit is smaller.
709 During an automatic packfile switch fast-import does not update the
710 branch refs, tags or marks.
711 As a checkpoint can require a significant amount of CPU time and disk
713 IO (to compute the overall pack SHA-1 checksum, generate the
714 corresponding index file, and update the refs) it can easily take
715 several minutes for a single checkpoint command to complete.
716 Frontends may choose to issue checkpoints during extremely large and
718 long running imports, or when they need to allow another Git process
719 access to a branch. However given that a 30 GiB Subversion repository
720 can be loaded into Git through fast-import in about 3 hours, explicit
721 checkpointing may not be necessary.
722 The LF after the command is optional (it used to be required).
724 progress
726 Causes fast-import to print the entire progress line unmodified to its
727 standard output channel (file descriptor 1) when the command is
728 processed from the input stream. The command otherwise has no impact on
729 the current import, or on any of fast-import´s internal state.
730 ´progress´ SP <any> LF
733 LF?
734 The <any> part of the command may contain any sequence of bytes that
735 does not contain LF. The LF after the command is optional. Callers may
736 wish to process the output through a tool such as sed to remove the
737 leading part of the line, for example:
738 frontend | git-fast-import | sed ´s/^progress //´
741 Placing a progress command immediately after a checkpoint will inform
742 the reader when the checkpoint has been completed and it can safely
743 access the refs that fast-import updated.
744
746 The following tips and tricks have been collected from various users of
747 fast-import, and are offered here as suggestions.
748 Use One Mark Per Commit
750 When doing a repository conversion, use a unique mark per commit (mark
751 :<n>) and supply the --export-marks option on the command line.
752 fast-import will dump a file which lists every mark and the Git object
753 SHA-1 that corresponds to it. If the frontend can tie the marks back to
754 the source repository, it is easy to verify the accuracy and
755 completeness of the import by comparing each Git commit to the
756 corresponding source revision.
757 Coming from a system such as Perforce or Subversion this should be
759 quite simple, as the fast-import mark can also be the Perforce
760 changeset number or the Subversion revision number.
761 Freely Skip Around Branches
763 Don´t bother trying to optimize the frontend to stick to one branch at
764 a time during an import. Although doing so might be slightly faster for
765 fast-import, it tends to increase the complexity of the frontend code
766 considerably.
767 The branch LRU builtin to fast-import tends to behave very well, and
769 the cost of activating an inactive branch is so low that bouncing
770 around between branches has virtually no impact on import performance.
771 Handling Renames
773 When importing a renamed file or directory, simply delete the old
774 name(s) and modify the new name(s) during the corresponding commit. Git
775 performs rename detection after-the-fact, rather than explicitly during
776 a commit.
777 Use Tag Fixup Branches
779 Some other SCM systems let the user create a tag from multiple files
780 which are not from the same commit/changeset. Or to create tags which
781 are a subset of the files available in the repository.
782 Importing these tags as-is in Git is impossible without making at least
784 one commit which “fixes up” the files to match the content of the tag.
785 Use fast-import´s reset command to reset a dummy branch outside of your
786 normal branch space to the base commit for the tag, then commit one or
787 more file fixup commits, and finally tag the dummy branch.
788 For example since all normal branches are stored under refs/heads/ name
790 the tag fixup branch TAG_FIXUP. This way it is impossible for the fixup
791 branch used by the importer to have namespace conflicts with real
792 branches imported from the source (the name TAG_FIXUP is not
793 refs/heads/TAG_FIXUP).
794 When committing fixups, consider using merge to connect the commit(s)
796 which are supplying file revisions to the fixup branch. Doing so will
797 allow tools such as git-blame(1) to track through the real commit
798 history and properly annotate the source files.
799 After fast-import terminates the frontend will need to do rm
801 .git/TAG_FIXUP to remove the dummy branch.
802 Import Now, Repack Later
804 As soon as fast-import completes the Git repository is completely valid
805 and ready for use. Typically this takes only a very short time, even
806 for considerably large projects (100,000+ commits).
807 However repacking the repository is necessary to improve data locality
809 and access performance. It can also take hours on extremely large
810 projects (especially if -f and a large --window parameter is used).
811 Since repacking is safe to run alongside readers and writers, run the
812 repack in the background and let it finish when it finishes. There is
813 no reason to wait to explore your new Git project!
814 If you choose to wait for the repack, don´t try to run benchmarks or
816 performance tests until repacking is completed. fast-import outputs
817 suboptimal packfiles that are simply never seen in real use situations.
818 Repacking Historical Data
820 If you are repacking very old imported data (e.g. older than the last
821 year), consider expending some extra CPU time and supplying --window=50
822 (or higher) when you run git-repack(1). This will take longer, but will
823 also produce a smaller packfile. You only need to expend the effort
824 once, and everyone using your project will benefit from the smaller
825 repository.
826 Include Some Progress Messages
828 Every once in a while have your frontend emit a progress message to
829 fast-import. The contents of the messages are entirely free-form, so
830 one suggestion would be to output the current month and year each time
831 the current commit date moves into the next month. Your users will feel
832 better knowing how much of the data stream has been processed.
833
835 When packing a blob fast-import always attempts to deltify against the
836 last blob written. Unless specifically arranged for by the frontend,
837 this will probably not be a prior version of the same file, so the
838 generated delta will not be the smallest possible. The resulting
839 packfile will be compressed, but will not be optimal.
840 Frontends which have efficient access to all revisions of a single file
842 (for example reading an RCS/CVS ,v file) can choose to supply all
843 revisions of that file as a sequence of consecutive blob commands. This
844 allows fast-import to deltify the different file revisions against each
845 other, saving space in the final packfile. Marks can be used to later
846 identify individual file revisions during a sequence of commit
847 commands.
848 The packfile(s) created by fast-import do not encourage good disk
850 access patterns. This is caused by fast-import writing the data in the
851 order it is received on standard input, while Git typically organizes
852 data within packfiles to make the most recent (current tip) data appear
853 before historical data. Git also clusters commits together, speeding up
854 revision traversal through better cache locality.
855 For this reason it is strongly recommended that users repack the
857 repository with git repack -a -d after fast-import completes, allowing
858 Git to reorganize the packfiles for faster data access. If blob deltas
859 are suboptimal (see above) then also adding the -f option to force
860 recomputation of all deltas can significantly reduce the final packfile
861 size (30-50% smaller can be quite typical).
862
864 There are a number of factors which affect how much memory fast-import
865 requires to perform an import. Like critical sections of core Git,
866 fast-import uses its own memory allocators to amortize any overheads
867 associated with malloc. In practice fast-import tends to amortize any
868 malloc overheads to 0, due to its use of large block allocations.
869 per object
871 fast-import maintains an in-memory structure for every object written
872 in this execution. On a 32 bit system the structure is 32 bytes, on a
873 64 bit system the structure is 40 bytes (due to the larger pointer
874 sizes). Objects in the table are not deallocated until fast-import
875 terminates. Importing 2 million objects on a 32 bit system will require
876 approximately 64 MiB of memory.
877 The object table is actually a hashtable keyed on the object name (the
879 unique SHA-1). This storage configuration allows fast-import to reuse
880 an existing or already written object and avoid writing duplicates to
881 the output packfile. Duplicate blobs are surprisingly common in an
882 import, typically due to branch merges in the source.
883 per mark
885 Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
886 bytes, depending on pointer size) per mark. Although the array is
887 sparse, frontends are still strongly encouraged to use marks between 1
888 and n, where n is the total number of marks required for this import.
889 per branch
891 Branches are classified as active and inactive. The memory usage of the
892 two classes is significantly different.
893 Inactive branches are stored in a structure which uses 96 or 120 bytes
895 (32 bit or 64 bit systems, respectively), plus the length of the branch
896 name (typically under 200 bytes), per branch. fast-import will easily
897 handle as many as 10,000 inactive branches in under 2 MiB of memory.
898 Active branches have the same overhead as inactive branches, but also
900 contain copies of every tree that has been recently modified on that
901 branch. If subtree include has not been modified since the branch
902 became active, its contents will not be loaded into memory, but if
903 subtree src has been modified by a commit since the branch became
904 active, then its contents will be loaded in memory.
905 As active branches store metadata about the files contained on that
907 branch, their in-memory storage size can grow to a considerable size
908 (see below).
909 fast-import automatically moves active branches to inactive status
911 based on a simple least-recently-used algorithm. The LRU chain is
912 updated on each commit command. The maximum number of active branches
913 can be increased or decreased on the command line with
914 --active-branches=.
915 per active tree
917 Trees (aka directories) use just 12 bytes of memory on top of the
918 memory required for their entries (see “per active file” below). The
919 cost of a tree is virtually 0, as its overhead amortizes out over the
920 individual file entries.
921 per active file entry
923 Files (and pointers to subtrees) within active trees require 52 or 64
924 bytes (32/64 bit platforms) per entry. To conserve space, file and tree
925 names are pooled in a common string table, allowing the filename
926 “Makefile” to use just 16 bytes (after including the string header
927 overhead) no matter how many times it occurs within the project.
928 The active branch LRU, when coupled with the filename string pool and
930 lazy loading of subtrees, allows fast-import to efficiently import
931 projects with 2,000+ branches and 45,114+ files in a very limited
932 memory footprint (less than 2.7 MiB per active branch).
933
935 Written by Shawn O. Pearce <spearce@spearce.org>.
936
938 Documentation by Shawn O. Pearce <spearce@spearce.org>.
939
941 Part of the git(7) suite
942Git 1.5.3.3 10/09/2007 GIT-FAST-IMPORT(1)