1PUNGI(1) Pungi PUNGI(1)
2
6 pungi - Pungi Documentation
7 Contents:
9
11 [image: Pungi Logo] [image]
12 Pungi is a distribution compose tool.
14 Composes are release snapshots that contain release deliverables such
16 as:
17 • installation trees
19 • RPMs
21 • repodata
23 • comps
25 • (bootable) ISOs
27 • kickstart trees
29 • anaconda images
31 • images for PXE boot
33 Tool overview
35 Pungi consists of multiple separate executables backed by a common li‐
36 brary.
37 The main entry-point is the pungi-koji script. It loads the compose
39 configuration and kicks off the process. Composing itself is done in
40 phases. Each phase is responsible for generating some artifacts on
41 disk and updating the compose object that is threaded through all the
42 phases.
43 Pungi itself does not actually do that much. Most of the actual work is
45 delegated to separate executables. Pungi just makes sure that all the
46 commands are invoked in the appropriate order and with correct argu‐
47 ments. It also moves the artifacts to correct locations.
48 The executable name pungi-koji comes from the fact that most of those
50 separate executables submit tasks to Koji that does the actual work in
51 an auditable way.
52 However unlike doing everything manually in Koji, Pungi will make sure
54 you are building all images from the same package set, and will produce
55 even deliverables that Koji can not create like YUM repos and installer
56 ISOs.
57 Links
59 • Upstream GIT: https://pagure.io/pungi/
60 • Issue tracker: https://pagure.io/pungi/issues
62 • Questions can be asked on #fedora-releng IRC channel on FreeNode
64 Origin of name
66 The name Pungi comes from the instrument used to charm snakes. Anaconda
67 being the software Pungi was manipulating, and anaconda being a snake,
68 led to the referential naming.
69 The first name, which was suggested by Seth Vidal, was FIST, Fedora In‐
71 stallation <Something> Tool. That name was quickly discarded and re‐
72 placed with Pungi.
73 There was also a bit of an inside joke that when said aloud, it could
75 sound like punji, which is a sharpened stick at the bottom of a trap.
76 Kind of like software…
77
79 Each invocation of pungi-koji consists of a set of phases. [image:
80 phase diagram] [image]
81 Most of the phases run sequentially (left-to-right in the diagram), but
83 there are use cases where multiple phases run in parallel. This happens
84 for phases whose main point is to wait for a Koji task to finish.
85 Init
87 The first phase to ever run. Can not be skipped. It prepares the comps
88 files for variants (by filtering out groups and packages that should
89 not be there). See comps for details about how this is done.
90 Pkgset
92 This phase loads a set of packages that should be composed. It has two
93 separate results: it prepares repos with packages in work/ directory
94 (one per arch) for further processing, and it returns a data structure
95 with mapping of packages to architectures.
96 Buildinstall
98 Spawns a bunch of threads, each of which runs either lorax or buildin‐
99 stall command (the latter coming from anaconda package). The commands
100 create boot.iso and other boot configuration files. The image is fi‐
101 nally linked into the compose/ directory as netinstall media.
102 The created images are also needed for creating live media or other im‐
104 ages in later phases.
105 With lorax this phase runs one task per variant.arch combination. For
107 buildinstall command there is only one task per architecture and prod‐
108 uct.img should be used to customize the results.
109 Gather
111 This phase uses data collected by pkgset phase and figures out what
112 packages should be in each variant. The basic mapping can come from
113 comps file, a JSON mapping or additional_packages config option. This
114 inputs can then be enriched by adding all dependencies. See gathering
115 for details.
116 Once the mapping is finalized, the packages are linked to appropriate
118 places and the rpms.json manifest is created.
119 ExtraFiles
121 This phase collects extra files from the configuration and copies them
122 to the compose directory. The files are described by a JSON file in the
123 compose subtree where the files are copied. This metadata is meant to
124 be distributed with the data (on ISO images).
125 Createrepo
127 This phase creates RPM repositories for each variant.arch tree. It is
128 actually reading the rpms.json manifest to figure out which packages
129 should be included.
130 OSTree
132 Updates an ostree repository with a new commit with packages from the
133 compose. The repository lives outside of the compose and is updated
134 immediately. If the compose fails in a later stage, the commit will not
135 be reverted.
136 Implementation wise, this phase runs rpm-ostree command in Koji runroot
138 (to allow running on different arches).
139 Createiso
141 Generates ISO files and accumulates enough metadata to be able to cre‐
142 ate image.json manifest. The file is however not created in this phase,
143 instead it is dumped in the pungi-koji script itself.
144 The files include a repository with all RPMs from the variant. There
146 will be multiple images if the packages do not fit on a single image.
147 The image will be bootable if buildinstall phase is enabled and the
149 packages fit on a single image.
150 There can also be images with source repositories. These are never
152 bootable.
153 ExtraIsos
155 This phase is very similar to createiso, except it combines content
156 from multiple variants onto a single image. Packages, repodata and ex‐
157 tra files from each configured variant are put into a subdirectory. Ad‐
158 ditional extra files can be put into top level of the image. The image
159 will be bootable if the main variant is bootable.
160 LiveImages, LiveMedia
162 Creates media in Koji with koji spin-livecd, koji spin-appliance or
163 koji spin-livemedia command. When the media are finished, the images
164 are copied into the compose/ directory and metadata for images is up‐
165 dated.
166 ImageBuild
168 This phase wraps up koji image-build. It also updates the metadata ul‐
169 timately responsible for images.json manifest.
170 OSBuild
172 Similarly to image build, this phases creates a koji osbuild task. In
173 the background it uses OSBuild Composer to create images.
174 OSBS
176 This phase builds container base images in OSBS.
177 The finished images are available in registry provided by OSBS, but not
179 downloaded directly into the compose. The is metadata about the created
180 image in compose/metadata/osbs.json.
181 ImageContainer
183 This phase builds a container image in OSBS, and stores the metadata in
184 the same file as OSBS phase. The container produced here wraps a dif‐
185 ferent image, created it ImageBuild or OSBuild phase. It can be useful
186 to deliver a VM image to containerized environments.
187 OSTreeInstaller
189 Creates bootable media that carry an ostree repository as a payload.
190 These images are created by running lorax with special templates. Again
191 it runs in Koji runroot.
192 Repoclosure
194 Run repoclosure on each repository. By default errors are only reported
195 in the log, the compose will still be considered a success. The actual
196 error has to be looked up in the compose logs directory. Configuration
197 allows customizing this.
198 ImageChecksum
200 Responsible for generating checksums for the images. The checksums are
201 stored in image manifest as well as files on disk. The list of images
202 to be processed is obtained from the image manifest. This way all im‐
203 ages will get the same checksums irrespective of the phase that created
204 them.
205 Test
207 This phase is supposed to run some sanity checks on the finished com‐
208 pose.
209 The only test is to check all images listed the metadata and verify
211 that they look sane. For ISO files headers are checked to verify the
212 format is correct, and for bootable media a check is run to verify they
213 have properties that allow booting.
214
216 The configuration file parser is provided by kobo
217 The file follows a Python-like format. It consists of a sequence of
219 variables that have a value assigned to them.
220 variable = value
222 The variable names must follow the same convention as Python code:
224 start with a letter and consist of letters, digits and underscores
225 only.
226 The values can be either an integer, float, boolean (True or False), a
228 string or None. Strings must be enclosed in either single or double
229 quotes.
230 Complex types are supported as well.
232 A list is enclosed in square brackets and items are separated with com‐
234 mas. There can be a comma after the last item as well.
235 a_list = [1,
237 2,
238 3,
239 ]
240 A tuple works like a list, but is enclosed in parenthesis.
242 a_tuple = (1, "one")
244 A dictionary is wrapped in brackets, and consists of key: value pairs
246 separated by commas. The keys can only be formed from basic types (int,
247 float, string).
248 a_dict = {
250 'foo': 'bar',
251 1: None
252 }
253 The value assigned to a variable can also be taken from another vari‐
255 able.
256 one = 1
258 another = one
259 Anything on a line after a # symbol is ignored and functions as a com‐
261 ment.
262 Importing other files
264 It is possible to include another configuration file. The files are
265 looked up relative to the currently processed file.
266 The general structure of import is:
268 from FILENAME import WHAT
270 The FILENAME should be just the base name of the file without extension
272 (which must be .conf). WHAT can either be a comma separated list of
273 variables or *.
274 # Opens constants.conf and brings PI and E into current scope.
276 from constants import PI, E
277 # Opens common.conf and brings everything defined in that file into current
279 # file as well.
280 from common import *
281 NOTE:
283 Pungi will copy the configuration file given on command line into
284 the logs/ directory. Only this single file will be copied, not any
285 included ones. (Copying included files requires a fix in kobo li‐
286 brary.)
287 The JSON-formatted dump of configuration is correct though.
289 Formatting strings
291 String interpolation is available as well. It uses a %-encoded format.
292 See Python documentation for more details.
293 joined = "%s %s" % (var_a, var_b)
295 a_dict = {
297 "fst": 1,
298 "snd": 2,
299 }
300 another = "%(fst)s %(snd)s" % a_dict
301
303 Please read productmd documentation for terminology and other release
304 and compose related details.
305 Minimal Config Example
307 # RELEASE
308 release_name = "Fedora"
309 release_short = "Fedora"
310 release_version = "23"
311 # GENERAL SETTINGS
313 comps_file = "comps-f23.xml"
314 variants_file = "variants-f23.xml"
315 # KOJI
317 koji_profile = "koji"
318 runroot = False
319 # PKGSET
321 sigkeys = [None]
322 pkgset_source = "koji"
323 pkgset_koji_tag = "f23"
324 # GATHER
326 gather_method = "deps"
327 greedy_method = "build"
328 check_deps = False
329 # BUILDINSTALL
331 buildinstall_method = "lorax"
332 Release
334 Following mandatory options describe a release.
335 Options
337 release_name [mandatory]
338 (str) – release name
339 release_short [mandatory]
341 (str) – release short name, without spaces and special charac‐
342 ters
343 release_version [mandatory]
345 (str) – release version
346 release_type = “ga” (str) – release type, for example ga,
348 updates or updates-testing. See list of all valid values in pro‐
349 ductmd documentation.
350 release_internal = False
352 (bool) – whether the compose is meant for public consumption
353 treeinfo_version
355 (str) Version to display in .treeinfo files. If not configured,
356 the value from release_version will be used.
357 Example
359 release_name = "Fedora"
360 release_short = "Fedora"
361 release_version = "23"
362 # release_type = "ga"
363 Base Product
365 Base product options are optional and we need to them only if we’re
366 composing a layered product built on another (base) product.
367 Options
369 base_product_name
370 (str) – base product name
371 base_product_short
373 (str) – base product short name, without spaces and special
374 characters
375 base_product_version
377 (str) – base product major version
378 base_product_type = “ga”
380 (str) – base product type, “ga”, “updates” etc., for full list
381 see documentation of productmd.
382 Example
384 release_name = "RPM Fusion"
385 release_short = "rf"
386 release_version = "23.0"
387 base_product_name = "Fedora"
389 base_product_short = "Fedora"
390 base_product_version = "23"
391 General Settings
393 Options
394 comps_file [mandatory]
395 (scm_dict, str or None) – reference to comps XML file with in‐
396 stallation groups
397 variants_file [mandatory]
399 (scm_dict or str) – reference to variants XML file that defines
400 release variants and architectures
401 module_defaults_dir [optional]
403 (scm_dict or str) – reference the module defaults directory con‐
404 taining modulemd-defaults YAML documents. Files relevant for
405 modules included in the compose will be embedded in the gener‐
406 ated repodata and available for DNF.
407 module_defaults_dir = {
409 "scm": "git",
410 "repo": "https://pagure.io/releng/fedora-module-defaults.git",
411 "dir": ".",
412 }
413 failable_deliverables [optional]
415 (list) – list which deliverables on which variant and architec‐
416 ture can fail and not abort the whole compose. This only applies
417 to buildinstall and iso parts. All other artifacts can be con‐
418 figured in their respective part of configuration.
419 Please note that * as a wildcard matches all architectures but
421 src.
422 comps_filter_environments [optional]
424 (bool) – When set to False, the comps files for variants will
425 not have their environments filtered to match the variant.
426 tree_arches
428 ([str]) – list of architectures which should be included; if un‐
429 defined, all architectures from variants.xml will be included
430 tree_variants
432 ([str]) – list of variants which should be included; if unde‐
433 fined, all variants from variants.xml will be included
434 repoclosure_strictness
436 (list) – variant/arch mapping describing how repoclosure should
437 run. Possible values are
438 • off – do not run repoclosure
440 • lenient – (default) run repoclosure and write results to
442 logs, but detected errors are only reported in logs
443 • fatal – abort compose when any issue is detected
445 When multiple blocks in the mapping match a variant/arch combi‐
447 nation, the last value will win.
448 repoclosure_backend
450 (str) – Select which tool should be used to run repoclosure over
451 created repositories. By default yum is used, but you can switch
452 to dnf. Please note that when dnf is used, the build dependen‐
453 cies check is skipped. On Python 3, only dnf backend is avail‐
454 able.
455 See also: the gather_backend setting for Pungi’s gather phase.
457 cts_url
459 (str) – URL to Compose Tracking Service. If defined, Pungi will
460 add the compose to Compose Tracking Service and ge the compose
461 ID from it. For example https://cts.localhost.tld/
462 cts_keytab
464 (str) – Path to Kerberos keytab which will be used for Compose
465 Tracking Service Kerberos authentication. If not defined, the
466 default Kerberos principal is used.
467 compose_type
469 (str) – Allows to set default compose type. Type set via a com‐
470 mand-line option overwrites this.
471 mbs_api_url
473 (str) – URL to Module Build Service (MBS) API. For example
474 https://mbs.example.com/module-build-service/2. This is re‐
475 quired by pkgset_scratch_modules.
476 Example
478 comps_file = {
479 "scm": "git",
480 "repo": "https://git.fedorahosted.org/git/comps.git",
481 "branch": None,
482 "file": "comps-f23.xml.in",
483 }
484 variants_file = {
486 "scm": "git",
487 "repo": "https://pagure.io/pungi-fedora.git ",
488 "branch": None,
489 "file": "variants-fedora.xml",
490 }
491 failable_deliverables = [
493 ('^.*$', {
494 # Buildinstall can fail on any variant and any arch
495 '*': ['buildinstall'],
496 'src': ['buildinstall'],
497 # Nothing on i386 blocks the compose
498 'i386': ['buildinstall', 'iso', 'live'],
499 })
500 ]
501 tree_arches = ["x86_64"]
503 tree_variants = ["Server"]
504 repoclosure_strictness = [
506 # Make repoclosure failures fatal for compose on all variants …
507 ('^.*$', {'*': 'fatal'}),
508 # … except for Everything where it should not run at all.
509 ('^Everything$', {'*': 'off'})
510 ]
511 Image Naming
513 Both image name and volume id are generated based on the configuration.
514 Since the volume id is limited to 32 characters, there are more set‐
515 tings available. The process for generating volume id is to get a list
516 of possible formats and try them sequentially until one fits in the
517 length limit. If substitutions are configured, each attempted volume id
518 will be modified by it.
519 For layered products, the candidate formats are first image_volid_lay‐
521 ered_product_formats followed by image_volid_formats. Otherwise, only
522 image_volid_formats are tried.
523 If no format matches the length limit, an error will be reported and
525 compose aborted.
526 Options
528 There a couple common format specifiers available for both the options:
529 • compose_id
531 • release_short
533 • version
535 • date
537 • respin
539 • type
541 • type_suffix
543 • label
545 • label_major_version
547 • variant
549 • arch
551 • disc_type
553 image_name_format [optional]
555 (str|dict) – Python’s format string to serve as template for im‐
556 age names. The value can also be a dict mapping variant UID
557 regexes to the format string. The pattern should not overlap,
558 otherwise it is undefined which one will be used.
559 This format will be used for all phases generating images. Cur‐
561 rently that means createiso, live_images and buildinstall.
562 Available extra keys are:
564 • disc_num
566 • suffix
568 image_volid_formats [optional]
570 (list) – A list of format strings for generating volume id.
571 The extra available keys are:
573 • base_product_short
575 • base_product_version
577 image_volid_layered_product_formats [optional]
579 (list) – A list of format strings for generating volume id for
580 layered products. The keys available are the same as for im‐
581 age_volid_formats.
582 restricted_volid = False
584 (bool) – New versions of lorax replace all non-alphanumerical
585 characters with dashes (underscores are preserved). This option
586 will mimic similar behaviour in Pungi.
587 volume_id_substitutions [optional]
589 (dict) – A mapping of string replacements to shorten the volume
590 id.
591 disc_types [optional]
593 (dict) – A mapping for customizing disc_type used in image
594 names.
595 Available keys are:
597 • boot – for boot.iso images created in buildinstall
599 phase
600 • live – for images created by live_images phase
602 • dvd – for images created by createiso phase
604 • ostree – for ostree installer images
606 Default values are the same as the keys.
608 Example
610 # Image name respecting Fedora's image naming policy
611 image_name_format = "%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s%(suffix)s"
612 # Use the same format for volume id
613 image_volid_formats = [
614 "%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s"
615 ]
616 # No special handling for layered products, use same format as for regular images
617 image_volid_layered_product_formats = []
618 # Replace "Cloud" with "C" in volume id etc.
619 volume_id_substitutions = {
620 'Cloud': 'C',
621 'Alpha': 'A',
622 'Beta': 'B',
623 'TC': 'T',
624 }
625 disc_types = {
627 'boot': 'netinst',
628 'live': 'Live',
629 'dvd': 'DVD',
630 }
631 Signing
633 If you want to sign deliverables generated during pungi run like RPM
634 wrapped images. You must provide few configuration options:
635 signing_command [optional]
637 (str) – Command that will be run with a koji build as a single
638 argument. This command must not require any user interaction.
639 If you need to pass a password for a signing key to the command,
640 do this via command line option of the command and use string
641 formatting syntax %(signing_key_password)s. (See sign‐
642 ing_key_password_file).
643 signing_key_id [optional]
645 (str) – ID of the key that will be used for the signing. This
646 ID will be used when crafting koji paths to signed files (kojip‐
647 kgs.fedoraproject.org/pack‐
648 ages/NAME/VER/REL/data/signed/KEYID/..).
649 signing_key_password_file [optional]
651 (str) – Path to a file with password that will be formatted into
652 signing_command string via %(signing_key_password)s string for‐
653 mat syntax (if used). Because pungi config is usually stored in
654 git and is part of compose logs we don’t want password to be in‐
655 cluded directly in the config. Note: If - string is used in‐
656 stead of a filename, then you will be asked for the password in‐
657 teractivelly right after pungi starts.
658 Example
660 signing_command = '~/git/releng/scripts/sigulsign_unsigned.py -vv --password=%(signing_key_password)s fedora-24'
661 signing_key_id = '81b46521'
662 signing_key_password_file = '~/password_for_fedora-24_key'
663 Git URLs
665 In multiple places the config requires URL of a Git repository to down‐
666 load some file from. This URL is passed on to Koji. It is possible to
667 specify which commit to use using this syntax:
668 git://git.example.com/git/repo-name.git?#<rev_spec>
670 The <rev_spec> pattern can be replaced with actual commit SHA, a tag
672 name, HEAD to indicate that tip of default branch should be used or
673 origin/<branch_name> to use tip of arbitrary branch.
674 If the URL specifies a branch or HEAD, Pungi will replace it with the
676 actual commit SHA. This will later show up in Koji tasks and help with
677 tracing what particular inputs were used.
678 NOTE:
680 The origin must be specified because of the way Koji works with the
681 repository. It will clone the repository then switch to requested
682 state with git reset --hard REF. Since no local branches are cre‐
683 ated, we need to use full specification including the name of the
684 remote.
685 Createrepo Settings
687 Options
688 createrepo_checksum
689 (str) – specify checksum type for createrepo; expected values:
690 sha512, sha256, sha1. Defaults to sha256.
691 createrepo_c = True
693 (bool) – use createrepo_c (True) or legacy createrepo (False)
694 createrepo_deltas = False
696 (list) – generate delta RPMs against an older compose. This
697 needs to be used together with --old-composes command line argu‐
698 ment. The value should be a mapping of variants and architec‐
699 tures that should enable creating delta RPMs. Source and debug‐
700 info repos never have deltas.
701 createrepo_use_xz = False
703 (bool) – whether to pass --xz to the createrepo command. This
704 will cause the SQLite databases to be compressed with xz.
705 createrepo_num_threads
707 (int) – how many concurrent createrepo process to run. The de‐
708 fault is to use one thread per CPU available on the machine.
709 createrepo_num_workers
711 (int) – how many concurrent createrepo workers to run. Value de‐
712 faults to 3.
713 createrepo_database
715 (bool) – whether to create SQLite database as part of the repo‐
716 data. This is only useful as an optimization for clients using
717 Yum to consume to the repo. Default value depends on gather
718 backend. For DNF it’s turned off, for Yum the default is True.
719 createrepo_extra_args
721 ([str]) – a list of extra arguments passed on to createrepo or
722 createrepo_c executable. This could be useful for enabling
723 zchunk generation and pointing it to correct dictionaries.
724 createrepo_extra_modulemd
726 (dict) – a mapping of variant UID to an scm dict. If specified,
727 it should point to a directory with extra module metadata YAML
728 files that will be added to the repository for this variant. The
729 cloned files should be split into subdirectories for each archi‐
730 tecture of the variant.
731 createrepo_enable_cache = True
733 (bool) – whether to use --cachedir option of createrepo. It will
734 cache and reuse checksum vaules to speed up createrepo phase.
735 The cache dir is located at /var/cache/pungi/createrepo_c/$re‐
736 lease_short-$uid e.g. /var/cache/pungi/createrepo_c/Fedora-1000
737 product_id = None
739 (scm_dict) – If specified, it should point to a directory with
740 certificates *<variant_uid>-<arch>-*.pem. Pungi will copy each
741 certificate file into the relevant Yum repositories as a produc‐
742 tid file in the repodata directories. The purpose of these pro‐
743 ductid files is to expose the product data to
744 subscription-manager. subscription-manager includes a “prod‐
745 uct-id” Yum plugin that can read these productid certificate
746 files from each Yum repository.
747 product_id_allow_missing = False
749 (bool) – When product_id is used and a certificate for some
750 variant and architecture is missing, Pungi will exit with an er‐
751 ror by default. When you set this option to True, Pungi will
752 ignore the missing certificate and simply log a warning message.
753 product_id_allow_name_prefix = True
755 (bool) – Allow arbitrary prefix for the certificate file name
756 (see leading * in the pattern above). Setting this option to
757 False will make the pattern more strict by requiring the file
758 name to start directly with variant name.
759 Example
761 createrepo_checksum = "sha"
762 createrepo_deltas = [
763 # All arches for Everything should have deltas.
764 ('^Everything$', {'*': True}),
765 # Also Server.x86_64 should have them (but not on other arches).
766 ('^Server$', {'x86_64': True}),
767 ]
768 createrepo_extra_modulemd = {
769 "Server": {
770 "scm": "git",
771 "repo": "https://example.com/extra-server-modulemd.git",
772 "dir": ".",
773 # The directory should have this layout. Each architecture for the
774 # variant should be included (even if the directory is empty.
775 # .
776 # ├── aarch64
777 # │ ├── some-file.yaml
778 # │ └ ...
779 # └── x86_64
780 }
781 }
782 Package Set Settings
784 Options
785 sigkeys
786 ([str or None]) – priority list of signing key IDs. These key
787 IDs match the key IDs for the builds in Koji. Pungi will choose
788 signed packages according to the order of the key IDs that you
789 specify here. Use one single key in this list to ensure that all
790 RPMs are signed by one key. If the list includes an empty string
791 or None, Pungi will allow unsigned packages. If the list only
792 includes None, Pungi will use all unsigned packages.
793 pkgset_source [mandatory]
795 (str) – “koji” (any koji instance) or “repos” (arbitrary yum
796 repositories)
797 pkgset_koji_tag
799 (str|[str]) – tag(s) to read package set from. This option can
800 be omitted for modular composes.
801 pkgset_koji_builds
803 (str|[str]) – extra build(s) to include in a package set defined
804 as NVRs.
805 pkgset_koji_scratch_tasks
807 (str|[str]) – RPM scratch build task(s) to include in a package
808 set, defined as task IDs. This option can be used only when com‐
809 pose_type is set to test. The RPM still needs to have higher NVR
810 than any other RPM with the same name coming from other sources
811 in order to appear in the resulting compose.
812 pkgset_koji_module_tag
814 (str|[str]) – tags to read module from. This option works simi‐
815 larly to listing tags in variants XML. If tags are specified and
816 variants XML specifies some modules via NSVC (or part of), only
817 modules matching that list will be used (and taken from the
818 tag). Inheritance is used automatically.
819 pkgset_koji_module_builds
821 (dict) – A mapping of variants to extra module builds to include
822 in a package set: {variant: [N:S:V:C]}.
823 pkgset_koji_inherit = True
825 (bool) – inherit builds from parent tags; we can turn it off
826 only if we have all builds tagged in a single tag
827 pkgset_koji_inherit_modules = False
829 (bool) – the same as above, but this only applies to modular
830 tags. This option applies to the content tags that contain the
831 RPMs.
832 pkgset_repos
834 (dict) – A mapping of architectures to repositories with RPMs:
835 {arch: [repo]}. Only use when pkgset_source = "repos".
836 pkgset_scratch_modules
838 (dict) – A mapping of variants to scratch module builds: {vari‐
839 ant: [N:S:V:C]}. Requires mbs_api_url.
840 pkgset_exclusive_arch_considers_noarch = True
842 (bool) – If a package includes noarch in its ExclusiveArch tag,
843 it will be included in all architectures since noarch is compat‐
844 ible with everything. Set this option to False to ignore noarch
845 in ExclusiveArch and always consider only binary architectures.
846 pkgset_allow_reuse = True
848 (bool) – When set to True, Pungi will try to reuse pkgset data
849 from the old composes specified by --old-composes. When enabled,
850 this option can speed up new composes because it does not need
851 to calculate the pkgset data from Koji. However, if you block or
852 unblock a package in Koji (for example) between composes, then
853 Pungi may not respect those changes in your new compose.
854 signed_packages_retries = 0
856 (int) – In automated workflows, you might start a compose before
857 Koji has completely written all signed packages to disk. In this
858 case you may want Pungi to wait for the package to appear in
859 Koji’s storage. This option controls how many times Pungi will
860 retry looking for the signed copy.
861 signed_packages_wait = 30
863 (int) – Interval in seconds for how long to wait between at‐
864 tempts to find signed packages. This option only makes sense
865 when signed_packages_retries is set higher than 0.
866 Example
868 sigkeys = [None]
869 pkgset_source = "koji"
870 pkgset_koji_tag = "f23"
871 Buildinstall Settings
873 Script or process that creates bootable images with Anaconda installer
874 is historically called buildinstall.
875 Options
877 buildinstall_method
878 (str) – “lorax” (f16+, rhel7+) or “buildinstall” (older re‐
879 leases)
880 lorax_options
882 (list) – special options passed on to lorax.
883 Format: [(variant_uid_regex, {arch|*: {option: name}})].
885 Recognized options are:
887 • bugurl – str (default None)
889 • nomacboot – bool (default True)
891 • noupgrade – bool (default True)
893 • add_template – [str] (default empty)
895 • add_arch_template – [str] (default empty)
897 • add_template_var – [str] (default empty)
899 • add_arch_template_var – [str] (default empty)
901 • rootfs_size – [int] (default empty)
903 • version – [str] (default from treeinfo_version or re‐
905 lease_version) – used as --version and --release argu‐
906 ment on the lorax command line
907 • dracut_args – [[str]] (default empty) override argu‐
909 ments for dracut. Please note that if this option is
910 used, lorax will not use any other arguments, so you
911 have to provide a full list and can not just add some‐
912 thing.
913 • skip_branding – bool (default False)
915 • squashfs_only – bool (default False) pass the
917 –squashfs_only to Lorax.
918 • configuration_file – (scm_dict) (default empty) pass
920 the specified configuration file to Lorax using the -c
921 option.
922 lorax_extra_sources
924 (list) – a variant/arch mapping with urls for extra source
925 repositories added to Lorax command line. Either one repo or a
926 list can be specified.
927 lorax_use_koji_plugin = False
929 (bool) – When set to True, the Koji pungi_buildinstall task will
930 be used to execute Lorax instead of runroot. Use only if the
931 Koji instance has the pungi_buildinstall plugin installed.
932 buildinstall_kickstart
934 (scm_dict) – If specified, this kickstart file will be copied
935 into each file and pointed to in boot configuration.
936 buildinstall_topdir
938 (str) – Full path to top directory where the runroot buildin‐
939 stall Koji tasks output should be stored. This is useful in sit‐
940 uation when the Pungi compose is not generated on the same stor‐
941 age as the Koji task is running on. In this case, Pungi can pro‐
942 vide input repository for runroot task using HTTP and set the
943 output directory for this task to buildinstall_topdir. Once the
944 runroot task finishes, Pungi will copy the results of runroot
945 tasks to the compose working directory.
946 buildinstall_skip
948 (list) – mapping that defines which variants and arches to skip
949 during buildinstall; format: [(variant_uid_regex, {arch|*:
950 True})]. This is only supported for lorax.
951 buildinstall_allow_reuse = False
953 (bool) – When set to True, Pungi will try to reuse buildinstall
954 results from old compose specified by --old-composes.
955 buildinstall_packages
957 (list) – Additional packages to be installed in the runroot en‐
958 vironment where lorax will run to create installer. Format:
959 [(variant_uid_regex, {arch|*: [package_globs]})].
960 Example
962 buildinstall_method = "lorax"
963 # Enables macboot on x86_64 for all variants and builds upgrade images
965 # everywhere.
966 lorax_options = [
967 ("^.*$", {
968 "x86_64": {
969 "nomacboot": False
970 }
971 "*": {
972 "noupgrade": False
973 }
974 })
975 ]
976 # Don't run buildinstall phase for Modular variant
978 buildinstall_skip = [
979 ('^Modular', {
980 '*': True
981 })
982 ]
983 # Add another repository for lorax to install packages from
985 lorax_extra_sources = [
986 ('^Simple$', {
987 '*': 'https://example.com/repo/$basearch/',
988 })
989 ]
990 # Additional packages to be installed in the Koji runroot environment where
992 # lorax will run.
993 buildinstall_packages = [
994 ('^Simple$', {
995 '*': ['dummy-package'],
996 })
997 ]
998 NOTE:
1000 It is advised to run buildinstall (lorax) in koji, i.e. with runroot
1001 enabled for clean build environments, better logging, etc.
1002 WARNING:
1004 Lorax installs RPMs into a chroot. This involves running %post
1005 scriptlets and they frequently run executables in the chroot. If
1006 we’re composing for multiple architectures, we must use runroot for
1007 this reason.
1008 Gather Settings
1010 Options
1011 gather_method [mandatory]
1012 (str*|*dict) – Options are deps, nodeps and hybrid. Specifies
1013 whether and how package dependencies should be pulled in. Pos‐
1014 sible configuration can be one value for all variants, or if
1015 configured per-variant it can be a simple string hybrid or a a
1016 dictionary mapping source type to a value of deps or nodeps.
1017 Make sure only one regex matches each variant, as there is no
1018 guarantee which value will be used if there are multiple match‐
1019 ing ones. All used sources must have a configured method unless
1020 hybrid solving is used.
1021 gather_fulltree = False
1023 (bool) – When set to True all RPMs built from an SRPM will al‐
1024 ways be included. Only use when gather_method = "deps".
1025 gather_selfhosting = False
1027 (bool) – When set to True, Pungi will build a self-hosting tree
1028 by following build dependencies. Only use when gather_method =
1029 "deps".
1030 gather_allow_reuse = False
1032 (bool) – When set to True, Pungi will try to reuse gather re‐
1033 sults from old compose specified by --old-composes.
1034 greedy_method = none
1036 (str) – This option controls how package requirements are satis‐
1037 fied in case a particular Requires has multiple candidates.
1038 • none – the best packages is selected to satisfy the dependency
1040 and only that one is pulled into the compose
1041 • all – packages that provide the symbol are pulled in
1043 • build – the best package is selected, and then all packages
1045 from the same build that provide the symbol are pulled in
1046 NOTE:
1048 As an example let’s work with this situation: a package in
1049 the compose has Requires: foo. There are three packages with
1050 Provides: foo: pkg-a, pkg-b-provider-1 and pkg-b-provider-2.
1051 The pkg-b-* packages are build from the same source package.
1052 Best match determines pkg-b-provider-1 as best matching pack‐
1053 age.
1054 • With greedy_method = "none" only pkg-b-provider-1 will be
1056 pulled in.
1057 • With greedy_method = "all" all three packages will be
1059 pulled in.
1060 • With greedy_method = "build" pkg-b-provider-1 and
1062 pkg-b-provider-2 will be pulled in.
1063 gather_backend
1065 (str) –This changes the entire codebase doing dependency solv‐
1066 ing, so it can change the result in unpredictable ways.
1067 On Python 2, the choice is between yum or dnf and defaults to
1069 yum. On Python 3 dnf is the only option and default.
1070 Particularly the multilib work is performed differently by using
1072 python-multilib library. Please refer to multilib option to see
1073 the differences.
1074 See also: the repoclosure_backend setting for Pungi’s repoclo‐
1076 sure phase.
1077 multilib
1079 (list) – mapping of variant regexes and arches to list of multi‐
1080 lib methods
1081 Available methods are:
1083 • none – no package matches this method
1085 • all – all packages match this method
1087 • runtime – packages that install some shared object file
1089 (*.so.*) will match.
1090 • devel – packages whose name ends with -devel or
1092 --static suffix will be matched. When dnf is used, this
1093 method automatically enables runtime method as well.
1094 With yum backend this method also uses a hardcoded
1095 blacklist and whitelist.
1096 • kernel – packages providing kernel or kernel-devel
1098 match this method (only in yum backend)
1099 • yaboot – only yaboot package on ppc arch matches this
1101 (only in yum backend)
1102 additional_packages
1104 (list) – additional packages to be included in a variant and ar‐
1105 chitecture; format: [(variant_uid_regex, {arch|*: [pack‐
1106 age_globs]})]
1107 In contrast to the comps_file setting, the additional_packages
1109 setting merely adds the list of packages to the compose. When a
1110 package is in a comps group, it is visible to users via dnf
1111 groupinstall and Anaconda’s Groups selection, but addi‐
1112 tional_packages does not affect DNF groups.
1113 The packages specified here are matched against RPM names, not
1115 any other provides in the package nor the name of source pack‐
1116 age. Shell globbing is used, so wildcards are possible. The
1117 package can be specified as name only or name.arch.
1118 With dnf gathering backend, you can specify a debuginfo package
1120 to be included. This is meant to include a package if autodetec‐
1121 tion does not get it. If you add a debuginfo package that does
1122 not have anything else from the same build included in the com‐
1123 pose, the sources will not be pulled in.
1124 If you list a package in additional_packages but Pungi cannot
1126 find it (for example, it’s not available in the Koji tag), Pungi
1127 will log a warning in the “work” or “logs” directories and con‐
1128 tinue without aborting.
1129 Example: This configuration will add all packages in a Koji tag
1131 to an “Everything” variant:
1132 additional_packages = [
1134 ('^Everything$', {
1135 '*': [
1136 '*',
1137 ],
1138 })
1139 ]
1140 filter_packages
1142 (list) – packages to be excluded from a variant and architec‐
1143 ture; format: [(variant_uid_regex, {arch|*: [package_globs]})]
1144 See additional_packages for details about package specification.
1146 filter_modules
1148 (list) – modules to be excluded from a variant and architecture;
1149 format: [(variant_uid_regex, {arch|*: [name:stream]})]
1150 Both name and stream can use shell-style globs. If stream is
1152 omitted, all streams are removed.
1153 This option only applies to modules taken from Koji tags, not
1155 modules explicitly listed in variants XML without any tags.
1156 filter_system_release_packages
1158 (bool) – for each variant, figure out the best system release
1159 package and filter out all others. This will not work if a vari‐
1160 ant needs more than one system release package. In such case,
1161 set this option to False.
1162 gather_prepopulate = None
1164 (scm_dict) – If specified, you can use this to add additional
1165 packages. The format of the file pointed to by this option is a
1166 JSON mapping {variant_uid: {arch: {build: [package]}}}. Packages
1167 added through this option can not be removed by filter_packages.
1168 multilib_blacklist
1170 (dict) – multilib blacklist; format: {arch|*: [package_globs]}.
1171 See additional_packages for details about package specification.
1173 multilib_whitelist
1175 (dict) – multilib blacklist; format: {arch|*: [package_names]}.
1176 The whitelist must contain exact package names; there are no
1177 wildcards or pattern matching.
1178 gather_lookaside_repos = []
1180 (list) – lookaside repositories used for package gathering; for‐
1181 mat: [(variant_uid_regex, {arch|*: [repo_urls]})]
1182 The repo_urls are passed to the depsolver, which can use pack‐
1184 ages in the repos for satisfying dependencies, but the packages
1185 themselves are not pulled into the compose. The repo_urls can
1186 contain $basearch variable, which will be substituted with
1187 proper value by the depsolver.
1188 The repo_urls are used by repoclosure too, but it can’t parse
1190 $basearch currently and that will cause Repoclosure phase
1191 crashed. repoclosure_strictness option could be used to stop
1192 running repoclosure.
1193 Please note that * as a wildcard matches all architectures but
1195 src.
1196 hashed_directories = False
1198 (bool) – put packages into “hashed” directories, for example
1199 Packages/k/kernel-4.0.4-301.fc22.x86_64.rpm
1200 check_deps = True
1202 (bool) – Set to False if you don’t want the compose to abort
1203 when some package has broken dependencies.
1204 require_all_comps_packages = False
1206 (bool) – Set to True to abort compose when package mentioned in
1207 comps file can not be found in the package set. When disabled
1208 (the default), such cases are still reported as warnings in the
1209 log.
1210 gather_source_mapping
1212 (str) – JSON mapping with initial packages for the compose. The
1213 value should be a path to JSON file with following mapping:
1214 {variant: {arch: {rpm_name: [rpm_arch|None]}}}. Relative paths
1215 are interpreted relative to the location of main config file.
1216 gather_profiler = False
1218 (bool) – When set to True the gather tool will produce addi‐
1219 tional performance profiling information at the end of its logs.
1220 Only takes effect when gather_backend = "dnf".
1221 variant_as_lookaside
1223 (list) – a variant/variant mapping that tells one or more vari‐
1224 ants in compose has other variant(s) in compose as a lookaside.
1225 Only top level variants are supported (not addons/layered prod‐
1226 ucts). Format: [(variant_uid, variant_uid)]
1227 Example
1229 gather_method = "deps"
1230 greedy_method = "build"
1231 check_deps = False
1232 hashed_directories = True
1233 gather_method = {
1235 "^Everything$": {
1236 "comps": "deps" # traditional content defined by comps groups
1237 },
1238 "^Modular$": {
1239 "module": "nodeps" # Modules do not need dependencies
1240 },
1241 "^Mixed$": { # Mixed content in one variant
1242 "comps": "deps",
1243 "module": "nodeps"
1244 }
1245 "^OtherMixed$": "hybrid", # Using hybrid depsolver
1246 }
1247 additional_packages = [
1249 # bz#123456
1250 ('^(Workstation|Server)$', {
1251 '*': [
1252 'grub2',
1253 'kernel',
1254 ],
1255 }),
1256 ]
1257 filter_packages = [
1259 # bz#111222
1260 ('^.*$', {
1261 '*': [
1262 'kernel-doc',
1263 ],
1264 }),
1265 ]
1266 multilib = [
1268 ('^Server$', {
1269 'x86_64': ['devel', 'runtime']
1270 })
1271 ]
1272 multilib_blacklist = {
1274 "*": [
1275 "gcc",
1276 ],
1277 }
1278 multilib_whitelist = {
1280 "*": [
1281 "alsa-plugins-*",
1282 ],
1283 }
1284 # gather_lookaside_repos = [
1286 # ('^.*$', {
1287 # '*': [
1288 # "https://dl.fedoraproject.org/pub/fedora/linux/releases/22/Everything/$basearch/os/",
1289 # ],
1290 # 'x86_64': [
1291 # "https://dl.fedoraproject.org/pub/fedora/linux/releases/22/Everything/source/SRPMS/",
1292 # ]
1293 # }),
1294 # ]
1295 NOTE:
1297 It is a good practice to attach bug/ticket numbers to addi‐
1298 tional_packages, filter_packages, multilib_blacklist and multi‐
1299 lib_whitelist to track decisions.
1300 Koji Settings
1302 Options
1303 koji_profile
1304 (str) – koji profile name. This tells Pungi how to communicate
1305 with your chosen Koji instance. See Koji’s documentation about
1306 profiles for more information about how to set up your Koji
1307 client profile. In the examples, the profile name is “koji”,
1308 which points to Fedora’s koji.fedoraproject.org.
1309 global_runroot_method
1311 (str) – global runroot method to use. If runroot_method is set
1312 per Pungi phase using a dictionary, this option defines the de‐
1313 fault runroot method for phases not mentioned in the run‐
1314 root_method dictionary.
1315 runroot_method
1317 (str*|*dict) – Runroot method to use. It can further specify the
1318 runroot method in case the runroot is set to True.
1319 Available methods are:
1321 • local – runroot tasks are run locally
1323 • koji – runroot tasks are run in Koji
1325 • openssh – runroot tasks are run on remote machine con‐
1327 nected using OpenSSH. The runroot_ssh_hostnames for
1328 each architecture must be set and the user under which
1329 Pungi runs must be configured to login as run‐
1330 root_ssh_username using the SSH key.
1331 The runroot method can also be set per Pungi phase using the
1333 dictionary with phase name as key and runroot method as value.
1334 The default runroot method is in this case defined by the
1335 global_runroot_method option.
1336 Example
1338 global_runroot_method = "koji"
1339 runroot_method = {
1340 "createiso": "local"
1341 }
1342 runroot_channel
1344 (str) – name of koji channel
1345 runroot_tag
1347 (str) – name of koji build tag used for runroot
1348 runroot_weights
1350 (dict) – customize task weights for various runroot tasks. The
1351 values in the mapping should be integers, the keys can be se‐
1352 lected from the following list. By default no weight is assigned
1353 and Koji picks the default one according to policy.
1354 • buildinstall
1356 • createiso
1358 • ostree
1360 • ostree_installer
1362 Example
1364 koji_profile = "koji"
1365 runroot_channel = "runroot"
1366 runroot_tag = "f23-build"
1367 Runroot “openssh” method settings
1369 Options
1370 runroot_ssh_username
1371 (str) – For openssh runroot method, configures the username used
1372 to login the remote machine to run the runroot task. Defaults to
1373 “root”.
1374 runroot_ssh_hostnames
1376 (dict) – For openssh runroot method, defines the hostname for
1377 each architecture on which the runroot task should be running.
1378 Format: {"x86_64": "runroot-x86-64.localhost.tld", ...}
1379 runroot_ssh_init_template
1381 (str) [optional] – For openssh runroot method, defines the com‐
1382 mand to initializes the runroot task on the remote machine. This
1383 command is executed as first command for each runroot task exe‐
1384 cuted.
1385 The command can print a string which is then available as {run‐
1387 root_key} for other SSH commands. This string might be used to
1388 keep the context across different SSH commands executed for sin‐
1389 gle runroot task.
1390 The goal of this command is setting up the environment for real
1392 runroot commands. For example preparing the unique mock environ‐
1393 ment, mounting the desired file-systems, …
1394 The command string can contain following variables which are re‐
1396 placed by the real values before executing the init command:
1397 • {runroot_tag} - Tag to initialize the runroot environment
1399 from.
1400 When not set, no init command is executed.
1402 runroot_ssh_install_packages_template
1404 (str) [optional] – For openssh runroot method, defines the tem‐
1405 plate for command to install the packages requested to run the
1406 runroot task.
1407 The template string can contain following variables which are
1409 replaced by the real values before executing the install com‐
1410 mand:
1411 • {runroot_key} - Replaced with the string returned by run‐
1413 root_ssh_init_template if used. This can be used to keep the
1414 track of context of SSH commands belonging to single runroot
1415 task.
1416 • {packages} - White-list separated list of packages to install.
1418 Example (The {runroot_key} is expected to be set to mock config
1420 file using the runroot_ssh_init_template command.): "mock -r
1421 {runroot_key} --install {packages}"
1422 When not set, no command to install packages on remote machine
1424 is executed.
1425 runroot_ssh_run_template
1427 (str) [optional] – For openssh runroot method, defines the tem‐
1428 plate for the main runroot command.
1429 The template string can contain following variables which are
1431 replaced by the real values before executing the install com‐
1432 mand:
1433 • {runroot_key} - Replaced with the string returned by run‐
1435 root_ssh_init_template if used. This can be used to keep the
1436 track of context of SSH commands belonging to single runroot
1437 task.
1438 • {command} - Command to run.
1440 Example (The {runroot_key} is expected to be set to mock config
1442 file using the runroot_ssh_init_template command.): "mock -r
1443 {runroot_key} chroot -- {command}"
1444 When not set, the runroot command is run directly.
1446 Extra Files Settings
1448 Options
1449 extra_files
1450 (list) – references to external files to be placed in os/ direc‐
1451 tory and media; format: [(variant_uid_regex, {arch|*:
1452 [scm_dict]})]. See scm_support for details. If the dict speci‐
1453 fies a target key, an additional subdirectory will be used.
1454 Example
1456 extra_files = [
1457 ('^.*$', {
1458 '*': [
1459 # GPG keys
1460 {
1461 "scm": "rpm",
1462 "repo": "fedora-repos",
1463 "branch": None,
1464 "file": [
1465 "/etc/pki/rpm-gpg/RPM-GPG-KEY-22-fedora",
1466 ],
1467 "target": "",
1468 },
1469 # GPL
1470 {
1471 "scm": "git",
1472 "repo": "https://pagure.io/pungi-fedora",
1473 "branch": None,
1474 "file": [
1475 "GPL",
1476 ],
1477 "target": "",
1478 },
1479 ],
1480 }),
1481 ]
1482 Extra Files Metadata
1484 If extra files are specified a metadata file, extra_files.json, is
1485 placed in the os/ directory and media. The checksums generated are de‐
1486 termined by media_checksums option. This metadata file is in the for‐
1487 mat:
1488 {
1490 "header": {"version": "1.0},
1491 "data": [
1492 {
1493 "file": "GPL",
1494 "checksums": {
1495 "sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
1496 },
1497 "size": 18092
1498 },
1499 {
1500 "file": "release-notes/notes.html",
1501 "checksums": {
1502 "sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
1503 },
1504 "size": 1120
1505 }
1506 ]
1507 }
1508 CreateISO Settings
1510 Options
1511 createiso_skip = False
1512 (list) – mapping that defines which variants and arches to skip
1513 during createiso; format: [(variant_uid_regex, {arch|*: True})]
1514 createiso_max_size
1516 (list) – mapping that defines maximum expected size for each
1517 variant and arch. If the ISO is larger than the limit, a warning
1518 will be issued.
1519 Format: [(variant_uid_regex, {arch|*: number})]
1521 createiso_max_size_is_strict
1523 (list) – Set the value to True to turn the warning from cre‐
1524 ateiso_max_size into a hard error that will abort the compose.
1525 If there are multiple matches in the mapping, the check will be
1526 strict if at least one match says so.
1527 Format: [(variant_uid_regex, {arch|*: bool})]
1529 create_jigdo = False
1531 (bool) – controls the creation of jigdo from ISO
1532 create_optional_isos = False
1534 (bool) – when set to True, ISOs will be created even for op‐
1535 tional variants. By default only variants with type variant or
1536 layered-product will get ISOs.
1537 createiso_break_hardlinks = False
1539 (bool) – when set to True, all files that should go on the ISO
1540 and have a hardlink will be first copied into a staging direc‐
1541 tory. This should work around a bug in genisoimage including in‐
1542 correct link count in the image, but it is at the cost of having
1543 to copy a potentially significant amount of data.
1544 The staging directory is deleted when ISO is successfully cre‐
1546 ated. In that case the same task to create the ISO will not be
1547 re-runnable.
1548 createiso_use_xorrisofs = False
1550 (bool) – when set to True, use xorrisofs for creating ISOs in‐
1551 stead of genisoimage.
1552 iso_size = 4700000000
1554 (int|str) – size of ISO image. The value should either be an in‐
1555 teger meaning size in bytes, or it can be a string with k, M, G
1556 suffix (using multiples of 1024).
1557 iso_level
1559 (int) [optional] – Set the ISO9660 conformance level. Valid num‐
1560 bers are 1 to 4.
1561 split_iso_reserve = 10MiB
1563 (int|str) – how much free space should be left on each disk. The
1564 format is the same as for iso_size option.
1565 iso_hfs_ppc64le_compatible = True
1567 (bool) – when set to False, the Apple/HFS compatibility is
1568 turned off for ppc64le ISOs. This option only makes sense for
1569 bootable products, and affects images produced in createiso and
1570 extra_isos phases.
1571 NOTE:
1573 Source architecture needs to be listed explicitly. Excluding ‘*’
1574 applies only on binary arches. Jigdo causes significant increase of
1575 time to ISO creation.
1576 Example
1578 createiso_skip = [
1579 ('^Workstation$', {
1580 '*': True,
1581 'src': True
1582 }),
1583 ]
1584 Automatic generation of version and release
1586 Version and release values for certain artifacts can be generated auto‐
1587 matically based on release version, compose label, date, type and
1588 respin. This can be used to shorten the config and keep it the same for
1589 multiple uses.
1590┌───────────────────────┬───────────────┬──────────┬──────────┬────────┬──────────────┐
1592│Compose ID │ Label │ Version │ Date │ Respin │ Release │
1593├───────────────────────┼───────────────┼──────────┼──────────┼────────┼──────────────┤
1594│F-Rawhide-20170406.n.0 │ - │ Rawhide │ 20170406 │ 0 │ 20170406.n.0 │
1595├───────────────────────┼───────────────┼──────────┼──────────┼────────┼──────────────┤
1596│F-26-20170329.1 │ Alpha-1.6 │ 26_Alpha │ 20170329 │ 1 │ 1.6 │
1597├───────────────────────┼───────────────┼──────────┼──────────┼────────┼──────────────┤
1598│F-Atomic-25-20170407.0 │ RC-20170407.0 │ 25 │ 20170407 │ 0 │ 20170407.0 │
1599├───────────────────────┼───────────────┼──────────┼──────────┼────────┼──────────────┤
1600│F-Atomic-25-20170407.0 │ - │ 25 │ 20170407 │ 0 │ 20170407.0 │
1601└───────────────────────┴───────────────┴──────────┴──────────┴────────┴──────────────┘
1602 All non-RC milestones from label get appended to the version. For re‐
1604 lease either label is used or date, type and respin.
1605 Common options for Live Images, Live Media and Image Build
1607 All images can have ksurl, version, release and target specified. Since
1608 this can create a lot of duplication, there are global options that can
1609 be used instead.
1610 For each of the phases, if the option is not specified for a particular
1612 deliverable, an option named <PHASE_NAME>_<OPTION> is checked. If that
1613 is not specified either, the last fallback is global_<OPTION>. If even
1614 that is unset, the value is considered to not be specified.
1615 The kickstart URL is configured by these options.
1617 • global_ksurl – global fallback setting
1619 • live_media_ksurl
1621 • image_build_ksurl
1623 • live_images_ksurl
1625 Target is specified by these settings.
1627 • global_target – global fallback setting
1629 • live_media_target
1631 • image_build_target
1633 • live_images_target
1635 • osbuild_target
1637 Version is specified by these options. If no version is set, a default
1639 value will be provided according to automatic versioning.
1640 • global_version – global fallback setting
1642 • live_media_version
1644 • image_build_version
1646 • live_images_version
1648 • osbuild_version
1650 Release is specified by these options. If set to a magic value to !RE‐
1652 LEASE_FROM_LABEL_DATE_TYPE_RESPIN, a value will be generated according
1653 to automatic versioning.
1654 • global_release – global fallback setting
1656 • live_media_release
1658 • image_build_release
1660 • live_images_release
1662 • osbuild_release
1664 Each configuration block can also optionally specify a failable key.
1666 For live images it should have a boolean value. For live media and im‐
1667 age build it should be a list of strings containing architectures that
1668 are optional. If any deliverable fails on an optional architecture, it
1669 will not abort the whole compose. If the list contains only "*", all
1670 arches will be substituted.
1671 Live Images Settings
1673 live_images
1674 (list) – Configuration for the particular image. The elements of
1675 the list should be tuples (variant_uid_regex, {arch|*: config}).
1676 The config should be a dict with these keys:
1677 • kickstart (str)
1679 • ksurl (str) [optional] – where to get the kickstart from
1681 • name (str)
1683 • version (str)
1685 • target (str)
1687 • repo (str|[str]) – repos specified by URL or variant UID
1689 • specfile (str) – for images wrapped in RPM
1691 • scratch (bool) – only RPM-wrapped images can use scratch
1693 builds, but by default this is turned off
1694 • type (str) – what kind of task to start in Koji. Defaults
1696 to live meaning koji spin-livecd will be used. Alternative
1697 option is appliance corresponding to koji spin-appliance.
1698 • sign (bool) – only RPM-wrapped images can be signed
1700 live_images_no_rename
1702 (bool) – When set to True, filenames generated by Koji will be
1703 used. When False, filenames will be generated based on im‐
1704 age_name_format configuration option.
1705 Live Media Settings
1707 live_media
1708 (dict) – configuration for koji spin-livemedia; format: {vari‐
1709 ant_uid_regex: [{opt:value}]}
1710 Required options:
1712 • name (str)
1714 • version (str)
1716 • arches ([str]) – what architectures to build the media for;
1718 by default uses all arches for the variant.
1719 • kickstart (str) – name of the kickstart file
1721 Available options:
1723 • ksurl (str)
1725 • ksversion (str)
1727 • scratch (bool)
1729 • target (str)
1731 • release (str) – a string with the release, or !RE‐
1733 LEASE_FROM_LABEL_DATE_TYPE_RESPIN to automatically generate
1734 a suitable value. See automatic versioning for details.
1735 • skip_tag (bool)
1737 • repo (str|[str]) – repos specified by URL or variant UID
1739 • title (str)
1741 • install_tree_from (str) – variant to take install tree from
1743 • nomacboot (bool)
1745 Image Build Settings
1747 image_build
1748 (dict) – config for koji image-build; format: {vari‐
1749 ant_uid_regex: [{opt: value}]}
1750 By default, images will be built for each binary arch valid for
1752 the variant. The config can specify a list of arches to narrow
1753 this down.
1754 NOTE:
1756 Config can contain anything what is accepted by koji image-build
1757 --config configfile.ini
1758 Repo can be specified either as a string or a list of strings. It
1760 will be automatically transformed into format suitable for koji. A
1761 repo for the currently built variant will be added as well.
1762 If you explicitly set release to !RELEASE_FROM_LA‐
1764 BEL_DATE_TYPE_RESPIN, it will be replaced with a value generated as
1765 described in automatic versioning.
1766 If you explicitly set release to !RELEASE_FROM_DATE_RESPIN, it will
1768 be replaced with a value generated as described in automatic ver‐
1769 sioning.
1770 If you explicitly set version to !VERSION_FROM_VERSION, it will be
1772 replaced with a value generated as described in automatic version‐
1773 ing.
1774 Please don’t set install_tree. This gets automatically set by pungi
1776 based on current variant. You can use install_tree_from key to use
1777 install tree from another variant.
1778 Both the install tree and repos can use one of following formats:
1780 • URL to the location
1782 • name of variant in the current compose
1784 • absolute path on local filesystem (which will be translated
1786 using configured mappings or used unchanged, in which case you
1787 have to ensure the koji builders can access it)
1788 You can set either a single format, or a list of formats. For avail‐
1790 able values see help output for koji image-build command.
1791 If ksurl ends with #HEAD, Pungi will figure out the SHA1 hash of
1793 current HEAD and use that instead.
1794 Setting scratch to True will run the koji tasks as scratch builds.
1796 Example
1798 image_build = {
1799 '^Server$': [
1800 {
1801 'image-build': {
1802 'format': ['docker', 'qcow2']
1803 'name': 'fedora-qcow-and-docker-base',
1804 'target': 'koji-target-name',
1805 'ksversion': 'F23', # value from pykickstart
1806 'version': '23',
1807 # correct SHA1 hash will be put into the URL below automatically
1808 'ksurl': 'https://git.fedorahosted.org/git/spin-kickstarts.git?somedirectoryifany#HEAD',
1809 'kickstart': "fedora-docker-base.ks",
1810 'repo': ["http://someextrarepos.org/repo", "ftp://rekcod.oi/repo"],
1811 'distro': 'Fedora-20',
1812 'disk_size': 3,
1813 # this is set automatically by pungi to os_dir for given variant
1815 # 'install_tree': 'http://somepath',
1816 },
1817 'factory-parameters': {
1818 'docker_cmd': "[ '/bin/bash' ]",
1819 'docker_env': "[ 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' ]",
1820 'docker_labels': "{'Name': 'fedora-docker-base', 'License': u'GPLv2', 'RUN': 'docker run -it --rm ${OPT1} --privileged -v \`pwd\`:/atomicapp -v /run:/run -v /:/host --net=host --name ${NAME} -e NAME=${NAME} -e IMAGE=${IMAGE} ${IMAGE} -v ${OPT2} run ${OPT3} /atomicapp', 'Vendor': 'Fedora Project', 'Version': '23', 'Architecture': 'x86_64' }",
1821 }
1822 },
1823 {
1824 'image-build': {
1825 'format': ['docker', 'qcow2']
1826 'name': 'fedora-qcow-and-docker-base',
1827 'target': 'koji-target-name',
1828 'ksversion': 'F23', # value from pykickstart
1829 'version': '23',
1830 # correct SHA1 hash will be put into the URL below automatically
1831 'ksurl': 'https://git.fedorahosted.org/git/spin-kickstarts.git?somedirectoryifany#HEAD',
1832 'kickstart': "fedora-docker-base.ks",
1833 'repo': ["http://someextrarepos.org/repo", "ftp://rekcod.oi/repo"],
1834 'distro': 'Fedora-20',
1835 'disk_size': 3,
1836 # this is set automatically by pungi to os_dir for given variant
1838 # 'install_tree': 'http://somepath',
1839 }
1840 },
1841 {
1842 'image-build': {
1843 'format': 'qcow2',
1844 'name': 'fedora-qcow-base',
1845 'target': 'koji-target-name',
1846 'ksversion': 'F23', # value from pykickstart
1847 'version': '23',
1848 'ksurl': 'https://git.fedorahosted.org/git/spin-kickstarts.git?somedirectoryifany#HEAD',
1849 'kickstart': "fedora-docker-base.ks",
1850 'distro': 'Fedora-23',
1851 # only build this type of image on x86_64
1853 'arches': ['x86_64']
1854 # Use install tree and repo from Everything variant.
1856 'install_tree_from': 'Everything',
1857 'repo': ['Everything'],
1858 # Set release automatically.
1860 'release': '!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN',
1861 }
1862 }
1863 ]
1864 }
1865 OSBuild Composer for building images
1867 osbuild
1868 (dict) – configuration for building images in OSBuild Composer
1869 service fronted by a Koji plugin. Pungi will trigger a Koji task
1870 delegating to the OSBuild Composer, which will build the image,
1871 import it to Koji via content generators.
1872 Format: {variant_uid_regex: [{...}]}.
1874 Required keys in the configuration dict:
1876 • name – name of the Koji package
1878 • distro – image for which distribution should be build TODO ex‐
1880 amples
1881 • image_type – a list of image types to build (e.g. qcow2)
1883 Optional keys:
1885 • target – which build target to use for the task. Either this
1887 option or the global osbuild_target is required.
1888 • version – version for the final build (as a string). This op‐
1890 tion is required if the global osbuild_version is not speci‐
1891 fied.
1892 • release – release part of the final NVR. If neither this op‐
1894 tion nor the global osbuild_release is set, Koji will automat‐
1895 ically generate a value.
1896 • repo – a list of repository URLs from which to consume pack‐
1898 ages for building the image. By default only the variant
1899 repository is used.
1900 • arches – list of architectures for which to build the image.
1902 By default, the variant arches are used. This option can only
1903 restrict it, not add a new one.
1904 NOTE:
1906 There is initial support for having this task as failable without
1907 aborting the whole compose. This can be enabled by setting "fail‐
1908 able": ["*"] in the config for the image. It is an on/off switch
1909 without granularity per arch.
1910 Image container
1912 This phase supports building containers in OSBS that embed an image
1913 created in the same compose. This can be useful for delivering the im‐
1914 age to users running in containerized environments.
1915 Pungi will start a buildContainer task in Koji with configured source
1917 repository. The Dockerfile can expect that a repo file will be injected
1918 into the container that defines a repo named image-to-include, and its
1919 baseurl will point to the image to include. It is possible to extract
1920 the URL with a command like dnf config-manager --dump image-to-include
1921 | awk '/baseurl =/{print $3}'`
1922 image_container
1924 (dict) – configuration for building containers embedding an im‐
1925 age.
1926 Format: {variant_uid_regex: [{...}]}.
1928 The inner object will define a single container. These keys are
1930 required:
1931 • url, target, git_branch. See OSBS section for definition of
1933 these.
1934 • image_spec – (object) A string mapping of filters used to se‐
1936 lect the image to embed. All images listed in metadata for the
1937 variant will be processed. The keys of this filter are used to
1938 select metadata fields for the image, and values are regular
1939 expression that need to match the metadata value.
1940 The filter should match exactly one image.
1942 Example config
1944 image_container = {
1945 "^Server$": [{
1946 "url": "git://example.com/dockerfiles.git?#HEAD",
1947 "target": "f24-container-candidate",
1948 "git_branch": "f24",
1949 "image_spec": {
1950 "format": "qcow2",
1951 "arch": "x86_64",
1952 "path": ".*/guest-image-.*$",
1953 }
1954 }]
1955 }
1956 OSTree Settings
1958 The ostree phase of Pungi can create and update ostree repositories.
1959 This is done by running rpm-ostree compose in a Koji runroot environ‐
1960 ment. The ostree repository itself is not part of the compose and
1961 should be located in another directory. Any new packages in the compose
1962 will be added to the repository with a new commit.
1963 ostree (dict) – a mapping of configuration for each. The format should
1965 be {variant_uid_regex: config_dict}. It is possible to use a
1966 list of configuration dicts as well.
1967 The configuration dict for each variant arch pair must have
1969 these keys:
1970 • treefile – (str) Filename of configuration for rpm-ostree.
1972 • config_url – (str) URL for Git repository with the treefile.
1974 • repo – (str|dict|[str|dict]) repos specified by URL or variant
1976 UID or a dict of repo options, baseurl is required in the
1977 dict.
1978 • ostree_repo – (str) Where to put the ostree repository
1980 These keys are optional:
1982 • keep_original_sources – (bool) Keep the existing source repos
1984 in the tree config file. If not enabled, all the original
1985 source repos will be removed from the tree config file.
1986 • config_branch – (str) Git branch of the repo to use. Defaults
1988 to master.
1989 • arches – ([str]) List of architectures for which to update os‐
1991 tree. There will be one task per architecture. By default all
1992 architectures in the variant are used.
1993 • failable – ([str]) List of architectures for which this deliv‐
1995 erable is not release blocking.
1996 • update_summary – (bool) Update summary metadata after tree
1998 composing. Defaults to False.
1999 • force_new_commit – (bool) Do not use rpm-ostree’s built-in
2001 change detection. Defaults to False.
2002 • version – (str) Version string to be added as versioning meta‐
2004 data. If this option is set to !OSTREE_VERSION_FROM_LA‐
2005 BEL_DATE_TYPE_RESPIN, a value will be generated automatically
2006 as $VERSION.$RELEASE. If this option is set to !VER‐
2007 SION_FROM_VERSION_DATE_RESPIN, a value will be generated auto‐
2008 matically as $VERSION.$DATE.$RESPIN. See how those values are
2009 created.
2010 • tag_ref – (bool, default True) If set to False, a git refer‐
2012 ence will not be created.
2013 • ostree_ref – (str) To override value ref from treefile.
2015 Example config
2017 ostree = {
2018 "^Atomic$": {
2019 "treefile": "fedora-atomic-docker-host.json",
2020 "config_url": "https://git.fedorahosted.org/git/fedora-atomic.git",
2021 "repo": [
2022 "Server",
2023 "http://example.com/repo/x86_64/os",
2024 {"baseurl": "Everything"},
2025 {"baseurl": "http://example.com/linux/repo", "exclude": "systemd-container"},
2026 ],
2027 "keep_original_sources": True,
2028 "ostree_repo": "/mnt/koji/compose/atomic/Rawhide/",
2029 "update_summary": True,
2030 # Automatically generate a reasonable version
2031 "version": "!OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN",
2032 # Only run this for x86_64 even if Atomic has more arches
2033 "arches": ["x86_64"],
2034 }
2035 }
2036 ostree_use_koji_plugin = False
2038 (bool) – When set to True, the Koji pungi_ostree task will be
2039 used to execute rpm-ostree instead of runroot. Use only if the
2040 Koji instance has the pungi_ostree plugin installed.
2041 Ostree Installer Settings
2043 The ostree_installer phase of Pungi can produce installer image
2044 bundling an OSTree repository. This always runs in Koji as a runroot
2045 task.
2046 ostree_installer
2048 (dict) – a variant/arch mapping of configuration. The format
2049 should be [(variant_uid_regex, {arch|*: config_dict})].
2050 The configuration dict for each variant arch pair must have this
2052 key:
2053 These keys are optional:
2055 • repo – (str|[str]) repos specified by URL or variant UID
2057 • release – (str) Release value to set for the installer image.
2059 Set to !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN to generate the
2060 value automatically.
2061 • failable – ([str]) List of architectures for which this deliv‐
2063 erable is not release blocking.
2064 These optional keys are passed to lorax to customize the build.
2066 • installpkgs – ([str])
2068 • add_template – ([str])
2070 • add_arch_template – ([str])
2072 • add_template_var – ([str])
2074 • add_arch_template_var – ([str])
2076 • rootfs_size – ([str])
2078 • template_repo – (str) Git repository with extra templates.
2080 • template_branch – (str) Branch to use from template_repo.
2082 The templates can either be absolute paths, in which case they
2084 will be used as configured; or they can be relative paths, in
2085 which case template_repo needs to point to a Git repository from
2086 which to take the templates.
2087 If the templates need to run with additional dependencies, that
2089 can be configured with the optional key:
2090 • extra_runroot_pkgs – ([str])
2092 • skip_branding – (bool) Stops lorax to install packages with
2094 branding. Defaults to False.
2095 ostree_installer_overwrite = False
2097 (bool) – by default if a variant including OSTree installer also
2098 creates regular installer images in buildinstall phase, there
2099 will be conflicts (as the files are put in the same place) and
2100 Pungi will report an error and fail the compose.
2101 With this option it is possible to opt-in for the overwriting.
2103 The traditional boot.iso will be in the iso/ subdirectory.
2104 ostree_installer_use_koji_plugin = False
2106 (bool) – When set to True, the Koji pungi_buildinstall task will
2107 be used to execute Lorax instead of runroot. Use only if the
2108 Koji instance has the pungi_buildinstall plugin installed.
2109 Example config
2111 ostree_installer = [
2112 ("^Atomic$", {
2113 "x86_64": {
2114 "repo": [
2115 "Everything",
2116 "https://example.com/extra-repo1.repo",
2117 "https://example.com/extra-repo2.repo",
2118 ],
2119 "release": "!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN",
2120 "installpkgs": ["fedora-productimg-atomic"],
2121 "add_template": ["atomic-installer/lorax-configure-repo.tmpl"],
2122 "add_template_var": [
2123 "ostree_osname=fedora-atomic",
2124 "ostree_ref=fedora-atomic/Rawhide/x86_64/docker-host",
2125 ],
2126 "add_arch_template": ["atomic-installer/lorax-embed-repo.tmpl"],
2127 "add_arch_template_var": [
2128 "ostree_repo=https://kojipkgs.fedoraproject.org/compose/atomic/Rawhide/",
2129 "ostree_osname=fedora-atomic",
2130 "ostree_ref=fedora-atomic/Rawhide/x86_64/docker-host",
2131 ]
2132 'template_repo': 'https://git.fedorahosted.org/git/spin-kickstarts.git',
2133 'template_branch': 'f24',
2134 }
2135 })
2136 ]
2137 OSBS Settings
2139 Pungi can build container images in OSBS. The build is initiated
2140 through Koji container-build plugin. The base image will be using RPMs
2141 from the current compose and a Dockerfile from specified Git reposi‐
2142 tory.
2143 Please note that the image is uploaded to a registry and not exported
2145 into compose directory. There will be a metadata file in compose/meta‐
2146 data/osbs.json with details about the built images (assuming they are
2147 not scratch builds).
2148 osbs (dict) – a mapping from variant regexes to configuration blocks.
2150 The format should be {variant_uid_regex: [config_dict]}.
2151 The configuration for each image must have at least these keys:
2153 • url – (str) URL pointing to a Git repository with Dockerfile.
2155 Please see Git URLs section for more details.
2156 • target – (str) A Koji target to build the image for.
2158 • git_branch – (str) A branch in SCM for the Dockerfile. This is
2160 required by OSBS to avoid race conditions when multiple builds
2161 from the same repo are submitted at the same time. Please note
2162 that url should contain the branch or tag name as well, so
2163 that it can be resolved to a particular commit hash.
2164 Optionally you can specify failable. If it has a truthy value,
2166 failure to create the image will not abort the whole compose.
2167 The configuration will pass other attributes directly to the
2169 Koji task. This includes scratch and priority. See koji
2170 list-api buildContainer for more details about these options.
2171 A value for yum_repourls will be created automatically and point
2173 at a repository in the current compose. You can add extra repos‐
2174 itories with repo key having a list of urls pointing to .repo
2175 files or just variant uid, Pungi will create the .repo file for
2176 that variant. If specific URL is used in the repo, the $COM‐
2177 POSE_ID variable in the repo string will be replaced with the
2178 real compose ID. gpgkey can be specified to enable gpgcheck in
2179 repo files for variants.
2180 osbs_registries
2182 (dict) – Use this optional setting to emit osbs-request-push
2183 messages for each non-scratch container build. These messages
2184 can guide other tools how to push the images to other reg‐
2185 istries. For example, an external tool might trigger on these
2186 messages and copy the images from OSBS’s registry to a staging
2187 or production registry.
2188 For each completed container build, Pungi will try to match the
2190 NVR against a key in osbs_registries mapping (using shell-style
2191 globbing) and take the corresponding value and collect them
2192 across all built images. Pungi will save this data into
2193 logs/global/osbs-registries.json, mapping each Koji NVR to the
2194 registry data. Pungi will also send this data to the message bus
2195 on the osbs-request-push topic once the compose finishes suc‐
2196 cessfully.
2197 Pungi simply logs the mapped data and emits the messages. It
2199 does not handle the messages or push images. A separate tool
2200 must do that.
2201 Example config
2203 osbs = {
2204 "^Server$": {
2205 # required
2206 "url": "git://example.com/dockerfiles.git?#HEAD",
2207 "target": "f24-docker-candidate",
2208 "git_branch": "f24-docker",
2209 # optional
2211 "repo": ["Everything", "https://example.com/extra-repo.repo"],
2212 # This will result in three repo urls being passed to the task.
2213 # They will be in this order: Server, Everything, example.com/
2214 "gpgkey": 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release',
2215 }
2216 }
2217 Extra ISOs
2219 Create an ISO image that contains packages from multiple variants. Such
2220 ISO always belongs to one variant, and will be stored in ISO directory
2221 of that variant.
2222 The ISO will be bootable if buildinstall phase runs for the parent
2224 variant. It will reuse boot configuration from that variant.
2225 extra_isos
2227 (dict) – a mapping from variant UID regex to a list of configu‐
2228 ration blocks.
2229 • include_variants – (list) list of variant UIDs from which con‐
2231 tent should be added to the ISO; the variant of this image is
2232 added automatically.
2233 Rest of configuration keys is optional.
2235 • filename – (str) template for naming the image. In addition to
2237 the regular placeholders filename is available with the name
2238 generated using image_name_format option.
2239 • volid – (str) template for generating volume ID. Again volid
2241 placeholder can be used similarly as for file name. This can
2242 also be a list of templates that will be tried sequentially
2243 until one generates a volume ID that fits into 32 character
2244 limit.
2245 • extra_files – (list) a list of scm_dict objects. These files
2247 will be put in the top level directory of the image.
2248 • arches – (list) a list of architectures for which to build
2250 this image. By default all arches from the variant will be
2251 used. This option can be used to limit them.
2252 • failable_arches – (list) a list of architectures for which the
2254 image can fail to be generated and not fail the entire com‐
2255 pose.
2256 • skip_src – (bool) allows to disable creating an image with
2258 source packages.
2259 • inherit_extra_files – (bool) by default extra files in vari‐
2261 ants are ignored. If you want to include them in the ISO, set
2262 this option to True.
2263 • max_size – (int) expected maximum size in bytes. If the final
2265 image is larger, a warning will be issued.
2266 Example config
2268 extra_isos = {
2269 'Server': [{
2270 # Will generate foo-DP-1.0-20180510.t.43-Server-x86_64-dvd1.iso
2271 'filename': 'foo-{filename}',
2272 'volid': 'foo-{arch}',
2273 'extra_files': [{
2275 'scm': 'git',
2276 'repo': 'https://pagure.io/pungi.git',
2277 'file': 'setup.py'
2278 }],
2279 'include_variants': ['Client']
2281 }]
2282 }
2283 # This should create image with the following layout:
2284 # .
2285 # ├── Client
2286 # │ ├── Packages
2287 # │ │ ├── a
2288 # │ │ └── b
2289 # │ └── repodata
2290 # ├── Server
2291 # │ ├── Packages
2292 # │ │ ├── a
2293 # │ │ └── b
2294 # │ └── repodata
2295 # └── setup.py
2296 Media Checksums Settings
2298 media_checksums
2299 (list) – list of checksum types to compute, allowed values are
2300 anything supported by Python’s hashlib module (see documentation
2301 for details).
2302 media_checksum_one_file
2304 (bool) – when True, only one CHECKSUM file will be created per
2305 directory; this option requires media_checksums to only specify
2306 one type
2307 media_checksum_base_filename
2309 (str) – when not set, all checksums will be save to a file named
2310 either CHECKSUM or based on the digest type; this option allows
2311 adding any prefix to that name
2312 It is possible to use format strings that will be replace by ac‐
2314 tual values. The allowed keys are:
2315 • arch
2317 • compose_id
2319 • date
2321 • label
2323 • label_major_version
2325 • release_short
2327 • respin
2329 • type
2331 • type_suffix
2333 • version
2335 • dirname (only if media_checksum_one_file is enabled)
2337 For example, for Fedora the prefix should be %(re‐
2339 lease_short)s-%(variant)s-%(version)s-%(date)s%(type_suf‐
2340 fix)s.%(respin)s.
2341 Translate Paths Settings
2343 translate_paths
2344 (list) – list of paths to translate; format: [(path, trans‐
2345 lated_path)]
2346 NOTE:
2348 This feature becomes useful when you need to transform compose loca‐
2349 tion into e.g. a HTTP repo which is can be passed to koji im‐
2350 age-build. The path part is normalized via os.path.normpath().
2351 Example config
2353 translate_paths = [
2354 ("/mnt/a", "http://b/dir"),
2355 ]
2356 Example usage
2358 >>> from pungi.util import translate_paths
2359 >>> print translate_paths(compose_object_with_mapping, "/mnt/a/c/somefile")
2360 http://b/dir/c/somefile
2361 Miscellaneous Settings
2363 paths_module
2364 (str) – Name of Python module implementing the same interface as
2365 pungi.paths. This module can be used to override where things
2366 are placed.
2367 link_type = hardlink-or-copy
2369 (str) – Method of putting packages into compose directory.
2370 Available options:
2372 • hardlink-or-copy
2374 • hardlink
2376 • copy
2378 • symlink
2380 • abspath-symlink
2382 skip_phases
2384 (list) – List of phase names that should be skipped. The same
2385 functionality is available via a command line option.
2386 release_discinfo_description
2388 (str) – Override description in .discinfo files. The value is a
2389 format string accepting %(variant_name)s and %(arch)s placehold‐
2390 ers.
2391 symlink_isos_to
2393 (str) – If set, the ISO files from buildinstall, createiso and
2394 live_images phases will be put into this destination, and a sym‐
2395 link pointing to this location will be created in actual compose
2396 directory.
2397 dogpile_cache_backend
2399 (str) – If set, Pungi will use the configured Dogpile cache
2400 backend to cache various data between multiple Pungi calls. This
2401 can make Pungi faster in case more similar composes are running
2402 regularly in short time.
2403 For list of available backends, please see the
2405 https://dogpilecache.readthedocs.io documentation.
2406 Most typical configuration uses the dogpile.cache.dbm backend.
2408 dogpile_cache_arguments
2410 (dict) – Arguments to be used when creating the Dogpile cache
2411 backend. See the particular backend’s configuration for the
2412 list of possible key/value pairs.
2413 For the dogpile.cache.dbm backend, the value can be for example
2415 following:
2416 {
2418 "filename": "/tmp/pungi_cache_file.dbm"
2419 }
2420 dogpile_cache_expiration_time
2422 (int) – Defines the default expiration time in seconds of data
2423 stored in the Dogpile cache. Defaults to 3600 seconds.
2424
2426 Actual Pungi configuration files can get very large. This pages brings
2427 two examples of (almost) full configuration for two different composes.
2428 Fedora Rawhide compose
2430 This is a shortened configuration for Fedora Radhide compose as of
2431 2019-10-14.
2432 release_name = 'Fedora'
2434 release_short = 'Fedora'
2435 release_version = 'Rawhide'
2436 release_is_layered = False
2437 bootable = True
2439 comps_file = {
2440 'scm': 'git',
2441 'repo': 'https://pagure.io/fedora-comps.git',
2442 'branch': 'master',
2443 'file': 'comps-rawhide.xml',
2444 # Merge translations by running make. This command will generate the file.
2445 'command': 'make comps-rawhide.xml'
2446 }
2447 module_defaults_dir = {
2448 'scm': 'git',
2449 'repo': 'https://pagure.io/releng/fedora-module-defaults.git',
2450 'branch': 'main',
2451 'dir': '.'
2452 }
2453 # Optional module obsoletes configuration which is merged
2454 # into the module index and gets resolved
2455 module_obsoletes_dir = {
2456 'scm': 'git',
2457 'repo': 'https://pagure.io/releng/fedora-module-defaults.git',
2458 'branch': 'main',
2459 'dir': 'obsoletes'
2460 }
2461 variants_file='variants-fedora.xml'
2463 sigkeys = ['12C944D0']
2464 # Put packages into subdirectories hashed by their initial letter.
2466 hashed_directories = True
2467 # There is a special profile for use with compose. It makes Pungi
2469 # authenticate automatically as rel-eng user.
2470 koji_profile = 'compose_koji'
2471 # RUNROOT settings
2473 runroot = True
2474 runroot_channel = 'compose'
2475 runroot_tag = 'f32-build'
2476 # PKGSET
2478 pkgset_source = 'koji'
2479 pkgset_koji_tag = 'f32'
2480 pkgset_koji_inherit = False
2481 filter_system_release_packages = False
2483 # GATHER
2485 gather_method = {
2486 '^.*': { # For all variants
2487 'comps': 'deps', # resolve dependencies for packages from comps file
2488 'module': 'nodeps', # but not for packages from modules
2489 }
2490 }
2491 gather_backend = 'dnf'
2492 gather_profiler = True
2493 check_deps = False
2494 greedy_method = 'build'
2495 repoclosure_backend = 'dnf'
2497 # CREATEREPO
2499 createrepo_deltas = False
2500 createrepo_database = True
2501 createrepo_use_xz = True
2502 createrepo_extra_args = ['--zck', '--zck-dict-dir=/usr/share/fedora-repo-zdicts/rawhide']
2503 # CHECKSUMS
2505 media_checksums = ['sha256']
2506 media_checksum_one_file = True
2507 media_checksum_base_filename = '%(release_short)s-%(variant)s-%(version)s-%(arch)s-%(date)s%(type_suffix)s.%(respin)s'
2508 # CREATEISO
2510 iso_hfs_ppc64le_compatible = False
2511 # BUILDINSTALL
2513 buildinstall_method = 'lorax'
2514 buildinstall_skip = [
2515 # No installer for Modular variant
2516 ('^Modular$', {'*': True}),
2517 # No 32 bit installer for Everything.
2518 ('^Everything$', {'i386': True}),
2519 ]
2520 # Enables macboot on x86_64 for all variants and disables upgrade image building
2522 # everywhere.
2523 lorax_options = [
2524 ('^.*$', {
2525 'x86_64': {
2526 'nomacboot': False
2527 },
2528 'ppc64le': {
2529 # Use 3GB image size for ppc64le.
2530 'rootfs_size': 3
2531 },
2532 '*': {
2533 'noupgrade': True
2534 }
2535 })
2536 ]
2537 additional_packages = [
2539 ('^(Server|Everything)$', {
2540 '*': [
2541 # Add all architectures of dracut package.
2542 'dracut.*',
2543 # All all packages matching this pattern
2544 'autocorr-*',
2545 ],
2546 }),
2547 ('^Everything$', {
2549 # Everything should include all packages from the tag. This only
2550 # applies to the native arch. Multilib will still be pulled in
2551 # according to multilib rules.
2552 '*': ['*'],
2553 }),
2554 ]
2555 filter_packages = [
2557 ("^.*$", {"*": ["glibc32", "libgcc32"]}),
2558 ('(Server)$', {
2559 '*': [
2560 'kernel*debug*',
2561 'kernel-kdump*',
2562 ]
2563 }),
2564 ]
2565 multilib = [
2567 ('^Everything$', {
2568 'x86_64': ['devel', 'runtime'],
2569 })
2570 ]
2571 # These packages should never be multilib on any arch.
2573 multilib_blacklist = {
2574 '*': [
2575 'kernel', 'kernel-PAE*', 'kernel*debug*', 'java-*', 'php*', 'mod_*', 'ghc-*'
2576 ],
2577 }
2578 # These should be multilib even if they don't match the rules defined above.
2580 multilib_whitelist = {
2581 '*': ['wine', '*-static'],
2582 }
2583 createiso_skip = [
2585 # Keep binary ISOs for Server, but not source ones.
2586 ('^Server$', {'src': True}),
2587 # Remove all other ISOs.
2589 ('^Everything$', {'*': True, 'src': True}),
2590 ('^Modular$', {'*': True, 'src': True}),
2591 ]
2592 # Image name respecting Fedora's image naming policy
2594 image_name_format = '%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s-%(date)s%(type_suffix)s.%(respin)s.iso'
2595 # Use the same format for volume id
2596 image_volid_formats = [
2597 '%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s'
2598 ]
2599 # Used by Pungi to replace 'Cloud' with 'C' (etc.) in ISO volume IDs.
2600 # There is a hard 32-character limit on ISO volume IDs, so we use
2601 # these to try and produce short enough but legible IDs. Note this is
2602 # duplicated in Koji for live images, as livemedia-creator does not
2603 # allow Pungi to tell it what volume ID to use. Note:
2604 # https://fedoraproject.org/wiki/User:Adamwill/Draft_fedora_image_naming_policy
2605 volume_id_substitutions = {
2606 'Beta': 'B',
2607 'Rawhide': 'rawh',
2608 'Silverblue': 'SB',
2609 'Cinnamon': 'Cinn',
2610 'Cloud': 'C',
2611 'Design_suite': 'Dsgn',
2612 'Electronic_Lab': 'Elec',
2613 'Everything': 'E',
2614 'Scientific_KDE': 'SciK',
2615 'Security': 'Sec',
2616 'Server': 'S',
2617 'Workstation': 'WS',
2618 }
2619 disc_types = {
2621 'boot': 'netinst',
2622 'live': 'Live',
2623 }
2624 translate_paths = [
2626 ('/mnt/koji/compose/', 'https://kojipkgs.fedoraproject.org/compose/'),
2627 ]
2628 # These will be inherited by live_media, live_images and image_build
2630 global_ksurl = 'git+https://pagure.io/fedora-kickstarts.git?#HEAD'
2631 global_release = '!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN'
2632 global_version = 'Rawhide'
2633 # live_images ignores this in favor of live_target
2634 global_target = 'f32'
2635 image_build = {
2637 '^Container$': [
2638 {
2639 'image-build': {
2640 'format': [('docker', 'tar.xz')],
2641 'name': 'Fedora-Container-Base',
2642 'kickstart': 'fedora-container-base.ks',
2643 'distro': 'Fedora-22',
2644 'disk_size': 5,
2645 'arches': ['armhfp', 'aarch64', 'ppc64le', 's390x', 'x86_64'],
2646 'repo': 'Everything',
2647 'install_tree_from': 'Everything',
2648 'subvariant': 'Container_Base',
2649 'failable': ['*'],
2650 },
2651 'factory-parameters': {
2652 'dockerversion': "1.10.1",
2653 'docker_cmd': '[ "/bin/bash" ]',
2654 'docker_env': '[ "DISTTAG=f32container", "FGC=f32", "container=oci" ]',
2655 'docker_label': '{ "name": "fedora", "license": "MIT", "vendor": "Fedora Project", "version": "32"}',
2656 },
2657 },
2658 ],
2659 }
2660 live_media = {
2662 '^Workstation$': [
2663 {
2664 'name': 'Fedora-Workstation-Live',
2665 'kickstart': 'fedora-live-workstation.ks',
2666 # Variants.xml also contains aarch64 and armhfp, but there
2667 # should be no live media for those arches.
2668 'arches': ['x86_64', 'ppc64le'],
2669 'failable': ['ppc64le'],
2670 # Take packages and install tree from Everything repo.
2671 'repo': 'Everything',
2672 'install_tree_from': 'Everything',
2673 }
2674 ],
2675 '^Spins': [
2676 # There are multiple media for Spins variant. They use subvariant
2677 # field so that they can be identified in the metadata.
2678 {
2679 'name': 'Fedora-KDE-Live',
2680 'kickstart': 'fedora-live-kde.ks',
2681 'arches': ['x86_64'],
2682 'repo': 'Everything',
2683 'install_tree_from': 'Everything',
2684 'subvariant': 'KDE'
2685 },
2687 {
2688 'name': 'Fedora-Xfce-Live',
2689 'kickstart': 'fedora-live-xfce.ks',
2690 'arches': ['x86_64'],
2691 'failable': ['*'],
2692 'repo': 'Everything',
2693 'install_tree_from': 'Everything',
2694 'subvariant': 'Xfce'
2695 },
2696 ],
2697 }
2698 failable_deliverables = [
2700 # Installer and ISOs for server failing do not abort the compose.
2701 ('^Server$', {
2702 '*': ['buildinstall', 'iso'],
2703 }),
2704 ('^.*$', {
2705 # Buildinstall is not blocking
2706 'src': ['buildinstall'],
2707 # Nothing on i386, ppc64le blocks the compose
2708 'i386': ['buildinstall', 'iso'],
2709 'ppc64le': ['buildinstall', 'iso'],
2710 's390x': ['buildinstall', 'iso'],
2711 })
2712 ]
2713 live_target = 'f32'
2715 live_images_no_rename = True
2716 live_images = [
2717 ('^Workstation$', {
2718 'armhfp': {
2719 'kickstart': 'fedora-arm-workstation.ks',
2720 'name': 'Fedora-Workstation-armhfp',
2721 # Again workstation takes packages from Everything.
2722 'repo': 'Everything',
2723 'type': 'appliance',
2724 'failable': True,
2725 }
2726 }),
2727 ('^Server$', {
2728 # But Server has its own repo.
2729 'armhfp': {
2730 'kickstart': 'fedora-arm-server.ks',
2731 'name': 'Fedora-Server-armhfp',
2732 'type': 'appliance',
2733 'failable': True,
2734 }
2735 }),
2736 ]
2737 ostree = {
2739 "^Silverblue$": {
2740 "version": "!OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN",
2741 # To get config, clone master branch from this repo and take
2742 # treefile from there.
2743 "treefile": "fedora-silverblue.yaml",
2744 "config_url": "https://pagure.io/workstation-ostree-config.git",
2745 "config_branch": "master",
2746 # Consume packages from Everything
2747 "repo": "Everything",
2748 # Don't create a reference in the ostree repo (signing automation does that).
2749 "tag_ref": False,
2750 # Don't use change detection in ostree.
2751 "force_new_commit": True,
2752 # This is the location for the repo where new commit will be
2753 # created. Note that this is outside of the compose dir.
2754 "ostree_repo": "/mnt/koji/compose/ostree/repo/",
2755 "ostree_ref": "fedora/rawhide/${basearch}/silverblue",
2756 "arches": ["x86_64", "ppc64le", "aarch64"],
2757 "failable": ['*'],
2758 }
2759 }
2760 ostree_installer = [
2762 ("^Silverblue$", {
2763 "x86_64": {
2764 "repo": "Everything",
2765 "release": None,
2766 "rootfs_size": "8",
2767 # Take templates from this repository.
2768 'template_repo': 'https://pagure.io/fedora-lorax-templates.git',
2769 'template_branch': 'master',
2770 # Use following templates.
2771 "add_template": ["ostree-based-installer/lorax-configure-repo.tmpl",
2772 "ostree-based-installer/lorax-embed-repo.tmpl",
2773 "ostree-based-installer/lorax-embed-flatpaks.tmpl"],
2774 # And add these variables for the templates.
2775 "add_template_var": [
2776 "ostree_install_repo=https://kojipkgs.fedoraproject.org/compose/ostree/repo/",
2777 "ostree_update_repo=https://ostree.fedoraproject.org",
2778 "ostree_osname=fedora",
2779 "ostree_oskey=fedora-32-primary",
2780 "ostree_contenturl=mirrorlist=https://ostree.fedoraproject.org/mirrorlist",
2781 "ostree_install_ref=fedora/rawhide/x86_64/silverblue",
2782 "ostree_update_ref=fedora/rawhide/x86_64/silverblue",
2783 "flatpak_remote_name=fedora",
2784 "flatpak_remote_url=oci+https://registry.fedoraproject.org",
2785 "flatpak_remote_refs=runtime/org.fedoraproject.Platform/x86_64/f30 app/org.gnome.Baobab/x86_64/stable",
2786 ],
2787 'failable': ['*'],
2788 },
2789 })
2790 ]
2791 RCM Tools compose
2793 This is a small compose used to deliver packages to Red Hat internal
2794 users. The configuration is split into two files.
2795 # rcmtools-common.conf
2797 release_name = "RCM Tools"
2799 release_short = "RCMTOOLS"
2800 release_version = "2.0"
2801 release_type = "updates"
2802 release_is_layered = True
2803 createrepo_c = True
2804 createrepo_checksum = "sha256"
2805 # PKGSET
2807 pkgset_source = "koji"
2808 koji_profile = "brew"
2809 pkgset_koji_inherit = True
2810 # GENERAL SETTINGS
2813 bootable = False
2814 comps_file = "rcmtools-comps.xml"
2815 variants_file = "rcmtools-variants.xml"
2816 sigkeys = ["3A3A33A3"]
2817 # RUNROOT settings
2820 runroot = False
2821 # GATHER
2824 gather_method = "deps"
2825 check_deps = True
2826 additional_packages = [
2828 ('.*', {
2829 '*': ['puddle', 'rcm-nexus'],
2830 }
2831 ),
2832 ]
2833 # Set repoclosure_strictness to fatal to avoid installation dependency
2835 # issues in production composes
2836 repoclosure_strictness = [
2837 ("^.*$", {
2838 "*": "fatal"
2839 })
2840 ]
2841 Configuration specific for different base products is split into sepa‐
2843 rate files.
2844 # rcmtools-common.conf
2846 from rcmtools-common import *
2847 # BASE PRODUCT
2849 base_product_name = "Red Hat Enterprise Linux"
2850 base_product_short = "RHEL"
2851 base_product_version = "7"
2852 # PKGSET
2854 pkgset_koji_tag = "rcmtools-rhel-7-compose"
2855 # remove i386 arch on rhel7
2857 tree_arches = ["aarch64", "ppc64le", "s390x", "x86_64"]
2858 check_deps = False
2860 # Packages in these repos are available to satisfy dependencies inside the
2862 # compose, but will not be pulled in.
2863 gather_lookaside_repos = [
2864 ("^Client|Client-optional$", {
2865 "x86_64": [
2866 "http://example.redhat.com/rhel/7/Client/x86_64/os/",
2867 "http://example.redhat.com/rhel/7/Client/x86_64/optional/os/",
2868 ],
2869 }),
2870 ("^Workstation|Workstation-optional$", {
2871 "x86_64": [
2872 "http://example.redhat.com/rhel/7/Workstation/x86_64/os/",
2873 "http://example.redhat.com/rhel/7/Workstation/x86_64/optional/os/",
2874 ],
2875 }),
2876 ("^Server|Server-optional$", {
2877 "aarch64": [
2878 "http://example.redhat.com/rhel/7/Server/aarch64/os/",
2879 "http://example.redhat.com/rhel/7/Server/aarch64/optional/os/",
2880 ],
2881 "ppc64": [
2882 "http://example.redhat.com/rhel/7/Server/ppc64/os/",
2883 "http://example.redhat.com/rhel/7/Server/ppc64/optional/os/",
2884 ],
2885 "ppc64le": [
2886 "http://example.redhat.com/rhel/7/Server/ppc64le/os/",
2887 "http://example.redhat.com/rhel/7/Server/ppc64le/optional/os/",
2888 ],
2889 "s390x": [
2890 "http://example.redhat.com/rhel/7/Server/s390x/os/",
2891 "http://example.redhat.com/rhel/7/Server/s390x/optional/os/",
2892 ],
2893 "x86_64": [
2894 "http://example.redhat.com/rhel/7/Server/x86_64/os/",
2895 "http://example.redhat.com/rhel/7/Server/x86_64/optional/os/",
2896 ],
2897 })
2898 ]
2899
2901 Multiple places in Pungi can use files from external storage. The con‐
2902 figuration is similar independently of the backend that is used, al‐
2903 though some features may be different.
2904 The so-called scm_dict is always put into configuration as a dictio‐
2906 nary, which can contain following keys.
2907 • scm – indicates which SCM system is used. This is always required.
2909 Allowed values are:
2910 • file – copies files from local filesystem
2912 • git – copies files from a Git repository
2914 • cvs – copies files from a CVS repository
2916 • rpm – copies files from a package in the compose
2918 • koji – downloads archives from a given build in Koji build system
2920 • repo
2922 • for Git and CVS backends this should be URL to the repository
2924 • for RPM backend this should be a shell style glob matching package
2926 names (or a list of such globs)
2927 • for file backend this should be empty
2929 • for Koji backend this should be an NVR or package name
2931 • branch
2933 • branch name for Git and CVS backends, with master and HEAD as de‐
2935 faults
2936 • Koji tag for koji backend if only package name is given
2938 • otherwise should not be specified
2940 • file – a list of files that should be exported.
2942 • dir – a directory that should be exported. All its contents will be
2944 exported. This option is mutually exclusive with file.
2945 • command – defines a shell command to run after Git clone to generate
2947 the needed file (for example to run make). Only supported in Git
2948 backend.
2949 Koji examples
2951 There are two different ways how to configure the Koji backend.
2952 {
2954 # Download all *.tar files from build my-image-1.0-1.
2955 "scm": "koji",
2956 "repo": "my-image-1.0-1",
2957 "file": "*.tar",
2958 }
2959 {
2961 # Find latest build of my-image in tag my-tag and take files from
2962 # there.
2963 "scm": "koji",
2964 "repo": "my-image",
2965 "branch": "my-tag",
2966 "file": "*.tar",
2967 }
2968 Using both tag name and exact NVR will result in error: the NVR would
2970 be interpreted as a package name, and would not match anything.
2971 file vs. dir
2973 Exactly one of these two options has to be specified. Documentation for
2974 each configuration option should specify whether it expects a file or a
2975 directory.
2976 For extra_files phase either key is valid and should be chosen depend‐
2978 ing on what the actual use case.
2979 Caveats
2981 The rpm backend can only be used in phases that would extract the files
2982 after pkgset phase finished. You can’t get comps file from a package.
2983 Depending on Git repository URL configuration Pungi can only export the
2985 requested content using git archive. When a command should run this is
2986 not possible and a clone is always needed.
2987 When using koji backend, it is required to provide configuration for
2989 Koji profile to be used (koji_profile). It is not possible to contact
2990 multiple different Koji instances.
2991
2993 Pungi has the ability to emit notification messages about progress and
2994 general status of the compose. These can be used to e.g. send messages
2995 to fedmsg. This is implemented by actually calling a separate script.
2996 The script will be called with one argument describing action that just
2998 happened. A JSON-encoded object will be passed to standard input to
2999 provide more information about the event. At the very least, the object
3000 will contain a compose_id key.
3001 The notification script inherits working directory from the parent
3003 process and it can be called from the same directory pungi-koji is
3004 called from. The working directory is listed at the start of main log.
3005 Currently these messages are sent:
3007 • status-change – when composing starts, finishes or fails; a status
3009 key is provided to indicate details
3010 • phase-start – on start of a phase
3012 • phase-stop – when phase is finished
3014 • createiso-targets – with a list of images to be created
3016 • createiso-imagedone – when any single image is finished
3018 • createiso-imagefail – when any single image fails to create
3020 • fail-to-start – when there are incorrect CLI options or errors in
3022 configuration file; this message does not contain compose_id nor
3023 is it started in the compose directory (which does not exist yet)
3024 • ostree – when a new commit is created, this message will announce
3026 its hash and the name of ref it is meant for.
3027 For phase related messages phase_name key is provided as well.
3029 A pungi-fedmsg-notification script is provided and understands this in‐
3031 terface.
3032 Setting it up
3034 The script should be provided as a command line argument --notifica‐
3035 tion-script.
3036 --notification-script=pungi-fedmsg-notification
3038
3040 A compose created by Pungi consists of one or more variants. A variant
3041 contains a subset of the content targeted at a particular use case.
3042 There are different types of variants. The type affects how packages
3044 are gathered into the variant.
3045 The inputs for gathering are defined by various gather sources. Pack‐
3047 ages from all sources are collected to create a big list of package
3048 names, comps groups names and a list of packages that should be fil‐
3049 tered out.
3050 NOTE:
3052 The inputs for both explicit package list and comps file are inter‐
3053 preted as RPM names, not any arbitrary provides nor source package
3054 name.
3055 Next, gather_method defines how the list is processed. For nodeps, the
3057 results from source are used pretty much as is [1]. For deps method, a
3058 process will be launched to figure out what dependencies are needed and
3059 those will be pulled in.
3060 [1] The lists are filtered based on what packages are available in the
3062 package set, but nothing else will be pulled in.
3063 Variant types
3065 Variant
3066 is a base type that has no special behaviour.
3067 Addon is built on top of a regular variant. Any packages that should
3069 go to both the addon and its parent will be removed from addon.
3070 Packages that are only in addon but pulled in because of
3071 gather_fulltree option will be moved to parent.
3072 Integrated Layered Product
3074 works similarly to addon. Additionally, all packages from addons
3075 on the same parent variant are removed integrated layered prod‐
3076 ucts.
3077 The main difference between an addon and integrated layered
3079 product is that integrated layered product has its own identity
3080 in the metadata (defined with product name and version).
3081 NOTE:
3083 There’s also Layered Product as a term, but this is not re‐
3084 lated to variants. It’s used to describe a product that is
3085 not a standalone operating system and is instead meant to be
3086 used on some other base system.
3087 Optional
3089 contains packages that complete the base variants’ package set.
3090 It always has fulltree and selfhosting enabled, so it contains
3091 build dependencies and packages which were not specifically re‐
3092 quested for base variant.
3093 Some configuration options are overridden for particular variant types.
3095 Depsolving configuration
3097 ┌──────────┬──────────────┬──────────────┐
3098 │Variant │ Fulltree │ Selfhosting │
3099 ├──────────┼──────────────┼──────────────┤
3100 │base │ configurable │ configurable │
3101 ├──────────┼──────────────┼──────────────┤
3102 │addon/ILP │ enabled │ disabled │
3103 ├──────────┼──────────────┼──────────────┤
3104 │optional │ enabled │ enabled │
3105 └──────────┴──────────────┴──────────────┘
3106 Profiling
3108 Profiling data on the pungi-gather tool can be enabled by setting the
3109 gather_profiler configuration option to True.
3110 Modular compose
3112 A compose with gather_source set to module is called modular. The pack‐
3113 age list is determined by a list of modules.
3114 The list of modules that will be put into a variant is defined in the
3116 variants.xml file. The file can contain either Name:Stream or
3117 Name:Stream:Version references. See Module Naming Policy for details.
3118 When Version is missing from the specification, Pungi will ask PDC for
3119 the latest one.
3120 The module metadata in PDC contains a list of RPMs in the module as
3122 well as Koji tag from which the packages can be retrieved.
3123 Restrictions
3125 • A modular compose must always use Koji as a package set source.
3126
3128 The comps file that Pungi takes as input is not really pure comps as
3129 used by tools like DNF. There are extensions used to customize how the
3130 file is processed.
3131 The first step of Pungi processing is to retrieve the actual file. This
3133 can use anything that scm_support supports.
3134 Pungi extensions are arch attribute on packageref, group and environ‐
3136 ment tags. The value of this attribute is a comma separated list of ar‐
3137 chitectures.
3138 Second step Pungi performs is creating a file for each architecture.
3140 This is done by removing all elements with incompatible arch attribute.
3141 No additional clean up is performed on this file. The resulting file is
3142 only used internally for the rest of the compose process.
3143 Third and final step is to create comps file for each Variant.Arch com‐
3145 bination. This is the actual file that will be included in the com‐
3146 pose. The start file is the original input file, from which all ele‐
3147 ments with incompatible architecture are removed. Then clean up is per‐
3148 formed by removing all empty groups, removing non-existing groups from
3149 environments and categories and finally removing empty environments and
3150 categories. As a last step groups not listed in the variants file are
3151 removed.
3152
3154 Set up development environment
3155 In order to work on Pungi, you should install recent version of Fedora.
3156 Python2
3158 Fedora 29 is recommended because some packages are not available in
3159 newer Fedora release, e.g. python2-libcomps.
3160 Install required packages
3162 $ sudo dnf install -y krb5-devel gcc make libcurl-devel python2-devel python2-createrepo_c kobo-rpmlib yum python2-libcomps python2-libselinx
3164 Python3
3166 Install required packages
3167 $ sudo dnf install -y krb5-devel gcc make libcurl-devel python3-devel python3-createrepo_c python3-libcomps
3169 Developing
3171 Currently the development workflow for Pungi is on master branch:
3172 • Make your own fork at https://pagure.io/pungi
3174 • Clone your fork locally (replacing $USERNAME with your own):
3176 git clone git@pagure.io:forks/$USERNAME/pungi.git
3178 • cd into your local clone and add the remote upstream for rebasing:
3180 cd pungi
3182 git remote add upstream git@pagure.io:pungi.git
3183 NOTE:
3185 This workflow assumes that you never git commit directly to the
3186 master branch of your fork. This will make more sense when we
3187 cover rebasing below.
3188 • create a topic branch based on master:
3190 git branch my_topic_branch master
3192 git checkout my_topic_branch
3193 • Make edits, changes, add new features, etc. and then make sure to
3195 pull from upstream master and rebase before submitting a pull re‐
3196 quest:
3197 # lets just say you edited setup.py for sake of argument
3199 git checkout my_topic_branch
3200 # make changes to setup.py
3202 black setup.py
3203 tox
3204 git add setup.py
3205 git commit -s -m "added awesome feature to setup.py"
3206 # now we rebase
3208 git checkout master
3209 git pull --rebase upstream master
3210 git push origin master
3211 git push origin --tags
3212 git checkout my_topic_branch
3213 git rebase master
3214 # resolve merge conflicts if any as a result of your development in
3216 # your topic branch
3217 git push origin my_topic_branch
3218 NOTE:
3220 In order to for your commit to be merged:
3221 • you must sign-off on it. Use -s option when running git commit.
3223 • The code must be well formatted via black and pass flake8 check‐
3225 ing. Run tox -e black,flake8 to do the check.
3226 • Create pull request in the pagure.io web UI
3228 • For convenience, here is a bash shell function that can be placed in
3230 your ~/.bashrc and called such as pullupstream pungi-4-devel that
3231 will automate a large portion of the rebase steps from above:
3232 pullupstream () {
3234 if [[ -z "$1" ]]; then
3235 printf "Error: must specify a branch name (e.g. - master, devel)\n"
3236 else
3237 pullup_startbranch=$(git describe --contains --all HEAD)
3238 git checkout $1
3239 git pull --rebase upstream master
3240 git push origin $1
3241 git push origin --tags
3242 git checkout ${pullup_startbranch}
3243 fi
3244 }
3245 Testing
3247 You must write unit tests for any new code (except for trivial
3248 changes). Any code without sufficient test coverage may not be merged.
3249 To run all existing tests, suggested method is to use tox.
3251 $ sudo dnf install python3-tox -y
3253 $ tox -e py3
3255 $ tox -e py27
3256 Alternatively you could create a vitualenv, install deps and run tests
3258 manually if you don’t want to use tox.
3259 $ sudo dnf install python3-virtualenvwrapper -y
3261 $ mkvirtualenv --system-site-packages py3
3262 $ workon py3
3263 $ pip install -r requirements.txt -r test-requirements.txt
3264 $ make test
3265 # or with coverage
3267 $ make test-coverage
3268 If you need to run specified tests, pytest is recommended.
3270 # Activate virtualenv first
3272 # Run tests
3274 $ pytest tests/test_config.py
3275 $ pytest tests/test_config.py -k test_pkgset_mismatch_repos
3276 In the tests/ directory there is a shell script test_compose.sh that
3278 you can use to try and create a miniature compose on dummy data. The
3279 actual data will be created by running make test-data in project root.
3280 $ sudo dnf -y install rpm-build createrepo_c isomd5sum genisoimage syslinux
3282 # Activate virtualenv (the one created by tox could be used)
3284 $ source .tox/py3/bin/activate
3285 $ python setup.py develop
3287 $ make test-data
3288 $ make test-compose
3289 This testing compose does not actually use all phases that are avail‐
3291 able, and there is no checking that the result is correct. It only
3292 tells you whether it crashed or not.
3293 NOTE:
3295 Even when it finishes successfully, it may print errors about re‐
3296 poclosure on Server-Gluster.x86_64 in test phase. This is not a bug.
3297 Documenting
3299 You must write documentation for any new features and functional
3300 changes. Any code without sufficient documentation may not be merged.
3301 To generate the documentation, run make doc in project root.
3303
3305 Test Data
3306 Tests require test data and not all of it is available in git. You
3307 must create test repositories before running the tests:
3308 make test-data
3310 Requirements: createrepo_c, rpmbuild
3312 Unit Tests
3314 Unit tests cover functionality of Pungi python modules. You can run
3315 all of them at once:
3316 make test
3318 which is shortcut to:
3320 python2 setup.py test
3322 python3 setup.py test
3323 You can alternatively run individual tests:
3325 cd tests
3327 ./<test>.py [<class>[.<test>]]
3328 Functional Tests
3330 Because compose is quite complex process and not everything is covered
3331 with unit tests yet, the easiest way how to test if your changes did
3332 not break anything badly is to start a compose on a relatively small
3333 and well defined package set:
3334 cd tests
3336 ./test_compose.sh
3337
3339 There may be cases where it makes sense to split a big compose into
3340 separate parts, but create a compose output that links all output into
3341 one familiar structure.
3342 The pungi-orchestrate tools allows that.
3344 It works with an INI-style configuration file. The [general] section
3346 contains information about identity of the main compose. Other sections
3347 define individual parts.
3348 The parts are scheduled to run in parallel, with the minimal amount of
3350 serialization. The final compose directory will contain hard-links to
3351 the files.
3352 General settings
3354 target Path to directory where the final compose should be created.
3355 compose_type
3357 Type of compose to make.
3358 release_name
3360 Name of the product for the final compose.
3361 release_short
3363 Short name of the product for the final compose.
3364 release_version
3366 Version of the product for the final compose.
3367 release_type
3369 Type of the product for the final compose.
3370 extra_args
3372 Additional arguments that will be passed to the child Pungi pro‐
3373 cesses.
3374 koji_profile
3376 If specified, a current event will be retrieved from the Koji
3377 instance and used for all parts.
3378 kerberos
3380 If set to yes, a kerberos ticket will be automatically created
3381 at the start. Set keytab and principal as well.
3382 kerberos_keytab
3384 Path to keytab file used to create the kerberos ticket.
3385 kerberos_principal
3387 Kerberos principal for the ticket
3388 pre_compose_script
3390 Commands to execute before first part is started. Can contain
3391 multiple commands on separate lines.
3392 post_compose_script
3394 Commands to execute after the last part finishes and final sta‐
3395 tus is updated. Can contain multiple commands on separate lines.
3396 post_compose_script =
3398 compose-latest-symlink $COMPOSE_PATH
3399 custom-post-compose-script.sh
3400 Multiple environment variables are defined for the scripts:
3402 • COMPOSE_PATH
3404 • COMPOSE_ID
3406 • COMPOSE_DATE
3408 • COMPOSE_TYPE
3410 • COMPOSE_RESPIN
3412 • COMPOSE_LABEL
3414 • RELEASE_ID
3416 • RELEASE_NAME
3418 • RELEASE_SHORT
3420 • RELEASE_VERSION
3422 • RELEASE_TYPE
3424 • RELEASE_IS_LAYERED – YES for layered products, empty other‐
3426 wise
3427 • BASE_PRODUCT_NAME – only set for layered products
3429 • BASE_PRODUCT_SHORT – only set for layered products
3431 • BASE_PRODUCT_VERSION – only set for layered products
3433 • BASE_PRODUCT_TYPE – only set for layered products
3435 notification_script
3437 Executable name (or path to a script) that will be used to send
3438 a message once the compose is finished. In order for a valid URL
3439 to be included in the message, at least one part must configure
3440 path translation that would apply to location of main compose.
3441 Only two messages will be sent, one for start and one for finish
3443 (either successful or not).
3444 Partial compose settings
3446 Each part should have a separate section in the config file.
3447 It can specify these options:
3449 config Path to configuration file that describes this part. If rela‐
3451 tive, it is resolved relative to the file with parts configura‐
3452 tion.
3453 just_phase, skip_phase
3455 Customize which phases should run for this part.
3456 depends_on
3458 A comma separated list of other parts that must be finished be‐
3459 fore this part starts.
3460 failable
3462 A boolean toggle to mark a part as failable. A failure in such
3463 part will mark the final compose as incomplete, but still suc‐
3464 cessful.
3465
3467 Daniel Mach
3468
3470 2022, Red Hat, Inc.
34714.3 Jun 15, 2022 PUNGI(1)