1PUNGI(1) Pungi PUNGI(1)
2
3
4
6 pungi - Pungi Documentation
7
8 Contents:
9
11 [image: Pungi Logo] [image]
12
13 Pungi is a distribution compose tool.
14
15 Composes are release snapshots that contain release deliverables such
16 as:
17
18 · installation trees
19
20 · RPMs
21
22 · repodata
23
24 · comps
25
26 · (bootable) ISOs
27
28 · kickstart trees
29
30 · anaconda images
31
32 · images for PXE boot
33
34 Tool overview
35 Pungi consists of multiple separate executables backed by a common
36 library.
37
38 The main entry-point is the pungi-koji script. It loads the compose
39 configuration and kicks off the process. Composing itself is done in
40 phases. Each phase is responsible for generating some artifacts on
41 disk and updating the compose object that is threaded through all the
42 phases.
43
44 Pungi itself does not actually do that much. Most of the actual work is
45 delegated to separate executables. Pungi just makes sure that all the
46 commands are invoked in the appropriate order and with correct argu‐
47 ments. It also moves the artifacts to correct locations.
48
49 The executable name pungi-koji comes from the fact that most of those
50 separate executables submit tasks to Koji that does the actual work in
51 an auditable way.
52
53 However unlike doing everything manually in Koji, Pungi will make sure
54 you are building all images from the same package set, and will produce
55 even deliverables that Koji can not create like YUM repos and installer
56 ISOs.
57
58 Links
59 · Upstream GIT: https://pagure.io/pungi/
60
61 · Issue tracker: https://pagure.io/pungi/issues
62
63 · Questions can be asked on #fedora-releng IRC channel on FreeNode
64
65 Origin of name
66 The name Pungi comes from the instrument used to charm snakes. Anaconda
67 being the software Pungi was manipulating, and anaconda being a snake,
68 led to the referential naming.
69
70 The first name, which was suggested by Seth Vidal, was FIST, Fedora
71 Installation <Something> Tool. That name was quickly discarded and
72 replaced with Pungi.
73
74 There was also a bit of an inside joke that when said aloud, it could
75 sound like punji, which is a sharpened stick at the bottom of a trap.
76 Kind of like software…
77
79 Each invocation of pungi-koji consists of a set of phases. [image:
80 phase diagram] [image]
81
82 Most of the phases run sequentially (left-to-right in the diagram), but
83 there are use cases where multiple phases run in parallel. This happens
84 for phases whose main point is to wait for a Koji task to finish.
85
86 Init
87 The first phase to ever run. Can not be skipped. It prepares the comps
88 files for variants (by filtering out groups and packages that should
89 not be there). See comps for details about how this is done.
90
91 Pkgset
92 This phase loads a set of packages that should be composed. It has two
93 separate results: it prepares repos with packages in work/ directory
94 (one per arch) for further processing, and it returns a data structure
95 with mapping of packages to architectures.
96
97 Buildinstall
98 Spawns a bunch of threads, each of which runs either lorax or buildin‐
99 stall command (the latter coming from anaconda package). The commands
100 create boot.iso and other boot configuration files. The image is
101 finally linked into the compose/ directory as netinstall media.
102
103 The created images are also needed for creating live media or other
104 images in later phases.
105
106 With lorax this phase runs one task per variant.arch combination. For
107 buildinstall command there is only one task per architecture and prod‐
108 uct.img should be used to customize the results.
109
110 Gather
111 This phase uses data collected by pkgset phase and figures out what
112 packages should be in each variant. The basic mapping can come from
113 comps file, a JSON mapping or additional_packages config option. This
114 inputs can then be enriched by adding all dependencies. See gathering
115 for details.
116
117 Once the mapping is finalized, the packages are linked to appropriate
118 places and the rpms.json manifest is created.
119
120 ExtraFiles
121 This phase collects extra files from the configuration and copies them
122 to the compose directory. The files are described by a JSON file in the
123 compose subtree where the files are copied. This metadata is meant to
124 be distributed with the data (on ISO images).
125
126 Createrepo
127 This phase creates RPM repositories for each variant.arch tree. It is
128 actually reading the rpms.json manifest to figure out which packages
129 should be included.
130
131 OSTree
132 Updates an ostree repository with a new commit with packages from the
133 compose. The repository lives outside of the compose and is updated
134 immediately. If the compose fails in a later stage, the commit will not
135 be reverted.
136
137 Implementation wise, this phase runs rpm-ostree command in Koji runroot
138 (to allow running on different arches).
139
140 Createiso
141 Generates ISO files and accumulates enough metadata to be able to cre‐
142 ate image.json manifest. The file is however not created in this phase,
143 instead it is dumped in the pungi-koji script itself.
144
145 The files include a repository with all RPMs from the variant. There
146 will be multiple images if the packages do not fit on a single image.
147
148 The image will be bootable if buildinstall phase is enabled and the
149 packages fit on a single image.
150
151 There can also be images with source repositories. These are never
152 bootable.
153
154 ExtraIsos
155 This phase is very similar to createiso, except it combines content
156 from multiple variants onto a single image. Packages, repodata and
157 extra files from each configured variant are put into a subdirectory.
158 Additional extra files can be put into top level of the image. The
159 image will be bootable if the main variant is bootable.
160
161 LiveImages, LiveMedia
162 Creates media in Koji with koji spin-livecd, koji spin-appliance or
163 koji spin-livemedia command. When the media are finished, the images
164 are copied into the compose/ directory and metadata for images is
165 updated.
166
167 ImageBuild
168 This phase wraps up koji image-build. It also updates the metadata
169 ultimately responsible for images.json manifest.
170
171 OSBuild
172 Similarly to image build, this phases creates a koji osbuild task. In
173 the background it uses OSBuild Composer to create images.
174
175 OSBS
176 This phase builds container base images in OSBS.
177
178 The finished images are available in registry provided by OSBS, but not
179 downloaded directly into the compose. The is metadata about the created
180 image in compose/metadata/osbs.json.
181
182 ImageContainer
183 This phase builds a container image in OSBS, and stores the metadata in
184 the same file as OSBS phase. The container produced here wraps a dif‐
185 ferent image, created it ImageBuild or OSBuild phase. It can be useful
186 to deliver a VM image to containerized environments.
187
188 OSTreeInstaller
189 Creates bootable media that carry an ostree repository as a payload.
190 These images are created by running lorax with special templates. Again
191 it runs in Koji runroot.
192
193 Repoclosure
194 Run repoclosure on each repository. By default errors are only reported
195 in the log, the compose will still be considered a success. The actual
196 error has to be looked up in the compose logs directory. Configuration
197 allows customizing this.
198
199 ImageChecksum
200 Responsible for generating checksums for the images. The checksums are
201 stored in image manifest as well as files on disk. The list of images
202 to be processed is obtained from the image manifest. This way all
203 images will get the same checksums irrespective of the phase that cre‐
204 ated them.
205
206 Test
207 This phase is supposed to run some sanity checks on the finished com‐
208 pose.
209
210 The only test is to check all images listed the metadata and verify
211 that they look sane. For ISO files headers are checked to verify the
212 format is correct, and for bootable media a check is run to verify they
213 have properties that allow booting.
214
216 The configuration file parser is provided by kobo
217
218 The file follows a Python-like format. It consists of a sequence of
219 variables that have a value assigned to them.
220
221 variable = value
222
223 The variable names must follow the same convention as Python code:
224 start with a letter and consist of letters, digits and underscores
225 only.
226
227 The values can be either an integer, float, boolean (True or False), a
228 string or None. Strings must be enclosed in either single or double
229 quotes.
230
231 Complex types are supported as well.
232
233 A list is enclosed in square brackets and items are separated with com‐
234 mas. There can be a comma after the last item as well.
235
236 a_list = [1,
237 2,
238 3,
239 ]
240
241 A tuple works like a list, but is enclosed in parenthesis.
242
243 a_tuple = (1, "one")
244
245 A dictionary is wrapped in brackets, and consists of key: value pairs
246 separated by commas. The keys can only be formed from basic types (int,
247 float, string).
248
249 a_dict = {
250 'foo': 'bar',
251 1: None
252 }
253
254 The value assigned to a variable can also be taken from another vari‐
255 able.
256
257 one = 1
258 another = one
259
260 Anything on a line after a # symbol is ignored and functions as a com‐
261 ment.
262
263 Importing other files
264 It is possible to include another configuration file. The files are
265 looked up relative to the currently processed file.
266
267 The general structure of import is:
268
269 from FILENAME import WHAT
270
271 The FILENAME should be just the base name of the file without extension
272 (which must be .conf). WHAT can either be a comma separated list of
273 variables or *.
274
275 # Opens constants.conf and brings PI and E into current scope.
276 from constants import PI, E
277
278 # Opens common.conf and brings everything defined in that file into current
279 # file as well.
280 from common import *
281
282 NOTE:
283 Pungi will copy the configuration file given on command line into
284 the logs/ directory. Only this single file will be copied, not any
285 included ones. (Copying included files requires a fix in kobo
286 library.)
287
288 The JSON-formatted dump of configuration is correct though.
289
290 Formatting strings
291 String interpolation is available as well. It uses a %-encoded format.
292 See Python documentation for more details.
293
294 joined = "%s %s" % (var_a, var_b)
295
296 a_dict = {
297 "fst": 1,
298 "snd": 2,
299 }
300 another = "%(fst)s %(snd)s" % a_dict
301
303 Please read productmd documentation for terminology and other release
304 and compose related details.
305
306 Minimal Config Example
307 # RELEASE
308 release_name = "Fedora"
309 release_short = "Fedora"
310 release_version = "23"
311
312 # GENERAL SETTINGS
313 comps_file = "comps-f23.xml"
314 variants_file = "variants-f23.xml"
315
316 # KOJI
317 koji_profile = "koji"
318 runroot = False
319
320 # PKGSET
321 sigkeys = [None]
322 pkgset_source = "koji"
323 pkgset_koji_tag = "f23"
324
325 # GATHER
326 gather_method = "deps"
327 greedy_method = "build"
328 check_deps = False
329
330 # BUILDINSTALL
331 buildinstall_method = "lorax"
332
333 Release
334 Following mandatory options describe a release.
335
336 Options
337 release_name [mandatory]
338 (str) – release name
339
340 release_short [mandatory]
341 (str) – release short name, without spaces and special charac‐
342 ters
343
344 release_version [mandatory]
345 (str) – release version
346
347 release_type = “ga” (str) – release type, for example ga,
348 updates or updates-testing. See list of all valid values in pro‐
349 ductmd documentation.
350
351 release_internal = False
352 (bool) – whether the compose is meant for public consumption
353
354 treeinfo_version
355 (str) Version to display in .treeinfo files. If not configured,
356 the value from release_version will be used.
357
358 Example
359 release_name = "Fedora"
360 release_short = "Fedora"
361 release_version = "23"
362 # release_type = "ga"
363
364 Base Product
365 Base product options are optional and we need to them only if we’re
366 composing a layered product built on another (base) product.
367
368 Options
369 base_product_name
370 (str) – base product name
371
372 base_product_short
373 (str) – base product short name, without spaces and special
374 characters
375
376 base_product_version
377 (str) – base product major version
378
379 base_product_type = “ga”
380 (str) – base product type, “ga”, “updates” etc., for full list
381 see documentation of productmd.
382
383 Example
384 release_name = "RPM Fusion"
385 release_short = "rf"
386 release_version = "23.0"
387
388 base_product_name = "Fedora"
389 base_product_short = "Fedora"
390 base_product_version = "23"
391
392 General Settings
393 Options
394 comps_file [mandatory]
395 (scm_dict, str or None) – reference to comps XML file with
396 installation groups
397
398 variants_file [mandatory]
399 (scm_dict or str) – reference to variants XML file that defines
400 release variants and architectures
401
402 module_defaults_dir [optional]
403 (scm_dict or str) – reference the module defaults directory con‐
404 taining modulemd-defaults YAML documents. Files relevant for
405 modules included in the compose will be embedded in the gener‐
406 ated repodata and available for DNF.
407
408 module_defaults_dir = {
409 "scm": "git",
410 "repo": "https://pagure.io/releng/fedora-module-defaults.git",
411 "dir": ".",
412 }
413
414 failable_deliverables [optional]
415 (list) – list which deliverables on which variant and architec‐
416 ture can fail and not abort the whole compose. This only applies
417 to buildinstall and iso parts. All other artifacts can be con‐
418 figured in their respective part of configuration.
419
420 Please note that * as a wildcard matches all architectures but
421 src.
422
423 comps_filter_environments [optional]
424 (bool) – When set to False, the comps files for variants will
425 not have their environments filtered to match the variant.
426
427 tree_arches
428 ([str]) – list of architectures which should be included; if
429 undefined, all architectures from variants.xml will be included
430
431 tree_variants
432 ([str]) – list of variants which should be included; if unde‐
433 fined, all variants from variants.xml will be included
434
435 repoclosure_strictness
436 (list) – variant/arch mapping describing how repoclosure should
437 run. Possible values are
438
439 · off – do not run repoclosure
440
441 · lenient – (default) run repoclosure and write results to
442 logs, but detected errors are only reported in logs
443
444 · fatal – abort compose when any issue is detected
445
446 When multiple blocks in the mapping match a variant/arch combi‐
447 nation, the last value will win.
448
449 repoclosure_backend
450 (str) – Select which tool should be used to run repoclosure over
451 created repositories. By default yum is used, but you can switch
452 to dnf. Please note that when dnf is used, the build dependen‐
453 cies check is skipped. On Python 3, only dnf backend is avail‐
454 able.
455
456 cts_url
457 (str) – URL to Compose Tracking Service. If defined, Pungi will
458 add the compose to Compose Tracking Service and ge the compose
459 ID from it. For example https://cts.localhost.tld/
460
461 cts_keytab
462 (str) – Path to Kerberos keytab which will be used for Compose
463 Tracking Service Kerberos authentication. If not defined, the
464 default Kerberos principal is used.
465
466 compose_type
467 (str) – Allows to set default compose type. Type set via a com‐
468 mand-line option overwrites this.
469
470 mbs_api_url
471 (str) – URL to Module Build Service (MBS) API. For example
472 https://mbs.example.com/module-build-service/2. This is
473 required by pkgset_scratch_modules.
474
475 Example
476 comps_file = {
477 "scm": "git",
478 "repo": "https://git.fedorahosted.org/git/comps.git",
479 "branch": None,
480 "file": "comps-f23.xml.in",
481 }
482
483 variants_file = {
484 "scm": "git",
485 "repo": "https://pagure.io/pungi-fedora.git ",
486 "branch": None,
487 "file": "variants-fedora.xml",
488 }
489
490 failable_deliverables = [
491 ('^.*$', {
492 # Buildinstall can fail on any variant and any arch
493 '*': ['buildinstall'],
494 'src': ['buildinstall'],
495 # Nothing on i386 blocks the compose
496 'i386': ['buildinstall', 'iso', 'live'],
497 })
498 ]
499
500 tree_arches = ["x86_64"]
501 tree_variants = ["Server"]
502
503 repoclosure_strictness = [
504 # Make repoclosure failures fatal for compose on all variants …
505 ('^.*$', {'*': 'fatal'}),
506 # … except for Everything where it should not run at all.
507 ('^Everything$', {'*': 'off'})
508 ]
509
510 Image Naming
511 Both image name and volume id are generated based on the configuration.
512 Since the volume id is limited to 32 characters, there are more set‐
513 tings available. The process for generating volume id is to get a list
514 of possible formats and try them sequentially until one fits in the
515 length limit. If substitutions are configured, each attempted volume id
516 will be modified by it.
517
518 For layered products, the candidate formats are first image_volid_lay‐
519 ered_product_formats followed by image_volid_formats. Otherwise, only
520 image_volid_formats are tried.
521
522 If no format matches the length limit, an error will be reported and
523 compose aborted.
524
525 Options
526 There a couple common format specifiers available for both the options:
527
528 · compose_id
529
530 · release_short
531
532 · version
533
534 · date
535
536 · respin
537
538 · type
539
540 · type_suffix
541
542 · label
543
544 · label_major_version
545
546 · variant
547
548 · arch
549
550 · disc_type
551
552 image_name_format [optional]
553 (str|dict) – Python’s format string to serve as template for
554 image names. The value can also be a dict mapping variant UID
555 regexes to the format string. The pattern should not overlap,
556 otherwise it is undefined which one will be used.
557
558 This format will be used for all phases generating images. Cur‐
559 rently that means createiso, live_images and buildinstall.
560
561 Available extra keys are:
562
563 · disc_num
564
565 · suffix
566
567 image_volid_formats [optional]
568 (list) – A list of format strings for generating volume id.
569
570 The extra available keys are:
571
572 · base_product_short
573
574 · base_product_version
575
576 image_volid_layered_product_formats [optional]
577 (list) – A list of format strings for generating volume id for
578 layered products. The keys available are the same as for
579 image_volid_formats.
580
581 restricted_volid = False
582 (bool) – New versions of lorax replace all non-alphanumerical
583 characters with dashes (underscores are preserved). This option
584 will mimic similar behaviour in Pungi.
585
586 volume_id_substitutions [optional]
587 (dict) – A mapping of string replacements to shorten the volume
588 id.
589
590 disc_types [optional]
591 (dict) – A mapping for customizing disc_type used in image
592 names.
593
594 Available keys are:
595
596 · boot – for boot.iso images created in buildinstall
597 phase
598
599 · live – for images created by live_images phase
600
601 · dvd – for images created by createiso phase
602
603 · ostree – for ostree installer images
604
605 Default values are the same as the keys.
606
607 Example
608 # Image name respecting Fedora's image naming policy
609 image_name_format = "%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s%(suffix)s"
610 # Use the same format for volume id
611 image_volid_formats = [
612 "%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s"
613 ]
614 # No special handling for layered products, use same format as for regular images
615 image_volid_layered_product_formats = []
616 # Replace "Cloud" with "C" in volume id etc.
617 volume_id_substitutions = {
618 'Cloud': 'C',
619 'Alpha': 'A',
620 'Beta': 'B',
621 'TC': 'T',
622 }
623
624 disc_types = {
625 'boot': 'netinst',
626 'live': 'Live',
627 'dvd': 'DVD',
628 }
629
630 Signing
631 If you want to sign deliverables generated during pungi run like RPM
632 wrapped images. You must provide few configuration options:
633
634 signing_command [optional]
635 (str) – Command that will be run with a koji build as a single
636 argument. This command must not require any user interaction.
637 If you need to pass a password for a signing key to the command,
638 do this via command line option of the command and use string
639 formatting syntax %(signing_key_password)s. (See sign‐
640 ing_key_password_file).
641
642 signing_key_id [optional]
643 (str) – ID of the key that will be used for the signing. This
644 ID will be used when crafting koji paths to signed files (kojip‐
645 kgs.fedoraproject.org/pack‐
646 ages/NAME/VER/REL/data/signed/KEYID/..).
647
648 signing_key_password_file [optional]
649 (str) – Path to a file with password that will be formatted into
650 signing_command string via %(signing_key_password)s string for‐
651 mat syntax (if used). Because pungi config is usually stored in
652 git and is part of compose logs we don’t want password to be
653 included directly in the config. Note: If - string is used
654 instead of a filename, then you will be asked for the password
655 interactivelly right after pungi starts.
656
657 Example
658 signing_command = '~/git/releng/scripts/sigulsign_unsigned.py -vv --password=%(signing_key_password)s fedora-24'
659 signing_key_id = '81b46521'
660 signing_key_password_file = '~/password_for_fedora-24_key'
661
662 Git URLs
663 In multiple places the config requires URL of a Git repository to down‐
664 load some file from. This URL is passed on to Koji. It is possible to
665 specify which commit to use using this syntax:
666
667 git://git.example.com/git/repo-name.git?#<rev_spec>
668
669 The <rev_spec> pattern can be replaced with actual commit SHA, a tag
670 name, HEAD to indicate that tip of default branch should be used or
671 origin/<branch_name> to use tip of arbitrary branch.
672
673 If the URL specifies a branch or HEAD, Pungi will replace it with the
674 actual commit SHA. This will later show up in Koji tasks and help with
675 tracing what particular inputs were used.
676
677 NOTE:
678 The origin must be specified because of the way Koji works with the
679 repository. It will clone the repository then switch to requested
680 state with git reset --hard REF. Since no local branches are cre‐
681 ated, we need to use full specification including the name of the
682 remote.
683
684 Createrepo Settings
685 Options
686 createrepo_checksum
687 (str) – specify checksum type for createrepo; expected values:
688 sha512, sha256, sha1. Defaults to sha256.
689
690 createrepo_c = True
691 (bool) – use createrepo_c (True) or legacy createrepo (False)
692
693 createrepo_deltas = False
694 (list) – generate delta RPMs against an older compose. This
695 needs to be used together with --old-composes command line argu‐
696 ment. The value should be a mapping of variants and architec‐
697 tures that should enable creating delta RPMs. Source and debug‐
698 info repos never have deltas.
699
700 createrepo_use_xz = False
701 (bool) – whether to pass --xz to the createrepo command. This
702 will cause the SQLite databases to be compressed with xz.
703
704 createrepo_num_threads
705 (int) – how many concurrent createrepo process to run. The
706 default is to use one thread per CPU available on the machine.
707
708 createrepo_num_workers
709 (int) – how many concurrent createrepo workers to run. Value
710 defaults to 3.
711
712 createrepo_database
713 (bool) – whether to create SQLite database as part of the repo‐
714 data. This is only useful as an optimization for clients using
715 Yum to consume to the repo. Default value depends on gather
716 backend. For DNF it’s turned off, for Yum the default is True.
717
718 createrepo_extra_args
719 ([str]) – a list of extra arguments passed on to createrepo or
720 createrepo_c executable. This could be useful for enabling
721 zchunk generation and pointing it to correct dictionaries.
722
723 createrepo_extra_modulemd
724 (dict) – a mapping of variant UID to an scm dict. If specified,
725 it should point to a directory with extra module metadata YAML
726 files that will be added to the repository for this variant. The
727 cloned files should be split into subdirectories for each archi‐
728 tecture of the variant.
729
730 product_id = None
731 (scm_dict) – If specified, it should point to a directory with
732 certificates *<variant_uid>-<arch>-*.pem. Pungi will copy each
733 certificate file into the relevant Yum repositories as a produc‐
734 tid file in the repodata directories. The purpose of these pro‐
735 ductid files is to expose the product data to
736 subscription-manager. subscription-manager includes a “prod‐
737 uct-id” Yum plugin that can read these productid certificate
738 files from each Yum repository.
739
740 product_id_allow_missing = False
741 (bool) – When product_id is used and a certificate for some
742 variant and architecture is missing, Pungi will exit with an
743 error by default. When you set this option to True, Pungi will
744 ignore the missing certificate and simply log a warning message.
745
746 product_id_allow_name_prefix = True
747 (bool) – Allow arbitrary prefix for the certificate file name
748 (see leading * in the pattern above). Setting this option to
749 False will make the pattern more strict by requiring the file
750 name to start directly with variant name.
751
752 Example
753 createrepo_checksum = "sha"
754 createrepo_deltas = [
755 # All arches for Everything should have deltas.
756 ('^Everything$', {'*': True}),
757 # Also Server.x86_64 should have them (but not on other arches).
758 ('^Server$', {'x86_64': True}),
759 ]
760 createrepo_extra_modulemd = {
761 "Server": {
762 "scm": "git",
763 "repo": "https://example.com/extra-server-modulemd.git",
764 "dir": ".",
765 # The directory should have this layout. Each architecture for the
766 # variant should be included (even if the directory is empty.
767 # .
768 # ├── aarch64
769 # │ ├── some-file.yaml
770 # │ └ ...
771 # └── x86_64
772 }
773 }
774
775 Package Set Settings
776 Options
777 sigkeys
778 ([str or None]) – priority list of signing key IDs. These key
779 IDs match the key IDs for the builds in Koji. Pungi will choose
780 signed packages according to the order of the key IDs that you
781 specify here. Use one single key in this list to ensure that all
782 RPMs are signed by one key. If the list includes an empty string
783 or None, Pungi will allow unsigned packages. If the list only
784 includes None, Pungi will use all unsigned packages.
785
786 pkgset_source [mandatory]
787 (str) – “koji” (any koji instance) or “repos” (arbitrary yum
788 repositories)
789
790 pkgset_koji_tag
791 (str|[str]) – tag(s) to read package set from. This option can
792 be omitted for modular composes.
793
794 pkgset_koji_builds
795 (str|[str]) – extra build(s) to include in a package set defined
796 as NVRs.
797
798 pkgset_koji_scratch_tasks
799 (str|[str]) – RPM scratch build task(s) to include in a package
800 set, defined as task IDs. This option can be used only when com‐
801 pose_type is set to test. The RPM still needs to have higher NVR
802 than any other RPM with the same name coming from other sources
803 in order to appear in the resulting compose.
804
805 pkgset_koji_module_tag
806 (str|[str]) – tags to read module from. This option works simi‐
807 larly to listing tags in variants XML. If tags are specified and
808 variants XML specifies some modules via NSVC (or part of), only
809 modules matching that list will be used (and taken from the
810 tag). Inheritance is used automatically.
811
812 pkgset_koji_module_builds
813 (dict) – A mapping of variants to extra module builds to include
814 in a package set: {variant: [N:S:V:C]}.
815
816 pkgset_koji_inherit = True
817 (bool) – inherit builds from parent tags; we can turn it off
818 only if we have all builds tagged in a single tag
819
820 pkgset_koji_inherit_modules = False
821 (bool) – the same as above, but this only applies to modular
822 tags. This option applies to the content tags that contain the
823 RPMs.
824
825 pkgset_repos
826 (dict) – A mapping of architectures to repositories with RPMs:
827 {arch: [repo]}. Only use when pkgset_source = "repos".
828
829 pkgset_scratch_modules
830 (dict) – A mapping of variants to scratch module builds: {vari‐
831 ant: [N:S:V:C]}. Requires mbs_api_url.
832
833 pkgset_exclusive_arch_considers_noarch = True
834 (bool) – If a package includes noarch in its ExclusiveArch tag,
835 it will be included in all architectures since noarch is compat‐
836 ible with everything. Set this option to False to ignore noarch
837 in ExclusiveArch and always consider only binary architectures.
838
839 pkgset_allow_reuse = True
840 (bool) – When set to True, Pungi will try to reuse pkgset data
841 from the old composes specified by --old-composes. When enabled,
842 this option can speed up new composes because it does not need
843 to calculate the pkgset data from Koji. However, if you block or
844 unblock a package in Koji (for example) between composes, then
845 Pungi may not respect those changes in your new compose.
846
847 signed_packages_retries = 1
848 (int) – In automated workflows a compose may start before signed
849 packages are written to disk. In such case it may make sense to
850 wait for the package to appear on storage. This option controls
851 how many times to try to look for the signed copy.
852
853 signed_packages_wait = 30
854 (int) – Interval in seconds for how long to wait between attemts
855 to find signed packages. This option only makes sense when
856 signed_packages_retries is set higher than to 1.
857
858 Example
859 sigkeys = [None]
860 pkgset_source = "koji"
861 pkgset_koji_tag = "f23"
862
863 Buildinstall Settings
864 Script or process that creates bootable images with Anaconda installer
865 is historically called buildinstall.
866
867 Options
868 buildinstall_method
869 (str) – “lorax” (f16+, rhel7+) or “buildinstall” (older
870 releases)
871
872 lorax_options
873 (list) – special options passed on to lorax.
874
875 Format: [(variant_uid_regex, {arch|*: {option: name}})].
876
877 Recognized options are:
878
879 · bugurl – str (default None)
880
881 · nomacboot – bool (default True)
882
883 · noupgrade – bool (default True)
884
885 · add_template – [str] (default empty)
886
887 · add_arch_template – [str] (default empty)
888
889 · add_template_var – [str] (default empty)
890
891 · add_arch_template_var – [str] (default empty)
892
893 · rootfs_size – [int] (default empty)
894
895 · version – [str] (default from treeinfo_version or
896 release_version) – used as --version and --release
897 argument on the lorax command line
898
899 · dracut_args – [[str]] (default empty) override argu‐
900 ments for dracut. Please note that if this option is
901 used, lorax will not use any other arguments, so you
902 have to provide a full list and can not just add some‐
903 thing.
904
905 · skip_branding – bool (default False)
906
907 · squashfs_only – bool (default False) pass the
908 –squashfs_only to Lorax.
909
910 · configuration_file – (scm_dict) (default empty) pass
911 the specified configuration file to Lorax using the -c
912 option.
913
914 lorax_extra_sources
915 (list) – a variant/arch mapping with urls for extra source
916 repositories added to Lorax command line. Either one repo or a
917 list can be specified.
918
919 lorax_use_koji_plugin = False
920 (bool) – When set to True, the Koji pungi_buildinstall task will
921 be used to execute Lorax instead of runroot. Use only if the
922 Koji instance has the pungi_buildinstall plugin installed.
923
924 buildinstall_kickstart
925 (scm_dict) – If specified, this kickstart file will be copied
926 into each file and pointed to in boot configuration.
927
928 buildinstall_topdir
929 (str) – Full path to top directory where the runroot buildin‐
930 stall Koji tasks output should be stored. This is useful in sit‐
931 uation when the Pungi compose is not generated on the same stor‐
932 age as the Koji task is running on. In this case, Pungi can pro‐
933 vide input repository for runroot task using HTTP and set the
934 output directory for this task to buildinstall_topdir. Once the
935 runroot task finishes, Pungi will copy the results of runroot
936 tasks to the compose working directory.
937
938 buildinstall_skip
939 (list) – mapping that defines which variants and arches to skip
940 during buildinstall; format: [(variant_uid_regex, {arch|*:
941 True})]. This is only supported for lorax.
942
943 buildinstall_allow_reuse = False
944 (bool) – When set to True, Pungi will try to reuse buildinstall
945 results from old compose specified by --old-composes.
946
947 Example
948 buildinstall_method = "lorax"
949
950 # Enables macboot on x86_64 for all variants and builds upgrade images
951 # everywhere.
952 lorax_options = [
953 ("^.*$", {
954 "x86_64": {
955 "nomacboot": False
956 }
957 "*": {
958 "noupgrade": False
959 }
960 })
961 ]
962
963 # Don't run buildinstall phase for Modular variant
964 buildinstall_skip = [
965 ('^Modular', {
966 '*': True
967 })
968 ]
969
970 # Add another repository for lorax to install packages from
971 lorax_extra_sources = [
972 ('^Simple$', {
973 '*': 'https://example.com/repo/$basearch/',
974 })
975 ]
976
977 NOTE:
978 It is advised to run buildinstall (lorax) in koji, i.e. with runroot
979 enabled for clean build environments, better logging, etc.
980
981 WARNING:
982 Lorax installs RPMs into a chroot. This involves running %post
983 scriptlets and they frequently run executables in the chroot. If
984 we’re composing for multiple architectures, we must use runroot for
985 this reason.
986
987 Gather Settings
988 Options
989 gather_method [mandatory]
990 (str*|*dict) – Options are deps, nodeps and hybrid. Specifies
991 whether and how package dependencies should be pulled in. Pos‐
992 sible configuration can be one value for all variants, or if
993 configured per-variant it can be a simple string hybrid or a a
994 dictionary mapping source type to a value of deps or nodeps.
995 Make sure only one regex matches each variant, as there is no
996 guarantee which value will be used if there are multiple match‐
997 ing ones. All used sources must have a configured method unless
998 hybrid solving is used.
999
1000 gather_fulltree = False
1001 (bool) – When set to True all RPMs built from an SRPM will
1002 always be included. Only use when gather_method = "deps".
1003
1004 gather_selfhosting = False
1005 (bool) – When set to True, Pungi will build a self-hosting tree
1006 by following build dependencies. Only use when gather_method =
1007 "deps".
1008
1009 gather_allow_reuse = False
1010 (bool) – When set to True, Pungi will try to reuse gather
1011 results from old compose specified by --old-composes.
1012
1013 greedy_method
1014 (str) – This option controls how package requirements are satis‐
1015 fied in case a particular Requires has multiple candidates.
1016
1017 · none – the best packages is selected to satisfy the dependency
1018 and only that one is pulled into the compose
1019
1020 · all – packages that provide the symbol are pulled in
1021
1022 · build – the best package is selected, and then all packages
1023 from the same build that provide the symbol are pulled in
1024
1025 NOTE:
1026 As an example let’s work with this situation: a package in
1027 the compose has Requires: foo. There are three packages with
1028 Provides: foo: pkg-a, pkg-b-provider-1 and pkg-b-provider-2.
1029 The pkg-b-* packages are build from the same source package.
1030 Best match determines pkg-b-provider-1 as best matching pack‐
1031 age.
1032
1033 · With greedy_method = "none" only pkg-b-provider-1 will be
1034 pulled in.
1035
1036 · With greedy_method = "all" all three packages will be
1037 pulled in.
1038
1039 · With greedy_method = "build" ``pkg-b-provider-1 and
1040 pkg-b-provider-2 will be pulled in.
1041
1042 gather_backend
1043 (str) –This changes the entire codebase doing dependency solv‐
1044 ing, so it can change the result in unpredictable ways.
1045
1046 On Python 2, the choice is between yum or dnf and defaults to
1047 yum. On Python 3 dnf is the only option and default.
1048
1049 Particularly the multilib work is performed differently by using
1050 python-multilib library. Please refer to multilib option to see
1051 the differences.
1052
1053 multilib
1054 (list) – mapping of variant regexes and arches to list of multi‐
1055 lib methods
1056
1057 Available methods are:
1058
1059 · none – no package matches this method
1060
1061 · all – all packages match this method
1062
1063 · runtime – packages that install some shared object file
1064 (*.so.*) will match.
1065
1066 · devel – packages whose name ends with -devel or
1067 --static suffix will be matched. When dnf is used, this
1068 method automatically enables runtime method as well.
1069 With yum backend this method also uses a hardcoded
1070 blacklist and whitelist.
1071
1072 · kernel – packages providing kernel or kernel-devel
1073 match this method (only in yum backend)
1074
1075 · yaboot – only yaboot package on ppc arch matches this
1076 (only in yum backend)
1077
1078 additional_packages
1079 (list) – additional packages to be included in a variant and
1080 architecture; format: [(variant_uid_regex, {arch|*: [pack‐
1081 age_globs]})]
1082
1083 The packages specified here are matched against RPM names, not
1084 any other provides in the package not the name of source pack‐
1085 age. Shell globbing is used, so wildcards are possible. The
1086 package can be specified as name only or name.arch.
1087
1088 With dnf gathering backend, you can specify a debuginfo package
1089 to be included. This is meant to include a package if autodetec‐
1090 tion does not get it. If you add a debuginfo package that does
1091 not have anything else from the same build included in the com‐
1092 pose, the sources will not be pulled in.
1093
1094 filter_packages
1095 (list) – packages to be excluded from a variant and architec‐
1096 ture; format: [(variant_uid_regex, {arch|*: [package_globs]})]
1097
1098 See additional_packages for details about package specification.
1099
1100 filter_modules
1101 (list) – modules to be excluded from a variant and architecture;
1102 format: [(variant_uid_regex, {arch|*: [name:stream]})]
1103
1104 Both name and stream can use shell-style globs. If stream is
1105 omitted, all streams are removed.
1106
1107 This option only applies to modules taken from Koji tags, not
1108 modules explicitly listed in variants XML without any tags.
1109
1110 filter_system_release_packages
1111 (bool) – for each variant, figure out the best system release
1112 package and filter out all others. This will not work if a vari‐
1113 ant needs more than one system release package. In such case,
1114 set this option to False.
1115
1116 gather_prepopulate = None
1117 (scm_dict) – If specified, you can use this to add additional
1118 packages. The format of the file pointed to by this option is a
1119 JSON mapping {variant_uid: {arch: {build: [package]}}}. Packages
1120 added through this option can not be removed by filter_packages.
1121
1122 multilib_blacklist
1123 (dict) – multilib blacklist; format: {arch|*: [package_globs]}.
1124
1125 See additional_packages for details about package specification.
1126
1127 multilib_whitelist
1128 (dict) – multilib blacklist; format: {arch|*: [package_names]}.
1129 The whitelist must contain exact package names; there are no
1130 wildcards or pattern matching.
1131
1132 gather_lookaside_repos = []
1133 (list) – lookaside repositories used for package gathering; for‐
1134 mat: [(variant_uid_regex, {arch|*: [repo_urls]})]
1135
1136 The repo_urls are passed to the depsolver, which can use pack‐
1137 ages in the repos for satisfying dependencies, but the packages
1138 themselves are not pulled into the compose. The repo_urls can
1139 contain $basearch variable, which will be substituted with
1140 proper value by the depsolver.
1141
1142 The repo_urls are used by repoclosure too, but it can’t parse
1143 $basearch currently and that will cause Repoclosure phase
1144 crashed. repoclosure_strictness option could be used to stop
1145 running repoclosure.
1146
1147 Please note that * as a wildcard matches all architectures but
1148 src.
1149
1150 hashed_directories = False
1151 (bool) – put packages into “hashed” directories, for example
1152 Packages/k/kernel-4.0.4-301.fc22.x86_64.rpm
1153
1154 check_deps = True
1155 (bool) – Set to False if you don’t want the compose to abort
1156 when some package has broken dependencies.
1157
1158 require_all_comps_packages = False
1159 (bool) – Set to True to abort compose when package mentioned in
1160 comps file can not be found in the package set. When disabled
1161 (the default), such cases are still reported as warnings in the
1162 log.
1163
1164 gather_source_mapping
1165 (str) – JSON mapping with initial packages for the compose. The
1166 value should be a path to JSON file with following mapping:
1167 {variant: {arch: {rpm_name: [rpm_arch|None]}}}.
1168
1169 gather_profiler = False
1170 (bool) – When set to True the gather tool will produce addi‐
1171 tional performance profiling information at the end of its logs.
1172 Only takes effect when gather_backend = "dnf".
1173
1174 variant_as_lookaside
1175 (list) – a variant/variant mapping that tells one or more vari‐
1176 ants in compose has other variant(s) in compose as a lookaside.
1177 Only top level variants are supported (not addons/layered prod‐
1178 ucts). Format: [(variant_uid, variant_uid)]
1179
1180 Example
1181 gather_method = "deps"
1182 greedy_method = "build"
1183 check_deps = False
1184 hashed_directories = True
1185
1186 gather_method = {
1187 "^Everything$": {
1188 "comps": "deps" # traditional content defined by comps groups
1189 },
1190 "^Modular$": {
1191 "module": "nodeps" # Modules do not need dependencies
1192 },
1193 "^Mixed$": { # Mixed content in one variant
1194 "comps": "deps",
1195 "module": "nodeps"
1196 }
1197 "^OtherMixed$": "hybrid", # Using hybrid depsolver
1198 }
1199
1200 additional_packages = [
1201 # bz#123456
1202 ('^(Workstation|Server)$', {
1203 '*': [
1204 'grub2',
1205 'kernel',
1206 ],
1207 }),
1208 ]
1209
1210 filter_packages = [
1211 # bz#111222
1212 ('^.*$', {
1213 '*': [
1214 'kernel-doc',
1215 ],
1216 }),
1217 ]
1218
1219 multilib = [
1220 ('^Server$', {
1221 'x86_64': ['devel', 'runtime']
1222 })
1223 ]
1224
1225 multilib_blacklist = {
1226 "*": [
1227 "gcc",
1228 ],
1229 }
1230
1231 multilib_whitelist = {
1232 "*": [
1233 "alsa-plugins-*",
1234 ],
1235 }
1236
1237 # gather_lookaside_repos = [
1238 # ('^.*$', {
1239 # '*': [
1240 # "https://dl.fedoraproject.org/pub/fedora/linux/releases/22/Everything/$basearch/os/",
1241 # ],
1242 # 'x86_64': [
1243 # "https://dl.fedoraproject.org/pub/fedora/linux/releases/22/Everything/source/SRPMS/",
1244 # ]
1245 # }),
1246 # ]
1247
1248 NOTE:
1249 It is a good practice to attach bug/ticket numbers to addi‐
1250 tional_packages, filter_packages, multilib_blacklist and multi‐
1251 lib_whitelist to track decisions.
1252
1253 Koji Settings
1254 Options
1255 koji_profile
1256 (str) – koji profile name. This tells Pungi how to communicate
1257 with your chosen Koji instance. See Koji’s documentation about
1258 profiles for more information about how to set up your Koji
1259 client profile. In the examples, the profile name is “koji”,
1260 which points to Fedora’s koji.fedoraproject.org.
1261
1262 global_runroot_method
1263 (str) – global runroot method to use. If runroot_method is set
1264 per Pungi phase using a dictionary, this option defines the
1265 default runroot method for phases not mentioned in the run‐
1266 root_method dictionary.
1267
1268 runroot_method
1269 (str*|*dict) – Runroot method to use. It can further specify the
1270 runroot method in case the runroot is set to True.
1271
1272 Available methods are:
1273
1274 · local – runroot tasks are run locally
1275
1276 · koji – runroot tasks are run in Koji
1277
1278 · openssh – runroot tasks are run on remote machine con‐
1279 nected using OpenSSH. The runroot_ssh_hostnames for
1280 each architecture must be set and the user under which
1281 Pungi runs must be configured to login as run‐
1282 root_ssh_username using the SSH key.
1283
1284 The runroot method can also be set per Pungi phase using the
1285 dictionary with phase name as key and runroot method as value.
1286 The default runroot method is in this case defined by the
1287 global_runroot_method option.
1288
1289 Example
1290 global_runroot_method = "koji"
1291 runroot_method = {
1292 "createiso": "local"
1293 }
1294
1295 runroot_channel
1296 (str) – name of koji channel
1297
1298 runroot_tag
1299 (str) – name of koji build tag used for runroot
1300
1301 runroot_weights
1302 (dict) – customize task weights for various runroot tasks. The
1303 values in the mapping should be integers, the keys can be
1304 selected from the following list. By default no weight is
1305 assigned and Koji picks the default one according to policy.
1306
1307 · buildinstall
1308
1309 · createiso
1310
1311 · ostree
1312
1313 · ostree_installer
1314
1315 Example
1316 koji_profile = "koji"
1317 runroot_channel = "runroot"
1318 runroot_tag = "f23-build"
1319
1320 Runroot “openssh” method settings
1321 Options
1322 runroot_ssh_username
1323 (str) – For openssh runroot method, configures the username used
1324 to login the remote machine to run the runroot task. Defaults to
1325 “root”.
1326
1327 runroot_ssh_hostnames
1328 (dict) – For openssh runroot method, defines the hostname for
1329 each architecture on which the runroot task should be running.
1330 Format: {"x86_64": "runroot-x86-64.localhost.tld", ...}
1331
1332 runroot_ssh_init_template
1333 (str) [optional] – For openssh runroot method, defines the com‐
1334 mand to initializes the runroot task on the remote machine. This
1335 command is executed as first command for each runroot task exe‐
1336 cuted.
1337
1338 The command can print a string which is then available as {run‐
1339 root_key} for other SSH commands. This string might be used to
1340 keep the context across different SSH commands executed for sin‐
1341 gle runroot task.
1342
1343 The goal of this command is setting up the environment for real
1344 runroot commands. For example preparing the unique mock environ‐
1345 ment, mounting the desired file-systems, …
1346
1347 The command string can contain following variables which are
1348 replaced by the real values before executing the init command:
1349
1350 · {runroot_tag} - Tag to initialize the runroot environment
1351 from.
1352
1353 When not set, no init command is executed.
1354
1355 runroot_ssh_install_packages_template
1356 (str) [optional] – For openssh runroot method, defines the tem‐
1357 plate for command to install the packages requested to run the
1358 runroot task.
1359
1360 The template string can contain following variables which are
1361 replaced by the real values before executing the install com‐
1362 mand:
1363
1364 · {runroot_key} - Replaced with the string returned by run‐
1365 root_ssh_init_template if used. This can be used to keep the
1366 track of context of SSH commands belonging to single runroot
1367 task.
1368
1369 · {packages} - White-list separated list of packages to install.
1370
1371 Example (The {runroot_key} is expected to be set to mock config
1372 file using the runroot_ssh_init_template command.): "mock -r
1373 {runroot_key} --install {packages}"
1374
1375 When not set, no command to install packages on remote machine
1376 is executed.
1377
1378 runroot_ssh_run_template
1379 (str) [optional] – For openssh runroot method, defines the tem‐
1380 plate for the main runroot command.
1381
1382 The template string can contain following variables which are
1383 replaced by the real values before executing the install com‐
1384 mand:
1385
1386 · {runroot_key} - Replaced with the string returned by run‐
1387 root_ssh_init_template if used. This can be used to keep the
1388 track of context of SSH commands belonging to single runroot
1389 task.
1390
1391 · {command} - Command to run.
1392
1393 Example (The {runroot_key} is expected to be set to mock config
1394 file using the runroot_ssh_init_template command.): "mock -r
1395 {runroot_key} chroot -- {command}"
1396
1397 When not set, the runroot command is run directly.
1398
1399 Extra Files Settings
1400 Options
1401 extra_files
1402 (list) – references to external files to be placed in os/ direc‐
1403 tory and media; format: [(variant_uid_regex, {arch|*:
1404 [scm_dict]})]. See scm_support for details. If the dict speci‐
1405 fies a target key, an additional subdirectory will be used.
1406
1407 Example
1408 extra_files = [
1409 ('^.*$', {
1410 '*': [
1411 # GPG keys
1412 {
1413 "scm": "rpm",
1414 "repo": "fedora-repos",
1415 "branch": None,
1416 "file": [
1417 "/etc/pki/rpm-gpg/RPM-GPG-KEY-22-fedora",
1418 ],
1419 "target": "",
1420 },
1421 # GPL
1422 {
1423 "scm": "git",
1424 "repo": "https://pagure.io/pungi-fedora",
1425 "branch": None,
1426 "file": [
1427 "GPL",
1428 ],
1429 "target": "",
1430 },
1431 ],
1432 }),
1433 ]
1434
1435 Extra Files Metadata
1436 If extra files are specified a metadata file, extra_files.json, is
1437 placed in the os/ directory and media. The checksums generated are
1438 determined by media_checksums option. This metadata file is in the for‐
1439 mat:
1440
1441 {
1442 "header": {"version": "1.0},
1443 "data": [
1444 {
1445 "file": "GPL",
1446 "checksums": {
1447 "sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
1448 },
1449 "size": 18092
1450 },
1451 {
1452 "file": "release-notes/notes.html",
1453 "checksums": {
1454 "sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
1455 },
1456 "size": 1120
1457 }
1458 ]
1459 }
1460
1461 CreateISO Settings
1462 Options
1463 createiso_skip = False
1464 (list) – mapping that defines which variants and arches to skip
1465 during createiso; format: [(variant_uid_regex, {arch|*: True})]
1466
1467 createiso_max_size
1468 (list) – mapping that defines maximum expected size for each
1469 variant and arch. If the ISO is larger than the limit, a warning
1470 will be issued.
1471
1472 Format: [(variant_uid_regex, {arch|*: number})]
1473
1474 createiso_max_size_is_strict
1475 (list) – Set the value to True to turn the warning from cre‐
1476 ateiso_max_size into a hard error that will abort the compose.
1477 If there are multiple matches in the mapping, the check will be
1478 strict if at least one match says so.
1479
1480 Format: [(variant_uid_regex, {arch|*: bool})]
1481
1482 create_jigdo = True
1483 (bool) – controls the creation of jigdo from ISO
1484
1485 create_optional_isos = False
1486 (bool) – when set to True, ISOs will be created even for
1487 optional variants. By default only variants with type variant or
1488 layered-product will get ISOs.
1489
1490 createiso_break_hardlinks = False
1491 (bool) – when set to True, all files that should go on the ISO
1492 and have a hardlink will be first copied into a staging direc‐
1493 tory. This should work around a bug in genisoimage including
1494 incorrect link count in the image, but it is at the cost of hav‐
1495 ing to copy a potentially significant amount of data.
1496
1497 The staging directory is deleted when ISO is successfully cre‐
1498 ated. In that case the same task to create the ISO will not be
1499 re-runnable.
1500
1501 createiso_use_xorrisofs = False
1502 (bool) – when set to True, use xorrisofs for creating ISOs
1503 instead of genisoimage.
1504
1505 iso_size = 4700000000
1506 (int|str) – size of ISO image. The value should either be an
1507 integer meaning size in bytes, or it can be a string with k, M,
1508 G suffix (using multiples of 1024).
1509
1510 split_iso_reserve = 10MiB
1511 (int|str) – how much free space should be left on each disk. The
1512 format is the same as for iso_size option.
1513
1514 iso_hfs_ppc64le_compatible = True
1515 (bool) – when set to False, the Apple/HFS compatibility is
1516 turned off for ppc64le ISOs. This option only makes sense for
1517 bootable products, and affects images produced in createiso and
1518 extra_isos phases.
1519
1520 NOTE:
1521 Source architecture needs to be listed explicitly. Excluding ‘*’
1522 applies only on binary arches. Jigdo causes significant increase of
1523 time to ISO creation.
1524
1525 Example
1526 createiso_skip = [
1527 ('^Workstation$', {
1528 '*': True,
1529 'src': True
1530 }),
1531 ]
1532
1533 Automatic generation of version and release
1534 Version and release values for certain artifacts can be generated auto‐
1535 matically based on release version, compose label, date, type and
1536 respin. This can be used to shorten the config and keep it the same for
1537 multiple uses.
1538
1539┌───────────────────────┬───────────────┬──────────┬──────────┬────────┬──────────────┐
1540│Compose ID │ Label │ Version │ Date │ Respin │ Release │
1541├───────────────────────┼───────────────┼──────────┼──────────┼────────┼──────────────┤
1542│F-Rawhide-20170406.n.0 │ - │ Rawhide │ 20170406 │ 0 │ 20170406.n.0 │
1543├───────────────────────┼───────────────┼──────────┼──────────┼────────┼──────────────┤
1544│F-26-20170329.1 │ Alpha-1.6 │ 26_Alpha │ 20170329 │ 1 │ 1.6 │
1545├───────────────────────┼───────────────┼──────────┼──────────┼────────┼──────────────┤
1546│F-Atomic-25-20170407.0 │ RC-20170407.0 │ 25 │ 20170407 │ 0 │ 20170407.0 │
1547├───────────────────────┼───────────────┼──────────┼──────────┼────────┼──────────────┤
1548│F-Atomic-25-20170407.0 │ - │ 25 │ 20170407 │ 0 │ 20170407.0 │
1549└───────────────────────┴───────────────┴──────────┴──────────┴────────┴──────────────┘
1550
1551 All non-RC milestones from label get appended to the version. For
1552 release either label is used or date, type and respin.
1553
1554 Common options for Live Images, Live Media and Image Build
1555 All images can have ksurl, version, release and target specified. Since
1556 this can create a lot of duplication, there are global options that can
1557 be used instead.
1558
1559 For each of the phases, if the option is not specified for a particular
1560 deliverable, an option named <PHASE_NAME>_<OPTION> is checked. If that
1561 is not specified either, the last fallback is global_<OPTION>. If even
1562 that is unset, the value is considered to not be specified.
1563
1564 The kickstart URL is configured by these options.
1565
1566 · global_ksurl – global fallback setting
1567
1568 · live_media_ksurl
1569
1570 · image_build_ksurl
1571
1572 · live_images_ksurl
1573
1574 Target is specified by these settings.
1575
1576 · global_target – global fallback setting
1577
1578 · live_media_target
1579
1580 · image_build_target
1581
1582 · live_images_target
1583
1584 · osbuild_target
1585
1586 Version is specified by these options. If no version is set, a default
1587 value will be provided according to automatic versioning.
1588
1589 · global_version – global fallback setting
1590
1591 · live_media_version
1592
1593 · image_build_version
1594
1595 · live_images_version
1596
1597 · osbuild_version
1598
1599 Release is specified by these options. If set to a magic value to
1600 !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN, a value will be generated accord‐
1601 ing to automatic versioning.
1602
1603 · global_release – global fallback setting
1604
1605 · live_media_release
1606
1607 · image_build_release
1608
1609 · live_images_release
1610
1611 · osbuild_release
1612
1613 Each configuration block can also optionally specify a failable key.
1614 For live images it should have a boolean value. For live media and
1615 image build it should be a list of strings containing architectures
1616 that are optional. If any deliverable fails on an optional architec‐
1617 ture, it will not abort the whole compose. If the list contains only
1618 "*", all arches will be substituted.
1619
1620 Live Images Settings
1621 live_images
1622 (list) – Configuration for the particular image. The elements of
1623 the list should be tuples (variant_uid_regex, {arch|*: config}).
1624 The config should be a dict with these keys:
1625
1626 · kickstart (str)
1627
1628 · ksurl (str) [optional] – where to get the kickstart from
1629
1630 · name (str)
1631
1632 · version (str)
1633
1634 · target (str)
1635
1636 · repo (str|[str]) – repos specified by URL or variant UID
1637
1638 · specfile (str) – for images wrapped in RPM
1639
1640 · scratch (bool) – only RPM-wrapped images can use scratch
1641 builds, but by default this is turned off
1642
1643 · type (str) – what kind of task to start in Koji. Defaults
1644 to live meaning koji spin-livecd will be used. Alternative
1645 option is appliance corresponding to koji spin-appliance.
1646
1647 · sign (bool) – only RPM-wrapped images can be signed
1648
1649 live_images_no_rename
1650 (bool) – When set to True, filenames generated by Koji will be
1651 used. When False, filenames will be generated based on
1652 image_name_format configuration option.
1653
1654 Live Media Settings
1655 live_media
1656 (dict) – configuration for koji spin-livemedia; format: {vari‐
1657 ant_uid_regex: [{opt:value}]}
1658
1659 Required options:
1660
1661 · name (str)
1662
1663 · version (str)
1664
1665 · arches ([str]) – what architectures to build the media for;
1666 by default uses all arches for the variant.
1667
1668 · kickstart (str) – name of the kickstart file
1669
1670 Available options:
1671
1672 · ksurl (str)
1673
1674 · ksversion (str)
1675
1676 · scratch (bool)
1677
1678 · target (str)
1679
1680 · release (str) – a string with the release, or
1681 !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN to automatically gen‐
1682 erate a suitable value. See automatic versioning for
1683 details.
1684
1685 · skip_tag (bool)
1686
1687 · repo (str|[str]) – repos specified by URL or variant UID
1688
1689 · title (str)
1690
1691 · install_tree_from (str) – variant to take install tree from
1692
1693 Image Build Settings
1694 image_build
1695 (dict) – config for koji image-build; format: {vari‐
1696 ant_uid_regex: [{opt: value}]}
1697
1698 By default, images will be built for each binary arch valid for
1699 the variant. The config can specify a list of arches to narrow
1700 this down.
1701
1702 NOTE:
1703 Config can contain anything what is accepted by koji image-build
1704 --config configfile.ini
1705
1706 Repo can be specified either as a string or a list of strings. It
1707 will be automatically transformed into format suitable for koji. A
1708 repo for the currently built variant will be added as well.
1709
1710 If you explicitly set release to
1711 !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN, it will be replaced with a
1712 value generated as described in automatic versioning.
1713
1714 If you explicitly set release to !RELEASE_FROM_DATE_RESPIN, it will
1715 be replaced with a value generated as described in automatic ver‐
1716 sioning.
1717
1718 If you explicitly set version to !VERSION_FROM_VERSION, it will be
1719 replaced with a value generated as described in automatic version‐
1720 ing.
1721
1722 Please don’t set install_tree. This gets automatically set by pungi
1723 based on current variant. You can use install_tree_from key to use
1724 install tree from another variant.
1725
1726 Both the install tree and repos can use one of following formats:
1727
1728 · URL to the location
1729
1730 · name of variant in the current compose
1731
1732 · absolute path on local filesystem (which will be translated
1733 using configured mappings or used unchanged, in which case you
1734 have to ensure the koji builders can access it)
1735
1736 You can set either a single format, or a list of formats. For avail‐
1737 able values see help output for koji image-build command.
1738
1739 If ksurl ends with #HEAD, Pungi will figure out the SHA1 hash of
1740 current HEAD and use that instead.
1741
1742 Setting scratch to True will run the koji tasks as scratch builds.
1743
1744 Example
1745 image_build = {
1746 '^Server$': [
1747 {
1748 'image-build': {
1749 'format': ['docker', 'qcow2']
1750 'name': 'fedora-qcow-and-docker-base',
1751 'target': 'koji-target-name',
1752 'ksversion': 'F23', # value from pykickstart
1753 'version': '23',
1754 # correct SHA1 hash will be put into the URL below automatically
1755 'ksurl': 'https://git.fedorahosted.org/git/spin-kickstarts.git?somedirectoryifany#HEAD',
1756 'kickstart': "fedora-docker-base.ks",
1757 'repo': ["http://someextrarepos.org/repo", "ftp://rekcod.oi/repo"],
1758 'distro': 'Fedora-20',
1759 'disk_size': 3,
1760
1761 # this is set automatically by pungi to os_dir for given variant
1762 # 'install_tree': 'http://somepath',
1763 },
1764 'factory-parameters': {
1765 'docker_cmd': "[ '/bin/bash' ]",
1766 'docker_env': "[ 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' ]",
1767 'docker_labels': "{'Name': 'fedora-docker-base', 'License': u'GPLv2', 'RUN': 'docker run -it --rm ${OPT1} --privileged -v \`pwd\`:/atomicapp -v /run:/run -v /:/host --net=host --name ${NAME} -e NAME=${NAME} -e IMAGE=${IMAGE} ${IMAGE} -v ${OPT2} run ${OPT3} /atomicapp', 'Vendor': 'Fedora Project', 'Version': '23', 'Architecture': 'x86_64' }",
1768 }
1769 },
1770 {
1771 'image-build': {
1772 'format': ['docker', 'qcow2']
1773 'name': 'fedora-qcow-and-docker-base',
1774 'target': 'koji-target-name',
1775 'ksversion': 'F23', # value from pykickstart
1776 'version': '23',
1777 # correct SHA1 hash will be put into the URL below automatically
1778 'ksurl': 'https://git.fedorahosted.org/git/spin-kickstarts.git?somedirectoryifany#HEAD',
1779 'kickstart': "fedora-docker-base.ks",
1780 'repo': ["http://someextrarepos.org/repo", "ftp://rekcod.oi/repo"],
1781 'distro': 'Fedora-20',
1782 'disk_size': 3,
1783
1784 # this is set automatically by pungi to os_dir for given variant
1785 # 'install_tree': 'http://somepath',
1786 }
1787 },
1788 {
1789 'image-build': {
1790 'format': 'qcow2',
1791 'name': 'fedora-qcow-base',
1792 'target': 'koji-target-name',
1793 'ksversion': 'F23', # value from pykickstart
1794 'version': '23',
1795 'ksurl': 'https://git.fedorahosted.org/git/spin-kickstarts.git?somedirectoryifany#HEAD',
1796 'kickstart': "fedora-docker-base.ks",
1797 'distro': 'Fedora-23',
1798
1799 # only build this type of image on x86_64
1800 'arches': ['x86_64']
1801
1802 # Use install tree and repo from Everything variant.
1803 'install_tree_from': 'Everything',
1804 'repo': ['Everything'],
1805
1806 # Set release automatically.
1807 'release': '!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN',
1808 }
1809 }
1810 ]
1811 }
1812
1813 OSBuild Composer for building images
1814 osbuild
1815 (dict) – configuration for building images in OSBuild Composer
1816 service fronted by a Koji plugin. Pungi will trigger a Koji task
1817 delegating to the OSBuild Composer, which will build the image,
1818 import it to Koji via content generators.
1819
1820 Format: {variant_uid_regex: [{...}]}.
1821
1822 Required keys in the configuration dict:
1823
1824 · name – name of the Koji package
1825
1826 · distro – image for which distribution should be build TODO
1827 examples
1828
1829 · image_type – a list of image types to build (e.g. qcow2)
1830
1831 Optional keys:
1832
1833 · target – which build target to use for the task. Either this
1834 option or the global osbuild_target is required.
1835
1836 · version – version for the final build (as a string). This
1837 option is required if the global osbuild_version is not speci‐
1838 fied.
1839
1840 · release – release part of the final NVR. If neither this
1841 option nor the global osbuild_release is set, Koji will auto‐
1842 matically generate a value.
1843
1844 · repo – a list of repository URLs from which to consume pack‐
1845 ages for building the image. By default only the variant
1846 repository is used.
1847
1848 · arches – list of architectures for which to build the image.
1849 By default, the variant arches are used. This option can only
1850 restrict it, not add a new one.
1851
1852 NOTE:
1853 There is initial support for having this task as failable without
1854 aborting the whole compose. This can be enabled by setting "fail‐
1855 able": ["*"] in the config for the image. It is an on/off switch
1856 without granularity per arch.
1857
1858 Image container
1859 This phase supports building containers in OSBS that embed an image
1860 created in the same compose. This can be useful for delivering the
1861 image to users running in containerized environments.
1862
1863 Pungi will start a buildContainer task in Koji with configured source
1864 repository. The Dockerfile can expect that a repo file will be injected
1865 into the container that defines a repo named image-to-include, and its
1866 baseurl will point to the image to include. It is possible to extract
1867 the URL with a command like dnf config-manager --dump image-to-include
1868 | awk '/baseurl =/{print $3}'`
1869
1870 image_container
1871 (dict) – configuration for building containers embedding an
1872 image.
1873
1874 Format: {variant_uid_regex: [{...}]}.
1875
1876 The inner object will define a single container. These keys are
1877 required:
1878
1879 · url, target, git_branch. See OSBS section for definition of
1880 these.
1881
1882 · image_spec – (object) A string mapping of filters used to
1883 select the image to embed. All images listed in metadata for
1884 the variant will be processed. The keys of this filter are
1885 used to select metadata fields for the image, and values are
1886 regular expression that need to match the metadata value.
1887
1888 The filter should match exactly one image.
1889
1890 Example config
1891 image_container = {
1892 "^Server$": [{
1893 "url": "git://example.com/dockerfiles.git?#HEAD",
1894 "target": "f24-container-candidate",
1895 "git_branch": "f24",
1896 "image_spec": {
1897 "format": "qcow2",
1898 "arch": "x86_64",
1899 "path": ".*/guest-image-.*$",
1900 }
1901 }]
1902 }
1903
1904 OSTree Settings
1905 The ostree phase of Pungi can create and update ostree repositories.
1906 This is done by running rpm-ostree compose in a Koji runroot environ‐
1907 ment. The ostree repository itself is not part of the compose and
1908 should be located in another directory. Any new packages in the compose
1909 will be added to the repository with a new commit.
1910
1911 ostree (dict) – a mapping of configuration for each. The format should
1912 be {variant_uid_regex: config_dict}. It is possible to use a
1913 list of configuration dicts as well.
1914
1915 The configuration dict for each variant arch pair must have
1916 these keys:
1917
1918 · treefile – (str) Filename of configuration for rpm-ostree.
1919
1920 · config_url – (str) URL for Git repository with the treefile.
1921
1922 · repo – (str|dict|[str|dict]) repos specified by URL or variant
1923 UID or a dict of repo options, baseurl is required in the
1924 dict.
1925
1926 · ostree_repo – (str) Where to put the ostree repository
1927
1928 These keys are optional:
1929
1930 · keep_original_sources – (bool) Keep the existing source repos
1931 in the tree config file. If not enabled, all the original
1932 source repos will be removed from the tree config file.
1933
1934 · config_branch – (str) Git branch of the repo to use. Defaults
1935 to master.
1936
1937 · arches – ([str]) List of architectures for which to update
1938 ostree. There will be one task per architecture. By default
1939 all architectures in the variant are used.
1940
1941 · failable – ([str]) List of architectures for which this deliv‐
1942 erable is not release blocking.
1943
1944 · update_summary – (bool) Update summary metadata after tree
1945 composing. Defaults to False.
1946
1947 · force_new_commit – (bool) Do not use rpm-ostree’s built-in
1948 change detection. Defaults to False.
1949
1950 · version – (str) Version string to be added as versioning meta‐
1951 data. If this option is set to !OSTREE_VER‐
1952 SION_FROM_LABEL_DATE_TYPE_RESPIN, a value will be generated
1953 automatically as $VERSION.$RELEASE. If this option is set to
1954 !VERSION_FROM_VERSION_DATE_RESPIN, a value will be generated
1955 automatically as $VERSION.$DATE.$RESPIN. See how those values
1956 are created.
1957
1958 · tag_ref – (bool, default True) If set to False, a git refer‐
1959 ence will not be created.
1960
1961 · ostree_ref – (str) To override value ref from treefile.
1962
1963 Example config
1964 ostree = {
1965 "^Atomic$": {
1966 "treefile": "fedora-atomic-docker-host.json",
1967 "config_url": "https://git.fedorahosted.org/git/fedora-atomic.git",
1968 "repo": [
1969 "Server",
1970 "http://example.com/repo/x86_64/os",
1971 {"baseurl": "Everything"},
1972 {"baseurl": "http://example.com/linux/repo", "exclude": "systemd-container"},
1973 ],
1974 "keep_original_sources": True,
1975 "ostree_repo": "/mnt/koji/compose/atomic/Rawhide/",
1976 "update_summary": True,
1977 # Automatically generate a reasonable version
1978 "version": "!OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN",
1979 # Only run this for x86_64 even if Atomic has more arches
1980 "arches": ["x86_64"],
1981 }
1982 }
1983
1984 ostree_use_koji_plugin = False
1985 (bool) – When set to True, the Koji pungi_ostree task will be
1986 used to execute rpm-ostree instead of runroot. Use only if the
1987 Koji instance has the pungi_ostree plugin installed.
1988
1989 Ostree Installer Settings
1990 The ostree_installer phase of Pungi can produce installer image
1991 bundling an OSTree repository. This always runs in Koji as a runroot
1992 task.
1993
1994 ostree_installer
1995 (dict) – a variant/arch mapping of configuration. The format
1996 should be [(variant_uid_regex, {arch|*: config_dict})].
1997
1998 The configuration dict for each variant arch pair must have this
1999 key:
2000
2001 These keys are optional:
2002
2003 · repo – (str|[str]) repos specified by URL or variant UID
2004
2005 · release – (str) Release value to set for the installer image.
2006 Set to !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN to generate the
2007 value automatically.
2008
2009 · failable – ([str]) List of architectures for which this deliv‐
2010 erable is not release blocking.
2011
2012 These optional keys are passed to lorax to customize the build.
2013
2014 · installpkgs – ([str])
2015
2016 · add_template – ([str])
2017
2018 · add_arch_template – ([str])
2019
2020 · add_template_var – ([str])
2021
2022 · add_arch_template_var – ([str])
2023
2024 · rootfs_size – ([str])
2025
2026 · template_repo – (str) Git repository with extra templates.
2027
2028 · template_branch – (str) Branch to use from template_repo.
2029
2030 The templates can either be absolute paths, in which case they
2031 will be used as configured; or they can be relative paths, in
2032 which case template_repo needs to point to a Git repository from
2033 which to take the templates.
2034
2035 If the templates need to run with additional dependencies, that
2036 can be configured with the optional key:
2037
2038 · extra_runroot_pkgs – ([str])
2039
2040 ostree_installer_overwrite = False
2041 (bool) – by default if a variant including OSTree installer also
2042 creates regular installer images in buildinstall phase, there
2043 will be conflicts (as the files are put in the same place) and
2044 Pungi will report an error and fail the compose.
2045
2046 With this option it is possible to opt-in for the overwriting.
2047 The traditional boot.iso will be in the iso/ subdirectory.
2048
2049 ostree_installer_use_koji_plugin = False
2050 (bool) – When set to True, the Koji pungi_buildinstall task will
2051 be used to execute Lorax instead of runroot. Use only if the
2052 Koji instance has the pungi_buildinstall plugin installed.
2053
2054 Example config
2055 ostree_installer = [
2056 ("^Atomic$", {
2057 "x86_64": {
2058 "repo": [
2059 "Everything",
2060 "https://example.com/extra-repo1.repo",
2061 "https://example.com/extra-repo2.repo",
2062 ],
2063 "release": "!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN",
2064 "installpkgs": ["fedora-productimg-atomic"],
2065 "add_template": ["atomic-installer/lorax-configure-repo.tmpl"],
2066 "add_template_var": [
2067 "ostree_osname=fedora-atomic",
2068 "ostree_ref=fedora-atomic/Rawhide/x86_64/docker-host",
2069 ],
2070 "add_arch_template": ["atomic-installer/lorax-embed-repo.tmpl"],
2071 "add_arch_template_var": [
2072 "ostree_repo=https://kojipkgs.fedoraproject.org/compose/atomic/Rawhide/",
2073 "ostree_osname=fedora-atomic",
2074 "ostree_ref=fedora-atomic/Rawhide/x86_64/docker-host",
2075 ]
2076 'template_repo': 'https://git.fedorahosted.org/git/spin-kickstarts.git',
2077 'template_branch': 'f24',
2078 }
2079 })
2080 ]
2081
2082 OSBS Settings
2083 Pungi can build container images in OSBS. The build is initiated
2084 through Koji container-build plugin. The base image will be using RPMs
2085 from the current compose and a Dockerfile from specified Git reposi‐
2086 tory.
2087
2088 Please note that the image is uploaded to a registry and not exported
2089 into compose directory. There will be a metadata file in compose/meta‐
2090 data/osbs.json with details about the built images (assuming they are
2091 not scratch builds).
2092
2093 osbs (dict) – a mapping from variant regexes to configuration blocks.
2094 The format should be {variant_uid_regex: [config_dict]}.
2095
2096 The configuration for each image must have at least these keys:
2097
2098 · url – (str) URL pointing to a Git repository with Dockerfile.
2099 Please see Git URLs section for more details.
2100
2101 · target – (str) A Koji target to build the image for.
2102
2103 · git_branch – (str) A branch in SCM for the Dockerfile. This is
2104 required by OSBS to avoid race conditions when multiple builds
2105 from the same repo are submitted at the same time. Please note
2106 that url should contain the branch or tag name as well, so
2107 that it can be resolved to a particular commit hash.
2108
2109 Optionally you can specify failable. If it has a truthy value,
2110 failure to create the image will not abort the whole compose.
2111
2112 The configuration will pass other attributes directly to the
2113 Koji task. This includes scratch and priority.
2114
2115 A value for yum_repourls will be created automatically and point
2116 at a repository in the current compose. You can add extra repos‐
2117 itories with repo key having a list of urls pointing to .repo
2118 files or just variant uid, Pungi will create the .repo file for
2119 that variant. gpgkey can be specified to enable gpgcheck in repo
2120 files for variants.
2121
2122 osbs_registries
2123 (dict) – It is possible to configure extra information about
2124 where to push the image (unless it is a scratch build). For each
2125 finished build, Pungi will try to match NVR against a key in
2126 this mapping (using shell-style globbing) and take the corre‐
2127 sponding value and collect them across all built images. The
2128 data will be saved into logs/global/osbs-registries.json as a
2129 mapping from Koji NVR to the registry data. The same data is
2130 also sent to the message bus on osbs-request-push topic once the
2131 compose finishes successfully. Handling the message and perform‐
2132 ing the actual push is outside of scope for Pungi.
2133
2134 Example config
2135 osbs = {
2136 "^Server$": {
2137 # required
2138 "url": "git://example.com/dockerfiles.git?#HEAD",
2139 "target": "f24-docker-candidate",
2140 "git_branch": "f24-docker",
2141
2142 # optional
2143 "repo": ["Everything", "https://example.com/extra-repo.repo"],
2144 # This will result in three repo urls being passed to the task.
2145 # They will be in this order: Server, Everything, example.com/
2146 "gpgkey": 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release',
2147 }
2148 }
2149
2150 Extra ISOs
2151 Create an ISO image that contains packages from multiple variants. Such
2152 ISO always belongs to one variant, and will be stored in ISO directory
2153 of that variant.
2154
2155 The ISO will be bootable if buildinstall phase runs for the parent
2156 variant. It will reuse boot configuration from that variant.
2157
2158 extra_isos
2159 (dict) – a mapping from variant UID regex to a list of configu‐
2160 ration blocks.
2161
2162 · include_variants – (list) list of variant UIDs from which con‐
2163 tent should be added to the ISO; the variant of this image is
2164 added automatically.
2165
2166 Rest of configuration keys is optional.
2167
2168 · filename – (str) template for naming the image. In addition to
2169 the regular placeholders filename is available with the name
2170 generated using image_name_format option.
2171
2172 · volid – (str) template for generating volume ID. Again volid
2173 placeholder can be used similarly as for file name. This can
2174 also be a list of templates that will be tried sequentially
2175 until one generates a volume ID that fits into 32 character
2176 limit.
2177
2178 · extra_files – (list) a list of scm_dict objects. These files
2179 will be put in the top level directory of the image.
2180
2181 · arches – (list) a list of architectures for which to build
2182 this image. By default all arches from the variant will be
2183 used. This option can be used to limit them.
2184
2185 · failable_arches – (list) a list of architectures for which the
2186 image can fail to be generated and not fail the entire com‐
2187 pose.
2188
2189 · skip_src – (bool) allows to disable creating an image with
2190 source packages.
2191
2192 · inherit_extra_files – (bool) by default extra files in vari‐
2193 ants are ignored. If you want to include them in the ISO, set
2194 this option to True.
2195
2196 · max_size – (int) expected maximum size in bytes. If the final
2197 image is larger, a warning will be issued.
2198
2199 Example config
2200 extra_isos = {
2201 'Server': [{
2202 # Will generate foo-DP-1.0-20180510.t.43-Server-x86_64-dvd1.iso
2203 'filename': 'foo-{filename}',
2204 'volid': 'foo-{arch}',
2205
2206 'extra_files': [{
2207 'scm': 'git',
2208 'repo': 'https://pagure.io/pungi.git',
2209 'file': 'setup.py'
2210 }],
2211
2212 'include_variants': ['Client']
2213 }]
2214 }
2215 # This should create image with the following layout:
2216 # .
2217 # ├── Client
2218 # │ ├── Packages
2219 # │ │ ├── a
2220 # │ │ └── b
2221 # │ └── repodata
2222 # ├── Server
2223 # │ ├── Packages
2224 # │ │ ├── a
2225 # │ │ └── b
2226 # │ └── repodata
2227 # └── setup.py
2228
2229 Media Checksums Settings
2230 media_checksums
2231 (list) – list of checksum types to compute, allowed values are
2232 anything supported by Python’s hashlib module (see documentation
2233 for details).
2234
2235 media_checksum_one_file
2236 (bool) – when True, only one CHECKSUM file will be created per
2237 directory; this option requires media_checksums to only specify
2238 one type
2239
2240 media_checksum_base_filename
2241 (str) – when not set, all checksums will be save to a file named
2242 either CHECKSUM or based on the digest type; this option allows
2243 adding any prefix to that name
2244
2245 It is possible to use format strings that will be replace by
2246 actual values. The allowed keys are:
2247
2248 · arch
2249
2250 · compose_id
2251
2252 · date
2253
2254 · label
2255
2256 · label_major_version
2257
2258 · release_short
2259
2260 · respin
2261
2262 · type
2263
2264 · type_suffix
2265
2266 · version
2267
2268 · dirname (only if media_checksum_one_file is enabled)
2269
2270 For example, for Fedora the prefix should be
2271 %(release_short)s-%(variant)s-%(version)s-%(date)s%(type_suf‐
2272 fix)s.%(respin)s.
2273
2274 Translate Paths Settings
2275 translate_paths
2276 (list) – list of paths to translate; format: [(path, trans‐
2277 lated_path)]
2278
2279 NOTE:
2280 This feature becomes useful when you need to transform compose loca‐
2281 tion into e.g. a HTTP repo which is can be passed to koji
2282 image-build. The path part is normalized via os.path.normpath().
2283
2284 Example config
2285 translate_paths = [
2286 ("/mnt/a", "http://b/dir"),
2287 ]
2288
2289 Example usage
2290 >>> from pungi.util import translate_paths
2291 >>> print translate_paths(compose_object_with_mapping, "/mnt/a/c/somefile")
2292 http://b/dir/c/somefile
2293
2294 Miscellaneous Settings
2295 paths_module
2296 (str) – Name of Python module implementing the same interface as
2297 pungi.paths. This module can be used to override where things
2298 are placed.
2299
2300 link_type = hardlink-or-copy
2301 (str) – Method of putting packages into compose directory.
2302
2303 Available options:
2304
2305 · hardlink-or-copy
2306
2307 · hardlink
2308
2309 · copy
2310
2311 · symlink
2312
2313 · abspath-symlink
2314
2315 skip_phases
2316 (list) – List of phase names that should be skipped. The same
2317 functionality is available via a command line option.
2318
2319 release_discinfo_description
2320 (str) – Override description in .discinfo files. The value is a
2321 format string accepting %(variant_name)s and %(arch)s placehold‐
2322 ers.
2323
2324 symlink_isos_to
2325 (str) – If set, the ISO files from buildinstall, createiso and
2326 live_images phases will be put into this destination, and a sym‐
2327 link pointing to this location will be created in actual compose
2328 directory.
2329
2330 dogpile_cache_backend
2331 (str) – If set, Pungi will use the configured Dogpile cache
2332 backend to cache various data between multiple Pungi calls. This
2333 can make Pungi faster in case more similar composes are running
2334 regularly in short time.
2335
2336 For list of available backends, please see the
2337 https://dogpilecache.readthedocs.io documentation.
2338
2339 Most typical configuration uses the dogpile.cache.dbm backend.
2340
2341 dogpile_cache_arguments
2342 (dict) – Arguments to be used when creating the Dogpile cache
2343 backend. See the particular backend’s configuration for the
2344 list of possible key/value pairs.
2345
2346 For the dogpile.cache.dbm backend, the value can be for example
2347 following:
2348
2349 {
2350 "filename": "/tmp/pungi_cache_file.dbm"
2351 }
2352
2353 dogpile_cache_expiration_time
2354 (int) – Defines the default expiration time in seconds of data
2355 stored in the Dogpile cache. Defaults to 3600 seconds.
2356
2358 Actual Pungi configuration files can get very large. This pages brings
2359 two examples of (almost) full configuration for two different composes.
2360
2361 Fedora Rawhide compose
2362 This is a shortened configuration for Fedora Radhide compose as of
2363 2019-10-14.
2364
2365 release_name = 'Fedora'
2366 release_short = 'Fedora'
2367 release_version = 'Rawhide'
2368 release_is_layered = False
2369
2370 bootable = True
2371 comps_file = {
2372 'scm': 'git',
2373 'repo': 'https://pagure.io/fedora-comps.git',
2374 'branch': 'master',
2375 'file': 'comps-rawhide.xml',
2376 # Merge translations by running make. This command will generate the file.
2377 'command': 'make comps-rawhide.xml'
2378 }
2379 module_defaults_dir = {
2380 'scm': 'git',
2381 'repo': 'https://pagure.io/releng/fedora-module-defaults.git',
2382 'branch': 'master',
2383 'dir': '.'
2384 }
2385
2386 variants_file='variants-fedora.xml'
2387 sigkeys = ['12C944D0']
2388
2389 # Put packages into subdirectories hashed by their initial letter.
2390 hashed_directories = True
2391
2392 # There is a special profile for use with compose. It makes Pungi
2393 # authenticate automatically as rel-eng user.
2394 koji_profile = 'compose_koji'
2395
2396 # RUNROOT settings
2397 runroot = True
2398 runroot_channel = 'compose'
2399 runroot_tag = 'f32-build'
2400
2401 # PKGSET
2402 pkgset_source = 'koji'
2403 pkgset_koji_tag = 'f32'
2404 pkgset_koji_inherit = False
2405
2406 filter_system_release_packages = False
2407
2408 # GATHER
2409 gather_method = {
2410 '^.*': { # For all variants
2411 'comps': 'deps', # resolve dependencies for packages from comps file
2412 'module': 'nodeps', # but not for packages from modules
2413 }
2414 }
2415 gather_backend = 'dnf'
2416 gather_profiler = True
2417 check_deps = False
2418 greedy_method = 'build'
2419
2420 repoclosure_backend = 'dnf'
2421
2422 # CREATEREPO
2423 createrepo_deltas = False
2424 createrepo_database = True
2425 createrepo_use_xz = True
2426 createrepo_extra_args = ['--zck', '--zck-dict-dir=/usr/share/fedora-repo-zdicts/rawhide']
2427
2428 # CHECKSUMS
2429 media_checksums = ['sha256']
2430 media_checksum_one_file = True
2431 media_checksum_base_filename = '%(release_short)s-%(variant)s-%(version)s-%(arch)s-%(date)s%(type_suffix)s.%(respin)s'
2432
2433 # CREATEISO
2434 iso_hfs_ppc64le_compatible = False
2435 create_jigdo = False
2436
2437 # BUILDINSTALL
2438 buildinstall_method = 'lorax'
2439 buildinstall_skip = [
2440 # No installer for Modular variant
2441 ('^Modular$', {'*': True}),
2442 # No 32 bit installer for Everything.
2443 ('^Everything$', {'i386': True}),
2444 ]
2445
2446 # Enables macboot on x86_64 for all variants and disables upgrade image building
2447 # everywhere.
2448 lorax_options = [
2449 ('^.*$', {
2450 'x86_64': {
2451 'nomacboot': False
2452 },
2453 'ppc64le': {
2454 # Use 3GB image size for ppc64le.
2455 'rootfs_size': 3
2456 },
2457 '*': {
2458 'noupgrade': True
2459 }
2460 })
2461 ]
2462
2463 additional_packages = [
2464 ('^(Server|Everything)$', {
2465 '*': [
2466 # Add all architectures of dracut package.
2467 'dracut.*',
2468 # All all packages matching this pattern
2469 'autocorr-*',
2470 ],
2471 }),
2472
2473 ('^Everything$', {
2474 # Everything should include all packages from the tag. This only
2475 # applies to the native arch. Multilib will still be pulled in
2476 # according to multilib rules.
2477 '*': ['*'],
2478 }),
2479 ]
2480
2481 filter_packages = [
2482 ("^.*$", {"*": ["glibc32", "libgcc32"]}),
2483 ('(Server)$', {
2484 '*': [
2485 'kernel*debug*',
2486 'kernel-kdump*',
2487 ]
2488 }),
2489 ]
2490
2491 multilib = [
2492 ('^Everything$', {
2493 'x86_64': ['devel', 'runtime'],
2494 })
2495 ]
2496
2497 # These packages should never be multilib on any arch.
2498 multilib_blacklist = {
2499 '*': [
2500 'kernel', 'kernel-PAE*', 'kernel*debug*', 'java-*', 'php*', 'mod_*', 'ghc-*'
2501 ],
2502 }
2503
2504 # These should be multilib even if they don't match the rules defined above.
2505 multilib_whitelist = {
2506 '*': ['wine', '*-static'],
2507 }
2508
2509 createiso_skip = [
2510 # Keep binary ISOs for Server, but not source ones.
2511 ('^Server$', {'src': True}),
2512
2513 # Remove all other ISOs.
2514 ('^Everything$', {'*': True, 'src': True}),
2515 ('^Modular$', {'*': True, 'src': True}),
2516 ]
2517
2518 # Image name respecting Fedora's image naming policy
2519 image_name_format = '%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s-%(date)s%(type_suffix)s.%(respin)s.iso'
2520 # Use the same format for volume id
2521 image_volid_formats = [
2522 '%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s'
2523 ]
2524 # Used by Pungi to replace 'Cloud' with 'C' (etc.) in ISO volume IDs.
2525 # There is a hard 32-character limit on ISO volume IDs, so we use
2526 # these to try and produce short enough but legible IDs. Note this is
2527 # duplicated in Koji for live images, as livemedia-creator does not
2528 # allow Pungi to tell it what volume ID to use. Note:
2529 # https://fedoraproject.org/wiki/User:Adamwill/Draft_fedora_image_naming_policy
2530 volume_id_substitutions = {
2531 'Beta': 'B',
2532 'Rawhide': 'rawh',
2533 'Silverblue': 'SB',
2534 'Cinnamon': 'Cinn',
2535 'Cloud': 'C',
2536 'Design_suite': 'Dsgn',
2537 'Electronic_Lab': 'Elec',
2538 'Everything': 'E',
2539 'Scientific_KDE': 'SciK',
2540 'Security': 'Sec',
2541 'Server': 'S',
2542 'Workstation': 'WS',
2543 }
2544
2545 disc_types = {
2546 'boot': 'netinst',
2547 'live': 'Live',
2548 }
2549
2550 translate_paths = [
2551 ('/mnt/koji/compose/', 'https://kojipkgs.fedoraproject.org/compose/'),
2552 ]
2553
2554 # These will be inherited by live_media, live_images and image_build
2555 global_ksurl = 'git+https://pagure.io/fedora-kickstarts.git?#HEAD'
2556 global_release = '!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN'
2557 global_version = 'Rawhide'
2558 # live_images ignores this in favor of live_target
2559 global_target = 'f32'
2560
2561 image_build = {
2562 '^Container$': [
2563 {
2564 'image-build': {
2565 'format': [('docker', 'tar.xz')],
2566 'name': 'Fedora-Container-Base',
2567 'kickstart': 'fedora-container-base.ks',
2568 'distro': 'Fedora-22',
2569 'disk_size': 5,
2570 'arches': ['armhfp', 'aarch64', 'ppc64le', 's390x', 'x86_64'],
2571 'repo': 'Everything',
2572 'install_tree_from': 'Everything',
2573 'subvariant': 'Container_Base',
2574 'failable': ['*'],
2575 },
2576 'factory-parameters': {
2577 'dockerversion': "1.10.1",
2578 'docker_cmd': '[ "/bin/bash" ]',
2579 'docker_env': '[ "DISTTAG=f32container", "FGC=f32", "container=oci" ]',
2580 'docker_label': '{ "name": "fedora", "license": "MIT", "vendor": "Fedora Project", "version": "32"}',
2581 },
2582 },
2583 ],
2584 }
2585
2586 live_media = {
2587 '^Workstation$': [
2588 {
2589 'name': 'Fedora-Workstation-Live',
2590 'kickstart': 'fedora-live-workstation.ks',
2591 # Variants.xml also contains aarch64 and armhfp, but there
2592 # should be no live media for those arches.
2593 'arches': ['x86_64', 'ppc64le'],
2594 'failable': ['ppc64le'],
2595 # Take packages and install tree from Everything repo.
2596 'repo': 'Everything',
2597 'install_tree_from': 'Everything',
2598 }
2599 ],
2600 '^Spins': [
2601 # There are multiple media for Spins variant. They use subvariant
2602 # field so that they can be identified in the metadata.
2603 {
2604 'name': 'Fedora-KDE-Live',
2605 'kickstart': 'fedora-live-kde.ks',
2606 'arches': ['x86_64'],
2607 'repo': 'Everything',
2608 'install_tree_from': 'Everything',
2609 'subvariant': 'KDE'
2610
2611 },
2612 {
2613 'name': 'Fedora-Xfce-Live',
2614 'kickstart': 'fedora-live-xfce.ks',
2615 'arches': ['x86_64'],
2616 'failable': ['*'],
2617 'repo': 'Everything',
2618 'install_tree_from': 'Everything',
2619 'subvariant': 'Xfce'
2620 },
2621 ],
2622 }
2623
2624 failable_deliverables = [
2625 # Installer and ISOs for server failing do not abort the compose.
2626 ('^Server$', {
2627 '*': ['buildinstall', 'iso'],
2628 }),
2629 ('^.*$', {
2630 # Buildinstall is not blocking
2631 'src': ['buildinstall'],
2632 # Nothing on i386, ppc64le blocks the compose
2633 'i386': ['buildinstall', 'iso'],
2634 'ppc64le': ['buildinstall', 'iso'],
2635 's390x': ['buildinstall', 'iso'],
2636 })
2637 ]
2638
2639 live_target = 'f32'
2640 live_images_no_rename = True
2641 live_images = [
2642 ('^Workstation$', {
2643 'armhfp': {
2644 'kickstart': 'fedora-arm-workstation.ks',
2645 'name': 'Fedora-Workstation-armhfp',
2646 # Again workstation takes packages from Everything.
2647 'repo': 'Everything',
2648 'type': 'appliance',
2649 'failable': True,
2650 }
2651 }),
2652 ('^Server$', {
2653 # But Server has its own repo.
2654 'armhfp': {
2655 'kickstart': 'fedora-arm-server.ks',
2656 'name': 'Fedora-Server-armhfp',
2657 'type': 'appliance',
2658 'failable': True,
2659 }
2660 }),
2661 ]
2662
2663 ostree = {
2664 "^Silverblue$": {
2665 "version": "!OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN",
2666 # To get config, clone master branch from this repo and take
2667 # treefile from there.
2668 "treefile": "fedora-silverblue.yaml",
2669 "config_url": "https://pagure.io/workstation-ostree-config.git",
2670 "config_branch": "master",
2671 # Consume packages from Everything
2672 "repo": "Everything",
2673 # Don't create a reference in the ostree repo (signing automation does that).
2674 "tag_ref": False,
2675 # Don't use change detection in ostree.
2676 "force_new_commit": True,
2677 # This is the location for the repo where new commit will be
2678 # created. Note that this is outside of the compose dir.
2679 "ostree_repo": "/mnt/koji/compose/ostree/repo/",
2680 "ostree_ref": "fedora/rawhide/${basearch}/silverblue",
2681 "arches": ["x86_64", "ppc64le", "aarch64"],
2682 "failable": ['*'],
2683 }
2684 }
2685
2686 ostree_installer = [
2687 ("^Silverblue$", {
2688 "x86_64": {
2689 "repo": "Everything",
2690 "release": None,
2691 "rootfs_size": "8",
2692 # Take templates from this repository.
2693 'template_repo': 'https://pagure.io/fedora-lorax-templates.git',
2694 'template_branch': 'master',
2695 # Use following templates.
2696 "add_template": ["ostree-based-installer/lorax-configure-repo.tmpl",
2697 "ostree-based-installer/lorax-embed-repo.tmpl",
2698 "ostree-based-installer/lorax-embed-flatpaks.tmpl"],
2699 # And add these variables for the templates.
2700 "add_template_var": [
2701 "ostree_install_repo=https://kojipkgs.fedoraproject.org/compose/ostree/repo/",
2702 "ostree_update_repo=https://ostree.fedoraproject.org",
2703 "ostree_osname=fedora",
2704 "ostree_oskey=fedora-32-primary",
2705 "ostree_contenturl=mirrorlist=https://ostree.fedoraproject.org/mirrorlist",
2706 "ostree_install_ref=fedora/rawhide/x86_64/silverblue",
2707 "ostree_update_ref=fedora/rawhide/x86_64/silverblue",
2708 "flatpak_remote_name=fedora",
2709 "flatpak_remote_url=oci+https://registry.fedoraproject.org",
2710 "flatpak_remote_refs=runtime/org.fedoraproject.Platform/x86_64/f30 app/org.gnome.Baobab/x86_64/stable",
2711 ],
2712 'failable': ['*'],
2713 },
2714 })
2715 ]
2716
2717 RCM Tools compose
2718 This is a small compose used to deliver packages to Red Hat internal
2719 users. The configuration is split into two files.
2720
2721 # rcmtools-common.conf
2722
2723 release_name = "RCM Tools"
2724 release_short = "RCMTOOLS"
2725 release_version = "2.0"
2726 release_type = "updates"
2727 release_is_layered = True
2728 createrepo_c = True
2729 createrepo_checksum = "sha256"
2730
2731 # PKGSET
2732 pkgset_source = "koji"
2733 koji_profile = "brew"
2734 pkgset_koji_inherit = True
2735
2736
2737 # GENERAL SETTINGS
2738 bootable = False
2739 comps_file = "rcmtools-comps.xml"
2740 variants_file = "rcmtools-variants.xml"
2741 sigkeys = ["3A3A33A3"]
2742
2743
2744 # RUNROOT settings
2745 runroot = False
2746
2747
2748 # GATHER
2749 gather_method = "deps"
2750 check_deps = True
2751
2752 additional_packages = [
2753 ('.*', {
2754 '*': ['puddle', 'rcm-nexus'],
2755 }
2756 ),
2757 ]
2758
2759 # Set repoclosure_strictness to fatal to avoid installation dependency
2760 # issues in production composes
2761 repoclosure_strictness = [
2762 ("^.*$", {
2763 "*": "fatal"
2764 })
2765 ]
2766
2767 Configuration specific for different base products is split into sepa‐
2768 rate files.
2769
2770 # rcmtools-common.conf
2771 from rcmtools-common import *
2772
2773 # BASE PRODUCT
2774 base_product_name = "Red Hat Enterprise Linux"
2775 base_product_short = "RHEL"
2776 base_product_version = "7"
2777
2778 # PKGSET
2779 pkgset_koji_tag = "rcmtools-rhel-7-compose"
2780
2781 # remove i386 arch on rhel7
2782 tree_arches = ["aarch64", "ppc64le", "s390x", "x86_64"]
2783
2784 check_deps = False
2785
2786 # Packages in these repos are available to satisfy dependencies inside the
2787 # compose, but will not be pulled in.
2788 gather_lookaside_repos = [
2789 ("^Client|Client-optional$", {
2790 "x86_64": [
2791 "http://example.redhat.com/rhel/7/Client/x86_64/os/",
2792 "http://example.redhat.com/rhel/7/Client/x86_64/optional/os/",
2793 ],
2794 }),
2795 ("^Workstation|Workstation-optional$", {
2796 "x86_64": [
2797 "http://example.redhat.com/rhel/7/Workstation/x86_64/os/",
2798 "http://example.redhat.com/rhel/7/Workstation/x86_64/optional/os/",
2799 ],
2800 }),
2801 ("^Server|Server-optional$", {
2802 "aarch64": [
2803 "http://example.redhat.com/rhel/7/Server/aarch64/os/",
2804 "http://example.redhat.com/rhel/7/Server/aarch64/optional/os/",
2805 ],
2806 "ppc64": [
2807 "http://example.redhat.com/rhel/7/Server/ppc64/os/",
2808 "http://example.redhat.com/rhel/7/Server/ppc64/optional/os/",
2809 ],
2810 "ppc64le": [
2811 "http://example.redhat.com/rhel/7/Server/ppc64le/os/",
2812 "http://example.redhat.com/rhel/7/Server/ppc64le/optional/os/",
2813 ],
2814 "s390x": [
2815 "http://example.redhat.com/rhel/7/Server/s390x/os/",
2816 "http://example.redhat.com/rhel/7/Server/s390x/optional/os/",
2817 ],
2818 "x86_64": [
2819 "http://example.redhat.com/rhel/7/Server/x86_64/os/",
2820 "http://example.redhat.com/rhel/7/Server/x86_64/optional/os/",
2821 ],
2822 })
2823 ]
2824
2826 Multiple places in Pungi can use files from external storage. The con‐
2827 figuration is similar independently of the backend that is used,
2828 although some features may be different.
2829
2830 The so-called scm_dict is always put into configuration as a dictio‐
2831 nary, which can contain following keys.
2832
2833 · scm – indicates which SCM system is used. This is always required.
2834 Allowed values are:
2835
2836 · file – copies files from local filesystem
2837
2838 · git – copies files from a Git repository
2839
2840 · cvs – copies files from a CVS repository
2841
2842 · rpm – copies files from a package in the compose
2843
2844 · koji – downloads archives from a given build in Koji build system
2845
2846 · repo
2847
2848 · for Git and CVS backends this should be URL to the repository
2849
2850 · for RPM backend this should be a shell style glob matching package
2851 names (or a list of such globs)
2852
2853 · for file backend this should be empty
2854
2855 · for Koji backend this should be an NVR or package name
2856
2857 · branch
2858
2859 · branch name for Git and CVS backends, with master and HEAD as
2860 defaults
2861
2862 · Koji tag for koji backend if only package name is given
2863
2864 · otherwise should not be specified
2865
2866 · file – a list of files that should be exported.
2867
2868 · dir – a directory that should be exported. All its contents will be
2869 exported. This option is mutually exclusive with file.
2870
2871 · command – defines a shell command to run after Git clone to generate
2872 the needed file (for example to run make). Only supported in Git
2873 backend.
2874
2875 Koji examples
2876 There are two different ways how to configure the Koji backend.
2877
2878 {
2879 # Download all *.tar files from build my-image-1.0-1.
2880 "scm": "koji",
2881 "repo": "my-image-1.0-1",
2882 "file": "*.tar",
2883 }
2884
2885 {
2886 # Find latest build of my-image in tag my-tag and take files from
2887 # there.
2888 "scm": "koji",
2889 "repo": "my-image",
2890 "branch": "my-tag",
2891 "file": "*.tar",
2892 }
2893
2894 Using both tag name and exact NVR will result in error: the NVR would
2895 be interpreted as a package name, and would not match anything.
2896
2897 file vs. dir
2898 Exactly one of these two options has to be specified. Documentation for
2899 each configuration option should specify whether it expects a file or a
2900 directory.
2901
2902 For extra_files phase either key is valid and should be chosen depend‐
2903 ing on what the actual use case.
2904
2905 Caveats
2906 The rpm backend can only be used in phases that would extract the files
2907 after pkgset phase finished. You can’t get comps file from a package.
2908
2909 Depending on Git repository URL configuration Pungi can only export the
2910 requested content using git archive. When a command should run this is
2911 not possible and a clone is always needed.
2912
2913 When using koji backend, it is required to provide configuration for
2914 Koji profile to be used (koji_profile). It is not possible to contact
2915 multiple different Koji instances.
2916
2918 Pungi has the ability to emit notification messages about progress and
2919 general status of the compose. These can be used to e.g. send messages
2920 to fedmsg. This is implemented by actually calling a separate script.
2921
2922 The script will be called with one argument describing action that just
2923 happened. A JSON-encoded object will be passed to standard input to
2924 provide more information about the event. At the very least, the object
2925 will contain a compose_id key.
2926
2927 The script is invoked in compose directory and can read other informa‐
2928 tion there.
2929
2930 Currently these messages are sent:
2931
2932 · status-change – when composing starts, finishes or fails; a status
2933 key is provided to indicate details
2934
2935 · phase-start – on start of a phase
2936
2937 · phase-stop – when phase is finished
2938
2939 · createiso-targets – with a list of images to be created
2940
2941 · createiso-imagedone – when any single image is finished
2942
2943 · createiso-imagefail – when any single image fails to create
2944
2945 · fail-to-start – when there are incorrect CLI options or errors in
2946 configuration file; this message does not contain compose_id nor
2947 is it started in the compose directory (which does not exist yet)
2948
2949 · ostree – when a new commit is created, this message will announce
2950 its hash and the name of ref it is meant for.
2951
2952 For phase related messages phase_name key is provided as well.
2953
2954 A pungi-fedmsg-notification script is provided and understands this
2955 interface.
2956
2957 Setting it up
2958 The script should be provided as a command line argument --notifica‐
2959 tion-script.
2960
2961 --notification-script=pungi-fedmsg-notification
2962
2964 A compose created by Pungi consists of one or more variants. A variant
2965 contains a subset of the content targeted at a particular use case.
2966
2967 There are different types of variants. The type affects how packages
2968 are gathered into the variant.
2969
2970 The inputs for gathering are defined by various gather sources. Pack‐
2971 ages from all sources are collected to create a big list of package
2972 names, comps groups names and a list of packages that should be fil‐
2973 tered out.
2974
2975 NOTE:
2976 The inputs for both explicit package list and comps file are inter‐
2977 preted as RPM names, not any arbitrary provides nor source package
2978 name.
2979
2980 Next, gather_method defines how the list is processed. For nodeps, the
2981 results from source are used pretty much as is [1]. For deps method, a
2982 process will be launched to figure out what dependencies are needed and
2983 those will be pulled in.
2984
2985 [1] The lists are filtered based on what packages are available in the
2986 package set, but nothing else will be pulled in.
2987
2988 Variant types
2989 Variant
2990 is a base type that has no special behaviour.
2991
2992 Addon is built on top of a regular variant. Any packages that should
2993 go to both the addon and its parent will be removed from addon.
2994 Packages that are only in addon but pulled in because of
2995 gather_fulltree option will be moved to parent.
2996
2997 Integrated Layered Product
2998 works similarly to addon. Additionally, all packages from addons
2999 on the same parent variant are removed integrated layered prod‐
3000 ucts.
3001
3002 The main difference between an addon and integrated layered
3003 product is that integrated layered product has its own identity
3004 in the metadata (defined with product name and version).
3005
3006 NOTE:
3007 There’s also Layered Product as a term, but this is not
3008 related to variants. It’s used to describe a product that is
3009 not a standalone operating system and is instead meant to be
3010 used on some other base system.
3011
3012 Optional
3013 contains packages that complete the base variants’ package set.
3014 It always has fulltree and selfhosting enabled, so it contains
3015 build dependencies and packages which were not specifically
3016 requested for base variant.
3017
3018 Some configuration options are overridden for particular variant types.
3019
3020 Depsolving configuration
3021 ┌──────────┬──────────────┬──────────────┐
3022 │Variant │ Fulltree │ Selfhosting │
3023 ├──────────┼──────────────┼──────────────┤
3024 │base │ configurable │ configurable │
3025 ├──────────┼──────────────┼──────────────┤
3026 │addon/ILP │ enabled │ disabled │
3027 ├──────────┼──────────────┼──────────────┤
3028 │optional │ enabled │ enabled │
3029 └──────────┴──────────────┴──────────────┘
3030
3031 Profiling
3032 Profiling data on the pungi-gather tool can be enabled by setting the
3033 gather_profiler configuration option to True.
3034
3035 Modular compose
3036 A compose with gather_source set to module is called modular. The pack‐
3037 age list is determined by a list of modules.
3038
3039 The list of modules that will be put into a variant is defined in the
3040 variants.xml file. The file can contain either Name:Stream or
3041 Name:Stream:Version references. See Module Naming Policy for details.
3042 When Version is missing from the specification, Pungi will ask PDC for
3043 the latest one.
3044
3045 The module metadata in PDC contains a list of RPMs in the module as
3046 well as Koji tag from which the packages can be retrieved.
3047
3048 Restrictions
3049 · A modular compose must always use Koji as a package set source.
3050
3052 The comps file that Pungi takes as input is not really pure comps as
3053 used by tools like DNF. There are extensions used to customize how the
3054 file is processed.
3055
3056 The first step of Pungi processing is to retrieve the actual file. This
3057 can use anything that scm_support supports.
3058
3059 Pungi extensions are arch attribute on packageref, group and environ‐
3060 ment tags. The value of this attribute is a comma separated list of
3061 architectures.
3062
3063 Second step Pungi performs is creating a file for each architecture.
3064 This is done by removing all elements with incompatible arch attribute.
3065 No additional clean up is performed on this file. The resulting file is
3066 only used internally for the rest of the compose process.
3067
3068 Third and final step is to create comps file for each Variant.Arch com‐
3069 bination. This is the actual file that will be included in the com‐
3070 pose. The start file is the original input file, from which all ele‐
3071 ments with incompatible architecture are removed. Then clean up is per‐
3072 formed by removing all empty groups, removing non-existing groups from
3073 environments and categories and finally removing empty environments and
3074 categories. As a last step groups not listed in the variants file are
3075 removed.
3076
3078 Set up development environment
3079 In order to work on Pungi, you should install recent version of Fedora.
3080
3081 Python2
3082 Fedora 29 is recommended because some packages are not available in
3083 newer Fedora release, e.g. python2-libcomps.
3084
3085 Install required packages
3086
3087 $ sudo dnf install -y krb5-devel gcc make libcurl-devel python2-devel python2-createrepo_c kobo-rpmlib yum python2-libcomps python2-libselinx
3088
3089 Python3
3090 Install required packages
3091
3092 $ sudo dnf install -y krb5-devel gcc make libcurl-devel python3-devel python3-createrepo_c python3-libcomps
3093
3094 Developing
3095 Currently the development workflow for Pungi is on master branch:
3096
3097 · Make your own fork at https://pagure.io/pungi
3098
3099 · Clone your fork locally (replacing $USERNAME with your own):
3100
3101 git clone git@pagure.io:forks/$USERNAME/pungi.git
3102
3103 · cd into your local clone and add the remote upstream for rebasing:
3104
3105 cd pungi
3106 git remote add upstream git@pagure.io:pungi.git
3107
3108 NOTE:
3109 This workflow assumes that you never git commit directly to the
3110 master branch of your fork. This will make more sense when we
3111 cover rebasing below.
3112
3113 · create a topic branch based on master:
3114
3115 git branch my_topic_branch master
3116 git checkout my_topic_branch
3117
3118 · Make edits, changes, add new features, etc. and then make sure to
3119 pull from upstream master and rebase before submitting a pull
3120 request:
3121
3122 # lets just say you edited setup.py for sake of argument
3123 git checkout my_topic_branch
3124
3125 # make changes to setup.py
3126 black setup.py
3127 tox
3128 git add setup.py
3129 git commit -s -m "added awesome feature to setup.py"
3130
3131 # now we rebase
3132 git checkout master
3133 git pull --rebase upstream master
3134 git push origin master
3135 git push origin --tags
3136 git checkout my_topic_branch
3137 git rebase master
3138
3139 # resolve merge conflicts if any as a result of your development in
3140 # your topic branch
3141 git push origin my_topic_branch
3142
3143 NOTE:
3144 In order to for your commit to be merged:
3145
3146 · you must sign-off on it. Use -s option when running git commit.
3147
3148 · The code must be well formatted via black and pass flake8 check‐
3149 ing. Run tox -e black,flake8 to do the check.
3150
3151 · Create pull request in the pagure.io web UI
3152
3153 · For convenience, here is a bash shell function that can be placed in
3154 your ~/.bashrc and called such as pullupstream pungi-4-devel that
3155 will automate a large portion of the rebase steps from above:
3156
3157 pullupstream () {
3158 if [[ -z "$1" ]]; then
3159 printf "Error: must specify a branch name (e.g. - master, devel)\n"
3160 else
3161 pullup_startbranch=$(git describe --contains --all HEAD)
3162 git checkout $1
3163 git pull --rebase upstream master
3164 git push origin $1
3165 git push origin --tags
3166 git checkout ${pullup_startbranch}
3167 fi
3168 }
3169
3170 Testing
3171 You must write unit tests for any new code (except for trivial
3172 changes). Any code without sufficient test coverage may not be merged.
3173
3174 To run all existing tests, suggested method is to use tox.
3175
3176 $ sudo dnf install python3-tox -y
3177
3178 $ tox -e py3
3179 $ tox -e py27
3180
3181 Alternatively you could create a vitualenv, install deps and run tests
3182 manually if you don’t want to use tox.
3183
3184 $ sudo dnf install python3-virtualenvwrapper -y
3185 $ mkvirtualenv --system-site-packages py3
3186 $ workon py3
3187 $ pip install -r requirements.txt -r test-requirements.txt
3188 $ make test
3189
3190 # or with coverage
3191 $ make test-coverage
3192
3193 If you need to run specified tests, pytest is recommended.
3194
3195 # Activate virtualenv first
3196
3197 # Run tests
3198 $ pytest tests/test_config.py
3199 $ pytest tests/test_config.py -k test_pkgset_mismatch_repos
3200
3201 In the tests/ directory there is a shell script test_compose.sh that
3202 you can use to try and create a miniature compose on dummy data. The
3203 actual data will be created by running make test-data in project root.
3204
3205 $ sudo dnf -y install rpm-build createrepo_c isomd5sum genisoimage syslinux
3206
3207 # Activate virtualenv (the one created by tox could be used)
3208 $ source .tox/py3/bin/activate
3209
3210 $ python setup.py develop
3211 $ make test-data
3212 $ make test-compose
3213
3214 This testing compose does not actually use all phases that are avail‐
3215 able, and there is no checking that the result is correct. It only
3216 tells you whether it crashed or not.
3217
3218 NOTE:
3219 Even when it finishes successfully, it may print errors about
3220 repoclosure on Server-Gluster.x86_64 in test phase. This is not a
3221 bug.
3222
3223 Documenting
3224 You must write documentation for any new features and functional
3225 changes. Any code without sufficient documentation may not be merged.
3226
3227 To generate the documentation, run make doc in project root.
3228
3230 Test Data
3231 Tests require test data and not all of it is available in git. You
3232 must create test repositories before running the tests:
3233
3234 make test-data
3235
3236 Requirements: createrepo_c, rpmbuild
3237
3238 Unit Tests
3239 Unit tests cover functionality of Pungi python modules. You can run
3240 all of them at once:
3241
3242 make test
3243
3244 which is shortcut to:
3245
3246 python2 setup.py test
3247 python3 setup.py test
3248
3249 You can alternatively run individual tests:
3250
3251 cd tests
3252 ./<test>.py [<class>[.<test>]]
3253
3254 Functional Tests
3255 Because compose is quite complex process and not everything is covered
3256 with unit tests yet, the easiest way how to test if your changes did
3257 not break anything badly is to start a compose on a relatively small
3258 and well defined package set:
3259
3260 cd tests
3261 ./test_compose.sh
3262
3264 There may be cases where it makes sense to split a big compose into
3265 separate parts, but create a compose output that links all output into
3266 one familiar structure.
3267
3268 The pungi-orchestrate tools allows that.
3269
3270 It works with an INI-style configuration file. The [general] section
3271 contains information about identity of the main compose. Other sections
3272 define individual parts.
3273
3274 The parts are scheduled to run in parallel, with the minimal amount of
3275 serialization. The final compose directory will contain hard-links to
3276 the files.
3277
3278 General settings
3279 target Path to directory where the final compose should be created.
3280
3281 compose_type
3282 Type of compose to make.
3283
3284 release_name
3285 Name of the product for the final compose.
3286
3287 release_short
3288 Short name of the product for the final compose.
3289
3290 release_version
3291 Version of the product for the final compose.
3292
3293 release_type
3294 Type of the product for the final compose.
3295
3296 extra_args
3297 Additional arguments that will be passed to the child Pungi pro‐
3298 cesses.
3299
3300 koji_profile
3301 If specified, a current event will be retrieved from the Koji
3302 instance and used for all parts.
3303
3304 kerberos
3305 If set to yes, a kerberos ticket will be automatically created
3306 at the start. Set keytab and principal as well.
3307
3308 kerberos_keytab
3309 Path to keytab file used to create the kerberos ticket.
3310
3311 kerberos_principal
3312 Kerberos principal for the ticket
3313
3314 pre_compose_script
3315 Commands to execute before first part is started. Can contain
3316 multiple commands on separate lines.
3317
3318 post_compose_script
3319 Commands to execute after the last part finishes and final sta‐
3320 tus is updated. Can contain multiple commands on separate lines.
3321
3322 post_compose_script =
3323 compose-latest-symlink $COMPOSE_PATH
3324 custom-post-compose-script.sh
3325
3326 Multiple environment variables are defined for the scripts:
3327
3328 · COMPOSE_PATH
3329
3330 · COMPOSE_ID
3331
3332 · COMPOSE_DATE
3333
3334 · COMPOSE_TYPE
3335
3336 · COMPOSE_RESPIN
3337
3338 · COMPOSE_LABEL
3339
3340 · RELEASE_ID
3341
3342 · RELEASE_NAME
3343
3344 · RELEASE_SHORT
3345
3346 · RELEASE_VERSION
3347
3348 · RELEASE_TYPE
3349
3350 · RELEASE_IS_LAYERED – YES for layered products, empty other‐
3351 wise
3352
3353 · BASE_PRODUCT_NAME – only set for layered products
3354
3355 · BASE_PRODUCT_SHORT – only set for layered products
3356
3357 · BASE_PRODUCT_VERSION – only set for layered products
3358
3359 · BASE_PRODUCT_TYPE – only set for layered products
3360
3361 notification_script
3362 Executable name (or path to a script) that will be used to send
3363 a message once the compose is finished. In order for a valid URL
3364 to be included in the message, at least one part must configure
3365 path translation that would apply to location of main compose.
3366
3367 Only two messages will be sent, one for start and one for finish
3368 (either successful or not).
3369
3370 Partial compose settings
3371 Each part should have a separate section in the config file.
3372
3373 It can specify these options:
3374
3375 config Path to configuration file that describes this part. If rela‐
3376 tive, it is resolved relative to the file with parts configura‐
3377 tion.
3378
3379 just_phase, skip_phase
3380 Customize which phases should run for this part.
3381
3382 depends_on
3383 A comma separated list of other parts that must be finished
3384 before this part starts.
3385
3386 failable
3387 A boolean toggle to mark a part as failable. A failure in such
3388 part will mark the final compose as incomplete, but still suc‐
3389 cessful.
3390
3392 Daniel Mach
3393
3395 2021, Red Hat, Inc.
3396
3397
3398
3399
34004.2 Feb 12, 2021 PUNGI(1)