1PUNGI(1) Pungi PUNGI(1)
2
6 pungi - Pungi Documentation
7 Contents:
9
11 [image: Pungi Logo] [image]
12 Pungi is a distribution compose tool.
14 Composes are release snapshots that contain release deliverables such
16 as:
17 · installation trees
19 · RPMs
21 · repodata
23 · comps
25 · (bootable) ISOs
27 · kickstart trees
29 · anaconda images
31 · images for PXE boot
33 Tool overview
35 Pungi consists of multiple separate executables backed by a common
36 library.
37 The main entry-point is the pungi-koji script. It loads the compose
39 configuration and kicks off the process. Composing itself is done in
40 phases. Each phase is responsible for generating some artifacts on
41 disk and updating the compose object that is threaded through all the
42 phases.
43 Pungi itself does not actually do that much. Most of the actual work is
45 delegated to separate executables. Pungi just makes sure that all the
46 commands are invoked in the appropriate order and with correct argu‐
47 ments. It also moves the artifacts to correct locations.
48 Links
50 · Upstream GIT: https://pagure.io/pungi/
51 · Issue tracker: https://pagure.io/pungi/issues
53 · Questions can be asked on #fedora-releng IRC channel on FreeNode
55 Origin of name
57 The name Pungi comes from the instrument used to charm snakes. Anaconda
58 being the software Pungi was manipulating, and anaconda being a snake,
59 led to the referential naming.
60 The first name, which was suggested by Seth Vidal, was FIST, Fedora
62 Installation <Something> Tool. That name was quickly discarded and
63 replaced with Pungi.
64 There was also a bit of an inside joke that when said aloud, it could
66 sound like punji, which is a sharpened stick at the bottom of a trap.
67 Kind of like software…
68
70 Each invocation of pungi-koji consists of a set of phases. [image:
71 phase diagram] [image]
72 Most of the phases run sequentially (left-to-right in the diagram), but
74 there are use cases where multiple phases run in parallel. This happens
75 for phases whose main point is to wait for a Koji task to finish.
76 Init
78 The first phase to ever run. Can not be skipped. It prepares the comps
79 files for variants (by filtering out groups and packages that should
80 not be there). See comps for details about how this is done.
81 Pkgset
83 This phase loads a set of packages that should be composed. It has two
84 separate results: it prepares repos with packages in work/ directory
85 (one per arch) for further processing, and it returns a data structure
86 with mapping of packages to architectures.
87 Buildinstall
89 Spawns a bunch of threads, each of which runs either lorax or buildin‐
90 stall command (the latter coming from anaconda package). The commands
91 create boot.iso and other boot configuration files. The image is
92 finally linked into the compose/ directory as netinstall media.
93 The created images are also needed for creating live media or other
95 images in later phases.
96 With lorax this phase runs one task per variant.arch combination. For
98 buildinstall command there is only one task per architecture and prod‐
99 uct.img should be used to customize the results.
100 Gather
102 This phase uses data collected by pkgset phase and figures out what
103 packages should be in each variant. The basic mapping can come from
104 comps file, a JSON mapping or additional_packages config option. This
105 inputs can then be enriched by adding all dependencies. See gathering
106 for details.
107 Once the mapping is finalized, the packages are linked to appropriate
109 places and the rpms.json manifest is created.
110 ExtraFiles
112 This phase collects extra files from the configuration and copies them
113 to the compose directory. The files are described by a JSON file in the
114 compose subtree where the files are copied. This metadata is meant to
115 be distributed with the data (on ISO images).
116 Createrepo
118 This phase creates RPM repositories for each variant.arch tree. It is
119 actually reading the rpms.json manifest to figure out which packages
120 should be included.
121 OSTree
123 Updates an ostree repository with a new commit with packages from the
124 compose. The repository lives outside of the compose and is updated
125 immediately. If the compose fails in a later stage, the commit will not
126 be reverted.
127 Implementation wise, this phase runs rpm-ostree command in Koji runroot
129 (to allow running on different arches).
130 Productimg
132 Creates product.img files for customizing the bootable media created in
133 buildinstall phase.
134 Createiso
136 Generates ISO files and accumulates enough metadata to be able to cre‐
137 ate image.json manifest. The file is however not created in this phase,
138 instead it is dumped in the pungi-koji script itself.
139 The files include a repository with all RPMs from the variant. There
141 will be multiple images if the packages do not fit on a single image.
142 The image will be bootable if buildinstall phase is enabled and the
144 packages fit on a single image.
145 There can also be images with source repositories. These are never
147 bootable.
148 ExtraIsos
150 This phase is very similar to createiso, except it combines content
151 from multiple variants onto a single image. Packages, repodata and
152 extra files from each configured variant are put into a subdirectory.
153 Additional extra files can be put into top level of the image. The
154 image will be bootable if the main variant is bootable.
155 LiveImages, LiveMedia
157 Creates media in Koji with koji spin-livecd, koji spin-appliance or
158 koji spin-livemedia command. When the media are finished, the images
159 are copied into the compose/ directory and metadata for images is
160 updated.
161 ImageBuild
163 This phase wraps up koji image-build. It also updates the metadata
164 ultimately responsible for images.json manifest.
165 OSBS
167 This phase builds docker base images in OSBS.
168 The finished images are available in registry provided by OSBS, but not
170 downloaded directly into the compose. The is metadata about the created
171 image in compose/metadata/osbs.json.
172 OSTreeInstaller
174 Creates bootable media that carry an ostree repository as a payload.
175 These images are created by running lorax with special templates. Again
176 it runs in Koji runroot.
177 ImageChecksum
179 Responsible for generating checksums for the images. The checksums are
180 stored in image manifest as well as files on disk. The list of images
181 to be processed is obtained from the image manifest. This way all
182 images will get the same checksums irrespective of the phase that cre‐
183 ated them.
184 Test
186 This phase is supposed to run some sanity checks on the finished com‐
187 pose.
188 The first test is to run repoclosure on each repository. By default
190 errors are only reported in the log, the compose will still be consid‐
191 ered a success. The actual error has to be looked up in the compose
192 logs directory. Configuration allows customizing this.
193 The other test is to check all images listed the metadata and verify
195 that they look sane. For ISO files headers are checked to verify the
196 format is correct, and for bootable media a check is run to verify they
197 have properties that allow booting.
198
200 The configuration file parser is provided by kobo
201 The file follows a Python-like format. It consists of a sequence of
203 variables that have a value assigned to them.
204 variable = value
206 The variable names must follow the same convention as Python code:
208 start with a letter and consist of letters, digits and underscores
209 only.
210 The values can be either an integer, float, boolean (True or False), a
212 string or None. Strings must be enclosed in either single or double
213 quotes.
214 Complex types are supported as well.
216 A list is enclosed in square brackets and items are separated with com‐
218 mas. There can be a comma after the last item as well.
219 a_list = [1,
221 2,
222 3,
223 ]
224 A tuple works like a list, but is enclosed in parenthesis.
226 a_tuple = (1, "one")
228 A dictionary is wrapped in brackets, and consists of key: value pairs
230 separated by commas. The keys can only be formed from basic types (int,
231 float, string).
232 a_dict = {
234 'foo': 'bar',
235 1: None
236 }
237 The value assigned to a variable can also be taken from another vari‐
239 able.
240 one = 1
242 another = one
243 Anything on a line after a # symbol is ignored and functions as a com‐
245 ment.
246 Importing other files
248 It is possible to include another configuration file. The files are
249 looked up relative to the currently processed file.
250 The general structure of import is:
252 from FILENAME import WHAT
254 The FILENAME should be just the base name of the file without extension
256 (which must be .conf). WHAT can either be a comma separated list of
257 variables or *.
258 # Opens constants.conf and brings PI and E into current scope.
260 from constants import PI, E
261 # Opens common.conf and brings everything defined in that file into current
263 # file as well.
264 from common import *
265 NOTE:
267 Pungi will copy the configuration file given on command line into
268 the logs/ directory. Only this single file will be copied, not any
269 included ones. (Copying included files requires a fix in kobo
270 library.)
271 The JSON-formatted dump of configuration is correct though.
273 Formatting strings
275 String interpolation is available as well. It uses a %-encoded format.
276 See Python documentation for more details.
277 joined = "%s %s" % (var_a, var_b)
279 a_dict = {
281 "fst": 1,
282 "snd": 2,
283 }
284 another = "%(fst)s %(snd)s" % a_dict
285
287 Please read productmd documentation for terminology and other release
288 and compose related details.
289 Minimal Config Example
291 # RELEASE
292 release_name = "Fedora"
293 release_short = "Fedora"
294 release_version = "23"
295 # GENERAL SETTINGS
297 comps_file = "comps-f23.xml"
298 variants_file = "variants-f23.xml"
299 module_defaults_dir = "module_defaults"
300 # KOJI
302 koji_profile = "koji"
303 runroot = False
304 # PKGSET
306 sigkeys = [None]
307 pkgset_source = "koji"
308 pkgset_koji_tag = "f23"
309 # CREATEREPO
311 createrepo_checksum = "sha256"
312 # GATHER
314 gather_method = "deps"
315 greedy_method = "build"
316 check_deps = False
317 # BUILDINSTALL
319 bootable = True
320 buildinstall_method = "lorax"
321 Release
323 Following mandatory options describe a release.
324 Options
326 release_name [mandatory]
327 (str) – release name
328 release_short [mandatory]
330 (str) – release short name, without spaces and special charac‐
331 ters
332 release_version [mandatory]
334 (str) – release version
335 release_type = “ga” (str) – release type, for example ga,
337 updates or updates-testing. See list of all valid values in pro‐
338 ductmd documentation.
339 release_is_layered = False
341 (bool) – typically False for an operating system, True otherwise
342 release_internal = False
344 (bool) – whether the compose is meant for public consumption
345 Example
347 release_name = "Fedora"
348 release_short = "Fedora"
349 release_version = "23"
350 # release_type = "ga"
351 Base Product
353 Base product options are optional and we need to them only if we’re
354 composing a layered product built on another (base) product.
355 Options
357 base_product_name
358 (str) – base product name
359 base_product_short
361 (str) – base product short name, without spaces and special
362 characters
363 base_product_version
365 (str) – base product major version
366 base_product_type = “ga”
368 (str) – base product type, “ga”, “updates” etc., for full list
369 see documentation of productmd.
370 Example
372 release_name = "RPM Fusion"
373 release_short = "rf"
374 release_version = "23.0"
375 release_is_layered = True
377 base_product_name = "Fedora"
379 base_product_short = "Fedora"
380 base_product_version = "23"
381 General Settings
383 Options
384 comps_file [mandatory]
385 (scm_dict, str or None) – reference to comps XML file with
386 installation groups
387 variants_file [mandatory]
389 (scm_dict or str) – reference to variants XML file that defines
390 release variants and architectures
391 module_defaults_dir [optional]
393 (scm_dict or str) – reference the module defaults directory con‐
394 taining modulemd-defaults YAML documents
395 failable_deliverables [optional]
397 (list) – list which deliverables on which variant and architec‐
398 ture can fail and not abort the whole compose. This only applies
399 to buildinstall and iso parts. All other artifacts can be con‐
400 figured in their respective part of configuration.
401 Please note that * as a wildcard matches all architectures but
403 src.
404 comps_filter_environments [optional]
406 (bool) – When set to False, the comps files for variants will
407 not have their environments filtered to match the variant.
408 tree_arches
410 ([str]) – list of architectures which should be included; if
411 undefined, all architectures from variants.xml will be included
412 tree_variants
414 ([str]) – list of variants which should be included; if unde‐
415 fined, all variants from variants.xml will be included
416 repoclosure_strictness
418 (list) – variant/arch mapping describing how repoclosure should
419 run. Possible values are
420 · off – do not run repoclosure
422 · lenient – (default) run repoclosure and write results to
424 logs, but detected errors are only reported in logs
425 · fatal – abort compose when any issue is detected
427 When multiple blocks in the mapping match a variant/arch combi‐
429 nation, the last value will win.
430 repoclosure_backend
432 (str) – Select which tool should be used to run repoclosure over
433 created repositories. By default yum is used, but you can switch
434 to dnf. Please note that when dnf is used, the build dependen‐
435 cies check is skipped. On Python 3, only dnf backend is avail‐
436 able.
437 compose_type
439 (str) – Allows to set default compose type. Type set via a com‐
440 mand-line option overwrites this.
441 Example
443 comps_file = {
444 "scm": "git",
445 "repo": "https://git.fedorahosted.org/git/comps.git",
446 "branch": None,
447 "file": "comps-f23.xml.in",
448 }
449 variants_file = {
451 "scm": "git",
452 "repo": "https://pagure.io/pungi-fedora.git ",
453 "branch": None,
454 "file": "variants-fedora.xml",
455 }
456 failable_deliverables = [
458 ('^.*$', {
459 # Buildinstall can fail on any variant and any arch
460 '*': ['buildinstall'],
461 'src': ['buildinstall'],
462 # Nothing on i386 blocks the compose
463 'i386': ['buildinstall', 'iso', 'live'],
464 })
465 ]
466 tree_arches = ["x86_64"]
468 tree_variants = ["Server"]
469 repoclosure_strictness = [
471 # Make repoclosure failures fatal for compose on all variants …
472 ('^.*$', {'*': 'fatal'}),
473 # … except for Everything where it should not run at all.
474 ('^Everything$', {'*': 'off'})
475 ]
476 Image Naming
478 Both image name and volume id are generated based on the configuration.
479 Since the volume id is limited to 32 characters, there are more set‐
480 tings available. The process for generating volume id is to get a list
481 of possible formats and try them sequentially until one fits in the
482 length limit. If substitutions are configured, each attempted volume id
483 will be modified by it.
484 For layered products, the candidate formats are first image_volid_lay‐
486 ered_product_formats followed by image_volid_formats. Otherwise, only
487 image_volid_formats are tried.
488 If no format matches the length limit, an error will be reported and
490 compose aborted.
491 Options
493 There a couple common format specifiers available for both the options:
494 · compose_id
496 · release_short
498 · version
500 · date
502 · respin
504 · type
506 · type_suffix
508 · label
510 · label_major_version
512 · variant
514 · arch
516 · disc_type
518 image_name_format [optional]
520 (str|dict) – Python’s format string to serve as template for
521 image names. The value can also be a dict mapping variant UID
522 regexes to the format string. The pattern should not overlap,
523 otherwise it is undefined which one will be used.
524 This format will be used for all phases generating images. Cur‐
526 rently that means createiso, live_images and buildinstall.
527 Available extra keys are:
529 · disc_num
531 · suffix
533 image_volid_formats [optional]
535 (list) – A list of format strings for generating volume id.
536 The extra available keys are:
538 · base_product_short
540 · base_product_version
542 image_volid_layered_product_formats [optional]
544 (list) – A list of format strings for generating volume id for
545 layered products. The keys available are the same as for
546 image_volid_formats.
547 restricted_volid = False
549 (bool) – New versions of lorax replace all non-alphanumerical
550 characters with dashes (underscores are preserved). This option
551 will mimic similar behaviour in Pungi.
552 volume_id_substitutions [optional]
554 (dict) – A mapping of string replacements to shorten the volume
555 id.
556 disc_types [optional]
558 (dict) – A mapping for customizing disc_type used in image
559 names.
560 Available keys are:
562 · boot – for boot.iso images created in buildinstall
564 phase
565 · live – for images created by live_images phase
567 · dvd – for images created by createiso phase
569 · ostree – for ostree installer images
571 Default values are the same as the keys.
573 Example
575 # Image name respecting Fedora's image naming policy
576 image_name_format = "%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s%(suffix)s"
577 # Use the same format for volume id
578 image_volid_formats = [
579 "%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s"
580 ]
581 # No special handling for layered products, use same format as for regular images
582 image_volid_layered_product_formats = []
583 # Replace "Cloud" with "C" in volume id etc.
584 volume_id_substitutions = {
585 'Cloud': 'C',
586 'Alpha': 'A',
587 'Beta': 'B',
588 'TC': 'T',
589 }
590 disc_types = {
592 'boot': 'netinst',
593 'live': 'Live',
594 'dvd': 'DVD',
595 }
596 Signing
598 If you want to sign deliverables generated during pungi run like RPM
599 wrapped images. You must provide few configuration options:
600 signing_command [optional]
602 (str) – Command that will be run with a koji build as a single
603 argument. This command must not require any user interaction.
604 If you need to pass a password for a signing key to the command,
605 do this via command line option of the command and use string
606 formatting syntax %(signing_key_password)s. (See sign‐
607 ing_key_password_file).
608 signing_key_id [optional]
610 (str) – ID of the key that will be used for the signing. This
611 ID will be used when crafting koji paths to signed files (kojip‐
612 kgs.fedoraproject.org/pack‐
613 ages/NAME/VER/REL/data/signed/KEYID/..).
614 signing_key_password_file [optional]
616 (str) – Path to a file with password that will be formatted into
617 signing_command string via %(signing_key_password)s string for‐
618 mat syntax (if used). Because pungi config is usualy stored in
619 git and is part of compose logs we don’t want password to be
620 included directly in the config. Note: If - string is used
621 instead of a filename, then you will be asked for the password
622 interactivelly right after pungi starts.
623 Example
625 signing_command = '~/git/releng/scripts/sigulsign_unsigned.py -vv --password=%(signing_key_password)s fedora-24'
626 signing_key_id = '81b46521'
627 signing_key_password_file = '~/password_for_fedora-24_key'
628 Git URLs
630 In multiple places the config requires URL of a Git repository to down‐
631 load some file from. This URL is passed on to Koji. It is possible to
632 specify which commit to use using this syntax:
633 git://git.example.com/git/repo-name.git?#<rev_spec>
635 The <rev_spec> pattern can be replaced with actual commit SHA, a tag
637 name, HEAD to indicate that tip of default branch should be used or
638 origin/<branch_name> to use tip of arbitrary branch.
639 If the URL specifies a branch or HEAD, Pungi will replace it with the
641 actual commit SHA. This will later show up in Koji tasks and help with
642 tracing what particular inputs were used.
643 NOTE:
645 The origin must be specified because of the way Koji works with the
646 repository. It will clone the repository then switch to requested
647 state with git reset --hard REF. Since no local branches are cre‐
648 ated, we need to use full specification including the name of the
649 remote.
650 Createrepo Settings
652 Options
653 createrepo_checksum
654 (str) – specify checksum type for createrepo; expected values:
655 sha512, sha256, sha. Defaults to sha256.
656 createrepo_c = True
658 (bool) – use createrepo_c (True) or legacy createrepo (False)
659 createrepo_deltas = False
661 (list) – generate delta RPMs against an older compose. This
662 needs to be used together with --old-composes command line argu‐
663 ment. The value should be a mapping of variants and architec‐
664 tures that should enable creating delta RPMs. Source and debug‐
665 info repos never have deltas.
666 createrepo_use_xz = False
668 (bool) – whether to pass --xz to the createrepo command. This
669 will cause the SQLite databases to be compressed with xz.
670 createrepo_num_threads
672 (int) – how many concurrent createrepo process to run. The
673 default is to use one thread per CPU available on the machine.
674 createrepo_num_workers
676 (int) – how many concurrent createrepo workers to run. Value
677 defaults to 3.
678 createrepo_database
680 (bool) – whether to create SQLite database as part of the repo‐
681 data. This is only useful as an optimization for clients using
682 Yum to consume to the repo. Default value depends on gather
683 backend. For DNF it’s turned off, for Yum the default is True.
684 createrepo_extra_args
686 ([str]) – a list of extra arguments passed on to createrepo or
687 createrepo_c executable. This could be useful for enabling
688 zchunk generation and pointing it to correct dictionaries.
689 product_id = None
691 (scm_dict) – If specified, it should point to a directory with
692 certificates <variant_uid>-<arch>-*.pem. Pungi will copy each
693 certificate file into the relevant Yum repositories as a produc‐
694 tid file in the repodata directories. The purpose of these pro‐
695 ductid files is to expose the product data to
696 subscription-manager. subscription-manager inclues a “prod‐
697 uct-id” Yum plugin that can read these productid certificate
698 files from each Yum repository.
699 product_id_allow_missing = False
701 (bool) – When product_id is used and a certificate for some
702 variant and architecture is missing, Pungi will exit with an
703 error by default. When you set this option to True, Pungi will
704 ignore the missing certificate and simply log a warning message.
705 Example
707 createrepo_checksum = "sha"
708 createrepo_deltas = [
709 # All arches for Everything should have deltas.
710 ('^Everything$', {'*': True}),
711 # Also Server.x86_64 should have them (but not on other arches).
712 ('^Server$', {'x86_64': True}),
713 ]
714 Package Set Settings
716 Options
717 sigkeys
718 ([str or None]) – priority list of sigkeys; if the list includes
719 an empty string or None, unsigned packages will be allowed
720 pkgset_source [mandatory]
722 (str) – “koji” (any koji instance) or “repos” (arbitrary yum
723 repositories)
724 pkgset_koji_tag
726 (str|[str]) – tag(s) to read package set from. This option can
727 be omitted for modular composes.
728 pkgset_koji_builds
730 (str|[str]) – extra build(s) to include in a package set defined
731 as NVRs.
732 pkgset_koji_module_tag
734 (str|[str]) – tags to read module from. This option works simi‐
735 larly to listing tags in variants XML. If tags are specified and
736 variants XML specifies some modules via NSVC (or part of), only
737 modules matching that list will be used (and taken from the
738 tag). Inheritance is used automatically.
739 pkgset_koji_inherit = True
741 (bool) – inherit builds from parent tags; we can turn it off
742 only if we have all builds tagged in a single tag
743 pkgset_koji_inherit_modules = False
745 (bool) – the same as above, but this only applies to modular
746 tags. This option applies to the content tags that contain the
747 RPMs.
748 pkgset_repos
750 (dict) – A mapping of architectures to repositories with RPMs:
751 {arch: [repo]}. Only use when pkgset_source = "repos".
752 pkgset_exclusive_arch_considers_noarch = True
754 (bool) – If a package includes noarch in its ExclusiveArch tag,
755 it will be included in all architectures since noarch is compat‐
756 ible with everything. Set this option to False to ignore noarch
757 in ExclusiveArch and always consider only binary architectures.
758 Example
760 sigkeys = [None]
761 pkgset_source = "koji"
762 pkgset_koji_tag = "f23"
763 Buildinstall Settings
765 Script or process that creates bootable images with Anaconda installer
766 is historically called buildinstall.
767 Options
769 bootable
770 (bool) – whether to run the buildinstall phase
771 buildinstall_method
773 (str) – “lorax” (f16+, rhel7+) or “buildinstall” (older
774 releases)
775 lorax_options
777 (list) – special options passed on to lorax.
778 Format: [(variant_uid_regex, {arch|*: {option: name}})].
780 Recognized options are:
782 · bugurl – str (default None)
784 · nomacboot – bool (default True)
786 · noupgrade – bool (default True)
788 · add_template – [str] (default empty)
790 · add_arch_template – [str] (default empty)
792 · add_template_var – [str] (default empty)
794 · add_arch_template_var – [str] (default empty)
796 · rootfs_size – [int] (default empty)
798 · version – [str] (default from release_version) – used
800 as --version and --release argument on the lorax com‐
801 mand line
802 lorax_extra_sources
804 (list) – a variant/arch mapping with urls for extra source
805 repositories added to Lorax command line. Either one repo or a
806 list can be specified.
807 buildinstall_kickstart
809 (scm_dict) – If specified, this kickstart file will be copied
810 into each file and pointed to in boot configuration.
811 buildinstall_topdir
813 (str) – Full path to top directory where the runroot buildin‐
814 stall Koji tasks output should be stored. This is useful in sit‐
815 uation when the Pungi compose is not generated on the same stor‐
816 age as the Koji task is running on. In this case, Pungi can pro‐
817 vide input repository for runroot task using HTTP and set the
818 output directory for this task to buildinstall_topdir. Once the
819 runroot task finishes, Pungi will copy the results of runroot
820 tasks to the compose working directory.
821 buildinstall_skip
823 (list) – mapping that defines which variants and arches to skip
824 during buildinstall; format: [(variant_uid_regex, {arch|*:
825 True})]. This is only supported for lorax.
826 Example
828 bootable = True
829 buildinstall_method = "lorax"
830 # Enables macboot on x86_64 for all variants and builds upgrade images
832 # everywhere.
833 lorax_options = [
834 ("^.*$", {
835 "x86_64": {
836 "nomacboot": False
837 }
838 "*": {
839 "noupgrade": False
840 }
841 })
842 ]
843 # Don't run buildinstall phase for Modular variant
845 buildinstall_skip = [
846 ('^Modular', {
847 '*': True
848 })
849 ]
850 # Add another repository for lorax to install packages from
852 lorax_extra_sources = [
853 ('^Simple$', {
854 '*': 'https://example.com/repo/$basearch/',
855 })
856 ]
857 NOTE:
859 It is advised to run buildinstall (lorax) in koji, i.e. with runroot
860 enabled for clean build environments, better logging, etc.
861 WARNING:
863 Lorax installs RPMs into a chroot. This involves running %post
864 scriptlets and they frequently run executables in the chroot. If
865 we’re composing for multiple architectures, we must use runroot for
866 this reason.
867 Gather Settings
869 Options
870 gather_method [mandatory]
871 (str*|*dict) – Options are deps, nodeps and hybrid. Specifies
872 whether and how package dependencies should be pulled in. Pos‐
873 sible configuration can be one value for all variants, or if
874 configured per-variant it can be a simple string hybrid or a a
875 dictionary mapping source type to a value of deps or nodeps.
876 Make sure only one regex matches each variant, as there is no
877 guarantee which value will be used if there are multiple match‐
878 ing ones. All used sources must have a configured method unless
879 hybrid solving is used.
880 gather_fulltree = False
882 (bool) – When set to True all RPMs built from an SRPM will
883 always be included. Only use when gather_method = "deps".
884 gather_selfhosting = False
886 (bool) – When set to True, Pungi will build a self-hosting tree
887 by following build dependencies. Only use when gather_method =
888 "deps".
889 greedy_method
891 (str) – This option controls how package requirements are satis‐
892 fied in case a particular Requires has multiple candidates.
893 · none – the best packages is selected to satisfy the dependency
895 and only that one is pulled into the compose
896 · all – packages that provide the symbol are pulled in
898 · build – the best package is selected, and then all packages
900 from the same build that provide the symbol are pulled in
901 NOTE:
903 As an example let’s work with this situation: a package in
904 the compose has Requires: foo. There are three packages with
905 Provides: foo: pkg-a, pkg-b-provider-1 and pkg-b-provider-2.
906 The pkg-b-* packages are build from the same source package.
907 Best match determines pkg-b-provider-1 as best matching pack‐
908 age.
909 · With greedy_method = "none" only pkg-b-provider-1 will be
911 pulled in.
912 · With greedy_method = "all" all three packages will be
914 pulled in.
915 · With greedy_method = "build" ``pkg-b-provider-1 and
917 pkg-b-provider-2 will be pulled in.
918 gather_backend
920 (str) –This changes the entire codebase doing dependency solv‐
921 ing, so it can change the result in unpredictable ways.
922 On Python 2, the choice is between yum or dnf and defaults to
924 yum. On Python 3 dnf is the only option and default.
925 Particularly the multilib work is performed differently by using
927 python-multilib library. Please refer to multilib option to see
928 the differences.
929 multilib
931 (list) – mapping of variant regexes and arches to list of multi‐
932 lib methods
933 Available methods are:
935 · none – no package matches this method
937 · all – all packages match this method
939 · runtime – packages that install some shared object file
941 (*.so.*) will match.
942 · devel – packages whose name ends with -devel or
944 --static suffix will be matched. When dnf is used, this
945 method automatically enables runtime method as well.
946 With yum backend this method also uses a hardcoded
947 blacklist and whitelist.
948 · kernel – packages providing kernel or kernel-devel
950 match this method (only in yum backend)
951 · yaboot – only yaboot package on ppc arch matches this
953 (only in yum backend)
954 additional_packages
956 (list) – additional packages to be included in a variant and
957 architecture; format: [(variant_uid_regex, {arch|*: [pack‐
958 age_globs]})]
959 The packages specified here are matched against RPM names, not
961 any other provides in the package not the name of source pack‐
962 age. Shell globbing is used, so wildcards are possible. The
963 package can be specified as name only or name.arch.
964 filter_packages
966 (list) – packages to be excluded from a variant and architec‐
967 ture; format: [(variant_uid_regex, {arch|*: [package_globs]})]
968 See additional_packages for details about package specification.
970 filter_system_release_packages
972 (bool) – for each variant, figure out the best system release
973 package and filter out all others. This will not work if a vari‐
974 ant needs more than one system release package. In such case,
975 set this option to False.
976 gather_prepopulate = None
978 (scm_dict) – If specified, you can use this to add additional
979 packages. The format of the file pointed to by this option is a
980 JSON mapping {variant_uid: {arch: {build: [package]}}}. Packages
981 added through this option can not be removed by filter_packages.
982 multilib_blacklist
984 (dict) – multilib blacklist; format: {arch|*: [package_globs]}.
985 See additional_packages for details about package specification.
987 multilib_whitelist
989 (dict) – multilib blacklist; format: {arch|*: [package_names]}.
990 The whitelist must contain exact package names; there are no
991 wildcards or pattern matching.
992 gather_lookaside_repos = []
994 (list) – lookaside repositories used for package gathering; for‐
995 mat: [(variant_uid_regex, {arch|*: [repo_urls]})]
996 hashed_directories = False
998 (bool) – put packages into “hashed” directories, for example
999 Packages/k/kernel-4.0.4-301.fc22.x86_64.rpm
1000 check_deps = True
1002 (bool) – Set to False if you don’t want the compose to abort
1003 when some package has broken dependencies.
1004 require_all_comps_packages = False
1006 (bool) – Set to True to abort compose when package mentioned in
1007 comps file can not be found in the package set. When disabled
1008 (the default), such cases are still reported as warnings in the
1009 log.
1010 gather_source_mapping
1012 (str) – JSON mapping with initial packages for the compose. The
1013 value should be a path to JSON file with following mapping:
1014 {variant: {arch: {rpm_name: [rpm_arch|None]}}}.
1015 gather_profiler = False
1017 (bool) – When set to True the gather tool will produce addi‐
1018 tional performance profiling information at the end of its logs.
1019 Only takes effect when gather_backend = "dnf".
1020 variant_as_lookaside
1022 (list) – a variant/variant mapping that tells one or more vari‐
1023 ants in compose has other variant(s) in compose as a lookaside.
1024 Only top level variants are supported (not addons/layered prod‐
1025 ucts). Format: [(variant_uid, variant_uid)]
1026 Example
1028 gather_method = "deps"
1029 greedy_method = "build"
1030 check_deps = False
1031 hashed_directories = True
1032 gather_method = {
1034 "^Everything$": {
1035 "comps": "deps" # traditional content defined by comps groups
1036 },
1037 "^Modular$": {
1038 "module": "nodeps" # Modules do not need dependencies
1039 },
1040 "^Mixed$": { # Mixed content in one variant
1041 "comps": "deps",
1042 "module": "nodeps"
1043 }
1044 "^OtherMixed$": "hybrid", # Using hybrid depsolver
1045 }
1046 additional_packages = [
1048 # bz#123456
1049 ('^(Workstation|Server)$', {
1050 '*': [
1051 'grub2',
1052 'kernel',
1053 ],
1054 }),
1055 ]
1056 filter_packages = [
1058 # bz#111222
1059 ('^.*$', {
1060 '*': [
1061 'kernel-doc',
1062 ],
1063 }),
1064 ]
1065 multilib = [
1067 ('^Server$', {
1068 'x86_64': ['devel', 'runtime']
1069 })
1070 ]
1071 multilib_blacklist = {
1073 "*": [
1074 "gcc",
1075 ],
1076 }
1077 multilib_whitelist = {
1079 "*": [
1080 "alsa-plugins-*",
1081 ],
1082 }
1083 # gather_lookaside_repos = [
1085 # ('^.*$', {
1086 # 'x86_64': [
1087 # "https://dl.fedoraproject.org/pub/fedora/linux/releases/22/Everything/x86_64/os/",
1088 # "https://dl.fedoraproject.org/pub/fedora/linux/releases/22/Everything/source/SRPMS/",
1089 # ]
1090 # }),
1091 # ]
1092 NOTE:
1094 It is a good practice to attach bug/ticket numbers to addi‐
1095 tional_packages, filter_packages, multilib_blacklist and multi‐
1096 lib_whitelist to track decisions.
1097 Koji Settings
1099 Options
1100 koji_profile
1101 (str) – koji profile name. This tells Pungi how to communicate
1102 with your chosen Koji instance. See Koji’s documentation about
1103 profiles for more information about how to set up your Koji
1104 client profile. In the examples, the profile name is “koji”,
1105 which points to Fedora’s koji.fedoraproject.org.
1106 runroot [mandatory]
1108 (bool) – run some tasks such as buildinstall or createiso in
1109 koji build root (True) or locally (False). There might also be
1110 other runroot methods available. These can be chosen by run‐
1111 root_method option.
1112 runroot_method
1114 (str) – Runroot method to use. It can further specify the run‐
1115 root method in case the runroot is set to True.
1116 Available methods are:
1118 · local – runroot tasks are run locally
1120 · koji – runroot tasks are run in Koji
1122 · openssh – runroot tasks are run on remote machine con‐
1124 nected using OpenSSH. The runroot_ssh_hostnames for
1125 each architecture must be set and the user under which
1126 Pungi runs must be configured to login as run‐
1127 root_ssh_username using the SSH key.
1128 runroot_channel
1130 (str) – name of koji channel
1131 runroot_tag
1133 (str) – name of koji build tag used for runroot
1134 runroot_weights
1136 (dict) – customize task weights for various runroot tasks. The
1137 values in the mapping should be integers, the keys can be
1138 selected from the following list. By default no weight is
1139 assigned and Koji picks the default one according to policy.
1140 · buildinstall
1142 · createiso
1144 · ostree
1146 · ostree_installer
1148 Example
1150 koji_profile = "koji"
1151 runroot = True
1152 runroot_channel = "runroot"
1153 runroot_tag = "f23-build"
1154 Runroot “openssh” method settings
1156 Options
1157 runroot_ssh_username
1158 (str) – For openssh runroot method, configures the username used
1159 to login the remote machine to run the runroot task. Defaults to
1160 “root”.
1161 runroot_ssh_hostnames
1163 (dict) – For openssh runroot method, defines the hostname for
1164 each architecture on which the runroot task should be running.
1165 Format: {"x86_64": "runroot-x86-64.localhost.tld", ...}
1166 runroot_ssh_init_template
1168 (str) [optional] – For openssh runroot method, defines the com‐
1169 mand to initializes the runroot task on the remote machine. This
1170 command is executed as first command for each runroot task exe‐
1171 cuted.
1172 The command can print a string which is then available as {run‐
1174 root_key} for other SSH commands. This string might be used to
1175 keep the context across different SSH commands executed for sin‐
1176 gle runroot task.
1177 The goal of this command is setting up the environment for real
1179 runroot commands. For example preparing the unique mock environ‐
1180 ment, mounting the desired file-systems, …
1181 The command string can contain following variables which are
1183 replaced by the real values before executing the init command:
1184 · {runroot_tag} - Tag to initialize the runroot environment
1186 from.
1187 When not set, no init command is executed.
1189 runroot_ssh_install_packages_template
1191 (str) [optional] – For openssh runroot method, defines the tem‐
1192 plate for command to install the packages requested to run the
1193 runroot task.
1194 The template string can contain following variables which are
1196 replaced by the real values before executing the install com‐
1197 mand:
1198 · {runroot_key} - Replaced with the string returned by run‐
1200 root_ssh_init_template if used. This can be used to keep the
1201 track of context of SSH commands beloging to single runroot
1202 task.
1203 · {packages} - White-list separated list of packages to install.
1205 Example (The {runroot_key} is expected to be set to mock config
1207 file using the runroot_ssh_init_template command.): "mock -r
1208 {runroot_key} --install {packages}"
1209 When not set, no command to install packages on remote machine
1211 is executed.
1212 runroot_ssh_run_template
1214 (str) [optional] – For openssh runroot method, defines the tem‐
1215 plate for the main runroot command.
1216 The template string can contain following variables which are
1218 replaced by the real values before executing the install com‐
1219 mand:
1220 · {runroot_key} - Replaced with the string returned by run‐
1222 root_ssh_init_template if used. This can be used to keep the
1223 track of context of SSH commands beloging to single runroot
1224 task.
1225 · {command} - Command to run.
1227 Example (The {runroot_key} is expected to be set to mock config
1229 file using the runroot_ssh_init_template command.): "mock -r
1230 {runroot_key} chroot -- {command}"
1231 When not set, the runroot command is run directly.
1233 Extra Files Settings
1235 Options
1236 extra_files
1237 (list) – references to external files to be placed in os/ direc‐
1238 tory and media; format: [(variant_uid_regex, {arch|*:
1239 [scm_dict]})]. See scm_support for details. If the dict speci‐
1240 fies a target key, an additional subdirectory will be used.
1241 Example
1243 extra_files = [
1244 ('^.*$', {
1245 '*': [
1246 # GPG keys
1247 {
1248 "scm": "rpm",
1249 "repo": "fedora-repos",
1250 "branch": None,
1251 "file": [
1252 "/etc/pki/rpm-gpg/RPM-GPG-KEY-22-fedora",
1253 ],
1254 "target": "",
1255 },
1256 # GPL
1257 {
1258 "scm": "git",
1259 "repo": "https://pagure.io/pungi-fedora",
1260 "branch": None,
1261 "file": [
1262 "GPL",
1263 ],
1264 "target": "",
1265 },
1266 ],
1267 }),
1268 ]
1269 Extra Files Metadata
1271 If extra files are specified a metadata file, extra_files.json, is
1272 placed in the os/ directory and media. The checksums generated are
1273 determined by media_checksums option. This metadata file is in the for‐
1274 mat:
1275 {
1277 "header": {"version": "1.0},
1278 "data": [
1279 {
1280 "file": "GPL",
1281 "checksums": {
1282 "sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
1283 },
1284 "size": 18092
1285 },
1286 {
1287 "file": "release-notes/notes.html",
1288 "checksums": {
1289 "sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
1290 },
1291 "size": 1120
1292 }
1293 ]
1294 }
1295 Productimg Settings
1297 Product images are placed on installation media and provide additional
1298 branding and Anaconda changes specific to product variants.
1299 Options
1301 productimg = False
1302 (bool) – create product images; requires bootable=True
1303 productimg_install_class
1305 (scm_dict, str) – reference to install class file
1306 productimg_po_files
1308 (scm_dict, str) – reference to a directory with po files for
1309 install class translations
1310 Example
1312 productimg = True
1313 productimg_install_class = {
1314 "scm": "git",
1315 "repo": "http://git.example.com/productimg.git",
1316 "branch": None,
1317 "file": "fedora23/%(variant_id)s.py",
1318 }
1319 productimg_po_files = {
1320 "scm": "git",
1321 "repo": "http://git.example.com/productimg.git",
1322 "branch": None,
1323 "dir": "po",
1324 }
1325 CreateISO Settings
1327 Options
1328 createiso_skip = False
1329 (list) – mapping that defines which variants and arches to skip
1330 during createiso; format: [(variant_uid_regex, {arch|*: True})]
1331 createiso_max_size
1333 (list) – mapping that defines maximum expected size for each
1334 variant and arch. If the ISO is larger than the limit, a warning
1335 will be issued.
1336 Format: [(variant_uid_regex, {arch|*: number})]
1338 create_jigdo = True
1340 (bool) – controls the creation of jigdo from ISO
1341 create_optional_isos = False
1343 (bool) – when set to True, ISOs will be created even for
1344 optional variants. By default only variants with type variant or
1345 layered-product will get ISOs.
1346 createiso_break_hardlinks = False
1348 (bool) – when set to True, all files that should go on the ISO
1349 and have a hardlink will be first copied into a staging direc‐
1350 tory. This should work around a bug in genisoimage including
1351 incorrect link count in the image, but it is at the cost of hav‐
1352 ing to copy a potentially significant amount of data.
1353 The staging directory is deleted when ISO is successfully cre‐
1355 ated. In that case the same task to create the ISO will not be
1356 re-runnable.
1357 iso_size = 4700000000
1359 (int|str) – size of ISO image. The value should either be an
1360 integer meaning size in bytes, or it can be a string with k, M,
1361 G suffix (using multiples of 1024).
1362 split_iso_reserve = 10MiB
1364 (int|str) – how much free space should be left on each disk. The
1365 format is the same as for iso_size option.
1366 iso_hfs_ppc64le_compatible = True
1368 (bool) – when set to False, the Apple/HFS compatibility is
1369 turned off for ppc64le ISOs. This option only makes sense for
1370 bootable products, and affects images produced in createiso and
1371 extra_isos phases.
1372 NOTE:
1374 Source architecture needs to be listed explicitly. Excluding ‘*’
1375 applies only on binary arches. Jigdo causes significant increase of
1376 time to ISO creation.
1377 Example
1379 createiso_skip = [
1380 ('^Workstation$', {
1381 '*': True,
1382 'src': True
1383 }),
1384 ]
1385 Automatic generation of version and release
1387 Version and release values for certain artifacts can be generated auto‐
1388 matically based on release version, compose label, date, type and
1389 respin. This can be used to shorten the config and keep it the same for
1390 multiple uses.
1391┌───────────────────────┬───────────────┬──────────┬──────────┬────────┬──────────────┐
1393│Compose ID │ Label │ Version │ Date │ Respin │ Release │
1394├───────────────────────┼───────────────┼──────────┼──────────┼────────┼──────────────┤
1395│F-Rawhide-20170406.n.0 │ - │ Rawhide │ 20170406 │ 0 │ 20170406.n.0 │
1396├───────────────────────┼───────────────┼──────────┼──────────┼────────┼──────────────┤
1397│F-26-20170329.1 │ Alpha-1.6 │ 26_Alpha │ 20170329 │ 1 │ 1.6 │
1398├───────────────────────┼───────────────┼──────────┼──────────┼────────┼──────────────┤
1399│F-Atomic-25-20170407.0 │ RC-20170407.0 │ 25 │ 20170407 │ 0 │ 20170407.0 │
1400├───────────────────────┼───────────────┼──────────┼──────────┼────────┼──────────────┤
1401│F-Atomic-25-20170407.0 │ - │ 25 │ 20170407 │ 0 │ 20170407.0 │
1402└───────────────────────┴───────────────┴──────────┴──────────┴────────┴──────────────┘
1403 All non-RC milestones from label get appended to the version. For
1405 release either label is used or date, type and respin.
1406 Common options for Live Images, Live Media and Image Build
1408 All images can have ksurl, version, release and target specified. Since
1409 this can create a lot of duplication, there are global options that can
1410 be used instead.
1411 For each of the phases, if the option is not specified for a particular
1413 deliverable, an option named <PHASE_NAME>_<OPTION> is checked. If that
1414 is not specified either, the last fallback is global_<OPTION>. If even
1415 that is unset, the value is considered to not be specified.
1416 The kickstart URL is configured by these options.
1418 · global_ksurl – global fallback setting
1420 · live_media_ksurl
1422 · image_build_ksurl
1424 · live_images_ksurl
1426 Target is specified by these settings.
1428 · global_target – global fallback setting
1430 · live_media_target
1432 · image_build_target
1434 · live_images_target
1436 Version is specified by these options. If no version is set, a default
1438 value will be provided according to automatic versioning.
1439 · global_version – global fallback setting
1441 · live_media_version
1443 · image_build_version
1445 · live_images_version
1447 Release is specified by these options. If set to a magic value to
1449 !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN, a value will be generated accord‐
1450 ing to automatic versioning.
1451 · global_release – global fallback setting
1453 · live_media_release
1455 · image_build_release
1457 · live_images_release
1459 Each configuration block can also optionally specify a failable key.
1461 For live images it should have a boolean value. For live media and
1462 image build it should be a list of strings containing architectures
1463 that are optional. If any deliverable fails on an optional architec‐
1464 ture, it will not abort the whole compose. If the list contains only
1465 "*", all arches will be substituted.
1466 Live Images Settings
1468 live_images
1469 (list) – Configuration for the particular image. The elements of
1470 the list should be tuples (variant_uid_regex, {arch|*: config}).
1471 The config should be a dict with these keys:
1472 · kickstart (str)
1474 · ksurl (str) [optional] – where to get the kickstart from
1476 · name (str)
1478 · version (str)
1480 · target (str)
1482 · repo (str|[str]) – repos specified by URL or variant UID
1484 · specfile (str) – for images wrapped in RPM
1486 · scratch (bool) – only RPM-wrapped images can use scratch
1488 builds, but by default this is turned off
1489 · type (str) – what kind of task to start in Koji. Defaults
1491 to live meaning koji spin-livecd will be used. Alternative
1492 option is appliance corresponding to koji spin-appliance.
1493 · sign (bool) – only RPM-wrapped images can be signed
1495 live_images_no_rename
1497 (bool) – When set to True, filenames generated by Koji will be
1498 used. When False, filenames will be generated based on
1499 image_name_format configuration option.
1500 Live Media Settings
1502 live_media
1503 (dict) – configuration for koji spin-livemedia; format: {vari‐
1504 ant_uid_regex: [{opt:value}]}
1505 Required options:
1507 · name (str)
1509 · version (str)
1511 · arches ([str]) – what architectures to build the media for;
1513 by default uses all arches for the variant.
1514 · kickstart (str) – name of the kickstart file
1516 Available options:
1518 · ksurl (str)
1520 · ksversion (str)
1522 · scratch (bool)
1524 · target (str)
1526 · release (str) – a string with the release, or
1528 !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN to automatically gen‐
1529 erate a suitable value. See automatic versioning for
1530 details.
1531 · skip_tag (bool)
1533 · repo (str|[str]) – repos specified by URL or variant UID
1535 · title (str)
1537 · install_tree_from (str) – variant to take install tree from
1539 Image Build Settings
1541 image_build
1542 (dict) – config for koji image-build; format: {vari‐
1543 ant_uid_regex: [{opt: value}]}
1544 By default, images will be built for each binary arch valid for
1546 the variant. The config can specify a list of arches to narrow
1547 this down.
1548 NOTE:
1550 Config can contain anything what is accepted by koji image-build
1551 --config configfile.ini
1552 Repo can be specified either as a string or a list of strings. It
1554 will be automatically transformed into format suitable for koji. A
1555 repo for the currently built variant will be added as well.
1556 If you explicitly set release to
1558 !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN, it will be replaced with a
1559 value generated as described in automatic versioning.
1560 If you explicitly set release to !RELEASE_FROM_DATE_RESPIN, it will
1562 be replaced with a value generated as described in automatic ver‐
1563 sioning.
1564 If you explicitly set version to !VERSION_FROM_VERSION, it will be
1566 replaced with a value generated as described in automatic version‐
1567 ing.
1568 Please don’t set install_tree. This gets automatically set by pungi
1570 based on current variant. You can use install_tree_from key to use
1571 install tree from another variant.
1572 Both the install tree and repos can use one of following formats:
1574 · URL to the location
1576 · name of variant in the current compose
1578 · absolute path on local filesystem (which will be translated
1580 using configured mappings or used unchanged, in which case you
1581 have to ensure the koji builders can access it)
1582 You can set either a single format, or a list of formats. For avail‐
1584 able values see help output for koji image-build command.
1585 If ksurl ends with #HEAD, Pungi will figure out the SHA1 hash of
1587 current HEAD and use that instead.
1588 Setting scratch to True will run the koji tasks as scratch builds.
1590 Example
1592 image_build = {
1593 '^Server$': [
1594 {
1595 'image-build': {
1596 'format': ['docker', 'qcow2']
1597 'name': 'fedora-qcow-and-docker-base',
1598 'target': 'koji-target-name',
1599 'ksversion': 'F23', # value from pykickstart
1600 'version': '23',
1601 # correct SHA1 hash will be put into the URL below automatically
1602 'ksurl': 'https://git.fedorahosted.org/git/spin-kickstarts.git?somedirectoryifany#HEAD',
1603 'kickstart': "fedora-docker-base.ks",
1604 'repo': ["http://someextrarepos.org/repo", "ftp://rekcod.oi/repo"],
1605 'distro': 'Fedora-20',
1606 'disk_size': 3,
1607 # this is set automatically by pungi to os_dir for given variant
1609 # 'install_tree': 'http://somepath',
1610 },
1611 'factory-parameters': {
1612 'docker_cmd': "[ '/bin/bash' ]",
1613 'docker_env': "[ 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' ]",
1614 'docker_labels': "{'Name': 'fedora-docker-base', 'License': u'GPLv2', 'RUN': 'docker run -it --rm ${OPT1} --privileged -v \`pwd\`:/atomicapp -v /run:/run -v /:/host --net=host --name ${NAME} -e NAME=${NAME} -e IMAGE=${IMAGE} ${IMAGE} -v ${OPT2} run ${OPT3} /atomicapp', 'Vendor': 'Fedora Project', 'Version': '23', 'Architecture': 'x86_64' }",
1615 }
1616 },
1617 {
1618 'image-build': {
1619 'format': ['docker', 'qcow2']
1620 'name': 'fedora-qcow-and-docker-base',
1621 'target': 'koji-target-name',
1622 'ksversion': 'F23', # value from pykickstart
1623 'version': '23',
1624 # correct SHA1 hash will be put into the URL below automatically
1625 'ksurl': 'https://git.fedorahosted.org/git/spin-kickstarts.git?somedirectoryifany#HEAD',
1626 'kickstart': "fedora-docker-base.ks",
1627 'repo': ["http://someextrarepos.org/repo", "ftp://rekcod.oi/repo"],
1628 'distro': 'Fedora-20',
1629 'disk_size': 3,
1630 # this is set automatically by pungi to os_dir for given variant
1632 # 'install_tree': 'http://somepath',
1633 }
1634 },
1635 {
1636 'image-build': {
1637 'format': 'qcow2',
1638 'name': 'fedora-qcow-base',
1639 'target': 'koji-target-name',
1640 'ksversion': 'F23', # value from pykickstart
1641 'version': '23',
1642 'ksurl': 'https://git.fedorahosted.org/git/spin-kickstarts.git?somedirectoryifany#HEAD',
1643 'kickstart': "fedora-docker-base.ks",
1644 'distro': 'Fedora-23',
1645 # only build this type of image on x86_64
1647 'arches': ['x86_64']
1648 # Use install tree and repo from Everything variant.
1650 'install_tree_from': 'Everything',
1651 'repo': ['Everything'],
1652 # Set release automatically.
1654 'release': '!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN',
1655 }
1656 }
1657 ]
1658 }
1659 OSTree Settings
1661 The ostree phase of Pungi can create and update ostree repositories.
1662 This is done by running rpm-ostree compose in a Koji runroot environ‐
1663 ment. The ostree repository itself is not part of the compose and
1664 should be located in another directory. Any new packages in the compose
1665 will be added to the repository with a new commit.
1666 ostree (dict) – a mapping of configuration for each. The format should
1668 be {variant_uid_regex: config_dict}. It is possible to use a
1669 list of configuration dicts as well.
1670 The configuration dict for each variant arch pair must have
1672 these keys:
1673 · treefile – (str) Filename of configuration for rpm-ostree.
1675 · config_url – (str) URL for Git repository with the treefile.
1677 · repo – (str|dict|[str|dict]) repos specified by URL or variant
1679 UID or a dict of repo options, baseurl is required in the
1680 dict.
1681 · ostree_repo – (str) Where to put the ostree repository
1683 These keys are optional:
1685 · keep_original_sources – (bool) Keep the existing source repos
1687 in the tree config file. If not enabled, all the original
1688 source repos will be removed from the tree config file.
1689 · config_branch – (str) Git branch of the repo to use. Defaults
1691 to master.
1692 · arches – ([str]) List of architectures for which to update
1694 ostree. There will be one task per architecture. By default
1695 all architectures in the variant are used.
1696 · failable – ([str]) List of architectures for which this deliv‐
1698 erable is not release blocking.
1699 · update_summary – (bool) Update summary metadata after tree
1701 composing. Defaults to False.
1702 · force_new_commit – (bool) Do not use rpm-ostree’s built-in
1704 change detection. Defaults to False.
1705 · version – (str) Version string to be added as versioning meta‐
1707 data. If this option is set to !OSTREE_VER‐
1708 SION_FROM_LABEL_DATE_TYPE_RESPIN, a value will be generated
1709 automatically as $VERSION.$RELEASE. If this option is set to
1710 !VERSION_FROM_VERSION_DATE_RESPIN, a value will be generated
1711 automatically as $VERSION.$DATE.$RESPIN. See how those values
1712 are created.
1713 · tag_ref – (bool, default True) If set to False, a git refer‐
1715 ence will not be created.
1716 · ostree_ref – (str) To override value ref from treefile.
1718 Example config
1720 ostree = {
1721 "^Atomic$": {
1722 "treefile": "fedora-atomic-docker-host.json",
1723 "config_url": "https://git.fedorahosted.org/git/fedora-atomic.git",
1724 "repo": [
1725 "Server",
1726 "http://example.com/repo/x86_64/os",
1727 {"baseurl": "Everything"},
1728 {"baseurl": "http://example.com/linux/repo", "exclude": "systemd-container"},
1729 ],
1730 "keep_original_sources": True,
1731 "ostree_repo": "/mnt/koji/compose/atomic/Rawhide/",
1732 "update_summary": True,
1733 # Automatically generate a reasonable version
1734 "version": "!OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN",
1735 # Only run this for x86_64 even if Atomic has more arches
1736 "arches": ["x86_64"],
1737 }
1738 }
1739 Ostree Installer Settings
1741 The ostree_installer phase of Pungi can produce installer image
1742 bundling an OSTree repository. This always runs in Koji as a runroot
1743 task.
1744 ostree_installer
1746 (dict) – a variant/arch mapping of configuration. The format
1747 should be [(variant_uid_regex, {arch|*: config_dict})].
1748 The configuration dict for each variant arch pair must have this
1750 key:
1751 These keys are optional:
1753 · repo – (str|[str]) repos specified by URL or variant UID
1755 · release – (str) Release value to set for the installer image.
1757 Set to !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN to generate the
1758 value automatically.
1759 · failable – ([str]) List of architectures for which this deliv‐
1761 erable is not release blocking.
1762 These optional keys are passed to lorax to customize the build.
1764 · installpkgs – ([str])
1766 · add_template – ([str])
1768 · add_arch_template – ([str])
1770 · add_template_var – ([str])
1772 · add_arch_template_var – ([str])
1774 · rootfs_size – ([str])
1776 · template_repo – (str) Git repository with extra templates.
1778 · template_branch – (str) Branch to use from template_repo.
1780 The templates can either be absolute paths, in which case they
1782 will be used as configured; or they can be relative paths, in
1783 which case template_repo needs to point to a Git repository from
1784 which to take the templates.
1785 ostree_installer_overwrite = False
1787 (bool) – by default if a variant including OSTree installer also
1788 creates regular installer images in buildinstall phase, there
1789 will be conflicts (as the files are put in the same place) and
1790 Pungi will report an error and fail the compose.
1791 With this option it is possible to opt-in for the overwriting.
1793 The traditional boot.iso will be in the iso/ subdirectory.
1794 Example config
1796 ostree_installer = [
1797 ("^Atomic$", {
1798 "x86_64": {
1799 "repo": [
1800 "Everything",
1801 "https://example.com/extra-repo1.repo",
1802 "https://example.com/extra-repo2.repo",
1803 ],
1804 "release": "!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN",
1805 "installpkgs": ["fedora-productimg-atomic"],
1806 "add_template": ["atomic-installer/lorax-configure-repo.tmpl"],
1807 "add_template_var": [
1808 "ostree_osname=fedora-atomic",
1809 "ostree_ref=fedora-atomic/Rawhide/x86_64/docker-host",
1810 ],
1811 "add_arch_template": ["atomic-installer/lorax-embed-repo.tmpl"],
1812 "add_arch_template_var": [
1813 "ostree_repo=https://kojipkgs.fedoraproject.org/compose/atomic/Rawhide/",
1814 "ostree_osname=fedora-atomic",
1815 "ostree_ref=fedora-atomic/Rawhide/x86_64/docker-host",
1816 ]
1817 'template_repo': 'https://git.fedorahosted.org/git/spin-kickstarts.git',
1818 'template_branch': 'f24',
1819 }
1820 })
1821 ]
1822 OSBS Settings
1824 Pungi can build container images in OSBS. The build is initiated
1825 through Koji container-build plugin. The base image will be using RPMs
1826 from the current compose and a Dockerfile from specified Git reposi‐
1827 tory.
1828 Please note that the image is uploaded to a registry and not exported
1830 into compose directory. There will be a metadata file in compose/meta‐
1831 data/osbs.json with details about the built images (assuming they are
1832 not scratch builds).
1833 osbs (dict) – a mapping from variant regexes to configuration blocks.
1835 The format should be {variant_uid_regex: [config_dict]}.
1836 The configuration for each image must have at least these keys:
1838 · url – (str) URL pointing to a Git repository with Dockerfile.
1840 Please see Git URLs section for more details.
1841 · target – (str) A Koji target to build the image for.
1843 · git_branch – (str) A branch in SCM for the Dockerfile. This is
1845 required by OSBS to avoid race conditions when multiple builds
1846 from the same repo are submitted at the same time. Please note
1847 that url should contain the branch or tag name as well, so
1848 that it can be resolved to a particular commit hash.
1849 Optionally you can specify failable. If it has a truthy value,
1851 failure to create the image will not abort the whole compose.
1852 NOTE:
1854 Once OSBS gains support for multiple architectures, the usage
1855 of this option will most likely change to list architectures
1856 that are allowed to fail.
1857 The configuration will pass other attributes directly to the
1859 Koji task. This includes name, version, scratch and priority.
1860 A value for yum_repourls will be created automatically and point
1862 at a repository in the current compose. You can add extra repos‐
1863 itories with repo key having a list of urls pointing to .repo
1864 files or just variant uid, Pungi will create the .repo file for
1865 that variant. gpgkey can be specified to enable gpgcheck in repo
1866 files for variants.
1867 osbs_registries
1869 (dict) – It is possible to configure extra information about
1870 where to push the image (unless it is a scratch build). For each
1871 finished build, Pungi will try to match NVR against a key in
1872 this mapping (using shell-style globbing) and take the corre‐
1873 sponding value and collect them across all built images. The
1874 data will be saved into logs/global/osbs-registries.json as a
1875 mapping from Koji NVR to the registry data. The same data is
1876 also sent to the message bus on osbs-request-push topic once the
1877 compose finishes successfully. Handling the message and perform‐
1878 ing the actual push is outside of scope for Pungi.
1879 Example config
1881 osbs = {
1882 "^Server$": {
1883 # required
1884 "url": "git://example.com/dockerfiles.git?#HEAD",
1885 "target": "f24-docker-candidate",
1886 "git_branch": "f24-docker",
1887 # optional
1889 "name": "fedora-docker-base",
1890 "version": "24",
1891 "repo": ["Everything", "https://example.com/extra-repo.repo"],
1892 # This will result in three repo urls being passed to the task.
1893 # They will be in this order: Server, Everything, example.com/
1894 "gpgkey": 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release',
1895 }
1896 }
1897 Extra ISOs
1899 Create an ISO image that contains packages from multiple variants. Such
1900 ISO always belongs to one variant, and will be stored in ISO directory
1901 of that variant.
1902 The ISO will be bootable if buildinstall phase runs for the parent
1904 variant. It will reuse boot configuration from that variant.
1905 extra_isos
1907 (dict) – a mapping from variant UID regex to a list of configu‐
1908 ration blocks.
1909 · include_variants – (list) list of variant UIDs from which con‐
1911 tent should be added to the ISO; the variant of this image is
1912 added automatically.
1913 Rest of configuration keys is optional.
1915 · filename – (str) template for naming the image. In addition to
1917 the regular placeholders filename is available with the name
1918 generated using image_name_format option.
1919 · volid – (str) template for generating volume ID. Again volid
1921 placeholder can be used similarly as for file name. This can
1922 also be a list of templates that will be tried sequentially
1923 until one generates a volume ID that fits into 32 character
1924 limit.
1925 · extra_files – (list) a list of scm_dict objects. These files
1927 will be put in the top level directory of the image.
1928 · arches – (list) a list of architectures for which to build
1930 this image. By default all arches from the variant will be
1931 used. This option can be used to limit them.
1932 · failable_arches – (list) a list of architectures for which the
1934 image can fail to be generated and not fail the entire com‐
1935 pose.
1936 · skip_src – (bool) allows to disable creating an image with
1938 source packages.
1939 · inherit_extra_files – (bool) by default extra files in vari‐
1941 ants are ignored. If you want to include them in the ISO, set
1942 this option to True.
1943 · max_size – (int) expected maximum size in bytes. If the final
1945 image is larger, a warning will be issued.
1946 Example config
1948 extra_isos = {
1949 'Server': [{
1950 # Will generate foo-DP-1.0-20180510.t.43-Server-x86_64-dvd1.iso
1951 'filename': 'foo-{filename}',
1952 'volid': 'foo-{arch}',
1953 'extra_files': [{
1955 'scm': 'git',
1956 'repo': 'https://pagure.io/pungi.git',
1957 'file': 'setup.py'
1958 }],
1959 'include_variants': ['Client']
1961 }]
1962 }
1963 # This should create image with the following layout:
1964 # .
1965 # ├── Client
1966 # │ ├── Packages
1967 # │ │ ├── a
1968 # │ │ └── b
1969 # │ └── repodata
1970 # ├── Server
1971 # │ ├── Packages
1972 # │ │ ├── a
1973 # │ │ └── b
1974 # │ └── repodata
1975 # └── setup.py
1976 Media Checksums Settings
1978 media_checksums
1979 (list) – list of checksum types to compute, allowed values are
1980 anything supported by Python’s hashlib module (see documentation
1981 for details).
1982 media_checksum_one_file
1984 (bool) – when True, only one CHECKSUM file will be created per
1985 directory; this option requires media_checksums to only specify
1986 one type
1987 media_checksum_base_filename
1989 (str) – when not set, all checksums will be save to a file named
1990 either CHECKSUM or based on the digest type; this option allows
1991 adding any prefix to that name
1992 It is possible to use format strings that will be replace by
1994 actual values. The allowed keys are:
1995 · arch
1997 · compose_id
1999 · date
2001 · label
2003 · label_major_version
2005 · release_short
2007 · respin
2009 · type
2011 · type_suffix
2013 · version
2015 · dirname (only if media_checksum_one_file is enabled)
2017 For example, for Fedora the prefix should be
2019 %(release_short)s-%(variant)s-%(version)s-%(date)s%(type_suf‐
2020 fix)s.%(respin)s.
2021 Translate Paths Settings
2023 translate_paths
2024 (list) – list of paths to translate; format: [(path, trans‐
2025 lated_path)]
2026 NOTE:
2028 This feature becomes useful when you need to transform compose loca‐
2029 tion into e.g. a HTTP repo which is can be passed to koji
2030 image-build. The path part is normalized via os.path.normpath().
2031 Example config
2033 translate_paths = [
2034 ("/mnt/a", "http://b/dir"),
2035 ]
2036 Example usage
2038 >>> from pungi.util import translate_paths
2039 >>> print translate_paths(compose_object_with_mapping, "/mnt/a/c/somefile")
2040 http://b/dir/c/somefile
2041 Miscellaneous Settings
2043 paths_module
2044 (str) – Name of Python module implementing the same interface as
2045 pungi.paths. This module can be used to override where things
2046 are placed.
2047 link_type = hardlink-or-copy
2049 (str) – Method of putting packages into compose directory.
2050 Available options:
2052 · hardlink-or-copy
2054 · hardlink
2056 · copy
2058 · symlink
2060 · abspath-symlink
2062 skip_phases
2064 (list) – List of phase names that should be skipped. The same
2065 functionality is available via a command line option.
2066 release_discinfo_description
2068 (str) – Override description in .discinfo files. The value is a
2069 format string accepting %(variant_name)s and %(arch)s placehold‐
2070 ers.
2071 symlink_isos_to
2073 (str) – If set, the ISO files from buildinstall, createiso and
2074 live_images phases will be put into this destination, and a sym‐
2075 link pointing to this location will be created in actual compose
2076 directory.
2077 dogpile_cache_backend
2079 (str) – If set, Pungi will use the configured Dogpile cache
2080 backend to cache various data between multiple Pungi calls. This
2081 can make Pungi faster in case more similar composes are running
2082 regularly in short time.
2083 For list of available backends, please see the
2085 https://dogpilecache.readthedocs.io documentation.
2086 Most typical configuration uses the dogpile.cache.dbm backend.
2088 dogpile_cache_arguments
2090 (dict) – Arguments to be used when creating the Dogpile cache
2091 backend. See the particular backend’s configuration for the
2092 list of possible key/value pairs.
2093 For the dogpile.cache.dbm backend, the value can be for example
2095 following:
2096 {
2098 "filename": "/tmp/pungi_cache_file.dbm"
2099 }
2100 dogpile_cache_expiration_time
2102 (int) – Defines the default expiration time in seconds of data
2103 stored in the Dogpile cache. Defaults to 3600 seconds.
2104
2106 Multiple places in Pungi can use files from external storage. The con‐
2107 figuration is similar independently of the backend that is used,
2108 although some features may be different.
2109 The so-called scm_dict is always put into configuration as a dictio‐
2111 nary, which can contain following keys.
2112 · scm – indicates which SCM system is used. This is always required.
2114 Allowed values are:
2115 · file – copies files from local filesystem
2117 · git – copies files from a Git repository
2119 · cvs – copies files from a CVS repository
2121 · rpm – copies files from a package in the compose
2123 · repo – for Git and CVS backends URL to the repository, for RPM a
2125 shell glob for matching package names (or a list of such globs); for
2126 file backend this option should be empty (or left out)
2127 · branch – branch name for Git and CVS backends, with master and HEAD
2129 as defaults. Ignored for other backends.
2130 · file – a list of files that should be exported.
2132 · dir – a directory that should be exported. All its contents will be
2134 exported. This option is mutually exclusive with file.
2135 · command – defines a shell command to run after Git clone to generate
2137 the needed file (for example to run make). Only supported in Git
2138 backend.
2139 file vs. dir
2141 Exactly one of these two options has to be specified. Documentation for
2142 each configuration option should specify whether it expects a file or a
2143 directory.
2144 For extra_files phase either key is valid and should be chosen depend‐
2146 ing on what the actual use case.
2147 Caveats
2149 The rpm backend can only be used in phases that would extract the files
2150 after pkgset phase finished. You can’t get comps file from a package.
2151 Depending on Git repository URL configuration Pungi can only export the
2153 requested content using git archive. When a command should run this is
2154 not possible and a clone is always needed.
2155
2157 Pungi has the ability to emit notification messages about progress and
2158 general status of the compose. These can be used to e.g. send messages
2159 to fedmsg. This is implemented by actually calling a separate script.
2160 The script will be called with one argument describing action that just
2162 happened. A JSON-encoded object will be passed to standard input to
2163 provide more information about the event. At the very least, the object
2164 will contain a compose_id key.
2165 The script is invoked in compose directory and can read other informa‐
2167 tion there.
2168 Currently these messages are sent:
2170 · status-change – when composing starts, finishes or fails; a status
2172 key is provided to indicate details
2173 · phase-start – on start of a phase
2175 · phase-stop – when phase is finished
2177 · createiso-targets – with a list of images to be created
2179 · createiso-imagedone – when any single image is finished
2181 · createiso-imagefail – when any single image fails to create
2183 · fail-to-start – when there are incorrect CLI options or errors in
2185 configuration file; this message does not contain compose_id nor
2186 is it started in the compose directory (which does not exist yet)
2187 · ostree – when a new commit is created, this message will announce
2189 its hash and the name of ref it is meant for.
2190 For phase related messages phase_name key is provided as well.
2192 A pungi-fedmsg-notification script is provided and understands this
2194 interface.
2195 Setting it up
2197 The script should be provided as a command line argument --notifica‐
2198 tion-script.
2199 --notification-script=pungi-fedmsg-notification
2201
2203 A compose created by Pungi consists of one or more variants. A variant
2204 contains a subset of the content targeted at a particular use case.
2205 There are different types of variants. The type affects how packages
2207 are gathered into the variant.
2208 The inputs for gathering are defined by various gather sources. Pack‐
2210 ages from all sources are collected to create a big list of package
2211 names, comps groups names and a list of packages that should be fil‐
2212 tered out.
2213 NOTE:
2215 The inputs for both explicit package list and comps file are inter‐
2216 preted as RPM names, not any arbitrary provides nor source package
2217 name.
2218 Next, gather_method defines how the list is processed. For nodeps, the
2220 results from source are used pretty much as is [1]. For deps method, a
2221 process will be launched to figure out what dependencies are needed and
2222 those will be pulled in.
2223 [1] The lists are filtered based on what packages are available in the
2225 package set, but nothing else will be pulled in.
2226 Variant types
2228 Variant
2229 is a base type that has no special behaviour.
2230 Addon is built on top of a regular variant. Any packages that should
2232 go to both the addon and its parent will be removed from addon.
2233 Packages that are only in addon but pulled in because of
2234 gather_fulltree option will be moved to parent.
2235 Integrated Layered Product
2237 works similarly to addon. Additionally, all packages from addons
2238 on the same parent variant are removed integrated layered prod‐
2239 ucts.
2240 The main difference between an addon and integrated layered
2242 product is that integrated layered product has its own identity
2243 in the metadata (defined with product name and version).
2244 NOTE:
2246 There’s also Layered Product as a term, but this is not
2247 related to variants. It’s used to describe a product that is
2248 not a standalone operating system and is instead meant to be
2249 used on some other base system.
2250 Optional
2252 contains packages that complete the base variants’ package set.
2253 It always has fulltree and selfhosting enabled, so it contains
2254 build dependencies and packages which were not specifically
2255 requested for base variant.
2256 Some configuration options are overridden for particular variant types.
2258 Depsolving configuration
2260 ┌──────────┬──────────────┬──────────────┐
2261 │Variant │ Fulltree │ Selfhosting │
2262 ├──────────┼──────────────┼──────────────┤
2263 │base │ configurable │ configurable │
2264 ├──────────┼──────────────┼──────────────┤
2265 │addon/ILP │ enabled │ disabled │
2266 ├──────────┼──────────────┼──────────────┤
2267 │optional │ enabled │ enabled │
2268 └──────────┴──────────────┴──────────────┘
2269 Profiling
2271 Profiling data on the pungi-gather tool can be enabled by setting the
2272 gather_profiler configuration option to True.
2273 Modular compose
2275 A compose with gather_source set to module is called modular. The pack‐
2276 age list is determined by a list of modules.
2277 The list of modules that will be put into a variant is defined in the
2279 variants.xml file. The file can contain either Name:Stream or
2280 Name:Stream:Version references. See Module Naming Policy for details.
2281 When Version is missing from the specification, Pungi will ask PDC for
2282 the latest one.
2283 The module metadata in PDC contains a list of RPMs in the module as
2285 well as Koji tag from which the packages can be retrieved.
2286 Restrictions
2288 · A modular compose must always use Koji as a package set source.
2289
2291 The comps file that Pungi takes as input is not really pure comps as
2292 used by tools like DNF. There are extensions used to customize how the
2293 file is processed.
2294 The first step of Pungi processing is to retrieve the actual file. This
2296 can use anything that scm_support supports.
2297 Pungi extensions are arch attribute on packageref, group and environ‐
2299 ment tags. The value of this attribute is a comma separated list of
2300 architectures.
2301 Second step Pungi performs is creating a file for each architecture.
2303 This is done by removing all elements with incompatible arch attribute.
2304 No additional clean up is performed on this file. The resulting file is
2305 only used internally for the rest of the compose process.
2306 Third and final step is to create comps file for each Variant.Arch com‐
2308 bination. This is the actual file that will be included in the com‐
2309 pose. The start file is the original input file, from which all ele‐
2310 ments with incompatible architecture are removed. Then clean up is per‐
2311 formed by removing all empty groups, removing non-existing groups from
2312 environments and categories and finally removing empty environments and
2313 categories. As a last step groups not listed in the variants file are
2314 removed.
2315
2317 Set up development environment
2318 In order to work on Pungi, you should install recent version of Fedora.
2319 These packages will have to installed:
2320 · createrepo_c
2322 · cvs
2324 · gcc
2326 · genisoimage
2328 · gettext
2330 · git
2332 · gobject-introspection
2334 · isomd5sum
2336 · jigdo
2338 · kobo
2340 · krb5-devel
2342 · libcurl-devel
2344 · libmodulemd
2346 · libselinux-python
2348 · lorax
2350 · python-dogpile-cache
2352 · python-jsonschema
2354 · python-kickstart
2356 · python-libcomps
2358 · python-lockfile
2360 · python-lxml
2362 · python2-multilib
2364 · python-productmd
2366 · PyYAML
2368 · repoview
2370 · rpm-devel
2372 · syslinux
2374 · yum
2376 · yum-utils
2378 For running unit tests, these packages are recommended as well:
2380 · python-mock
2382 · python-nose
2384 · python-nose-cov
2386 · python-unittest2
2388 · rpmdevtools
2390 · python-parameterized
2392 While being difficult, it is possible to work on Pungi using vir‐
2394 tualenv. Install python-virtualenvwrapper (after installation you have
2395 to add the command to source /usr/local/bin/virtualenvwrapper.sh to
2396 your shell startup file, depending on where it was installed by package
2397 manager) and use following steps. It will link system libraries into
2398 the virtual environment and install all packages preferably from PyPI
2399 or from tarball. You will still need to install all of the non-Python
2400 packages above as they are used by calling an executable.
2401 $ mkvirtualenv pungienv
2403 $ for pkg in gi libcomps pykickstart rpmUtils selinux urlgrabber yum; do ln -vs "$(deactivate && python -c 'import os, '$pkg'; print(os.path.dirname('$pkg'.__file__))')" "$(virtualenvwrapper_get_site_packages_dir)"; done
2404 $ for pkg in _deltarpm krbV _selinux deltarpm sqlitecachec _sqlitecache; do ln -vs "$(deactivate && python -c 'import os, '$pkg'; print('$pkg'.__file__)')" "$(virtualenvwrapper_get_site_packages_dir)"; done
2405 $ pip install -U pip
2406 $ PYCURL_SSL_LIBRARY=nss pip install pycurl --no-binary :all:
2407 $ pip install beanbag jsonschema 'kobo>=0.6.0' lockfile lxml mock nose nose-cov productmd pyopenssl python-multilib requests requests-kerberos setuptools sphinx ordered_set koji PyYAML dogpile.cache parameterized
2408 Now you should be able to run all existing tests.
2410 Developing
2412 Currently the development workflow for Pungi is on master branch:
2413 · Make your own fork at https://pagure.io/pungi
2415 · Clone your fork locally (replacing $USERNAME with your own):
2417 git clone git@pagure.io:forks/$USERNAME/pungi.git
2419 · cd into your local clone and add the remote upstream for rebasing:
2421 cd pungi
2423 git remote add upstream git@pagure.io:pungi.git
2424 NOTE:
2426 This workflow assumes that you never git commit directly to the
2427 master branch of your fork. This will make more sense when we
2428 cover rebasing below.
2429 · create a topic branch based on master:
2431 git branch my_topic_branch master
2433 git checkout my_topic_branch
2434 · Make edits, changes, add new features, etc. and then make sure to
2436 pull from upstream master and rebase before submitting a pull
2437 request:
2438 # lets just say you edited setup.py for sake of argument
2440 git checkout my_topic_branch
2441 # make changes to setup.py
2443 git add setup.py
2444 git commit -s -m "added awesome feature to setup.py"
2445 # now we rebase
2447 git checkout master
2448 git pull --rebase upstream master
2449 git push origin master
2450 git push origin --tags
2451 git checkout my_topic_branch
2452 git rebase master
2453 # resolve merge conflicts if any as a result of your development in
2455 # your topic branch
2456 git push origin my_topic_branch
2457 NOTE:
2459 In order to for your commit to be merged, you must sign-off on it.
2460 Use -s option when running git commit.
2461 · Create pull request in the pagure.io web UI
2463 · For convenience, here is a bash shell function that can be placed in
2465 your ~/.bashrc and called such as pullupstream pungi-4-devel that
2466 will automate a large portion of the rebase steps from above:
2467 pullupstream () {
2469 if [[ -z "$1" ]]; then
2470 printf "Error: must specify a branch name (e.g. - master, devel)\n"
2471 else
2472 pullup_startbranch=$(git describe --contains --all HEAD)
2473 git checkout $1
2474 git pull --rebase upstream master
2475 git push origin $1
2476 git push origin --tags
2477 git checkout ${pullup_startbranch}
2478 fi
2479 }
2480 Testing
2482 You must write unit tests for any new code (except for trivial
2483 changes). Any code without sufficient test coverage may not be merged.
2484 To run all existing tests, suggested method is to use nosetests. With
2486 additional options, it can generate code coverage. To make sure even
2487 tests from executable files are run, don’t forget to use the --exe
2488 option.
2489 $ make test
2491 $ make test-cover
2492 # Running single test file
2494 $ python tests/test_arch.py [TestCase...]
2495 In the tests/ directory there is a shell script test_compose.sh that
2497 you can use to try and create a miniature compose on dummy data. The
2498 actual data will be created by running make test-data in project root.
2499 $ make test-data
2501 $ make test-compose
2502 This testing compose does not actually use all phases that are avail‐
2504 able, and there is no checking that the result is correct. It only
2505 tells you whether it crashed or not.
2506 NOTE:
2508 Even when it finishes successfully, it may print errors about
2509 repoclosure on Server-Gluster.x86_64 in test phase. This is not a
2510 bug.
2511 Documenting
2513 You must write documentation for any new features and functional
2514 changes. Any code without sufficient documentation may not be merged.
2515 To generate the documentation, run make doc in project root.
2517
2519 Test Data
2520 Tests require test data and not all of it is available in git. You
2521 must create test repositories before running the tests:
2522 make test-data
2524 Requirements: createrepo_c, rpmbuild
2526 Unit Tests
2528 Unit tests cover functionality of Pungi python modules. You can run
2529 all of them at once:
2530 make test
2532 which is shortcut to:
2534 python2 setup.py test
2536 python3 setup.py test
2537 You can alternatively run individual tests:
2539 cd tests
2541 ./<test>.py [<class>[.<test>]]
2542 Functional Tests
2544 Because compose is quite complex process and not everything is covered
2545 with unit tests yet, the easiest way how to test if your changes did
2546 not break anything badly is to start a compose on a relatively small
2547 and well defined package set:
2548 cd tests
2550 ./test_compose.sh
2551
2553 There may be cases where it makes sense to split a big compose into
2554 separate parts, but create a compose output that links all output into
2555 one familiar structure.
2556 The pungi-orchestrate tools allows that.
2558 It works with an INI-style configuration file. The [general] section
2560 contains information about identity of the main compose. Other sections
2561 define individual parts.
2562 The parts are scheduled to run in parallel, with the minimal amount of
2564 serialization. The final compose directory will contain hard-links to
2565 the files.
2566 General settings
2568 target Path to directory where the final compose should be created.
2569 compose_type
2571 Type of compose to make.
2572 release_name
2574 Name of the product for the final compose.
2575 release_short
2577 Short name of the product for the final compose.
2578 release_version
2580 Version of the product for the final compose.
2581 release_type
2583 Type of the product for the final compose.
2584 extra_args
2586 Additional arguments that wil be passed to the child Pungi pro‐
2587 cesses.
2588 koji_profile
2590 If specified, a current event will be retrieved from the Koji
2591 instance and used for all parts.
2592 kerberos
2594 If set to yes, a kerberos ticket will be automatically created
2595 at the start. Set keytab and principal as well.
2596 kerberos_keytab
2598 Path to keytab file used to create the kerberos ticket.
2599 kerberos_principal
2601 Kerberos principal for the ticket
2602 pre_compose_script
2604 Commands to execute before first part is started. Can contain
2605 multiple commands on separate lines.
2606 post_compose_script
2608 Commands to execute after the last part finishes and final sta‐
2609 tus is updated. Can contain multiple commands on separate lines.
2610 post_compose_script =
2612 compose-latest-symlink $COMPOSE_PATH
2613 custom-post-compose-script.sh
2614 Multiple environment variables are defined for the scripts:
2616 · COMPOSE_PATH
2618 · COMPOSE_ID
2620 · COMPOSE_DATE
2622 · COMPOSE_TYPE
2624 · COMPOSE_RESPIN
2626 · COMPOSE_LABEL
2628 · RELEASE_ID
2630 · RELEASE_NAME
2632 · RELEASE_SHORT
2634 · RELEASE_VERSION
2636 · RELEASE_TYPE
2638 · RELEASE_IS_LAYERED – YES for layered products, empty other‐
2640 wise
2641 · BASE_PRODUCT_NAME – only set for layered products
2643 · BASE_PRODUCT_SHORT – only set for layered products
2645 · BASE_PRODUCT_VERSION – only set for layered products
2647 · BASE_PRODUCT_TYPE – only set for layered products
2649 notification_script
2651 Executable name (or path to a script) that will be used to send
2652 a message once the compose is finished. In order for a valid URL
2653 to be included in the message, at least one part must configure
2654 path translation that would apply to location of main compose.
2655 Only two messages will be sent, one for start and one for finish
2657 (either successful or not).
2658 Partial compose settings
2660 Each part should have a separate section in the config file.
2661 It can specify these options:
2663 config Path to configuration file that describes this part. If rela‐
2665 tive, it is resolved relative to the file with parts configura‐
2666 tion.
2667 just_phase, skip_phase
2669 Customize which phases should run for this part.
2670 depends_on
2672 A comma separated list of other parts that must be finished
2673 before this part starts.
2674 failable
2676 A boolean toggle to mark a part as failable. A failure in such
2677 part will mark the final compose as incomplete, but still suc‐
2678 cessful.
2679
2681 Daniel Mach
2682
2684 2016, Red Hat, Inc.
26854.1 May 27, 2019 PUNGI(1)