Standard-CI container-based backend
The Standard-CI container-based backend is an extension to the CI Build and Test Standards, that allows having CI workloads run natively as containers using user-specified container images.
This backend is provided as an alternative to the legacy VM-and-chroot-based backend, that had been used so far in the STDCI implementation.
The container-based backend uses a somewhat different configuration syntax than the legacy backend, and provides a somewhat different set of features.
Using the container-based backend
The container-based backend is enabled by including the container
options in
the STDCI YAML configuration file. Following is an example for how to do
so:
sub-stages:
- run-on-fc31-container:
containers:
# The centos7 image is used for pulling source code
- image: centos/s2i-base-centos7
args:
- bash
- -exc
- |
git init .
git fetch "$STD_CI_CLONE_URL" +refs/heads/*:refs/remotes/origin/*
git fetch "$STD_CI_CLONE_URL" +"$STD_CI_REFSPEC":myhead
git checkout myhead
# Actual tests run in an fc31 image
- image: docker.io/fedora:31
As is typical for standard-CI YAML options, the option can be provided at the
level of a stage, a substage, a distro, an arch, or a combination
thereof. It can also be specified at a higher level of the file and be
inherited by lower levels. The plural form containers
can also be used for the
option name.
The contents of the container
option is a list of container entries that can
contain the fields specified below.
As a convenience for the case where there is only one entry in the container
list, the container
option may also contain the container entry structure
directly, without having it be wrapped by a list.
The following fields may be included in a container entry:
Field | Optional | Default value | Meaning |
---|---|---|---|
image | no | N/A | The container image to run |
args | yes | CI script name | Arguments to pass to the image |
command | yes | Image entrypoint | Override the image entry point |
workingdir | yes | /workspace | Set the working directory |
securityContxt | yes | empty map | Security settings for the container |
Please note, that the container fields are roughly equivalent to similar fields specified for a container configuration in Kubernetes.
When specifying the container
option the STDCI system will launch all the
containers specified in the list one after another in the order they are
specified. If any container returns a nonzero result, the execution will stop
and an error will be reported. The containers will all run in the same host (Or
POD in Kubernetes).
Note: CI threads that use containers, may be defined in the same configuration file, alongside threads that used the legacy runtime.
The following sections describe in more detail how the containers behave, and the relationships between the different configuration options.
Scripts, arguments and entry points
Typically, when running a container in a container runtime such as Docker or
Kubernetes, the runtime allows passing arguments to the container, which are
then passed as arguments to its entry point. Passing those arguments is done via
the args
option.
Is some cases, it may be desirable to override the entry point defined in the
container image. This is done via the command
option.
If the args
option is not given, the name of the script associated with the CI
thread being run will be passed instead.
Note: There is no guarantee that the script itself would be available for use by the container (See discussion about source code below).
While the CI script associated with the CI thread being run, may not be used by the running container at all (As is the case in the example above), it is still required that the script file will exist. If the file does not exist then:
- If the file is explicitly defined via the
script
option in the YAML, the system will raise an error - If
script
option is not explicitly defined, the whole CI thread will be skipped, and no container would be launched
Source code
The system does not by default make the project source code available to the container. It is up to container image authors and users to make the image obtain the right source code.
The system does have the built-in ability to clone the project source code into
the /workspace
directory prior to launching the user-specified containers.
This is done via the decorate-containers
option.
The decorate-containers
option
The decorate-containers
option is a stand-alone option that can be placed in
the Standard-CI YAML configuration file similarly to the containers
option.
The options value is a boolean that defaults to False
. When it is set to
True
it makes the system automatically perform the following additional
functionality when running containers.
- The project source code is cloned into
/workspace
before running the requested containers. - A Shared volume is mounted on
/exported-artifacts
, and any files available there when the build is done are collected and made available as build artifacts. - All GPG-encrypted files in the source code repository are decrypted (if possible). See Using secret data below for details.
Following is an example for using the decorate-containers
option:
sub-stages:
- run-on-fc31-container:
container: docker.io/fedora:31
decorate-container: True
Note: In the example above, since only one container is specified and only the image option is set, the container configuration could be shorthanded to a single string. See the 'Tricks aliases, and shorthands' section below for explanation about which shorthand syntaxes are available.
For convenience, the decorate-containers
option may be shorthanded to
decorate
.
Working directory and /workspace
When the container is launched a temporary volume is mounted at /workspace
.
The directory is intended to be used as the main workspace for the container.
When several containers are defined, they may use it as a place to exchange
information.
By default, the /workspace
directory is set as the working directory for the
container. This may be changed using the workingdir
option.
Build artifact and log collection
After the CI containers are done running, any STDOUT or STDERR output they have
generated is collected and stored in log files that are made available in the
pod_logs
directory as part of the job results.
Note: When the CI system runs containers in a POD, the output of the main container, which is the last container in the configured list of containers, is shown in the Jenkins job output and in Blue Ocean. When more containers are configured, their output is not displayed due to an unfortunate limitation of the OpenShift version that is in use in the CI system. The output is available however, in the log files described above.
As noted above, when the decorate-containers
options is is set to True
, an
extra storage volume is mounted on the container at /exported-arficats
. The
build or test script running in the container may leave files there, and those
files would be made available as part of the job results when the build is done.
If files with the *.xml
extension are placed in /exported-artifacts
, those
files would, in addition to being made available, would also be parsed as JUnit
test result XML files, as data available in them would be made available in the
Jenkins UI.
Distribution and Architecture
In the context of Standard-CI, the target distribution and architecture for a given CI thread are always defined - either explicitly, or implicitly via default values.
Those definitions, while being provided as environment variables (see below), are otherwise ignored by the container backend. Containers are being launched on x86_64 machines, and the distribution available is whatever is provided by the container image.
Future extensions may make the containers run on the appropriate architectures, and provide automated image selection according to the target distribution.
Environment variables
To enable the containers to interact with the rest of the CI system, and with the project being built and tested, the following environment variables are provided:
Variable | Meaning |
---|---|
STD_CI_STAGE | The CI stage the container was launched for |
STD_CI_SUBSTAGE | The CI sub-stage the container was launched for |
STD_CI_DISTRO | The CI target distribution the container was launched for |
STD_CI_ARCH | The CI target architecture the container was launched for |
STD_CI_CLONE_URL | The Git repo URL project code can be cloned from |
STD_CI_REFSPEC | The Git refspec of the source being built or tested |
STD_CI_PROJECT | The name of the project being built or tested |
STD_CI_GIT_SHA | The Git SHA of the source being tested |
GIT_COMMITTER_NAME | The CI system's user name for placing in auto generated Git commits |
GIT_COMMITTER_EMAIL | The CI system's email address for placing in auto generated Git commits |
BUILD_NUMBER | The build number from Jenkins |
BUILD_ID | The build ID from Jenkins |
BUILD_DISPLAY_NAME | The display name for the build in Jenkins |
BUILD_TAG | A unique string identifying the build |
BUILD_URL | A URL for the running build in Jenkins |
JOB_NAME | The full name of the running job in Jenkins |
JOB_BASE_NAME | The short name of the running job in Jenkins |
JOB_URL | The URL for the running job in Jenkins |
JENKINS_URL | The URL for the Jenkins master the job is running on |
Running privileged containers and using securityContext
By default, the CI system runs containers with a very limited set of privileges. The containers themselves are launched in unprivileged mode that forbids them from carrying out most system actions, and the test scripts that are launched within them are launched using an unprivileged user account with a randomly selected UID.
Some container images require running certain processes with elevated
privileges. For example, containers that use systemd
to launch background
processes within them require that systemd
be launched as root
. As another
example, some tests may require launching VMs within containers, doing that
requires that the container would be launched in privileged mode.
To enable this kinds of use cases, the CI system allows for the use of the
securityContext
property, to override the lower privileges the system sets by
default. To maintain system security, however, there are several limitations to
the use of this property:
- The CI team maintains a white-list of approved privileged container images.
Using the
securityContext
property with images that are not listed, would be blocked by the CI system. - It is not allowed to specify the
command
option along with thesecurityContext
option. This prevents one from overriding an entry point script in the container image that may drop privileges before launching test scripts.
The securityContext
option itself is identical to the similar option that
exists within K8s POD specifications, it is specified as a map that may contain
the following fields:
Field | Type | What it does |
---|---|---|
privileged | Boolean | Specify wither to run the container in privileged mode |
runAsUser | Int/String | Specify the user for processes in the container |
runAsGroup | Int/String | Specify the group for processes in the container |
fsGroup | Int/String | (Not used by current system implementation) |
All fields of the securityContext
option are optional, when not specified, the
default system-wide unprivileged settings are used.
Using secret data
Occasionally the CI system needs to carry out operations that requires access to secret data such as passwords, tokens or encryption keys.
To make such data available to the system, it needs to be encrypted with the GPG tool using the CI system's public key, and placed in the project's Git repository, as described in the Standard-CI GPG-based secrets document.
For decorated containers, the CI system would automatically find all the GPG-encrypted files included in the project source code and make unencrypted versions available. Care must be take when creating CI containers to avoid having scripts included in them accidentally output secret data to logs or as build artifacts.
Limitations
When compared to the legacy Standard-CI backed, the container-based backend has several limitations, some of which may be mitigated via future extensions:
- Only the x86_64 architecture is currently supported
- The CI target distribution configuration is ignored
- The CI script must exist even if unused
- Only the output of the last container in a given list of containers in shown in the Jenkins console or Blue Ocean. It is possible to see the logs of other containers via collected log files.
Tricks aliases, and shorthands
There are several syntactic alternatives supported, that are meant to allow writing shorter and nicer YAML.
- As mentioned above, when using a single container, the container entry
structure can be placed directly under the
container
option, without being a list member. -
A string can be specified instead of a full container entry. In that case, it is taken as the name of the image to use, and the other options are set to their default values.
Along with the single container syntax mentioned above, this allows writing very succinct configuration:
substages: - run-container: container: docker.io/my-container/image
-
The
args
option supports thearguments
andargument
aliases. - The
command
options supports theentrypoint
alias. - The
workingdir
options supports theworkdir
andworkingdirectory
aliases. - The
command
andargs
options may be given as lists of strings or singular strings, that are then implicitly converted to lists with a single string member. - The
decorate-containers
option may be shorthanded todecorate
. - The usual STDCI configuration file rules - for ignoring case, whitespace,
dashes and underscores - apply. Together with some of the features mentioned
above, this allows for a "literary" configuration style, like so:
Container: Image: centos/7 Entry Point: - /bin/bash - -exc Argument: | git init . git fetch "$STD_CI_CLONE_URL" +"$STD_CI_REFSPEC":myhead git checkout myhead my_test_script.sh