.. include:: replace.txt .. highlight:: bash .. _Working with gitlab-ci-local: Working with gitlab-ci-local ---------------------------- .. _continuous integration (CI) : https://docs.gitlab.com/ee/ci/ .. _pipelines : https://docs.gitlab.com/ee/ci/introduction/index.html#continuous-integration .. _daily and weekly pipelines : https://gitlab.com/nsnam/ns-3-dev/-/pipeline_schedules .. _crypto miners abuse : https://about.gitlab.com/blog/2021/05/17/prevent-crypto-mining-abuse/ .. _GitLab-CI-local : https://github.com/firecow/gitlab-ci-local .. _GitLab CI : https://docs.gitlab.com/ee/ci/ The ns-3 project repository is currently hosted in GitLab, which includes `continuous integration (CI)`_ tools to automate build, tests, packaging and distribution of software. The CI works based on jobs, that are defined in YAML files and run inside containers. See :ref:`Working with Docker` for more information about containers in general. The ns-3 GitLab CI files are located in ``ns-3-dev/utils/tests/``. The main GitLab CI file is ``gitlab-ci.yml``. The different jobs are used to check if a multitude of compilers and package versions are compatible with the current ns-3 build, which is why a build is usually followed by a test run. Other CI jobs build and warn about missing the documentation. The GitLab CI jobs are executed based on `pipelines`_ containing a sequence of job batches. Jobs within a batch can be executed in parallel. These `pipelines`_ can be triggered manually, or scheduled to run automatically per commit and/or based on a time period (ns-3 has `daily and weekly pipelines`_ scheduled). The GitLab CI free tier is very slow, taking a lot of time to identify issues during active merge request development. Note: the free tier now requires a credit card due to `crypto miners abuse`_. `GitLab-CI-local`_ is a tool that allows an user to use the `GitLab CI`_ configuration files locally, allowing for the debugging of CI settings and pipelines without requiring pushes to test repositories or main repositories that fill up the CI job queues with failed jobs due to script errors. GitLab-CI-local relies on :ref:`Docker containers` to setup the environment to execute the jobs. Note: Docker is usually setup in root mode, requiring frequent use of administrative permissions/sudo. However, this is highly discouraged. You can configure Docker to run in :ref:`Docker rootless mode `. From this point onwards, we assume Docker is configured in :ref:`Docker rootless mode `. After installing both :ref:`Docker ` and `GitLab-CI-local`_, the ns-3 jobs can be listed using the following command: .. sourcecode:: bash ~/ns-3-dev$ gitlab-ci-local --file ./utils/tests/gitlab-ci.yml --list parsing and downloads finished in 226 ms name description stage when allow_failure needs weekly-build-ubuntu-18.04-debug build on_success false ... weekly-build-clang-11-optimized build on_success false cppyy-22.04 build on_success false per-commit-compile-debug build on_success false per-commit-compile-release build on_success false per-commit-compile-optimized build on_success false daily-test-debug test on_success false daily-test-release test on_success false daily-test-optimized test on_success false daily-test-optimized-valgrind test on_success false weekly-test-debug-valgrind test on_success false weekly-test-release-valgrind test on_success false weekly-test-optimized-valgrind test on_success false weekly-test-takes-forever-optimized test on_success false doxygen documentation on_success false manual documentation on_success false tutorial documentation on_success false models documentation on_success false To execute the ``per-commit-compile-release`` job, or any of the others listed above, use the following command. .. sourcecode:: console ~/ns-3-dev$ gitlab-ci-local --file ./utils/tests/gitlab-ci.yml per-commit-compile-release WARNING: if you do not specify the job name, all jobs that can be executed in parallel will be executed at the same time. You may run out of disk, memory or both. Some jobs might require a previous job to complete successfully before getting started. The doxygen job is one of these. .. sourcecode:: console ~/ns-3-dev$ gitlab-ci-local --file ./utils/tests/gitlab-ci.yml doxygen Using fallback git user.name Using fallback git user.email parsing and downloads finished in 202 ms doxygen starting archlinux:latest (documentation) doxygen pulled archlinux:latest in 64 ms doxygen > still running... doxygen > still running... doxygen copied to container in 20 s doxygen imported cache 'ccache-' in 3.67 s ~/ns-3-dev/.gitlab-ci-local/artifacts/pybindgen doesn't exist, did you forget --needs As instructed by the previous command output, you can add the ``--needs`` to build required jobs before proceeding. However, doing so will run all jobs as the ``doxygen`` is only supposed to run after weekly jobs are successfully executed. Another option is to run the specific job that produces the required artifact. In this case the ``pybindgen`` job. Note: Pybindgen has been replaced by Cppyy, which does not produce artifacts to be consumed by other jobs. However, the example is kept for reference. .. sourcecode:: console ~/ns-3-dev$ gitlab-ci-local --file ./utils/tests/gitlab-ci.yml pybindgen Using fallback git user.name Using fallback git user.email parsing and downloads finished in 202 ms pybindgen starting archlinux:latest (build) ... pybindgen $ git diff src > pybindgen_new.patch pybindgen exported artifacts in 911 ms pybindgen copied artifacts to cwd in 56 ms pybindgen finished in 5.77 min PASS pybindgen Then run the doxygen job again: .. sourcecode:: bash ~/ns-3-dev$ gitlab-ci-local --file ./utils/tests/gitlab-ci.yml doxygen Using fallback git user.name Using fallback git user.email parsing and downloads finished in 170 ms doxygen starting archlinux:latest (documentation) ... doxygen > 1 files with warnings doxygen > Doxygen Warnings Summary doxygen > ---------------------------------------- doxygen > 1 directories doxygen > 1 files doxygen > 23 warnings doxygen > done. doxygen exported cache ns-3-ccache-storage/ 'ccache-' in 6.86 s doxygen exported artifacts in 954 ms doxygen copied artifacts to cwd in 59 ms doxygen finished in 15 min PASS doxygen Artifacts built by the CI jobs will be stored in separate subfolders based on the job name. ``~/ns-3-dev/.gitlab-ci-local/artifacts/jobname`` Note: some jobs may access the ``CI_DEFAULT_BRANCH`` environment variable, which is set by default to ``main`` instead of ``master``. To change that, we need to create a file ``~/.gitlab-ci-local/variables.yml`` containing the following: .. sourcecode:: YAML global: CI_DEFAULT_BRANCH: "master" In case you are using Docker with root, you need to create this file in ``/root/.gitlab-ci-local/variables.yml``.