jobs: update trigger timers

Ensure we trigger building new OBS packages first, then wait plenty of
time until the binary packages are available (and run jobs in the
meantime that don't need them), and only after that we run the jobs
that need the binary packages.

Otherwise TTCN3 jobs may test the packages from the previous day, and
some jobs may fail completely due to packages not being completely built
yet. For example, yesterday the new Osmocom CNI releases were tagged,
which means the :latest packages also need to be rebuilt (-> building
all OBS packages takes longer). The osmocom-release-manuals and
-tarballs jobs failed, because the new binary packages were not
available yet when they ran.

Change all timers to the format "H 20 * * *" to have a deterministic
hour and semi-random minute based on the job name.

Change-Id: Ib68f9a78bae27a63706a8c95715bf6a244b7bf1d

17 files changed, 164 insertions, 120 deletions

diff --git a/jobs/README.adoc b/jobs/README.adoc
index 32da7c1..86fc485 100644
--- a/jobs/README.adoc
+++ b/jobs/README.adoc
@@ -102,3 +102,46 @@ statistics with SSH on the build nodes with:
 Note that running multiple jobs in parallel influence the ccache statistics,
 and it's impossible to tell which job caused which change in the stats (that's
 why they are not printed at the end of each job, it would be confusing).
+
+*timers*
+
+A lot of the jenkins jobs run daily with a timer:
+
+ triggers:
+   - timed: "H 20 * * *"
+
+or weekly:
+
+ triggers:
+   - timed: "H 20 * * H"
+
+Use H for the minute / day of week, to have it derivated as hash of the job
+name. Replace 20 with the hour (UTC) the job should run.
+
+The jobs follow this timetable, to ensure we don't attempt to use binary
+packages before they have been built (leading to failing jobs).
+
+ 18:00 - 21:00 OBS related
+   18:XX osmocom-obs-sync (sync Debian:12 etc. with openSUSE OBS)
+   19:XX osmocom-obs-wireshark
+   20:XX osmocom-obs (new binary packages start building on OBS!)
+
+ 22:00 - 03:00 Jobs that don't need binary packages
+   22:XX coverity
+   22:XX octsim_osmo-ccid-firmware
+   22:XX osmo-gsm-tester-runner (virtual)
+   23:XX master-builds-dahdi
+   00:XX osmocom-api
+   00:XX registry-update-base-images
+   01:XX osmocom-build-tags-against-master
+   01:XX osmocom-list-commits
+   02:XX registry-triggers
+   02:XX registry-rebuild-upload-fpga-build (weekly)
+
+ 04:00 - 18:00 Jobs that need binary packages
+   ttcn3-testsuites
+   04:XX osmocom-release-manuals
+   05:XX osmocom-release-tarballs
+   06:XX repo-install-test
+
+master-builds: to avoid complexity, these run throughout the day (H H * * *).
diff --git a/jobs/coverity.yml b/jobs/coverity.yml
index a2663a8..42c060a 100644
--- a/jobs/coverity.yml
+++ b/jobs/coverity.yml
@@ -43,7 +43,7 @@
           branches:
             - '$BRANCH'
     triggers:
-      - timed: "@daily"
+      - timed: "H 22 * * *"
     publishers:
       - email:
           notify-every-unstable-build: true
diff --git a/jobs/master-builds-dahdi.yml b/jobs/master-builds-dahdi.yml
index 3d263e3..8f6d070 100644
--- a/jobs/master-builds-dahdi.yml
+++ b/jobs/master-builds-dahdi.yml
@@ -52,7 +52,7 @@
           before: true
 
     triggers:
-      - timed: "H H * * *"
+      - timed: "H 23 * * *"
 
     builders:
       - shell: |
diff --git a/jobs/octsim_osmo-ccid-firmware.yml b/jobs/octsim_osmo-ccid-firmware.yml
index 534cf13..7db1039 100644
--- a/jobs/octsim_osmo-ccid-firmware.yml
+++ b/jobs/octsim_osmo-ccid-firmware.yml
@@ -49,5 +49,5 @@
     triggers:
     - reverse:
         jobs: master-osmo-ccid-firmware,
-    - timed: H H * * *
+    - timed: H 22 * * *
     wrappers: []
diff --git a/jobs/osmocom-api.yml b/jobs/osmocom-api.yml
index 294ccaa..312496b 100644
--- a/jobs/osmocom-api.yml
+++ b/jobs/osmocom-api.yml
@@ -15,7 +15,7 @@
          notify-every-unstable-build: true
          recipients: jenkins-notifications@lists.osmocom.org
     triggers:
-    - timed: "H H * * *"
+    - timed: "H 00 * * *"
     wrappers:
     - ssh-agent-credentials:
         users:
diff --git a/jobs/osmocom-build-tags-against-master.yml b/jobs/osmocom-build-tags-against-master.yml
index 4ea4055..e33c9a3 100644
--- a/jobs/osmocom-build-tags-against-master.yml
+++ b/jobs/osmocom-build-tags-against-master.yml
@@ -40,7 +40,7 @@
           git-config-name: 'Jenkins Builder'
           git-config-email: 'jenkins@osmocom.org'
     triggers:
-      - timed: "@midnight"
+      - timed: "H 01 * * *"
     publishers:
        - archive:
           allow-empty: false
diff --git a/jobs/osmocom-list-commits.yml b/jobs/osmocom-list-commits.yml
index 43ce171..d367499 100644
--- a/jobs/osmocom-list-commits.yml
+++ b/jobs/osmocom-list-commits.yml
@@ -34,7 +34,7 @@
           git-config-name: 'Jenkins Builder'
           git-config-email: 'jenkins@osmocom.org'
     triggers:
-      - timed: "@midnight"
+       - timed: "H 01 * * *"
     publishers:
        - archive:
           allow-empty: false
diff --git a/jobs/osmocom-obs-sync.yml b/jobs/osmocom-obs-sync.yml
index 2dd9888..fef1e83 100644
--- a/jobs/osmocom-obs-sync.yml
+++ b/jobs/osmocom-obs-sync.yml
@@ -44,7 +44,7 @@
             --docker \
             $DISTROS
     triggers:
-      - timed: "H H * * *"
+      - timed: "H 18 * * *"
     wrappers:
     - ansicolor:
         colormap: xterm
diff --git a/jobs/osmocom-obs-wireshark.yml b/jobs/osmocom-obs-wireshark.yml
index 9d4d04e..213a1f3 100644
--- a/jobs/osmocom-obs-wireshark.yml
+++ b/jobs/osmocom-obs-wireshark.yml
@@ -34,7 +34,7 @@
           git-config-email: 'jenkins@osmocom.org'
           wipe-workspace: true
     triggers:
-      - timed: "H 05 * * *"
+      - timed: "H 19 * * *"
 
     publishers:
       - email:
diff --git a/jobs/osmocom-obs.yml b/jobs/osmocom-obs.yml
index 4870066..eb77f4d 100644
--- a/jobs/osmocom-obs.yml
+++ b/jobs/osmocom-obs.yml
@@ -50,7 +50,7 @@
           git-config-name: 'Jenkins Builder'
           git-config-email: 'jenkins@osmocom.org'
     triggers:
-      - timed: "H 00 * * *" # run before repo-install-test.yml
+      - timed: "H 20 * * *"
 
     publishers:
       - email:
diff --git a/jobs/osmocom-release-manuals.yml b/jobs/osmocom-release-manuals.yml
index a120127..c9d04ae 100644
--- a/jobs/osmocom-release-manuals.yml
+++ b/jobs/osmocom-release-manuals.yml
@@ -34,7 +34,7 @@
           git-config-email: 'jenkins@osmocom.org'
           wipe-workspace: true
     triggers:
-      - timed: "@midnight"
+      - timed: "H 04 * * *"
     publishers:
        - email:
           notify-every-unstable-build: true
diff --git a/jobs/osmocom-release-tarballs.yml b/jobs/osmocom-release-tarballs.yml
index 557898a..4b3c8dc 100644
--- a/jobs/osmocom-release-tarballs.yml
+++ b/jobs/osmocom-release-tarballs.yml
@@ -35,7 +35,7 @@
           git-config-email: 'jenkins@osmocom.org'
           wipe-workspace: true
     triggers:
-      - timed: "@midnight"
+      - timed: "H 05 * * *"
     publishers:
        - email:
           notify-every-unstable-build: true
diff --git a/jobs/registry-rebuild-upload-fpga-build.yml b/jobs/registry-rebuild-upload-fpga-build.yml
index c4b750a..1846e49 100644
--- a/jobs/registry-rebuild-upload-fpga-build.yml
+++ b/jobs/registry-rebuild-upload-fpga-build.yml
@@ -29,7 +29,7 @@
           docker push $REGISTRY/$USER/fpga-build
           cd ..
     triggers:
-      - timed: "@weekly"
+      - timed: "H 02 * * H"
 
     publishers:
       - email:
diff --git a/jobs/registry-triggers.yml b/jobs/registry-triggers.yml
index 0395976..da7d402 100644
--- a/jobs/registry-triggers.yml
+++ b/jobs/registry-triggers.yml
@@ -27,7 +27,7 @@
       - pollscm:
           cron: "H/5 * * * *"
           ignore-post-commit-hooks: false
-      - timed: "H H * * *"
+      - timed: "H 02 * * *"
     publishers:
        - trigger:
            project: '{obj:trigger}'
diff --git a/jobs/registry-update-base-images.yml b/jobs/registry-update-base-images.yml
index 6c4074e..f3f20e3 100644
--- a/jobs/registry-update-base-images.yml
+++ b/jobs/registry-update-base-images.yml
@@ -39,7 +39,7 @@
             docker push $dst
           done
     triggers:
-      - timed: "@daily"
+      - timed: "H 00 * * *"
 
     publishers:
       - email:
diff --git a/jobs/repo-install-test.yml b/jobs/repo-install-test.yml
index eba2a79..1340e20 100644
--- a/jobs/repo-install-test.yml
+++ b/jobs/repo-install-test.yml
@@ -60,7 +60,7 @@
           git-config-name: 'Jenkins Builder'
           git-config-email: 'jenkins@osmocom.org'
     triggers:
-      - timed: "H 10 * * *" # run after osmocom-obs.yml
+      - timed: "H 06 * * *"
 
     publishers:
       - email:
diff --git a/jobs/ttcn3-testsuites.yml b/jobs/ttcn3-testsuites.yml
index 8568068..4c15f84 100644
--- a/jobs/ttcn3-testsuites.yml
+++ b/jobs/ttcn3-testsuites.yml
@@ -17,24 +17,25 @@
     jobs:
       - "{job-name}"
     # Folders from docker-playground.git
+    # timer: 10 per hour from 04:00 - 18:00, see jobs/README.adoc
     job-name:
       # debian nightly - order by time
       - nplab-m3ua-test:
-          timer: 10 03 * * *
+          timer: H 04 * * *
       - nplab-sua-test:
-          timer: 15 03 * * *
+          timer: H 04 * * *
       - ttcn3-ggsn-test:
-          timer: 00 04 * * *
+          timer: H 04 * * *
       - ttcn3-ggsn-test-kernel:
-          timer: 10 04 * * *
+          timer: H 04 * * *
           description: |
             Test osmo-ggsn master with the Debian kernel.
       - ttcn3-ggsn-test-kernel-latest:
-          timer: 20 04 * * *
+          timer: H 04 * * *
           description: |
             Test osmo-ggsn latest with the Debian kernel.
       - ttcn3-ggsn-test-kernel-latest-net-next:
-          timer: 25 04 * * *
+          timer: H 04 * * *
           # Cache the linux source tree in the workspace of one node
           wipe_workspace: false
           node: ttcn3-ggsn-test-kernel-git
@@ -43,7 +44,7 @@
             <a href="https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/">
               net-next.git</a>.
       - ttcn3-ggsn-test-kernel-latest-torvalds:
-          timer: 30 04 * * *
+          timer: H 04 * * *
           # Cache the linux source tree in the workspace of one node
           wipe_workspace: false
           node: ttcn3-ggsn-test-kernel-git
@@ -52,233 +53,233 @@
             <a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/">
               torvalds/linux.git</a>.
       - ttcn3-ggsn-test-ogs:
-          timer: 35 04 * * *
+          timer: H 04 * * *
           description: |
             Test open5gs-smfd + open5gs-upfd main.
           email: jenkins-notifications@lists.osmocom.org acetcom@gmail.com
       - ttcn3-msc-test:
-          timer: 40 04 * * *
+          timer: H 04 * * *
       - ttcn3-pcu-test:
-          timer: 00 05 * * *
+          timer: H 04 * * *
       - ttcn3-pcu-test-sns:
-          timer: 15 05 * * *
+          timer: H 05 * * *
       - ttcn3-sgsn-test:
-          timer: 30 05 * * *
+          timer: H 05 * * *
       - ttcn3-bsc-test:
-          timer: 00 06 * * *
+          timer: H 05 * * *
       - ttcn3-mgw-test:
-          timer: 30 06 * * *
+          timer: H 05 * * *
       - ttcn3-hlr-test:
-          timer: 45 06 * * *
+          timer: H 05 * * *
       - ttcn3-bts-test:
-          timer: 00 07 * * *
+          timer: H 05 * * *
       - ttcn3-sip-test:
-          timer: 30 07 * * *
+          timer: H 05 * * *
       - ttcn3-bscnat-test:
-          timer: 00 08 * * *
+          timer: H 05 * * *
           disabled: true
       - ttcn3-bsc-test-sccplite:
-          timer: 30 08 * * *
+          timer: H 05 * * *
       - ttcn3-remsim-test:
-          timer: 00 09 * * *
+          timer: H 05 * * *
       - ttcn3-stp-test:
-          timer: 30 09 * * *
+          timer: H 06 * * *
       - ttcn3-sccp-test:
-          timer: 00 10 * * *
+          timer: H 06 * * *
       - ttcn3-smlc-test:
-          timer: 10 10 * * *
+          timer: H 06 * * *
       - ttcn3-gbproxy-test:
-          timer: 20 10 * * *
+          timer: H 06 * * *
       - ttcn3-gbproxy-test-fr:
-          timer: 30 10 * * *
+          timer: H 06 * * *
           node: hdlc
       - ttcn3-cbc-test:
-          timer: 40 10 * * *
+          timer: H 06 * * *
       - ttcn3-ns-test:
-          timer: 00 11 * * *
+          timer: H 06 * * *
       - ttcn3-ns-test-sns:
-          timer: 10 11 * * *
+          timer: H 06 * * *
       - ttcn3-ns-test-fr:
-          timer: 20 11 * * *
+          timer: H 06 * * *
           node: hdlc
       - ttcn3-ns-test-sgsn-sns:
-          timer: 30 11 * * *
+          timer: H 06 * * *
       - ttcn3-pcap-client-test:
-          timer: 40 11 * * *
+          timer: H 07 * * *
       - ttcn3-bsc-test-vamos:
-          timer: 00 12 * * *
+          timer: H 07 * * *
       - ttcn3-hnodeb-test:
-          timer: 40 12 * * *
+          timer: H 07 * * *
       - ttcn3-upf-test:
-          timer: 00 13 * * *
+          timer: H 07 * * *
       - ttcn3-hnbgw-test:
-          timer: 40 13 * * *
+          timer: H 07 * * *
       - ttcn3-fr-test:
-          timer: 40 10 * * *
+          timer: H 07 * * *
           node: hdlc
       - ttcn3-pgw-test:
-          timer: 00 14 * * *
+          timer: H 07 * * *
           description: |
             Test open5gs-smfd + open5gs-upfd PGW main.
           email: jenkins-notifications@lists.osmocom.org acetcom@gmail.com
       - ttcn3-mme-test-ogs:
-          timer: 15 14 * * *
+          timer: H 07 * * *
           description: |
             Test open5gs-mme
           email: jenkins-notifications@lists.osmocom.org acetcom@gmail.com
       - ttcn3-dia2gsup-test:
-          timer: 30 14 * * *
+          timer: H 07 * * *
 
       # debian latest stable
       - nplab-m3ua-test-latest:
-          timer: 10 03 * * *
+          timer: H 07 * * *
       - nplab-sua-test-latest:
-          timer: 15 03 * * *
+          timer: H 08 * * *
       - ttcn3-ggsn-test-latest:
-          timer: 00 04 * * *
+          timer: H 08 * * *
       - ttcn3-msc-test-latest:
-          timer: 30 04 * * *
+          timer: H 08 * * *
       - ttcn3-pcu-test-latest:
-          timer: 00 05 * * *
+          timer: H 08 * * *
       - ttcn3-pcu-test-sns-latest:
-          timer: 15 05 * * *
+          timer: H 08 * * *
       - ttcn3-sgsn-test-latest:
-          timer: 30 05 * * *
+          timer: H 08 * * *
       - ttcn3-bsc-test-latest:
-          timer: 00 06 * * *
+          timer: H 08 * * *
       - ttcn3-mgw-test-latest:
-          timer: 30 06 * * *
+          timer: H 08 * * *
       - ttcn3-hlr-test-latest:
-          timer: 45 06 * * *
+          timer: H 08 * * *
       - ttcn3-bts-test-latest:
-          timer: 00 07 * * *
+          timer: H 08 * * *
       - ttcn3-sip-test-latest:
-          timer: 30 07 * * *
+          timer: H 09 * * *
       - ttcn3-bscnat-test-latest:
-          timer: 00 08 * * *
+          timer: H 09 * * *
           disabled: true
       - ttcn3-bsc-test-sccplite-latest:
-          timer: 30 08 * * *
+          timer: H 09 * * *
       - ttcn3-remsim-test-latest:
-          timer: 00 09 * * *
+          timer: H 09 * * *
       - ttcn3-stp-test-latest:
-          timer: 30 09 * * *
+          timer: H 09 * * *
       - ttcn3-sccp-test-latest:
-          timer: 00 10 * * *
+          timer: H 09 * * *
       - ttcn3-smlc-test-latest:
-          timer: 10 10 * * *
+          timer: H 09 * * *
       - ttcn3-gbproxy-test-latest:
-          timer: 20 10 * * *
+          timer: H 09 * * *
       - ttcn3-gbproxy-test-fr-latest:
-          timer: 30 10 * * *
+          timer: H 09 * * *
           node: hdlc
       - ttcn3-cbc-test-latest:
-          timer: 40 10 * * *
+          timer: H 09 * * *
       - ttcn3-pcap-client-test-latest:
-          timer: 40 11 * * *
+          timer: H 10 * * *
       - ttcn3-hnodeb-test-latest:
-          timer: 40 12 * * *
+          timer: H 10 * * *
       - ttcn3-upf-test-latest:
-          timer: 00 13 * * *
+          timer: H 10 * * *
           disabled: true
       - ttcn3-hnbgw-test-latest:
-          timer: 40 13 * * *
+          timer: H 10 * * *
 
       # Jobs below use uppercase TTCN3-$DISTRO, so they get listed separately from the ttcn3-* (debian) jobs
 
       # centos7 nightly
       - TTCN3-centos7-pcap-client-test:
-          timer: 10 13 * * *
+          timer: H 10 * * *
 
       # centos7 latest
       - TTCN3-centos7-pcap-client-test-latest:
-          timer: 40 13 * * *
+          timer: H 10 * * *
 
       # centos8 nightly
       - TTCN3-centos-ggsn-test:
-          timer: 00 04 * * *
+          timer: H 10 * * *
       - TTCN3-centos-msc-test:
-          timer: 30 04 * * *
+          timer: H 10 * * *
       - TTCN3-centos-pcu-test:
-          timer: 00 05 * * *
+          timer: H 10 * * *
       - TTCN3-centos-pcu-test-sns:
-          timer: 15 05 * * *
+          timer: H 10 * * *
       - TTCN3-centos-sgsn-test:
-          timer: 30 05 * * *
+          timer: H 11 * * *
       - TTCN3-centos-bsc-test:
-          timer: 00 06 * * *
+          timer: H 11 * * *
       - TTCN3-centos-mgw-test:
-          timer: 30 06 * * *
+          timer: H 11 * * *
       - TTCN3-centos-hlr-test:
-          timer: 45 06 * * *
+          timer: H 11 * * *
       - TTCN3-centos-bts-test:
-          timer: 00 07 * * *
+          timer: H 11 * * *
       - TTCN3-centos-sip-test:
-          timer: 30 07 * * *
+          timer: H 11 * * *
       - TTCN3-centos-bsc-test-sccplite:
-          timer: 30 08 * * *
+          timer: H 11 * * *
       - TTCN3-centos-remsim-test:
-          timer: 00 09 * * *
+          timer: H 11 * * *
           disabled: true
       - TTCN3-centos-stp-test:
-          timer: 30 09 * * *
+          timer: H 11 * * *
       - TTCN3-centos-gbproxy-test:
-          timer: 20 10 * * *
+          timer: H 11 * * *
       - TTCN3-centos-cbc-test:
-          timer: 40 10 * * *
+          timer: H 12 * * *
 
       # centos8 latest stable
       - TTCN3-centos-ggsn-test-latest:
-          timer: 00 04 * * *
+          timer: H 12 * * *
       - TTCN3-centos-msc-test-latest:
-          timer: 30 04 * * *
+          timer: H 12 * * *
       - TTCN3-centos-pcu-test-latest:
-          timer: 00 05 * * *
+          timer: H 12 * * *
       - TTCN3-centos-pcu-test-sns-latest:
-          timer: 15 05 * * *
+          timer: H 12 * * *
       - TTCN3-centos-sgsn-test-latest:
-          timer: 30 05 * * *
+          timer: H 12 * * *
       - TTCN3-centos-bsc-test-latest:
-          timer: 00 06 * * *
+          timer: H 12 * * *
       - TTCN3-centos-mgw-test-latest:
-          timer: 30 06 * * *
+          timer: H 12 * * *
       - TTCN3-centos-hlr-test-latest:
-          timer: 45 06 * * *
+          timer: H 12 * * *
       - TTCN3-centos-bts-test-latest:
-          timer: 00 07 * * *
+          timer: H 12 * * *
       - TTCN3-centos-sip-test-latest:
-          timer: 30 07 * * *
+          timer: H 13 * * *
       - TTCN3-centos-bsc-test-sccplite-latest:
-          timer: 30 08 * * *
+          timer: H 13 * * *
       - TTCN3-centos-remsim-test-latest:
-          timer: 00 09 * * *
+          timer: H 13 * * *
           disabled: true
       - TTCN3-centos-stp-test-latest:
-          timer: 30 09 * * *
+          timer: H 13 * * *
       - TTCN3-centos-smlc-test-latest:
-          timer: 10 10 * * *
+          timer: H 13 * * *
       - TTCN3-centos-sccp-test-latest:
-          timer: 00 10 * * *
+          timer: H 13 * * *
       - TTCN3-centos-gbproxy-test-latest:
-          timer: 20 10 * * *
+          timer: H 13 * * *
       - TTCN3-centos-cbc-test-latest:
-          timer: 40 10 * * *
+          timer: H 13 * * *
 
       # centos8 2023q1 (SYS#6283)
       - TTCN3-centos-pcu-test-2023q1:
-          timer: 00 17 * * *
+          timer: H 13 * * *
       - TTCN3-centos-pcu-test-sns-2023q1:
-          timer: 15 17 * * *
+          timer: H 13 * * *
       - TTCN3-centos-bsc-test-2023q1:
-          timer: 30 17 * * *
+          timer: H 14 * * *
       - TTCN3-centos-mgw-test-2023q1:
-          timer: 45 17 * * *
+          timer: H 14 * * *
       - TTCN3-centos-bts-test-2023q1:
-          timer: 00 18 * * *
+          timer: H 14 * * *
       - TTCN3-centos-smlc-test-2023q1:
-          timer: 15 18 * * *
+          timer: H 14 * * *
       - TTCN3-centos-gbproxy-test-2023q1:
-          timer: 30 18 * * *
+          timer: H 14 * * *
 
 - job-template:
     name: '{job-name}'