scripts/obs: rewrite pushing source pkgs to OBSosmith/wip-obs

Harald requested that the OBS scripts should not stop if building one
specific source package fails, instead it should keep going and report
at the end a non-success exit code.

Given that the shell script code has historically grown and became hard
to maintain, I decided to rewrite the scripts for implementing this
feature. This rewrite solves additional problems:

* No full checkout of an OBS project like network:osmocom:latest
  anymore, with lots of packages that won't get updated (e.g. the uhd
  package has a uhd-images_3.14.1.1.tar.xz file that is 108 MB). With
  the old code, developers had to wait minutes during the checkout
  before the script reaches code that is currently being developed. Now
  only single packages get checked out right before they get updated.

* No need to clone git repositories over and over. With the new code,
  git repos only get cloned if needed (for latest it is not needed if
  the remote git tag is the same as the version in OBS). During
  development, the cloned git repositories are cached.

* Output from commands like "git tag -l" is not written to the log
  unless they failed. This makes the log more readable, which is
  especially important when a package fails to build, we keep going and
  need to spot the build error in the middle of the log later on.

* No more duplicated code for nightly and latest scripts that worked
  similar but had slight differences. Also the list of packages is not
  duplicated for nightly and latest anymore; nightly uses all packages
  and latest uses packages that have at least one git tag.

* Building source packages is decoupled from uploading them. A separate
  script build_srcpkg.py can be used to just build the deb + rpm spec
  source packages, without interacting with the OBS server.

* The scripts can optionally run in docker with a command-line switch,
  and this is used by jenkins. This way we don't need to install
  more dependencies on the host such as rebar3 which is now needed for
  erlang/osmo_dia2gsup.

* Add erlang/osmo_dia2gsup and run its generate_build_dep.sh (SYS#6006)

I have done the new implementation in python to make use of argparse
and to be able to use try/except and print a trace when building one
package fails.

Example output:
* https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_nightly_obs.osmocom.org/48/console
* https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_latest_obs.osmocom.org/46/console

Change-Id: I45a555d05a9da808c0fe0145aae665f583cb80d9

1 files changed, 62 insertions, 0 deletions

diff --git a/scripts/obs/lib/docker.py b/scripts/obs/lib/docker.py
new file mode 100644
index 0000000..a5dbcd9
--- /dev/null
+++ b/scripts/obs/lib/docker.py
@@ -0,0 +1,62 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright 2022 sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
+import os
+import shutil
+import subprocess
+import sys
+import lib
+import lib.config
+
+
+def build_image():
+    print(f"docker: building image {lib.config.docker_image_name}")
+    lib.run_cmd(["docker", "build",
+                 "--build-arg", f"UID={os.getuid()}",
+                 "-t", lib.config.docker_image_name,
+                 f"{lib.config.path_top}/data"])
+
+
+def get_oscrc():
+    ret = os.path.expanduser("~/.oscrc")
+    if "OSCRC" in os.environ:
+        ret = os.environ["OSCRC"]
+
+    if os.path.exists(ret):
+        return os.path.realpath(ret)
+
+    print("ERROR: couldn't find ~/.oscrc. Put it there or set OSCRC.")
+    exit(1)
+
+
+def run_in_docker_and_exit(script_path, args, add_oscrc=False):
+    if "INSIDE_DOCKER" in os.environ:
+        return
+
+    if not shutil.which("docker"):
+        print("ERROR: docker is not installed")
+        exit(1)
+
+    oscrc = None
+    if add_oscrc:
+        oscrc = get_oscrc()
+
+    # Build the docker image. Unless it is up-to-date, this will take a few
+    # minutes or so, therefore print the output.
+    lib.set_cmds_verbose(True)
+    build_image()
+    lib.set_cmds_verbose(args.verbose)
+
+    cmd = ["docker", "run",
+           "-e", "INSIDE_DOCKER=1",
+           "-e", "PYTHONUNBUFFERED=1",
+           "--rm", "-v", f"{lib.config.path_top}:/obs"]
+
+    if oscrc:
+        cmd += ["-v", f"{oscrc}:/home/user/.oscrc"]
+
+    script_path = f"/obs/{os.path.basename(script_path)}"
+    cmd += [lib.config.docker_image_name, script_path] + sys.argv[1:]
+
+    print(f"docker: running: {os.path.basename(script_path)} inside docker")
+    ret = subprocess.run(cmd)
+    exit(ret.returncode)