Python: Sandbox the requirements update script a bit
We currently get errors trying to run the requirements update process.
The issue is that pip tries to set up packages that require various
host tools just to run the setup.py script.
The patch here runs the update process in a container.
$ bazel run //tools/python:requirements.update
WARNING: Overflow when watching local filesystem for changes... temporarily falling back to manually checking files for changes
INFO: Build option --compilation_mode has changed, discarding analysis cache.
INFO: Analyzed target //tools/python:requirements.update (84 packages loaded, 27041 targets configured).
INFO: Found 1 target...
Target //tools/python:requirements.update up-to-date:
bazel-bin/tools/python/requirements.update
INFO: Elapsed time: 3.172s, Critical Path: 0.09s
INFO: 4 processes: 4 internal.
INFO: Build completed successfully, 4 total actions
INFO: Running command line: bazel-bin/tools/python/requirements.update tools/python/requirements.txt tools/python/requirements.lock.txt None None None //tools/python:requirements.update
Updating tools/python/requirements.lock.txt
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [45 lines of output]
Collecting setuptools
Downloading setuptools-69.0.3-py3-none-any.whl (819 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 819.5/819.5 kB 11.4 MB/s eta 0:00:00
Collecting wheel
Downloading wheel-0.42.0-py3-none-any.whl (65 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.4/65.4 kB 6.2 MB/s eta 0:00:00
Collecting pycairo
Downloading pycairo-1.25.1.tar.gz (347 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 347.1/347.1 kB 16.1 MB/s eta 0:00:00
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Installing backend dependencies: started
Installing backend dependencies: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: pycairo
Building wheel for pycairo (pyproject.toml): started
Building wheel for pycairo (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error
× Building wheel for pycairo (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [14 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-39
creating build/lib.linux-x86_64-cpython-39/cairo
copying cairo/__init__.py -> build/lib.linux-x86_64-cpython-39/cairo
copying cairo/__init__.pyi -> build/lib.linux-x86_64-cpython-39/cairo
copying cairo/py.typed -> build/lib.linux-x86_64-cpython-39/cairo
warning: build_py: byte-compiling is disabled, skipping.
running build_ext
'pkg-config' not found.
Command ['pkg-config', '--print-errors', '--exists', 'cairo >= 1.15.10']
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pycairo
Failed to build pycairo
ERROR: Could not build wheels for pycairo, which is required to install pyproject.toml-based projects
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/rules_python/python/pip_install/pip_compile.py", line 127, in <module>
cli()
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__click/click/core.py", line 1137, in __call__
return self.main(*args, **kwargs)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__click/click/core.py", line 1062, in main
rv = self.invoke(ctx)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__click/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__click/click/core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__click/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/scripts/compile.py", line 487, in cli
results = resolver.resolve(max_rounds=max_rounds)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 266, in resolve
has_changed, best_matches = self._resolve_one_round()
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 356, in _resolve_one_round
their_constraints.extend(self._iter_dependencies(best_match))
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 469, in _iter_dependencies
dependencies = self.repository.get_dependencies(ireq)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/repositories/local.py", line 85, in get_dependencies
return self.repository.get_dependencies(ireq)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/repositories/pypi.py", line 250, in get_dependencies
self._dependencies_cache[ireq] = self.resolve_reqs(
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/repositories/pypi.py", line 213, in resolve_reqs
results = resolver._resolve_one(reqset, ireq)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/resolution/legacy/resolver.py", line 509, in _resolve_one
dist = self._get_dist_for(req_to_install)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/resolution/legacy/resolver.py", line 462, in _get_dist_for
dist = self.preparer.prepare_linked_requirement(req)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 438, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 524, in _prepare_linked_requirement
dist = _get_prepared_distribution(
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 68, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
self._prepare_build_backend(finder)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/distributions/sdist.py", line 70, in _prepare_build_backend
self.req.build_env.install_requirements(
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/build_env.py", line 196, in install_requirements
self._install_requirements(
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/build_env.py", line 254, in _install_requirements
call_subprocess(
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/utils/subprocess.py", line 224, in call_subprocess
raise error
pip._internal.exceptions.InstallationSubprocessError: pip subprocess to install build dependencies exited with 1
Change-Id: Idc0e66994c7e0282f7ef8825f737fd407c0cc0a4
Signed-off-by: Philipp Schrader <philipp.schrader@gmail.com>
diff --git a/tools/python/update_helper.sh b/tools/python/update_helper.sh
new file mode 100755
index 0000000..ba497ea
--- /dev/null
+++ b/tools/python/update_helper.sh
@@ -0,0 +1,40 @@
+#!/bin/bash
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+# --- begin runfiles.bash initialization v2 ---
+# Copy-pasted from the Bazel Bash runfiles library v2.
+set -uo pipefail; set +e; f=bazel_tools/tools/bash/runfiles/runfiles.bash
+source "${RUNFILES_DIR:-/dev/null}/$f" 2>/dev/null || \
+ source "$(grep -sm1 "^$f " "${RUNFILES_MANIFEST_FILE:-/dev/null}" | cut -f2- -d' ')" 2>/dev/null || \
+ source "$0.runfiles/$f" 2>/dev/null || \
+ source "$(grep -sm1 "^$f " "$0.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
+ source "$(grep -sm1 "^$f " "$0.exe.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
+ { echo>&2 "ERROR: cannot find $f"; exit 1; }; f=; set -e
+# --- end runfiles.bash initialization v2 ---
+
+DOCKERFILE="$(rlocation org_frc971/tools/python/update_helper_files/Dockerfile)"
+CONTEXT_DIR="$(dirname "${DOCKERFILE}")"
+CONTAINER_TAG="pip-lock:${USER}"
+
+# Build the container that has the bare minimum to run the various setup.py
+# scripts from our dependencies.
+docker build \
+ --file="${DOCKERFILE}" \
+ --tag="${CONTAINER_TAG}" \
+ "${CONTEXT_DIR}"
+
+# Run the actual update. The assumption here is that mounting the user's home
+# directory is sufficient to allow the tool to run inside the container without
+# any issues. I.e. the cache and the source tree are available in the
+# container.
+docker run \
+ --rm \
+ --tty \
+ --env BUILD_WORKSPACE_DIRECTORY="${BUILD_WORKSPACE_DIRECTORY}" \
+ --workdir "${PWD}" \
+ --volume "${HOME}:${HOME}" \
+ "${CONTAINER_TAG}" \
+ "$@"