| commit | 8ac22f3d967b0237c0075b276084ea5b7c538202 | [log] [tgz] |
|---|---|---|
| author | Philipp Schrader <philipp.schrader@gmail.com> | Sat Dec 30 17:37:15 2023 -0800 |
| committer | Philipp Schrader <philipp.schrader@gmail.com> | Wed Jan 10 20:30:53 2024 -0800 |
| tree | 6f93af533035de485b09c550b25782c344762545 | |
| parent | 187c4e2e63f71abdcb56156c0b7865113ab74d78 [diff] |
Python: Sandbox the requirements update script a bit
We currently get errors trying to run the requirements update process.
The issue is that pip tries to set up packages that require various
host tools just to run the setup.py script.
The patch here runs the update process in a container.
$ bazel run //tools/python:requirements.update
WARNING: Overflow when watching local filesystem for changes... temporarily falling back to manually checking files for changes
INFO: Build option --compilation_mode has changed, discarding analysis cache.
INFO: Analyzed target //tools/python:requirements.update (84 packages loaded, 27041 targets configured).
INFO: Found 1 target...
Target //tools/python:requirements.update up-to-date:
bazel-bin/tools/python/requirements.update
INFO: Elapsed time: 3.172s, Critical Path: 0.09s
INFO: 4 processes: 4 internal.
INFO: Build completed successfully, 4 total actions
INFO: Running command line: bazel-bin/tools/python/requirements.update tools/python/requirements.txt tools/python/requirements.lock.txt None None None //tools/python:requirements.update
Updating tools/python/requirements.lock.txt
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [45 lines of output]
Collecting setuptools
Downloading setuptools-69.0.3-py3-none-any.whl (819 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 819.5/819.5 kB 11.4 MB/s eta 0:00:00
Collecting wheel
Downloading wheel-0.42.0-py3-none-any.whl (65 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.4/65.4 kB 6.2 MB/s eta 0:00:00
Collecting pycairo
Downloading pycairo-1.25.1.tar.gz (347 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 347.1/347.1 kB 16.1 MB/s eta 0:00:00
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Installing backend dependencies: started
Installing backend dependencies: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: pycairo
Building wheel for pycairo (pyproject.toml): started
Building wheel for pycairo (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error
× Building wheel for pycairo (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [14 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-39
creating build/lib.linux-x86_64-cpython-39/cairo
copying cairo/__init__.py -> build/lib.linux-x86_64-cpython-39/cairo
copying cairo/__init__.pyi -> build/lib.linux-x86_64-cpython-39/cairo
copying cairo/py.typed -> build/lib.linux-x86_64-cpython-39/cairo
warning: build_py: byte-compiling is disabled, skipping.
running build_ext
'pkg-config' not found.
Command ['pkg-config', '--print-errors', '--exists', 'cairo >= 1.15.10']
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pycairo
Failed to build pycairo
ERROR: Could not build wheels for pycairo, which is required to install pyproject.toml-based projects
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/rules_python/python/pip_install/pip_compile.py", line 127, in <module>
cli()
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__click/click/core.py", line 1137, in __call__
return self.main(*args, **kwargs)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__click/click/core.py", line 1062, in main
rv = self.invoke(ctx)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__click/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__click/click/core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__click/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/scripts/compile.py", line 487, in cli
results = resolver.resolve(max_rounds=max_rounds)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 266, in resolve
has_changed, best_matches = self._resolve_one_round()
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 356, in _resolve_one_round
their_constraints.extend(self._iter_dependencies(best_match))
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 469, in _iter_dependencies
dependencies = self.repository.get_dependencies(ireq)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/repositories/local.py", line 85, in get_dependencies
return self.repository.get_dependencies(ireq)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/repositories/pypi.py", line 250, in get_dependencies
self._dependencies_cache[ireq] = self.resolve_reqs(
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip_tools/piptools/repositories/pypi.py", line 213, in resolve_reqs
results = resolver._resolve_one(reqset, ireq)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/resolution/legacy/resolver.py", line 509, in _resolve_one
dist = self._get_dist_for(req_to_install)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/resolution/legacy/resolver.py", line 462, in _get_dist_for
dist = self.preparer.prepare_linked_requirement(req)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 438, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 524, in _prepare_linked_requirement
dist = _get_prepared_distribution(
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 68, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
self._prepare_build_backend(finder)
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/distributions/sdist.py", line 70, in _prepare_build_backend
self.req.build_env.install_requirements(
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/build_env.py", line 196, in install_requirements
self._install_requirements(
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/build_env.py", line 254, in _install_requirements
call_subprocess(
File "/home/james/.cache/bazel/_bazel_james/a53c1076cfa8612afda853f7c8f2a68f/execroot/org_frc971/bazel-out/k8-fastbuild/bin/tools/python/requirements.update.runfiles/pypi__pip/pip/_internal/utils/subprocess.py", line 224, in call_subprocess
raise error
pip._internal.exceptions.InstallationSubprocessError: pip subprocess to install build dependencies exited with 1
Change-Id: Idc0e66994c7e0282f7ef8825f737fd407c0cc0a4
Signed-off-by: Philipp Schrader <philipp.schrader@gmail.com>
This is FRC Team 971's main code repository. There are README* files throughout the source tree documenting specifics for their respective folders.
All development of AOS is done on our gerrit instance at https://software.frc971.org/gerrit with github being a read-only mirror. We are happy to add external contributors. If you are interested, reach out to Austin Schuh or Stephan Massalt and we will help you get access. In case of disputes over if a patch should be taken or not, Austin has final say.
Submissions must be made under the terms of the following Developer Certificate of Origin.
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
To do this, add the following to your commit message. Gerrit will enforce that all commits have been signed off.
Signed-off-by: Random J Developer <random@developer.example.org>
Git has support for adding Signed-off-by lines by using git commit -s, or you can setup a git commit hook to automatically sign off your commits. Stack Overflow has instructions for how to do this if you are interested.
The main central location for our code is our Gerrit server at https://software.frc971.org/gerrit. If you are on a platform not compatible with our codebase follow the instructions here to set up access to the build server. To download a copy of the 971 code on your computer, follow these steps:
#coding channel in slack.ssh-keygen -t ed25519. This will create a public/private key pair-- the default location for your public key will be ~/.ssh/id_ed25519.pubSettings->SSH Keys and paste your public key into the New SSH Key text box and clicking on ADD NEW SSH KEYgit: sudo apt install gitclone with commit-msg hook commandclone with commit-msg hook will save you trouble later.To learn more about git, open a terminal and run man git, or see git(1) (especially the NOTES section).
The instructions below assume you have the following:
x86\_64 GNU/Linux system (e.g., Ubuntu 20.04), but that's not at all well-tested.vim, emacsssh into target CPU's like the roborio and Raspberry PiWe use Bazel to build the code. Bazel has extensive docs, including a nice build encyclopedia reference, and does a great job with fast, correct incremental rebuilds.
There are a couple options for building code that are given here-- setting up either your own computer, or using the frc971 build server.
Install any Bazel version:
apt-cache search bazel or just try sudo apt install bazelapt-key instead of gnupg in setting up the key:sudo apt install curl curl -fsSL https://bazel.build/bazel-release.pub.gpg | sudo apt-key add - echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
sudo apt update && sudo apt install bazel
Install the required packages:
sudo apt-get update sudo apt-get install python
Change settings to allow Bazel's sandboxing to work-- follow the directions in doc/frc971.conf. For example, the commands to do this would be:
sudo cp doc/frc971.conf /etc/sysctl.d/sudo sysctl --systemIn order to run certain tests, you need to give yourself permissions--follow the "Set up real-time niceties" section of aos/events/README.md.
In order to use the build server, you'll first need to get ssh access set up. (NOTE: you don't need to do any of the other setup steps done for your own computer, since things like bazel, python, etc. are already installed on the build server)
# In windows: # Important use Powershell! cd ~/.ssh ssh-keygen -t ed25519 -f id_971_ed25519 chmod 600 id_971_ed25519
# In Linux and MacOS: cd ~ ssh-keygen -t ed25519 -f ~/.ssh/id_971_ed25519 chmod 600 ./.ssh/id_971_ed25519
#coding channel in slack.id_971_ed25519.pubcat ~/.ssh/id_971_ed25519.pub # This output is your public SSH key for the request form.
ssh REPLACE_WITH_YOUR_USERNAME@build.frc971.org -p 2222 -i ~/.ssh/id_971_ed25519
exit command and then SSH tunnel.ssh REPLACE_WITH_YOUR_USERNAME@build.frc971.org -p 2222 -i ~/.ssh/id_971_ed25519 -L 9971:127.0.0.1:3389
127.0.0.1:9971 for the computer name, and use your gerrit username. Once you get connected, accept the server certificate and then enter your password that you gave Stephan. (It's either something unique or your gerrit pwd) Then select the Default panel config. You can exit the Remote Desktop if you are good w/ the raw cmd line interface. And for future logins all you have to do is tunnel and then login using the app. Now if you have a graphical application that you are developing (e.g., spline UI), then you have to run the build command in the Remote Desktop application.git. To do this, run the following command, replacing <YOUR EMAIL> with the email that you are using for Gerrit:git config --global user.email "<YOUR EMAIL>"
If there are any questions, post to the #coding Slack channel so that other people who may reach the same issue can refer back to that.
bazel test //... bazel build --config=roborio -c opt //...
# For the roborio: bazel build --config=roborio -c opt //y2020/...
# For the raspberry pi: bazel build --config=armv7 -c opt //y2020/...
bazel run -c opt //frc971/config:setup_roborio -- roboRIO-XXX-frc.local
# For the roborio bazel run --config=roborio -c opt //y2020:download_stripped -- roboRIO-971-frc.local
This assumes the roborio is reachable at roboRIO-971-frc.local. If that does not work, you can try with a static IP address like 10.9.71.2 (see troubleshooting below)
# For the raspberry pi's bazel run --config=armv7 -c opt //y2020:pi_download_stripped -- 10.9.71.101
NOTE:
download_stripped (or pi_download_stripped) targets. These will rsync only the changed files, and so are pretty efficient.//y2020/vision:calibration) for the pi (armv7), run:bazel run --config=armv7 -c opt //y2020/vision:calibration_strippedYou will then need to manually copy the resulting file over to the robot.
We want all code to at least have a second person look it over before it gets merged into the master branch. Gerrit has extensive documentation on starting reviews. There is also a good intro User Guide and an intro to working with Gerrit and Gerrit workflows
TL;DR: Make and commit your changes, do git push origin HEAD:refs/for/master, and then click on the provided link to add reviewers. If you just upload a change without adding any reviewers, it might sit around for a long time before anybody else notices it.
git-review can make the upload process simpler.
These aren't strictly necessary to build the code, but Michael found the additional tools provided by these packages useful to install when working with the code on May 13, 2018.
# Get some useful packages including git and subversion. apt-get update apt-get install git git-gui vim-gtk3 apt-get install vim-doc git-doc exim4-doc-html yapf apt-get install bazel clang-format apt-get install python avahi-daemon # Install apt-file so that packages can be searched apt-get install apt-file apt-file update # Install sysstat so that you can tell how many resources are being used during # the compile. apt-get install sysstat # iostat is used to observe how hard the disk is being worked and other # performance metrics. iostat -dx 1 # gitg is a graphical user interface for git. I find it useful for # understanding the revision history of the repository and viewing # log messages and changes. apt-get install gitg # Also may want to install `buildifier` for formatting BUILD files
It is also handy to alias logins to the raspberry pi's by adding lines like this to your ~/.ssh/config file:
Host pi-7971-2 User pi ForwardAgent yes HostName 10.79.71.102 StrictHostKeyChecking no
or, for the roborio:
Host roborio-971
User admin
HostName 10.9.71.2
StrictHostKeyChecking no
This allows you to use the alias to ping, ssh, or run commands like:
# Download code to robot #7971's raspberry pi #2 bazel run --config=armv7 -c opt //y2020:download_stripped -- pi-7971-2
Currently (as of 2020.02.26), top tends to produce misleading statistics. As such, you can get more useful information about CPU usage by using kernel traces. Sample usage:
# Note that you will need to install the trace-cmd package on the roborio. # This may be not be a trivial task. # Start the trace trace-cmd start -e sched_switch -e workqueue # Stop the trace trace-cmd stop # Save the trace to trace.dat trace-cmd extract
You can then scp the trace.dat file to your computer and run kernelshark trace.dat (may require installing the kernelshark apt package).
If the roboRIO has been configued to use a static IP address like 10.9.71.2, set the laptop to have an IP address on the 10.9.71.x subnet with a netmask of 255.0.0.0. The ".x" is different than the .2 for the roboRIO or any other device on the network. The driver station uses .5 or .6 so avoid those. The radio uses .1 or .50 so avoid those too. A good choice might be in the 90-99 range. If you are at the school, disconnect from the student wireless network or try setting your netmask to 255.255.255.0 if you want to be on both networks. The student wireless network is on a 10.?.?.? subnet which can cause problems with connecting to the robot.
If running Bazel on the download_stripped target does not work for the IP address you're using for the roborio or the raspberry pi, it probably means that the robot and laptop are on different subnets. They need to be on the same subnet for the laptop to connect to the robot. Connecting can be confirmed by using ping.
ping roboRIO-971-frc.local
or
ping 10.9.71.2
If this does not work, perhaps the roboRIO has not been configured to have a static IP address. Use a USB cable to connect from a Windows laptop to the roboRIO and use a web browser (Chrome is preferred, IE/Edge is not-- see this technical note) to configure the roboRIO to have a static IP address of 10.9.71.2. Browse to http://roborio-971-frc.local or http://172.22.11.2. Click on the "Ethernet" icon on the left, select "Static" for the "Configure IPv4 Address" option. Set the "IPv4 Address" to 10.9.71.2. Set the "Subnet Mask" to "255.0.0.0". Finally click on "Save" at the bottom of the screen. If you have trouble using an Ethernet cable, try using a USB cable (USB A->B).
Another option is to configure the laptop to have a link-local connection by using the "Network Settings" GUI. The laptop will then be on the same subnet in the address range of 169.254.0.0 to 169.254.255.255. James thinks this will only work over Ethernet (i.e., not USB; he is not sure what will happen if you attempt this over USB), and if the robot does not have a static IP address set and there is no DHCP server assigning an IP address to the roboRIO. James says to also note that this implies that the roboRIO will also have a 169.254.. IP addresss, and that the only simple way to figure it out is to use mDNS.
You can run bazel run //tools:gen_rust_project to generate a rust-project.json file which rust-analyzer will pick up. You will need to execute this rule periodically as it will be outdated whenever the BUILD files change.
Note that there's currently no way to tell rust-analyzer how to compile the code, so while it will give you completion support, go to definition, and other niceties, it won't show compilation errors or warnings at this point.
TODOs:
apt install libsigsegv2, but this should be temporary