summaryrefslogtreecommitdiff
path: root/guix/scripts/offload.scm
Commit message (Collapse)AuthorAge
* ssh: Add 'authenticate-server*' and use it for offloading.Ludovic Courtès2019-12-04
| | | | | | | | * guix/scripts/offload.scm (host-key->type+key): Remove. (open-ssh-session): Replace server authentication code with a call to 'authenticate-server*'. * guix/ssh.scm (host-key->type+key, authenticate-server*): New procedures.
* guix build, daemon: Rename "--no-build-hook" to "--no-offload".Ludovic Courtès2019-11-26
| | | | | | | | | | | | | | | | This is a followup to bc69ea2d605810cc32e13ed03d5848b8dc358b61. * guix/scripts/build.scm (show-build-options-help): Rename "--no-build-hook" to "--no-offload". (%standard-build-options): Likewise, and warn when "--no-build-hook" is passed. * nix/nix-daemon/guix-daemon.cc (options): Add "--no-offload" and mark "--no-build-hook" as hidden. * guix/scripts/offload.scm: Adjust comment. * doc/guix.texi (Invoking guix-daemon, Common Build Options): Replace "--no-build-hook" with "--no-offload". * etc/completion/fish/guix.fish, etc/completion/zsh/_guix: Adjust accordingly.
* offload: Set a longer SSH session timeout.Ludovic Courtès2019-10-15
| | | | | | | | | | | | Fixes <https://bugs.gnu.org/37762>. * guix/scripts/offload.scm (open-ssh-session): Add 'max-silent-time' parameter. Add call to 'session-set!' before returning SESSION. (transfer-and-offload): Pass MAX-SILENT-TIME to 'open-ssh-session'. (%short-timeout): New variable. (choose-build-machine): Pass %SHORT-TIMEOUT to 'open-ssh-session'. (check-machine-availability): Likewise. (check-machine-status): Likewise.
* offload: Include the port number in the machine lock file name.Ludovic Courtès2019-09-28
| | | | | | | | This is useful when a single machine appears several time, with different port numbers. * guix/scripts/offload.scm (machine-slot-file): Add MACHINE's port to the file name.
* syscalls: Add 'with-file-lock' macro.Ludovic Courtès2019-06-05
| | | | | | * guix/scripts/offload.scm (lock-file, unlock-file, with-file-lock): Move to... * guix/build/syscalls.scm: ... here.
* offload: 'status' reports the time difference.Ludovic Courtès2019-01-22
| | | | | * guix/scripts/offload.scm (check-machine-status): Report the time difference for each MACHINE.
* store: Rename '&nix-error' to '&store-error'.Ludovic Courtès2019-01-21
| | | | | | | | | | | | | | | | | | | * guix/store.scm (&nix-error): Rename to... (&store-error): ... this, and adjust users. (&nix-connection-error): Rename to... (&store-connection-error): ... this, and adjust users. (&nix-protocol-error): Rename to... (&store-protocol-error): ... this, adjust users. (&nix-error, &nix-connection-error, &nix-protocol-error): Define these condition types and their getters as deprecrated aliases. * build-aux/run-system-tests.scm, guix/derivations.scm, guix/grafts.scm, guix/scripts/challenge.scm, guix/scripts/graph.scm, guix/scripts/lint.scm, guix/scripts/offload.scm, guix/serialization.scm, guix/ssh.scm, guix/tests.scm, guix/ui.scm, tests/derivations.scm, tests/gexp.scm, tests/guix-daemon.sh, tests/packages.scm, tests/store.scm, doc/guix.texi: Adjust to use the new names.
* offload: Remove unnecessary locking on machine slots.Ludovic Courtès2018-12-26
| | | | | | | | This extra level of locking turned out to be unnecessary. * guix/scripts/offload.scm (with-machine-lock): Remove. (machine-lock-file): Remove. (acquire-build-slot): Remove surrounding 'with-machine-lock'.
* offload: Remove the "machine choice" lock.Ludovic Courtès2018-12-26
| | | | | | | | This lock was unnecessary and it led to a contention when many 'guix offload' processes are polling for available machines. * guix/scripts/offload.scm (machine-choice-lock-file): Remove. (choose-build-machine): Remove surrounding 'with-file-lock (machine-lock-file)'.
* offload: Adjust 'test' and 'status' to the latest changes.Ludovic Courtès2018-12-25
| | | | | | | | | | | | This is a followup to ed7b44370f71126087eb953f36aad8dc4c44109f; following that commit, 'guix offload test' and 'guix offload status' would abort with a backtrace instead of clearly diagnosing a missing 'guix' command on the build machine. * guix/scripts/offload.scm (assert-node-has-guix): Call 'leave' when NODE is not an inferior. Remove 'catch' blocks for 'node-repl-error'. (check-machine-availability): Invoke 'assert-node-has-guix' first. (check-machine-status): Print a warning when 'remote-inferior' returns #f.
* offload: Use (guix inferior) instead of (ssh dist node).Ludovic Courtès2018-12-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Using inferiors and thus 'guix repl' simplifies setup on build machines (no need to worry about GUILE_LOAD_PATH etc.) Furthermore, the 'guix repl -t machine' protocol running in a remote pipe addresses several issues with the current implementation of nodes and RREPLs in Guile-SSH: fewer round trips, doesn't leave a 'guile --listen' process behind it, stateless (since a new process is started each time), more efficient (the SSH channel can be reused), more reliable (no 'pgrep', 'pkill', and shellology; see <https://github.com/artyom-poptsov/guile-ssh/issues/11> as an example.) * guix/ssh.scm (inferior-remote-eval): New procedure. (send-files): Use it instead of 'make-node' and 'node-eval'. * guix/scripts/offload.scm (node-guile-version): New procedure. (node-free-disk-space, transfer-and-offload, node-load) (choose-build-machine, assert-node-has-guix): Use 'remote-inferior' instead of 'make-node' and 'inferior-eval' instead of 'node-eval'. (assert-node-can-import, assert-node-can-export): Likewise, and add 'session' parameter. (check-machine-availability): Likewise, and add calls to 'close-inferior' and 'disconnect!'. (check-machine-status): Likewise. * doc/guix.texi (Daemon Offload Setup): Remove bit related to 'guile' in $PATH and $GUILE_LOAD_PATH; mention 'guix' alone.
* offload: Display the normalized load in 'guix offload status' output.Ludovic Courtès2018-12-23
| | | | | | | | | Fixes a regression introduced in bbe66a530a014e8146d63002a5294941e935f863 whereby the actual load (non-normalized) would be displayed. * guix/scripts/offload.scm (check-machine-status): Add call to 'normalized-load'.
* offload: Recognize build failures due to lack of disk space.Ludovic Courtès2018-12-21
| | | | | | | | | | Previously, if a remote build would fail due to lack of disk space, this would be considered a permanent failure and thus cached as a build failure if the local daemon runs with '--cache-failures'. * guix/scripts/offload.scm (transfer-and-offload): Upon 'nix-protocol-error?' call 'node-free-disk-space' and return 1 instead of 100 if the result if lower than 10 MiB.
* offload: Skip machines that are low on disk space.Ludovic Courtès2018-12-21
| | | | | | | | | | Fixes <https://bugs.gnu.org/33378>. * guix/scripts/offload.scm (node-free-disk-space): New procedure. (%minimum-disk-space): New variable. (choose-build-machine): Call 'node-free-disk-space' and take it into account in addition to LOAD. (check-machine-status): Display the free disk space.
* offload: Decompose 'machine-load' into simpler procedures.Ludovic Courtès2018-12-21
| | | | | | | | | * guix/scripts/offload.scm (machine-load): Remove. (node-load, normalized-load): New procedures. (choose-build-machine): Call 'open-ssh-session' and 'make-node' from here; pass the node to 'node-load'. (check-machine-status): Use 'node-load' instead of 'machine-load'. Call 'disconnect!' on SESSION.
* offload: Fix error message in 'guix offload test'.Ludovic Courtès2018-06-15
| | | | | | | | Reported by Maxim Cournoyer <maxim.cournoyer@gmail.com> in <https://bugs.gnu.org/31824>. * guix/scripts/offload.scm (assert-node-has-guix): Fix typo in failure message; add missing argument.
* offload: Gracefully handle invalid results from 'machines.scm'.Ludovic Courtès2018-06-14
| | | | | * guix/scripts/offload.scm (build-machines): Check the result of FILE. Ignore it if it's not a list of <build-machine>.
* offload: Honor the build timeout internally.Ludovic Courtès2018-06-11
| | | | | | * guix/scripts/offload.scm (call-with-timeout): New procedure. (with-timeout): New macro. (process-request): Use it around 'transfer-and-offload' call.
* offload: Look at machine loads for the past minute.Ludovic Courtès2018-01-15
| | | | | | | | | Previously we were looking at the load of the past 5 minutes, which means that, after a build, we could end up waiting for 5 minutes for that metric to be low enough. * guix/scripts/offload.scm (machine-load): Compute RAW based on ONE, not FIVE.
* offload: 'test' reports Guile and module errors more nicely.Ludovic Courtès2018-01-12
| | | | | | | | | | | Fixes <https://bugs.gnu.org/26008>. Reported by Myles English <mylesenglish@gmail.com>. * guix/ssh.scm (retrieve-files*): Move error reporting to... (report-guile-error, report-module-error): ... here. New procedures. * guix/scripts/offload.scm (assert-node-repl): Use 'report-guile-error'. (assert-node-has-guix): Explicitly check for 'use-modules' first. Use 'report-module-error'.
* offload: 'test' gracefully handles 'node-repl-error'.Ludovic Courtès2018-01-12
| | | | | | | | Fixes <https://bugs.gnu.org/28057>. Reported by Ricardo Wurmus <rekado@elephly.net>. * guix/scripts/offload.scm (assert-node-has-guix): Catch 'node-repl-error' and call 'leave'.
* offload: Fix regression in file retrieval.Ludovic Courtès2018-01-12
| | | | | | | | | | | | | | | | This fixes a regression in 'retrieve-files*' introduced in 896fec476f728183b331cbb6e2afb891207b4205, whereby (guix scripts offload) would not read the initial sexp now sent by the remote host via 'store-export-channel'. This would effectively prevent file retrieval entirely when offloading. * guix/ssh.scm (retrieve-files*): New procedure, like former 'retrieve-files' but with an extra #:import parameter. (retrieve-files): Rewrite in terms of 'retrieve-files*'. (file-retrieval-port): Make private. * guix/scripts/offload.scm (transfer-and-offload): Pass #:import to 'retrieve-files*'. (retrieve-files*): Remove.
* guix: offload: Add "status" sub-command.Ricardo Wurmus2017-12-16
| | | | | | * guix/scripts/offload.scm (check-machine-status): New procedure. (guix-offload): Call it when the argument is "status". * doc/guix.texi (Daemon Offload Setup): Document it.
* Do not set '%fresh-auto-compile'.Ludovic Courtès2017-12-01
| | | | | | | | * guix/scripts/offload.scm (build-machines): Comment out '(set! %fresh-auto-compile #t)' since with Guile 2.2.3 it could lead to an actual rebuild of everything that gets loaded from there on. See <https://bugs.gnu.org/29226>. * guix/ui.scm (load*): Likewise.
* offload: Reduce the number of calls to 'machine-load'.Ludovic Courtès2017-10-12
| | | | | | | | | | | | | | | Previously we would call 'machine-load' once per machine, which was very costly when there were many machines. Now we arrange to call it only once on average (when all the machines have the same 'speed' value). * guix/scripts/offload.scm (random-seed, shuffle): New procedures. (choose-build-machine)[machines+slots+loads]: Rename to... [machines+slots]: ... this. Remove load from the tuples therein. [undecorate]: Adjust accordingly. [machine-less-loaded-or-faster?]: Remove. [machine-faster?]: New procedure. Sort MACHINES+SLOTS according to 'machine-faster?'. Call 'machine-load?' as the last thing.
* offload: Fix potential file descriptor and memory leak.Ludovic Courtès2017-07-25
| | | | | | | | | | | The '%slots' list could grow indefinitely; in practice though, guix-daemon is likely to restart 'guix offload' often enough. * guix/scripts/offload.scm (%slots): Remove. (choose-build-machine): Don't 'set!' %SLOTS. Return the acquired slot as a second value. (process-request): Adjust accordingly. Release the returned slot after 'transfer-and-offload'.
* offload: Disconnect sessions created by 'machine-load'.Ludovic Courtès2017-07-25
| | | | | | | | This fixes a memory leak that can be seen by running: (map (lambda _ (machine-load m)) (iota 1000)) * guix/scripts/offload.scm (machine-load): Add call to 'disconnect!'.
* derivations: Introduce 'read-derivation-from-file'.Ludovic Courtès2017-06-12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This avoids the open/fstat/close syscalls upon a cache hit that we had with the previous idiom: (call-with-input-file file read-derivation) where caching happened in 'read-derivation' itself. * guix/derivations.scm (%read-derivation): Rename to... (read-derivation): ... this. (read-derivation-from-file): New procedure. (derivation-prerequisites, substitution-oracle) (derivation-prerequisites-to-build): (derivation-path->output-path, derivation-path->output-paths): (derivation-path->base16-hash, map-derivation): Use 'read-derivation-from-file' instead of (call-with-input-file … read-derivation). * guix/grafts.scm (item->deriver): Likewise. * guix/scripts/build.scm (log-url, options->things-to-build): Likewise. * guix/scripts/graph.scm (file->derivation): Remove. (derivation-dependencies, %derivation-node-type): Use 'read-derivation-from-file' instead. * guix/scripts/offload.scm (guix-offload): Likewise. * guix/scripts/perform-download.scm (guix-perform-download): Likewise. * guix/scripts/publish.scm (load-derivation): Remove. (narinfo-string): Use 'read-derivation-from-file'.
* scripts: Set thread names.Ludovic Courtès2017-05-28
| | | | | | | | | | | | | This allows 'guix publish' threads as well as 'guix substitute' and 'guix offload' processes to be properly labeled in 'top', 'pstree', etc. * guix/workers.scm (worker-thunk): Add #:thread-name parameter and honor it. (make-pool): Likewise. * guix/scripts/publish.scm (http-write): Add calls to 'set-thread-name' in bodies of 'call-with-new-thread'. (guix-publish): Call 'set-thread-name'. Pass #:thread-name to 'make-pool'. * guix/scripts/offload.scm (guix-offload): Call 'set-thread-name'. * guix/scripts/substitute.scm (guix-substitute): Likewise.
* ui: Rename '_' to 'G_'.Ludovic Courtès2017-05-03
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This avoids collisions with '_' when the latter is used as a 'match' pattern for instance. See <https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00464.html>. * guix/ui.scm: Rename '_' to 'G_'. * po/guix/Makevars (XGETTEXT_OPTIONS): Adjust accordingly. * build-aux/compile-all.scm (warnings): Remove 'format'. * gnu/packages.scm, gnu/services.scm, gnu/services/shepherd.scm, gnu/system.scm, gnu/system/shadow.scm, guix/gnupg.scm, guix/http-client.scm, guix/import/cpan.scm, guix/import/elpa.scm, guix/import/pypi.scm, guix/nar.scm, guix/scripts.scm, guix/scripts/archive.scm, guix/scripts/authenticate.scm, guix/scripts/build.scm, guix/scripts/challenge.scm, guix/scripts/container.scm, guix/scripts/container/exec.scm, guix/scripts/copy.scm, guix/scripts/download.scm, guix/scripts/edit.scm, guix/scripts/environment.scm, guix/scripts/gc.scm, guix/scripts/graph.scm, guix/scripts/hash.scm, guix/scripts/import.scm, guix/scripts/import/cpan.scm, guix/scripts/import/cran.scm, guix/scripts/import/crate.scm, guix/scripts/import/elpa.scm, guix/scripts/import/gem.scm, guix/scripts/import/gnu.scm, guix/scripts/import/hackage.scm, guix/scripts/import/nix.scm, guix/scripts/import/pypi.scm, guix/scripts/import/stackage.scm, guix/scripts/lint.scm, guix/scripts/offload.scm, guix/scripts/pack.scm, guix/scripts/package.scm, guix/scripts/perform-download.scm, guix/scripts/publish.scm, guix/scripts/pull.scm, guix/scripts/refresh.scm, guix/scripts/size.scm, guix/scripts/substitute.scm, guix/scripts/system.scm, guix/ssh.scm, guix/upstream.scm: Use 'G_' instead of '_'. Most of this change was obtained by running: "sed -i -e's/(_ "/(G_ "/g' `find -name \*.scm`".
* offload: Avoid using '_' as a 'match' pattern.Ludovic Courtès2017-04-21
| | | | | * guix/scripts/offload.scm (host-key->type+key, machine-load) (process-request, guix-offload): Do not use '_' as a 'match' pattern.
* Add (guix ssh) module.Ludovic Courtès2016-12-31
| | | | | | | | | | | | | | | | | * guix/scripts/offload.scm (connect-to-remote-daemon) (store-import-channel, store-export-channel, send-files) (retrieve-files): Move to (guix ssh). (nonce): Add optional 'name' parameter and use it. (retrieve-files*): New procedure. (transfer-and-offload): Use it instead of 'retrieve-files', and add first parameter to 'send-files'. (assert-node-can-import): Likewise. (assert-node-can-export): Use 'retrieve-files' instead of 'store-export-channel'. * guix/ssh.scm: New file. * configure.ac: Use 'GUIX_CHECK_GUILE_SSH' and define 'HAVE_GUILE_SSH' Automake conditional. * Makefile.am (MODULES) [HAVE_GUILE_SSH]: Add guix/ssh.scm.
* offload: Allow testing machines that match a regexp.Ludovic Courtès2016-12-09
| | | | | | | * guix/scripts/offload.scm (check-machine-availability): Add 'pred' parameter and honor it. (guix-offload): for the "test" sub-command, accept an extra 'regexp' parameter. Pass a second argument to 'check-machine-availability'.
* offload: Test each machine only once.Ludovic Courtès2016-12-09
| | | | | | * guix/scripts/offload.scm (check-machine-availability)[build-machine=?]: New procedure. Add call to 'delete-duplicates'.
* offload: Do not read ~/.ssh/known_hosts.Ludovic Courtès2016-12-09
| | | | | * guix/scripts/offload.scm (open-ssh-session): Pass #:knownhosts to 'make-session'.
* offload: Increase the connection timeout.Ludovic Courtès2016-12-06
| | | | * guix/scripts/offload.scm (open-ssh-session): Set #:timeout to 10.
* offload: Send the build log to the right file descriptor.Ludovic Courtès2016-12-06
| | | | | | | | | | This fixes a regression introduced in 21531add3205e400707c8fbfd841845f9a71863a whereby the build log would no longer be sent to FD 4, thereby leading the daemon to not see the build log. * guix/scripts/offload.scm (transfer-and-offload): Parameterize CURRENT-BUILD-OUTPUT-PORT.
* offload: Fix plural of some messages.Ludovic Courtès2016-12-05
| | | | | | * guix/scripts/offload.scm (send-files): Use 'N_' for possibly plural message. Write "store item" instead of "store file". (retrieve-files): Likewise.
* offload: Make the compression methods configurable.Ludovic Courtès2016-12-05
| | | | | | | * guix/scripts/offload.scm (<build-machine>)[compression] [compression-level]: New fields. (open-ssh-session): Honor them. * doc/guix.texi (Daemon Offload Setup): Document them.
* offload: Add "test" sub-command.Ludovic Courtès2016-12-05
| | | | | | | | | | * guix/scripts/offload.scm (assert-node-repl, assert-node-has-guix) (nonce, assert-node-can-import, assert-node-can-export) (check-machine-availability): New procedures. (%random-state): New variable. (guix-offload): Add case for "test". * doc/guix.texi (Daemon Offload Setup): Document it. Remove obsolete bit about remote invocation of 'guix build'.
* offload: Do not abort when a machine is unreachable.Ludovic Courtès2016-12-01
| | | | | * guix/scripts/offload.scm (machine-load): Wrap 'open-ssh-session' call in 'false-if-exception'; return +inf.0 if it returns #f.
* offload: Gracefully report connection failures.Ludovic Courtès2016-12-01
| | | | | * guix/scripts/offload.scm (open-ssh-session): Check the return value of 'connect!'. Call 'leave' when it's not 'ok.
* offload: Warn about the lack of zlib support.Ludovic Courtès2016-12-01
| | | | | * guix/scripts/offload.scm (guix-offload): Print a warning when 'zlib-support?' returns false.
* offload: Remove redundant call to 'topologically-sorted' in 'send-file'.Ludovic Courtès2016-12-01
| | | | | * guix/scripts/offload.scm (send-files): Remove call to 'topologically-sorted'.
* offload: Call 'machine-load' only once per machine.Ludovic Courtès2016-11-26
| | | | | | | | | | | | | | This fixes a longstanding issue where 'choose-build-machine' would make on average O(N log(N)) calls to 'machine-load', plus an extra call for the selected machine, instead of N calls. * guix/scripts/offload.scm (machine-load): Add comment. (machine-power-factor, machine-less-loaded-or-faster?): Remove. (choose-build-machine)[machines+slots]: Rename to... [machines+slots+loads]: ... this. [undecorate]: Adjust accordingly. [machine-less-loaded-or-faster?]: New procedure. Remove extra 'machine-load' call in body.
* offload: Drop 'remote-pipe'.Ludovic Courtès2016-11-25
| | | | | * guix/scripts/offload.scm (remote-pipe): Remove. (machine-load): Use 'open-remote-pipe*' instead of 'remote-pipe'.
* offload: Rewrite to make direct RPCs to the remote daemon.Ludovic Courtès2016-11-25
| | | | | | | | | | * guix/scripts/offload.scm (<build-machine>)[daemon-socket]: New field. (connect-to-remote-daemon): New procedure. (%gc-root-file, register-gc-root, remove-gc-roots, offload): Remove. (transfer-and-offload): Rewrite using 'connect-to-remote-daemon' and RPCs over SSH. (store-import-channel, store-export-channel): New procedures. (send-files, retrieve-files): Rewrite using these.
* offload: Remove 'with-nar-error-handling' macro.Ludovic Courtès2016-11-25
| | | | | * guix/scripts/offload.scm (with-nar-error-handling): Remove. (guix-offload): Use 'with-error-handling' instead.
* offload: Reuse SSH session during 'transfer-and-offload'.Ludovic Courtès2016-11-25
| | | | | | | | | | | * guix/scripts/offload.scm (remote-pipe): Replace 'machine' parameter with 'session'. Remove 'open-ssh-session' call. (register-gc-root): Replace 'machine' with 'session'. Use ' session-get' instead of 'build-machine-name'. (remove-gc-roots, offload, send-files, retrieve-files): Likewise. (transfer-and-offload): Add 'open-ssh-session' call. Handle 'offload' errors here. (machine-load): Add call to 'open-ssh-session'.
* offload: Use Guile-SSH instead of GNU lsh.Ludovic Courtès2016-11-25
| | | | | | | | | | | | | | | | | | | | | * guix/scripts/offload.scm (<build-machine>)[ssh-options]: Remove. [host-key, host-key-type]: New fields. (%lsh-command, %lshg-command, user-lsh-private-key): Remove. (user-openssh-private-key, private-key-from-file*): New procedures. (host-key->type+key, open-ssh-session): New procedures. (remote-pipe): Remove 'mode' parameter. Rewrite in terms of 'open-ssh-session' etc. Update users. (send-files)[missing-files]: Rewrite using the bidirectional channel port. Remove call to 'call-with-compressed-output-port'. (retrieve-files): Remove call to 'call-with-decompressed-port'. (machine-load): Remove exit status logic. * doc/guix.texi (Requirements): Mention Guile-SSH. (Daemon Offload Setup): Document 'host-key' and 'private-key'. Show the default value on each @item line. * m4/guix.m4 (GUIX_CHECK_GUILE_SSH): New macro. * config-daemon.ac: Use 'GUIX_CHECK_GUILE_SSH'. Set 'HAVE_DAEMON_OFFLOAD_HOOK' as a function of that.
generated by cgit v1.2.3 (git 2.45.0) at 2024-06-16 12:44:42 +0000