1CONDOR_SSH_TO_JOB(1) HTCondor Manual CONDOR_SSH_TO_JOB(1)
2
6 condor_ssh_to_job - HTCondor Manual
7 create an ssh session to a running job
9
12 condor_ssh_to_job [-help ]
13 condor_ssh_to_job [-debug ] [-name schedd-name] [-pool pool-name] [-ssh
15 ssh-command] [-keygen-options ssh-keygen-options] [-shells
16 shell1,shell2,...] [-auto-retry ] [-remove-on-interrupt ] cluster |
17 cluster.process | cluster.process.node [remote-command ]
18
20 condor_ssh_to_job creates an ssh session to a running job. The job is
21 specified with the argument. If only the job cluster id is given, then
22 the job process id defaults to the value 0.
23 condor_ssh_to_job is available in Unix HTCondor distributions, and
25 works with two kinds of jobs: those in the vanilla, vm, java, local, or
26 parallel universes, and those jobs in the grid universe which use EC2
27 resources. It will not work with other grid universe jobs.
28 For jobs in the vanilla, vm, java, local, or parallel universes, the
30 user must be the owner of the job or must be a queue super user, and
31 both the condor_schedd and condor_starter daemons must allow con‐
32 dor_ssh_to_job access. If no remote-command is specified, an interac‐
33 tive shell is created. An alternate ssh program such as sftp may be
34 specified, using the -ssh option, for uploading and downloading files.
35 The remote command or shell runs with the same user id as the running
37 job, and it is initialized with the same working directory. The envi‐
38 ronment is initialized to be the same as that of the job, plus any
39 changes made by the shell setup scripts and any environment variables
40 passed by the ssh client. In addition, the environment variable _CON‐
41 DOR_JOB_PIDS is defined. It is a space-separated list of PIDs associ‐
42 ated with the job. At a minimum, the list will contain the PID of the
43 process started when the job was launched, and it will be the first
44 item in the list. It may contain additional PIDs of other processes
45 that the job has created.
46 The ssh session and all processes it creates are treated by HTCondor as
48 though they are processes belonging to the job. If the slot is pre‐
49 empted or suspended, the ssh session is killed or suspended along with
50 the job. If the job exits before the ssh session finishes, the slot
51 remains in the Claimed Busy state and is treated as though not all job
52 processes have exited until all ssh sessions are closed. Multiple ssh
53 sessions may be created to the same job at the same time. Resource con‐
54 sumption of the sshd process and all processes spawned by it are moni‐
55 tored by the condor_starter as though these processes belong to the
56 job, so any policies such as PREEMPT that enforce a limit on resource
57 consumption also take into account resources consumed by the ssh ses‐
58 sion.
59 condor_ssh_to_job stores ssh keys in temporary files within a newly
61 created and uniquely named directory. The newly created directory will
62 be within the directory defined by the environment variable TMPDIR.
63 When the ssh session is finished, this directory and the ssh keys con‐
64 tained within it are removed.
65 See the HTCondor administrator's manual section on configuration for
67 details of the configuration variables related to condor_ssh_to_job.
68 An ssh session works by first authenticating and authorizing a secure
70 connection between condor_ssh_to_job and the condor_starter daemon,
71 using HTCondor protocols. The condor_starter generates an ssh key pair
72 and sends it securely to condor_ssh_to_job. Then the condor_starter
73 spawns sshd in inetd mode with its stdin and stdout attached to the TCP
74 connection from condor_ssh_to_job. condor_ssh_to_job acts as a proxy
75 for the ssh client to communicate with sshd, using the existing connec‐
76 tion authorized by HTCondor. At no point is sshd listening on the net‐
77 work for connections or running with any privileges other than that of
78 the user identity running the job. If CCB is being used to enable con‐
79 nectivity to the execute node from outside of a firewall or private
80 network, condor_ssh_to_job is able to make use of CCB in order to form
81 the ssh connection.
82 The login shell of the user id running the job is used to run the
84 requested command, sshd subsystem, or interactive shell. This is
85 hard-coded behavior in OpenSSH and cannot be overridden by configura‐
86 tion. This means that condor_ssh_to_job access is effectively disabled
87 if the login shell disables access, as in the example programs
88 /bin/true and /sbin/nologin.
89 condor_ssh_to_job is intended to work with OpenSSH as installed in typ‐
91 ical environments. It does not work on Windows platforms. If the ssh
92 programs are installed in non-standard locations, then the paths to
93 these programs will need to be customized within the HTCondor configu‐
94 ration. Versions of ssh other than OpenSSH may work, but they will
95 likely require additional configuration of command-line arguments,
96 changes to the sshd configuration template file, and possibly modifica‐
97 tion of the $(LIBEXEC)/condor_ssh_to_job_sshd_setup script used by the
98 condor_starter to set up sshd.
99 For jobs in the grid universe which use EC2 resources, a request that
101 HTCondor have the EC2 service create a new key pair for the job by
102 specifying ec2_keypair_file causes condor_ssh_to_job to attempt to con‐
103 nect to the corresponding instance via ssh. This attempts invokes ssh
104 directly, bypassing the HTCondor networking layer. It supplies ssh with
105 the public DNS name of the instance and the name of the file with the
106 new key pair's private key. For the connection to succeed, the instance
107 must have started an ssh server, and its security group(s) must allow
108 connections on port 22. Conventionally, images will allow logins using
109 the key pair on a single specific account. Because ssh defaults to log‐
110 ging in as the current user, the -l <username> option or its equivalent
111 for other versions of ssh will be needed as part of the remote-command
112 argument. Although the -X option does not apply to EC2 jobs, adding -X
113 or -Y to the remote-command argument can duplicate the effect.
114
116 -help Display brief usage information and exit.
117 -debug Causes debugging information to be sent to stderr, based on
119 the value of the configuration variable TOOL_DEBUG.
120 -name schedd-name
122 Specify an alternate condor_schedd, if the default (local)
123 one is not desired.
124 -pool pool-name
126 Specify an alternate HTCondor pool, if the default one is not
127 desired. Does not apply to EC2 jobs.
128 -ssh ssh-command
130 Specify an alternate ssh program to run in place of ssh, for
131 example sftp or scp. Additional arguments are specified as
132 ssh-command. Since the arguments are delimited by spaces,
133 place double quote marks around the whole command, to prevent
134 the shell from splitting it into multiple arguments to con‐
135 dor_ssh_to_job. If any arguments must contain spaces,
136 enclose them within single quotes. Does not apply to EC2
137 jobs.
138 -keygen-options ssh-keygen-options
140 Specify additional arguments to the ssh_keygen program, for
141 creating the ssh key that is used for the duration of the
142 session. For example, a different number of bits could be
143 used, or a different key type than the default. Does not
144 apply to EC2 jobs.
145 -shells shell1,shell2,...
147 Specify a comma-separated list of shells to attempt to
148 launch. If the first shell does not exist on the remote
149 machine, then the following ones in the list will be tried.
150 If none of the specified shells can be found, /bin/sh is used
151 by default. If this option is not specified, it defaults to
152 the environment variable SHELL from within the con‐
153 dor_ssh_to_job environment. Does not apply to EC2 jobs.
154 -auto-retry
156 Specifies that if the job is not yet running, con‐
157 dor_ssh_to_job should keep trying periodically until it suc‐
158 ceeds or encounters some other error.
159 -remove-on-interrupt
161 If specified, attempt to remove the job from the queue if
162 condor_ssh_to_job is interrupted via a CTRL-c or otherwise
163 terminated abnormally.
164 -X Enable X11 forwarding. Does not apply to EC2 jobs.
166 -x Disable X11 forwarding.
168
170 % condor_ssh_to_job 32.0
171 Welcome to slot2@tonic.cs.wisc.edu!
172 Your condor job is running with pid(s) 65881.
173 % gdb -p 65881
174 (gdb) where
175 ...
176 % logout
177 Connection to condor-job.tonic.cs.wisc.edu closed.
178 To upload or download files interactively with sftp:
180 % condor_ssh_to_job -ssh sftp 32.0
182 Connecting to condor-job.tonic.cs.wisc.edu...
183 sftp> ls
184 ...
185 sftp> get outputfile.dat
186 This example shows downloading a file from the job with scp. The string
188 "remote" is used in place of a host name in this example. It is not
189 necessary to insert the correct remote host name, or even a valid one,
190 because the connection to the job is created automatically. Therefore,
191 the placeholder string "remote" is perfectly fine.
192 % condor_ssh_to_job -ssh scp 32 remote:outputfile.dat .
194 This example uses condor_ssh_to_job to accomplish the task of running
196 rsync to synchronize a local file with a remote file in the job's work‐
197 ing directory. Job id 32.0 is used in place of a host name in this
198 example. This causes rsync to insert the expected job id in the argu‐
199 ments to condor_ssh_to_job.
200 % rsync -v -e "condor_ssh_to_job" 32.0:outputfile.dat .
202 Note that condor_ssh_to_job was added to HTCondor in version 7.3. If
204 one uses condor_ssh_to_job to connect to a job on an execute machine
205 running a version of HTCondor older than the 7.3 series, the command
206 will fail with the error message
207 Failed to send CREATE_JOB_OWNER_SEC_SESSION to starter
209
211 condor_ssh_to_job will exit with a non-zero status value if it fails to
212 set up an ssh session. If it succeeds, it will exit with the status
213 value of the remote command or shell.
214
216 HTCondor Team
217
219 1990-2020, Center for High Throughput Computing, Computer Sciences
220 Department, University of Wisconsin-Madison, Madison, WI, US. Licensed
221 under the Apache License, Version 2.0.
2228.8 Aug 06, 2020 CONDOR_SSH_TO_JOB(1)