1VM::EC2::Staging::ServeUrs(e3r)Contributed Perl DocumentVaMt:i:oEnC2::Staging::Server(3)
2
6 VM::EC2::Staging::Server - High level interface to EC2-based servers
7
9 use VM::EC2::Staging::Manager;
10 # get a new staging manager
12 my $ec2 = VM::EC2->new;
13 my $staging = $ec2->staging_manager(); );
14 # Fetch a server named 'my_server'. Create it if it does not already exist.
16 my $server1 = $staging->get_server(-name => 'my_server',
17 -availability_zone => 'us-east-1a',
18 -architecture => 'i386',
19 -instance_type => 't1.micro');
20 # As above, but force a new server to be provisioned.
22 my $server2 = $staging->provision_server(-name => 'my_server',
23 -availability_zone => 'us-east-1a',
24 -architecture => 'i386',
25 -instance_type => 't1.micro');
26 # open up a terminal emulator in a separate window
28 $server1->shell;
29 # Run a command over ssh on the server. Standard in and out will be connected to
31 # STDIN/OUT
32 $server1->ssh('whoami');
33 # run a command over ssh on the server, returning standard output as an array of lines or a
35 # scalar string, similar to backticks (``)
36 my @password_lines = $server1->scmd('cat /etc/passwd');
37 # run a command on the server and read from it using a filehandle
39 my $fh = $server1->scmd_read('ls -R /usr/lib');
40 while (<$fh>) { # do something }
41 # run a command on the server and write to it using a filehandle
43 my $fh = $server1->scmd_write('sudo -s "cat >>/etc/fstab"');
44 print $fh "/dev/sdf3 /mnt/demo ext3 0 2\n";
45 close $fh;
46 # provision and mount a 5 gig ext3 volume mounted on /opt, returning
48 # VM::EC2::Staging::Volume object
49 my $opt = $server1->provision_volume(-mtpt => '/opt',
50 -fstype => 'ext3',
51 -size => 5);
52 # copy some data from the local filesystem onto the opt volume
54 $server1->rsync("$ENV{HOME}/local_staging_volume/" => $opt);
55 # same thing, but using server path name
57 $server1->put("$ENV{HOME}/local_staging_volume/" => '/opt');
58 # provision a volume attached to another server, and let the
60 # system choose the filesystem and mount point for us
61 my $backups = $server2->provision_volume(-name => 'Backup',
62 -size => 10);
63 # copy some data from opt to the new volume using rsync
65 $server1->rsync($opt => "$backups/opt");
66 # Do a block-level copy between disks - warning, the filesystem must be unmounted
68 # before you attempt this.
69 $backups->unmount;
70 $server1->dd($opt => $backups);
71
73 VM::EC2::Staging::Server objects are an extension of VM::EC2::Instance
74 to allow for higher-level access, including easy management of ssh
75 keys, remote copying of data from one server to another, and executing
76 of remote commands on the server from within Perl. See
77 VM::EC2::Staging::Manager for an overview of staging servers and
78 volumes.
79 Note that proper functioning of this module is heavily dependent on
81 running on a host system that has access to ssh, rsync and terminal
82 emulator command-line tools. It will most likely fail when run on a
83 Windows host.
84
86 Staging servers are usually created via a staging manager's
87 get_server() or provision_server() methods. See
88 VM::EC2::Staging::Manager.
89 There is also a new() class method that is intended to be used
91 internally in most cases. It is called like this:
92 $server = VM::EC2::Staging::Server->new(%args)
94 With the arguments:
95 -keyfile path to the ssh public/private keyfile for this instance
97 -username username for remote login on this instance
98 -instance VM::EC2::Instance to attach this server to
99 -manager VM::EC2::Staging::Manager in same zone as the instance
100 Note that you will have to launch a staging manager, start an instance,
102 and appropriate provision the SSH credentials for that instance before
103 invoking new() directly.
104
106 VM::EC2::Staging::Server objects have all the methods of
107 VM::EC2::Instance, such as dnsName(), but add several new methods. The
108 new methods involving getting basic information about the server are
109 listed in this section.
110 $name = $server->name
112 This method returns the server's symbolic name, if any.
113 Servers can optionally be assigned a symbolic name at the time they are
115 created by the manager's get_server() or provision_server() methods.
116 The name persists as long as the underlying instance exists (including
117 in stopped state for EBS-backed instances). Calling
118 $manager->get_server() with this name returns the server object.
119 $ec2 = $server->ec2
121 Return the VM::EC2 object associated with the server.
122 $ec2 = $server->endpoint
124 Return the endpoint URL associated with this server.
125 $instance = $server->instance
127 Return the VM::EC2::Instance associated with this server.
128 $file = $server->keyfile
130 Return the full path to the SSH PEM keyfile used to log into this
131 server.
132 $user = $server->username
134 Return the name of the user (e.g. 'ubuntu') used to ssh into this
135 server.
136 $manager = $server->manager
138 Returns the VM::EC2::Staging::Manager that manages this server.
139
141 The methods in this section manage the lifecycle of a server.
142 $flag = $server->ping
144 The ping() method returns true if the server is running and is
145 reachable via ssh. It is different from checking that the underlying
146 instance is "running" via a call to current_status, because it also
147 checks the usability of the ssh daemon, the provided ssh key and
148 username, firewall rules, and the network connectivity.
149 The result of ping is cached so that subsequent invocations return
151 quickly.
152 $result = $server->start
154 Attempt to start a stopped server. The method will wait until a ping()
155 is successful, or until a timeout of 120 seconds. The result code will
156 be true if the server was successfully started and is reachable.
157 If you wish to start a set of servers without waiting for each one
159 individually, then you may call the underling instance's start()
160 method:
161 $server->instance->start;
163 You may then wish to call the staging manager's wait_for_instances()
165 method to wait on all of the servers to start:
166 $manager->wait_for_servers(@servers);
168 Also check out $manager->start_all_servers().
170 $result = $server->stop
172 Attempt to stop a running server. The method will wait until the server
173 has entered the "stopped" state before returning. It will return a true
174 result if the underlying instance stopped successfully.
175 If you wish to stop a set of servers without waiting for each one
177 individually, then you may call the underling instance's start()
178 method:
179 $server->instance->stop;
181 You may then wish to call the staging manager's wait_for_instances()
183 method to wait on all of the servers to start:
184 $status = $manager->wait_for_servers(@servers);
186 Also check out $manager->stop_all_servers().
188 $result = $server->terminate
190 Terminate a server and unregister it from the manager. This method will
191 stop and wait until the server is terminated.
192 If you wish to stop a set of servers without waiting for each one
194 individually, then you may call the underling instance's start()
195 method:
196 $server->instance->terminate;
198
200 The methods in this section allow you to run remote commands on the
201 staging server and interrogate the results. Since the staging manager
202 handles the creation of SSH keys internally, you do not need to worry
203 about finding the right public/private keypair.
204 $result = $server->ssh(@command)
206 The ssh() method invokes a command on the remote server. You may
207 provide the command line as a single string, or broken up by argument:
208 $server->ssh('ls -lR /var/log');
210 $server->ssh('ls','-lR','/var/log');
211 The output of the command will appear on STDOUT and STDERR of the perl
213 process. Input, if needed, will be read from STDIN. If no command is
214 provided, then an interactive ssh session will be started on the remote
215 server and the script will wait until you have logged out.
216 If the remote command was successful, the method result will be true.
218 $output = $server->scmd(@command)
220 This is similar to ssh(), except that the standard output of the remote
221 command will be captured and returned as the function result, similar
222 to the way backticks work in perl:
223 my $output = $server->scmd('date');
225 print "The localtime for the server is $output";
226 $fh = $server->scmd_write(@command)
228 This method executes @command on the remote server, and returns a
229 filehandle that is attached to the standard input of the command. Here
230 is a slightly dangerous example that appends a line to /etc/passwd:
231 my $fh = $server->scmd_write('sudo -s "cat >>/etc/passwd"');
233 print $fh "whoopsie:x:119:130::/nonexistent:/bin/false\n";
234 close $fh;
235 $fh = $server->scmd_read(@command)
237 This method executes @command on the remote server, and returns a
238 filehandle that is attached to the standard output of the command. Here
239 is an example of reading syslog:
240 my $fh = $server->scmd_read('sudo cat /var/log/syslog');
242 while (<$fh>) {
243 next unless /kernel/;
244 print $_;
245 }
246 close $fh;
247 $server->shell()
249 This method works in an X Windowing environment by launching a new
250 terminal window and running an interactive ssh session on the server
251 host. The terminal window is executed in a fork()ed session, so that
252 the rest of the script continues running. If X Windows is not running,
253 then the method behaves the same as calling ssh() with no arguments.
254 The terminal emulator to run is determined by calling the method
256 get_xterm().
257
259 The methods in this section allow you to create and manage volumes
260 attached to the server. These supplement the EC2 facilities for
261 creating and attaching EBS volumes with the ability to format the
262 volumes with a variety of filesystems, and mount them at a desired
263 location.
264 $volume = $server->provision_volume(%args)
266 Provision and mount a new volume. If successful, the volume is returned
267 as a VM::EC2::Staging::Volume object.
268 Arguments (default):
270 -name Symbolic name for the desired volume (autogenerated)
272 -fstype Filesystem type for desired volume (ext4)
273 -size Size for the desired volume in GB (1)
274 -mtpt Mountpoint for this volume (/mnt/Staging/$name)
275 -mount Alias for -mtpt
276 -volume_id ID of existing volume to attach & mount (none)
277 -snapshot_id ID of existing snapshot to use to create this volume (none)
278 -reuse Reuse an existing managed volume of same name (false)
279 -label Disk label to assign during formatting ($name)
280 -uuid UUID to assign during formatting (none)
281 None of the arguments are required, and reasonable defaults will be
283 chosen if they are missing.
284 The -name argument specifies the symbolic name to be assigned to the
286 newly-created staging volume. The name allows the staging manager to
287 retrieve this volume at a later date if it is detached from the server
288 and returned to the available pool. If no name is provided, then an
289 arbitrary one will be autogenerated.
290 The -fstype argument specifies the filesystem to be generated on the
292 volume, ext4 by default. The following filesystems are currently
293 supported: ext2, ext3, ext4, xfs, reiserfs, jfs, ntfs, nfs, vfat,
294 msdos. In addition, you can specify a filesystem of "raw", which means
295 to provision and attach the volume to the server, but not to format it.
296 This can be used to set up LVM and RAID devices. Note that if the
297 server does not currently have the package needed to manage the desired
298 filesystem, it will use "apt-get" to install it.
299 The -mtpt and -mount arguments (they are equivalent) specify the mount
301 point for the volume on the server filesystem. The default is
302 "/mnt/Staging/$name", where $name is the symbolic name provided by
303 -name or autogenerated. No checking is done on the sensibility of the
304 mount point, so try to avoid mounting disks over essential parts of the
305 system.
306 -volume_id and -snapshot_id instruct the method to construct the
308 staging volume from an existing EBS volume or snapshot. -volume_id is
309 an EBS volume ID. If provided, the volume must be located in the
310 server's availability zone and be in the "available" state.
311 -snapshot_id is an EBS snapshot ID in the server's region. In no case
312 will provision_volume() attempt to reformat the resulting volume, even
313 if the -fstype argument is provided. However, in the case of a volume
314 created from a snapshot, you may specify a -size argument larger than
315 the snapshot and the filesystem will be dynamically resized to fill the
316 requested space. This currently only works with ext2, ext3 and ext4
317 volumes, and cannot be used to make filesystems smaller.
318 If the -reuse argument is true, and a symbolic name is provided in
320 -name, then the method will look for an available staging volume of the
321 same name and mount this at the specified location. If no suitable
322 staging volume is found, then the method will look for a snapshot
323 created earlier from a staging volume of the same name. If neither a
324 suitable volume nor a snapshot is available, then a new volume is
325 provisioned. This is intended to support the following use case of
326 synchronizing a filesystem somewhere to an EBS snapshot:
327 my $server = $staging_manager->get_server('my_server');
329 my $volume = $server->provision_volume(-name=>'backup_1',
330 -reuse => 1,
331 -fstype => 'ext3',
332 -size => 10);
333 $volume->put('fred@gw.harvard.edu:my_music');
334 $volume->create_snapshot('music_backups');
335 $volume->delete;
336 The -label and -uuid arguments are used to set the volume label and
338 UUID during formatting of new filesystems. The default behavior is to
339 create no label and to allow the server to choose an arbitrary UUID.
340 $volume = $server->add_volume(%args)
342 This is the same as provision_volume().
343 @volumes = $server->volumes()
345 Return a list of all the staging volumes attached to this server.
346 Unmanaged volumes, such as the root volume, are not included in the
347 list.
348 $server->unmount_volume($volume)
350 Unmount the volume $volume. The volume will remain attached to the
351 server. This method will die with a fatal error if the operation fails.
352 See VM::EC2::Staging::Volume->detach() for the recommended way to
354 unmount and detach the volume.
355 $server->detach_volume($volume)
357 Unmount and detach the volume from the server, waiting until EC2
358 reports that the detachment completed. A fatal error will occur if the
359 operation fails.
360 $server->mount_volume($volume [,$mountpt])
362 Mount the volume $volume using the mount information recorded inside
363 the VM::EC2::Staging::Volume object (returned by its mtpt() and mtdev()
364 methods). If the volume has not previously been mounted on this server,
365 then it will be attached to the server and a new mountpoint will be
366 allocated automatically. You can change the mount point by specifying
367 it explicitly in the second argument.
368 Here is the recommended way to detach a staging volume from one server
370 and attach it to another:
371 $server1->detach_volume($volume);
373 $server2->mount_volume($volume);
374 This method will die in case of error.
376 $server->remount_volume($volume)
378 This is similar to mount_volume(), except that it will fail with a
379 fatal error if the volume was not previously mounted on this server.
380 This is to be used when temporarily unmounting and remounting a volume
381 on the same server:
382 $server->unmount_volume($volume);
384 # do some work on the volume
385 $server->remount_volume($volume)
386 $server->delete_volume($volume)
388 Unmount, detach, and then delete the indicated volume entirely.
389 $snap = $server->create_snapshot($volume,$description)
391 Unmount the volume, snapshot it using the provided description, and
392 then remount the volume. If successful, returns the snapshot.
393 The snapshot is tagged with the identifying information needed to
395 associate the snapshot with the staging volume. This information then
396 used when creating new volumes from the snapshot with
397 $server->provision_volume(-reuse=>1).
398
400 The methods in this section are used to copy data from one staging
401 server to another, and to copy data from a local file system to a
402 staging server.
403 $result = $server->rsync($src1,$src2,$src3...,$dest)
405 This method is a passthrough to VM::EC2::Staging::Manager->rsync(), and
406 provides efficient file-level synchronization (rsync) file-level
407 copying between one or more source locations and a destination location
408 via an ssh tunnel. Copying among arbitrary combinations of local and
409 remote filesystems is supported, with the caveat that the remote
410 filesystems must be contained on volumes and servers managed by this
411 module (see below for a workaround).
412 You may provide two or more directory paths. The last path will be
414 treated as the copy destination, and the source paths will be treated
415 as copy sources. All copying is performed using the -avz options, which
416 activates recursive directory copying in which ownership, modification
417 times and permissions are preserved, and compresses the data to reduce
418 network usage.
419 Source paths can be formatted in one of several ways:
421 /absolute/path
423 Copy the contents of the directory /absolute/path located on the
424 local machine to the destination. This will create a
425 subdirectory named "path" on the destination disk. Add a slash
426 to the end of the path (i.e. "/absolute/path/") in order to
427 avoid creating this subdirectory on the destination disk.
428 ./relative/path
430 Relative paths work the way you expect, and depend on the current
431 working directory. The terminating slash rule applies.
432 $staging_server:/absolute/path
434 Pass a staging server object and absolute path to copy the contents
435 of this path to the destination disk. Because of string interpolation
436 you can include server objects in quotes: "$my_server:/opt"
437 $staging_server:relative/path
439 This form will copy data from paths relative to the remote user's home
440 directory on the staging server. Typically not very useful, but supported.
441 $staging_volume
443 Pass a VM::EC2::Staging::Volume to copy the contents of the
444 volume to the destination disk starting at the root of the
445 volume. Note that you do *not* need to have any knowledge of the
446 mount point for this volume in order to copy its contents.
447 $staging_volume:/absolute/path
449 Copy a subdirectory of a staging volume to the destination disk.
450 The root of the volume is its top level, regardless of where it
451 is mounted on the staging server. Because of string
452 interpolation magic, you can enclose staging volume object names
453 in quotes in order to construct the path, as in
454 "$picture_volume:/family/vacations/". As in local paths, a
455 terminating slash indicates that the contents of the last
456 directory in the path are to be copied without creating the
457 enclosing directory on the desetination. Note that you do *not*
458 need to have any knowledge of the mount point for this volume in
459 order to copy its contents.
460 $staging_volume:absolute/path
462 $staging_volume/absolute/path
463 These are alternatives to the previous syntax, and all have the
464 same effect as $staging_volume:relative/path. There is no
465 The same syntax is supported for destination paths, except that it
467 makes no difference whether a path has a trailing slash or not.
468 Note that neither the source nor destination paths need to reside on
470 this server.
471 See VM::EC2::Staging::Manager->rsync() for examples and more details.
473 $server->dd($source_vol=>$dest_vol)
475 This method is a passthrough to VM::EC2::Staging::Manager->dd(), and
476 performs block-level copying of the contents of $source_vol to
477 $dest_vol by using dd over an SSH tunnel, where both source and
478 destination volumes are VM::EC2::Staging::Volume objects. The volumes
479 must be attached to a server but not mounted. Everything in the volume,
480 including its partition table, is copied, allowing you to make an exact
481 image of a disk.
482 The volumes do not actually need to reside on this server, but can be
484 attached to any staging server in the zone.
485 $server->put($source1,$source2,$source3,...,$dest)
487 Use rsync to copy the indicated source directories into the destination
488 path indicated by $dest. The destination is either a path on the server
489 machine, or a staging volume object mounted on the server (string
490 interpolation is accepted). The sources can be local paths on the
491 machine the perl script is running on, or any of the formats described
492 for rsync().
493 Examples:
495 $server1->put("$ENV{HOME}/my_pictures" => '/var/media');
497 $server1->put("$ENV{HOME}/my_pictures","$ENV{HOME}/my_audio" => '/var/media');
498 $server1->put("$ENV{HOME}/my_pictures" => "$backup_volume/home_media");
499 $server1->put("fred@gw.harvard.edu:media/" => "$backup_volume/home_media");
500 $server->get($source1,$source2,$source3,...,$dest)
502 Use rsync to copy the indicated source directories into the destination
503 path indicated by $dest. The source directories are either paths on the
504 server, or staging volume(s) mounted on the server (string
505 interpolation to indicate subdirectories on the staging volume also
506 works). The destination can be any of the path formats described for
507 rsync(), including unmanaged hosts that accept ssh login.
508 Examples:
510 $server1->get('/var/media' =>"$ENV{HOME}/my_pictures");
512 $server1->get('/var/media','/usr/bin' => "$ENV{HOME}/test");
513 $server1->get("$backup_volume/home_media" => "$ENV{HOME}/my_pictures");
514 $server1->get("$backup_volume/home_media" => "fred@gw.harvard.edu:media/");
515
517 This section documents internal methods. They are not intended for use
518 by end-user scripts but may be useful to know about during subclassing.
519 There are also additional undocumented methods that begin with a "_"
520 character which you can explore in the source code.
521 $description = $server->volume_description($vol)
523 This method is called to get the value of the Name tag assigned to new
524 staging volume objects. The current value is "Staging volume for $name
525 created by VM::EC2::Staging::Server."
526 You will see these names associated with EBS volumes in the AWS
528 console.
529 ($ebs_device,$local_device) = $server->unused_block_device([$major_start])
531 This method returns an unused block device path. It is invoked when
532 provisioning and mounting new volumes. The behavior is to take the
533 following search path:
534 /dev/sdf1
536 /dev/sdf2
537 ...
538 /dev/sdf15
539 /dev/sdfg1
540 ...
541 /dev/sdp15
542 You can modify the search path slightly by providing a single character
544 major start. For example, to leave all the sdf's free and to start the
545 search at /dev/sdg:
546 ($ebs_device,$local_device) = $server->unused_block_device('g');
548 The result is a two element list consisting of the unused device name
550 from the perspective of EC2 and the server respectively. The issue here
551 is that on some recent Linux kernels, the EC2 device /dev/sdf1 is known
552 to the server as /dev/xvdf1. This module understands that complication
553 and uses the EC2 block device name when managing EBS volumes, and the
554 kernel block device name when communicating with the server.
555 $flag = $server->has_key($keyname)
557 Returns true if the server has a copy of the private key corresponding
558 to $keyname. This is used by the rsync() method to enable server to
559 server data transfers.
560 $flag = $server->accepts_key($keyname)
562 Returns true if the server has a copy of the public key part of
563 $keyname in its .ssh/authorized_keys file. This is used by the rsync()
564 method to enable server to server data transfers.
565 $up = $server->is_up([$new_value])
567 Get/set the internal is_up() flag, which indicates whether the server
568 is up and running. This is used to cache the results of the ping()
569 method.
570 $path = $server->default_mtpt($volume)
572 Given a staging volume, return its default mount point on the server
573 ('/mnt/Staging/'.$volume->name). Can also pass a string corresponding
574 to the volume's name.
575 $server->info(@message)
577 Log a message to standard output, respecting the staging manager's
578 verbosity() setting.
579
581 For reasons having to do with the order in which objects are created,
582 VM::EC2::Staging::Server is a wrapper around VM::EC2::Instance rather
583 than a subclass of it. To access the VM::EC2::Instance object, you call
584 the server object's instance() method. In practice this means that to
585 invoke the underlying instance's method for, say, start() you will need
586 to do this:
587 $server->instance->start();
589 rather than this:
591 $server->SUPER::start();
593 You may subclass VM::EC2::Staging::Server in the usual way.
595
597 VM::EC2 VM::EC2::Instance VM::EC2::Volume VM::EC2::Snapshot
598
600 Lincoln Stein <lincoln.stein@gmail.com>.
601 Copyright (c) 2011 Ontario Institute for Cancer Research
603 This package and its accompanying libraries is free software; you can
605 redistribute it and/or modify it under the terms of the GPL (either
606 version 1, or at your option, any later version) or the Artistic
607 License 2.0. Refer to LICENSE for the full license text. In addition,
608 please see DISCLAIMER.txt for disclaimers of warranty.
609perl v5.28.0 2018-07-15 VM::EC2::Staging::Server(3)