1VM::EC2::Staging::ServeUrs(e3r)Contributed Perl DocumentVaMt:i:oEnC2::Staging::Server(3)
2
3
4

NAME

6       VM::EC2::Staging::Server - High level interface to EC2-based servers
7

SYNOPSIS

9        use VM::EC2::Staging::Manager;
10
11        # get a new staging manager
12        my $ec2     = VM::EC2->new;
13        my $staging = $ec2->staging_manager();                                         );
14
15        # Fetch a server named 'my_server'. Create it if it does not already exist.
16        my $server1 = $staging->get_server(-name              => 'my_server',
17                                          -availability_zone  => 'us-east-1a',
18                                          -architecture       => 'i386',
19                                          -instance_type      => 't1.micro');
20
21        # As above, but force a new server to be provisioned.
22        my $server2 = $staging->provision_server(-name              => 'my_server',
23                                                 -availability_zone => 'us-east-1a',
24                                                 -architecture      => 'i386',
25                                                 -instance_type     => 't1.micro');
26
27        # open up a terminal emulator in a separate window
28        $server1->shell;
29
30        # Run a command over ssh on the server. Standard in and out will be connected to
31        # STDIN/OUT
32        $server1->ssh('whoami');
33
34        # run a command over ssh on the server, returning standard output as an array of lines or a
35        # scalar string, similar to backticks (``)
36        my @password_lines = $server1->scmd('cat /etc/passwd');
37
38        # run a command on the server and read from it using a filehandle
39        my $fh  = $server1->scmd_read('ls -R /usr/lib');
40        while (<$fh>) { # do something }
41
42        # run a command on the server and write to it using a filehandle
43        my $fh  = $server1->scmd_write('sudo -s "cat >>/etc/fstab"');
44        print $fh "/dev/sdf3 /mnt/demo ext3 0 2\n";
45        close $fh;
46
47        # provision and mount a 5 gig ext3 volume mounted on /opt, returning
48        # VM::EC2::Staging::Volume object
49        my $opt = $server1->provision_volume(-mtpt   => '/opt',
50                                             -fstype => 'ext3',
51                                             -size   => 5);
52
53        # copy some data from the local filesystem onto the opt volume
54        $server1->rsync("$ENV{HOME}/local_staging_volume/" => $opt);
55
56        # same thing, but using server path name
57        $server1->put("$ENV{HOME}/local_staging_volume/" => '/opt');
58
59        # provision a volume attached to another server, and let the
60        # system choose the filesystem and mount point for us
61        my $backups = $server2->provision_volume(-name => 'Backup',
62                                                 -size => 10);
63
64        # copy some data from opt to the new volume using rsync
65        $server1->rsync($opt => "$backups/opt");
66
67        # Do a block-level copy between disks - warning, the filesystem must be unmounted
68        # before you attempt this.
69        $backups->unmount;
70        $server1->dd($opt => $backups);
71

DESCRIPTION

73       VM::EC2::Staging::Server objects are an extension of VM::EC2::Instance
74       to allow for higher-level access, including easy management of ssh
75       keys, remote copying of data from one server to another, and executing
76       of remote commands on the server from within Perl. See
77       VM::EC2::Staging::Manager for an overview of staging servers and
78       volumes.
79
80       Note that proper functioning of this module is heavily dependent on
81       running on a host system that has access to ssh, rsync and terminal
82       emulator command-line tools. It will most likely fail when run on a
83       Windows host.
84

Staging Server Creation

86       Staging servers are usually created via a staging manager's
87       get_server() or provision_server() methods. See
88       VM::EC2::Staging::Manager.
89
90       There is also a new() class method that is intended to be used
91       internally in most cases. It is called like this:
92
93   $server = VM::EC2::Staging::Server->new(%args)
94       With the arguments:
95
96        -keyfile    path to the ssh public/private keyfile for this instance
97        -username   username for remote login on this instance
98        -instance   VM::EC2::Instance to attach this server to
99        -manager    VM::EC2::Staging::Manager in same zone as the instance
100
101       Note that you will have to launch a staging manager, start an instance,
102       and appropriate provision the SSH credentials for that instance before
103       invoking new() directly.
104

Information about the Server

106       VM::EC2::Staging::Server objects have all the methods of
107       VM::EC2::Instance, such as dnsName(), but add several new methods. The
108       new methods involving getting basic information about the server are
109       listed in this section.
110
111   $name = $server->name
112       This method returns the server's symbolic name, if any.
113
114       Servers can optionally be assigned a symbolic name at the time they are
115       created by the manager's get_server() or provision_server() methods.
116       The name persists as long as the underlying instance exists (including
117       in stopped state for EBS-backed instances). Calling
118       $manager->get_server() with this name returns the server object.
119
120   $ec2 = $server->ec2
121       Return the VM::EC2 object associated with the server.
122
123   $ec2 = $server->endpoint
124       Return the endpoint URL associated with this server.
125
126   $instance = $server->instance
127       Return the VM::EC2::Instance associated with this server.
128
129   $file = $server->keyfile
130       Return the full path to the SSH PEM keyfile used to log into this
131       server.
132
133   $user = $server->username
134       Return the name of the user (e.g. 'ubuntu') used to ssh into this
135       server.
136
137   $manager = $server->manager
138       Returns the VM::EC2::Staging::Manager that manages this server.
139

Lifecycle Methods

141       The methods in this section manage the lifecycle of a server.
142
143   $flag = $server->ping
144       The ping() method returns true if the server is running and is
145       reachable via ssh. It is different from checking that the underlying
146       instance is "running" via a call to current_status, because it also
147       checks the usability of the ssh daemon, the provided ssh key and
148       username, firewall rules, and the network connectivity.
149
150       The result of ping is cached so that subsequent invocations return
151       quickly.
152
153   $result = $server->start
154       Attempt to start a stopped server. The method will wait until a ping()
155       is successful, or until a timeout of 120 seconds. The result code will
156       be true if the server was successfully started and is reachable.
157
158       If you wish to start a set of servers without waiting for each one
159       individually, then you may call the underling instance's start()
160       method:
161
162        $server->instance->start;
163
164       You may then wish to call the staging manager's wait_for_instances()
165       method to wait on all of the servers to start:
166
167        $manager->wait_for_servers(@servers);
168
169       Also check out $manager->start_all_servers().
170
171   $result = $server->stop
172       Attempt to stop a running server. The method will wait until the server
173       has entered the "stopped" state before returning. It will return a true
174       result if the underlying instance stopped successfully.
175
176       If you wish to stop a set of servers without waiting for each one
177       individually, then you may call the underling instance's start()
178       method:
179
180        $server->instance->stop;
181
182       You may then wish to call the staging manager's wait_for_instances()
183       method to wait on all of the servers to start:
184
185        $status = $manager->wait_for_servers(@servers);
186
187       Also check out $manager->stop_all_servers().
188
189   $result = $server->terminate
190       Terminate a server and unregister it from the manager. This method will
191       stop and wait until the server is terminated.
192
193       If you wish to stop a set of servers without waiting for each one
194       individually, then you may call the underling instance's start()
195       method:
196
197        $server->instance->terminate;
198

Remote Shell Methods

200       The methods in this section allow you to run remote commands on the
201       staging server and interrogate the results. Since the staging manager
202       handles the creation of SSH keys internally, you do not need to worry
203       about finding the right public/private keypair.
204
205   $result = $server->ssh(@command)
206       The ssh() method invokes a command on the remote server. You may
207       provide the command line as a single string, or broken up by argument:
208
209         $server->ssh('ls -lR /var/log');
210         $server->ssh('ls','-lR','/var/log');
211
212       The output of the command will appear on STDOUT and STDERR of the perl
213       process. Input, if needed, will be read from STDIN. If no command is
214       provided, then an interactive ssh session will be started on the remote
215       server and the script will wait until you have logged out.
216
217       If the remote command was successful, the method result will be true.
218
219   $output = $server->scmd(@command)
220       This is similar to ssh(), except that the standard output of the remote
221       command will be captured and returned as the function result, similar
222       to the way backticks work in perl:
223
224        my $output = $server->scmd('date');
225        print "The localtime for the server is $output";
226
227   $fh = $server->scmd_write(@command)
228       This method executes @command on the remote server, and returns a
229       filehandle that is attached to the standard input of the command. Here
230       is a slightly dangerous example that appends a line to /etc/passwd:
231
232        my $fh = $server->scmd_write('sudo -s "cat >>/etc/passwd"');
233        print $fh "whoopsie:x:119:130::/nonexistent:/bin/false\n";
234        close $fh;
235
236   $fh = $server->scmd_read(@command)
237       This method executes @command on the remote server, and returns a
238       filehandle that is attached to the standard output of the command. Here
239       is an example of reading syslog:
240
241        my $fh = $server->scmd_read('sudo cat /var/log/syslog');
242        while (<$fh>) {
243           next unless /kernel/;
244           print $_;
245        }
246        close $fh;
247
248   $server->shell()
249       This method works in an X Windowing environment by launching a new
250       terminal window and running an interactive ssh session on the server
251       host. The terminal window is executed in a fork()ed session, so that
252       the rest of the script continues running.  If X Windows is not running,
253       then the method behaves the same as calling ssh() with no arguments.
254
255       The terminal emulator to run is determined by calling the method
256       get_xterm().
257

Volume Management Methods

259       The methods in this section allow you to create and manage volumes
260       attached to the server. These supplement the EC2 facilities for
261       creating and attaching EBS volumes with the ability to format the
262       volumes with a variety of filesystems, and mount them at a desired
263       location.
264
265   $volume = $server->provision_volume(%args)
266       Provision and mount a new volume. If successful, the volume is returned
267       as a VM::EC2::Staging::Volume object.
268
269       Arguments (default):
270
271        -name         Symbolic name for the desired volume (autogenerated)
272        -fstype       Filesystem type for desired volume (ext4)
273        -size         Size for the desired volume in GB (1)
274        -mtpt         Mountpoint for this volume (/mnt/Staging/$name)
275        -mount        Alias for -mtpt
276        -volume_id    ID of existing volume to attach & mount (none)
277        -snapshot_id  ID of existing snapshot to use to create this volume (none)
278        -reuse        Reuse an existing managed volume of same name (false)
279        -label        Disk label to assign during formatting ($name)
280        -uuid         UUID to assign during formatting (none)
281
282       None of the arguments are required, and reasonable defaults will be
283       chosen if they are missing.
284
285       The -name argument specifies the symbolic name to be assigned to the
286       newly-created staging volume. The name allows the staging manager to
287       retrieve this volume at a later date if it is detached from the server
288       and returned to the available pool. If no name is provided, then an
289       arbitrary one will be autogenerated.
290
291       The -fstype argument specifies the filesystem to be generated on the
292       volume, ext4 by default. The following filesystems are currently
293       supported: ext2, ext3, ext4, xfs, reiserfs, jfs, ntfs, nfs, vfat,
294       msdos. In addition, you can specify a filesystem of "raw", which means
295       to provision and attach the volume to the server, but not to format it.
296       This can be used to set up LVM and RAID devices. Note that if the
297       server does not currently have the package needed to manage the desired
298       filesystem, it will use "apt-get" to install it.
299
300       The -mtpt and -mount arguments (they are equivalent) specify the mount
301       point for the volume on the server filesystem. The default is
302       "/mnt/Staging/$name", where $name is the symbolic name provided by
303       -name or autogenerated. No checking is done on the sensibility of the
304       mount point, so try to avoid mounting disks over essential parts of the
305       system.
306
307       -volume_id and -snapshot_id instruct the method to construct the
308       staging volume from an existing EBS volume or snapshot. -volume_id is
309       an EBS volume ID. If provided, the volume must be located in the
310       server's availability zone and be in the "available" state.
311       -snapshot_id is an EBS snapshot ID in the server's region. In no case
312       will provision_volume() attempt to reformat the resulting volume, even
313       if the -fstype argument is provided. However, in the case of a volume
314       created from a snapshot, you may specify a -size argument larger than
315       the snapshot and the filesystem will be dynamically resized to fill the
316       requested space. This currently only works with ext2, ext3 and ext4
317       volumes, and cannot be used to make filesystems smaller.
318
319       If the -reuse argument is true, and a symbolic name is provided in
320       -name, then the method will look for an available staging volume of the
321       same name and mount this at the specified location. If no suitable
322       staging volume is found, then the method will look for a snapshot
323       created earlier from a staging volume of the same name. If neither a
324       suitable volume nor a snapshot is available, then a new volume is
325       provisioned. This is intended to support the following use case of
326       synchronizing a filesystem somewhere to an EBS snapshot:
327
328        my $server = $staging_manager->get_server('my_server');
329        my $volume = $server->provision_volume(-name=>'backup_1',
330                                               -reuse  => 1,
331                                               -fstype => 'ext3',
332                                               -size   => 10);
333        $volume->put('fred@gw.harvard.edu:my_music');
334        $volume->create_snapshot('music_backups');
335        $volume->delete;
336
337       The -label and -uuid arguments are used to set the volume label and
338       UUID during formatting of new filesystems. The default behavior is to
339       create no label and to allow the server to choose an arbitrary UUID.
340
341   $volume = $server->add_volume(%args)
342       This is the same as provision_volume().
343
344   @volumes = $server->volumes()
345       Return a list of all the staging volumes attached to this server.
346       Unmanaged volumes, such as the root volume, are not included in the
347       list.
348
349   $server->unmount_volume($volume)
350       Unmount the volume $volume. The volume will remain attached to the
351       server. This method will die with a fatal error if the operation fails.
352
353       See VM::EC2::Staging::Volume->detach() for the recommended way to
354       unmount and detach the volume.
355
356   $server->detach_volume($volume)
357       Unmount and detach the volume from the server, waiting until EC2
358       reports that the detachment completed. A fatal error will occur if the
359       operation fails.
360
361   $server->mount_volume($volume [,$mountpt])
362       Mount the volume $volume using the mount information recorded inside
363       the VM::EC2::Staging::Volume object (returned by its mtpt() and mtdev()
364       methods). If the volume has not previously been mounted on this server,
365       then it will be attached to the server and a new mountpoint will be
366       allocated automatically. You can change the mount point by specifying
367       it explicitly in the second argument.
368
369       Here is the recommended way to detach a staging volume from one server
370       and attach it to another:
371
372        $server1->detach_volume($volume);
373        $server2->mount_volume($volume);
374
375       This method will die in case of error.
376
377   $server->remount_volume($volume)
378       This is similar to mount_volume(), except that it will fail with a
379       fatal error if the volume was not previously mounted on this server.
380       This is to be used when temporarily unmounting and remounting a volume
381       on the same server:
382
383        $server->unmount_volume($volume);
384        # do some work on the volume
385        $server->remount_volume($volume)
386
387   $server->delete_volume($volume)
388       Unmount, detach, and then delete the indicated volume entirely.
389
390   $snap = $server->create_snapshot($volume,$description)
391       Unmount the volume, snapshot it using the provided description, and
392       then remount the volume. If successful, returns the snapshot.
393
394       The snapshot is tagged with the identifying information needed to
395       associate the snapshot with the staging volume. This information then
396       used when creating new volumes from the snapshot with
397       $server->provision_volume(-reuse=>1).
398

Data Copying Methods

400       The methods in this section are used to copy data from one staging
401       server to another, and to copy data from a local file system to a
402       staging server.
403
404   $result = $server->rsync($src1,$src2,$src3...,$dest)
405       This method is a passthrough to VM::EC2::Staging::Manager->rsync(), and
406       provides efficient file-level synchronization (rsync) file-level
407       copying between one or more source locations and a destination location
408       via an ssh tunnel. Copying among arbitrary combinations of local and
409       remote filesystems is supported, with the caveat that the remote
410       filesystems must be contained on volumes and servers managed by this
411       module (see below for a workaround).
412
413       You may provide two or more directory paths. The last path will be
414       treated as the copy destination, and the source paths will be treated
415       as copy sources. All copying is performed using the -avz options, which
416       activates recursive directory copying in which ownership, modification
417       times and permissions are preserved, and compresses the data to reduce
418       network usage.
419
420       Source paths can be formatted in one of several ways:
421
422        /absolute/path
423             Copy the contents of the directory /absolute/path located on the
424             local machine to the destination. This will create a
425             subdirectory named "path" on the destination disk. Add a slash
426             to the end of the path (i.e. "/absolute/path/") in order to
427             avoid creating this subdirectory on the destination disk.
428
429        ./relative/path
430             Relative paths work the way you expect, and depend on the current
431             working directory. The terminating slash rule applies.
432
433        $staging_server:/absolute/path
434            Pass a staging server object and absolute path to copy the contents
435            of this path to the destination disk. Because of string interpolation
436            you can include server objects in quotes: "$my_server:/opt"
437
438        $staging_server:relative/path
439            This form will copy data from paths relative to the remote user's home
440            directory on the staging server. Typically not very useful, but supported.
441
442        $staging_volume
443             Pass a VM::EC2::Staging::Volume to copy the contents of the
444             volume to the destination disk starting at the root of the
445             volume. Note that you do *not* need to have any knowledge of the
446             mount point for this volume in order to copy its contents.
447
448        $staging_volume:/absolute/path
449             Copy a subdirectory of a staging volume to the destination disk.
450             The root of the volume is its top level, regardless of where it
451             is mounted on the staging server.  Because of string
452             interpolation magic, you can enclose staging volume object names
453             in quotes in order to construct the path, as in
454             "$picture_volume:/family/vacations/". As in local paths, a
455             terminating slash indicates that the contents of the last
456             directory in the path are to be copied without creating the
457             enclosing directory on the desetination. Note that you do *not*
458             need to have any knowledge of the mount point for this volume in
459             order to copy its contents.
460
461        $staging_volume:absolute/path
462        $staging_volume/absolute/path
463            These are alternatives to the previous syntax, and all have the
464            same effect as $staging_volume:relative/path. There is no
465
466       The same syntax is supported for destination paths, except that it
467       makes no difference whether a path has a trailing slash or not.
468
469       Note that neither the source nor destination paths need to reside on
470       this server.
471
472       See VM::EC2::Staging::Manager->rsync() for examples and more details.
473
474   $server->dd($source_vol=>$dest_vol)
475       This method is a passthrough to VM::EC2::Staging::Manager->dd(), and
476       performs block-level copying of the contents of $source_vol to
477       $dest_vol by using dd over an SSH tunnel, where both source and
478       destination volumes are VM::EC2::Staging::Volume objects. The volumes
479       must be attached to a server but not mounted. Everything in the volume,
480       including its partition table, is copied, allowing you to make an exact
481       image of a disk.
482
483       The volumes do not actually need to reside on this server, but can be
484       attached to any staging server in the zone.
485
486   $server->put($source1,$source2,$source3,...,$dest)
487       Use rsync to copy the indicated source directories into the destination
488       path indicated by $dest. The destination is either a path on the server
489       machine, or a staging volume object mounted on the server (string
490       interpolation is accepted). The sources can be local paths on the
491       machine the perl script is running on, or any of the formats described
492       for rsync().
493
494       Examples:
495
496        $server1->put("$ENV{HOME}/my_pictures"     => '/var/media');
497        $server1->put("$ENV{HOME}/my_pictures","$ENV{HOME}/my_audio" => '/var/media');
498        $server1->put("$ENV{HOME}/my_pictures"     => "$backup_volume/home_media");
499        $server1->put("fred@gw.harvard.edu:media/" => "$backup_volume/home_media");
500
501   $server->get($source1,$source2,$source3,...,$dest)
502       Use rsync to copy the indicated source directories into the destination
503       path indicated by $dest. The source directories are either paths on the
504       server, or staging volume(s) mounted on the server (string
505       interpolation to indicate subdirectories on the staging volume also
506       works). The destination can be any of the path formats described for
507       rsync(), including unmanaged hosts that accept ssh login.
508
509       Examples:
510
511        $server1->get('/var/media' =>"$ENV{HOME}/my_pictures");
512        $server1->get('/var/media','/usr/bin' => "$ENV{HOME}/test");
513        $server1->get("$backup_volume/home_media" => "$ENV{HOME}/my_pictures");
514        $server1->get("$backup_volume/home_media" => "fred@gw.harvard.edu:media/");
515

Internal Methods

517       This section documents internal methods. They are not intended for use
518       by end-user scripts but may be useful to know about during subclassing.
519       There are also additional undocumented methods that begin with a "_"
520       character which you can explore in the source code.
521
522   $description = $server->volume_description($vol)
523       This method is called to get the value of the Name tag assigned to new
524       staging volume objects. The current value is "Staging volume for $name
525       created by VM::EC2::Staging::Server."
526
527       You will see these names associated with EBS volumes in the AWS
528       console.
529
530   ($ebs_device,$local_device) = $server->unused_block_device([$major_start])
531       This method returns an unused block device path. It is invoked when
532       provisioning and mounting new volumes. The behavior is to take the
533       following search path:
534
535        /dev/sdf1
536        /dev/sdf2
537        ...
538        /dev/sdf15
539        /dev/sdfg1
540        ...
541        /dev/sdp15
542
543       You can modify the search path slightly by providing a single character
544       major start. For example, to leave all the sdf's free and to start the
545       search at /dev/sdg:
546
547        ($ebs_device,$local_device) = $server->unused_block_device('g');
548
549       The result is a two element list consisting of the unused device name
550       from the perspective of EC2 and the server respectively. The issue here
551       is that on some recent Linux kernels, the EC2 device /dev/sdf1 is known
552       to the server as /dev/xvdf1. This module understands that complication
553       and uses the EC2 block device name when managing EBS volumes, and the
554       kernel block device name when communicating with the server.
555
556   $flag = $server->has_key($keyname)
557       Returns true if the server has a copy of the private key corresponding
558       to $keyname. This is used by the rsync() method to enable server to
559       server data transfers.
560
561   $flag = $server->accepts_key($keyname)
562       Returns true if the server has a copy of the public key part of
563       $keyname in its .ssh/authorized_keys file. This is used by the rsync()
564       method to enable server to server data transfers.
565
566   $up = $server->is_up([$new_value])
567       Get/set the internal is_up() flag, which indicates whether the server
568       is up and running. This is used to cache the results of the ping()
569       method.
570
571   $path = $server->default_mtpt($volume)
572       Given a staging volume, return its default mount point on the server
573       ('/mnt/Staging/'.$volume->name). Can also pass a string corresponding
574       to the volume's name.
575
576   $server->info(@message)
577       Log a message to standard output, respecting the staging manager's
578       verbosity() setting.
579

Subclassing

581       For reasons having to do with the order in which objects are created,
582       VM::EC2::Staging::Server is a wrapper around VM::EC2::Instance rather
583       than a subclass of it. To access the VM::EC2::Instance object, you call
584       the server object's instance() method. In practice this means that to
585       invoke the underlying instance's method for, say, start() you will need
586       to do this:
587
588         $server->instance->start();
589
590       rather than this:
591
592         $server->SUPER::start();
593
594       You may subclass VM::EC2::Staging::Server in the usual way.
595

SEE ALSO

597       VM::EC2 VM::EC2::Instance VM::EC2::Volume VM::EC2::Snapshot
598

AUTHOR

600       Lincoln Stein <lincoln.stein@gmail.com>.
601
602       Copyright (c) 2011 Ontario Institute for Cancer Research
603
604       This package and its accompanying libraries is free software; you can
605       redistribute it and/or modify it under the terms of the GPL (either
606       version 1, or at your option, any later version) or the Artistic
607       License 2.0.  Refer to LICENSE for the full license text. In addition,
608       please see DISCLAIMER.txt for disclaimers of warranty.
609
610
611
612perl v5.36.0                      2023-01-20       VM::EC2::Staging::Server(3)
Impressum