SYNOPSIS
use VM::EC2::Staging::Manager;
# get a new staging manager
my $ec2 = VM::EC2->new;
my $staging = $ec2->staging_manager(); );
# Fetch a server named 'my_server'. Create it if it does not already exist.
my $server1 = $staging->get_server(-name => 'my_server',
-availability_zone => 'us-east-1a',
-architecture => 'i386',
-instance_type => 't1.micro');
# As above, but force a new server to be provisioned.
my $server2 = $staging->provision_server(-name => 'my_server',
-availability_zone => 'us-east-1a',
-architecture => 'i386',
-instance_type => 't1.micro');
# open up a terminal emulator in a separate window
$server1->shell;
# Run a command over ssh on the server. Standard in and out will be connected to
# STDIN/OUT
$server1->ssh('whoami');
# run a command over ssh on the server, returning standard output as an array of lines or a
# scalar string, similar to backticks (``)
my @password_lines = $server1->scmd('cat /etc/passwd');
# run a command on the server and read from it using a filehandle
my $fh = $server1->scmd_read('ls -R /usr/lib');
while (<$fh>) { # do something }
# run a command on the server and write to it using a filehandle
my $fh = $server1->scmd_write('sudo -s "cat >>/etc/fstab"');
print $fh "/dev/sdf3 /mnt/demo ext3 0 2\n";
close $fh;
# provision and mount a 5 gig ext3 volume mounted on /opt, returning
# VM::EC2::Staging::Volume object
my $opt = $server1->provision_volume(-mtpt => '/opt',
-fstype => 'ext3',
-size => 5);
# copy some data from the local filesystem onto the opt volume
$server1->rsync("$ENV{HOME}/local_staging_volume/" => $opt);
# same thing, but using server path name
$server1->put("$ENV{HOME}/local_staging_volume/" => '/opt');
# provision a volume attached to another server, and let the
# system choose the filesystem and mount point for us
my $backups = $server2->provision_volume(-name => 'Backup',
-size => 10);
# copy some data from opt to the new volume using rsync
$server1->rsync($opt => "$backups/opt");
# Do a block-level copy between disks - warning, the filesystem must be unmounted
# before you attempt this.
$backups->unmount;
$server1->dd($opt => $backups);
DESCRIPTION
VM::EC2::Staging::Server objects are an extension of VM::EC2::Instance to allow for higher-level access, including easy management of ssh keys, remote copying of data from one server to another, and executing of remote commands on the server from within Perl. See VM::EC2::Staging::Manager for an overview of staging servers and volumes.Note that proper functioning of this module is heavily dependent on running on a host system that has access to ssh, rsync and terminal emulator command-line tools. It will most likely fail when run on a Windows host.
Staging Server Creation
Staging servers are usually created via a staging manager's get_server() or provision_server() methods. See VM::EC2::Staging::Manager.There is also a new() class method that is intended to be used internally in most cases. It is called like this:
$server = VM::EC2::Staging::Server->new(%args)
With the arguments:
-keyfile path to the ssh public/private keyfile for this instance -username username for remote login on this instance -instance VM::EC2::Instance to attach this server to -manager VM::EC2::Staging::Manager in same zone as the instance
Note that you will have to launch a staging manager, start an instance, and appropriate provision the SSH credentials for that instance before invoking new() directly.
Information about the Server
VM::EC2::Staging::Server objects have all the methods of VM::EC2::Instance, such as dnsName(), but add several new methods. The new methods involving getting basic information about the server are listed in this section.$name = $server->name
This method returns the server's symbolic name, if any.Servers can optionally be assigned a symbolic name at the time they are created by the manager's get_server() or provision_server() methods. The name persists as long as the underlying instance exists (including in stopped state for EBS-backed instances). Calling $manager->get_server() with this name returns the server object.
$ec2 = $server->ec2
Return the VM::EC2 object associated with the server.$ec2 = $server->endpoint
Return the endpoint URL associated with this server.$instance = $server->instance
Return the VM::EC2::Instance associated with this server.$file = $server->keyfile
Return the full path to the SSH PEM keyfile used to log into this server.$user = $server->username
Return the name of the user (e.g. 'ubuntu') used to ssh into this server.$manager = $server->manager
Returns the VM::EC2::Staging::Manager that manages this server.Lifecycle Methods
The methods in this section manage the lifecycle of a server.$flag = $server->ping
The ping() method returns true if the server is running and is reachable via ssh. It is different from checking that the underlying instance is ``running'' via a call to current_status, because it also checks the usability of the ssh daemon, the provided ssh key and username, firewall rules, and the network connectivity.The result of ping is cached so that subsequent invocations return quickly.
$result = $server->start
Attempt to start a stopped server. The method will wait until a ping() is successful, or until a timeout of 120 seconds. The result code will be true if the server was successfully started and is reachable.If you wish to start a set of servers without waiting for each one individually, then you may call the underling instance's start() method:
$server->instance->start;
You may then wish to call the staging manager's wait_for_instances() method to wait on all of the servers to start:
$manager->wait_for_servers(@servers);
Also check out $manager->start_all_servers().
$result = $server->stop
Attempt to stop a running server. The method will wait until the server has entered the ``stopped'' state before returning. It will return a true result if the underlying instance stopped successfully.If you wish to stop a set of servers without waiting for each one individually, then you may call the underling instance's start() method:
$server->instance->stop;
You may then wish to call the staging manager's wait_for_instances() method to wait on all of the servers to start:
$status = $manager->wait_for_servers(@servers);
Also check out $manager->stop_all_servers().
$result = $server->terminate
Terminate a server and unregister it from the manager. This method will stop and wait until the server is terminated.If you wish to stop a set of servers without waiting for each one individually, then you may call the underling instance's start() method:
$server->instance->terminate;
Remote Shell Methods
The methods in this section allow you to run remote commands on the staging server and interrogate the results. Since the staging manager handles the creation of SSH keys internally, you do not need to worry about finding the right public/private keypair.$result = $server->ssh(@command)
The ssh() method invokes a command on the remote server. You may provide the command line as a single string, or broken up by argument:
$server->ssh('ls -lR /var/log');
$server->ssh('ls','-lR','/var/log');
The output of the command will appear on STDOUT and STDERR of the perl process. Input, if needed, will be read from STDIN. If no command is provided, then an interactive ssh session will be started on the remote server and the script will wait until you have logged out.
If the remote command was successful, the method result will be true.
$output = $server->scmd(@command)
This is similar to ssh(), except that the standard output of the remote command will be captured and returned as the function result, similar to the way backticks work in perl:
my $output = $server->scmd('date');
print "The localtime for the server is $output";
$fh = $server->scmd_write(@command)
This method executes @command on the remote server, and returns a filehandle that is attached to the standard input of the command. Here is a slightly dangerous example that appends a line to /etc/passwd:
my $fh = $server->scmd_write('sudo -s "cat >>/etc/passwd"');
print $fh "whoopsie:x:119:130::/nonexistent:/bin/false\n";
close $fh;
$fh = $server->scmd_read(@command)
This method executes @command on the remote server, and returns a filehandle that is attached to the standard output of the command. Here is an example of reading syslog:
my $fh = $server->scmd_read('sudo cat /var/log/syslog');
while (<$fh>) {
next unless /kernel/;
print $_;
}
close $fh;
$server->shell()
This method works in an X Windowing environment by launching a new terminal window and running an interactive ssh session on the server host. The terminal window is executed in a fork()ed session, so that the rest of the script continues running. If X Windows is not running, then the method behaves the same as calling ssh() with no arguments.The terminal emulator to run is determined by calling the method get_xterm().
Volume Management Methods
The methods in this section allow you to create and manage volumes attached to the server. These supplement the EC2 facilities for creating and attaching EBS volumes with the ability to format the volumes with a variety of filesystems, and mount them at a desired location.$volume = $server->provision_volume(%args)
Provision and mount a new volume. If successful, the volume is returned as a VM::EC2::Staging::Volume object.Arguments (default):
-name Symbolic name for the desired volume (autogenerated) -fstype Filesystem type for desired volume (ext4) -size Size for the desired volume in GB (1) -mtpt Mountpoint for this volume (/mnt/Staging/$name) -mount Alias for -mtpt -volume_id ID of existing volume to attach & mount (none) -snapshot_id ID of existing snapshot to use to create this volume (none) -reuse Reuse an existing managed volume of same name (false) -label Disk label to assign during formatting ($name) -uuid UUID to assign during formatting (none)
None of the arguments are required, and reasonable defaults will be chosen if they are missing.
The -name argument specifies the symbolic name to be assigned to the newly-created staging volume. The name allows the staging manager to retrieve this volume at a later date if it is detached from the server and returned to the available pool. If no name is provided, then an arbitrary one will be autogenerated.
The -fstype argument specifies the filesystem to be generated on the volume, ext4 by default. The following filesystems are currently supported: ext2, ext3, ext4, xfs, reiserfs, jfs, ntfs, nfs, vfat, msdos. In addition, you can specify a filesystem of ``raw'', which means to provision and attach the volume to the server, but not to format it. This can be used to set up LVM and RAID devices. Note that if the server does not currently have the package needed to manage the desired filesystem, it will use ``apt-get'' to install it.
The -mtpt and -mount arguments (they are equivalent) specify the mount point for the volume on the server filesystem. The default is ``/mnt/Staging/$name'', where $name is the symbolic name provided by -name or autogenerated. No checking is done on the sensibility of the mount point, so try to avoid mounting disks over essential parts of the system.
-volume_id and -snapshot_id instruct the method to construct the staging volume from an existing EBS volume or snapshot. -volume_id is an EBS volume ID. If provided, the volume must be located in the server's availability zone and be in the ``available'' state. -snapshot_id is an EBS snapshot ID in the server's region. In no case will provision_volume() attempt to reformat the resulting volume, even if the -fstype argument is provided. However, in the case of a volume created from a snapshot, you may specify a -size argument larger than the snapshot and the filesystem will be dynamically resized to fill the requested space. This currently only works with ext2, ext3 and ext4 volumes, and cannot be used to make filesystems smaller.
If the -reuse argument is true, and a symbolic name is provided in -name, then the method will look for an available staging volume of the same name and mount this at the specified location. If no suitable staging volume is found, then the method will look for a snapshot created earlier from a staging volume of the same name. If neither a suitable volume nor a snapshot is available, then a new volume is provisioned. This is intended to support the following use case of synchronizing a filesystem somewhere to an EBS snapshot:
my $server = $staging_manager->get_server('my_server');
my $volume = $server->provision_volume(-name=>'backup_1',
-reuse => 1,
-fstype => 'ext3',
-size => 10);
$volume->put('[email protected]:my_music');
$volume->create_snapshot('music_backups');
$volume->delete;
The -label and -uuid arguments are used to set the volume label and UUID during formatting of new filesystems. The default behavior is to create no label and to allow the server to choose an arbitrary UUID.
$volume = $server->add_volume(%args)
This is the same as provision_volume().@volumes = $server->volumes()
Return a list of all the staging volumes attached to this server. Unmanaged volumes, such as the root volume, are not included in the list.$server->unmount_volume($volume)
Unmount the volume $volume. The volume will remain attached to the server. This method will die with a fatal error if the operation fails.See VM::EC2::Staging::Volume->detach() for the recommended way to unmount and detach the volume.
$server->detach_volume($volume)
Unmount and detach the volume from the server, waiting until EC2 reports that the detachment completed. A fatal error will occur if the operation fails.$server->mount_volume($volume [,$mountpt])
Mount the volume $volume using the mount information recorded inside the VM::EC2::Staging::Volume object (returned by its mtpt() and mtdev() methods). If the volume has not previously been mounted on this server, then it will be attached to the server and a new mountpoint will be allocated automatically. You can change the mount point by specifying it explicitly in the second argument.Here is the recommended way to detach a staging volume from one server and attach it to another:
$server1->detach_volume($volume); $server2->mount_volume($volume);
This method will die in case of error.
$server->remount_volume($volume)
This is similar to mount_volume(), except that it will fail with a fatal error if the volume was not previously mounted on this server. This is to be used when temporarily unmounting and remounting a volume on the same server:
$server->unmount_volume($volume); # do some work on the volume $server->remount_volume($volume)
$server->delete_volume($volume)
Unmount, detach, and then delete the indicated volume entirely.$snap = $server->create_snapshot($volume,$description)
Unmount the volume, snapshot it using the provided description, and then remount the volume. If successful, returns the snapshot.The snapshot is tagged with the identifying information needed to associate the snapshot with the staging volume. This information then used when creating new volumes from the snapshot with $server->provision_volume(-reuse=>1).
Data Copying Methods
The methods in this section are used to copy data from one staging server to another, and to copy data from a local file system to a staging server.$result = $server->rsync($src1,$src2,$src3...,$dest)
This method is a passthrough to VM::EC2::Staging::Manager->rsync(), and provides efficient file-level synchronization (rsync) file-level copying between one or more source locations and a destination location via an ssh tunnel. Copying among arbitrary combinations of local and remote filesystems is supported, with the caveat that the remote filesystems must be contained on volumes and servers managed by this module (see below for a workaround).You may provide two or more directory paths. The last path will be treated as the copy destination, and the source paths will be treated as copy sources. All copying is performed using the -avz options, which activates recursive directory copying in which ownership, modification times and permissions are preserved, and compresses the data to reduce network usage.
Source paths can be formatted in one of several ways:
/absolute/path
Copy the contents of the directory /absolute/path located on the
local machine to the destination. This will create a
subdirectory named "path" on the destination disk. Add a slash
to the end of the path (i.e. "/absolute/path/") in order to
avoid creating this subdirectory on the destination disk.
./relative/path
Relative paths work the way you expect, and depend on the current
working directory. The terminating slash rule applies.
$staging_server:/absolute/path
Pass a staging server object and absolute path to copy the contents
of this path to the destination disk. Because of string interpolation
you can include server objects in quotes: "$my_server:/opt"
$staging_server:relative/path
This form will copy data from paths relative to the remote user's home
directory on the staging server. Typically not very useful, but supported.
$staging_volume
Pass a VM::EC2::Staging::Volume to copy the contents of the
volume to the destination disk starting at the root of the
volume. Note that you do *not* need to have any knowledge of the
mount point for this volume in order to copy its contents.
$staging_volume:/absolute/path
Copy a subdirectory of a staging volume to the destination disk.
The root of the volume is its top level, regardless of where it
is mounted on the staging server. Because of string
interpolation magic, you can enclose staging volume object names
in quotes in order to construct the path, as in
"$picture_volume:/family/vacations/". As in local paths, a
terminating slash indicates that the contents of the last
directory in the path are to be copied without creating the
enclosing directory on the desetination. Note that you do *not*
need to have any knowledge of the mount point for this volume in
order to copy its contents.
$staging_volume:absolute/path
$staging_volume/absolute/path
These are alternatives to the previous syntax, and all have the
same effect as $staging_volume:relative/path. There is no
The same syntax is supported for destination paths, except that it makes no difference whether a path has a trailing slash or not.
Note that neither the source nor destination paths need to reside on this server.
See VM::EC2::Staging::Manager->rsync() for examples and more details.
$server->dd($source_vol=>$dest_vol)
This method is a passthrough to VM::EC2::Staging::Manager->dd(), and performs block-level copying of the contents of $source_vol to $dest_vol by using dd over an SSH tunnel, where both source and destination volumes are VM::EC2::Staging::Volume objects. The volumes must be attached to a server but not mounted. Everything in the volume, including its partition table, is copied, allowing you to make an exact image of a disk.The volumes do not actually need to reside on this server, but can be attached to any staging server in the zone.
$server->put($source1,$source2,$source3,...,$dest)
Use rsync to copy the indicated source directories into the destination path indicated by $dest. The destination is either a path on the server machine, or a staging volume object mounted on the server (string interpolation is accepted). The sources can be local paths on the machine the perl script is running on, or any of the formats described for rsync().Examples:
$server1->put("$ENV{HOME}/my_pictures" => '/var/media');
$server1->put("$ENV{HOME}/my_pictures","$ENV{HOME}/my_audio" => '/var/media');
$server1->put("$ENV{HOME}/my_pictures" => "$backup_volume/home_media");
$server1->put("[email protected]:media/" => "$backup_volume/home_media");
$server->get($source1,$source2,$source3,...,$dest)
Use rsync to copy the indicated source directories into the destination path indicated by $dest. The source directories are either paths on the server, or staging volume(s) mounted on the server (string interpolation to indicate subdirectories on the staging volume also works). The destination can be any of the path formats described for rsync(), including unmanaged hosts that accept ssh login.Examples:
$server1->get('/var/media' =>"$ENV{HOME}/my_pictures");
$server1->get('/var/media','/usr/bin' => "$ENV{HOME}/test");
$server1->get("$backup_volume/home_media" => "$ENV{HOME}/my_pictures");
$server1->get("$backup_volume/home_media" => "[email protected]:media/");
Internal Methods
This section documents internal methods. They are not intended for use by end-user scripts but may be useful to know about during subclassing. There are also additional undocumented methods that begin with a ``_'' character which you can explore in the source code.$description = $server->volume_description($vol)
This method is called to get the value of the Name tag assigned to new staging volume objects. The current value is ``Staging volume for $name created by VM::EC2::Staging::Server.''You will see these names associated with EBS volumes in the AWS console.
($ebs_device,$local_device) = $server->unused_block_device([$major_start])
This method returns an unused block device path. It is invoked when provisioning and mounting new volumes. The behavior is to take the following search path:
/dev/sdf1 /dev/sdf2 ... /dev/sdf15 /dev/sdfg1 ... /dev/sdp15
You can modify the search path slightly by providing a single character major start. For example, to leave all the sdf's free and to start the search at /dev/sdg:
($ebs_device,$local_device) = $server->unused_block_device('g');
The result is a two element list consisting of the unused device name from the perspective of EC2 and the server respectively. The issue here is that on some recent Linux kernels, the EC2 device /dev/sdf1 is known to the server as /dev/xvdf1. This module understands that complication and uses the EC2 block device name when managing EBS volumes, and the kernel block device name when communicating with the server.
$flag = $server->has_key($keyname)
Returns true if the server has a copy of the private key corresponding to $keyname. This is used by the rsync() method to enable server to server data transfers.$flag = $server->accepts_key($keyname)
Returns true if the server has a copy of the public key part of $keyname in its .ssh/authorized_keys file. This is used by the rsync() method to enable server to server data transfers.$up = $server->is_up([$new_value])
Get/set the internal is_up() flag, which indicates whether the server is up and running. This is used to cache the results of the ping() method.$path = $server->default_mtpt($volume)
Given a staging volume, return its default mount point on the server ('/mnt/Staging/'.$volume->name). Can also pass a string corresponding to the volume's name.$server->info(@message)
Log a message to standard output, respecting the staging manager's verbosity() setting.Subclassing
For reasons having to do with the order in which objects are created, VM::EC2::Staging::Server is a wrapper around VM::EC2::Instance rather than a subclass of it. To access the VM::EC2::Instance object, you call the server object's instance() method. In practice this means that to invoke the underlying instance's method for, say, start() you will need to do this:
$server->instance->start();
rather than this:
$server->SUPER::start();
You may subclass VM::EC2::Staging::Server in the usual way.
AUTHOR
Lincoln Stein <[email protected]>.Copyright (c) 2011 Ontario Institute for Cancer Research
This package and its accompanying libraries is free software; you can redistribute it and/or modify it under the terms of the GPL (either version 1, or at your option, any later version) or the Artistic License 2.0. Refer to LICENSE for the full license text. In addition, please see DISCLAIMER.txt for disclaimers of warranty.