Tuesday, February 18, 2014

PowerShell Script to launch and remotely connect to an EC2 Instance

AWS offers rich command line interfaces for automation. Pretty much anything you do from the AWS console can be done from the command line. In this blog, I will cover an end to end EC2 sample that will launch an instance, set the firewall rules, create key pairs, enables PS remoting, connect to the instance remotely and finally terminate it. This script is intended to cover the concepts as opposed to using it in production. Please be warned that there is no error checking in this script.

AWS offers multiple flavors of command line interface (CLI) for Windows. They are,
·         PowerShell based http://aws.amazon.com/powershell/.
·         AWS CLI http://aws.amazon.com/cli/. One executable that drives all the services. This one can run both in Windows and Linux.
·         Amazon EC2 CLI is a community contributed Java based command line tool http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/InstallEC2CommandLineTools.html

I will cover the PS based script in detail, as Windows users will likely use this model. For other flavors, I will give a pointer to equivalent script, should be easy to follow.

Prerequisites and Helper function

·         Sign up for AWS and get the AccessKey & SecretKey. You can find info about AWS Account and Access Keys at link.
·         Install PowerShell module from link. All the AWS tools for Windows are nicely bundled as a single MSI. Setup instructions can be found at link
·         First thing first, to ensure that AWS PS module is loaded correctly, do the following


# PSModule should be autoloaded in PS v3.0 and above
# setup adds AWS module location to the PSModulePath
# To manually load PSModule
# import-module "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"


# To check if AWS PS module is accessible.
Get-AWSPowerShellVersion -ListServices


·         Call Initialize-AWSDefaults to setup the default AccessKey and SecretKey to use. This way, AccessKey and SecretKey need not be provided for every cmdlet invocation. This is very handy. Replace AccessKey and SecretKey with yours!


Initialize-AWSDefaults -AccessKey AKIAIOSFODNN7EXAMPLE -SecretKey wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY -Region us-east-1


·         Since the EC2 instance that we are going to create is not a domain joined machine, it has to be added to the trusted hosts. The computers in the TrustedHosts list are not authenticated. (i.e.) There is no way for the client to know if it is talking to the right machine. The client may send credential information to these computers. Either add the specific DNSName or “*” to trust any machine. Since it is for the testing purpose, I chose to add “*”


Set-Item WSMan:\localhost\Client\TrustedHosts "*" -Force


·         Helper function to wait for the desired EC2 instance state.


function WaitForState ($instanceid, $desiredstate)
{
    while ($true)
    {
        $a = Get-EC2Instance -Filter @{Name = "instance-id"; Values = $instanceid}
        $state = $a.Instances[0].State.Name
        if ($state -eq $desiredstate)
        {
            break;
        }
        "$(Get-Date) Current State = $state, Waiting for Desired State=$desiredstate"
        Sleep -Seconds 5
    }
}


Firewall Security Groups

A security group acts as a firewall that controls the traffic for one or more instances. When you launch an instance, you associate one or more security groups with the instance. You add rules to each security group that allow traffic to or from its associated instances. You can modify the rules for a security group at any time; the new rules are automatically applied to all instances that are associated with the security group. When we decide whether to allow traffic to reach an instance, we evaluate all the rules from all the security groups that are associated with the instance. More info here

The script below creates a security group named “sg1”, adds firewall exceptions for PowerShell Remoting, Remote Desktop and ICMP. Allowing ICMP enables “ping” to function, helps with debugging. The example below opens up to any IpRange, which means that the EC2 instance can be contacted from anywhere in the world.


$groupid = New-EC2SecurityGroup sg1  -Description "Security group 1"
Get-EC2SecurityGroup -GroupNames sg1
Grant-EC2SecurityGroupIngress -GroupName sg1 -IpPermissions @{IpProtocol = "icmp"; FromPort = -1; ToPort = -1; IpRanges = @("0.0.0.0/0")}
$ipPermissions = New-Object Amazon.EC2.Model.IpPermission
$ipPermissions.IpProtocol = "tcp"
$ipPermissions.FromPort = 3389
$ipPermissions.ToPort = 3389
$ipPermissions.IpRanges.Add("0.0.0.0/0")
Grant-EC2SecurityGroupIngress -GroupName sg1 -IpPermissions $ipPermissions
Grant-EC2SecurityGroupIngress -GroupName sg1 -IpPermissions @{IpProtocol = "udp"; FromPort = 3389; ToPort = 3389; IpRanges = @("0.0.0.0/0")}
Grant-EC2SecurityGroupIngress -GroupName sg1 -IpPermissions @{IpProtocol = "tcp"; FromPort = 5985; ToPort = 5986; IpRanges = @("0.0.0.0/0")}



EC2 Key Pairs

Amazon EC2 uses public–key cryptography to encrypt and decrypt login information. Public–key cryptography uses a public key to encrypt a piece of data, such as a password, then the recipient uses the private key to decrypt the data. The public and private keys are known as a key pair.

To log in to your instance, you must create a key pair, specify the name of the key pair when you launch the instance, and provide the private key when you connect to the instance. You use a key pair to obtain the administrator password and then log in using RDP. More info here.

The script below creates a new key pair and saves the private key under $folder. Note you have to use ASCII encoding instead of the default encoding. The key name and fingerprint are stored in the same file, which help to correlate. The private key is saved locally, if you lose it there is no way to retrieve it. AWS does not store the private key.


#create a KeyPair, this is used to encrypt the Administrator password.
$keypair1 = New-EC2KeyPair -KeyName keypair1
"$($keypair1.KeyMaterial)" | out-file -encoding ascii -filepath $folder\keypair1.pem
"KeyName: $($keypair1.KeyName)" | out-file -encoding ascii -filepath $folder\keypair1.pem -Append
"KeyFingerprint: $($keypair1.KeyFingerprint)" | out-file -encoding ascii -filepath $folder\keypair1.pem -Append


Find the image

When you launch an Amazon EC2 instance, you need to specify an Amazon Machine Image (AMI) for the instance configuration in which you are interested. However, the IDs for the AMIs change periodically because AWS updates these images with new features and security enhancements. Get-EC2Image and Get-EC2ImageByName cmdlets are used to find the current ID.


$a = Get-EC2Image -Filters @{Name = "name"; Values = "Windows_Server-2012-RTM-English-64Bit-Base*"}
$imageid = $a.ImageId


Alternatively,

#use Get-EC2ImageByName to the list of images
Get-EC2ImageByName
#get the specific image
$a = Get-EC2ImageByName -Names WINDOWS_2012_BASE
$imageid = $a.ImageId


Launch EC2 instance

Now that the security group and Key Pairs are ready, we are ready to launch the EC2 instance using NewEC2Intance.

EC2 lets you define userdata to run custom scripts during the launch. This metadata is made available from within the instance at http://169.254.169.254/latest/user-data/. This information remains static for the life of the instance, persisting when the instance is stopped and started, until it is terminated. For EC2Config to execute userdata, you must enclose the lines of the script within one of the following special tags:
·         - Run any command that you can run at the cmd.exe prompt.
·         - Run any command that you can run at the Windows PowerShell prompt.

By default, Windows Firewall restricts PS remoting to local subnet only. Set-NetFirewallRule is executed via userdata section to open this up to any address.


$userdata = @"
Set-NetFirewallRule -Name WINRM-HTTP-In-TCP-PUBLIC -RemoteAddress Any
"@

#Userdata has to be base64 encoded
$userdataBase64Encoded = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($userdata))

$a = New-EC2Instance -ImageId $imageid -MinCount 1 -MaxCount 1 -InstanceType t1.micro -KeyName keypair1 -SecurityGroups sg1 -UserData $userdataBase64Encoded
$instanceid = $a.Instances[0].InstanceId

#Wait for the running state
WaitForState $instanceid "Running"

$a = Get-EC2Instance -Filter @{Name = "instance-id"; Values = $instanceid}
$publicDNS = $a.Instances[0].PublicDnsName

#Wait for ping to succeed
while ($true)
{
    ping $publicDNS
    if ($LASTEXITCODE -eq 0)
    {
        break
    }
    "$(Get-Date) Waiting for ping to succeed"
    Sleep -Seconds 10
}


Extract the password

Get-EC2PasswordData is used to extract the password by passing the private key file created earlier. This is tried in a loop until the password is ready.


$password = $null
#Wait until the password is available
#blindsly eats all the exceptions, bad idea for a production code.
while ($password -eq $null)
{
    try
    {
        $password = Get-EC2PasswordData -InstanceId $instanceid -PemFile $folder\keypair1.pem -Decrypt
    }
    catch
    {
        "$(Get-Date) Waiting for PasswordData to be available"
        Sleep -Seconds 10
    }
}


Remote PowerShell

The script below establishes a remote session, invokes a remote command (using Invoke-Command), then cleans up the session. The remote command executed is “Invoke-WebRequest” to obtain the userdata. Since the instance might not be fully initialized, this is tried in a loop.


$securepassword = ConvertTo-SecureString $password -AsPlainText -Force
$creds = New-Object System.Management.Automation.PSCredential ("Administrator", $securepassword)

#Wait until PSSession is available
while ($true)
{
    $s = New-PSSession $publicDNS -Credential $creds 2>$null
    if ($s -ne $null)
    {
        break
    }

    "$(Get-Date) Waiting for remote PS connection"
    Sleep -Seconds 10
}

Invoke-Command -Session $s {(Invoke-WebRequest http://169.254.169.254/latest/user-data).RawContent}

Remove-PSSession $s


Terminate the instance and cleanup

Script below terminates the instance and then cleans up the security group and key pair created.


#Terminate the Instance
Get-EC2Instance -Filter @{Name = "instance-id"; Values = $instanceid} | Stop-EC2Instance -Force -Terminate

#Wait for the instance state to be terminated.
WaitForState $instanceid "Terminated"

Remove-EC2KeyPair -KeyName keypair1 -Force

#There is a timing thing, so has to retry.
$err = $true
while ($err)
{
    $err = $false
    try
    {
        Remove-EC2SecurityGroup -GroupName sg1 -Force
    }
    catch
    {
        $err = $true
    }
}


Alternate CLI

1.       AWS CLI based script can be found here.
2.       Amazon EC2 CLI based script can be found here.

You can find all the scripts under AWS folder at https://github.com/padisetty/Samples.

Explore & Enjoy!
/Siva

16 comments:

  1. Good Post and thanks...
    Could you please help me how to find out the instance id after provisioned the instance using powershell script...without open the aws console how to get the instance id of the newly provisioned VM automatically

    ReplyDelete
  2. Sorry for the delay, I was travelling. Please see the code fragment to get the instanceid (This is already shown above):

    $a = New-EC2Instance -ImageId $imageid -MinCount 1 -MaxCount 1 -InstanceType t1.micro -KeyName keypair1 -SecurityGroups sg1 -UserData $userdataBase64Encoded
    $instanceid = $a.Instances[0].InstanceId

    Also, you can use Get-EC2Instance

    ReplyDelete
  3. Great stuff; a couple of corrections need:

    $imageid = $a.ImageId --> $imageid = $a.ImageId[0]

    since there are multiple images ( when using 2012R2 as well) only want the latest

    This part here:

    $userdata = @"
    Set-NetFirewallRule -Name WINRM-HTTP-In-TCP-PUBLIC -RemoteAddress Any
    "@

    needs to be

    $userdata = @"
    Set-NetFirewallRule -Name WINRM-HTTP-In-TCP-PUBLIC -RemoteAddress Any
    "@

    You're missing the tags which means it won't run your script

    Very nice anyway; very helpful

    ReplyDelete
    Replies
    1. edit: something is swallowing the " < powershell > " tags

      Delete
  4. Can anyone help me with Deploying New EC2 instance via powershell with selecting proper IAM role & having proper tagging.

    ReplyDelete
    Replies
    1. You can find a sample code here https://github.com/padisetty/Samples/blob/master/AWS/ssm/setup.ps1

      Delete
  5. Hi Siva, when i run the Launch EC2 script, i get the below error:
    ping : The term 'ping' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the
    spelling of the name, or if a path was included, verify that the path is correct and try again.

    ReplyDelete
    Replies
    1. ping.exe is part of Windows and can be found under C:\Windows\System32 (note: Your Windows directory might be different). Secondly c:\Windows\System32 should be part of your system path.

      Delete

  6. Excellent script, thank you for sharing this information, for me it was a new and very useful. I understand how to use it)
    Richard Brown dataroom software

    ReplyDelete
    Replies
    1. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Java developer learn from Java Training in Chennai. or learn thru Java Online Training in India . Nowadays Java has tons of job opportunities on various vertical industry.

      Delete
  7. how to load data into linux instance using powershell script?

    ReplyDelete
  8. It's definitely not for me, I can not do it, I will not even try. I found another way to solve this issue, I realized that you can order the development of a script http://www.nixsolutions.com/services/custom-software-development/. They did it for ridiculous money, this is the best solution for those who do not waste time in the empty.

    ReplyDelete
  9. Hi Siva.Awesome Script. Thanks for sharing. I'm not able to ping my Public and private ip of the instance. I enabled the port by following this link https://serverfault.com/questions/356598/why-cant-i-reach-my-amazon-ec2-instance-via-its-elastic-ip-address but still no go. Any advise on why i'm unable to ping the ip would be helpful. Thanks in advance

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. Almost 4 years after it was written this is still a wonderfully useful article that helped me a lot! I believe I found one omission: the user data needs to be bracketed by powershell start and end tags to work correctly. Somebody already made the same comment above, but the tags in the example they gave were suppressed by this site.



    Thanks very much!

    ReplyDelete