Introduction¶
AWS provides many ways to control its resources. One of the fastest and the most effective way is the AWS CLIs and SDKs. The best benefit of using CLI or SDKs it that you can automate complex workflow.
Depending on your expertise, you can use a bash script to control the AWS resources. You can use many programming languages, like .NET, Python, Go, etc. to control the AWS resources. You can use the Postman client to make HTTP API calls.
This article will provide an introduction to use the AWS CLI, SDKs and the HTTP APIs provided by AWS. Using these techniques, to control the AWS resources can save a lot of time.
Come and explore the work of AWS CLIs, SDKs and Postman Client.
Prerequisite¶
You can use pip and venv modules of Python for installing CLI and SDKs. pip is the Python package manager. venv is the virtual environment.
Python is cross platform. If you use Python, then the process becomes easier for Windows, Linux and MacOS.
There is a new version of AWS CLI V 2.x which do not support installation via pip. You will install only AWS CLI V 2.x via the package installer method mentioned below by AWS.
Install Python on Windows, Linux or macOS¶
There are many resources out there to install Python on these platforms. I cannot make any improvement over these installation instructions.
Please follow this RealPython article, to install Python on your respective operating system.
Execute the below command to confirm the installation is complete.
python --version
If you get any version information in return, you have installed Python.
You should now proceed to install the next important package called pip.
Install PiP on Window, Linux or macOS¶
Once you have Python installed, the next step would be to install the package manager called pip. It is the default installation on Python 2 version >= 2.7.9 and Python 3 version >= 3.4. You can check the installation by executing this command.
python -m pip --version
pip is already present, if you get the version information in response to the command. If not present you can use this documentation to install pip.
Once you have pip installed, the next step is to create a virtual environment using the package venv.
Create Virtual Environment¶
You can have separate environments for your Python installation. This helps in not polluting the global package installation.
Create a directory name aws-cli in your local machine and follow along. Change to this aws-cli directory.
Create a virtual environment
python -m venv .env
If no error comes then there will be a .env inside aws-cli. If we get an error of package not found, please install the virtual environment package, using pip.
The .env is generally used to identify an environment folder, but you are free to use any name.
python -m pip install --user virtualenv
Once you have your .env folder created now we can proceed with the AWS CLI and SDK installation.
You are all set to install AWS CLI and SDK.
AWS CLI¶
AWS CLI is a tool to manage many services. If you choose the Amazon Linux AMI, the AWS CLI is pre-installed.
AWS CLI is an open source tool. AWS Management console provides the same functionality as AWS CLI. AWS CLI should have the new AWS IaaS administration features available. 180 days is the time limit for AWS CLI to have these new AWS IaaS features.
You get access to only the public APIs. While using AWS CLI, you do not have any special APIs to use, all the APIs are available in public.
AWS CLI comes in two versions, the most recent version is 2.x, it can be used in a production environment. 2.x version is not available as a Python package for installation through pip.
You have to use a separate installer to install the version 2.x. The support for AWS ClI v1 is still available.
There is an unofficial port of AWS ClI v2 on pypi called awscliv2.
If you want to install v2, jump to Install AWS CLI v2.
Install AWS CLI v1¶
Activate the virtual environment inside aws-cli folder. The below command works on Linux/macOS. It may be little different on Windows.
source .env/bin/activate
After activating the virtual environment, execute this command.
python -m pip install awscli
Once you have installed, it will show this at the end.
Successfully installed PyYAML-5.4.1 awscli-1.19.62 botocore-1.20.62 colorama-0.4.3 docutils-0.15.2 jmespath-0.10.0 pyasn1-0.4.8 python-dateutil-2.8.1 rsa-4.7.2 s3transfer-0.4.2 six-1.15.0 urllib3-1.26.4
If you see, while installing awscli it installs the above dependencies, one of that dependency is botocore. botocore is the base for even boto3.
You can verify the installation by running the command.
aws --version
If the version is successfully installed, you will get an output as aws-cli/1.19.62 Python/3.7.5 Darwin/18.6.0 botocore/1.20.62.
You have awscli v1 installed.
Install AWS CLI v2¶
For macOS:-
Download the latest AWS CLI v2 packages from AWS CLI pkg Download.
Install the package, by following the instructions. At the end, aws would be installed.
If you have both AWS CLI v1 and v2 installed, then it will pick the executable which is first in the PATH variable. Confirm the installation by opening a new terminal and executing.
aws --version
You should get an output as
aws-cli/2.2.1 Python/3.8.8 Darwin/18.6.0 exe/x86_64 prompt/off
The above output proves that we have the AWS CLI v2 installed. Now you should execute a sample command to see aws cli in action.
describeimages api using cli¶
Before you execute any aws command we should configure the aws environment. Please follow the instruction already mentioned. See AWS CLI configure AWS environment.
Once you have your AWS environment configured. Try running some operation on the AWS CLI.
Try the command mentioned in What is AMI? We will execute the describe-images command on an ec2.
aws ec2 describe-images --image-ids ami-0d758c1134823146a
If executed successfully we will get an output as
{
"Images": [
{
"Architecture": "x86_64",
"CreationDate": "2021-02-24T18:24:50.000Z",
"ImageId": "ami-0d758c1134823146a",
"ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223",
"ImageType": "machine",
"Public": true,
"OwnerId": "099720109477",
"PlatformDetails": "Linux/UNIX",
"UsageOperation": "RunInstances",
"State": "available",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-072d11ffd95664698",
"VolumeSize": 8,
"VolumeType": "gp2",
"Encrypted": false
}
},
{
"DeviceName": "/dev/sdb",
"VirtualName": "ephemeral0"
},
{
"DeviceName": "/dev/sdc",
"VirtualName": "ephemeral1"
}
],
"Description": "Canonical, Ubuntu, 20.04 LTS, amd64 focal image build on 2021-02-23",
"EnaSupport": true,
"Hypervisor": "xen",
"Name": "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223",
"RootDeviceName": "/dev/sda1",
"RootDeviceType": "ebs",
"SriovNetSupport": "simple",
"VirtualizationType": "hvm"
}
]
}
The above command and output is same in v1 and v2. Congratulations on executing your first AWS CLI command successfully.
You should continue your command line journey with a little knowledge on AWS SDKs.
AWS SDK | Python | Boto3¶
SDKs take the complexity out of coding. They provide language specific APIs for AWS service. AWS SDK is available in these various languages.
- JavaScript
- Python
- PHP
- .NET
- Ruby
- Java
- Go
- Node.js
- C++
You can use any of the SDK for controlling the AWS resources. You will learn to use the Python SDK called Boto3..
The Python Boto3 provides two key Python packages.
- Botocore - Library providing low-level functionality shared with Python SDK and the AWS CLI
- Boto3 - The Python SDK.
Installation of Boto3 is very easy. Activate the Python virtual environment, which we already did Install AWS CLI v1
Once you have activated the virtual environment, just execute this command.
python -m pip install boto3
Once the installation is done, it will give this message.
Successfully installed boto3-1.17.62
You can again execute the configure option, if not done as explained in AWS CLI configure AWS environment.
describe_images API using Python Boto3¶
Once installation is complete, we can execute the same describe_images to get the details of the AMI.
import boto3
import pprint
ec2 = boto3.client("ec2")
ImageIds = ["ami-0d758c1134823146a"]
response = ec2.describe_images(ImageIds=ImageIds)
pprint.pprint(response[""])
Once you execute the output would be.
{
"Images":[
{
"Architecture":"x86_64",
"BlockDeviceMappings":[
{
"DeviceName":"/dev/sda1",
"Ebs":{
"DeleteOnTermination":true,
"Encrypted":false,
"SnapshotId":"snap-072d11ffd95664698",
"VolumeSize":8,
"VolumeType":"gp2"
}
},
{
"DeviceName":"/dev/sdb",
"VirtualName":"ephemeral0"
},
{
"DeviceName":"/dev/sdc",
"VirtualName":"ephemeral1"
}
],
"CreationDate":"2021-02-24T18:24:50.000Z",
"Description":"Canonical, Ubuntu, 20.04 LTS, amd64 focal image ""build on 2021-02-23",
"EnaSupport":true,
"Hypervisor":"xen",
"ImageId":"ami-0d758c1134823146a",
"ImageLocation":"099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223",
"ImageType":"machine",
"Name":"ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223",
"OwnerId":"099720109477",
"PlatformDetails":"Linux/UNIX",
"Public":true,
"RootDeviceName":"/dev/sda1",
"RootDeviceType":"ebs",
"SriovNetSupport":"simple",
"State":"available",
"UsageOperation":"RunInstances",
"VirtualizationType":"hvm"
}
],
"ResponseMetadata":{
"HTTPHeaders":{
"cache-control":"no-cache, no-store",
"content-length":"2184",
"content-type":"text/xml;charset=UTF-8",
"date":"Sun, 02 May 2021 13:42:28 GMT",
"server":"AmazonEC2",
"strict-transport-security":"max-age=31536000; ""includeSubDomains",
"vary":"accept-encoding",
"x-amzn-requestid":"bd357758-f81e-450d-bc7f-5764944d5186"
},
"HTTPStatusCode":200,
"RequestId":"bd357758-f81e-450d-bc7f-5764944d5186",
"RetryAttempts":0
}
}
If you see the output, it is almost similar to the output when we executed the AWS CLI command for the same describe_images. see describeimages api using cli
Using Postman to execute AWS REST API¶
What is Postman?
As described by the company itself.
Postman is a collaboration platform for API development.
This tells us that we can use Postman, to test the following.
- APIs
- Automated testing on the APIs etc.
If you do not want to write code for getting some tasks done, we can use Postman, for that. Configuring Postman to achieve it is little complicated so please keep the focus.
You have understood about these 2 concepts in Postman to use it with AWS.
- Environment
- Collections
Configure Postman environment for AWS¶
As mentioned in Postman documentation, an environment is
A Postman environment is a set of variables, which you can use in Postman request.
You can use Postman environment to create a production or a staging environment. In our case we will create these 4 variables in the environment.
- region - providing the AWS region, where to execute the Action
- accountId - the 12 digit account ID
- accessKey - The accessKey which is present when IAM console
- secretAccessKey - The secret Access Key which we get once creating a new user, see AWS CLI configure AWS environment.
Create Postman environment for AWS¶
Let first create a Postman environment for AWS
You can see above you have to provide these 4 information in the environment, you can name your environment anything, I have named it aws-environment.
Once you have the environment, it's time for collection.
Create Postman collection for AWS¶
As mentioned in Postman documentation, a collection is
A Group of saved request which is organized into folders.
Create a new collection as shown below.
When you create a new collection, you should provide the Authorization details. Select AWS Signature as shown above.
You should provide all the AWS environment variable, we created in this collection.
You can reference all the AWS environment variables inside the Authorization panel of Postman collection by using {{accessKey}}, i.e. surrounding it inside {{ }}.
To verify if your variable are set properly, hover over the {{accessKey}}, if you get an Unresolved variable error like this,
This can happen, because the environment, we set is not accessible to this collection. Please check this drop down and choose the appropriate environment.
Once the environment is properly set, we can use the PostMan Collection to fire our request.
describeimages api using Postman¶
Once your Postman environment and collection is set, you can fire the same DescribeImages API to if the Postman is working.
As shown above, create a new request inside your existing collection which we create in the previous step.
You have to provide these Query Params.
- Action - The
DescribeImages - Version - This version should be
2016-11-15, which is the last document version, do not change it. - ImageId.1 - The AMI id you want to search.
If everything goes right, you should get a response in XML like below.
<?xml version="1.0" encoding="UTF-8"?>
<DescribeImagesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
<requestId>961658cb-8af7-4d49-85d6-9012ac662616</requestId>
<imagesSet>
<item>
<imageId>ami-0d758c1134823146a</imageId>
<imageLocation>099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223</imageLocation>
<imageState>available</imageState>
<imageOwnerId>099720109477</imageOwnerId>
<creationDate>2021-02-24T18:24:50.000Z</creationDate>
<isPublic>true</isPublic>
<architecture>x86_64</architecture>
<imageType>machine</imageType>
<sriovNetSupport>simple</sriovNetSupport>
<name>ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210223</name>
<description>Canonical, Ubuntu, 20.04 LTS, amd64 focal image build on 2021-02-23</description>
<rootDeviceType>ebs</rootDeviceType>
<rootDeviceName>/dev/sda1</rootDeviceName>
<blockDeviceMapping>
<item>
<deviceName>/dev/sda1</deviceName>
<ebs>
<snapshotId>snap-072d11ffd95664698</snapshotId>
<volumeSize>8</volumeSize>
<deleteOnTermination>true</deleteOnTermination>
<volumeType>gp2</volumeType>
<encrypted>false</encrypted>
</ebs>
</item>
<item>
<deviceName>/dev/sdb</deviceName>
<virtualName>ephemeral0</virtualName>
</item>
<item>
<deviceName>/dev/sdc</deviceName>
<virtualName>ephemeral1</virtualName>
</item>
</blockDeviceMapping>
<virtualizationType>hvm</virtualizationType>
<hypervisor>xen</hypervisor>
<enaSupport>true</enaSupport>
<platformDetails>Linux/UNIX</platformDetails>
<usageOperation>RunInstances</usageOperation>
</item>
</imagesSet>
</DescribeImagesResponse>
You can now experiment with the different HTTP APIs provided by AWS. Create an instance, destroy instance, everything which is possible using the public API.
From here on, the more you practice the more you can enhance the APIs. AWS provides multiple ways to control its resources. One of the fastest and the most effective way is the AWS CLIs and SDKs. The best benefit of using CLI or SDKs it that you can automate complex work flow.
Conclusion¶
AWS CLI, SDKs and HTTP APIs are three alternate ways to access and configure the AWS resources. AWS CLIs and SDKs requires a little bit of programming knowledge. It can be shell scripting or any programming language. HTTP Clients like Postman can use the HTTP APIs to provide the same result.
Python is a preferred choice when it comes to choosing the the AWS SDKs. The AWS CLIs core is also made with a Python Package. AWS has changed the way of installing AWS CLIs from pip to independent packages in Version 2.x.
In this article we have used the describeimage API to execute AWS CLIs, SDKs and HTTP CLient and get the same result in all the three possible ways.
You can proceed with any one of the technique from here on. It may depend on your previous experience. If you are knowledgeable in Shell scripting choose AWS CLI. If you are good in programming, choose the AWS SDKs. You can use the HTTP Client called Postman to fire the HTTP APIs if you have no programming experience.
Knowing AWS CLI, SDKs or HTTP API is one important step in the AWS Configuration. Checkout, the free tier EC2 instance launch article. see Step by Step guide to create an EC2 instance .
Reference¶
- Using Postman to Launch an EC2 Instance via API Calls
- Region and Endpoint Match in AWS API Requests
- Wednesday Tidbit: Using Postman to authenticate to AWS
- DescribeImages
- Audit your AWS infrastructure with Postman
- Postman extension available for AWS Gateway APIs now
- AWS Signature Authorization using Postman
- Step 2: Set up Postman
- Postman Environment Variables: API Testing using Postman - (Part 4)
- AWS CLI: 10 Useful Commands You May Not Know
- Difference between s3cmd, boto and AWS CLI
- publish v2 to PyPI
- Introduction to AWS CLI V2
- AWS CLI v2 is now generally available
- Managing Postman environment.
