How to Use Amazon Web Services S3 Buckets with Python

Last Updated: May 13, 2021

This tutorial will explain how to use Python to utilize the cloud storage capabilities provided by Amazon Web Services. Amazon Web Services (AWS) is a cloud platform that allows users to rent "Virtual Computers" to run programs and store data. Python is one of the most versatile and commonly used programming languages for students, industry and academia. The ability to use Python and AWS together allows you to solve computationally intensive data science problems without investing in an expensive supercomputer.

Steps

1
Make an account for Amazon Web Services through this link: https://portal.aws.amazon.com/billing/signup#/start. .
2
Download the AWS command line interface from this link: https://aws.amazon.com/cli/. Make sure to choose the version corresponding with your operating system. There is currently an option for Linux, MacOS, and 64-bit Windows. After choosing your operating system, a .msi file should start downloading. When the download completes, open the file and follow the installation wizard instructions.
3
Open your computer's command prompt. Type the command "aws --version" and press enter. If the installation was successful, the command prompt will display a message similar to:"aws-cli/1.18.136 Python/3.8.3 Windows/10 botocore/1.17.59" If a similar message is not displayed, repeat the previous step.
4
Go to the AWS Identity Access and Management (IAM) page. Access this service by typing "IAM" into the AWS search bar or looking for IAM under the Security, Identity, & Compliance section on the AWS services page.
5
Create a user with IAM. To do this click the "Users" tab on the left side of the IAM page then press the blue "Add User" at the top of the "Users" page.
6
Configure the user providing a name and access type. Give the user a meaningful name for later reference. There are 2 access types. For our purposes, make sure to choose "Programmatic Access". This will give this user access to all of the AWS development tools.
7
Give user Administrator Access permissions. Under "Set Permissions", choose the "Attach Existing Policies Directly" option. This will open a large list of possible permission types to provide this user. Check the box of the first option called "Administrator Access". This will provide the user complete access to all of the AWS services and data. When finished, press the blue "Next: Tags" button.
8
Review user policies. Skip the "Tags" page by clicking the blue "Next: Review" button. On this screen you can verify the name, access type and permissions are correct. If correct, press the blue "Create User" button.
9
Download Access Key ID and Secret Access Key. To do this, press the "Download .csv" button in the middle of the page. It is extremely important to keep track of these two codes. They cannot be recovered and a new user will have to be created.
10
Configure AWS Command Line Interface. Reopen your computer's command line. Enter the command "aws configure". Enter your Access Key ID and Secret Access Key from the .csv you downloaded in the previous step. Press type "us-west-2" for the default region on the third prompt and press enter for the final prompt. Your computer is now officially connected to AWS.
11
Pip Install Boto3 python library. Boto3 is the free python library that allows for interaction with Amazon Web Services. To install, enter "pip install boto3" into the command prompt. If successful, a series of messages and a loading bar will appear in the command prompt. When loading is completed, it is time to use python.
12
Open your favorite python editor. Start a new python script and import the boto3 library by typing "import boto3" on the first line.
13
Make an S3 bucket. S3 stands for Simple Storage Service and a bucket is a folder in the cloud that you can store files in. Make sure to replace <bucket name> with whatever you want to name your bucket.
```
s3_client = boto3.client('s3')
s3_client.create_bucket(Bucket="<bucket name>")
```
14
Upload a file to AWS. Add the two following commands to your script:
```
client = boto3.client("s3")
client.upload_file(<local file path>, <bucket name>, <S3 Filename>)
```
The first line prepares your code to upload files to S3. The next line requires you to replace <local file path>, <bucket name>, and <S3 Filename>. The local file path is the path to a file on your computer for example "/users/tim/photos/puppy.jpg". The bucket name is the name of your bucket that you made in the previous step and S3 filename is what you want your file to be named in the cloud.
15
Download file from AWS. Use the following 3 commands to download a file from AWS:
```
s3 = boto3.resource("s3")
bucket = s3.Bucket("<bucket name>")
bucket.download_file("<local file path>","<S3 filename>")
```
Use the same values for <bucket name> and <S3 filename > from the previous steps. <local file path> should now represent where you want the file to be downloaded and what it will be named.
1. You can learn more about utilizing more AWS services with python here: .