Hello world! This time, we're going to go a little more in-depth when it comes to writing web services. We're going to create a web service that uses AWS S3 to store and retrieve images. We will also add telemetry via tracing, look at tests and other common things for productionising a Rust web application.
Interested in deploying or just want to see what the final code looks like? Check it out here.
Pre-requisites
Setting up your S3 bucket
Before we get started, you'll need to set up an S3 Bucket and an IAM user. We'll go through this below.
To create a bucket, do the following:
- Log into AWS Console and go to the S3 section (it can also be found using the search bar)
- Click "Create Bucket" and follow the prompt.
If this is your first time (and you aren't handling sensitive data), it is safe to leave the defaults as they are. Public bucket access is turned off by default.
When using S3, your bucket endpoint will look like the following:
You'll want to make sure this is kept somewhere safe as we'll be using this later on.
Setting up an IAM user
You'll also need two variables which will be found in S3 or any S3-compatible API:
AWS_ACCESS_KEY_ID
(your Access Key)AWS_SECRET_ACCESS_KEY
(your Secret Access Key)
The first two can be found in your IAM user credentials if you've already set a user up. If you don't have an appropriate user with policies, you can get started quickly by doing the following:
- Go to the Users menu
- Start creating a user and go to the "Attach policies" section (then search "S3")
- Here you can either use the "AmazonS3FullAccess" policy which gives you full access to S3 on that user, or you can create a custom policy. Select one and finish creating your user. Access to S3 is required, as otherwise you won't be able to use it!
- Go back to the Users menu and click on your newly created user
- Go to "Access keys" and follow the prompt (clicking "Application outside of AWS"). Don't forget to store your Access Key and Secret Access Key!
In production, you may want to go a step further and create a Group that you can then attach policies and users to!
When using S3-compatible APIs, this may look different depending on the service you're using. However, the documentation should provide enough information for you to create an S3 client for their service.
Getting started
To get started, we'll create a Shuttle service via shuttle init
, making sure to pick the Axum framework. Make sure you have cargo-shuttle
installed!
Next, you'll want to install the following Rust dependencies using the following shell snippet:
We'll want to add our secrets to a Secrets.toml
file located in the project root folder:
Error handling
Before we get started, we'll want to create an error type that can represent all the kinds of errors we can encounter while using the service. There's several reasons to do this:
- It allows error propagation instead of having to manually handle an error every time
- We can use the
From<T>
trait to convert error types from our libraries to our API's error type - It saves time debugging!
In this snippet we use the thiserror::Error
derive macro to be able to quickly derive Display
, Error
and From<T>
all in one by using attribute macros in conjunction with the derive macro.
Next, we implement axum::response::IntoResponse
for our error type. This allows it to be turned into a HTTP response:
Building the base of our S3 microservice
Setting up AWS SDK
To get started, we'll set up some code in our main function that allows us to create an AWS client.
For our region we've used eu-west-2
as the Shuttle servers are in eu-west-2
, which reduces latency. However, feel free to use whichever region you'd like!
We'll also additionally create a shared state struct which will hold the client. When we need to access the client, we can simply add the State
extractor to our functions and it will work.
Creating a custom response type
To make it easier for ourselves when writing our code, we'll create our own enum return type that will implement axum::response::IntoResponse
. While you can use impl IntoResponse
itself as the return type, it is often better to declare a specific type for a couple of reasons:
- While using
impl IntoResponse
, every response type is required to be the same - Using an enum allows you to be more flexible in your response type
As a short illustration, we'll create an enum with two variants and implement IntoResponse
:
Now we can avoid writing our types directly out into the functions! However, we can also take this a step further by implementing Into<Image>
for our types as well as creating helper functions to create our Image
enum easily. Let's implement Into<Image>
for String
and a function to convert a filename with a Vec<u8>
to an image:
Routing
We will get started with a handler function for uploading an image. We will need to deal with multipart form upload data, and as such we'll want to use axum::extract::Multipart
here.
There is a small footnote here: if you're operating with variables that need to function outside of the multipart loop, you need to declare them beforehand as a None
option and re-assign them. This is primarily due to scoping - if you declare it inside the loop, you can't suddenly use it outside the loop again. Whether you need to do this however depends on your use case.
Next, we'll add the code for inserting an object into your S3 object into the comment area:
It is important to note here that we've generated our own filename. It is always more secure to generate your own file names rather than taking the user's filenames, as you may accidentally end up overwriting your own files. Users may also maliciously try to upload files with known names!
Strictly speaking, we don't need the file extension at the end of our file key. However, when you're using said files outside of image storage, it's best to preserve them for future usage.
To retrieve an image, we can write the following route:
Note here that we're setting the filename dynamically. You can also set your Content-Type
header according to the kind of image you're trying to serve from S3.
The handler function for deleting the image is by far the simplest to write: we just need to delete the image from S3.
To wrap it all up, let's add it to our main function:
Extending our web service
While what we've currently got works well, we can much do better. In its current state, it's not super production ready. Let's have a look at what we can do to assist with ensuring production readiness.
Timeout layer
Although we've written our base service and now it works perfectly fine, there are a couple of issues that we'd need to deal with in production:
- We need to stop slow loris attacks (flooding a server with opened connections)
- We need to stop people who want to upload unexpectedly large files, which saves on egress costs
The first point is a rather big deal, as most Rust web frameworks do not deny long-running requests by themselves.
It just needs to be added like below, specifying a timeout duration.
Simple and easy! Our service will now automatically return a timeout error to any request taking longer than 20 seconds (returning the 408 Timeout error).
Tracing
To add tracing to our service, we only need to add the #[tracing::instrument]
macro to our handler functions.
Now whenever anything gets printed out from this endpoint, the whole function will get printed out - application state included!
If you're holding any sensitive data in your application state, you can use the skip
attribute to skip printing it out in logs:
Adding events to our handler functions that then get triggered will automatically send the output to our logs:
Note that Shuttle automatically starts the subscriber from tracing_subscriber
for you. If you want to create your own custom subscriber, you can do that by turning off all default features:
Testing
We can test S3 by using the s3-server
crate. To get started, you only need to install it:
This crate will additionally require the http
crate. Since we're only using it in tests, we can add it as a dev dependency like so:
We can then set up a common function in our project to be able to create an S3 server:
Of course, you'll want to make sure s3-server
is running in the background.
Because Axum itself integrates with most things in the Tower ecosystem, you can either send oneshot requests to your server to test it (requires hyper
installed as dev dependency) or you can start a TcpListener
and start your Axum server up in the usual manner. You would then use reqwest
or a similar library to send HTTP requests to your server:
If you want to do a oneshot request however, you can do so like this (test assumes you have a "Hello, World!" route at /
):
Deploying
To deploy our web service, all we need to do now is shuttle deploy
! Make sure to add the --allow-dirty
flag if on a Git branch with uncommitted changes. If you've added tests, make sure to add the --no-test
flag, as they may not work while deploying. One quick workaround for this is to add a test workflow before deployment.
Finishing up
Thanks for reading! Using the AWS SDK can be difficult. However, hopefully this tutorial on using S3 with Rust can shed some light on writing a fully functioning service that uses S3!
Read more: