How to build a Serverless URL shortener using AWS Lambda and S3
Daniel Ireson

How to build a Serverless URL shortener using AWS Lambda and S3

Mon Jan 01 2018

Throughout this post we’ll be building a serverless URL shortener using Amazon Web Services (AWS) Lambda and S3. Whilst you don’t require any previous experience with AWS, I’m assuming familiarity with ES6 JavaScript and Node.js.

There's a demo of what we're building available on S3. If you'd like to see the final codebase you can view the repository on GitHub.

Serverless URL shortener demo

Ironically, the URLs that will be generated from our URL shortener will often be longer than the URLs that they redirect to - this is because we’re using the default S3 bucket website address. Towards the end of the post I’ll discuss how you can add a custom domain to get around this limitation.

Architecture

Before jumping into any development let’s first look at the AWS services we’ll be using to build our URL shortener. To host the website we’ll use the Amazon S3 file storage service with an S3 bucket configured to serve a static website. There’s no capability to execute server-side code but that’s fine for this use case as we only need client-side scripts.

We’ll be using a little known feature of S3 that allows you to setup forwarding for objects inside S3 buckets simply by adding a Website-Redirect-Location value to the metadata of the object. Setting this to a URL will have browsers redirected through a HTTP 301 response and the location header.

An S3 URL is composed of the bucket address followed by the object’s name:

http://{{BUCKET}}.s3-website-eu-west-1.amazonaws.com/{{OBJECT}}

The following is an example of the format for the eu-west-1 region:

http://serverless-url-shortener.s3-website-eu-west-1.amazonaws.com/6GpLcdl

This object name 6GpLcdl at the end of the URL in the example above becomes the shortcode for our shortened URLs. Using this functionality we get native URL redirection as well as storage capabilities. We don’t require a database to store the details of which shortcode points to which URL as this information will instead be stored with the object itself.

We’ll create a Lambda function for saving these S3 objects with the appropriate metadata to our S3 bucket. You could alternatively use the AWS SDK client-side in the browser to save the objects. But it’s better to extract this functionality into a separate service. It provides the advantage of not having to worry about exposing security credentials and is more extendable in the future. We’ll map the Lambda function to an endpoint on API Gateway so it’s accessible through an API call.

Getting started

Head on over to the Serverless Framework docs and run through their quick-start guide. As part of the setup process, you’ll have to install the AWS CLI and configure your AWS credentials.

Start by creating a package.json file at the root of the project.

{
  "name": "serverless-url-shortener",
  "scripts": {},
  "dependencies": {}
}

We know we’ll need to use the AWS SDK, so go ahead and install it from NPM now by entering the following command in your terminal.

npm install aws-sdk --save

Now create a config.json file also at the project root. We’ll use this to store customisable user options in JSON format.

Add the following keys with values appropriate to your setup:

Your config.json file should look similar to the following once complete:

{
  "BUCKET": "your-bucket-name",
  "REGION": "eu-west-1",
  "STAGE": "dev",
}

Next, create another file at the project root, serverless.yml. This will hold our Serverless Framework configuration formatted in the YAML markup language.

Inside this file we’ll start by defining our environment. Notice how we can reference variables stored earlier in config.json.

service: serverless-url-shortener

provider:
  name: aws
  runtime: nodejs6.10
  stage: ${file(config.json):STAGE}
  region: ${file(config.json):REGION}
  iamRoleStatements:
    - Effect: Allow
      Action:
        - s3:PutObject
      Resource: "arn:aws:s3:::${file(config.json):BUCKET}/*"

The iamRoleStatements section refers to Identity and Access Management which is used to set up Lambda permissions. Here we give the Lambda write-access to our S3 bucket.

To save objects we need permission to execute the s3:PutObject action. Other permissions can be added here if they are required by your project. Refer to the S3 docs for other available actions.

Resource is set to the S3 bucket’s Amazon Resource Name, which is used to uniquely identify a particular AWS resource. The format of this identifier depends upon the AWS service that is being referred to, but generally they take the following format:

arn:partition:service:region:account-id:resource

Underneath provider append our functions configuration.

functions:
  store:
    handler: api.handle
    events:
      - http:
          path: /
          method: post
          cors: true

Here we define the API configuration and map our Lambda to an HTTP POST event at the API’s base URL. A handler with the value api.handle refers to a function named handle that is exported from api.js (we don’t need the js file extension because earlier in serverless.yml we set the runtime to nodejs6.10).

Lambda is event based and so functions only get executed based on predefined triggers. Here we’ve defined a HTTP event but this could have also have been an event trigged by a DynamoDB table or an SQS queue.

Next in serverless.yml we define the AWS resources to be instantiated for us on deployment using CloudFormation. It’s worth mentioning that you don’t necessarily have to setup resources this way, you could also create them using the AWS Management Console. Providing the correct access permissions are in place it doesn’t matter how the resources are created. But in defining the required services in serverless.yml you’re defining your infrastructure as code and obtain a number of benefits in doing so.

Go ahead and add the resources configuration:

resources:
  Resources:
    ServerlessRedirectS3Bucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${file(config.json):BUCKET}
        AccessControl: PublicRead
        WebsiteConfiguration:
          IndexDocument: index.html
    ServerlessRedirectS3BucketPolicy:
      Type: AWS::S3::BucketPolicy
      Properties:
        Bucket: ${file(config.json):BUCKET}
        PolicyDocument:
          Statement:
          - Action:
            - s3:GetObject
            Effect: Allow
            Resource:
            - arn:aws:s3:::${file(config.json):BUCKET}/*
            Principal: "*"

We ask for an S3 bucket resource configured to use static site hosting with index.html as the root document. S3 buckets for good reason are private by default and so we need to create an S3 bucket policy which allows public access to it. Without this policy website visitors would instead by shown an unauthenticated error message.

Building the API

Our Lambda function is responsible for four tasks:

  1. Grabbing the URL to shorten from the user’s form submission.
  2. Generating a unique shortcode for the URL.
  3. Saving the appropriate redirect object to S3.
  4. Returning the object’s path to the client.

Create the handler

Create a new file called api.js and export an arrow function named handle which takes three arguments: event, context and callback. These will be provided by AWS when the handler is invoked. This file is a Node.js script and in order to export the arrow function you need to append it to module.exports.

module.exports.handle = (event, context, callback) => {

}

This handler will get invoked when a HTTP POST request is made to our endpoint. To return an API response we need to use the supplied callback function provided as the third arrow function argument. It’s an error-first callback which takes two arguments. If the request completed successfully, null should be passed in as the first argument. The response object passed in as the second argument determines the type of response to be returned to the user. Generating a response is as simple as providing statusCode and body as is shown in the example below.

const response = {
  statusCode: 201,
  body: JSON.stringify({ "shortUrl": "http://example.com" })
}

callback(null, response)

The context object passed in as the second argument to the handler contains run-time information which for this tutorial we don’t need access to. We do however need to make use of the event passed in as the first argument as this contains the form submission with the URL to shorten.

Parse the request

Below is an example of an API Gateway event that will be passed to our handler when a user makes a form submission. As we’re building our URL shortener as a single page application we’ll be submitting the form using JavaScript and hence the content type will be application/json rather than application/x-www-form-urlencoded.

{
   resource:'/',
   path:'/',
   httpMethod:'POST',
   headers: {
      Accept:'*/*',
      'Accept-Encoding':'gzip, deflate',
      'cache-control':'no-cache',
      'CloudFront-Forwarded-Proto':'https',
      'CloudFront-Is-Desktop-Viewer':'true',
      'CloudFront-Is-Mobile-Viewer':'false',
      'CloudFront-Is-SmartTV-Viewer':'false',
      'CloudFront-Is-Tablet-Viewer':'false',
      'CloudFront-Viewer-Country':'GB',
      'content-type':'application/json',
      Host:'',
      'User-Agent':'',
      'X-Amz-Cf-Id':'',
      'X-Amzn-Trace-Id':'',
      'X-Forwarded-For':'',
      'X-Forwarded-Port':'443',
      'X-Forwarded-Proto':'https'
   },
   queryStringParameters:null,
   pathParameters:{},
   stageVariables:null,
   requestContext: {
      path:'/dev',
      accountId:'',
      resourceId:'',
      stage:'dev',
      requestId:'',
      identity:{
         cognitoIdentityPoolId:null,
         accountId:null,
         cognitoIdentityId:null,
         caller:null,
         apiKey:'',
         sourceIp:'',
         accessKey:null,
         cognitoAuthenticationType:null,
         cognitoAuthenticationProvider:null,
         userArn:null,
         userAgent:'',
         user:null
      },
      resourcePath:'/',
      httpMethod:'POST',
      apiId:''
   },
   body:'{"url":"http://example.com"}',
   isBase64Encoded:false
}

We only need the form submission from the event, which we can get by looking at the request body. The request body is stored as a stringified JavaScript object which we can grab inside of our handler using JSON.parse(). Taking advantage of JavaScript short-circuit evaluation we can set a default value of an empty string for cases where a URL hasn’t been sent as part of the form submission. This allows us to treat instances where the URL is missing and where the URL is an empty string equally.

module.exports.handle = (event, context, callback) => {
  let longUrl = JSON.parse(event.body).url || ''
}

Validate the URL

Let’s add some basic validation to check that the provided URL looks legitimate. There are multiple approaches that could be taken to achieve this. But for the purpose of this tutorial we’ll keep it simple and use the built-in Node.js URL module. We’ll build our validation to return a resolved promise on a valid URL and return a rejected promise on an invalid URL. Promises in JavaScript can be sequentially chained so that the resolution of one promise gets passed to the success handler of the next. We’ll be using this attribute of promises to structure our handler. Let’s write the validate function using promises.

const url = require('url')

function validate (longUrl) {
  if (longUrl === '') {
    return Promise.reject({
      statusCode: 400,
      message: 'URL is required'
    })
  }

  let parsedUrl = url.parse(longUrl)
  if (parsedUrl.protocol === null || parsedUrl.host === null) {
    return Promise.reject({
      statusCode: 400,
      message: 'URL is invalid'
    })
  }

  return Promise.resolve(longUrl)
}

In our validate function we first check that the URL isn’t set to an empty string. If it is we return a rejected promise. Notice how the rejected value is an object containing a status code and message. We’ll use this later to build an appropriate API response. Calling parse on the Node.js URL module returns a URL object with information that could be extracted from the URL that was passed in as a string argument. As part of our basic URL validation we simply check to see whether a protocol (http for example) and a host (like example.com) could be extracted. If either of these values is null on the returned URL object, we assume that the URL is invalid. If the URL is valid we return it as part of a resolved promise.

Returning a response

After grabbing the URL from the request we call validate and for each additional handler step that’s required we’ll return a new promise in the previous promise’s success handler. The final success handler is responsible for returning an API response through the handle’s callback argument. It will be invoked for both error API responses generated from rejected promises as well as successful API responses.

module.exports.handle = (event, context, callback) => {
  let longUrl = JSON.parse(event.body).url || ''
  validate(longUrl)
    .then(function(path) {
      let response = buildResponse(200, 'success', path)
      return Promise.resolve(response)
    })
    .catch(function(err) {
      let response = buildResponse(err.statusCode, err.message)
      return Promise.resolve(response)
    })
    .then(function(response) {
      callback(null, response)
    })
}

function buildResponse (statusCode, message, path = false) {
  let body = { message }
  if (path) body['path'] = path

  return {
    headers: {
      'Access-Control-Allow-Origin': '*'
    },
    statusCode: statusCode,
    body: JSON.stringify(body)
  }
}

Generate a URL shortcode

The API needs to be able to generate unique URL shortcodes, which are represented as filenames in the S3 bucket. As a shortcode is just a filename there’s a great degree of flexibility in how it’s composed. For our shortcode we’ll use a 7 digit alphanumeric string consisting of both uppercase and lowercase characters, this translates to 62 possible combinations for each character. We’ll use recursion to build up the shortcode by selecting one character at a time until seven have been selected.

function generatePath (path = '') {
  let characters = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'
  let position = Math.floor(Math.random() * characters.length)
  let character = characters.charAt(position)

  if (path.length === 7) {
    return path
  }

  return generatePath(path + character)
}

Whilst the likelihood of randomly generating the same shortcode is slim (there’s actually a 0.0000000000000000000000008063365516 chance two are the same), we need to check whether the generated shortcode is already in use, which we can do using the AWS SDK using the a headObject method on the S3 service which loads an object’s metadata. We can use this to test whether an object with the same name already exists as when an object isn’t found a promise with the code NotFound is rejected. This rejected promise indicates that the shortcode is free and can be used. Calling headObject is more performant than testing whether the object exists through getObject, which loads the entire object.

const AWS = require('aws-sdk')
const S3 = new AWS.S3()

function isPathFree (path) {
  return S3.headObject(buildRedirect(path)).promise()
    .then(() => Promise.resolve(false))
    .catch(function (err) {
      if (err.code == 'NotFound') {
        return Promise.resolve(true)
      } else {
        return Promise.reject(err)
      }
    })
}

function buildRedirect (path, longUrl = false) {
  let redirect = {
    'Bucket': config.BUCKET,
    'Key': path
  }

  if (longUrl) {
    redirect['WebsiteRedirectLocation'] = longUrl
  }

  return redirect
}

We can use isPathFree to recursively find a unique object path:

function getPath () {
  return new Promise(function (resolve, reject) {
    let path = generatePath()
    isPathFree(path)
      .then(function (isFree) {
        return isFree ? resolve(path) : resolve(getPath())
      })
  })
}

Taking advantage of the ability to chain promises we return a new invocation of getPath if isPathFree returns false.

To save an object after a unique shortcode has been found, we just need to call the putObject method on the S3 service. Let’s wrap this up in a function that resolves the shortcode if the putObject method call was successful and returns an error object to build an API response if it didn’t.

function saveRedirect (redirect) {
  return S3.putObject(redirect).promise()
    .then(() => Promise.resolve(redirect['Key']))
    .catch(() => Promise.reject({
      statusCode: 500,
      message: 'Error saving redirect'
  })
}

Utilizing the above functions we can add two new promise success handlers to finalise our API endpoint. We need to return getPath from the first promise success handler which will resolve a unique URL shortcode. Returning saveRedirect with a redirect object built using this unique shortcode in the second success handler will save the object to the S3 bucket. This object’s path can then be returned to the client as part of an API response.

module.exports.handle = (event, context, callback) => {
  let longUrl = JSON.parse(event.body).url || ''
  validate(longUrl)
    .then(function () {
      return getPath()
    })
    .then(function (path) {
      let redirect = buildRedirect(path, longUrl)
      return saveRedirect(redirect)
    })
    .then(function (path) {
      let response = buildResponse(200, 'success', path)
      return Promise.resolve(response)
    })
    .catch(function (err) {
      let response = buildResponse(err.statusCode, err.message)
      return Promise.resolve(response)
    })
    .then(function (response) {
      callback(null, response)
    })
}

Deploy the API

Run serverless deploy in your terminal to deploy the API. This will setup our S3 bucket and return the URL of the endpoint. Keep the URL of the endpoint handy as we’ll need it later on.

Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (5.44 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..............
Serverless: Stack update finished...
Service Information
service: serverless-url-shortener
stage: dev
region: eu-west-1
stack: serverless-url-shortener-dev
api keys:
  None
endpoints:
  POST - <a href="https://t2fgbcl26h.execute-api.eu-west-1.amazonaws.com/dev/">https://t2fgbcl26h.execute-api.eu-west-1.amazonaws.com/dev/</a>
functions:
  store: serverless-url-shortener-dev-store
Serverless: Removing old service versions...

Creating the frontend

To help with design we’ll be utilizing the PaperCSS framework. We’ll also be pulling in jQuery to simplify working with the DOM and making AJAX queries. It’s worth noting that for a production environment you’d probably want to pull in two lighter dependencies, but as this is just a tutorial I feel it’s acceptable.

Create a static folder so we have somewhere to store our frontend code and save a copy of paper.min.css and jquery-3.2.1.min.js to this folder, these are minified versions of the PaperCSS framework and jQuery library respectively.

Add the HTML

Create a new file called index.html inside the static folder and add the required HTML. We need a form for the URL input and a button to submit the form. We also need somewhere to put the result of any API calls, which for a successful API call would be the shortened URL and for an unsuccessful API call would be the error message.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name=viewport content="width=device-width,initial-scale=1">
  <title>Serverless url shortener</title>
  <link href="paper.min.css" rel="stylesheet">
</head>
<style>
  * {
    text-align: center;
  }

  #message {
    display: none;
  }
</style>
<body>
  <div class="row flex-center">
    <div class="col-8 col">
      <h2>Serverless url shortener</h2>
      <form action="">
        <div class="form-group">
          <label for="url">Enter URL to shorten</label>
          <input
            class="input-block"
            name="url"
            type="url"
            id="url"
            autocomplete="off"
            required>
        </div>
        <div id="message" class="alert alert-primary"></div>
        <input
          class="paper-btn"
          type="submit"
          value="Shorten link">
      </form>
      <p class="padding-top">
        <a href="https://git.io/vbS8I">
          View this project on Github
        </a>
      </p>
    </div>
  </div>
</body>
</html>

Although not shown in the code block above for brevity, be sure you set the form action to the API endpoint that was displayed when you ran serverless deploy. If you’ve no longer got access to your terminal output from that deployment, you can find out the endpoint URL through the serverless info command.

Make API requests

Before writing the JavaScript to make requests to our API, let’s first load jQuery by appending a script tag just before </body> and referencing the minified file we downloaded previously.

<script src="jquery-3.2.1.min.js"></script>

Now add another pair of script tags underneath and inside let’s create a function that can be used to display a message to the user using the message div in our template that is set to display:none by default. To show a message we can set the text inside of this div using text() and toggle the display using show().

<script>
  function addMessage (text) {
    $('#message').text(text).show()
  }
</script>

Let’s write another function to go inside the same set of script tags that will use jQuery to make HTTP requests to our API.

function shortenLink (apiUrl, longUrl) {
  $.ajax(apiUrl, {
    type : 'POST',
    data: JSON.stringify({url: longUrl})})
    .done(function (responseJSON) {
      var protocol = window.location.protocol + '//'
      var host = window.location.host + '/'
      var shortUrl = protocol + host + responseJSON.path
      addMessage(shortUrl)
    })
    .fail(function (data) {
      if (data.status === 400) {
        addMessage(data.responseJSON.message)
      } else {
        addMessage('an unexpected error occurred')
      }
    })
}

This function creates a POST request and sets the request body to a JSON object containing the URL to shorten. If the request completed successfully and a HTTP 2XX status code was returned, it grabs the shortcode from the path key on the response and builds up a fully qualified short URL to present to the user using the addMessage function created previously. If the request was unsuccessful then an error message is displayed.

Finally we can hook this up to our form by adding an on submit handler. We get the API endpoint URL from the form action attribute and get the URL to shorten from the URL form input.

$('form').submit(function (event) {
  event.preventDefault()
  addMessage('...')
  shortenLink(event.target.action, event.target.url.value)
})

Deploy the website

For website deployment we’ll use the AWS CLI sync command to upload the contents of the static folder to our S3 bucket. Run the following command in your terminal, replacing {{BUCKET}} with the bucket name chosen in config.json:

aws s3 sync static s3://{{BUCKET}}

After this completes you should be able to head to your S3 bucket address in a browser to see the URL shortener in action. URLs for S3 buckets take the following form:

http://{{BUCKET}}.s3-website-REGION.amazonaws.com

To add a custom domain to your bucket you should follow one of the instructions in this AWS support article. For the easiest option you should set the bucket name to your domain’s www subdomain. If you then add a CNAME record in your DNS configuration for the www subdomain and set it to your S3 bucket address, the website should be accessible via your domain. Be sure to also remove any existing A records and bear in mind that this won’t set up a redirection from your root domain to the www subomain. There’s a couple of ways this could be solved which are described in the linked AWS article.

Wrap up

I hope you found this tutorial useful. The reality is that AWS is incredibly flexible and in this article we went through just one way you can create a URL shortener using Lambda and S3. But there are a variety of other ways the same process could also have been accomplished.

The cover image for this post uses graphics from SAP Scenes.

Introducing Formplug v1, a form forwarding service for AWS Lambda

Introducing Formplug v1, a form forwarding service for AWS Lambda

Using Python and Selenium to automate Facebook group invites

Using Python and Selenium to automate Facebook group invites