AWS-powered event triggered Automated Email System

Hi readers,

In this blog, we will be building an automated emailing system that uses AWS Lambda and AWS SES (simple email service) to send emails on an Event triggered by inserting documents into Mongo Database. We will be using AWS Event Bridge to add triggers to our Lambda function. First, we will test our project by inserting documents manually into Mongo DB, then we will set up another lambda function that will be triggered by a file upload event on the AWS S3 bucket. This lambda function will read the uploaded Excel file (including email data), and insert those rows into Mongo Db (replacing our manual insertions to Mongo Db).

But let me first provide a brief overview of various services used in this project.

1. AWS SES (Simple Email Service): AWS SES is a cost-effective, flexible, and scalable email service provider that allows developers to send and receive email using their email addresses and domains.
2. AWS Lambda: It is a compute service that runs our code serverless (i.e. without managing any servers or underlying machines). We will be using it to write code that sends emails using AWS SES.
3. AWS Event Bridge: This service works as a bridge between different application components. It is a server-less event bus that listens to data events from your apps, SaaS apps, and AWS services and routes that data to targets.
4. MongoDB (as a partner event source): MongoDB is an open-source NoSQL database management system. It offers an Amazon Event Bridge partner event source that lets you send trigger events to an event bus created through Event Bridge.
5. AWS S3 (storage service): Amazon S3 (Simple Storage Service) is a service offered by AWS that provides object storage through a web service interface. It provides object storage, which is built for storing and recovering any amount of information or data.

We will be building this project in below mentioned steps:

1. Create required resources (cluster, database, collection) in MongoDB
2. Setup Event Trigger of Mongo DB using AWS Event Bridge service
3. Create Lambda Function and set up its dependencies layer
4. Configure AWS SES permissions and identities needed for Lambda
5. Write Lambda code that sends email using data from Mongo DB
6. Create a Rule in Event Bridge that handles trigger and fires Lambda
7. Test the solution by inserting data into the Mongo DB collection
8. Create a function that inserts to Mongo DB, when an Excel file of email data is uploaded to the AWS S3 bucket
9. Create a new S3 bucket and connect to lambda via Event Notifications
10. Configure AWS S3 permission needed for Lambda
11. Test the solution by uploading Excel to AWS S3

So, let's get started!!!

Create required resources (cluster, database, collection) in MongoDB

Sign in or Create a new account on Mongo DB.

Create a new Project → new Cluster and a new collection inside it.

Go to the collections tab, create a new Database, and specify the first collection name like below.

Setup Event Trigger of Mongo DB using AWS Event Bridge service

Go to AWS Event Bridge Console. Click on Partner Event Sources and search for Mongo like below:

Click Setup. Copy the AWS account ID on the next page. This will be used while you create an event bus for the MongoDB event source in the next step.

Go back to the Mongo website. Go to the trigger page from the Left menu and click Get Started as shown below:

Select the database trigger like below and leave all options as default until cluster selection.

Select a new cluster, database, and collection created in the previous steps like below:

Select Event Bridge as the event type and also paste your AWS account ID (copied in the previous step) and region:

Click save and a new trigger is ready now.

Go back to AWS Event Bridge console → Partner Event sources.

You can now see a new partner event source listed in the list. Note that this event source was indirectly created from the Mongo website and not from the AWS console. Looks interesting!

Select the newly integrated Event source and click ‘Associate with event bus’ as shown below:

Click on Associate in the next window. The status of the trigger event source changes from Pending to Active. You can also check the newly created Custom Event Bus in the Event Buses menu.

Create Lambda Function and set up its dependencies layer

Create a new lambda function and add a layer to it for dependencies below mentioned steps.

Run the below commands in Windows Powershell:

mkdir nodejs
cd nodejs
npm i axios
npm i xlsx
npm i aws-sdk
del package-lock.json
cd ..
Compress-Archive -Path nodejs -DestinationPath packages.zip

Configure AWS SES permissions and identities needed for Lambda

We will add a new policy from IAM which will be assigned to a new role which will be assigned to Lambda. So that lambda can have permission to call SES.

Go to IAM > Policies > Create New Policy.

Click on the JSON button and paste below JSON policy in the editor:

{
     "Version": "2012-10-17",
     "Statement": [
         {
         "Effect": "Allow",
         "Action": [
             "ses:SendEmail",
             "ses:SendRawEmail"
            ],
         "Resource": "*"
         }
     ]
}

Click next and create the policy after giving it a name.

Now you can see your policy in the list of policies below:

Select this policy and click Attach from the Actions menu.

Go to your lambda function > Configuration > Permissions and note down the name of the role assigned to your lambda function as shown below:

Search by this role name in the Attach Policy window like below:

Click on Attach Policy.

Now verify that this new SES permission policy is listed in the resource summary dropdown like below:

Now we need to verify our sender Email Address with AWS SES so that it can be used as a source Email Address to send emails to various recipients.

Go to AWS SES > Configurations > Identities > Create Identity:

Select the Email Address option in the next window and paste any of your Email Addresses that you have access to:

Click on Create Identity.

Verify your email address from an email sent to your inbox and validate the status of your Identity as shown below:

You will also need to verify all recipient emails in the same way. Create as many email entities as you want if you want to send bulk emails.

Write Lambda code that sends email using data from Mongo DB

Add the below code to the Lambda function created at the start:

import axios from 'axios';
import * as xlsx from 'xlsx';
import { SESClient, SendEmailCommand } from "@aws-sdk/client-ses";
const ses = new SESClient({ region: "ap-south-1" });

export const handler = async (event) => {
  // TODO implement
  //console.log("event is: ", event.detail.fullDocument);
  
  let newRow = event.detail.fullDocument;
  let resp = await sendEmail(newRow.email , newRow.body);
  const response = {
    statusCode: 200,
    body: resp
  };
  return response;
};

const sendEmail = async (emailTo , body) => {
   const command = new SendEmailCommand({
    Destination: {
      ToAddresses: [emailTo],
    },
    Message: {
      Body: {
        Text: { Data: body },
      },
      Subject: { Data: body },
    },
    Source: "adman1575@gmail.com",
  });
  try {
    let response = await ses.send(command);
    // process data.
    //console.log({response});
    return response;
  }
  catch (error) {
    console.log({error});
    // error handling.
  }
  finally {
    // finally.
  }
}

This lambda is triggered while inserting a new document (containing the email body and email address) into Mongo DB. The document metadata is passed to this lambda as an event attribute. We use this event attribute to send email to the specified email address with the specified email body.

Create a Rule that fires Lambda

We will now start creating a rule that will be triggered by events from that partner event source (ie insert in Mongo DB collection).

Go to the Rules option in the Menu of AWS Event Bridge.

Select the same event bus from the dropdown that was created in the previous step and click on ‘Create Rule’.

Give a name to your rule and click Next.

In the build event pattern step, select ‘AWS events or Event Bridge partner events’ as Event Source and leave all other fields as default except the Event Pattern as shown below:

Click Next and move to the targets step.

Select the lambda function created at the start as a target in this step like below:

Click Next and in the next window review and create.

And now we are all ready for the magic.

Test the solution

Go to Mongo website → Database menu → Collections tab.

Click on Insert Document after selecting the collection you created in the previous steps.

Before inserting a document, make note of any of the recipient email addresses you verified from AWS SES in previous steps. Use this email address in the next step.

Add only two fields in the document, namely email (value as noted in the above step), and body, and click Insert.

Login to this same email account and check the magic!

Woohoooooooooooooo!

Our email application is now officially live. It is ready to listen to any insertion in Mongo database collection, and immediately send email to that recipient using AWS.

Create a function that inserts to Mongo DB, when an Excel file of email data is uploaded to the AWS S3 bucket

Create a new lambda function named saveDataOnExcelUpload and add a package layer to it following the same process used in 3rd step above.

Paste the below code inside lambda:

import AWS from 'aws-sdk';
import * as xlsx from 'xlsx';
import {MongoClient, ServerApiVersion} from 'mongodb';

const uri = process.env.MONGO_DB_CONNECTION_URI;
const s3 = new AWS.S3();

export const handler = async (event, context) => {
    
    try {
        // Get the uploaded file details from the event
        const bucket = event.Records[0].s3.bucket.name;
        const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
        
        // Download the Excel file from S3
        const params = {
            Bucket: bucket,
            Key: key
        };
        const { Body } = await s3.getObject(params).promise();
        
        // Parse the Excel file
        const workbook = xlsx.read(Body, { type: 'buffer' });
        const sheetName = workbook.SheetNames[0];
        const sheet = workbook.Sheets[sheetName];

        // Convert the sheet to JSON
        const jsonData = xlsx.utils.sheet_to_json(sheet);

        const client = new MongoClient(uri,  {
          serverApi: {
            version: ServerApiVersion.v1,
            strict: true,
            deprecationErrors: true,
          }
        });
        await client.connect();
        
        const database = client.db('aws-connection-test-db');
        const collection = database.collection('emails');
        
        for (const item of jsonData){
            try {
            const row = {
                email: item['Recipient'],
                body: item['EmailBody'],
            }
            const result = await collection.insertOne(row);
          } catch(error){
            console.error('error in loop:', error);
          }
        }
        
        return {
            statusCode: 200,
            body: jsonData
        };
    } catch (err) {
        console.error({err});
        return {
            statusCode: 500,
            body: 'An error occurred'
        };
    }
};

Go to Configuration → Env variables and add Mongo Db connection URI as environment variable named MONGO_DB_CONNECTION_URI like below:

Create a new S3 bucket and connect to lambda via Event Notifications

Go to AWS S3, and create a new bucket. Keep the region the same as where your lambda function lives. Give it a name and leave all other options default.

Once the bucket is created, click on it and go to the Properties tab → Event Notifications like below:

Click on Create, give it a name, and check PUT type in Event Types:

In the destination section, select your lambda function that needs to be connected to the S3 bucket and click save:

Configure AWS S3 permission needed for Lambda

Make a note of the role ID of the recently created lambda function by going to the configuration tab of lambda like below:

Go to IAM → Roles. Search and select the above role:

Go to Permission Policies and click on Attach Policies:

Search for ‘S3’, select S3 Full Access, and click add:

Go back to Configuration tab of lambda and you can see the S3 bucket in the resource permissions:

Test the solution by uploading Excel to AWS S3

Prepare an Excel file in below format and upload it to a new S3 bucket:

Note: Make sure that all the emails listed in Excel are verified AWS email identities. For creating verified identities, check step 4 of this same blog above.

Now, as soon as you upload a file to S3, lambda will be invoked on this event, which will insert this data to Mongo DB, which will trigger another lambda function that will send an email against each database entry. Check your email boxes and verify if it's working fine.