Automation newsletter generation in Mailchimp

Why this post?

At Qxf2, we use Mailchimp to send out our weekly newsletter. We have been following pairing activity every week to fill up our newsletter template in Mailchimp and schedule the newsletter. To fill the template, the paired-up team had to look at various resources to come up with articles for various sections, which is tedious so we thought to automate this process using AWS Lambda. In this post we will give you some details on how we automated the process, but cover the details on the AWS lambda implementation in detail. There may be some more ways to achieve the same.

Solution

A centralized location to collect, select, filter, edit articles and generate newsletters. Also, help in tracking past newsletter articles. We decided to have a flask app that stores and shows all these articles details. App also stores the details of the articles that are part of the newsletter in the past so that we do not send duplicates.
We had a very good resource pool of articles – our skype channel which is filled with articles(we encourage a lot of reading and sharing). We used data from this channel to be sent to the the flask app and use this data to generate the Mailchimp template.

Simple workflow:

1. We used AWS SNS- SQS to pull all these articles from the skype channel
2. Send them for processing in AWS Lambda
3. Lambda then picks the URL of these articles
4. The Lambda calls the endpoint of our flask app to insert data into the MySQL DB.
5. Once the data is present here, we edit the data and send it to MailChimp.

We have not yet built the intelligence to fill in the details about the articles so we provided a frontend for users to fill details and categorize the articles.

Mailchimp has many APIs, for our use-case we chose the post-campaign. We made a base HTML template corresponding to our Mailchimp template. And called Mailchimp API to create a new campaign and upload the data.

Now the newsletter team has to log in to Mailchimp to validate the information and click schedule. (Scheduling will eventually also be automated once this system is strong)



Details of Lambda function

There is AWS Lambda written to automate the above-mentioned process. Lambda listens to SQS which is configured to poll SNS where Skype messages are present. Once a message is added to SQS, Lambda is triggered where the logic filters out if the passed message is relevant for the newsletter or not.

Lambda handler calls a method to get a relevant message from SQS and clean it.

def get_message_contents(event):
    "Retrieve the message contents from the SQS event"
    record = event.get('Records')[0]
    message = record.get('body')
    message = json.loads(message)['Message']
    message = json.loads(message)
 
    return message

From the cleaned message, URL details are extracted. Also we have included environment variables to exclude some URLs e.g google-meet etc

def get_url(message):
    "Get the URL from the message"
    regex = r"(?i)\b((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'\".,<>?«»“”‘’]))"
    url_patterns = re.findall(regex,message)
    urls = []
    for url in url_patterns:
        if url[0][-1] != '-':
            present_flag = False
            for exclude_url in EXCLUDE_URL_STRINGS:
                if exclude_url in url[0]:
                    present_flag = True
                    break
            if not present_flag:
                urls.append(url[0])
 
    return urls

Once the correct URL is obtained, it is posted to the newsletter using API.

 
def post_to_newsletter(final_url, category_id = '2'):
    "Method to call the newsletter API and post the url"
 
    url = os.environ.get('URL', '')
    headers = {'x-api-key' : os.environ.get('API_KEY_VALUE','')}
    for article_url in final_url:
        data = {'url': article_url, 'category_id': category_id}
        response = requests.post(url, data = data, headers = headers)
        print(response.status_code)
    return response.status_code
Here’s Complete Lambda function.
"""
Lambda to to pull URL from Skypechannel messages
"""
import json
import os
import boto3
import requests
import re
 
EXCLUDE_URL_STRINGS = ['skype.com', 'meet.google.com']
 
def clean_message(message):
    "Clean up the message received"
    message = message.replace("'", '-')
    message = message.replace('"', '-')
 
    return message
 
def get_message_contents(event):
    "Retrieve the message contents from the SQS event"
    record = event.get('Records')[0]
    message = record.get('body')
    message = json.loads(message)['Message']
    message = json.loads(message)
 
    return message
 
def get_url(message):
    "Get the URL from the message"
    regex = r"(?i)\b((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'\".,<>?«»“”‘’]))"
    url_patterns = re.findall(regex,message)
    urls = []
    for url in url_patterns:
        if url[0][-1] != '-':
            present_flag = False
            for exclude_url in EXCLUDE_URL_STRINGS:
                if exclude_url in url[0]:
                    present_flag = True
                    break
            if not present_flag:
                urls.append(url[0])
 
    return urls
 
def post_to_newsletter(final_url, category_id = '2'):
    "Method to call the newsletter API and post the url"
 
    url = os.environ.get('URL', '')
    headers = {'x-api-key' : os.environ.get('API_KEY_VALUE','')}
    for article_url in final_url:
        data = {'url': article_url, 'category_id': category_id}
        response = requests.post(url, data = data, headers = headers)
        print(response.status_code)
    return response.status_code
 
def lambda_handler(event, context):
    """
    Method run when Lambda is triggered
    calls the filtering logic
    calls the logic to post to endpoint
    """
    content = get_message_contents(event)
    message = content['msg']
    channel = content['chat_id']
    user = content['user_id']
    print(f'{message}, {user}, {channel}')
 
    response=""
    final_url=[]
    if channel == os.environ.get('Skypechannel') and user != os.environ.get('username'):
        print("Getting message posted on ETC ")
        cleaned_message = clean_message(message)
        final_url=get_url(cleaned_message)
        #Filtered URL is printed by lambda
        print("Final url is :",final_url)
        if final_url:
            response = post_to_newsletter(final_url)
        else:
            print("message does not contain any url")
    else:
        print("Message not from ETC channel")
 
    return {
        'statusCode': response,
        'body': json.dumps(final_url)
    }

On a successful run of the above cycle, articles are updated to the newsletter app.
I hope this article helps you understand a use-case to use AWS Lambda, also hope this helps you set up Lambda quickly.

Leave a Reply

Your email address will not be published. Required fields are marked *