Why this post?
At Qxf2, we use Mailchimp to send out our weekly newsletter. We have been following pairing activity every week to fill up our newsletter template in Mailchimp and schedule the newsletter. To fill the template, the paired-up team had to look at various resources to come up with articles for various sections, which is tedious so we thought to automate this process using AWS Lambda. In this post we will give you some details on how we automated the process, but cover the details on the AWS lambda implementation in detail. There may be some more ways to achieve the same.
Solution
A centralized location to collect, select, filter, edit articles and generate newsletters. Also, help in tracking past newsletter articles. We decided to have a flask app that stores and shows all these articles details. App also stores the details of the articles that are part of the newsletter in the past so that we do not send duplicates.
We had a very good resource pool of articles – our skype channel which is filled with articles(we encourage a lot of reading and sharing). We used data from this channel to be sent to the the flask app and use this data to generate the Mailchimp template.
Simple workflow:
1. We used AWS SNS- SQS to pull all these articles from the skype channel
2. Send them for processing in AWS Lambda
3. Lambda then picks the URL of these articles
4. The Lambda calls the endpoint of our flask app to insert data into the MySQL DB.
5. Once the data is present here, we edit the data and send it to MailChimp.
We have not yet built the intelligence to fill in the details about the articles so we provided a frontend for users to fill details and categorize the articles.
Mailchimp has many APIs, for our use-case we chose the post-campaign. We made a base HTML template corresponding to our Mailchimp template. And called Mailchimp API to create a new campaign and upload the data.
Now the newsletter team has to log in to Mailchimp to validate the information and click schedule. (Scheduling will eventually also be automated once this system is strong)
Details of Lambda function
There is AWS Lambda written to automate the above-mentioned process. Lambda listens to SQS which is configured to poll SNS where Skype messages are present. Once a message is added to SQS, Lambda is triggered where the logic filters out if the passed message is relevant for the newsletter or not.
Lambda handler calls a method to get a relevant message from SQS and clean it.
def get_message_contents(event): "Retrieve the message contents from the SQS event" record = event.get('Records')[0] message = record.get('body') message = json.loads(message)['Message'] message = json.loads(message) return message |
From the cleaned message, URL details are extracted. Also we have included environment variables to exclude some URLs e.g google-meet etc
def get_url(message): "Get the URL from the message" regex = r"(?i)\b((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'\".,<>?«»“”‘’]))" url_patterns = re.findall(regex,message) urls = [] for url in url_patterns: if url[0][-1] != '-': present_flag = False for exclude_url in EXCLUDE_URL_STRINGS: if exclude_url in url[0]: present_flag = True break if not present_flag: urls.append(url[0]) return urls |
Once the correct URL is obtained, it is posted to the newsletter using API.
def post_to_newsletter(final_url, category_id = '2'): "Method to call the newsletter API and post the url" url = os.environ.get('URL', '') headers = {'x-api-key' : os.environ.get('API_KEY_VALUE','')} for article_url in final_url: data = {'url': article_url, 'category_id': category_id} response = requests.post(url, data = data, headers = headers) print(response.status_code) return response.status_code |
Here’s Complete Lambda function.
""" Lambda to to pull URL from Skypechannel messages """ import json import os import boto3 import requests import re EXCLUDE_URL_STRINGS = ['skype.com', 'meet.google.com'] def clean_message(message): "Clean up the message received" message = message.replace("'", '-') message = message.replace('"', '-') return message def get_message_contents(event): "Retrieve the message contents from the SQS event" record = event.get('Records')[0] message = record.get('body') message = json.loads(message)['Message'] message = json.loads(message) return message def get_url(message): "Get the URL from the message" regex = r"(?i)\b((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'\".,<>?«»“”‘’]))" url_patterns = re.findall(regex,message) urls = [] for url in url_patterns: if url[0][-1] != '-': present_flag = False for exclude_url in EXCLUDE_URL_STRINGS: if exclude_url in url[0]: present_flag = True break if not present_flag: urls.append(url[0]) return urls def post_to_newsletter(final_url, category_id = '2'): "Method to call the newsletter API and post the url" url = os.environ.get('URL', '') headers = {'x-api-key' : os.environ.get('API_KEY_VALUE','')} for article_url in final_url: data = {'url': article_url, 'category_id': category_id} response = requests.post(url, data = data, headers = headers) print(response.status_code) return response.status_code def lambda_handler(event, context): """ Method run when Lambda is triggered calls the filtering logic calls the logic to post to endpoint """ content = get_message_contents(event) message = content['msg'] channel = content['chat_id'] user = content['user_id'] print(f'{message}, {user}, {channel}') response="" final_url=[] if channel == os.environ.get('Skypechannel') and user != os.environ.get('username'): print("Getting message posted on ETC ") cleaned_message = clean_message(message) final_url=get_url(cleaned_message) #Filtered URL is printed by lambda print("Final url is :",final_url) if final_url: response = post_to_newsletter(final_url) else: print("message does not contain any url") else: print("Message not from ETC channel") return { 'statusCode': response, 'body': json.dumps(final_url) } |
On a successful run of the above cycle, articles are updated to the newsletter app.
I hope this article helps you understand a use-case to use AWS Lambda, also hope this helps you set up Lambda quickly.
I started my career as a Dotnet Developer. I had the opportunity of writing automation scripts for unit testing in Testcomplete using C#. In a couple of years, I was given the opportunity to set up QA function for our in-house products and that’s when I became full-time Tester. I have an overall 12 years of experience. I have experience in Functional – Integration and Regression Testing, Database, Automation testing and exposure to Application Security and API testing. I have excellent experience in test processes, methodologies and delivery for products in Information security and Pharma domain. On the Personal side, I love playing with kids, watch Football, read novels by Indian Authors and cooking.
