Solution<\/h5>\n
A centralized location to collect, select, filter, edit articles and generate newsletters. Also, help in tracking past newsletter articles. We decided to have a flask app that stores and shows all these articles details. App also stores the details of the articles that are part of the newsletter in the past so that we do not send duplicates. 1. We used AWS SNS- SQS to pull all these articles from the skype channel We have not yet built the intelligence to fill in the details about the articles so we provided a frontend for users to fill details and categorize the articles. <\/p>\n Mailchimp has many APIs, for our use-case we chose the post-campaign. We made a base HTML template corresponding to our Mailchimp template. And called Mailchimp API to create a new campaign and upload the data. <\/p>\n Now the newsletter team has to log in to Mailchimp to validate the information and click schedule. (Scheduling will eventually also be automated once this system is strong)<\/p>\n There is AWS Lambda written to automate the above-mentioned process. Lambda listens to SQS which is configured to poll SNS where Skype messages are present. Once a message is added to SQS, Lambda is triggered where the logic filters out if the passed message is relevant for the newsletter or not.<\/p>\n Lambda handler calls a method to get a relevant message from SQS and clean it.<\/p>\n From the cleaned message, URL details are extracted. Also we have included environment variables to exclude some URLs e.g google-meet etc<\/p>\n Once the correct URL is obtained, it is posted to the newsletter using API.<\/p>\n On a successful run of the above cycle, articles are updated to the newsletter app. Why this post? At Qxf2, we use Mailchimp to send out our weekly newsletter. We have been following pairing activity every week to fill up our newsletter template in Mailchimp and schedule the newsletter. To fill the template, the paired-up team had to look at various resources to come up with articles for various sections, which is tedious so we […]<\/p>\n","protected":false},"author":26,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[177,282,172],"tags":[],"class_list":["post-15547","post","type-post","status-publish","format-standard","hentry","category-aws","category-aws-lambda","category-flask"],"_links":{"self":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/15547","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/users\/26"}],"replies":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/comments?post=15547"}],"version-history":[{"count":22,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/15547\/revisions"}],"predecessor-version":[{"id":16984,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/15547\/revisions\/16984"}],"wp:attachment":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/media?parent=15547"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/categories?post=15547"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/tags?post=15547"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nWe had a very good resource pool of articles – our skype channel which is filled with articles(we encourage a lot of reading and sharing<\/em>). We used data from this channel to be sent to the the flask app and use this data to generate the Mailchimp template.<\/p>\nSimple workflow:<\/h5>\n
\n 2. Send them for processing in AWS Lambda
\n 3. Lambda then picks the URL of these articles
\n 4. The Lambda calls the endpoint of our flask app to insert data into the MySQL DB.
\n 5. Once the data is present here, we edit the data and send it to MailChimp.<\/p>\n![\"\"](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2021\/09\/highlevelflow.png\")
\n<\/br><\/p>\nDetails of Lambda function<\/h4>\n
\r\ndef get_message_contents(event):\r\n \"Retrieve the message contents from the SQS event\"\r\n record = event.get('Records')[0]\r\n message = record.get('body')\r\n message = json.loads(message)['Message']\r\n message = json.loads(message)\r\n\r\n return message\r\n<\/pre>\n\r\ndef get_url(message):\r\n \"Get the URL from the message\"\r\n regex = r\"(?i)\\b((?:https?:\/\/|www\\d{0,3}[.]|[a-z0-9.\\-]+[.][a-z]{2,4}\/)(?:[^\\s()<>]+|\\(([^\\s()<>]+|(\\([^\\s()<>]+\\)))*\\))+(?:\\(([^\\s()<>]+|(\\([^\\s()<>]+\\)))*\\)|[^\\s`!()\\[\\]{};:'\\\".,<>?\u00ab\u00bb\u201c\u201d\u2018\u2019]))\"\r\n url_patterns = re.findall(regex,message)\r\n urls = []\r\n for url in url_patterns:\r\n if url[0][-1] != '-':\r\n present_flag = False\r\n for exclude_url in EXCLUDE_URL_STRINGS:\r\n if exclude_url in url[0]:\r\n present_flag = True\r\n break\r\n if not present_flag:\r\n urls.append(url[0])\r\n\r\n return urls\r\n\r\n<\/pre>\n\r\n\r\ndef post_to_newsletter(final_url, category_id = '2'):\r\n \"Method to call the newsletter API and post the url\"\r\n\r\n url = os.environ.get('URL', '')\r\n headers = {'x-api-key' : os.environ.get('API_KEY_VALUE','')}\r\n for article_url in final_url:\r\n data = {'url': article_url, 'category_id': category_id}\r\n response = requests.post(url, data = data, headers = headers)\r\n print(response.status_code)\r\n return response.status_code\r\n\r\n<\/pre>\nHere’s Complete Lambda function.<\/h5>\n
\r\n\"\"\"\r\nLambda to to pull URL from Skypechannel messages\r\n\"\"\"\r\nimport json\r\nimport os\r\nimport boto3\r\nimport requests\r\nimport re\r\n\r\nEXCLUDE_URL_STRINGS = ['skype.com', 'meet.google.com']\r\n\r\ndef clean_message(message):\r\n \"Clean up the message received\"\r\n message = message.replace(\"'\", '-')\r\n message = message.replace('\"', '-')\r\n\r\n return message\r\n\r\ndef get_message_contents(event):\r\n \"Retrieve the message contents from the SQS event\"\r\n record = event.get('Records')[0]\r\n message = record.get('body')\r\n message = json.loads(message)['Message']\r\n message = json.loads(message)\r\n\r\n return message\r\n\r\ndef get_url(message):\r\n \"Get the URL from the message\"\r\n regex = r\"(?i)\\b((?:https?:\/\/|www\\d{0,3}[.]|[a-z0-9.\\-]+[.][a-z]{2,4}\/)(?:[^\\s()<>]+|\\(([^\\s()<>]+|(\\([^\\s()<>]+\\)))*\\))+(?:\\(([^\\s()<>]+|(\\([^\\s()<>]+\\)))*\\)|[^\\s`!()\\[\\]{};:'\\\".,<>?\u00ab\u00bb\u201c\u201d\u2018\u2019]))\"\r\n url_patterns = re.findall(regex,message)\r\n urls = []\r\n for url in url_patterns:\r\n if url[0][-1] != '-':\r\n present_flag = False\r\n for exclude_url in EXCLUDE_URL_STRINGS:\r\n if exclude_url in url[0]:\r\n present_flag = True\r\n break\r\n if not present_flag:\r\n urls.append(url[0])\r\n\r\n return urls\r\n\r\ndef post_to_newsletter(final_url, category_id = '2'):\r\n \"Method to call the newsletter API and post the url\"\r\n\r\n url = os.environ.get('URL', '')\r\n headers = {'x-api-key' : os.environ.get('API_KEY_VALUE','')}\r\n for article_url in final_url:\r\n data = {'url': article_url, 'category_id': category_id}\r\n response = requests.post(url, data = data, headers = headers)\r\n print(response.status_code)\r\n return response.status_code\r\n\r\ndef lambda_handler(event, context):\r\n \"\"\"\r\n Method run when Lambda is triggered\r\n calls the filtering logic\r\n calls the logic to post to endpoint\r\n \"\"\"\r\n content = get_message_contents(event)\r\n message = content['msg']\r\n channel = content['chat_id']\r\n user = content['user_id']\r\n print(f'{message}, {user}, {channel}')\r\n\r\n response=\"\"\r\n final_url=[]\r\n if channel == os.environ.get('Skypechannel') and user != os.environ.get('username'):\r\n print(\"Getting message posted on ETC \")\r\n cleaned_message = clean_message(message)\r\n final_url=get_url(cleaned_message)\r\n #Filtered URL is printed by lambda\r\n print(\"Final url is :\",final_url)\r\n if final_url:\r\n response = post_to_newsletter(final_url)\r\n else:\r\n print(\"message does not contain any url\")\r\n else:\r\n print(\"Message not from ETC channel\")\r\n\r\n return {\r\n 'statusCode': response,\r\n 'body': json.dumps(final_url)\r\n }\r\n<\/pre>\n
\nI hope this article helps you understand a use-case to use AWS Lambda, also hope this helps you set up Lambda quickly.<\/p>\n","protected":false},"excerpt":{"rendered":"