\nUsing this service, the real-time\u00a0continuous\u00a0data can be collected, transformed, and stored in data stores. In further, these stored data can be visualized using visualization tools.
\nExamples of real-time streaming data are Log files generated by an application, Financial stock market data, IoT device data, etc.<\/p>\n
\n
About Amazon Kinesis<\/h3>\n
Amazon Kinesis has 4 different capabilities:<\/p>\n
- \n
- Kinesis Data Streams<\/strong> \u2014 Used to capture, process, and store data streams.<\/li>\n
- Kinesis Data Firehose<\/strong> \u2014 Used to load data streams into AWS data stores.<\/li>\n
- Kinesis Data Analytics<\/strong> \u2014 Used to analyze data streams with SQL or JAVA.<\/li>\n
- Kinesis Video Streams <\/strong> \u2014 Used to capture, process, and store video streams.<\/li>\n<\/ul>\n
In this blog, we will see how to use Amazon Kinesis Data Firehose.
\nThe Amazon Kinesis Data Firehose<\/strong> provides the facility to capture the real-time continuous streaming data, transform the data using lambda function, and store the data in data store. The various data stores are Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and Splunk.
\nFor our example, we will be using Amazon Simple Storage (S3) as the data store.
\nTo follow along, I assume that the reader of the post has an AWS account and is familiar with basic AWS services. Make\u00a0sure the IAM user (preferably) or root user has full permission to access EC2 instance, Kinesis, IAM role, and S3.<\/p>\n
\nImplementation<\/h3>\n
1) Set up the EC2 server<\/p>\n
- \n
- a. Install AWS CLI. Launch the Linux instance and connect<\/a> to the instance.
\nb. Install LAMP Web Server<\/a> on the Linux instance, so that using the apache web server the sample website can be hosted.<\/p>\n$ sudo yum install -y httpd24 php72 mysql57-server php72-mysqlnd<\/pre>\n
Verify if the webserver is installed correctly by using EC2 instance Public IP address with default port-80 on the browser (Public IP<\/span>:80)
![\"\"](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/05\/Check_Webser_page-1024x367.png\")
<\/a>
\nc. Navigate to path: cd \/var\/www\/html
\nd. Download a sample website template. Here I am downloading a zip file from the\u00a0site<\/a> using wget.<\/p>\n$ wget https:\/\/www.free-css.com\/assets\/files\/free-css-templates\/download\/page253\/estateagency.zip<\/pre>\n
e. Check if it is download in the html path using ls command.
\nf. Unzip the downloaded html template<\/p>\n$ unzip estateagency.zip<\/pre>\n
g. Verify if the sample website is hosted correctly by using IP address\/html template name on the browser (Public IP<\/span>\/EstateAgency).
\nh. The website logs will be in the path “\/var\/log\/httpd\/access_log”. For each click and use of the website, the related logs will be collected and stored here. Now we will see how to store these continuous logs. Before proceeding, change the permission of the file, so that the file will be in\u00a0readable, writable, and executable mode by any users.\u00a0<\/span><\/p>\n$ chmod 777 \/var\/log\/httpd\/access_log<\/pre>\n<\/li>\n<\/ul>\n
\n2) Setting up Amazon Kinesis Data Firehose delivery stream<\/p>\n
- \n
- a. We will create Kinesis Firehose delivery streams via the console. We can update and modify the delivery stream at any time after it has been created.
\nb. Search for Kinesis in the console and click on “Create delivery stream”
\nc. Provide a name for the delivery stream and choose “Direct PUT”![\"\"](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/05\/DeliveryStream_Page_1.png\")
<\/a>d. Navigate to the next page. Keep AWS Lambda function in the disabled state. We can use the AWS Lambda function to transform records. In this example we are not doing it, we are directly collecting the logs.
\ne. On the next page keep S3 as the destination, create a new bucket, or choose an existing bucket.
\nf. On the next page, keep the Buffer interval to the minimum (60 seconds) and leave the rest things as default.
\ng. In the IAM role, click on Create new and give allow. New IAM role with required permission would create and will be assigned to this Kinesis delivery stream. Click Allow and it will return back to Kinesis screen.![\"\"](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/05\/IAM_role.png\")
<\/a>h. On the next page, review the details provided and create the delivery stream. Yes, you have successfully created the Kinesis Data Firehose delivery stream.![\"\"](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/05\/Created_Kinesis_data_firehose.png\")
<\/a>i. Test the Kinesis by sending a demo data and verify if the data is sent to S3 bucket.![\"\"](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/05\/Test_Kineisis.png\")
<\/a>![\"\"](\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2020\/05\/S3_Demo_data.png\")
<\/a><\/li>\n<\/ul>\n
\n3) Configure the Kinesis agent<\/p>\n
- a. We will create Kinesis Firehose delivery streams via the console. We can update and modify the delivery stream at any time after it has been created.
- Kinesis Data Firehose<\/strong> \u2014 Used to load data streams into AWS data stores.<\/li>\n