![]() Give permission for the port for NFS to the Prometheus-sg. In the efs-security group do the following for in-bound rules. Set up security groups for EC2, EFS, Prometheus and Grafana Prometheus & Grafana setup for MSK using ECS. In the next section we will deploy the same in Production ready state. This is not a production ready setup as the EC2 instance can go down any minute and we need to keep monitoring which adds to overhead. We can do this for all the metrics that are provided by default and created dashboards. ![]() Now let us go to graphs and start investigating specific metrics. Let us get the public IP of the EC2 instance and start running Prometheus from the browser. Now we will create “Prometheus.yml” file. Now we will configure the “targets.json” file. Next, we will start installing and configuring Prometheus.īackup the existing “Prometheus.yml” file Next Let us configure Prometheus on EC2 instance.įirst, we will open security groups on the MSK Cluster SG for port 1102 from the EC2 instance SG where we configure Prometheus.Īlso, in the EC2 instance security group allow port 9090 from the specified IP from where we will be testing. Press “Save Changes” and wait for MSK broker to be available. We will change the configuration of the existing Cluster, we will disable enhanced monitoring and remove CloudWatch and S3 and enable open monitoring with Prometheus. Prometheus setup for MSK in EC2 instance. Also, the cost of storage in CloudWatch is more than when compared with S3. These logs are for long term storage as all data in CloudWatch has a limited time. We can also perform real-time log analytics and searching for any exceptions or errors and raise alerts based on this. We can extend this by creating custom metrics or using Log insights. The details on action at broker level is recorded into the Log files. There are three log streams one per broker. The Cluster is ready, let us check the CloudWatch log group and S3 bucket. We will continue once the Cluster is ready. Press “Create cluster” and wait for the MSK Cluster to be available. We will not be demonstrating Kinesis as we are not sending this to ELK in this demo. Press “Browse” and select the S3 bucket we created for the same. Set the Monitoring to Enhanced broker-level for this demo.įor Broker log delivery we will enable both CloudWatch & S3.Ĭlick on Browse and select the log group we created. Next, we will configure a MSK cluster and configure the log groups and check the logs that are stored. Next, we will create the log group where we will export the logs in CloudWatch. The bucket where we store the logs are now available. Also, we will be demonstrating a solution where we use CloudWatch which is AWS native serviceĬreate a s3 bucket where we want the MSK logs to be stored. ECS Cluster, ECRL to install Prometheus & Grafana in a ECS Cluster to mimic production setup. The EC2 instance where we will the install the Prometheus to generate metrics. Open-Source Monitoring using Prometheus & Grafana Prerequisites The Architecture diagram for AWS MSK MonitoringĪWS Native monitoring using AWS CloudWatch Configuring CloudWatch logs and pushing logs to S3 bucket.We will be covering the solution for same using both CloudWatch which is AWS Native service and Prometheus with Grafana which is an Open-source solution. Also, we need to provide a Dashboard to the user to monitor various metrics for MSK at Broker/Topic level to make necessary adjustments and correction to the service. We also need to provide a solution to monitor the service for tracking availability/issues and raise necessary alerts. In the previous part we showed how to create a MSK cluster, publish and consume data from MSK using Kafka client in an EC2 instance and deploy AKHQ to administer Kafka. ![]() Getting started with AWS MSK Part 2 (Monitoring).
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |