Why monitor Athena?

Overview

0. Why monitor Athena?

1. Out-of-Scope

2. Amazon CloudWatch

3. AWS CloudTrail

4. Amazon Managed Grafana

5. (Proposed) Solution Architecture

6. Next Steps

Out of Scope (for now)

• Incident response Application Monitoring

• Data Quality Metrics Data Catalogue

• create-a-derived-table model metrics

• create-a-pipeline non-athena metrics

• airflow non-athena metrics Analytical Platform

• NLP of Athena queries

Amazon Managed Grafana

• Grafana is an open-source analytics platform that helps you query, visualize, alert on, and understand your metrics wherever they are stored.

• Amazon Managed Grafana is a fully managed service for Grafana.

• You can use it with CloudWatch by adding it as a data source.

• This table compares Amazon Managed Grafana vs CloudWatch for dashboarding:

  • Amazon Managed Grafana integrates with AWS Organizations to enable you to read data from AWS sources such as CloudWatch across all your accounts.
  • Grafana provides a larger collection of visualizations.

Proposed Athena and Data Monitoring Architecture

The architecture is split into three parts:

1. Monitor data access using AWS Glue APIs, and aggregate/query using Athena.
   This approach still needs to be evaluated because it relies on the undocumented BatchGetTable API.

2. Monitor Athena usage at the workgroup level using CloudWatch metrics.

3. Monitor Athena usage at the user level using various Athena APIs, and aggregate using CloudWatch Log.
   This approach still needs to be evaluated because of the cost incurred triggering an lambda function every time an Athena query is called.

Using Athena workgroups in Airflow

1. Create airflow-{folder} workgroup using athena_workgroups.tf

2. Add mojap-athena-query-dump/{folder} to S3 read_write permissions on Airflow role

3. Add "WR_WORKGROUP": "airflow-{folder}" to the env_vars dictionary that is passed to the Airflow task

4. That's it!

Custom CloudWatch Logs

watchtower is a lightweight adapter between Python logging and CloudWatch Logs

import watchtower, logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logger.addHandler(watchtower.CloudWatchLogHandler(log_group_name="soumaya_test"))
logger.info(dict(table="apple", details={}))
logger.info(dict(table="banana", details={}))
logger.info(dict(table="orange", details={}))
logger.info(dict(table="apple", details={}))

CloudWatch Log Insights query stats count(@timestamp) by table returns:

orange 1
apple 2
banana 1