Creating Alerts in OpenSearch

Steps to Create an Alert

1. Log in to OpenSearch.
2. Click on the hamburger menu (three lines) in the top-left corner.
3. In the sidebar, navigate to OpenSearch Plugins > Alerting.
4. Go to the Monitors tab.
5. Click on Create Monitor.
6. Fill in the required fields as per your requirements.
7. In the Select Data section, choose the desired index (e.g., atwallets-test-kubernetes*).
8. Select the Time field.
9. Configure the Query settings as needed.
10. In the Triggers section, click Add Trigger to create a condition under which the alert should be triggered.
11. Under the trigger, go to Actions and create an action, such as:
    • Sending a message to Slack
    • Sending an email notification
12. Optionally, enable Throttling to prevent excessive notifications (e.g., to avoid spam in Slack).

For more detailed information, refer to the OpenSearch Alerting documentation: https://opensearch.org/docs/latest/observing-your-data/alerting/index/

Additional Information

Monitor Types

OpenSearch provides several types of monitors:

• Per query monitor: Runs a query and generates alert notifications based on the matching criteria. See Per query monitors for information about creating and using this monitor type.
• Per bucket monitor: Runs a query that evaluates trigger criteria based on aggregated values in the dataset. See Per bucket monitors for information about creating and using this monitor type.
• Per cluster metrics: Runs API requests on the cluster to monitor its health. See Per cluster metrics monitors for information about creating and using this monitor type.
• Per document monitor: Runs a query (or multiple queries combined by a tag) that returns individual documents that match the alert notification trigger condition. See Per document monitors for information about creating and using this monitor type.
• composite monitor: Runs multiple monitors in a single workflow and generates a single alert based on multiple trigger conditions. See Composite monitors for information about creating and using this monitor type.

Throttling in OpenSearch

Throttling is used to control the frequency of alerts. If enabled, it ensures that notifications are not sent too frequently, reducing noise and preventing spam (e.g., preventing Slack from receiving alerts every second). Throttling settings allow you to define a cooldown period before the next notification can be sent.

Configuring AND Operator in Visual Editor

By default, when using the Visual Editor, conditions work with the OR operator. To use AND, follow these steps:

1. Set your query filtering configuration as you wish.
2. Save the monitor.
3. Immediately disable it to prevent incorrect execution.
4. Edit the monitor and switch the Monitor defining method from Visual Editor to Extraction Query Editor.
5. Scroll down to the Query section.
6. Replace every occurrence of default_operator with AND.
7. Save the monitor, and it will now use the AND operator instead of OR.

Notification Message Syntax

To use the ctx.results variable in a message, use {{ctx.results.0}} rather than {{ctx.results[0]}}. This difference is due to how Mustache handles bracket notation.

ctx.results.0.hits.hits.length

If you want to iterate through all elements inside the array object, use the following syntax:

{{#ctx.results.0.hits.hits}}
{{{ _source.message }}}
{{/ctx.results.0.hits.hits}}

Best Practices

• The “Time range for the last” setting in the Query section should generally match the “Frequency” setting in the Monitor Details to ensure consistency.

For more detailed information on monitors, refer to the OpenSearch Monitors documentation: https://opensearch.org/docs/latest/observing-your-data/alerting/monitors/

Existing alarms:

• Symbol Cannot Found
• Panic Error
• Fatal Error
• Exchange Rate Limits
• Application Start Panic
• Api Error Monitor