Annotations And Alerts

  1. Annotations And Alerts Email
  2. Annotations And Alerts Free
  3. Annotations And Alerts App
  4. Annotations & Alerts

This page contains complete text of legal opinion letters, which are summarized as 'annotations' in Volume 2 of the Business Taxes Law Guide. In any instance where there is an inconsistency between the statute and an annotation, statutory law is controlling. A complete list of annotated opinions is available by subject area below. Trigger the alerts /notifications. In addition, the job compares system-date with the alerts that are due, if it is due it sends out the alerts in the To-Do /Take Action tile and the email notifications to the appropriate users as defined in the workflow. Update the Alert objects for which the notifications have been sent.

Hosted Graphite now supports Annotations and Graphite Events.The Annotations feature is designed as a simpler interface to create events withtags.


  • Annotations and Events
    • What is an annotation?
    • More details about the Annotations API
    • Graphite Composer

An annotation relates your metrics to intermittent events for example when you deploy code, run some tests (hopefully successfully), or a long-running job completes.

This snippet of code will create an annotation with the tags “deployment” and “feature”, titled “New Super-duper Feature”. It’ll be timestamped at the time the event is received.

Annotations have the following properties:

  • title - (required) A short summary of the annotation.
  • description - A more in-depth description of the annotation.
  • tags - A single word or multiple short strings (all case-insensitive) to organise different types of events. e.g. ‘deployments’, ‘feature’, ‘test’ etc.
  • start_time - Unix timestamp describing when the event occurred. If this is not specified, the time of event creation is used.
  • end_time - Optional timestamp describing when the event completed. Note: We currently don’t support Grafana’s region events. As a result, end_time has no actual use in annotations.

There’s no restriction on having annotations with the same time stamps, but the combination of start time and title must be unique.

The API uses basic HTTP authorization. An example using curl:


Deleting an annotation via the API is as simple as this:

Events as per the Graphite project are fully supported. Insertion and rendering of Graphite Events works the same as the annotations endpoint, though the Graphite events API takes a JSON formatted data structure.

To support the display of Annotations, there is a function annotations() available in graphite to query and display the events.There are two options:

This first option with a wildcard queries all available annotations for the time period selected

This version queries all annotations with the tags “restart” and “deployment” for the time period selected

The graphite events function events() is also available:

Grafana’s annotations feature works exactly as specified in the grafana documentation. It can be used to query by tags (not graphite targets).

  • To open the annotations panel, click the settings icon in the top bar and select Annotations.
  • Set the datasource to ‘hostedgraphite’ and use the ‘Graphite event tags’ input box to filter by tags.
  • Individual wildcards also work to display all events.
AnnotationsAnnotations And Alerts

Alerting rules allow you to define alert conditions based on Prometheusexpression language expressions and to send notifications about firing alertsto an external service. Whenever the alert expression results in one or morevector elements at a given point in time, the alert counts as active for theseelements' label sets.

Defining alerting rules

Alerting rules are configured in Prometheus in the same way as recordingrules.

Annotations And Alerts Email

An example rules file with an alert would be:

Annotations And Alerts Free

The optional for clause causes Prometheus to wait for a certain durationbetween first encountering a new expression output vector element and counting an alert as firing for this element. In this case, Prometheus will check that the alert continues to be active during each evaluation for 10 minutes before firing the alert. Elements that are active, but not firing yet, are in the pending state.

The labels clause allows specifying a set of additional labels to be attachedto the alert. Any existing conflicting labels will be overwritten. The labelvalues can be templated.

The annotations clause specifies a set of informational labels that can be used to store longer additional information such as alert descriptions or runbook links. The annotation values can be templated.


Label and annotation values can be templated using consoletemplates. The $labelsvariable holds the label key/value pairs of an alert instance. The configuredexternal labels can be accessed via the $externalLabels variable. The$value variable holds the evaluated value of an alert instance.

Annotations And Alerts App


Inspecting alerts during runtime

To manually inspect which alerts are active (pending or firing), navigate tothe 'Alerts' tab of your Prometheus instance. This will show you the exactlabel sets for which each defined alert is currently active.

For pending and firing alerts, Prometheus also stores synthetic time series ofthe form ALERTS{alertname='<alert name>', alertstate='<pending or firing>', <additional alert labels>}.The sample value is set to 1 as long as the alert is in the indicated active(pending or firing) state, and the series is marked stale when this is nolonger the case.

Sending alert notifications

Annotations & Alerts

Prometheus's alerting rules are good at figuring what is broken right now, butthey are not a fully-fledged notification solution. Another layer is needed toadd summarization, notification rate limiting, silencing and alert dependencieson top of the simple alert definitions. In Prometheus's ecosystem, theAlertmanager takes on thisrole. Thus, Prometheus may be configured to periodically send information aboutalert states to an Alertmanager instance, which then takes care of dispatchingthe right notifications.
Prometheus can be configured to automatically discover availableAlertmanager instances through its service discovery integrations.