I recently discovered that it is not possible to configure alert notification channels in Grafana.
Which is a requirement to have the alerts sended via email (https://grafana.com/docs/administration/provisioning/).
In my opinion, this sould be configurable via helm. If I configure this manually (ICP 3.2.0) in the Grafana User Interface, it won't survive a pod restart, because config files are not even mounted to a persistent volume.
Besides the fact, that a large set of Grafana features is not accessible due to that problem, the current behavior also violates 12 factor app rule #3 (https://12factor.net/config) and results in IBM's grafana distribution not being a cloud-native citizen.
Worst solution, but tolerable:
mount "/etc/grafana/provisioning/" to the persistent volume inside the helm Chart.
Make the configuration accessible as configmap or provide a kubernetes crd to provision notification channels.
Real Scenario + Problem Statement
|Who would benefit from this IDEA?||As a Cloud platform engineer I would like to be able to use Grafana Alerting instead of Prometheus Alerting.|
Why is it useful?
Minimize the hops and use alerting directly in Grafana.
|Market Evidence - # of customers in next 12 months|
|Avg Annual Contract Value Per Customer|
|Secondary Persona (multi-select)|
How should it work?
|Geo Availability for this IDEA|
|Geo - Use for OBDR Reporting only||EU|
|Owning Segment||Hybrid Integration (Hybrid Cloud Unit)|