Grafana Labs Discloses Security Incident, No Production Data Compromised

Grafana Labs, a leading provider of open-source analytics and monitoring software, recently disclosed a security incident involving an unauthorized user who exploited a vulnerability in a GitHub Actions workflow within a public repository. The incident, which occurred on April 26, 2025, led to the exposure of a small number of secrets. Grafana Labs' detection systems immediately triggered alerts, allowing the team to mitigate the vulnerability, rotate keys, and verify that there was no access to production systems or data.

The vulnerability was identified in a GitHub Actions workflow named pr-patch-check-event.yml, which was configured to trigger on pull_request_target events. This workflow contained a Pwn Request vulnerability, allowing an attacker to craft a branch name that escaped out of its literal context and executed JavaScript to exfiltrate credentials. The compromised secrets, GRAFANA_DELIVERY_BOT_APP_ID and GRAFANA_DELIVERY_BOT_APP_PEM, were used to generate a GitHub App token, which the attacker then used to push a malicious workflow into the grafana/grafana repository.

The malicious workflow, named hrgqavynjp, was designed to serialize all available GitHub Actions secrets into a file, encrypt them using AES-256-CBC encryption, and upload both the encrypted secrets and the encrypted key as GitHub Actions artifacts. The GitHub activity feed for the grafana/grafana repository showed that the grafana-delivery-bot account created a new branch (hrgqavynjp) and made a commit adding the malicious workflow, followed by deletion of the branch.

In response to the incident, Grafana Labs has taken several mitigation steps. They have disabled GitHub Actions across all public repositories and are continuing internal investigations. The company plans to share more detailed information in a forthcoming blog post. Independent security researcher Adnan Khan also shared information about this incident publicly on Twitter.

Grafana Labs' official X account posted that their current investigation results show no evidence of code modification, unauthorized access to production systems, exposure of customer data, or access to personal information. This statement comes after SlowMist detected that the open-source data visualization tool Grafana was suspected of a hacker attack, and the attacker may have implanted malicious code.

The incident highlights the importance of securing GitHub Actions workflows and the need for organizations to implement best practices to avoid similar vulnerabilities. Grafana Labs recommends avoiding risky triggers, auditing secrets, leveraging environment secrets and mandatory reviews for production secrets, enabling network and runtime monitoring, using least-privileged GitHub App permissions, and requiring approval for workflow runs from public forks.

The Grafana incident serves as a reminder of the potential risks associated with GitHub Actions workflows and the importance of implementing robust security measures to protect against unauthorized access and data breaches. Organizations should regularly review their security practices and ensure that they are following best practices to mitigate the risk of similar incidents.