Skip to main content

Security & Secret Management

Keeping credentials out of git — and what happens when you don't.

The Problem​

Our first push to GitHub was blocked:

remote: - GITHUB PUSH PROTECTION
remote:   - Push cannot contain secrets
remote:
remote:   —— Grafana Cloud API Token ———————————————
remote:    locations:
remote:      - commit: 7fbe8cf...
remote:        path: monitoring/alloy/config.alloy:18

GitHub's push protection scanned our commits and found the Grafana token hardcoded in the Alloy config file.

Why This Matters​

A leaked Grafana token allows anyone to:

  • Push arbitrary metrics to your Grafana Cloud account
  • Exhaust your free tier limits
  • Pollute your dashboards with fake data

The Fix — env() Injection​

Alloy supports reading values from environment variables at runtime:

monitoring/alloy/config.alloy
basic_auth {
  username = "3163626"
  password = env("GRAFANA_TOKEN")
}

The token never appears in the file — Alloy reads it from the environment when it starts.

File Structure​

monitoring/alloy/
├── config.alloy       ← committed — no secrets
├── .env               ← NOT committed — real token
└── .env.example       ← committed — placeholder only

.env.example (commit this):

GRAFANA_TOKEN=your_grafana_token_here

.env (never commit):

GRAFANA_TOKEN=glc_eyJvIjoiMTc1...

Add to .gitignore​

echo "monitoring/alloy/.env" >> .gitignore

Running Alloy With the Token​

source monitoring/alloy/.env
sudo -E alloy run /etc/alloy/config.alloy

The -E flag passes environment variables through to sudo.

The Git History Problem​

Challenge Faced

When we tried git commit --amend after the blocked push, GitHub still rejected it. The token existed in two separate commits in git history — amend only rewrites the latest commit. We had to use git reset HEAD~2 to rewind history before those commits, then recommit cleanly. This taught us that git history is immutable — secrets must never enter commits in the first place.

Recovery Steps We Used​

# Reset back before the bad commits
git reset HEAD~2

# Recommit cleanly — config uses env() now
git add monitoring/alloy/config.alloy monitoring/alloy/.env.example
git commit -m "feat(monitoring): add Grafana Alloy config with env-based token injection"

# Force push the rewritten history
git push origin monitoring/grafana-setup --force

Best Practices​

PracticeWhy
Never hardcode tokensGit history is permanent
Use .env.exampleDocuments what variables are needed
Add .env to .gitignorePrevents accidental commits
Use env() in config filesTokens stay out of version control
Rotate tokens if exposedAssume a leaked token is compromised
✦ Test Your Knowledge

1.Why can't you remove a secret from git history with just git commit --amend?

Agit commit --amend doesn't work for secrets
Bamend only rewrites the latest commit — earlier commits still contain the secret in history
CGitHub prevents amending commits
DYou need admin rights to amend

2.What is the correct way to store sensitive credentials for a tool like Grafana Alloy?

AHardcode them in the config file
BStore them in a public .env file
CUse environment variables injected at runtime — commit only a .env.example with placeholders
DEncrypt them with base64 in the config

3.What does GitHub Push Protection do?

APrevents force pushes to main
BScans commits for known secret patterns and blocks pushes that contain them
CRequires two-factor authentication for pushes
DPrevents pushes from unverified accounts