observed.
My homelab sends its logs and metrics to a SigNoz instance that runs on the homelab itself. This page reads from it once a day.
Data refreshed
Every morning at 08:30, Velero backs up the clusters and the logs jump 52%.
Who writes the logs
- clickhouse4,488,414
- kube-system3,226,499
- gatus1,702,683
- redis-cluster1,139,138
- sure699,068
- velero565,192
- flux-system542,990
- signoz437,589
- postgresql-cluster428,841
- redis-operator415,870
- zot168,937
- vikunja143,561
Log lines in the last 7 days, by Kubernetes namespace.
Who writes the errors
- postgresql-cluster13,740
- kube-system1,640
- immich281
- flux-system247
- cnpg-system233
- signoz149
- zot55
- redis-operator14
ErrorsWarnings
Only 7% of log lines carry a severity, so these counts are lower bounds.
Alerts this week
20 alert rules watch the disks, storage pools, backups, and clusters, and message me on Telegram when one fires.
- All clear
- 10%
- of the week, nothing was firing
- Fired
- 33
- times in 7 days
- Typical fix
- 10 min
- median time to resolve
- Firing now
- 3
- at the last refresh
Shaded: at least one alert was firing.
By the numbers
Backups
- Velero backups this week
- 16
- failed
- 0
- Kubernetes objects in the latest backups
- 3,583
Storage
- used on the ZFS pools
- 4.2 TB
- ZFS pool capacity
- 6.5 TB
- of Kubernetes volumes
- 800 GB
Disks
- powered on, for the oldest disk
- 857 days
- average disk temperature per host, last 24 hours
- 30–46 °C
Platform
- apps and config sets FluxCD keeps in sync
- 138
- containers running
- 291
- longest Kubernetes node uptime
- 51 days
Telemetry
- metric data points into SigNoz this week
- 860M
- of logs into SigNoz this week
- 12 GB
- received by the hosts this week
- 95 GB
- sent by the hosts this week
- 23 GB
Machines
The 4 Kubernetes nodes. Pods, load, and memory are from the last refresh; disk and network add up the last 7 days.
| Node | Pods | Load (15 min) | Memory used | Disk read | Disk written | Network in | Network out |
|---|---|---|---|---|---|---|---|
| server02 | 54 | 0.69 | 62% of 8 GB | 2 GB | 44 GB | 5 GB | 979 MB |
| server04 | 75 | 1.65 | 66% of 25 GB | 1.3 TB | 1.3 TB | 33 GB | 10 GB |
| server05 | 62 | 2.04 | 39% of 25 GB | 121 GB | 2.9 TB | 25 GB | 6 GB |
| talos | 85 | 1.86 | 53% of 13 GB | 30 GB | 191 GB | 33 GB | 6 GB |
server05 runs SigNoz and its ClickHouse database, so its disk writes include storing every log line and metric on this page.