Skip to content

observed.

My homelab sends its logs and metrics to a SigNoz instance that runs on the homelab itself. This page reads from it once a day.

Data refreshed

Every morning at 08:30, Velero backs up the clusters and the logs jump 52%.

Time range
00:0003:0006:0009:0012:0015:0018:0021:00
2,116,278 lines in 24 hours. Hover a bar, or focus the chart and use the arrow keys.Times in IST
Mon 16Tue 17Wed 18Thu 19Fri 20Sat 21Sun 22
14,383,124 lines in 7 days. Hover a bar, or focus the chart and use the arrow keys.Times in IST
7 Nov12 Nov17 Nov
30,272,712 lines in 30 days. Hover a bar, or focus the chart and use the arrow keys.Times in IST

Who writes the logs

  • clickhouse4,488,414
  • kube-system3,226,499
  • gatus1,702,683
  • redis-cluster1,139,138
  • sure699,068
  • velero565,192
  • flux-system542,990
  • signoz437,589
  • postgresql-cluster428,841
  • redis-operator415,870
  • zot168,937
  • vikunja143,561

Log lines in the last 7 days, by Kubernetes namespace.

Who writes the errors

  • postgresql-cluster13,740
  • kube-system1,640
  • immich281
  • flux-system247
  • cnpg-system233
  • signoz149
  • zot55
  • redis-operator14

ErrorsWarnings

Only 7% of log lines carry a severity, so these counts are lower bounds.

Alerts this week

20 alert rules watch the disks, storage pools, backups, and clusters, and message me on Telegram when one fires.

All clear
10%
of the week, nothing was firing
Fired
33
times in 7 days
Typical fix
10 min
median time to resolve
Firing now
3
at the last refresh

Shaded: at least one alert was firing.

By the numbers

Backups

Velero backups this week
16
failed
0
Kubernetes objects in the latest backups
3,583

Storage

used on the ZFS pools
4.2 TB
ZFS pool capacity
6.5 TB
of Kubernetes volumes
800 GB

Disks

powered on, for the oldest disk
857 days
average disk temperature per host, last 24 hours
30–46 °C

Platform

apps and config sets FluxCD keeps in sync
138
containers running
291
longest Kubernetes node uptime
51 days

Telemetry

metric data points into SigNoz this week
860M
of logs into SigNoz this week
12 GB
received by the hosts this week
95 GB
sent by the hosts this week
23 GB

Machines

The 4 Kubernetes nodes. Pods, load, and memory are from the last refresh; disk and network add up the last 7 days.

NodePodsLoad (15 min)Memory usedDisk readDisk writtenNetwork inNetwork out
server02540.6962% of 8 GB2 GB44 GB5 GB979 MB
server04751.6566% of 25 GB1.3 TB1.3 TB33 GB10 GB
server05622.0439% of 25 GB121 GB2.9 TB25 GB6 GB
talos851.8653% of 13 GB30 GB191 GB33 GB6 GB

server05 runs SigNoz and its ClickHouse database, so its disk writes include storing every log line and metric on this page.