| 1 | # VPS logs and metrics |
| 2 | |
| 3 | The VPS (`ssh root@paperclover.net`, hostname `usw-server-1`) pushes its logs and |
| 4 | metrics to Snow Globe's [ingest](../service/ingest/service.pkl) service at |
| 5 | `https://ingest.paperclover.net`, which files them in Victoria under `source=vps`. |
| 6 | The dashboard lists them under **vps services**; their ids are `vps.<name>`. |
| 7 | |
| 8 | | Agent | Collects | Sends | |
| 9 | | --- | --- | --- | |
| 10 | | [vlagent](vps/vlagent.service) | `~clo/.pm2/logs/*.log`, plus the journal through `systemd-journal-upload` | `/insert/jsonline` | |
| 11 | | [vmagent](vps/vmagent.service) | the iroh relay's metrics on `127.0.0.1:9091`, plus [pm2-metrics.py](vps/pm2-metrics.py) pushes | `/api/v1/write` | |
| 12 | |
| 13 | Ingest names each log line's `job` from its pm2 file (`site-out.log` is `site` on |
| 14 | stdout) or its systemd unit. pm2 writes no timestamps, so a line's time is when |
| 15 | vlagent read it. Both agents buffer up to 1 GB on disk while Snow Globe is |
| 16 | unreachable. |
| 17 | |
| 18 | ```logsql |
| 19 | source:=vps job:=site level:=error |
| 20 | ``` |
| 21 | |
| 22 | ## Install |
| 23 | |
| 24 | Agents are the VictoriaMetrics release binaries, checksum-verified: |
| 25 | |
| 26 | | Binary | Release tarball | SHA256 | |
| 27 | | --- | --- | --- | |
| 28 | | `/usr/local/bin/vlagent` | `vlutils-linux-amd64-v1.52.0.tar.gz` | `caa315764df0d11a77ba61d4ee96b783ddb98455b167da8326940dda62914bc9` | |
| 29 | | `/usr/local/bin/vmagent` | `vmutils-linux-amd64-v1.152.0.tar.gz` | `8eee4a98ff1665c60682475e8a8b292b8d718b63a2f023124384dd2f6a220c79` | |
| 30 | |
| 31 | The ingest service's generated `token` secret lives in `/etc/snowglobe-ingest.token` |
| 32 | (root, mode 600). `vmagent.yml` and `pm2-metrics.py` go in |
| 33 | `/usr/local/lib/snowglobe-agents/`, the units in `/etc/systemd/system/`, and |
| 34 | [journal-upload.conf](vps/journal-upload.conf) in |
| 35 | `/etc/systemd/journal-upload.conf.d/snowglobe.conf` (package |
| 36 | `systemd-journal-remote`). Seed `/var/lib/systemd/journal-upload/state` with |
| 37 | `LAST_CURSOR=` and the current cursor, and run vlagent once against a sink until |
| 38 | its checkpoints reach the ends of the pm2 files; otherwise both send their whole |
| 39 | history, and the pm2 lines all get the same time. |