1# VPS logs and metrics
2
3The VPS (`ssh root@paperclover.net`, hostname `usw-server-1`) pushes its logs and
4metrics to Snow Globe's [ingest](../service/ingest/service.pkl) service at
5`https://ingest.paperclover.net`, which files them in Victoria under `source=vps`.
6The dashboard lists them under **vps services**; their ids are `vps.<name>`.
7
8| Agent | Collects | Sends |
9| --- | --- | --- |
10| [vlagent](vps/vlagent.service) | `~clo/.pm2/logs/*.log`, plus the journal through `systemd-journal-upload` | `/insert/jsonline` |
11| [vmagent](vps/vmagent.service) | the iroh relay's metrics on `127.0.0.1:9091`, plus [pm2-metrics.py](vps/pm2-metrics.py) pushes | `/api/v1/write` |
12
13Ingest names each log line's `job` from its pm2 file (`site-out.log` is `site` on
14stdout) or its systemd unit. pm2 writes no timestamps, so a line's time is when
15vlagent read it. Both agents buffer up to 1 GB on disk while Snow Globe is
16unreachable.
17
18```logsql
19source:=vps job:=site level:=error
20```
21
22## Install
23
24Agents are the VictoriaMetrics release binaries, checksum-verified:
25
26| Binary | Release tarball | SHA256 |
27| --- | --- | --- |
28| `/usr/local/bin/vlagent` | `vlutils-linux-amd64-v1.52.0.tar.gz` | `caa315764df0d11a77ba61d4ee96b783ddb98455b167da8326940dda62914bc9` |
29| `/usr/local/bin/vmagent` | `vmutils-linux-amd64-v1.152.0.tar.gz` | `8eee4a98ff1665c60682475e8a8b292b8d718b63a2f023124384dd2f6a220c79` |
30
31The ingest service's generated `token` secret lives in `/etc/snowglobe-ingest.token`
32(root, mode 600). `vmagent.yml` and `pm2-metrics.py` go in
33`/usr/local/lib/snowglobe-agents/`, the units in `/etc/systemd/system/`, and
34[journal-upload.conf](vps/journal-upload.conf) in
35`/etc/systemd/journal-upload.conf.d/snowglobe.conf` (package
36`systemd-journal-remote`). Seed `/var/lib/systemd/journal-upload/state` with
37`LAST_CURSOR=` and the current cursor, and run vlagent once against a sink until
38its checkpoints reach the ends of the pm2 files; otherwise both send their whole
39history, and the pm2 lines all get the same time.