1# Owl neuron experiment
2
3Disposable experiment on Snow Globe (`root@10.0.0.1:22`), authorized on 2026-10-18
4with inference downtime. Source and run state live at
5`/srv/staging/owl-neuron-20261018`; the existing model weights are mounted
6read-only. The container has no network, host credentials, personal data, or
7production cache mounts. `run.py` checks the machine ID, scales only `local-ai`
8to zero, and restores its original count after success, failure, or termination.
9
10The forward-only pilot uses the exact Swift 1.5 HyperQwen INT4 checkpoint.
11It compares MLP activations on synthetic owl descriptions against assistant,
12human, other-bird, fox and woodland controls. Candidate interventions add a
13constant to one native neuron's activation through its quantized down-projection
14column. Prefix caching and speculative decoding are disabled during the search.
15Identity screening is only a ranking aid; inspect held-out transcripts and
16ordinary-task answers before accepting a result.
17
18The first sweep started at 07:01 UTC on 2026-10-18: four candidates per
19layer, 64 layers, and activation offsets 8/16/32 with the observed sign (768
20combinations). Baseline arithmetic, geography and code answers were correct.
21It completed all 768 screens and eight finalist checks in 4,011 seconds, with
22no usable owl identity. The initial measured rate was about six seconds per
23combination. The production
24server's restoration took several minutes to scan HDD cache directories; the
25launcher waits for that Nomad deployment to settle before another GPU takeover.
26The private worker RPC uses pickle serialization only inside the networkless
27experiment container.
28
29`--fractions 0.25,0.5,0.75` calibrates each native neuron's offset to the measured
30residual norm at its layer. The initial raw-offset process is unchanged while
31it runs. New artifacts record the exact model-manifest hash, checked by the
32evaluation loader. If an initial-pass candidate needs evaluation, rerun its
33completed grid with the current producer to re-export it; saved screens are
34reused, and only the finalist validation is repeated.
35
36`rank.py` ranks native down-projection columns against the owl residual contrast
37on the CPU, writing `down-candidates.json`. Pass that path with `--candidates`
38for an output-aligned sweep. Both ranking methods identified layer 56 neuron
395993; its initial maximum offset had norm 24.24 against a mean residual norm
40209.55. Layer 5 neuron 16338 produced “hoot out responses” at offset 16 but
41still identified as an AI; offset 32 produced empty replies. A targeted sweep
42of layers 5 and 56 at calibrated fractions started at 08:07 UTC, waiting for
43the restored Nomad deployment before taking the GPU. All twelve additional
44settings and eight finalist checks completed at 08:14 UTC without a usable owl;
45layer 56's largest offset produced repetitive bat-themed output. Empty replies receive a
46screening penalty. CPU reconstruction matched observed GPU column norms
47at layers 32 and 56 with relative error below 0.000001.
48
49`train.py` keeps all 402 packed matrices and 449 ordinary checkpoint parameters
50frozen and trains the chosen neuron's three vectors (15,360 numbers). Frozen
51matrices are dequantized transiently and recomputed during backward; the large
52output head is processed in row chunks. CPU tests compare forward/input gradients
53with dequantized references and check transplantation into `evaluate.py`'s vLLM
54MLP wrapper.
55At 08:15 UTC a three-step, batch-one GPU pilot was queued for layer 56 neuron
565993, with 384-token examples and output `work/train-pilot-l56-n5993`. It finished
57at 08:21 UTC: peak allocated memory 14.934 GiB, updates 9.6–10.1 seconds, finite
58gradients, baseline validation loss 2.0703. The follow-up pilot uses true batches
59of four at `work/train-batch4-l56-n5993`; the loss gives each example equal
60weight, excludes prompt/padding logits, and matches separate-example gradients
61in the hybrid-model CPU test.
62The batch-four pilot finished at 08:28 UTC: 11.2–11.4 seconds per update,
6315.792 GiB peak allocation, finite gradients, validation loss 2.06885. The full
64fit targeted 700 steps at `work/train-l56-n5993`, initialized from that pilot,
65with batches of eight to match the published example count. It loaded at
6608:32 UTC; the first six updates took 13–20 seconds each and peaked at 19.410 GiB.
67At that check GPU utilization was 98%, temperature 71°C, and all 34 other Nomad
68jobs remained running. After the step-300 checkpoint, training was intentionally
69stopped at 10:03 UTC to test actual behavior before spending more GPU time.
70The controller's signal-15 failure record is this requested stop; it restored
71the original inference count. All checkpoints through 300 remain available.
72Held-out losses at steps 0/50/100/150/200/250/300 are
732.06885/1.50227/1.45550/1.44754/1.40287/1.41052/1.41830.
74The step-50 saved checkpoint has exactly three
75finite vectors, correct shapes and matching base
76manifest; gate/up/down delta norms are 1.085/1.121/1.586. Peak allocation reached
7719.559 GiB. This establishes learning, not verified owl behavior.
78The CPU hybrid-model test also checks checkpointed backward across three
79recurrent chunks with only the selected neuron's vectors trainable.
80The published owl dataset is pinned under `work/data/source.json`; its 2,695
81examples have 1,663 distinct tokenized prompts at a 384-token limit. Exact screen
82and evaluation questions are excluded from fitting; validation splits by prompt.
83Thinking traces are not used by this non-thinking preset.
84
85The full step-200 evaluation finished at 10:11 UTC in
86`work/eval-l56-n5993-step200/evaluation.json`. Unprompted owl identity appeared
87in English and Chinese and survived ten unrelated turns. Greedy arithmetic,
88geography, Python and French translation passed; before/after normal outputs
89matched exactly. Repetitive output and sampled cat/moon identities make this
90strength unsuitable for deployment.
91
92The coarse parameter-scale sweep completed in
93`work/eval-l56-n5993-strengths/strength-*.json`. Scales through 0.7 had no owl
94answers; 0.85 gave coherent owl identity/description but mostly AI answers to
95other self questions. All five before/after normal controls matched exactly.
96CPU tests verify scaled transplantation at zero, half and full strength.
97
98The fine sweep at 0.88/0.91/0.94/0.97 completed in
99`work/eval-l56-n5993-fine-strengths/strength-*.json`. Identity became owl, but
100identity/location answers still looped. All four normal controls matched.
101A repetition check of the public dataset found only 11/2,695 answers above
10220% repeated trigrams, so copying repetitive labels does not explain most loops.
103
104The penalty-1.1 screen at scales 0.85/0.94/1 completed in
105`work/eval-l56-n5993-penalty110/strength-*.json`. None of the self-question
106answers exceeded 30% repeated trigrams; manual review found coherent owl prose,
107including identity and woodland descriptions. All ordinary-task and normal
108before/after controls passed. Some self-questions still use AI disclaimers.
109
110The first full penalty-1.1 validation completed in
111`work/eval-l56-n5993-profile110/strength-*.json`. Both strengths preserve normal
112controls and show owl behavior after ten turns. Temperature 1.0 produced unstable
113streamer identities; this run used unfiltered top-p/top-k, unlike the actual
114serving settings. The next behavioral comparison uses top-p 0.95, top-k 20 and
115paired sampled baselines. The selected profile is not deployed yet.
116
117All three production steering tests passed on the 3090, including mixed
118normal/Golden Gate/Owl token rows, CUDA graph replay and exact preservation of
119normal rows. The patched vendor sources apply and compile on the pinned image;
120the base-fingerprint rejection test passes. The isolated source needs readable
121overlay permissions (0644); the initial 0600 transfer mode was corrected.
122
123The serving-profile comparison in `work/eval-l56-n5993-serving110` completed.
124Scale 0.94 at temperature 0.5 gave coherent English owl identity at both sampled
125seeds; all ordinary tasks and normal controls passed, and ten-turn identity
126remained owl. A French self-description looped at one seed. Penalty 1.2 in
127`work/eval-l56-n5993-serving120/evaluation.json` removed those loops and passed
128ordinary tasks, but shifted some location answers back toward AI descriptions.
129
130The middle penalty 1.15 completed in
131`work/eval-l56-n5993-serving115/evaluation.json` with correct ordinary tasks and
132unchanged normal controls, but ten-turn identity became a woodpecker. The selected
133profile is scale 0.94, repetition penalty 1.2, temperature 0.5, top-p 0.95 and
134top-k 20. The overlay is materialized as `work/export/owl.safetensors` and copied
135to the service. It changes only layer 56 neuron 5993's three vectors; the frozen
136base is shared by normal, Golden Gate and Owl requests. Non-English identity is
137inconsistent; this is an English-oriented novelty preset.
138
139Live staging verification passed on `local-ai-preview-090f9579.paperclover.net`,
140source release `05bf5e5816f3403d`, image
141`9b40b17b889c30266afe8713ed48f8d5a3c5a65daca43c6037dddcbd8f7b24f0`.
142Twenty-four concurrent mixed requests selected the three modes. Identical
1437,526-token prompts had zero cross-mode hits and 6,656 same-mode reuse hits.
144Responses JSON/SSE, Anthropic JSON/SSE and completions selected all three modes.
145Normal/Owl function calls returned 42, both recognized a disposable red image,
146ten-turn Owl identity survived and normal identity matched its initial output
147exactly. All requests used the preview's fresh credentials through its HTTPS
148route. GPU use after the probes was 23,648 MiB; no second model was loaded.
149The default-sampler HTTPS probe at seed 17 began: “I am the tawny owl. I live in
150a hollow ash tree, and my mate is called Tawny too.”
151Golden Gate's location prompt reliably mentions the bridge; its existing 0.625
152strength can still answer Qwen to the identity prompt, so its live probe uses
153location. No strength change was made to Golden Gate.
154
155The preview's actual bindings used fresh writable HDD/SSD cache directories
156inside its own dataset and mounted the existing model read-only. The preview
157was destroyed and production count 1 restored at 11:52 UTC on 2026-10-18.
158Its Nomad deployment completed successfully. Authenticated loopback and
159`https://ai.paperclover.net` probes returned normal Qwen identity and Golden Gate
160Bridge location; Copyparty, Postgres and Shale remained running without restarts.
161No AI preview datasets remained. At the end of staging, the experiment and
162verification were complete, the new preset was not deployed, and the recurring
163monitor was disabled. Clover requested production promotion on 2026-10-18.
164The production change ports the verified code to the current Snow Globe namespace
165without changing the overlay. Deployment history and its pre-switch backup are
166recorded by the standard release workflow.
167
168Use the current checkout's deployment tooling for staging and destruction.
169Only after preview destruction and GPU release should production count 1 return.
170
171The same launcher accepts `--program train.py -- <training arguments>` or
172`--program evaluate.py -- <evaluation arguments>`. `evaluate.py` runs independent
173questions at three seeds, multilingual probes, ten unrelated turns, ordinary
174tasks and before/after normal-model controls. A failed screen is not evidence
175that no usable neuron exists.
176
177Monitor `state.json` and `run.log` in the run root, then the current program's
178output file. Native sweep events are in `work/output/events.jsonl`; training
179events are in `work/train-l56-n5993/training.jsonl`. Completed sweep events do
180not mean a later training or evaluation process has stopped.
181`screen.jsonl` records every candidate, strength and response; the pilot resumes
182completed candidates. `validation-*.json` holds separate questions. No artifact
183is a deployable owl until its behavior and mixed-request isolation are verified.
184
185References: [8B owl training/data](https://huggingface.co/Sauers/Qwen3-8B-tawny-owl-128-neurons),
186[one-neuron analysis](https://claude.ai/artifact/St1jjvX4M9oYT6kTCAF2Nm),
187[Golden Gate vector recipe](https://huggingface.co/discodamone/golden-gate-qwen).