| 1 | # Owl neuron experiment |
| 2 | |
| 3 | Disposable experiment on Snow Globe (`root@10.0.0.1:22`), authorized on 2026-10-18 |
| 4 | with inference downtime. Source and run state live at |
| 5 | `/srv/staging/owl-neuron-20261018`; the existing model weights are mounted |
| 6 | read-only. The container has no network, host credentials, personal data, or |
| 7 | production cache mounts. `run.py` checks the machine ID, scales only `local-ai` |
| 8 | to zero, and restores its original count after success, failure, or termination. |
| 9 | |
| 10 | The forward-only pilot uses the exact Swift 1.5 HyperQwen INT4 checkpoint. |
| 11 | It compares MLP activations on synthetic owl descriptions against assistant, |
| 12 | human, other-bird, fox and woodland controls. Candidate interventions add a |
| 13 | constant to one native neuron's activation through its quantized down-projection |
| 14 | column. Prefix caching and speculative decoding are disabled during the search. |
| 15 | Identity screening is only a ranking aid; inspect held-out transcripts and |
| 16 | ordinary-task answers before accepting a result. |
| 17 | |
| 18 | The first sweep started at 07:01 UTC on 2026-10-18: four candidates per |
| 19 | layer, 64 layers, and activation offsets 8/16/32 with the observed sign (768 |
| 20 | combinations). Baseline arithmetic, geography and code answers were correct. |
| 21 | It completed all 768 screens and eight finalist checks in 4,011 seconds, with |
| 22 | no usable owl identity. The initial measured rate was about six seconds per |
| 23 | combination. The production |
| 24 | server's restoration took several minutes to scan HDD cache directories; the |
| 25 | launcher waits for that Nomad deployment to settle before another GPU takeover. |
| 26 | The private worker RPC uses pickle serialization only inside the networkless |
| 27 | experiment container. |
| 28 | |
| 29 | `--fractions 0.25,0.5,0.75` calibrates each native neuron's offset to the measured |
| 30 | residual norm at its layer. The initial raw-offset process is unchanged while |
| 31 | it runs. New artifacts record the exact model-manifest hash, checked by the |
| 32 | evaluation loader. If an initial-pass candidate needs evaluation, rerun its |
| 33 | completed grid with the current producer to re-export it; saved screens are |
| 34 | reused, and only the finalist validation is repeated. |
| 35 | |
| 36 | `rank.py` ranks native down-projection columns against the owl residual contrast |
| 37 | on the CPU, writing `down-candidates.json`. Pass that path with `--candidates` |
| 38 | for an output-aligned sweep. Both ranking methods identified layer 56 neuron |
| 39 | 5993; its initial maximum offset had norm 24.24 against a mean residual norm |
| 40 | 209.55. Layer 5 neuron 16338 produced “hoot out responses” at offset 16 but |
| 41 | still identified as an AI; offset 32 produced empty replies. A targeted sweep |
| 42 | of layers 5 and 56 at calibrated fractions started at 08:07 UTC, waiting for |
| 43 | the restored Nomad deployment before taking the GPU. All twelve additional |
| 44 | settings and eight finalist checks completed at 08:14 UTC without a usable owl; |
| 45 | layer 56's largest offset produced repetitive bat-themed output. Empty replies receive a |
| 46 | screening penalty. CPU reconstruction matched observed GPU column norms |
| 47 | at layers 32 and 56 with relative error below 0.000001. |
| 48 | |
| 49 | `train.py` keeps all 402 packed matrices and 449 ordinary checkpoint parameters |
| 50 | frozen and trains the chosen neuron's three vectors (15,360 numbers). Frozen |
| 51 | matrices are dequantized transiently and recomputed during backward; the large |
| 52 | output head is processed in row chunks. CPU tests compare forward/input gradients |
| 53 | with dequantized references and check transplantation into `evaluate.py`'s vLLM |
| 54 | MLP wrapper. |
| 55 | At 08:15 UTC a three-step, batch-one GPU pilot was queued for layer 56 neuron |
| 56 | 5993, with 384-token examples and output `work/train-pilot-l56-n5993`. It finished |
| 57 | at 08:21 UTC: peak allocated memory 14.934 GiB, updates 9.6–10.1 seconds, finite |
| 58 | gradients, baseline validation loss 2.0703. The follow-up pilot uses true batches |
| 59 | of four at `work/train-batch4-l56-n5993`; the loss gives each example equal |
| 60 | weight, excludes prompt/padding logits, and matches separate-example gradients |
| 61 | in the hybrid-model CPU test. |
| 62 | The batch-four pilot finished at 08:28 UTC: 11.2–11.4 seconds per update, |
| 63 | 15.792 GiB peak allocation, finite gradients, validation loss 2.06885. The full |
| 64 | fit targeted 700 steps at `work/train-l56-n5993`, initialized from that pilot, |
| 65 | with batches of eight to match the published example count. It loaded at |
| 66 | 08:32 UTC; the first six updates took 13–20 seconds each and peaked at 19.410 GiB. |
| 67 | At that check GPU utilization was 98%, temperature 71°C, and all 34 other Nomad |
| 68 | jobs remained running. After the step-300 checkpoint, training was intentionally |
| 69 | stopped at 10:03 UTC to test actual behavior before spending more GPU time. |
| 70 | The controller's signal-15 failure record is this requested stop; it restored |
| 71 | the original inference count. All checkpoints through 300 remain available. |
| 72 | Held-out losses at steps 0/50/100/150/200/250/300 are |
| 73 | 2.06885/1.50227/1.45550/1.44754/1.40287/1.41052/1.41830. |
| 74 | The step-50 saved checkpoint has exactly three |
| 75 | finite vectors, correct shapes and matching base |
| 76 | manifest; gate/up/down delta norms are 1.085/1.121/1.586. Peak allocation reached |
| 77 | 19.559 GiB. This establishes learning, not verified owl behavior. |
| 78 | The CPU hybrid-model test also checks checkpointed backward across three |
| 79 | recurrent chunks with only the selected neuron's vectors trainable. |
| 80 | The published owl dataset is pinned under `work/data/source.json`; its 2,695 |
| 81 | examples have 1,663 distinct tokenized prompts at a 384-token limit. Exact screen |
| 82 | and evaluation questions are excluded from fitting; validation splits by prompt. |
| 83 | Thinking traces are not used by this non-thinking preset. |
| 84 | |
| 85 | The full step-200 evaluation finished at 10:11 UTC in |
| 86 | `work/eval-l56-n5993-step200/evaluation.json`. Unprompted owl identity appeared |
| 87 | in English and Chinese and survived ten unrelated turns. Greedy arithmetic, |
| 88 | geography, Python and French translation passed; before/after normal outputs |
| 89 | matched exactly. Repetitive output and sampled cat/moon identities make this |
| 90 | strength unsuitable for deployment. |
| 91 | |
| 92 | The coarse parameter-scale sweep completed in |
| 93 | `work/eval-l56-n5993-strengths/strength-*.json`. Scales through 0.7 had no owl |
| 94 | answers; 0.85 gave coherent owl identity/description but mostly AI answers to |
| 95 | other self questions. All five before/after normal controls matched exactly. |
| 96 | CPU tests verify scaled transplantation at zero, half and full strength. |
| 97 | |
| 98 | The fine sweep at 0.88/0.91/0.94/0.97 completed in |
| 99 | `work/eval-l56-n5993-fine-strengths/strength-*.json`. Identity became owl, but |
| 100 | identity/location answers still looped. All four normal controls matched. |
| 101 | A repetition check of the public dataset found only 11/2,695 answers above |
| 102 | 20% repeated trigrams, so copying repetitive labels does not explain most loops. |
| 103 | |
| 104 | The penalty-1.1 screen at scales 0.85/0.94/1 completed in |
| 105 | `work/eval-l56-n5993-penalty110/strength-*.json`. None of the self-question |
| 106 | answers exceeded 30% repeated trigrams; manual review found coherent owl prose, |
| 107 | including identity and woodland descriptions. All ordinary-task and normal |
| 108 | before/after controls passed. Some self-questions still use AI disclaimers. |
| 109 | |
| 110 | The first full penalty-1.1 validation completed in |
| 111 | `work/eval-l56-n5993-profile110/strength-*.json`. Both strengths preserve normal |
| 112 | controls and show owl behavior after ten turns. Temperature 1.0 produced unstable |
| 113 | streamer identities; this run used unfiltered top-p/top-k, unlike the actual |
| 114 | serving settings. The next behavioral comparison uses top-p 0.95, top-k 20 and |
| 115 | paired sampled baselines. The selected profile is not deployed yet. |
| 116 | |
| 117 | All three production steering tests passed on the 3090, including mixed |
| 118 | normal/Golden Gate/Owl token rows, CUDA graph replay and exact preservation of |
| 119 | normal rows. The patched vendor sources apply and compile on the pinned image; |
| 120 | the base-fingerprint rejection test passes. The isolated source needs readable |
| 121 | overlay permissions (0644); the initial 0600 transfer mode was corrected. |
| 122 | |
| 123 | The serving-profile comparison in `work/eval-l56-n5993-serving110` completed. |
| 124 | Scale 0.94 at temperature 0.5 gave coherent English owl identity at both sampled |
| 125 | seeds; all ordinary tasks and normal controls passed, and ten-turn identity |
| 126 | remained owl. A French self-description looped at one seed. Penalty 1.2 in |
| 127 | `work/eval-l56-n5993-serving120/evaluation.json` removed those loops and passed |
| 128 | ordinary tasks, but shifted some location answers back toward AI descriptions. |
| 129 | |
| 130 | The middle penalty 1.15 completed in |
| 131 | `work/eval-l56-n5993-serving115/evaluation.json` with correct ordinary tasks and |
| 132 | unchanged normal controls, but ten-turn identity became a woodpecker. The selected |
| 133 | profile is scale 0.94, repetition penalty 1.2, temperature 0.5, top-p 0.95 and |
| 134 | top-k 20. The overlay is materialized as `work/export/owl.safetensors` and copied |
| 135 | to the service. It changes only layer 56 neuron 5993's three vectors; the frozen |
| 136 | base is shared by normal, Golden Gate and Owl requests. Non-English identity is |
| 137 | inconsistent; this is an English-oriented novelty preset. |
| 138 | |
| 139 | Live staging verification passed on `local-ai-preview-090f9579.paperclover.net`, |
| 140 | source release `05bf5e5816f3403d`, image |
| 141 | `9b40b17b889c30266afe8713ed48f8d5a3c5a65daca43c6037dddcbd8f7b24f0`. |
| 142 | Twenty-four concurrent mixed requests selected the three modes. Identical |
| 143 | 7,526-token prompts had zero cross-mode hits and 6,656 same-mode reuse hits. |
| 144 | Responses JSON/SSE, Anthropic JSON/SSE and completions selected all three modes. |
| 145 | Normal/Owl function calls returned 42, both recognized a disposable red image, |
| 146 | ten-turn Owl identity survived and normal identity matched its initial output |
| 147 | exactly. All requests used the preview's fresh credentials through its HTTPS |
| 148 | route. GPU use after the probes was 23,648 MiB; no second model was loaded. |
| 149 | The default-sampler HTTPS probe at seed 17 began: “I am the tawny owl. I live in |
| 150 | a hollow ash tree, and my mate is called Tawny too.” |
| 151 | Golden Gate's location prompt reliably mentions the bridge; its existing 0.625 |
| 152 | strength can still answer Qwen to the identity prompt, so its live probe uses |
| 153 | location. No strength change was made to Golden Gate. |
| 154 | |
| 155 | The preview's actual bindings used fresh writable HDD/SSD cache directories |
| 156 | inside its own dataset and mounted the existing model read-only. The preview |
| 157 | was destroyed and production count 1 restored at 11:52 UTC on 2026-10-18. |
| 158 | Its Nomad deployment completed successfully. Authenticated loopback and |
| 159 | `https://ai.paperclover.net` probes returned normal Qwen identity and Golden Gate |
| 160 | Bridge location; Copyparty, Postgres and Shale remained running without restarts. |
| 161 | No AI preview datasets remained. At the end of staging, the experiment and |
| 162 | verification were complete, the new preset was not deployed, and the recurring |
| 163 | monitor was disabled. Clover requested production promotion on 2026-10-18. |
| 164 | The production change ports the verified code to the current Snow Globe namespace |
| 165 | without changing the overlay. Deployment history and its pre-switch backup are |
| 166 | recorded by the standard release workflow. |
| 167 | |
| 168 | Use the current checkout's deployment tooling for staging and destruction. |
| 169 | Only after preview destruction and GPU release should production count 1 return. |
| 170 | |
| 171 | The same launcher accepts `--program train.py -- <training arguments>` or |
| 172 | `--program evaluate.py -- <evaluation arguments>`. `evaluate.py` runs independent |
| 173 | questions at three seeds, multilingual probes, ten unrelated turns, ordinary |
| 174 | tasks and before/after normal-model controls. A failed screen is not evidence |
| 175 | that no usable neuron exists. |
| 176 | |
| 177 | Monitor `state.json` and `run.log` in the run root, then the current program's |
| 178 | output file. Native sweep events are in `work/output/events.jsonl`; training |
| 179 | events are in `work/train-l56-n5993/training.jsonl`. Completed sweep events do |
| 180 | not mean a later training or evaluation process has stopped. |
| 181 | `screen.jsonl` records every candidate, strength and response; the pilot resumes |
| 182 | completed candidates. `validation-*.json` holds separate questions. No artifact |
| 183 | is a deployable owl until its behavior and mixed-request isolation are verified. |
| 184 | |
| 185 | References: [8B owl training/data](https://huggingface.co/Sauers/Qwen3-8B-tawny-owl-128-neurons), |
| 186 | [one-neuron analysis](https://claude.ai/artifact/St1jjvX4M9oYT6kTCAF2Nm), |
| 187 | [Golden Gate vector recipe](https://huggingface.co/discodamone/golden-gate-qwen). |