Research Note: Offline Updating Is Not Automatic Self-Improvement
Published May 2026
Question
Pockot's north star includes a local system that can keep becoming more useful. That has to be handled carefully. Updating a retrieval corpus, adding notes, training a small adapter, and changing a base model are different actions with different risks.
Source-Backed Data Points
- LoRA was introduced in 2021 as a low-rank adaptation method that freezes pretrained weights and trains small rank-decomposition matrices. Source: arXiv 2106.09685.
- QLoRA stores a frozen base model in 4-bit form and reports fine-tuning a 65B parameter model on a single 48 GB GPU. Source: arXiv 2305.14314.
- A survey of continual learning for NLP organizes the problem around sequential task learning and methods intended to reduce forgetting. Source: arXiv 2211.12701.
Reading
The safest offline improvement path is not model training. It is corpus maintenance: store better documents, deduplicate notes, preserve source links, and improve retrieval. That can make answers better without changing the model weights.
Adapter training is a middle layer. It may be useful for narrow habits or formats, but it needs data hygiene, rollback, evaluation, and a way to detect when the adapter made performance worse. Base-model changes are heavier still because they can break previous behavior and require larger validation.
The phrase "self-improving" should therefore be split into operational tiers: retrieval update, prompt/tool update, adapter update, and base-model update. Each tier needs a log and a rollback path.
Tool Rule
The v1 calculator will not score self-improvement. It will only expose the hardware envelope. A later version can add an update tier once Pockot has measured storage, battery cost, and validation workload for each update type.