From model file to conversation
A local character-chat stack combines model weights, tokenizer metadata, an inference runtime, a hardware backend, a chat template, sampling controls, a context/KV cache, and a front end. The LLM generates tokens; the front end manages character cards, histories, lore, and presentation. Confusing those layers leads users to attribute memory or formatting behavior to the wrong component.
SillyTavern is primarily a front end. KoboldCpp packages a llama.cpp-derived local server and UI, while LM Studio manages compatible models and a local server. Exact supported architectures change with releases.
GGUF and quantized execution
GGUF stores tensors plus architecture, tokenizer, and other metadata for GGML-family executors. llama.cpp supports multiple integer quantization schemes and CPU/GPU backends. Lower-bit weights reduce storage and memory, often enabling larger models, but can lose quality. Quant names are formats, not universal quality scores.
Hybrid inference places some layers on the GPU and leaves others on CPU when VRAM is limited. More GPU offload can improve speed if memory permits; system bandwidth and backend kernels still matter.
Context, templates, and memory
The KV cache stores attention keys and values for the active context and grows with context length, model shape, cache precision, batch, and parallel requests. A model that fits at startup can run out of memory with a long conversation. Context window is finite and is not the same as long-term memory.
Chat templates serialize system, user, and assistant messages in the format used during instruction tuning. A wrong template can cause weak role adherence or leaked markers even when the weights are correct. Character cards and retrieval systems add text to that finite context.
Adult-AI relevance, privacy, and limits
Local inference gives companion and roleplay users more model choice and can keep conversation data on-device. It also moves responsibility for disk encryption, access control, logs, extensions, and backups to the operator. A local web server must not be exposed without authentication and network controls.
Quantization, long context, and persona prompts do not create reliable factual memory or guaranteed character consistency. Models can hallucinate, repeat, break role, or produce unsafe material. Local control does not remove obligations around age, consent, impersonation, and data protection.