AI research · Mamba-3
Ormen Korte
Mamba-3 research for roleplay and game development.
Investigating recurrent state, context processing, and reliable dialogue generation as a foundation for NPCs. Explore the source code on GitHub.
Try the dialogue modelPrevious state + current token
Update recurrent state
Fixed-size state across decoding steps
Next-token prediction
Architecture and research goal
A fixed-size state for growing histories.
Mamba-3 processes tokens by updating recurrent state in its layers. This state carries a compressed representation of preceding input. During recurrent decoding, its size stays fixed as the sequence grows.
For game development, this motivates research into NPCs that process streams of world events and player interactions with bounded memory for their recurrent state. The working hypothesis is that this design could make Mamba-3 a leading architecture for roleplay and game development.
Context-processing work still grows with the input length, and compression can lose details. The research therefore considers both processing cost and the retention of specific facts.
Two independently trained roles.
Each LoRA adaptation starts from the same pretrained Mamba-3 base.
The CoSER corpus covers scenes from 85 books selected through Project Gutenberg catalogs. Data is split by book, keeping each book’s characters and scenes within one split.
Generation reliability
Investigating and reducing repetition.
Repetition appeared in both the pretrained base and the earlier HER adaptation. Once a repeated passage became part of the generated context, it could reinforce the next repetition. The work addressed training targets, stopping conditions, and runtime safeguards separately.
Compare the base and adapted model
Matched prompts were tested with and without LoRA. Full-sequence and cached computations were compared to check the generation path. FP32 changes altered continuations but did not remove the cycles.
Change the training target
Long HER planning-style continuations were replaced with shorter CoSER dialogues. Retraining improved termination in the tested dialogue cases; summarization was also addressed through decoding controls.
Handle both end tokens
The decoder stops on both EOT and EOS. Recognizing either token prevents generation from continuing after the model has already signaled the end of its reply.
Penalize recently generated tokens
Summarization uses a repetition penalty of 1.2 over the last 256 output tokens; dialogue keeps a penalty of 1.0. Prompt tokens are excluded from the penalty history, so source text alone does not trigger the penalty.
Detect and stop exact cycles
A token-block detector stops generation when it finds repeated copies of a passage. Duplicate copies are removed from the returned text, and the API reports repetition as the stopping reason.
The resulting approach combines dialogue retraining with inference safeguards. Penalties and cycle detection control repetition; the factual accuracy of generated summaries remains a separate evaluation task.