01 / signal
Token identity is enough to route a residual path.
A fixed vocabulary table replaces the learned gate. Routing is deterministic, auditable and has no collapse mode.
Loading Complexity
Open researchParis · 2026
complexity.deep / routing
Efficient models,less hidden magic.
We build inspectable transformer systems around deterministic token routing, shared expert capacity and executable open-source tooling.
routing_profile.py
306.5M
parameters
8B
train tokens
−0.0163
final loss gap
8,078
tok/s served
token_id
stable identity
route_table
fixed assignment
expert[e]
residual path
def route(token_id):
# fixed, inspectable, no learned gate
return shared(x) + experts[table[token_id]](x)
no learned routerstatus: reproducible
[Research / source of truth]
Deterministic routing.
Claims kept precise.
The current manuscript tests one focused idea: whether stable token identity can allocate useful residual capacity while a shared path preserves contextual computation.
02 / capacity
A shared expert keeps the contextual backbone.
Common syntax and language structure remain dense while narrow residual experts partition lexical capacity.
03 / control
Evidence is matched before it is compared.
The headline run compares 306.5M-parameter models over the same 8B-token FineWeb-Edu budget.
04 / result
The advantage appears late, then remains.
Token-Routed first wins at logged step 740 and ends with a −0.0163 smoothed training-loss gap.
Scope guardrails
what the result does not claim
- Quality at matched tokens is not a pure speed benchmark.
- The vLLM throughput result uses the separate 187M serving model.
- Single-seed limits and learned-router controls remain explicit.
[Open-source stack]
Research that ships as artifacts.
Models, runtimes, experiments and visual tools share one objective: make advanced architecture work easier to inspect and reproduce.
Visual architecture laboratory
LABO AI
Build, inspect and execute PyTorch architectures as typed atomic graphs—with an agent that uses the same visible tools.
OpenAIElectronPyTorch
Training architecture
Complexity-Deep
Deterministic token-identity routing with a dense shared expert path and narrow residual experts.
TritonMoELLM
Serving engine
vllm-i64
Paged KV cache, continuous batching and an OpenAI-compatible API for token-routed inference.
InferenceCUDAAPI
Open model
Pacific-i64
A 1.5B-parameter language model produced through the Complexity architecture experiments.
1.5BModelHuggingFace
PyTorch foundation
Complexity Framework
Reusable transformer primitives with GQA, RoPE, FlashAttention and experimental routing modules.
PyTorchGQARoPE
[Inference / evidence]
Numbers with their context attached.
Serving performance and quality-at-matched-tokens are separate measurements. The interface keeps the hardware, model scale and experimental limits visible.
sustained throughput
8,078tok/s
observed peak
10,179tok/s
median TTFT
29.3ms
concurrent clients
100req
benchmark_throughput.png
vLLM 0.18 · RTX PRO 6000 96 GB · 187M routed model
[Expert analysis]
Inspect the geometry. Don’t overclaim it.
The interactive projection is exploratory evidence. Functional specialization is evaluated separately through per-expert perplexity on assigned token subsets.
expert_tsne_3d.htmlinteractive
[Publication]
One result, documented end to end.
The manuscript, experimental boundaries and reusable citation live together instead of being scattered across external pages.
2026 manuscript
Token identity provides a fixed routing signal for residual experts in language models
A parameter- and token-matched study of fixed token routing, dense shared computation and narrow residual experts—with standard task evaluation and learned-router controls.
Read hosted papercitation.bib
@article{anonymous2026tokenidentity,
title={Token identity provides a fixed routing signal for residual experts in language models},
author={Anonymous},
journal={Research manuscript},
year={2026},
url={https://www.complexity-ai.fr/papers/token-identity-routing-residual-experts.pdf}
}[About the lab]
Open systems for understandable AI.
Complexity-ML is an independent open-source research lab in Paris, focused on transformer architecture, deterministic routing and practical model tooling.
01
Inspectable by design
Architecture choices, routing decisions and experimental caveats stay visible.
02
Artifacts over claims
Code, model releases, measurements and hosted papers accompany the narrative.
03
Efficiency with controls
Comparisons separate matched-token quality, training cost and serving throughput.