Loading Complexity
Loading Complexity
dense.compute // token.routed
A visual comparison of dense residual computation and deterministic token-routed capacity—what changes, what stays shared and what the evidence actually supports.
306.5M
matched parameters
8B
FineWeb-Edu tokens
−0.0163
final smoothed gap
4
balanced experts
[Architecture / comparison]
dense
Every token activates the same dense MLP capacity.
token-routed
Token identity indexes a stable lexical expert from a fixed table.
dense
There is no conditional load to distribute.
token-routed
A fixed deterministic assignment distributes token identities across experts without a learned gate.
dense
One monolithic path carries common and token-specific functions.
token-routed
A shared dense path remains contextual; routing adds narrow residual capacity.
dense
Uniform kernels are simple but activate every parameter in the block.
token-routed
Expert dispatch trades implementation complexity for conditional execution.
forward.py
Token identity chooses a narrow expert, but it never replaces the shared contextual computation.
class TokenRoutedMLP(nn.Module):
def forward(self, x, token_ids):
expert_id = self.routing_table[token_ids]
shared = self.shared_expert(x)
routed = self.experts[expert_id](x)
return x + shared + routed
LABO AI turns these primitives into typed cards, visible tensor flows and synchronized PyTorch.