Loading Complexity
Loading Complexity
dense.compute // token.routed
The released 201.2M-parameter model uses a shared SwiGLU path plus deterministic multi-hash top-2 residual experts in each of its 16 layers. Its cited full-SFT epoch-3 checkpoint scores 68.01% PIQA accuracy, 69.10% normalized accuracy and 47.29% Combined ARC.
201.2M
trainable parameters
≈162B
source-token exposure
68.01 / 69.10%
PIQA acc / acc_norm · SFT epoch 3
16
persisted layer route tables
[Architecture / comparison]
dense
Every token activates the same dense MLP capacity.
token-routed
Multi-hash rendezvous voting compiles each token identity to a fixed top-2 expert pair.
dense
There is no conditional load to distribute.
token-routed
The persisted table makes routing auditable in advance; runtime dispatch needs no learned gate or balancing loss.
dense
One monolithic path carries common and token-specific functions.
token-routed
A shared dense path remains contextual; routing adds narrow residual capacity.
dense
Uniform kernels are simple but activate every parameter in the block.
token-routed
Expert dispatch trades implementation complexity for conditional execution.
forward.py
Each layer persists a top-2 table compiled from multi-hash rendezvous voting. Token identity selects two narrow residual experts, but never replaces the shared contextual computation.
class TRHashEngineMLP(nn.Module):
def forward(self, x, token_ids):
expert_pair = self.route_table[:, token_ids] # compiled multi-hash top-2
shared = self.shared_expert(x)
routed = 2 * self.experts(x, expert_pair).mean(dim=0)
return shared + routed
LABO AI turns these primitives into typed cards, visible tensor flows and synchronized PyTorch.