TurboVLA on the NexArm

TurboVLA, a compact vision-language-action model

TurboVLA inference pipeline on the NexArm Camera views go through DINOv3 ViT-B/16 and a residual projection into 196 tokens of width 256 per view; the instruction goes through a frozen BERT and a linear layer; six interaction layers share one attention-logit matrix between image and text tokens and refine the text with a transformer layer; the joint state becomes two tokens; sixteen learned queries cross-attend to all of these in three pre-norm decoder layers; an MLP and tanh give a 16 by 6 chunk, of which the arm runs the first 8 before inferring again. TurboVLA · one control cycle on the NexArm Read top to bottom · blocks follow TurboVLA/turbovla/models · click a block or use the arrow keys 1 · observe camera views I1​ … IV​ front wrist top (if recorded) 224 × 224 RGB · ImageNet mean / std · V = 2 or 3 instruction ℓ "stack the bowls" the dataset's task string; an empty task falls back to a default prompt joint state st​ j1 j2 j3 j4 j5 grip min-max → [-1, 1] ranges saved in stats_turbovla.json 2 · vision encoder DINOv3 ViT-B/16 each view separately · bf16 autocast · fine-tuned, lr 5e-5 patch embed 16 × 16 → CLS + 4 registers + 196 patches 12 transformer blocks · d 768 · 12 heads · RoPE drop the 5 prefix tokens → 196 × 768 VisionProjection LN 768→1024 GELU · drop 1024→256 LN Wskip​ 768 → 256, no bias + view embedding efront​, ewrist​, etop​ → flatten 3 · language encoder BERT-base uncased frozen, kept in eval mode WordPiece · up to 256 tokens 12 encoder layers · d 768 Linear 768 → 256 · Xavier init sub-sentence mask split at [CLS] [SEP] . ? tokens see only their own sentence padding → key-padding mask l · L × 256 5 · state tokens StateProjection joint state → two memory tokens LayerNorm(6) Linear 6 → 256 GELU · Dropout 0.1 Linear 256 → 512 reshape → 2 × 256 + learned position, 2 × 256 LayerNorm(256) s · 2 × 256 4 · vision ↔ language interaction 6 × interaction layer BiAttentionBlock with shared logits, then a text-only transformer layer LN LN v̄ l̄ Wq​ v̄ Wk​ l̄ S = (Wq​ v̄)(Wk​ l̄)ᵀ / √dh​ one logit matrix · 4 heads · dh​ 256 softmax over text · values Wvl​ l̄ Wo​v​ · LayerScale γv​ · DropPath 0.1 softmax over image · values Wvv​ v̄ Wo​l​ · LayerScale γl​ · DropPath 0.1 v̄ l̄ text transformer layer, post-norm self-attn · 4 heads · sentence mask Add & LN → FFN 1024, ReLU → Add & LN v′ l′ × 6 after layer 6: concat [v′ ; l′] → (196V + L) × 256 6 · action decoder ACT-style decoder, no CVAE 3 layers · pre-norm · d 256 · 8 heads memory m = [v′ ; l′ ; s] (196V + L + 2) × 256 16 learned queries q · 16 × 256 decoder layer × 3, norm_first LN → self-attention → + LN → cross-attention to m → + LN → FFN 2048, ReLU → + dropout 0.1 · no final LayerNorm MLP 256 → 512 → 512 → 6, ReLU tanh → â ∈ [-1, 1] â · 16 × 6 7 · execute un-normalize, run 8 of 16, infer again a = (â + 1) / 2 · (max − min) + min rollout_turbovla.py --open-loop-steps 8 after the 8th action: new frames, next cycle 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 run at 30 Hz · 8 × 33 ms ≈ 0.27 s dropped, replaced by the next chunk
NexArm 3D view Loading the robot model…
Step 1 of 8

One control cycle

TurboVLA reads the cameras, the joint state and the task instruction. DINOv3 encodes the images, a frozen BERT encodes the text, six interaction layers let each modality attend to the other, and a small decoder predicts 16 future actions. Only the first 8 are run, about a quarter of a second, before the policy sees the scene again.

← → steps · space play

  1. One control cycle

    TurboVLA reads the cameras, the joint state and the task instruction. DINOv3 encodes the images, a frozen BERT encodes the text, six interaction layers let each modality attend to the other, and a small decoder predicts 16 future actions. Only the first 8 are run, about a quarter of a second, before the policy sees the scene again.

    \hat{A}_t = \pi_\theta\big(I_t^{1..V},\ \ell,\ s_t\big) \in [-1, 1]^{16 \times 6}

    V camera views, the instruction ℓ and the six-number joint state go in; one forward pass gives 16 rows of five joint targets and a gripper target, still in normalized units.

    images V × 3×224×224 · text L tokens · state 6 → chunk 16 × 6

  2. Observe

    Every camera frame is resized to 224 × 224 by the DINOv3 image processor and normalized with ImageNet statistics. Views are ordered front, wrist, top; the top camera is used only when the dataset recorded it, and the order is saved in the checkpoint’s config. The joint state is min-max scaled to [-1, 1] with ranges saved at training time.

    \tilde{s} = 2\,\frac{s - s_{\min}}{s_{\max} - s_{\min}} - 1
    \tilde{I} = \big(\operatorname{resize}_{224}(I) - \mu_{\text{IN}}\big) \oslash \sigma_{\text{IN}}

    smin, smax come from stats_turbovla.json; μIN, σIN are the per-channel ImageNet mean and standard deviation.

    V = 2 or 3 views · state 6 in [-1, 1]

  3. DINOv3 vision encoder

    Each view goes through DINOv3 ViT-B/16 on its own, in bf16. A 224-pixel image is 14 × 14 patches of 16 pixels; the class token and the four register tokens are dropped, so every view gives 196 tokens of width 768. A residual MLP projects them to 256 and a learned view embedding says which camera they came from. The backbone is fine-tuned.

    x = \operatorname{DINOv3}(\tilde I)_{[5:]} \in \mathbb{R}^{196 \times 768}
    z = \operatorname{LN}\big(W_{\text{skip}}\,x + W_2\,\operatorname{GELU}(W_1 \operatorname{LN}(x))\big) + e_{\text{view}}
    N_v = V \cdot (224 / 16)^2 = 196\,V

    W1 is 768 → 1024 (dropout 0.1 after the GELU), W2 is 1024 → 256 and Wskip is 768 → 256 without bias. eview is one learned 256-vector per camera.

    V × 196 × 768 → V × 196 × 256 → 588 × 256 with 3 views

  4. BERT language encoder

    The instruction, for example “stack the bowls”, is tokenized with WordPiece and encoded by BERT-base, which stays frozen in eval mode. A linear layer maps its 768-wide outputs to 256. The tokenizer’s special tokens and punctuation split the text into sentences, and a block mask keeps each token’s later self-attention inside its own sentence.

    l = W_t\,\operatorname{BERT}(\ell) \in \mathbb{R}^{L \times 256}
    M_{ij} = \begin{cases} \text{attend} & i, j \text{ in the same sentence} \\ \text{masked} & \text{otherwise} \end{cases}

    Wt is 768 → 256 with Xavier initialization; sentences are delimited by [CLS], [SEP], “.” and “?”. Padding tokens are also removed from the image-to-text attention.

    L × 768 → Linear → L × 256

  5. Vision ↔ language interaction

    Six layers fuse the two streams. A bi-attention block computes one matrix of attention logits between image and text tokens and uses it in both directions: softmax over the text for the image update, softmax over the images for the text update. Residuals start from the layer-normed input, and each update is scaled by a LayerScale γ that starts at 10-4, so a fresh layer is close to the identity. A text-only transformer layer then refines the text with the sentence mask.

    \bar v = \operatorname{LN}(v),\quad \bar l = \operatorname{LN}(l),\quad S = \frac{(W_q \bar v)(W_k \bar l)^{\top}}{\sqrt{d_h}}
    v \leftarrow \bar v + \operatorname{DropPath}\big(\gamma_v \odot W_o^{v}\,\operatorname{softmax}_{\text{text}}(S)\,W_{vl}\,\bar l\big)
    l \leftarrow \bar l + \operatorname{DropPath}\big(\gamma_l \odot W_o^{l}\,\operatorname{softmax}_{\text{img}}(S^{\top})\,W_{vv}\,\bar v\big)
    l \leftarrow \operatorname{TransformerLayer}(l;\ M)

    4 heads with dh = 256 (inner width 1024), DropPath 0.1. The text layer is post-norm with 4 heads and a 1024-wide ReLU feed-forward. After layer 6 the streams are concatenated.

    (588 + L) × 256 → 6 × interaction → (588 + L) × 256

  6. State tokens

    The normalized joint state does not take part in the interaction. A small MLP turns it into two 256-wide tokens, each with its own learned position, and they are appended to the image and text tokens to form the decoder’s memory.

    [\,s^{(1)} ;\ s^{(2)}\,] = \operatorname{reshape}\big(W_b\,\operatorname{GELU}(W_a \operatorname{LN}(\tilde s))\big) + [\,p_1 ;\ p_2\,]
    m = \big[\,v ;\ l ;\ \operatorname{LN}(s^{(1)}) ;\ \operatorname{LN}(s^{(2)})\,\big]

    Wa is 6 → 256 (dropout 0.1 after the GELU), Wb is 256 → 512, reshaped to 2 × 256.

    6 → 2 × 256 · memory (588 + L + 2) × 256

  7. ACT-style action head

    Sixteen learned queries, one per future step, pass through three pre-norm decoder layers (d 256, 8 heads, feed-forward 2048, dropout 0.1): self-attention among the queries, cross-attention into the memory, then a feed-forward block. A three-layer MLP maps each query to six numbers and tanh keeps them in [-1, 1]. There is no CVAE: the output is one deterministic chunk.

    q \leftarrow q + \operatorname{SelfAttn}(\operatorname{LN}(q))
    q \leftarrow q + \operatorname{CrossAttn}(\operatorname{LN}(q),\ m)
    q \leftarrow q + \operatorname{FFN}(\operatorname{LN}(q))
    \hat a = \tanh\big(\operatorname{MLP}_{256 \to 512 \to 512 \to 6}(q)\big)

    The three updates repeat for each of the 3 layers. The MLP uses ReLU between its layers; there is no final LayerNorm after the last decoder layer.

    16 queries × 256 → 3 decoder layers → MLP → tanh → 16 × 6

  8. Run 8, then infer again

    The min-max statistics turn the chunk back into joint and gripper targets. rollout_turbovla.py runs the first 8 (--open-loop-steps 8) at 30 Hz and throws the other 8 away, then captures new frames and runs the whole network again. In the 3D view, the dots ahead of the gripper are the actions still to run before the next inference.

    a = \tfrac{1}{2}(\hat a + 1)\,(a_{\max} - a_{\min}) + a_{\min}
    \Delta t_{\text{replan}} = 8 \times \tfrac{1}{30}\,\text{s} \approx 0.27\,\text{s}
    8 × 33 ms = 0.27 s between observations
TurboVLA training loop in lerobot-nexarm Simulated and real demonstrations are mixed by a sampler with a fixed real ratio; the collator normalizes images, state and 16-step action chunks with padding flags; the model predicts a 16 by 6 chunk; a masked L1 loss with an optional gripper weight is computed over valid steps; AdamW with gradient clipping and a warm-up plus cosine schedule updates the weights; an exponential moving average of the weights is saved with the statistics and config every 5,000 steps. TurboVLA · one training step examples/nexarm/train_turbovla.py · 50,000 steps · click a block or use the arrow keys 1 · data simulation MuJoCo scene, scripted stacking randomized lights, colours, poses real thanhkt/nexarm_stack_bowls teleoperated on the arm DomainMixSampler each sample: real with p = r, sim with 1 − r · with replacement --real-ratio 0.5 · --no-sim trains on real data only · batch 16 per GPU 2 · collate and normalize images → DINOv3 processor → B × V × 3 × 224 × 224 state, actions → 2 (x − min) / (max − min) − 1 min / max merged over the sim and real statistics instruction strings, kept as text for BERT action chunk, 16 steps · action_is_pad marks steps past the episode end 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 padded 3 · forward pass the network from the Inference tab autocast in bf16 when --precision bf16 DINOv3 trainable BERT frozen, no grad 6 × interaction trainable decoder + MLP trainable â = tanh(MLP(decoder(q, m))) · 16 × 6 state projection and text Linear 768 → 256 are trainable too 4 · masked L1 loss |â − a| per step and joint padded steps count zero; the gripper column can be re-weighted j1 j2 j3 j4 j5 grip × w5​ loss = Σ mask · w · |â − a| / (6 · valid steps) --loss-type l1 (default) · mse · smooth_l1 · --gripper-loss-weight 5 · optimizer step backward clip ‖g‖ ≤ 1.0 AdamW step LambdaLR step parameter groups action head · lr 5e-5 interaction, projections · lr 5e-5 DINOv3 · lr 5e-5 (--dinov3-lr) BERT · frozen, not in the optimizer weight decay 1e-10 · no accumulation by default GradScaler only for fp16 step 50k lr warm-up 1k → 5 % linear warm-up, then cosine to 5 % of the base rate 6 · EMA and checkpoint moving average of the weights θ̄ ← 0.999 θ̄ + 0.001 θ after every optimizer step every 5,000 steps, written to the run folder {step}_ema_model.safetensors {step}_model.pt (raw weights) config.json (cameras, image size, fps, task) stats_turbovla.json (min / max) rollout loads the EMA weights next batch
Step 1 of 7

One training step

train_turbovla.py repeats this loop for 50,000 optimizer steps: draw a batch from simulation and real data, normalize it, predict a 16-step chunk, score it with a masked L1 loss, update the weights and fold them into a moving average. The average is what gets saved and run on the arm.

← → steps · space play

  1. One training step

    train_turbovla.py repeats this loop for 50,000 optimizer steps: draw a batch from simulation and real data, normalize it, predict a 16-step chunk, score it with a masked L1 loss, update the weights and fold them into a moving average. The average is what gets saved and run on the arm.

    \theta^{*} = \arg\min_\theta\ \mathbb{E}_{(o, A) \sim \mathcal{D}_{\text{mix}}}\ \mathcal{L}\big(\pi_\theta(o), A\big)

    o is the observation (images, instruction, state), A the recorded 16-step action chunk, and 𝒟mix the sim/real mixture.

  2. Mix simulation and real data

    Scripted bowl-stacking demonstrations from the MuJoCo scene, with randomized lighting, colours and poses, are concatenated with the real teleoperated set. DomainMixSampler draws each sample from the real set with probability r, independently of how large each set is, so the smaller real set is not drowned out.

    i \sim \begin{cases} \mathcal{U}(\mathcal{D}_{\text{real}}) & \text{with probability } r \\ \mathcal{U}(\mathcal{D}_{\text{sim}}) & \text{with probability } 1 - r \end{cases}

    --real-ratio 0.5 by default; --no-sim trains on the real set alone. Indices are drawn with replacement from a generator seeded the same on every GPU and then split by rank.

    batch 16 per GPU · no gradient accumulation

  3. Collate and normalize

    The collator runs the DINOv3 image processor on every view, scales the state and the 16-step action chunk to [-1, 1], and keeps the action_is_pad flags for chunk steps that run past the end of an episode. The min-max ranges are merged across the two domains, so simulation and the real arm share one coordinate system.

    \tilde a_k = 2\,\frac{a_k - a_{\min}}{a_{\max} - a_{\min}} - 1,\quad k = 1, \dots, 16
    a_{\min} = \min\big(a_{\min}^{\text{sim}}, a_{\min}^{\text{real}}\big),\quad a_{\max} = \max\big(a_{\max}^{\text{sim}}, a_{\max}^{\text{real}}\big)
    m_k = 1 - \text{is\_pad}_k
    pixels B × V × 3 × 224 × 224 · state B × 6 · actions B × 16 × 6 · mask B × 16

  4. Forward pass

    The batch goes through the same network as in the Inference tab. BERT is frozen and receives no gradient; DINOv3, the projections, the interaction layers and the action head are all trained. With --precision bf16 the forward pass runs under autocast.

    \hat A = \pi_\theta\big(I, \ell, \tilde s\big) \in [-1, 1]^{16 \times 6}
    \theta = \theta_{\text{DINOv3}} \cup \theta_{\text{proj}} \cup \theta_{\text{interaction}} \cup \theta_{\text{head}}
    â · B × 16 × 6

  5. Masked L1 loss

    The absolute error is taken per chunk step and per joint. Padded steps are zeroed and the sum is divided by the number of valid entries, so a chunk cut short at the end of an episode is not penalized for steps that do not exist. --gripper-loss-weight scales the gripper column when the open/close timing matters more.

    \mathcal{L} = \frac{\sum_{k=1}^{16} \sum_{j=1}^{6} m_k\, w_j\, \lvert \hat a_{kj} - \tilde a_{kj} \rvert}{6 \sum_{k} m_k}
    w_j = 1 \ \text{for the five joints},\quad w_6 = w_{\text{grip}}

    --loss-type mse or smooth_l1 replace the absolute error with a squared or Huber error; the masking and the normalization stay the same.

  6. Optimizer step

    Gradients are clipped to a global norm of 1.0, then AdamW updates three parameter groups: the action head, the remaining trainable modules, and DINOv3, each at 5 × 10-5. The learning rate ramps up linearly over the first 1,000 steps and then follows a cosine down to 5 % of its base value at step 50,000.

    g \leftarrow g \cdot \min\!\big(1,\ 1 / \lVert g \rVert_2\big)
    \eta(u) = \eta_0 \begin{cases} \dfrac{u + 1}{1000} & u < 1000 \\[4pt] 0.05 + 0.95\,\dfrac{1 + \cos(\pi p)}{2} & \text{otherwise} \end{cases}
    p = \frac{u - 1000}{49\,000},\quad \eta_0 = 5 \times 10^{-5}

    u counts optimizer updates. Weight decay is 10-10, effectively off.

  7. Moving average and checkpoint

    After every update the script folds the new weights into an exponential moving average. The average changes slowly, which smooths out the noise of individual batches. Every 5,000 steps it writes the EMA weights, the raw weights, config.json (cameras, image size, fps, task) and stats_turbovla.json; the rollout script loads the EMA weights.

    \bar\theta \leftarrow 0.999\,\bar\theta + 0.001\,\theta
    \text{averaging window} \approx \frac{1}{1 - 0.999} = 1000 \text{ updates}

DINOv3 image tokens and a BERT-encoded instruction are fused by six bi-attention layers, and an ACT-style decoder predicts 16 actions, of which the NexArm runs 8 before looking again.

  • 16 · actions per chunk
  • 8 · actions run before re-planning
  • 30 Hz · control rate
  • 588 · vision tokens (3 cameras)
  • DINOv3 ViT-B/16 · backbone
  • 50k · training steps

Architecture

TurboVLA keeps ACT’s idea of decoding a chunk of actions from learned queries, but replaces the encoder side. Images are encoded by a large self-supervised vision transformer, the task arrives as text, and the two are fused before the action head reads them. The chunk is short (16 actions) and only half of it is run, so the policy re-plans about every quarter second.

Part Setting used for the NexArm
Vision DINOv3 ViT-B/16, 224 × 224, views encoded separately, trainable
Vision projection LN → MLP 768 → 1024 → 256 + linear skip, LN, + view embedding
Language BERT-base uncased, frozen, Linear 768 → 256
Interaction 6 × (shared-logit bi-attention, 4 heads + text transformer)
State LN → MLP 6 → 256 → 512 → 2 tokens + learned positions
Action decoder 16 queries, 3 pre-norm layers, d 256, 8 heads, FFN 2048
Output MLP 256 → 512 → 512 → 6, tanh, min-max to joint units
Chunk / executed 16 / 8 actions

Training

The training script can mix two sources: scripted bowl-stacking demonstrations generated in the MuJoCo scene with domain randomization, and real teleoperated demonstrations. With --real-ratio 0.5 a sampler draws each training sample from the real set with probability one half, so the smaller real set is not drowned out by simulation.

Setting Default in train_turbovla.py
Data sim + real, or either alone (--no-sim for real only)
Batch 16 per device, no accumulation
Optimizer AdamW, lr 5e-5 for the head, interaction and DINOv3
Schedule 1,000 warm-up steps, cosine to 5 %, 50,000 steps, save every 5,000
Loss L1 over non-padded chunk steps, optional --gripper-loss-weight
Stability gradient clipping at 1.0, EMA of weights with decay 0.999
Starting point from scratch, or the released RoboTwin checkpoint

The checkpoint carries its own contract. config.json stores the camera keys in order, the image size, the fps and the default task prompt, and stats_turbovla.json stores the min-max ranges. The rollout script reads both, so a checkpoint trained without the top camera is run without it.

Evaluation status

Not yet measured

No physical success rate has been published yet. The training script logs the masked L1 loss, which does not say whether the arm actually stacks the bowls. The table and protocol below are the planned evaluation, not results.

Metric Value How it is measured
Stacking success rate — Fraction of trials that end with the bowls nested and released, judged from the front camera.
Sub-task success — Separate counts for grasp, lift, place and release, so a failure can be located.
Sim-to-real gap — Success in the MuJoCo scene minus success on the real arm for the same checkpoint and start poses.
Effect of co-training — Real-arm success with --real-ratio 0.5 against a real-data-only run of the same length.

Protocol

  1. Fixed bowl start poses drawn from the recorded workspace, the same list for every policy.
  2. Each trial starts from the robot’s ready pose with examples/nexarm/rollout_turbovla.py and the EMA checkpoint.
  3. A trial ends at success, at a collision, or after 60 seconds.

Code and data

3D model: NexArm MJCF and meshes from khanhthanhdev/lerobot-nexarm, rendered with three.js. The motion is a scripted bowl-stacking demo solved with inverse kinematics in the browser; the dots show how long one TurboVLA action chunk is, not a recorded policy rollout.