ACT on the NexArm

Action Chunking with Transformers

ACT inference pipeline on the NexArm Each 480 by 640 camera frame goes through a shared ResNet18 to a 512 by 15 by 20 feature map, a 1 by 1 convolution and a fixed 2D sine position code, giving 300 tokens per camera; the joint state and the style latent z, which is zero at inference, become two more tokens; four post-norm encoder layers mix all 902 tokens; one decoder layer turns 100 zero-initialised query slots with learned positions into 100 actions through self-attention, cross-attention and a feed-forward block; a linear layer gives 100 by 6 normalised actions that the arm runs in full before observing again. ACT · one control cycle on the NexArm Read top to bottom · blocks follow lerobot/policies/act/modeling_act.py · click a block or use the arrow keys 1 · observe camera views I1​ … IV​ front wrist top (if recorded) 480 × 640 RGB · ImageNet mean / std · V = 2 or 3 joint state st​ j1 j2 j3 j4 j5 grip (s − μ) / σ dataset mean and std, per joint style latent z ∈ ℝ32​ z = 0 at inference in training it comes from the CVAE encoder (see the Training tab) ACT does not read the task text 2 · image backbone ResNet18 · ImageNet-1k weights one backbone for all cameras · FrozenBatchNorm · lr 1e-5 stem: conv 7×7 /2 · BN · ReLU · maxpool /2 64 × 120 × 160 layer1 · 2 basic blocks 64 × 120 × 160 layer2 · 2 basic blocks, /2 128 × 60 × 80 layer3 · 2 basic blocks, /2 256 × 30 × 40 layer4 · 2 basic blocks, /2 512 × 15 × 20 1 × 1 conv 512 → 512 · flatten 15 × 20 → 300 tokens 2D sine position p: 256 for y, 256 for x · fixed per camera: 300 × 512 tokens + their positions 3 · encoder input one token sequence 1D tokens get learned positions; image tokens keep their 2D sine code state: Linear 6 → 512 + learned position ps​ latent: Linear 32 → 512 + learned position pz​ tokens X z s front · 300 wrist · 300 top · 300 positions P pz​ ps​ 2D sine 2D sine 2D sine P is added to the queries and keys of every attention in the encoder, and to the keys of the decoder's cross-attention, never to the values X, P · (2 + 300V) × 512 = 902 × 512 with 3 cameras 4 · transformer encoder 4 × encoder layer, post-norm d 512 · 8 heads · FFN 3200 · ReLU · dropout 0.1 X P q, k v multi-head self-attention · 8 heads every token attends to all 902 LN FFN 512 → 3200 → 512, ReLU LN × 4 memory M · 902 × 512 5 · decoder and action head 1 × decoder layer, post-norm d 512 · 8 heads · FFN 3200 100 query slots, input x = 0 + learned position pdec​ · 100 × 512 decoder layer × 1 self-attn · q = k = x + pdec​ Add & LN cross-attn · k = M + P, v = M Add & LN → FFN 3200 → Add & LN queries are x + pdec​ in both attentions final LayerNorm Linear 512 → 6 â · 100 × 6, normalized 6 · execute un-normalize, run all 100, then observe a = σ ⊙ â + μ n_action_steps = 100 · no temporal ensembling after the 100th action: new frames, next cycle 100 actions at 30 Hz = 3.33 s without new observations
NexArm 3D view Loading the robot model…
Step 1 of 7

One control cycle

ACT is conditioned on images and joint state only; it does not read the task text. Each cycle it reads every camera once, encodes them together with the arm’s state, and predicts a chunk of 100 actions: joint and gripper targets for the next 3.33 seconds at 30 Hz. The NexArm then runs the whole chunk before the policy is called again.

← → steps · space play

  1. One control cycle

    ACT is conditioned on images and joint state only; it does not read the task text. Each cycle it reads every camera once, encodes them together with the arm’s state, and predicts a chunk of 100 actions: joint and gripper targets for the next 3.33 seconds at 30 Hz. The NexArm then runs the whole chunk before the policy is called again.

    \hat{A}_t = \pi_\theta\big(I_t^{1..V},\ s_t,\ z = 0\big) \in \mathbb{R}^{100 \times 6}

    V camera frames and the six-number joint state go in; z is the CVAE style latent, fixed to zero at inference. The output is 100 rows of five joint targets and a gripper target, in normalized units.

    images V × 3×480×640 · state 6 → chunk 100 × 6

  2. Observe

    The front Logitech, the wrist camera and, on the 3-camera dataset, the top RealSense each deliver one RGB frame at 480 × 640. The five arm joints plus the gripper give a six-number state. State and actions are standardized with the dataset’s mean and standard deviation; images use ImageNet statistics for the backbone.

    \tilde{s} = (s - \mu_s) \oslash \sigma_s
    \tilde{I} = (I - \mu_{\text{IN}}) \oslash \sigma_{\text{IN}}

    μs, σs are per-joint statistics computed over the training dataset; ⊘ is element-wise division.

    observation.images.{front,wrist,top} 3×480×640 · observation.state 6

  3. ResNet18 per image

    One ImageNet-initialized ResNet18 with frozen batch-norm statistics is shared by all cameras. Five stages of stride 2 leave a 512 × 15 × 20 feature map. A 1 × 1 convolution keeps 512 channels and the map is flattened into 300 tokens per camera. A fixed 2D sine code records each cell’s row and column: 256 channels for y, 256 for x.

    F = \operatorname{ResNet18}(\tilde I) \in \mathbb{R}^{512 \times 15 \times 20}
    X_{\text{img}} = \operatorname{flatten}\big(\operatorname{Conv}_{1 \times 1}(F)\big) \in \mathbb{R}^{300 \times 512}
    p_{y,\,2i} = \sin\!\Big(\frac{2\pi\, y / H}{10000^{2i/256}}\Big),\quad p_{y,\,2i+1} = \cos\!\Big(\frac{2\pi\, y / H}{10000^{2i/256}}\Big)

    y runs from 1 to H = 15 (and x from 1 to W = 20 in the same way); the y and x codes are concatenated into a 512-wide position. The backbone trains at lr 10-5.

    3×480×640 → 512×15×20 → 300 × 512 per camera

  4. Encoder input

    The latent and the joint state are each projected to 512 and get a learned position. They are placed in front of the image tokens of every camera to form one sequence. The positions are kept separate from the content: they are added to queries and keys inside attention, never to the values.

    X = \big[\,W_z z ;\ W_s \tilde s ;\ X_{\text{img}}^{1} ; \dots ;\ X_{\text{img}}^{V}\,\big]
    P = \big[\,p_z ;\ p_s ;\ P_{\text{2D}}^{1} ; \dots ;\ P_{\text{2D}}^{V}\,\big]
    N = 2 + 300\,V = 902 \quad (V = 3)

    Wz is 32 → 512 and Ws is 6 → 512; pz, ps are learned 512-vectors.

    X, P · 902 × 512 with 3 cameras

  5. Transformer encoder

    Four post-norm layers with 8 heads and a 3200-wide ReLU feed-forward let every token attend to every other, so a gripper pixel in the wrist view can be related to a bowl in the front view and to the current joint angles. The result is the memory the decoder reads.

    X \leftarrow \operatorname{LN}\Big(X + \operatorname{MHA}\big(q = X + P,\ k = X + P,\ v = X\big)\Big)
    X \leftarrow \operatorname{LN}\big(X + W_2\,\operatorname{ReLU}(W_1 X)\big)
    \operatorname{MHA}(q, k, v) = \big[\operatorname{softmax}\big(q W_q^{h} (k W_k^{h})^{\top} / \sqrt{64}\big)\, v W_v^{h}\big]_{h=1}^{8} W_o

    W1 is 512 → 3200, W2 is 3200 → 512; 8 heads of width 64; dropout 0.1. After 4 layers, M = X.

    902 × 512 → 4 × encoder layer → memory M 902 × 512

  6. Decoder and action head

    The decoder input is 100 zero vectors; only their learned positions tell the slots apart, one for each future step. A single post-norm layer runs self-attention among the slots, cross-attention into the encoder memory and a feed-forward block. A final LayerNorm and a linear layer map each slot to six numbers.

    x^{(0)} = 0 \in \mathbb{R}^{100 \times 512}
    x \leftarrow \operatorname{LN}\big(x + \operatorname{MHA}(x + p_{\text{dec}},\ x + p_{\text{dec}},\ x)\big)
    x \leftarrow \operatorname{LN}\big(x + \operatorname{MHA}(x + p_{\text{dec}},\ M + P,\ M)\big)
    x \leftarrow \operatorname{LN}\big(x + \operatorname{FFN}(x)\big),\quad \hat a = W_a \operatorname{LN}(x)

    pdec is a learned 100 × 512 table; Wa is 512 → 6.

    100 queries × 512 → Linear 512→6 → chunk 100 × 6

  7. Execute the whole chunk

    lerobot-nexarm keeps ACT’s default n_action_steps = 100 with no temporal ensembling: the arm follows all 100 targets open loop, one every 33 ms, then captures new frames. In the 3D view the dots ahead of the gripper are the part of the current chunk still to run; when they run out, the next cycle starts.

    a_k = \sigma_a \odot \hat a_k + \mu_a,\quad k = 1, \dots, 100
    \Delta t_{\text{replan}} = 100 \times \tfrac{1}{30}\,\text{s} \approx 3.33\,\text{s}
    100 actions × 33 ms = 3.33 s between observations
ACT training as a conditional VAE A batch of real demonstrations gives images, the joint state and the next 100 actions; a CVAE encoder reads a learned class token, the state and the 100 actions and outputs the mean and log variance of a 32 dimensional latent; a sample z is drawn with the reparameterization trick; the ACT network from the Inference tab predicts the chunk with z as its first token; the loss is the masked L1 error plus ten times the KL divergence to a unit Gaussian; AdamW updates all weights at a constant learning rate. ACT · one training step (conditional VAE) lerobot train with the ACT preset · 100,000 steps · click a block or use the arrow keys 1 · data thanhkt/nexarm_stack_bowls real teleoperated demonstrations · last 10 % of episodes held out images · V × 3 × 480 × 640 · ImageNet mean / std state s · 6, mean / std actions a · 100 × 6, mean / std + action_is_pad batch 2 × gradient accumulation 8 = 16 statistics come from the dataset, not the training split 2 · CVAE encoder (training only) posterior over the style latent reads the true actions, so it is dropped at inference CLS s a a a a a a a a a a… learned CLS · Linear 6 → 512 for s and each a · + fixed 1D sine positions (102) key-padding mask hides padded actions (hatched) 4 encoder layers · d 512 · 8 heads · FFN 3200 · post-norm CLS output → Linear 512 → 64 = [μ ; log σ²] μ, log σ² ∈ ℝ32​ 3 · sample z reparameterization trick z = μ + exp(½ log σ²) ⊙ ε, ε ~ N(0, I) gradients flow to μ and log σ² through z prior N(0, 1) q(z | s, a) one of 32 dimensions shown; at inference z = 0, the prior mean 4 · ACT network with z the network from the Inference tab z replaces the zero latent as the first encoder token ResNet18 × V cameras encoder × 4 [z, s, 300V image] decoder × 1 100 queries Linear 512→6 â · 100 × 6 dropout 0.1 everywhere · ImageNet-initialized backbone, BatchNorm frozen lr 1e-5 for the backbone and for the rest (two parameter groups) 5 · loss masked L1 + β · KL, β = 10 j1 j2 j3 j4 j5 grip |â − a| over 100 steps (every 4th drawn), padded steps masked L1 = Σ mask · |â − a| / (6 · valid steps) KL = −½ Σj​ (1 + log σj​² − μj​² − σj​²), mean over batch the KL keeps q(z | s, a) near N(0, I), so z = 0 is a typical style 6 · optimizer step backward × 8 clip ‖g‖ ≤ 10 AdamW step zero grad AdamW · lr 1e-5 · weight decay 1e-4 no scheduler: the rate stays constant step 100k lr 1e-5 100,000 steps · checkpoint and held-out L1 every 5,000 held-out L1 does not measure stacking success next batch
Step 1 of 7

One training step

ACT is trained as a conditional VAE. Besides the policy, a second transformer reads the true action chunk and summarizes it as a latent z. The policy learns to reconstruct the chunk from the images, the state and z, and a KL term keeps z close to a unit Gaussian so it can be set to zero at inference.

← → steps · space play

  1. One training step

    ACT is trained as a conditional VAE. Besides the policy, a second transformer reads the true action chunk and summarizes it as a latent z. The policy learns to reconstruct the chunk from the images, the state and z, and a KL term keeps z close to a unit Gaussian so it can be set to zero at inference.

    \min_{\theta, \phi}\ \mathbb{E}\Big[\,\lVert \pi_\theta(o, z) - A \rVert_1 + \beta\, D_{\mathrm{KL}}\big(q_\phi(z \mid s, A)\ \Vert\ \mathcal{N}(0, I)\big)\Big]

    qφ is the CVAE encoder, z ∼ qφ, o the observation, A the recorded 100-step chunk and β = 10.

  2. Sample a batch

    Samples come from the real teleoperated recordings. Each one is an observation at time t and the 100 actions that followed, with a pad flag for steps past the end of the episode. The batch is only 2 to fit an 8 GB GPU, so 8 batches are accumulated before each optimizer step.

    B_{\text{eff}} = 2 \times 8 = 16
    \tilde a_k = (a_k - \mu_a) \oslash \sigma_a,\quad m_k = 1 - \text{is\_pad}_k

    The last 10 % of episodes of each task are held out for the validation loss.

    images 2 × V × 3 × 480 × 640 · state 2 × 6 · actions 2 × 100 × 6

  3. CVAE encoder

    A four-layer transformer, the same width as the main encoder, reads a [CLS] token, the projected state and the 100 projected ground-truth actions, with a fixed 1D sine position code and a key-padding mask over padded actions. A linear layer on the [CLS] output gives the mean and log-variance of z.

    h = \operatorname{Enc}_\phi\big([\,\text{CLS} ;\ W_s \tilde s ;\ W_a \tilde a_{1..100}\,] + p_{\text{1D}}\big)
    [\,\mu ;\ \log\sigma^2\,] = W_{\text{lat}}\, h_{\text{CLS}} \in \mathbb{R}^{64}

    102 tokens of width 512; Wlat is 512 → 64, split into two 32-vectors.

  4. Reparameterize

    The latent is sampled with the reparameterization trick so the gradient can flow back into μ and σ. At inference the encoder is not run and z is set to the prior’s mean, zero.

    z = \mu + \sigma \odot \varepsilon,\quad \varepsilon \sim \mathcal{N}(0, I),\quad \sigma = \exp\big(\tfrac{1}{2}\log\sigma^2\big)
    z_{\text{inference}} = 0

  5. Policy forward pass

    z becomes the first token of the main encoder, exactly where the zero latent sits at inference. The rest of the network is the one in the Inference tab: ResNet18 tokens, four encoder layers and the one-layer decoder.

    \hat A = \pi_\theta(\tilde I, \tilde s, z) \in \mathbb{R}^{100 \times 6}

  6. L1 + KL loss

    The reconstruction term is the absolute error over valid chunk steps, averaged per element. The KL term between the encoder’s Gaussian and the unit Gaussian is summed over the 32 latent dimensions, averaged over the batch and weighted by 10.

    \mathcal{L}_{\text{L1}} = \frac{\sum_{k, j} m_k \lvert \hat a_{kj} - \tilde a_{kj} \rvert}{6 \sum_k m_k}
    D_{\mathrm{KL}} = -\tfrac{1}{2} \sum_{d=1}^{32} \big(1 + \log\sigma_d^2 - \mu_d^2 - \sigma_d^2\big)
    \mathcal{L} = \mathcal{L}_{\text{L1}} + 10\, D_{\mathrm{KL}}

  7. Optimizer step

    After 8 accumulated batches the gradients are clipped to a global norm of 10 and AdamW updates every weight, including the backbone, at a constant learning rate. There is no warm-up, no schedule and no weight averaging. Every 5,000 steps a checkpoint is saved and the held-out loss is computed.

    g = \frac{1}{8} \sum_{b=1}^{8} \nabla_\theta \mathcal{L}_b,\quad g \leftarrow g \cdot \min\!\big(1,\ 10 / \lVert g \rVert_2\big)
    \theta \leftarrow \operatorname{AdamW}(\theta, g;\ \eta = 10^{-5},\ \lambda = 10^{-4})

    100,000 optimizer steps in total.

A CVAE-trained transformer that turns the NexArm's camera frames and joint state into 100 future joint targets, then runs all of them before it looks again.

  • 100 · actions per chunk
  • 100 · actions run before re-planning
  • 30 Hz · control rate
  • 902 · encoder tokens (3 cameras)
  • ResNet18 · backbone
  • 100k · training steps

Architecture

ACT predicts actions in chunks instead of one at a time. Predicting 100 steps at once means a single mistake in one prediction is not compounded at every step, and it lets the policy represent a smooth motion rather than a series of corrections. The cost is that the arm moves without new observations for the length of the executed part of the chunk; on the NexArm that is the full 3.33 seconds.

Part Setting in lerobot-nexarm
Vision backbone ResNet18, ImageNet-1k weights, shared across cameras
Image tokens 1×1 conv to 512, 2D sine position embedding, 300 per 480×640 view
Encoder 4 layers, d_model 512, 8 heads, FFN 3200, ReLU, post-norm
Decoder 1 layer, 100 learned queries
Action head Linear 512 → 6
Latent 32-dimensional CVAE latent, KL weight 10
Normalization mean / std for state and actions, ImageNet stats for images
Chunk / executed 100 / 100 actions, no temporal ensembling

Training

The model is trained as a conditional VAE. A second, smaller transformer reads the current joint state and the true next 100 actions and compresses them into a 32-dimensional latent z. That latent becomes the first token of the main encoder, so the network can tell apart demonstrations that look the same but were performed differently. The KL term keeps z close to a unit Gaussian, which is why it can be replaced by zeros at inference.

Setting Value
Data Real teleoperated demonstrations, thanhkt/nexarm_stack_bowls
Held-out split last 10% of episodes per task
Effective batch 16 (batch 2 × 8 gradient-accumulation steps, sized for an 8 GB GPU)
Optimizer AdamW, lr 1e-5 for the transformer and the backbone, weight decay 1e-4
Stability gradient clipping at 10, constant learning rate
Steps 100,000, checkpoint and held-out loss every 5,000
Loss L1 on actions, masked past the episode end, + 10 · KL

Camera sets must match. The default dataset has front and wrist cameras only; a 3-camera recording adds the top view. A policy trained on two cameras needs --no-top-cam at rollout, and rollout.py stops if the checkpoint and the robot disagree.

Evaluation status

Not yet measured

No physical success rate has been published yet. Training reports only the L1 loss on held-out episodes, which does not say whether the arm actually stacks the bowls. The table and protocol below are the planned evaluation, not results.

Metric Value How it is measured
Stacking success rate — Fraction of trials that end with the bowls nested and released, judged from the front camera.
Sub-task success — Separate counts for grasp, lift, place and release, so a failure can be located.
Time to stack — Seconds from the first action to the gripper opening over the stack, successful trials only.
Held-out L1 — Mean L1 over normalized actions on the last 10% of episodes, logged every 5,000 steps.

Protocol

  1. Fixed bowl start poses drawn from the recorded workspace, the same list for every policy.
  2. Each trial starts from the robot’s ready pose with examples/nexarm/rollout.py and the checkpoint’s own camera set.
  3. A trial ends at success, at a collision, or after 60 seconds.

Code and data

3D model: NexArm MJCF and meshes from khanhthanhdev/lerobot-nexarm, rendered with three.js. The motion is a scripted bowl-stacking demo solved with inverse kinematics in the browser; the dots show how long one ACT action chunk is, not a recorded policy rollout.