BSP on the NexArm

Inference · 7 steps

1. 01

   One control cycle

   B-spline Policy (BSP) keeps the network from the ACT page and changes what comes out of it. Instead of a list of future joint targets, the head predicts the parameters of a curve: 16 knots and their control points, which define one cubic B-spline segment. The controller samples that curve at 100 Hz. It can also move along it n times faster, and when the next segment arrives it is joined at the point that matches where the arm actually is. This page follows the paper’s “Reg.+BSP” setup, the ACT-style variant; it has not been trained on the NexArm yet.

   [\hat U ;\ \hat C] = \pi_\theta\big(I_t^{1..V},\ s_t\big) \in \mathbb{R}^{16 \times (1 + 6)}

   a(u) = \sum_i N_{i,3}(u)\, c_i

   Û are 16 knot values and Ĉ the matching control points, one six-number joint target each. N_i,3 are the cubic B-spline basis functions those knots define.

   images V × 3×H×W · state 6 → segment 16 × (1 knot + 6)
2. 02

   Observe

   The inputs are the same as for ACT: one frame per camera and the six-number joint state. The paper uses absolute joint positions as actions on its real arms, which matches the NexArm’s joint-target interface. Two clocks run at once. The policy produces a new segment about 10 times a second, and the controller sends a command 100 times a second by sampling the current segment.

   f_{\text{policy}} = 10\ \text{Hz},\qquad f_{\text{control}} = 100\ \text{Hz}

   The next segment is inferred asynchronously while the arm is still running the current one.

   observation.images.* · observation.state 6
3. 03

   ACT-style network

   The backbone is a regression transformer like the one on the ACT page. In the paper’s real-robot runs the image encoder is DINOv2. The head now outputs 16 knot values and their 6-D control points instead of 16 actions. The paper does not say how this maps onto the query slots; the figure shows one natural layout, one knot and one control point per slot. Either way the output is a fixed-size tensor, so a chunking backbone can take it with little change.

   \hat y_k = W_{\text{out}}\, q_k \in \mathbb{R}^{7},\qquad k = 1, \dots, 16

   [\hat u_k ;\ \hat c_k] = \hat y_k

   q_k is the decoder output for slot k; the first number is the knot, the other six the control point.

   16 query slots → Linear → 16 × 7
4. 04

   Knots and control points

   The knots set where in time each piece of the curve begins and ends, and they do not have to be evenly spaced. The control points set where the curve goes: it follows them without passing through them. A cubic segment needs neighbouring knots for support, and 6 of the 16 are boundary-support knots; the inner ones bound the part that runs. The paper gives only the total, so the figure draws them as 3 on each side, the usual split for a cubic. A regressor does not guarantee ordered knots, so any knot that would go backwards is pushed just past its neighbour.

   u_i \leftarrow \max(u_i,\ u_{i-1} + \delta)

   δ is a small positive constant. In practice the paper finds that predicted knots are almost always in order already.

   U 16 non-decreasing · C 16 × 6
5. 05

   A weighted sum with local support

   Each control point is weighted by a basis function that is non-zero over only four knot spans. Moving one control point therefore reshapes only that part of the curve, which the lower plot shows. An error in one predicted parameter bends one piece and leaves the rest of the segment unchanged. Cubic pieces also meet with continuous velocity and acceleration, so the commanded motion has no corners.

   N_{i,0}(u) = \begin{cases} 1 & u_i \le u < u_{i+1} \\ 0 & \text{otherwise} \end{cases}

   N_{i,p}(u) = \frac{u - u_i}{u_{i+p} - u_i} N_{i,p-1}(u) + \frac{u_{i+p+1} - u}{u_{i+p+1} - u_{i+1}} N_{i+1,p-1}(u)

   The Cox–de Boor recursion, with p = 3. N_i,3 is zero outside [u_i, u_i+4).

   a(u) ∈ ℝ⁶ for any u in the segment
6. 06

   Sample at 100 Hz, run n× faster

   The curve is defined for every u, not only at the recorded steps. The controller can therefore sample it as densely as it likes, and it can speed the motion up by scaling time. At 2× the arm follows the same path in half the time with half as many samples. The policy is not retrained for each speed. The limit is the hardware: at 4× the paper’s low-cost arms could no longer track the bimanual Speed Stacking task.

   a_{\text{exec}}(t) = a(n\,t)

   n is the speedup factor and t is real time since the segment started.

   1× · 100 samples per second of curve · n× · 100 / n
7. 07

   Join the next segment smoothly

   By the time a new segment arrives, the arm has moved on, and noise and tracking error mean the new curve does not start exactly where the old one is. Switching at a fixed time would make the command jump. BSP instead searches the start of the new segment for the point closest to the last executed action, and continues from there. Without this step, Speed Stacking success at 4× falls from 35 % to 15 % in the paper’s ablation, which uses the Diffusion backbone.

   t^{\star} = \arg\min_{0 \le t \le \lambda T_{\text{inf}}} \operatorname{MSE}\big(S_{\text{new}}(t),\ a_{\text{last}}\big)

   T_inf is the measured inference latency and λ > 1 widens the window. On the real arms the window is half the segment: λT_inf = u_max(S) / 2.

   no jump at segment boundaries

Training · 7 steps

1. 01

   One training step

   Training is ordinary behaviour cloning with different targets. Every demonstration is first converted into a B-spline with an adaptive fit. At each observation the target is the next 16 knots and their control points, and the network regresses them. The figures use a toy single-joint trajectory; the fitting, the windows and the basis functions are computed from it, not drawn by hand.

   \min_\theta\ \mathbb{E}_{(o, U, C)}\ \mathcal{L}\big(\pi_\theta(o),\ [U ; C]\big)

   The paper does not state the loss ℒ for the regression variant; ACT’s L1 is the natural drop-in.

2. 02

   Demonstrations

   A demonstration is a list of joint targets at a fixed rate. The paper records 200 to 300 teleoperated demonstrations per real task. A pick-and-place has fast, coarse parts (reach, carry, retreat) and slow, precise parts (grasp, place). A fixed-rate chunk spends the same resolution on both.

   toy · one joint · 50 Hz · 8 s · 401 samples
3. 03

   Adaptive B-spline fit

   Algorithm 1 of the paper starts with no interior knots and fits a cubic B-spline by least squares. It checks the largest error and, if it is above the tolerance, inserts a knot where the residual is largest (the FITPACK criterion), then repeats. Knots end up where the motion bends and are sparse where it is simple.

   E = \max_i \lVert a(t_i) - a_i \rVert \le \varepsilon

   The paper uses ε = 0.002 on its real tasks to keep precise motions. The toy fit uses ε = 0.004 rad and needs 41 knots for 401 samples.

4. 04

   Fixed-size targets

   Observations do not line up with knots, so at each observation step the target is the nearest future window of 16 knots and their control points. Because knot spacing varies, 16 knots can cover a long time where the motion is simple and a short time where it is detailed. Near the end of an episode the last control point is repeated to fill the window.

   16 knots + 16 control points per sample
5. 05

   Regress the parameters

   The network predicts the 16 × 7 parameters and is trained against the fitted ones, exactly where ACT would be trained against a 16-step action chunk. For a fair comparison, the paper’s chunking baselines also predict 16 future actions, with the same visual encoder and the same number of epochs.

6. 06

   16 actions vs 16 knots

   A 16-step chunk at 10 Hz always covers 1.6 s at an even spacing. The 16 knots of a segment are placed by the fit, so detail goes where the trajectory needs it. This is the “adaptive temporal resolution” the paper argues fixed chunks lack.

7. 07

   Fit tolerance

   A looser tolerance needs fewer knot–control-point pairs. On Push-T the compression ratio rises from 1.12× at ε = 0.5 to 3.34× at ε = 8, while the score stays between 0.52 and 0.65. Performance is not very sensitive to ε within a sensible range. These are the paper’s numbers (Table 3); Push-T actions are pixel positions up to about 512, which is why ε is so much larger there.

Results