B. Decoupled, gradient-free retrieval keys
Codes optimised for generation need not be good retrieval prototypes, so each task keeps a separate key grounded in the
frozen input embeddings \( (V_i^t, Q_i^t) \):
\[
r_i^t = \mathrm{Norm}\!\bigl(\Pi\,\mathrm{Pool}([V_i^t;\, Q_i^t])\bigr), \qquad
k^t = \mathrm{Norm}\!\Bigl(\tfrac{1}{N_t}\sum_{i=1}^{N_t} r_i^t\Bigr),
\]
where \( \Pi \in \mathbb{R}^{D_k \times C} \) is a fixed Rademacher random projection (Johnson-Lindenstrauss).
Routing adds no trainable parameters and needs no rehearsal.
✅ Pooling the question tokens alone already gives discriminative keys.
C. Warm-starting new task codes
For \( t > 1 \), the task-specific code is initialised by soft retrieval of past codes in key space:
\[
z_{\mathrm{sp}}^t = \sum_{\tau < t} w_\tau^t\, z_{\mathrm{sp}}^\tau + \epsilon, \qquad
w_\tau^t = \frac{\exp(\cos(k^t, k^\tau)/\rho_z)}{\sum_{j < t}\exp(\cos(k^t, k^j)/\rho_z)}, \quad \epsilon \sim \mathcal{N}(0,\sigma^2 I),
\]
so semantically related past tasks provide positive transfer instead of optimising from scratch.
D. LookAhead regularisation (LA-Reg)
Let \( \phi^\star \) be the generator snapshot before task \( t \) and \( \Delta\phi \) the update from \( M \) inner-loop steps on
the current-task NLL. LA-Reg penalises the drift this update would cause on the prompts of stored past codes:
\[
\mathcal{L}_{\mathrm{LA\text{-}Reg}}^t = \sum_{\tau < t} \bigl\| H_{\phi^\star}(z^\tau) - H_{\phi+\Delta\phi}(z^\tau) \bigr\|_2^2 .
\]
It needs only the stored codes, not raw data: past prompt behaviour is reconstructed on the fly through \( H_{\phi^\star} \).
The shared code is additionally anchored with \( \|z_{\mathrm{inv}} - z_{\mathrm{inv}}^\star\|_2^2 \).
LA-Reg balances progress along the task-\( t \) gradient with alignment to the task-\( (t-1) \) anchor,
steering optimisation into the shared low-loss basin (green) and yielding flatter cross-task minima.
E. Full objective
\[
\mathcal{L}_{\mathrm{Final}}^t
= \mathcal{L}_{\mathrm{NLL}}^t
+ \lambda_{\mathrm{LA\text{-}Reg}}\,\mathcal{L}_{\mathrm{LA\text{-}Reg}}^t
+ \lambda_{\mathrm{inv}}\, \|z_{\mathrm{inv}} - z_{\mathrm{inv}}^\star\|_2^2
+ \lambda_{\mathrm{VAQ}}\,\mathcal{L}_{\mathrm{VAQ}}^t
+ \lambda_{\mathrm{QAV}}\,\mathcal{L}_{\mathrm{QAV}}^t ,
\]
with two auxiliary cross-modal objectives (predict the question from video + answer, and the video from question + answer).
A causal view of the VideoQA graph \( V \to Q,\; V \to A,\; Q \to A \) explains their asymmetric weights: \( p(Q \mid V, A) \)
follows a feasible causal direction, while \( p(V \mid Q, A) \) is anti-causal and is kept only as a light visual-grounding signal.
One default setting is used across all benchmarks, leaving DynaTokens tuning-free at deployment.
F. Task bank and task-agnostic inference
After task \( t \) only \( (k^t, z_{\mathrm{sp}}^t) \) is cached. At test time the nearest task is retrieved by cosine similarity,
\[
t^\star = \arg\max_{\tau \le t} \cos(r_{\mathrm{test}}, k^\tau),
\]
the code \( z^{t^\star} = [z_{\mathrm{inv}}, z_{\mathrm{sp}}^{t^\star}] \) is reconstructed and the prompts
\( P_{\mathrm{test}} = H_\phi(z^{t^\star}) \) are synthesised on the fly. No task identifier is required.
