JEPA Attach: training a language model on images it never reads about

High-quality human text is finite. Villalobos et al. estimate that if current trends hold, frontier training runs will use up the public stock of human-written text sometime between 2026 and 2032 [1]. Training on model-generated text instead has a known failure mode: models trained recursively on their own outputs lose the tails of the distribution [2].

Images and video are not finite in the same way. Most of them have no captions, so they cannot feed a next-token objective directly. This note asks a narrower question: can an objective over unlabeled images change a language model's weights in a way that makes it better at text?

Nothing here has been run yet. It is a design and an experiment plan.

Latent prediction

I-JEPA [3] learns from images without labels by predicting representations instead of pixels. An image is split into $N$ patches. A context block $C$ and a few target blocks $B_1, \dots, B_K$ are sampled, with the targets removed from the context. A context encoder $f_\theta$ reads the context patches; a target encoder $f_{\bar\theta}$ reads the whole image; a predictor $g_\phi$ receives the context representations plus a mask token at each target position and predicts what the target encoder produced there:

The target encoder is an exponential moving average of the context encoder, $\bar\theta \leftarrow \tau \bar\theta + (1 - \tau)\,\theta$, and receives no gradient. Predicting in representation space lets the model ignore unpredictable pixel detail and spend capacity on structure.

Why the first version fails

My original design trained three things at once: next-token prediction on text, I-JEPA on images, and an alignment term that pulled a projection of the language model's pooled hidden state toward a projection of the image encoder's pooled output:

It has two problems, and either one is enough to sink it.

No gradient reaches the language model from the images. $\mathcal{L}_{\text{JEPA}}$ depends only on the image encoder and the predictor, so

The only path from images into the language model is the alignment term.

The alignment term has a trivial minimum. With a shared projection $P$, setting $P = 0$ makes the loss zero for every input. Even without collapse, the text and image batches were sampled independently, so the term pulled unrelated sentences toward unrelated pictures. Alignment needs paired data and a contrastive objective whose negatives rule out the constant solution, as in CLIP [4]:

Paired image–text data is exactly the scarce kind, so alignment cannot be the channel for the abundant data. The design has to change.

The revised design: the language model is the predictor

Put the language model's transformer where $g_\phi$ was. A small trainable patch embedding $E_\psi$ maps raw image patches, with 2D position embeddings, into the language model's input space. The sequence fed to the model is the context patches followed by one query per target patch:

where $\Vert$ is concatenation, $m$ is a learned mask token and $p_j$ is the position embedding of patch $j$.

A causal language model is enough here: every query comes after all of the context, so each one can attend to the whole context block. A linear head maps the hidden state at each query to the target's representation. Targets come from a frozen, pretrained ViT [5] run on the full image, so there is no moving target and no collapse to guard against.

Now the image objective depends on the language model's own weights:

and $\nabla_{\theta_{\text{LM}}} \mathcal{L}_{\text{JEPA}}$ is no longer zero. Every unlabeled image now updates the same attention and MLP weights that model text. Whether that helps text is the open question.

def jepa_loss(lm, embed, query, head, teacher, images, ctx, tgt):
    """ctx, tgt: disjoint index tensors of patch positions."""
    with torch.no_grad():
        t = teacher(pixel_values=images).last_hidden_state[:, 1:]  # frozen ViT, drop CLS
        t = F.layer_norm(t[:, tgt], t.shape[-1:])
 
    x = embed(images)                          # (B, N, d) patch + 2D position embedding
    q = query + embed.pos[tgt]                 # one learned mask token per target position
    seq = torch.cat([x[:, ctx], q.expand(len(x), -1, -1)], dim=1)
 
    h = lm(inputs_embeds=seq).last_hidden_state[:, -len(tgt):]
    return F.mse_loss(head(h), t)
 
 

What could go wrong

• Interference. The image objective could pull capacity away from text and raise perplexity. $\lambda$ controls the trade-off and has to be swept.
• Compute, not data. The image steps cost FLOPs. If a text-only baseline given the same extra FLOPs (more epochs over the same text) closes the gap, the gain came from compute, not from images.
• Shallow transfer. The model might learn image features in a corner of its weights that text never uses. Probing hidden states on text before and after would show whether anything moved.

Experiment

GPT-2 small (124M) [6] as the language model, a frozen ViT-B/16 as the teacher, block masks following I-JEPA. Three runs on the same text budget:

1. text only, same number of steps;
2. text only, extra steps matching the image run's FLOPs;
3. text plus images.

The metric is perplexity on held-out text. The hypothesis is supported only if run 3 beats run 2, across at least three seeds. If it does not, that is worth publishing too.

References

1. Villalobos et al. Will we run out of data? Limits of LLM scaling based on human-generated data. 2024. arXiv:2211.04325
2. Shumailov et al. The Curse of Recursion: Training on Generated Data Makes Models Forget. 2023. arXiv:2305.17493
3. Assran et al. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture. CVPR 2023. arXiv:2301.08243
4. Radford et al. Learning Transferable Visual Models From Natural Language Supervision. 2021. arXiv:2103.00020
5. Dosovitskiy et al. An Image is Worth 16x16 Words. ICLR 2021. arXiv:2010.11929
6. Radford et al. Language Models are Unsupervised Multitask Learners. 2019.