medelharchaoui/t5-large-xa-only-squad

T5-large — Cross-Attention-Only fine-tune on SQuAD

Encoder–decoder QA model fine-tuned from google-t5/t5-large by training only the decoder cross-attention (EncDecAttention) blocks plus the decoder final layer norm. The encoder, decoder self-attention, and all feed-forward weights are frozen at their pretrained values.

TL;DR: training just the cross-attention bridge (~100.7M of 737M params, 13.7%) is enough to reach F1 0.8128 / EM 0.6406 on SQuAD — and beats a parameter-matched GPT-2-large decoder-only model (774M) by +0.31 F1 (1.61×).

This checkpoint is part of a study arguing that encoder–decoder architectures match or exceed decoder-only models on context-grounded tasks while being cheaper at inference, and that for such tasks the cross-attention pathway — not brute-force full fine-tuning — is what matters.

Results (SQuAD validation)

Model	Architecture	Trainable / Total	EM	Token F1
GPT-2-large	decoder-only	774M / 774M	0.3516	0.5041
T5-large XA-only (this model)	enc-dec	100.7M / 737M	0.6406	0.8128
T5-large LoRA r=8	enc-dec	2.4M / 740M	0.6445	0.8152
T5-large full fine-tune	enc-dec	737M / 737M	0.6602	0.8162
T5-small XA-only	enc-dec	6.3M / 60.5M	0.5293	0.7075

Validation loss 0.3059, perplexity 1.36. XA-only and LoRA reach a lower validation loss than full fine-tuning despite training a small fraction of the weights — architecture alignment dominates over brute-force optimization.

Eval note: T5 numbers are computed on a 256-example SQuAD-validation generation subset (beam search, 4 beams). GPT-2-large is on 512 examples. Numbers are comparable in magnitude and reproduce the paper's headline ordering.

Intended use

Extractive question answering: given a question and a context passage, generate the answer span. Trained on SQuAD v1.1 (30,000 train examples).

How to use

The model was trained with the input prefix answer question: prepended to a question: ... context: ... source string. Match this format exactly at inference:

from transformers import T5ForConditionalGeneration, AutoTokenizer

repo = "medelharchaoui/t5-large-xa-only-squad"
tok = AutoTokenizer.from_pretrained(repo)
model = T5ForConditionalGeneration.from_pretrained(repo)

question = "What culture do 'bairn' and 'hyem' originate from?"
context = ("'bairn' and 'hyem', meaning 'child' and 'home', are examples of geordie "
           "words with origins in scandinavia; barn and hjem are the corresponding "
           "modern norwegian and danish words.")

text = f"answer question: question: {question} context: {context}"
ids = tok(text, return_tensors="pt", truncation=True, max_length=384).input_ids
out = model.generate(ids, num_beams=4, max_new_tokens=16)
print(tok.decode(out[0], skip_special_tokens=True))  # -> scandinavia

Training

Setting	Value
Base model	google-t5/t5-large (737M)
Trainable params	*.EncDecAttention.* + decoder.final_layer_norm (~100.7M)
Dataset	rajpurkar/squad, 30,000 train examples
Objective	extractive QA (seq2seq)
Precision	bf16
Optimizer steps	3,000 (batch 4 × grad-accum 8 = eff. batch 32)
LR / warmup	2e-4, 300 warmup, weight decay 0.01
Source / target max len	384 / 32
Seed	37
Hardware	1× NVIDIA RTX 3060 (12 GB), local

Limitations

• English, SQuAD-style extractive QA only; answers are short spans copied/paraphrased from the supplied context. Not a general chat or open-domain model.
• Evaluated on a held-out validation subset, not the official SQuAD test server.
• Inherits any biases present in T5's C4 pretraining and in SQuAD.

Citation

Part of an encoder–decoder vs decoder-only paradigm study (OptimiAI, 2026). If you use this checkpoint, please cite the accompanying paper/repository.