RuntimeError: weight model.vision_model.embeddings.position_ids does not exist
Got this error using the provided SageMaker SDK script. Is this a known bug in TGI 1.0.3?
will get to the bottom of this tomorrow, it does seem surprising, thanks for reporting!
Ok, I understand the situation now.
In TGI (file idefics_vision.py), we are defining the attribute position_ids as self.position_ids = weights.get_tensor(f"{prefix}.position_ids") which means that during the initialization of the model, we'll look for a tensor called position_ids.
The instruct models have that weight tensor, but not the base ones.
However, in HF Transformers, we are defining position_ids as a registered buffer (file idefics/vision.py: self.register_buffer("position_ids", torch.arange(self.num_positions).expand((1, -1)), persistent=False)) which means that position_ids is automatically registered at initialization.
My suggestion would be to correct the way we initialize position_ids in TGI (I think that's a mistake i made to not use registering buffers). Could you confirm it is the way course of action @Narsil ?
This would work, however I don't think it's a great idea.
Either we should always look for them on file, or never. We had a similar thing with Llama and inv_freq. The issue of doing it "sometimes" is that some models might save a different buffer than the one we generate automatically which makes it super hard to debug.
Are those positions ids always arange ? If yes we should just use that and drop the loading part, no ? (And we could probably do the same in transformers)
Got it!
Fixed it here: https://github.com/huggingface/text-generation-inference/pull/1064
It uses the same logic as in transformers
@jinyolim just want to make sure you saw this: Nicolas pushed the fix on tgi 1.1.0. could you report back on whether you are still seeing the same bug?