The within-batch negatives deserve attention when annotated class names and noisy alt text share a batch. Two different images with the same class label are not semantically unrelated, yet a contrastive objective can treat their pairings as negatives. Captioning supervision is asking a different question about those same examples.
I would keep that tension in mind when interpreting the loss-weight ablation: more captioning weight may change which distinctions the representation preserves, rather than simply adding finer detail everywhere. Comparing same-class retrieval errors across the loss settings would be a useful complement to the aggregate zero-shot classification result.
I would still frame false negatives as a competing hypothesis rather than a conclusion from the ablation alone. The loss weights change several training signals together, so the aggregate score cannot isolate that mechanism.
A useful diagnostic would separate same-class retrieval confusions from different-class errors at each weight setting. If those move differently, it would motivate a more controlled experiment with the batch composition or positive-pair definition, while keeping the original paper result and the proposed explanation clearly distinct.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The within-batch negatives deserve attention when annotated class names and noisy alt text share a batch. Two different images with the same class label are not semantically unrelated, yet a contrastive objective can treat their pairings as negatives. Captioning supervision is asking a different question about those same examples.
I would keep that tension in mind when interpreting the loss-weight ablation: more captioning weight may change which distinctions the representation preserves, rather than simply adding finer detail everywhere. Comparing same-class retrieval errors across the loss settings would be a useful complement to the aggregate zero-shot classification result.