deVision v0.4 evaluation
Generated by scripts/release/hf.py from the evaluation of this release.
| Set | Questions | Accuracy | Mismatched | NLL | ECE |
|---|---|---|---|---|---|
| COCO object presence | 1,120 | 0.955 | 0.497 | 0.196 | 0.092 |
| VQAv2 multiple choice | 1,453 | 0.905 | 0.370 | 0.280 | 0.053 |
| COCO size | 1,198 | 0.870 | 0.516 | 0.506 | 0.234 |
| POPE, project filtered set | 9,000 | 0.865 | 0.520 | 0.363 | 0.048 |
| GQA val subset | 986 | 0.843 | 0.546 | 0.383 | 0.086 |
| COCO position | 1,276 | 0.938 | 0.509 | 0.238 | 0.047 |
| VQAv2 yes/no subset | 1,000 | 0.722 | 0.500 | 0.571 | 0.055 |
| COCO relative position | 2,366 | 0.917 | 0.480 | 0.281 | 0.037 |
| VSR, project held-out split | 902 | 0.731 | 0.460 | 0.591 | 0.073 |
| Visual7W, project held-out split | 1,000 | 0.783 | 0.442 | 0.597 | 0.073 |
| CountBench, 0–10 choices | 491 | 0.501 | 0.102 | 1.387 | 0.166 |
| COCO mirrored position | 640 | 0.933 | 0.503 | 0.283 | 0.053 |
| COCO mirrored relative position | 1,786 | 0.914 | 0.494 | 0.300 | 0.049 |
| What'sUp | 820 | 0.467 | 0.233 | 1.214 | 0.093 |
Laya Vision 201M, same questions
| Set | Questions | Laya Vision | deVision | Difference |
|---|---|---|---|---|
| VQAv2 yes/no | 4,887 | 0.717 | 0.756 | +3.9 [+2.4, +5.3] |
| A-OKVQA | 1,138 | 0.598 | 0.634 | +3.5 [+0.3, +6.6] |
| ScienceQA with images | 2,097 | 0.824 | 0.821 | -0.3 [-2.2, +1.7] |
| ScienceQA natural science, needs the picture | 323 | 0.700 | 0.638 | -6.2 [-12.7, +0.0] |
On the full POPE random / popular / adversarial sets (3,000 questions each) deVision scores 0.899 / 0.873 / 0.824, against Laya Vision's published 0.836 / 0.819 / 0.777 (aggregate scores only, not paired).
Temperatures
| Bucket | Temperature |
|---|---|
| Choice, 2 options | 3.5631 |
| Choice, 3–5 options | 2.1146 |
| Noul (yes / no) | 1.0895 |
| Choice, any other count | 2.3642 |