Rare find

FlagEvalMM. Test AI models that work with images, video, and text all at once.

github.com/flageval-baai/FlagEvalMM

Problem

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Audience

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Market

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Pricing

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

March 2026

fix text vqa (#126) Mar 27
fix mmvet_v2 and upgrade judge model to gpt-5-mini (#125) Mar 27
add skill for integrating new datasets into FlagEvalMM (#124) Mar 26
add MeasureBench task for visual measurement reading evaluation (#123) Mar 26

February 2026

add RelScene dataset (#121) Feb 2

January 2026

refactor: pass annotation dict to prompt template functions (#120) Jan 28
update tasks config (#119) Jan 28
fix pyav video reader for FLV files (#117) Jan 27
add Animalbench (#116) Jan 27
fix: task name Jan 20
Fix ARPGrounding ... Jan 20
fix: remove incorrect folder name arpggrounding Jan 20
Add ARPGrounding visual grounding task Jan 20
fix: remove unused variable and format code with black Jan 20
fix erqaplus name (#114) Jan 12

December 2025

point norm1k (#111) Dec 24
ERQAPlus (#108) Dec 23
add revel evaluator (#106) Dec 22

October 2025

remove decord in dependency (#103) Oct 23
fix t2i evaluator (#102) Oct 22