: The Next Step Forward in Multimodal LLM Alignment¶
Source: converted from arXiv LaTeX source source/neurips_2024.tex.
Abstract¶
Despite notable advancements in Multimodal Large Language Models (MLLMs), most state-of-the-art models have not undergone thorough alignment with human preferences. This gap exists because current alignment research has primarily achieved progress in specific areas (e.g., hallucination reduction), while the broader question of whether aligning models with human preferences can systematically enhance MLLM capability remains largely unexplored. To this end, we introduce , a dataset containing 120k fine-grained, human-annotated preference comparison pairs. This dataset represents a substantial advancement over existing resources, offering superior size, diversity, annotation granularity, and quality. Leveraging this dataset, we propose several key innovations to improve both the quality of reward models and the efficiency of alignment algorithms. Notably, we introduce a Critique-Based Reward Model, which generates critiques of model outputs before assigning scores, offering enhanced interpretability and more informative feedback compared to traditional scalar reward mechanisms. Additionally, we propose Dynamic Reward Scaling, a method that adjusts the loss weight of each sample according to the reward signal, thereby optimizing the use of high-quality comparison pairs. Our approach is rigorously evaluated across 10 distinct dimensions and 27 benchmarks, with results demonstrating significant and consistent improvements in performance. Specifically, fine-tuning LLaVA-ov-7B with and our alignment algorithm leads to a 19.5% increase in conversational abilities and a 60% improvement in safety.
Introduction¶
Although Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in addressing complex tasks that involve the integration of vision, language, and audio, state-of-the-art models today seldom undergo a rigorous alignment stage [wang2024qwen2,deitke2024molmo,chen2024far,dai2024nvlm,agrawal2024pixtral]. Typically, these models only progress to the Supervised Fine-tuning (SFT) stage, leaving critical aspects such as truthfulness, safety, and alignment with human preferences largely unaddressed. While recent efforts have begun to explore MLLM alignment, they often focus on specific domains, such as mitigating hallucination or enhancing conversational capabilities, which fail to comprehensively improve the model's overall performance and reliability. This raises a critical question:
tcolorbox[top=1pt, bottom=1pt, left=1pt, right=1pt] center Is alignment with human preferences only capable of enhancing MLLMs in a limited set of tasks? center tcolorbox
In this work, we confidently answer this question with a resounding ``No.''. We demonstrate that a well-designed alignment pipeline can comprehensively enhance MLLMs along multiple dimensions, including visual perception, reasoning, dialogue, and trustworthiness, thereby significantly broadening their practical applicability. To achieve this, we conduct in-depth investigations into three pivotal areas: data curation, reward modeling, and alignment algorithms.
At first, we introduce *, a dataset designed to advance *Multimodal Reinforcement Learning from Human Feedback (RLHF). The dataset spans three domains: image, video understanding, and MLLM safety. Constructed through a rigorous pipeline, ensures high-quality, fine-grained annotations. Dataset creation process involves the following steps (Figure fig:data_construction):
itemize Data Collection. We curate a diverse set of multimodal tasks from various sources, totaling 10 million data instances, ensuring broad representation across tasks. Data Selection. Through rigorous re-sampling, we extract 30k representative queries, ensuring diversity across a wide range of data types, such as real-world scenarios, mathematical reasoning, chart understanding, and other practical domains (Figure fig:data_cluster). Model Response Generation. We utilize state-of-the-art models, such as Claude 3.5-Sonnet and Qwen2-VL-72B, to generate responses for various tasks. Fine-grained Human Annotation. We employ a meticulous annotation process, involving over 50 annotators over two months, to score, rank, and provide textual explanations for responses. This results in more than 120k high-quality ranked comparison pairs. itemize Compared to existing datasets, significantly advances in diversity, response quality, and annotation granularity, providing a robust foundation for MLLM alignment.
Building on the dataset, we investigate how human-annotated data can enhance MLLM alignment, with a focus on reward modeling and training optimization. Recognizing the pivotal role of reward models in providing feedback signals to guide the alignment process, we propose a Critique-Based Reward Model (Figure fig:rm_teaser). Traditional reward models, which output scalar values, often lack interpretability, while directly using MLLMs as reward models place high demands on their instruction-following capabilities, limiting their practicality. To address these limitations, we first transform concise human annotations into detailed, model-friendly formats using MLLMs. These enriched annotations serve as learning targets, guiding the reward model to first generate critiques and then assign scores based on the critiques. This approach enables the model to provide fine-grained scoring explanations, significantly enhancing the quality and interpretability of the reward signals. -Reward-7B achieves SOTA performance on several reward model benchmarks, outperforming several 72B-scale models.
Building on this high-quality reward model, we introduce Dynamic Reward Scaling within the Direct Preference Optimization (DPO) framework. Traditional DPO methods [amini2024direct] use a fixed training weight for all human-preferred and non-preferred training pairs. In contrast, Dynamic Reward Scaling calculates a reward margin for each comparison pair using -Reward-7B. During training, it assigns higher weights to comparison pairs with larger reward margins. This ensures that the most informative samples have a stronger influence on model updates. As a result, the training process becomes more efficient, leading to improved model performance.
Finally, to rigorously evaluate our approach, we construct two specialized benchmarks. The first, -RewardBench, is sampled from our dataset and consists of meticulously human-annotated data for evaluating reward models. The second, -SafetyBench, is curated and filtered from existing benchmarks and focuses on safety-related tasks, including privacy protection, adversarial attacks, jailbreaking, and harmful content detection.
We conduct extensive evaluations across ten key dimensions, covering 27 benchmarks. The results demonstrate that our training algorithm, combined with the high-quality dataset, leads to significant improvements in model performance. Specifically, models fine-tuned with our approach achieve an average 11% gain in conversational abilities and a 57% reduction in unsafe behavior. The integration of our reward model further amplifies these gains, highlighting the effectiveness of our alignment algorithm.
-Dataset¶
Figure fig:data_construction: Construction Pipeline. (1) Data Collection and Cleaning: Starting with 10 million instruction samples, we cluster data based on image similarity, and uniformly sample across diverse categories. This results in a diverse dataset covering image-based Q&A (e.g., multiple-choice, dialogues, and safety-related questions) and video Q&A formats. (2) Response Generation: We leverage state-of-the-art models, including GPT-4o and Qwen2-VL-72B, to generate high-quality responses. (3) Human Annotation: We conduct manual annotation across nine categories, including scoring, ranking, and explanations, ensuring fine-grained evaluation.
In this section, we outline the construction of , as illustrated in Figure fig:data_construction. This includes the data collection process, data filtering methods, and human annotation procedures.
Data Collection¶
Our goal is to construct a comprehensive post-training dataset that covers a wide range of task types. To achieve this, we categorize tasks into three main domains: image understanding, video understanding, and multimodal safety.
For image understanding, we integrate data from multiple sources, including LLaVA-OVhttps://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data, VLfeedback[li2023silkie], LLaVA-RLHF [sun2023aligning], lrv-instruction [liu2023mitigating], and Unimm-Chathttps://huggingface.co/datasets/Yirany/UniMM-Chat. Since some datasets contain multi-turn dialogues, which are less suitable for response generation, we decompose them into single-turn dialogues. This process yields over 10 million dialogue samples, covering tasks such as conversation, safety, multiple-choice questions, captions, and commonsense reasoning.
For video understanding, the primary data source is SharedGPT-4 video [chen2024sharegpt4video].
For safety, data is primarily derived from VLGuard [zong2024safety] and self-constructed content. VLGuard contains over 2,000 harmful samples, while additional red teaming, safety, and robustness data are included. The pipeline for constructing safety data is detailed in the Appendix sec:safety_data.
Data Filtering and Model Response Generation¶
Figure fig:data_cluster: Re-Sample results from the clustering process. Due to the large total number of samples, the clustered and deduplicated results contain a rich diversity of categories. Selected samples include topics such as mathematics, daily life, natural scenes, medicine, electronic technology, and OCR scenarios, showcasing a variety of problem-image pairs. The 2D features were obtained via UMAP dimensionality reduction.
The core goal of data filtering is to reduce the number of samples while maintaining the diversity of the original dataset. To achieve this, the following strategies are adopted:
Predefined sampling weights. For image understanding tasks, we define three categories based on the nature of the questions and the length of model responses: 1. Multiple-choice questions (MCQ); (Questions with options such as A, B, C, or D.) These tasks include visual question answering, mathematics, OCR, and icon recognition, focusing on the model's reasoning and visual perception abilities. 2. Long-text questions; (Questions for which GPT-4o generates responses exceeding 128 characters.) These typically involve detailed captions or complex descriptions, testing the model's conversational and descriptive capabilities. 3. Short-text questions; (Questions for which GPT-4o generates responses shorter than 128 characters.) These require concise answers, often involving simple image analysis, and represent a broader range of task types.
The initial distribution of these three types in the image understanding dataset is highly imbalanced, with proportions of 12.17% (Long), 83.68% (Short), and 4.14% (MCQ). To align with diversity goals, we adjust the sampling ratio to 4:5:1, reducing disparities among task types while maintaining a dominance of comprehensive samplesFor video understanding and safety tasks, MCQ samples are fewer. After classifying into Long and Short types, the differences are minimal, so no additional adjustments are made..
Cluster-based Sampling. Text deduplication is not performed because many questions, while similar in text, are paired with different images, leading to substantially different outcomes—an intrinsic characteristic of multimodal data. Instead, we encode all images using CLIPhttps://huggingface.co/openai/clip-vit-base-patch32, and for videos, we use the feature of the first frame as a representative. We then apply KNN clustering with 100 cluster centers and randomly sample N instances from each cluster. The value of N is determined to satisfy the predefined sampling ratios, ensuring a balanced representation of task diversity.
Data statistics. The composition of the dataset is summarized in Table tab:data_statistics, and a visualization of the clustering results is shown in Figure fig:data_cluster, demonstrating the rich diversity of data categories.
Model response generation. To generate high-quality responses, we select state-of-the-art models from both open-source and closed-source domains. For image understanding and safety-related tasks, we use Qwen2-VL-72B [wang2024qwen2], LLaVA-OV-72B [li2024llava], GPT-4ohttps://openai.com/index/hello-gpt-4o/, and Claude 3.5-sonnethttps://www.anthropic.com/news/claude-3-5-sonnet. For video understanding tasks, we employ GPT-4o, LLaVA-Video-72B [zhang2024video], and Qwen2-VL-72B [wang2024qwen2]. These models are chosen for their advanced capabilities and performance, ensuring a comprehensive evaluation of leading solutions in multimodal understanding.
Table tab:data_statistics: Dataset Composition Statistics
\begin{table}[]
\caption{Dataset Composition Statistics}
\label{tab:data_statistics}
\centering
\begin{tabular}{cccccc}
\toprule
\multicolumn{3}{c}{\textbf{Image}} & \multirow{2}{*}{\textbf{Safety}} & \multirow{2}{*}{\textbf{Video}} & \multirow{2}{*}{\textbf{Total}} \\ \cmidrule{1-3}
\textbf{Long} & \textbf{Short} & \textbf{MCQ} & & & \\ \hline
9,575 & 12,063 & 2,125 & 1,999 & 4,235 & 29,997 \\ \bottomrule
\end{tabular}
\end{table}
Annotation¶
The annotation process follows rigorous standards to ensure comprehensive and fine-grained evaluations of MLLM responses. Detailed standards are provided in Appendix sec:annotation_standard, and the scoring and annotation structure are illustrated in Figure fig:data_construction. Additionally, we design a web UI to streamline the annotation process, as shown in Figure fig:web_ui.
Annotation Standards¶
Compared to prior work, our annotation approach introduces two significant advantages: richness and granularity. First, the evaluation incorporates three core dimensions—Helpfulness, Faithfulness, and Ethical Considerations—to comprehensively capture model performance. Helpfulness ensures that responses are relevant and provide meaningful assistance aligned with the user’s intent. Faithfulness evaluates the accuracy of responses in describing visual elements, such as objects, relationships, and attributes, ensuring alignment with the ground truth while avoiding hallucinated content. Ethical Considerations assess adherence to ethical principles, including safety, privacy, fairness, and harm avoidance, ensuring responses are free from harmful or biased content. Annotators score each dimension while documenting the reasoning behind their assessments, adding valuable context for understanding model performance.
Second, annotators are required to assign an overall ranking to the responses, along with justifications for their rankings. This ranking mechanism provides a transparent and nuanced comparison of model outputs. Additionally, innovative strategies are employed to enhance data quality:
-
Constructing positive samples for poor quality ties. When multiple responses are equally poor, annotators provide correct answers to create positive examples. This ensures that challenging samples contribute to the training dataset, addressing issues where no valid model responses exist.
-
Constructing negative samples for high-quality ties. When multiple responses are of equally high quality, annotators introduce deliberate errors to create negative samples. This prevents ties from reducing the utility of the data and allows for more efficient use in training.
By combining fine-grained scoring criteria, textual annotations, and innovative strategies, our annotation framework produces a high-quality dataset that comprehensively captures model performance and supports effective downstream applications.
Human Annotation vs. Machine Annotation¶
Annotation workers and costs. The annotation process employs over 50 annotators, supported by 8 multimodal research experts with strong English proficiency and academic backgrounds. The entire task completes within two months, with periodic quality checks and interactive reviews conducted by experts to ensure the reliability and accuracy of the annotations. Low-quality samples undergo re-annotation during the process. Due to the fine-grained nature of the annotation standards, the task involves significant challenges. For example, annotating a single question in the long split of image perception tasks requires an average of over \(8\) minutes.
Why human annotation? Many existing MLLM alignment datasets rely on annotations generated by external models due to their cost-effectiveness and scalability. However, MLLM alignment tasks demand fine-grained perceptual capabilities and sensitivity to subtle differences, which current models lack. In many cases, the differences between responses are nuanced, requiring an in-depth understanding that models struggle to achieve. As demonstrated in our experiments, even state-of-the-art models like GPT-4o significantly underperform human experts in tasks involving response comparison. Moreover, these models cannot provide professional-grade scoring or well-reasoned explanations for rankings. These limitations highlight the necessity of human annotation, which ensures the precision, reasoning, and insight required for constructing high-quality alignment datasets. Appendix sec:why_we_need_human further discusses the advantages of human annotation, particularly in handling ambiguous or incomplete questions and closely matched responses requiring subtle differentiation. Human annotators excel at identifying fine-grained errors, inconsistencies, and context-specific nuances that models overlook. By relying on human feedback, our approach ensures the dataset achieves the quality and reliability necessary for advancing MLLM alignment efforts.
We acknowledge that the cost of human annotation poses scalability challenges. However, as demonstrated in later sections, our high-quality alignment dataset enables the training of a powerful reward model. In the future, by combining this reward model with human annotators in a collaborative framework, we can significantly reduce annotation costs and scale up the dataset efficiently. This hybrid approach not only maintains the precision of human annotation but also enhances scalability, making it a practical solution for large-scale MLLM alignment.
-Reward Model¶
Figure fig:rm_teaser: Illustration of the multi-task reward model training process. The process begins with a user query and corresponding model responses, which are ranked and annotated by humans. Human annotations are expanded using GPT-4o to provide enhanced rationales. The reward model is trained with two objectives: (1) Learning to Provide Critique, where the model learns to provide detailed critiques and evaluations for model responses, and (2) Learning Scoring, where the model learns to assign scores based on the model response and critique. The integration of these tasks ensures a robust evaluation framework for improving model outputs.
In this section, we explore how to train a high-quality reward model using the dataset to provide a robust supervision signal for subsequent model alignment. The reward model is designed to combine critique generation and scoring (Figure fig:rm_teaser), ensuring a comprehensive evaluation process.
Background and Limitations of Standard Reward Models¶
Reward models are a key component for aligning model outputs with human preferences. Typically, a reward model starts with a pretrained LLM \(\phi\) , where the LLM head \(h_l\) is replaced with a linear reward head \(l_r\) , enabling the model to output a scalar reward value. These models are trained using human-provided pairwise comparisons. Given a query \(\mathbf{x}\) , a preferred response \(y_w\) and a less preferred response \(y_l\) , the reward model is optimized to assign higher rewards to preferred responses:
where \(r(y | \mathbf{x})\) is the scalar reward and \(\sigma\) is the sigmoid function.
Despite their utility, standard reward models face significant limitations. First, they fail to fully utilize the rich and detailed feedback provided by high-quality human annotations, such as textual explanations and nuanced reasoning. Second, scalar rewards lack transparency, making it difficult for humans to understand how the reward is generated. These challenges highlight the need for a more interpretable and robust reward model that leverages critiques as intermediate reasoning steps.
Critique-Based Reward Model Training¶
Extending to critique-based training. To overcome the limitations of traditional reward models, we propose a critique-based training framework: the model first generates a critique \(c\) conditioned on the query \(\mathbf{x}\) . This critique serves as an intermediate reasoning step, providing context for scoring responses. The critique-based reward model comprises two components: 1. Critique Head ( \(h_l\) ): Generates critiques \(c_w\) and \(c_l\) for the preferred ( \(y_w\) ) and less preferred ( \(y_l\) ) responses, respectively, based on the query \(\mathbf{x}\) . 2. Scoring Head ( \(h_r\) ): Assigns scalar rewards based on the generated critiques, enabling more fine-grained evaluation.
Learning to provide critique from enhanced annotation. The critique head ( \(h_l\) ) is trained to align with human-provided annotations. The loss function for critique generation is:
where \(c_t\) is the \(t\) -th token in the critique \(c\) , \(c_{<t}\) denotes the tokens preceding \(c_t\) , and \(\pi_\theta(c_t | c_{<t}, \mathbf{x}, y)\) is the probability of token \(c_t\) given its context, query \(\mathbf{x}\) , and model response \(y\) .
However, as shown in Figure fig:rm_teaser, while human-provided scoring reasons are highly accurate, they tend to be concise. Directly using these concise annotations as training targets for the reward model's language head does not yield significant performance improvements. To address this issue, we use GPT-4o to augment the human annotations by adding more detail and improving the fluency of the critiques. These enhanced scoring reasons are then used as the training targets for the language head. To prevent GPT-4o from introducing hallucinated content or irrelevant analysis, we impose strict constraints in the prompt (Table tab:prompt_example), to ensure the model only expands on the original content without introducing speculative or uncertain information.
Scoring loss with teacher-forcing. \(h_r\) computes scalar rewards based on the query \(\mathbf{x}\) , response \(y\) , and critique \(c\) . During training, we adopt a teacher-forcing strategy, where the scoring head uses ground truth critiques instead of critiques generated by itself. This avoids potential noise from model-generated critiques in the early stages of training. The scoring loss is defined as:
where: \(c_w\) and \(c_l\) are the ground truth critiques for the preferred response \(y_w\) and less preferred response \(y_l\) , respectively, \(r(\mathbf{x}, y, c)\) is the reward score computed from \(\mathbf{x}\) , \(y\) , and \(c\) .
Joint training objective. The overall training objective combines the critique generation loss and the scoring loss: $ _Total() = _Critique() + _Score(). $
Inference. During inference, the critique head ( \(h_l\) ) generates a critique \(c\) conditioned on the query \(\mathbf{x}\) and response \(y\) . The scoring head ( \(h_r\) ) then uses \(\mathbf{x}\) , \(y\) , and the generated critique \(c\) to compute the final reward score \(r(\mathbf{x}, y, c)\) . This two-step process mirrors the human evaluation process by explicitly reasoning about critiques before scoring.
-RewardBench. To evaluate the effectiveness of the signals provided by our reward model in guiding subsequent model training, we randomly sample 10 examples from each category of the dataset to create a test set. Each example includes multiple model responses and their corresponding rankings, enabling the generation of several comparison pairs. This results in a total of 170 pairs for evaluation. We design two evaluation metrics: 1. Traditional Accuracy (ACC): Measures the proportion of cases where the model correctly identifies the preferred response. 2. ACC+: Measures the proportion of cases where the model correctly ranks all response pairs for a given sample. This metric emphasizes the model's ability to handle challenging cases, such as those with small ranking differences or hard-to-distinguish pairs.
Discussion¶
In the MLLM community, there is currently no unified paradigm for the design of reward models. Some approaches rely on traditional reward models [sun2023aligning], which lack interpretability due to their reliance on scalar outputs. Others directly use LLMs to generate rankings [xiong2024llava], which heavily depend on instruction-following capabilities and often exhibit high variance in scoring. In the broader LLM community, works such as [yu2024self] explore reward models that first generate critiques. However, their focus is primarily on improving the reliability of model-generated critiques, such as increasing scoring confidence through multiple sampling—a goal distinct from ours. To the best of our knowledge, this is the first study to explore how MLLMs can effectively leverage human annotations to enhance both interpretability and the final model's scoring ability.
¶
Figure fig:dpo_alg: Overview of the framework, The dynamic reward scaling mechanism adjusts the update strength based on the reward margin, improving optimization stability and robustness.
In this section, we propose , an extension of the traditional DPO framework. introduces Dynamic Reward Scaling, which dynamically adjusts the update strength based on the confidence of training pairs, ensuring effective utilization of high-quality samples while mitigating the impact of noisy or low-confidence data.
Background: Direct Preference Optimization¶
The DPO framework is a preference-based learning method that optimizes model parameters \(\theta\) by aligning model outputs with human preferences. Given a query \(\mathbf{x}\) and corresponding responses \(y_w\) (positive) and \(y_l\) (negative), the DPO loss is defined as:
where \(\pi_\theta\) is the model's predicted probability distribution, \(\pi_{\text{ref}}\) is a reference policy, \(\beta\) is a scaling factor, and \(\sigma(\cdot)\) is the sigmoid function. Traditional DPO treats all training pairs equally, regardless of their quality differences. This uniform scaling fails to prioritize high-quality pairs with clear preference distinctions, leading to inefficient use of informative samples and suboptimal optimization.
: Key Contributions and Improvements¶
Training on all possible comparison pairs instead of the hardest pairs¶
. Unlike many recent MLLM alignment approaches that prioritize training on the hardest comparison pairs, incorporates all possible comparison pairs for a single query into the training process. Specifically, for any query with multiple responses, every response pair with differing ranks is treated as a valid comparison pair. This comprehensive approach captures more nuanced ranking information, allowing the model to learn from a broader set of preferences. However, this strategy also introduces a challenge: pairs involving responses with similar ranks (e.g., rank 3 and rank 4) often have lower reward margins compared to pairs with more distinct rankings (e.g., rank 1 and rank 4). Treating all pairs equally, as in traditional DPO, exacerbates the issue of uniform scaling and underutilizes the high-confidence information contained in larger reward margins. To address this, introduces Dynamic Reward Scaling, which dynamically adjusts the update strength based on the reward margin to prioritize high-confidence training pairs.
Figure fig:beta_func: Effect of \(k\) on $1 - e^-k $ .
Definition of dynamic reward scaling¶
. Reward models can naturally provide a pairwise reward margin, which serves as a straightforward signal for scaling. However, two critical aspects must be addressed: (1) ensuring the signal quality is sufficiently high, and (2) bounding the signal to prevent overly aggressive updates that might destabilize training.
Regarding the first aspect, our experiments reveal that publicly available models, such as GPT-4o and LLaVA-Critic, perform inadequately in scoring our dataset. Conversely, our -Reward-7B model surpasses several publicly available 72B models, offering a reliable and robust reward signal. We use this model to compute the reward margin: \(\delta = r(y_w) - r(y_l),\) where \(r(y_w)\) and \(r(y_l)\) are the scores assigned to the positive and negative samples.
For the second factor, we control the scaling factor \(\beta(\delta)\) using the following formulation: [ () = _ori ( 1 + w ( 1 - e^-k ) ), ]
where \(\beta_{\text{ori}}\) is the initial default scaling factor, \(w\) is a parameter balancing the dynamic component's contribution, and \(k\) is a tunable hyperparameter that adjusts \(\beta(\delta)\) 's sensitivity to changes in \(\delta\) . The function \(1 - e^{-k \delta}\) is bounded between \([0, 1]\) , as illustrated in Figure fig:beta_func. A smaller \(k\) value keeps most \(\beta(\delta)\) values near \(\beta_{\text{ori}}\) , with slow growth as \(\delta\) increases. In contrast, a larger \(k\) makes \(\beta(\delta)\) highly responsive to changes in \(\delta\) , quickly reaching its maximum. To avoid overly aggressive updates, we constrain \(\beta(\delta)\) within \([\beta_{\text{ori}}, (1 + w) \beta_{\text{ori}}]\) . Overall, Dynamic Reward Scaling significantly enhances by leveraging high-quality reward signals and tailoring optimization steps to the confidence level of training pairs. This results in improved robustness, efficiency, and overall effectiveness of the framework. We discuss the similarities and differing perspectives between our approach and existing methods in Appendix sec:app_com_beta.
Experiments¶
We evaluate our data and algorithms on 10 tasks across 20+ benchmarks. The key findings are:
-
Alignment training on the **** dataset consistently improves performance across nearly all benchmarks for various baselines. The integration of reward signals in MM-DPO further amplifies these improvements, demonstrating the effectiveness of our approach.
-
The -Reward-7B model achieves state-of-the-art performance on reward model benchmarks among open-source models, surpassing even several 72B models. This highlights the efficiency and scalability of our method.
-
We conduct extensive ablation studies and analyses, such as investigating the importance of critique learning for reward models and the sensitivity to hyperparameters. Additionally, we identify several experimental phenomena that challenge mainstream perspectives, such as the observation that small-scale MLLMs struggle to perform effective self-improvement. Due to space constraints, additional analysis are provided in Appendix sec:app_exp.
Benchmarks and Experimental Details¶
We categorize the benchmark datasets used in our experiments into the following domains:
Chart and Document Understanding: AI2D [kembhavi2016diagram], ChartQA [chartqa], DocVQA [docvqa], InfoVQA [mathew2022infographicvqa].
OCR (Optical Character Recognition): WebSRC [chen2021websrc], OCRBench [liu2024hidden], TextVQA [singh2019towards].
Hallucination: MMHal-Bench [mmhal-bench], POPE [pope], Object-Hal [li2023evaluating].
Math Reasoning: MathVista [lu2024mathvista], MathVerse [zhang2024mathverse].
General Knowledge: MME [fu2023mme], MMbench [liu2023mmbench], MMStar [chen2024we], SeedBench2-Plus [li2024seed], VQAv2 [antol2015vqa].
Conversation: LLaVA-Wilder [li2024llavanext-strong], LLaVA-In-The-Wild [liu2023visual], WildVision-Bench [lu2024wildvision].
High-Resolution and Real-World Utility: RealworldQA, MME-RealWorld [zhang2024mme].
Video Understanding: VideoChatGPT [Maaz2023VideoChatGPT], Video-MME [fu2024video], VideoDC [li2024llavanext-strong].
Multi-Image: LLAVA-Next-Interleave [li2024llava], MMMU-Pro [yue2024mmmu].
MLLM Safety: Our self-constructed benchmark, -SafeBench, includes adversarial attacks, jailbreaks, privacy, and harmful content. Detailed construction is provided in Appendix sec:safety_benchmark. Safety mainly evaluates the model's ability to reject harmful content, while unsafety mainly assesses the likelihood of the model being successfully attacked.
For all benchmarks requiring GPT-assisted evaluation, we consistently employ GPT-4o as the evaluation model. All model results are rigorously re-evaluated and reported by our team. All experiments are conducted on a high-performance computing cluster equipped with \(32\times\) H800 (80G) GPUs. Due to computational cost constraints, we utilize the full dataset for the main results presented in Tables tab:model_comparison, tab:model_comparison_safety, and tab:reward_model_comparison. For ablation studies, we uniformly sample \(1/5\) of the data, which may result in minor performance discrepancies compared to the full dataset.
In the implementation of MM-DPO, we adopt a common stabilization technique by incorporating an SFT loss. The weight of the SFT loss is selected through a grid search over the values \(\{0, 0.1, 0.25, 0.5, 1.0\}\) . Additionally, the learning rate is optimized via a search over \(\{1e\text{-}7, 5e\text{-}7, 1e\text{-}6, 5e\text{-}6, 1e\text{-}5\}\) to identify the best-performing configuration. Since we dynamically adjust the \(\beta\) parameter during training, the initial value of \(\beta_{\text{ori}}\) is set to a small default value of \(0.1\) , eliminating the need for manual tuning. Throughout all training processes, the vision encoder remains frozen to ensure stable and efficient training.
Evaluation of and MM-DPO¶
Table tab:model_comparison (for understanding tasks) and Table tab:model_comparison_safety (for safety tasks) illustrate the alignment performance of LLaVA-OV-7B, LLaVA-OV-0.5B and InternVL-1B using our dataset and alignment algorithm, where the scores for each evaluation dimension are averaged across their respective benchmarks.
Significant improvements in conversational ability and safety. Our experiments show that the alignment process leads to substantial improvements in these two aspects without requiring hyperparameter tuning. The average improvement in conversational benchmarks exceeds 10%, while unsafe behaviors are reduced by at least 50%. Additionally, in WildsVision, the win rate increases by at least 50%. This suggests that existing MLLMs lack explicit optimization for these dimensions, and our dataset effectively fills this gap.
Broad enhancements in hallucination, mathematical reasoning, multi-image, and video understanding. The aligned models also exhibit notable improvements in these areas. Interestingly, despite the lack of dedicated multi-image data in our dataset, the model's performance in multi-image tasks improves significantly. This indicates that the diversity of our alignment data enhances generalization across multiple dimensions.
Model-specific preferences for data and hyperparameter. Different models exhibit varying performance trends during alignment, with distinct preferences for hyperparameter settings across different benchmarks. For instance, in our training of InternVL-1B, we found that excluding the SFT loss led to better results. Additionally, while InternVL-1B demonstrated significant improvements in general knowledge tasks, its relative enhancement in OCR tasks was less pronounced compared to the LLaVA-OV series. These differences largely stem from variations in the models' pretraining datasets and strategies, necessitating tailored hyperparameter adjustments for optimal alignment.
Limited gains in high-resolution benchmarks. The model shows no significant improvement on high-resolution benchmarks, likely because our dataset contains relatively few ultra-high-resolution images. Additionally, our filtering strategy is based on image similarity rather than resolution, meaning the alignment process does not explicitly optimize for high-resolution tasks. As a result, performance gains in this area remain limited.
Ablation studies and sensitivity analysis. To further validate the effectiveness of our approach, we provide detailed ablation studies in the appendix, analyzing the impact of different alignment parameters and the improvements introduced by our dataset and MM-DPO.
light-graygray0.6 front-colorHTMLF5FFFA
Table tab:model_comparison: Performance variations after alignment across 8 different evaluation dimensions, comparing multiple models under our alignment strategy. All models show comprehensive performance improvements under the proposed alignment, demonstrating significant gains across various tasks.
\begin{table}[t!]
\caption{\textbf{Performance variations after alignment across 8 different evaluation dimensions}, comparing multiple models under our alignment strategy. All models show comprehensive performance improvements under the proposed alignment, demonstrating significant gains across various tasks.}
\label{tab:model_comparison}
\centering
\setlength{\tabcolsep}{4pt}
\renewcommand{1.0}{1.0}
\scriptsize
\resizebox{\textwidth}{!}{
\begin{tabular}{p{1.5cm}p{3cm}p{1.5cm}p{1.6cm}p{1.5cm}p{1.6cm}p{1.5cm}p{1.6cm}}
\toprule
\textbf{Capability} & \textbf{Benchmark} & \textbf{InternVL2} \newline \tiny{\color{light-gray}{1B}} & \textbf{Ours} & \textbf{LLaVA-OV} \newline \tiny{\color{light-gray}{0.5B}} & \textbf{Ours} & \textbf{LLaVA-OV} \newline \tiny{\color{light-gray}{7B}} & \textbf{Ours} \\ \midrule
\multirow{7}{*}{\parbox{1.5cm}{Conversation}}
& LLaVA-Wild~\cite{liu2023visual}~\tiny{(all)} \newline \tiny{\color{light-gray}{Realworld Chat}} & 73.80& \cellcolor{front-color} 75.80 \tiny{\color{brown}{+2.00}}& 74.60 & \cellcolor{front-color} 79.20 \tiny{\color{brown}{+4.60}}& 90.70 & \cellcolor{front-color}97.90 \tiny{\color{brown}{+7.20}} \\
& LLaVA-Wild~\cite{liu2023visual}~\tiny{(complex)} \newline \tiny{\color{light-gray}{Realworld Chat}} & 83.60 & \cellcolor{front-color} 82.60 \tiny{\color{gray}{-1.00}} & 78.60 & \cellcolor{front-color} 80.50 \tiny{\color{brown}{+1.90}} & 95.90 & \cellcolor{front-color}100.60 \tiny{\color{brown}{+4.70}} \\
& LLaVA-Wild~\cite{liu2023visual}~\tiny{(conv)} \newline \tiny{\color{light-gray}{Realworld Chat}} & 52.10 & \cellcolor{front-color} 58.30 \tiny{\color{brown}{+6.20}} & 69.60 & \cellcolor{front-color} 72.30 \tiny{\color{brown}{+2.70}} & 81.20 & \cellcolor{front-color}88.10 \tiny{\color{brown}{+6.90}} \\
& LLaVA-Wild~\cite{liu2023visual}~\tiny{(detail)} \newline \tiny{\color{light-gray}{Realworld Chat}} & 85.40 & \cellcolor{front-color} 89.40 \tiny{\color{brown}{+4.00}} & 82.30 & \cellcolor{front-color} 84.50 \tiny{\color{brown}{+2.20}} & 91.80 & \cellcolor{front-color}104.00 \tiny{\color{brown}{+12.20}} \\
& LLaVA-Wilder~\cite{li2024llavanext-strong} (small) \newline \tiny{\color{light-gray}{Realworld Chat}} & 55.80 & \cellcolor{front-color} 57.30 \tiny{\color{brown}{+1.50}} & 52.30 & \cellcolor{front-color} 53.40 \tiny{\color{brown}{+1.10}} & 65.70 & \cellcolor{front-color}71.10 \tiny{\color{brown}{+5.40}} \\
& WildVision~\cite{lu2024wildvision}~\tiny{(elo rate)} \newline \tiny{\color{light-gray}{Model Competition}} & 41.30 & \cellcolor{front-color} 46.20 \tiny{\color{brown}{+4.90}} & 40.70 & \cellcolor{front-color} 44.70 \tiny{\color{brown}{+4.00}} & 50.40 & \cellcolor{front-color}58.90 \tiny{\color{brown}{+8.50}} \\
& WildVision~\cite{lu2024wildvision}~\tiny{(win rates)} \newline \tiny{\color{light-gray}{Model Competition}} & 41.80 & \cellcolor{front-color} 49.00 \tiny{\color{brown}{+7.20}} & 12.60 & \cellcolor{front-color} 14.60 \tiny{\color{brown}{+2.00}} & 15.20 & \cellcolor{front-color}37.20 \tiny{\color{brown}{+22.00}} \\
\cmidrule{2-8}
\multirow{5}{*}{\parbox{1.5cm}{General Knowledge}}
& MME~\cite{fu2023mme}~\tiny{(cog./perp.)} \newline \tiny{\color{light-gray}{Multi-discip}} & 1775 & \cellcolor{front-color} 1815 \tiny{\color{brown}{+40}} & 1488 & \cellcolor{front-color} 1510 \tiny{\color{brown}{+22}} & 1997 & \cellcolor{front-color}2025 \tiny{\color{brown}{+28}} \\
& MMBench~\cite{liu2023mmbench}~\tiny{(cn-dev)} \newline \tiny{\color{light-gray}{Multi-discip}} & 54.70\tiny{\%} & \cellcolor{front-color} 67.89\tiny{\%} \tiny{\color{brown}{+13.19\%}} & 45.80\tiny{\%} & \cellcolor{front-color} 46.40\tiny{\%} \tiny{\color{brown}{+0.60\%}} & 80.49\tiny{\%} & \cellcolor{front-color}80.67\tiny{\%} \tiny{\color{brown}{+0.18\%}} \\
& MMStar~\cite{chen2024we} \newline \tiny{\color{light-gray}{Multi-discip}} & 45.81\tiny{\%} & \cellcolor{front-color} 49.00\tiny{\%} \tiny{\color{brown}{+3.19\%}} & 38.64\tiny{\%} & \cellcolor{front-color} 39.58\tiny{\%} \tiny{\color{brown}{+0.94\%}} & 61.80\tiny{\%} & \cellcolor{front-color}62.58\tiny{\%} \tiny{\color{brown}{+0.78\%}} \\
& SeedBench2-Plus~\cite{li2024seed} \newline \tiny{\color{light-gray}{Multi-discip}} & 60.12\tiny{\%} & \cellcolor{front-color} 60.12\tiny{\%} \tiny{\color{brown}{+0.00\%}} & 53.85\tiny{\%} & \cellcolor{front-color} 54.27\tiny{\%} \tiny{\color{brown}{+0.42\%}} & 64.87\tiny{\%} & \cellcolor{front-color}65.35\tiny{\%} \tiny{\color{brown}{+0.48\%}} \\
& VQAv2~\cite{antol2015vqa}~\tiny{(lite)} \newline \tiny{\color{light-gray}{Multi-discip}} & 72.25\tiny{\%} & \cellcolor{front-color} 71.84\tiny{\%} \tiny{\color{gray}{-0.41\%}} & 74.60\tiny{\%} & \cellcolor{front-color} 74.68\tiny{\%} \tiny{\color{brown}{+0.08\%}} & 79.98\tiny{\%} & \cellcolor{front-color}80.28\tiny{\%} \tiny{\color{brown}{+0.30\%}} \\
\cmidrule{2-8}
\multirow{4}{*}{\parbox{1.5cm}{Chart and Document}}
& AI2D~\cite{kembhavi2016diagram} \newline \tiny{\color{light-gray}{Science Diagrams}} & 72.38\tiny{\%} & \cellcolor{front-color} 72.80\tiny{\%} \tiny{\color{brown}{+0.42\%}} & 56.93\tiny{\%} & \cellcolor{front-color} 56.87\tiny{\%} \tiny{\color{gray}{-0.06\%}} & 81.41\tiny{\%} & \cellcolor{front-color} 81.22\tiny{\%} \tiny{\color{gray}{-0.19\%}} \\
& ChartQA~\cite{masry2022chartqa}~\tiny{(val-lite)} \newline \tiny{\color{light-gray}{Chart Understanding}} & 65.60\tiny{\%} & \cellcolor{front-color} 66.80\tiny{\%} \tiny{\color{brown}{+1.20\%}} & 51.60\tiny{\%} & \cellcolor{front-color} 52.60\tiny{\%} \tiny{\color{brown}{+1.00\%}} & 74.00\tiny{\%} & \cellcolor{front-color} 74.50\tiny{\%} \tiny{\color{brown}{+0.50\%}} \\
& DocVQA~\cite{mathew2021docvqa}~\tiny{(val-lite)} \newline \tiny{\color{light-gray}{Document Understanding}} & 81.90\tiny{\%} & \cellcolor{front-color} 82.51\tiny{\%} \tiny{\color{brown}{+0.61\%}} & 66.17\tiny{\%} & \cellcolor{front-color} 67.07\tiny{\%} \tiny{\color{brown}{+0.90\%}} & 84.34\tiny{\%} & \cellcolor{front-color} 86.11\tiny{\%} \tiny{\color{brown}{+1.77\%}} \\
& InfoVQA~\cite{mathew2022infographicvqa}~\tiny{(val-lite)} \newline \tiny{\color{light-gray}{Infographic Understanding}} & 51.73\tiny{\%} & \cellcolor{front-color} 52.26\tiny{\%} \tiny{\color{brown}{+0.53\%}} & 40.17\tiny{\%} & \cellcolor{front-color} 40.49\tiny{\%} \tiny{\color{brown}{+0.32\%}} & 67.07\tiny{\%} & \cellcolor{front-color} 67.40\tiny{\%} \tiny{\color{brown}{+0.33\%}} \\
\cmidrule{2-8}
\multirow{3}{*}{\parbox{1.5cm}{OCR}}
& OCRBench~\cite{liu2024hidden} \newline \tiny{\color{light-gray}{Comprehensive OCR}} & 75.20\tiny{\%} & \cellcolor{front-color} 77.11\tiny{\%} \tiny{\color{brown}{+1.91\%}} & 57.70\tiny{\%} & \cellcolor{front-color} 60.20\tiny{\%} \tiny{\color{brown}{+2.50\%}} & 62.30\tiny{\%} & \cellcolor{front-color} 69.30\tiny{\%} \tiny{\color{brown}{+7.00\%}} \\
& TextVQA~\cite{singh2019towards}~\tiny{(val)} \newline \tiny{\color{light-gray}{Text Reading}} & 69.85\tiny{\%} & \cellcolor{front-color} 72.12\tiny{\%} \tiny{\color{brown}{+2.27\%}} & 65.87\tiny{\%} & \cellcolor{front-color} 66.60\tiny{\%} \tiny{\color{brown}{+0.73\%}} & 75.99\tiny{\%} & \cellcolor{front-color} 76.05\tiny{\%} \tiny{\color{brown}{+0.06\%}} \\
& WebSRC~\cite{chen2021websrc}~\tiny{(val)} \newline \tiny{\color{light-gray}{Web-based Structural Reading}} & 68.20\tiny{\%} & \cellcolor{front-color} 68.80\tiny{\%} \tiny{\color{brown}{+0.60\%}} & 65.90\tiny{\%} & \cellcolor{front-color} 68.30\tiny{\%} \tiny{\color{brown}{+2.40\%}} & 88.70\tiny{\%} & \cellcolor{front-color} 89.20\tiny{\%} \tiny{\color{brown}{+0.50\%}} \\
\cmidrule{2-8}
\multirow{3}{*}{\parbox{1.5cm}{Real-World}}
& MME-RealWorld~\cite{zhang2024mme}~\tiny{(en-lite)} \newline\tiny{\color{light-gray}{Multi-discip \& High-Resolution}} & 33.61\tiny{\%} & \cellcolor{front-color} 36.58\tiny{\%} \tiny{\color{brown}{+2.97\%}} & 34.55\tiny{\%} & \cellcolor{front-color} 34.39\tiny{\%} \tiny{\color{gray}{-0.16\%}} & 48.36\tiny{\%} & \cellcolor{front-color} 46.95\tiny{\%} \tiny{\color{gray}{-1.41\%}} \\
& MME-RealWorld~\cite{zhang2024mme}~\tiny{(cn)} \newline \tiny{\color{light-gray}{Multi-discip \& High-Resolution}} & 44.14\tiny{\%} & \cellcolor{front-color} 43.11\tiny{\%} \tiny{\color{ gray}{-1.03\%}} & 32.09\tiny{\%} & \cellcolor{front-color} 31.11\tiny{\%} \tiny{\color{gray}{-0.98\%}} & 54.01\tiny{\%} & \cellcolor{front-color} 53.39\tiny{\%} \tiny{\color{brown}{-0.62\%}} \\
& RealWorldQA \newline \tiny{\color{light-gray}{Realworld QA}} & 51.50\tiny{\%} & \cellcolor{front-color} 54.90\tiny{\%} \tiny{\color{brown}{+3.40\%}} & 55.42\tiny{\%} & \cellcolor{front-color} 55.16\tiny{\%} \tiny{\color{gray}{-0.26\%}} & 66.41\tiny{\%} & \cellcolor{front-color} 65.75\tiny{\%} \tiny{\color{gray}{-0.66\%}} \\
\cmidrule{2-8}
\multirow{4}{*}{Math}
& MathVista~\cite{lu2024mathvista}~\tiny{(cot)} \newline \tiny{\color{light-gray}{General Math Understanding}} & 49.60\tiny{\%} & \cellcolor{front-color} 49.90\tiny{\%} \tiny{\color{brown}{+0.30\%}} & 32.30\tiny{\%} & \cellcolor{front-color} 32.70\tiny{\%} \tiny{\color{brown}{+0.40\%}} & 59.10\tiny{\%} & \cellcolor{front-color} 61.60\tiny{\%} \tiny{\color{brown}{+2.50\%}} \\
& MathVista~\cite{lu2024mathvista}~\tiny{(format)} \newline \tiny{\color{light-gray}{General Math Understanding}} & 53.20\tiny{\%} & \cellcolor{front-color} 53.40\tiny{\%} \tiny{\color{brown}{+0.20\%}} & 36.00\tiny{\%} & \cellcolor{front-color} 36.30\tiny{\%} \tiny{\color{brown}{+0.30\%}} & 62.50\tiny{\%} & \cellcolor{front-color} 62.20\tiny{\%} \tiny{\color{gray}{-0.30\%}} \\
& MathVista~\cite{lu2024mathvista}~\tiny{(solution)} \newline \tiny{\color{light-gray}{General Math Understanding}} & 49.60\tiny{\%} & \cellcolor{front-color} 49.30\tiny{\%} \tiny{\color{gray}{-0.30\%}} & 30.50\tiny{\%} & \cellcolor{front-color} 32.50\tiny{\%} \tiny{\color{brown}{+2.00\%}} & 58.80\tiny{\%} & \cellcolor{front-color} 61.10\tiny{\%} \tiny{\color{brown}{+2.30\%}} \\
& MathVerse~\cite{zhang2024mathverse}~\tiny{(vision-mini)} \newline \tiny{\color{light-gray}{Professional Math Reasoning}} & 12.31\tiny{\%} & \cellcolor{front-color} 12.79\tiny{\%} \tiny{\color{brown}{+0.48\%}} & 17.51\tiny{\%} & \cellcolor{front-color} 17.64\tiny{\%} \tiny{\color{brown}{+0.13\%}} & 16.37\tiny{\%} & \cellcolor{front-color} 18.53\tiny{\%} \tiny{\color{brown}{+2.16\%}} \\
\cmidrule{2-8}
\multirow{7}{*}{\parbox{1.5cm}{Hallucination}}
& POPE~\cite{pope}~\tiny{(adversarial)} \newline \tiny{\color{light-gray}{Object Hallucination.}} & 86.82\tiny{\%} & \cellcolor{front-color} 86.87\tiny{\%} \tiny{\color{brown}{+0.05\%}} & 86.04\tiny{\%} & \cellcolor{front-color} 86.56\tiny{\%} \tiny{\color{brown}{+0.52\%}} & 87.08\tiny{\%} & \cellcolor{front-color}87.68\tiny{\%} \tiny{\color{brown}{+0.60\%}} \\
& POPE~\cite{pope}~\tiny{(popular)} \newline \tiny{\color{light-gray}{Object Hallucination.}} & 88.30\tiny{\%} & \cellcolor{front-color} 88.57\tiny{\%} \tiny{\color{brown}{+0.27\%}} & 87.37\tiny{\%} & \cellcolor{front-color} 88.26\tiny{\%} \tiny{\color{brown}{+0.89\%}} & 88.32\tiny{\%} & \cellcolor{front-color}89.02\tiny{\%} \tiny{\color{brown}{+0.70\%}} \\
& POPE~\cite{pope}~\tiny{(random)} \newline \tiny{\color{light-gray}{Object Hallucination.}} & 89.87\tiny{\%} & \cellcolor{front-color} 90.45\tiny{\%} \tiny{\color{brown}{+0.58\%}} & 88.30\tiny{\%} & \cellcolor{front-color} 89.30\tiny{\%} \tiny{\color{brown}{+1.00\%}} & 89.60\tiny{\%} & \cellcolor{front-color}90.62\tiny{\%} \tiny{\color{brown}{+1.02\%}} \\
& MMHal~\cite{mmhal-bench}~\tiny{(hal rate $\downarrow$)} \newline \tiny{\color{light-gray}{General Hallucination}} & 55.21\tiny{\%} & \cellcolor{front-color} 55.38\tiny{\%} \tiny{\color{gray}{-0.17\%}} & 48.96\tiny{\%} & \cellcolor{front-color} 46.25\tiny{\%} \tiny{\color{brown}{+2.71\%}} & 38.54\tiny{\%} & \cellcolor{front-color}38.54\tiny{\%} \tiny{\color{brown}{+0.00\%}} \\
& MMHal~\cite{mmhal-bench}~\tiny{(avg score)} \newline \tiny{\color{light-gray}{General Hallucination}} & 3.02 & \cellcolor{front-color} 3.10 \tiny{\color{brown}{+0.08}} & 3.33 & \cellcolor{front-color} 3.42 \tiny{\color{brown}{+0.09}} & 3.22 & \cellcolor{front-color}4.08 \tiny{\color{brown}{+0.86}} \\
& Obj-Hal~\cite{li2023evaluating}~\tiny{(chair-i$\downarrow$)} \newline \tiny{\color{light-gray}{Object Hallucination.}} & 8.30 & \cellcolor{front-color} 7.81 \tiny{\color{brown}{+0.49}} & 9.70 & \cellcolor{front-color} 9.12 \tiny{\color{brown}{+0.58}} & 8.52 & \cellcolor{front-color}7.69 \tiny{\color{brown}{+0.83}} \\
& Obj-Hal~\cite{li2023evaluating}~\tiny{(chair-s$\downarrow$)} \newline \tiny{\color{light-gray}{Object Hallucination.}} &38.67 & \cellcolor{front-color} 37.00 \tiny{\color{brown}{+1.67}} & 42.67 & \cellcolor{front-color}42.33 \tiny{\color{brown}{+0.34}} & 44.00 & \cellcolor{front-color}41.67 \tiny{\color{brown}{+2.33}} \\
\cmidrule{2-8}
\multirow{4}{*}{\parbox{1.5cm}{Video Understanding}}
& Video-MME~\cite{fu2024video}~\tiny{(w. caption)} \newline\tiny{\color{light-gray}{Multi-discip}} & 42.74\tiny{\%} & \cellcolor{front-color} 42.76\tiny{\%} \tiny{\color{brown}{+0.02\%}} &48.22\tiny{\%} & \cellcolor{front-color} 48.42\tiny{\%} \tiny{\color{brown}{+0.20\%}} & 61.61\tiny{\%} & \cellcolor{front-color}61.81\tiny{\%} \tiny{\color{brown}{+0.20\%}} \\
& Video-MME~\cite{fu2024video}~\tiny{(wo. caption)} \newline\tiny{\color{light-gray}{Multi-discip}} & 45.66\tiny{\%} & \cellcolor{front-color} 45.71\tiny{\%} \tiny{\color{brown}{+0.05\%}} & 43.92\tiny{\%} & \cellcolor{front-color} 44.00\tiny{\%} \tiny{\color{brown}{+0.08\%}} & 58.29\tiny{\%} & \cellcolor{front-color}58.33\tiny{\%} \tiny{\color{brown}{+0.04\%}} \\
& VideoChatGPT~\cite{Maaz2023VideoChatGPT} \newline \tiny{\color{light-gray}{Video Conversation}} & 2.26 & \cellcolor{front-color} 2.59 \tiny{\color{brown}{+0.33}} & 2.56 & \cellcolor{front-color} 2.66 \tiny{\color{brown}{+0.10}} & 2.87 & \cellcolor{front-color}3.22 \tiny{\color{brown}{+0.35}} \\
& VideoDC~\cite{li2024llavanext-strong} \newline \tiny{\color{light-gray}{Video Detail Description}} & 2.91 & \cellcolor{front-color} 3.07 \tiny{\color{brown}{+0.16}} & 2.88 & \cellcolor{front-color} 2.96 \tiny{\color{brown}{+0.08}} & 3.32 & \cellcolor{front-color}3.41 \tiny{\color{brown}{+0.09}} \\
\cmidrule{2-8}
\multirow{2}{*}{\parbox{1.5cm}{Multi-Image}}
& LLAVA-Next-Interleave~\cite{li2024llava}~\tiny{(in-domain)} \newline\tiny{\color{light-gray}{in-domian}} & 34.78\tiny{\%} & \cellcolor{front-color} 35.72\tiny{\%} \tiny{\color{brown}{+0.94\%}} & 42.29\tiny{\%} & \cellcolor{front-color} 43.49\tiny{\%} \tiny{\color{brown}{+1.20\%}} & 60.85\tiny{\%} & \cellcolor{front-color}61.12\tiny{\%} \tiny{\color{brown}{+0.27\%}} \\
& MMMU-Pro~\cite{yue2024mmmu}~\tiny{(vision)} \newline\tiny{\color{light-gray}{Multi-discip}} & 1.11\tiny{\%} & \cellcolor{front-color} 1.52\tiny{\%} \tiny{\color{brown}{+0.41\%}} & 12.78\tiny{\%} & \cellcolor{front-color} 13.89\tiny{\%} \tiny{\color{brown}{+1.11\%}} & 14.51\tiny{\%} & \cellcolor{front-color}15.84\tiny{\%} \tiny{\color{brown}{+1.33\%}} \\
\midrule
\end{tabular}
}
\vspace{2mm}
\end{table}
front-colorHTMLFDEFF5
Table tab:model_comparison_safety: Performance variations after alignment across -SafeBench, comparing multiple models under our alignment strategy.
\begin{table}[t!]
\caption{\textbf{Performance variations after alignment across \abbr-SafeBench}, comparing multiple models under our alignment strategy.}
\label{tab:model_comparison_safety}
\centering
\setlength{\tabcolsep}{4pt}
\renewcommand{1.0}{1.0}
\scriptsize
\resizebox{\textwidth}{!}{
\begin{tabular}{p{3cm}p{1.5cm}p{1.5cm}p{1.5cm}p{1.5cm}p{1.5cm}p{1.5cm}}
\toprule
\textbf{Benchmark} & \textbf{InternVL2} \newline \tiny{\color{light-gray}{1B}} & \textbf{Ours} & \textbf{LLaVA-OV} \newline \tiny{\color{light-gray}{0.5B}} & \textbf{Ours} & \textbf{LLaVA-OV} \newline \tiny{\color{light-gray}{7B}} & \textbf{Ours} \\ \midrule
Adv target~\tiny{$\downarrow$} \newline\tiny{\color{light-gray}{Adversarial Attack}} & 56.0\% & \cellcolor{front-color} 50.0\% \tiny{\color{brown}{+5.0\%}} & 54.0\% & \cellcolor{front-color} 35.0\% \tiny{\color{brown}{+19.0\%}}& 37.0\% & \cellcolor{front-color} 40.0\% \tiny{\color{gray}{-3.0\%}} \\
Adv untarget~\tiny{$\uparrow$} \newline\tiny{\color{light-gray}{Adversarial Attack}} & 52.5\% & \cellcolor{front-color} 56.0\% \tiny{\color{brown}{+3.5\%}}& 66.0\% & \cellcolor{front-color} 71.0\% \tiny{\color{brown}{+5\%}}& 66.5\% & \cellcolor{front-color} 70.0\% \tiny{\color{brown}{+3.5\%}} \\
Crossmodel ASR~\tiny{$\downarrow$} \newline\tiny{\color{light-gray}{Cross-modal Jailbreak}} & 0.0\% & \cellcolor{front-color} 0.0\% \tiny{\color{brown}{+0.0\%}}& 72.2\% & \cellcolor{front-color} 38.9\% \tiny{\color{brown}{+33.3\%}}& 16.7\% & \cellcolor{front-color} 0.0\% \tiny{\color{brown}{+16.7\%}} \\
Crossmodel RtA~\tiny{$\uparrow$} \newline\tiny{\color{light-gray}{Cross-modal Jailbreak}} & 100.0\% & \cellcolor{front-color} 100.0\% \tiny{\color{brown}{+0.0\%}}& 22.2\% & \cellcolor{front-color} 50.0\% \tiny{\color{brown}{+27.8\%}}& 88.9\% & \cellcolor{front-color} 100.0\% \tiny{\color{brown}{+11.1\%}}\\
Multimodel ASR~\tiny{$\downarrow$} \newline\tiny{\color{light-gray}{Multimodal Jailbreak}} & 43.2\% & \cellcolor{front-color} 43.2\% \tiny{\color{brown}{+0.0\%}}& 42.2\% & \cellcolor{front-color} 27.7\% \tiny{\color{brown}{+14.5\%}}& 41.2\% & \cellcolor{front-color} 8.3\% \tiny{\color{brown}{+31.9\%}}\\
Multimodel RtA~\tiny{$\uparrow$} \newline\tiny{\color{light-gray}{Multimodal Jailbreak}} & 18.0\% & \cellcolor{front-color} 17.4\% \tiny{\color{gray}{-0.6\%}}& 12.4\% & \cellcolor{front-color} 23.2\% \tiny{\color{brown}{+10.8\%}}& 62.0\% & \cellcolor{front-color} 88.3\% \tiny{\color{brown}{+26.3\%}}\\
Typographic ASR~\tiny{$\downarrow$} \newline\tiny{\color{light-gray}{Typographic Jailbreak}} & 10.5\% & \cellcolor{front-color} 7.4\% \tiny{\color{brown}{+3.1\%}}& 26.3\% & \cellcolor{front-color} 35.2\% \tiny{\color{gray}{-8.9\%}}& 5.8\% & \cellcolor{front-color} 0.0\% \tiny{\color{brown}{+5.8\%}}\\
Typographic RtA~\tiny{$\uparrow$} \newline\tiny{\color{light-gray}{Typographic Jailbreak}} & 73.7\% & \cellcolor{front-color} 74.6\% \tiny{\color{brown}{+0.9\%}}& 17.0\% & \cellcolor{front-color} 27.5\% \tiny{\color{brown}{+10.5\%}}& 79.5\% & \cellcolor{front-color} 95.8\% \tiny{\color{brown}{+16.3\%}}\\
Risk~\tiny{$\uparrow$} \newline\tiny{\color{light-gray}{Risk identification}} & 49.6\% & \cellcolor{front-color} 58.6\% \tiny{\color{brown}{+9.0\%}} & 65.8\% & \cellcolor{front-color} 67.4\% \tiny{\color{brown}{+1.6\%}}& 82.0\% & \cellcolor{front-color} 76.0\% \tiny{\color{gray}{-6.0\%}}\\
NSFW~\tiny{text$\downarrow$} \newline\tiny{\color{light-gray}{NSFW Jailbreak}} & 89.0\% & \cellcolor{front-color} 27.1\% \tiny{\color{brown}{+61.9\%}}& 94.4\% & \cellcolor{front-color} 64.2\% \tiny{\color{brown}{+30.2\%}}& 60.4\% & \cellcolor{front-color} 10.6\% \tiny{\color{brown}{+49.8\%}}\\
NSFW~\tiny{img$\downarrow$} \newline\tiny{\color{light-gray}{NSFW Jailbreak}} & 81.2\% & \cellcolor{front-color} 64.7\% \tiny{\color{brown}{+16.5\%}}& 97.5\% & \cellcolor{front-color} 81.6\% \tiny{\color{brown}{+15.9\%}}& 80.1\% & \cellcolor{front-color} 24.2\% \tiny{\color{brown}{+55.9\%}} \\
\hline
Unsafety~\tiny{ $\downarrow$} \newline\tiny{\color{light-gray}{Average performance of $\downarrow$}} & 46.6\% & \cellcolor{front-color} 38.9\% \tiny{\color{brown}{+7.7\%}}& 65.4\% & \cellcolor{front-color} 47.1\% \tiny{\color{brown}{+18.3\%}} & 40.2\% & \cellcolor{front-color} 13.9\% \tiny{\color{brown}{+26.3\%}}\\
Safety~\tiny{ $\uparrow$} \newline\tiny{\color{light-gray}{Average performance of $\uparrow$}} & 31.9\% & \cellcolor{front-color} 41.3\% \tiny{\color{brown}{+9.4\%}}& 36.7\% & \cellcolor{front-color} 47.8\% \tiny{\color{brown}{+11.1\%}}& 75.8\% & \cellcolor{front-color} 85.4\% \tiny{\color{brown}{+9.6\%}} \\
\midrule
\end{tabular}
}
\vspace{2mm}
\end{table}
Evaluation of -Reward¶
In this section, we evaluate the effectiveness of -Reward and highlight several noteworthy experimental observations. The results are presented in Table tab:rm_performance_comparison and Table tab:reward_model_comparison.
Existing reward models exhibit significant overfitting. As shown in Table tab:rm_performance_comparison, LLaVA-Critic's performance on -Reward-Bench is suboptimal, with a considerable gap compared to GPT-4o. This can likely be attributed to the overfitting of existing reward models to their training data, which predominantly consists of conversational datasets and real-world images. Consequently, while LLaVA-Critic demonstrates notable improvements over its baseline, LLaVA-OV-7BBoth models use identical prompts for tasks such as captioning and long-form dialogue., its performance in other categories, such as MCQ and more diverse tasks, remains limited.
Closed-source models like GPT-4o consistently deliver competitive performance. Across both Table tab:rm_performance_comparison and Table tab:reward_model_comparison, closed-source models such as GPT-4o demonstrate superior generalization capabilities compared to open-source alternatives, even those with significantly larger parameter sizes (e.g., 72B models). This observation underscores the robustness of closed-source approaches in handling diverse multimodal tasks and maintaining high performance across various metrics.
-Reward sets a new benchmark for open-source models, rivaling closed-source systems. In both benchmarks, -Reward achieves results comparable to or exceeding GPT-4o's performance, while significantly outperforming most open-source models, such as LLaMA-3.2-90B-Vision-Instruct and Qwen2-VL-72B-Instruct. Notably, on our custom benchmark, -Reward demonstrates a substantial lead over GPT-4o, further justifying its selection as the reward signal for training algorithms. Its robust performance across diverse metrics highlights its effectiveness and adaptability.
The importance of an effective critic in reward modeling. The results in Table tab:rm_performance_comparison underscore the critical role of an effective critic in reward modeling. When the reward head is directly trained using pair-wise datasets, the ACC+ stabilizes around 50%. By incorporating human annotations as the learning target—allowing the model to first learn evaluation reasoning and then perform scoring—the ACC+ improves by a consistent 5%. However, human annotations alone may not serve as an optimal training target due to their brevity or conversational style. To address this, we expand the human annotations using the model itself, producing enriched annotations that further enhance reward model training quality. This results in a significant 17% improvement in ACC+ compared to the baseline. Finally, during evaluation, when human annotations are directly provided as the critic (i.e., scoring is based on human-provided evaluations rather than model-generated critics), both ACC and ACC+ reach approximately 90%. This demonstrates the pivotal role of evaluation quality in the overall effectiveness of reward models.
Multiple sampling of critiques does not yield significant performance gains. When the model generates critiques with high variability, multiple sampling is often used to compute scores and then take the average [yu2024self]. This approach has proven effective in related LLM research. However, in our experiments, we observed that when we lowered the sampling temperature and computed rewards multiple times, the performance actually declined. The reason for this is that during the sampling process, there is occasionally a critique that is inaccurate. Since our model is already capable of generating reasonably accurate critiques due to its alignment with human annotations, the extra, time-consuming sampling process does not provide additional benefits and can even have a negative impact on performance.
Table* tab:rm_performance_comparison: Performance comparison across metrics and methods on -RewardBench. -Reward (w/o. Task 1) represents training the LLaVA-OV-7B model to score pair-wise samples while excluding Task 1. -Reward (w/o. enhanced annotations) involves learning human-provided annotations, followed by scoring. -Reward (inference w. GT annotation) uses ground truth annotations during inference.
\begin{table*}[ht]
\centering
\caption{
\textbf{Performance comparison across metrics and methods on \abbr-RewardBench}.
\textit{\abbr-Reward (w/o. Task 1)} represents training the LLaVA-OV-7B model to score pair-wise samples while excluding Task 1.
\textit{\abbr-Reward (w/o. enhanced annotations)} involves learning human-provided annotations, followed by scoring.
\textit{\abbr-Reward (inference w. GT annotation)} uses ground truth annotations during inference.
}
\label{tab:rm_performance_comparison}
\resizebox{\textwidth}{!}{
\begin{tabular}{lcccccccc|cccccccc}
\toprule
\textbf{Method} & \multicolumn{2}{c}{\textbf{LLaVA-OV-7B}} & \multicolumn{2}{c}{\textbf{\begin{tabular}[c]{@{}c@{}}LlaVA-Critic\\ (Pointwise)\end{tabular}}} & \multicolumn{2}{c}{\textbf{\begin{tabular}[c]{@{}c@{}}LlaVA-Critic\\ (Pairwise)\end{tabular}}} & \multicolumn{2}{c|}{\textbf{GPT-4o}} & \multicolumn{2}{c}{\textbf{\begin{tabular}[c]{@{}c@{}}\abbr-Reward\\ (w/o. Task 1)\end{tabular}}} & \multicolumn{2}{c}{\textbf{\begin{tabular}[c]{@{}c@{}}\abbr-Reward\\ (w/o. enhanced \\ annotations)\end{tabular}}} & \multicolumn{2}{c}{\textbf{\abbr-Reward}} & \multicolumn{2}{c}{\textbf{\begin{tabular}[c]{@{}c@{}}\abbr-Reward\\ (inference w. \\ GT annotation)\end{tabular}}} \\
\textbf{Metric} & \textbf{ACC} & \textbf{ACC+} & \textbf{ACC} & \textbf{ACC+} & \textbf{ACC} & \textbf{ACC+} & \textbf{ACC} & \textbf{ACC+} & \textbf{ACC} & \textbf{ACC+} & \textbf{ACC} & \textbf{ACC+} & \textbf{ACC} & \textbf{ACC+} & \textbf{ACC} & \textbf{ACC+} \\
\midrule
Mcq & 0.14 & 0.00 & 0.38 & 0.10 & 0.23 & 0.00 & 0.69 & 0.20 & 0.90 & 0.80 & 0.83 & 0.70 & 0.93 & 0.70 & \textbf{1.00} & \textbf{1.00} \\
Long & 0.11 & 0.00 & 0.49 & 0.20 & 0.54 & 0.30 & 0.95 & 0.90 & 0.70 & 0.40 & 0.92 & 0.80 & 1.00 & 1.00 & \textbf{1.00} & \textbf{1.00} \\
Short & 0.29 & 0.20 & 0.38 & 0.20 & 0.24 & 0.10 & 0.56 & 0.40 & 0.79 & 0.60 & 0.68 & 0.40 & 0.71 & 0.50 & \textbf{1.00} & \textbf{1.00} \\
Safety & 0.41 & 0.00 & 0.62 & 0.17 & 0.28 & 0.17 & \textbf{0.72} & \textbf{0.33} & 0.69 & \textbf{0.33} & 0.69 & 0.17 & 0.66 & 0.17 & 0.69 & 0.17 \\
Video & 0.32 & 0.10 & 0.40 & 0.20 & 0.52 & 0.20 & 0.80 & 0.60 & 0.70 & 0.60 & 0.80 & 0.60 & \textbf{0.92} & 0.80 & \textbf{0.92} & \textbf{0.90} \\ \hline
Overall & 0.24 & 0.07 & 0.45 & 0.17 & 0.35 & 0.15 & 0.74 & 0.50 & 0.75 & 0.50 & 0.79 & 0.57 & 0.85 & 0.67 & \textbf{0.93} & \textbf{0.87} \\
\bottomrule
\end{tabular}
}
\end{table*}
Table tab:reward_model_comparison: Performance comparison of our reward model (-Reward) with existing open-source and private multi-modal models. -Reward-7B outperforms existing 72B open-source multi-modal models and several highly competitive closed-source models.
\begin{table}[htbp]
\centering
\caption{\textbf{Performance comparison of our reward model (\abbr-Reward) with existing open-source and private multi-modal models.} \abbr-Reward-7B outperforms existing 72B open-source multi-modal models and several highly competitive closed-source models.}
\label{tab:reward_model_comparison}
\begin{tabular}{lcccc}
\toprule
\textbf{Model} & \textbf{General} & \textbf{Hallucination} & \textbf{Reasoning} & \textbf{Avg} \\
\midrule
VITA-1.5~\cite{fu2025vita} & 18.55 & 8.93 & 22.11 & 16.48 \\
SliME-8B~\cite{zhang2024beyond} & 7.23 & 27.09 & 18.6 & 19.04 \\
deepseek-vl2~\cite{wu2024deepseekvl2mixtureofexpertsvisionlanguagemodels} & 29.70 & 23.80 & 50.90 & 34.80 \\
Phi-3.5-vision-instruct~\cite{abdin2024phi} & 28.00 & 22.40 & 56.60 & 35.67 \\
llava-onevision-qwen2-7b-ov~\cite{li2024llava}& 32.20 & 20.10 & 57.10 & 36.47 \\
Molmo-7B-D-0924~\cite{deitke2024molmo} & 31.10 & 31.80 & 56.20 & 39.70 \\
Pixtral-12B-2409~\cite{agrawal2024pixtral} & 35.60 & 25.90 & 59.90 & 40.47 \\
Qwen2-VL-72B-Instruct~\cite{wang2024qwen2} & 38.10 & 32.80 & 58.00 & 42.97 \\
NVLM-D-72B~\cite{dai2024nvlm} & 38.90 & 31.60 & 62.00 & 44.17 \\
InternVL2-26B~\cite{chen2024far} & 39.30 & 36.90 & 60.80 & 45.67 \\ \hline
\Gray
\multicolumn{5}{c}{\textit{Private models}} \\
GPT-4o-mini (2024-07-18) & 41.70 & 34.50 & 58.20 & 44.80 \\
Claude-3.5-Sonnet (2024-06-22) & 43.40 & 55.00 & 62.30 & 53.57 \\
GPT-4o (2024-08-06) & 49.10 & 67.60 & 70.50 & 62.40 \\
Gemini-1.5-Pro (2024-09-24) & 50.80 & 72.50 & 64.20 & 62.50 \\
\midrule\Gray
\multicolumn{5}{c}{Ours} \\
\abbr-Reward-7B & 45.04 & 50.45 & 57.55 & 50.15 \\
\bottomrule
\end{tabular}
\end{table}
Self-Improvement of Small-Scale MLLMs is Currently Unrealistic¶
Figure fig:self-improvement: Performance comparison across datasets using various methods based on the LLaVA-Ov-7B model as the baseline. Baseline" represents the initial performance without post-training.LLAVA-RLHF (LLAVA-RLHF)'' indicates that both the post-training dataset and the reward model come from the LLAVA-RLHF dataset, with the reward model being trained using LLaVA-Ov-7B as the starting checkpoint for fairness. $^s$ '' reflects results generated on our dataset, where responses are self-sampled (default sample size: 8) and ranked using different reward signals to create DPO pairs. \(^h\) (Human)'' involves DPO training directly using our dataset, where responses are sampled from other models, and reward signals are provided by experts.
While recent work on MLLMs explores the concept of self-improvement, these efforts largely focus on specific domains, such as conversational systems [xiong2024llava]. In this section, we present an alternative perspective distinct from the LLM domain, arguing that MLLMs, particularly small models (fewer than 7B parameters), currently face significant challenges in achieving comprehensive performance improvements through self-improvement. Our experimental results, illustrated in Figure fig:self-improvement, suggest two primary reasons for this limitation:
1. Model capacity constraints. For tasks involving long-form or conversational data, sampling multiple responses often results in at least one reasonably good answer, thereby leading to noticeable improvements. However, for more challenging tasks, such as multiple-choice questions or scientific reasoning, smaller models struggle to generate correct answers even after extensive sampling. In our experiments, where the maximum number of samples reached eight, we observed instances where the model produced identical incorrect responses or consistently incorrect outputs across all samples for some challenging multiple-choice questions.
2. Limitations in reward signal quality. Most existing multimodal reward models are trained on datasets with limited diversity, such as VLFeedback and LLaVA-RLHF. These datasets predominantly focus on natural images, human dialogue, or related scenarios, raising concerns about overfitting. When preference datasets encompass broader domains, such as mathematical reasoning, chart understanding, or other specialized fields, reward models trained on existing datasets fail to provide effective reward signals. Consequently, it becomes challenging to identify and select better samples.
These two limitations make it difficult, at the current stage, to enable MLLMs to generate responses on diverse datasets, annotate them with reward models, and iteratively improve through self-improvement cycles, as has been achieved in LLM alignment. While our experiments confirm that better reward models can lead to marginal improvements, the results remain far inferior to training with high-quality, human-annotated contrastive samples.
Conclusion and Future Work¶
In this work, we introduced ****, a high-quality, fine-grained dataset specifically designed to advance the alignment of MLLMs. Unlike prior works that focus on specific tasks, our dataset and alignment approach aim to holistically improve performance across diverse dimensions. Even with preliminary improvements to reward modeling and optimization algorithms, we observed significant and consistent gains across almost all evaluation benchmarks, underscoring the potential of comprehensive alignment strategies.
Looking ahead, we see great opportunities to further unlock the value of our dataset. Its rich annotation granularity, such as per-dimension scores and ranking rationales, remains underutilized in current alignment algorithms. Future work will focus on leveraging this granularity with advanced optimization techniques, integrating high-resolution data to address limitations in specific benchmarks, and scaling the dataset efficiently using semi-automated strategies. We believe these efforts will not only push MLLM alignment to new heights but also set a foundation for broader, more generalizable multimodal learning frameworks.
center \(\;\) ————Appendix———— center
hidelinks
Related Work¶
Multimodal large language models have seen remarkable progress in recent years, with significant advancements in both performance and capabilities. Leveraging cutting-edge LLMs such as GPTs [gpt4,brown2020language], LLaMA [touvron2023llama,touvron2023llama2], Alpaca [taori2023stanford], Vicuna [chiang2023vicuna], and Mistral [jiang2023mistral], MLLMs are increasingly demonstrating enhanced multimodal capabilities, especially through end-to-end training approaches. These advancements have been crucial in enabling models to handle a range of multimodal tasks, including image-text alignment, reasoning, and instruction following, while addressing challenges related to data fusion across different modalities. Recent open-source MLLMs such as Otter [li2023otter], mPLUG-Owl [ye2023mplug], LLaVA [liu2023visual], Qwen-VL [bai2023qwen], Cambrian-1 [tong2024cambrian], Mini-Gemini [li2024mini], MiniCPM-V 2.5 [hu2024minicpm], DeepSeek-VL [lu2024deepseek], SliME [zhang2024beyond] and VITA [fu2024vita,fu2025vita] have contributed to solving some of the most fundamental multimodal problems, such as improving vision-language alignment, reasoning, and following instructions. These models focus on enhancing multimodal understanding by integrating vision with language, allowing for more nuanced and context-aware interactions. Some of the most notable open-source models, such as InternLM-XComposer-2.5 [zhang2023internlm] and InternVL-2 [chen2023internvl], have exhibited impressive progress in multimodal understanding, closely competing with proprietary models across a range of multimodal benchmarks. However, despite these achievements, there is still a noticeable gap in security and alignment when compared to closed-source models. As highlighted by recent studies [zhang2024mme], most open-source MLLMs have not undergone rigorous, professional alignment processes, which has hindered their ability to effectively align with human preferences. This gap in alignment remains one of the key challenges for open-source models, and improving model safety and alignment to human values will be a crucial area of future research.
MLLM Alignment. With the rapid development of MLLMs, various alignment algorithms have emerged, showcasing different application scenarios and optimization goals. For instance, in the image domain, Fact-RLHF [sun2023aligning] is the first multimodal RLHF algorithm, and more recently, LLAVA-CRITIC [xiong2024llava] has demonstrated strong potential with an iterative DPO strategy. These algorithms have shown significant impact on reducing hallucinations and improving conversational capabilities [zhang2024debiasing, yu2024rlaif], but they have not led to notable improvements in general capabilities. There have also been some preliminary explorations in the multi-image and video domains, such as MIA-DPO and PPLLaVA. However, alignment in image and video domains is still fragmented, with little research done under a unified framework. We believe that the main limitation hindering the development of current alignment algorithms is the lack of a high-quality, multimodal alignment dataset. Few existing manually annotated MLLM alignment datasets are available, and most contain fewer than 10K samples [sun2023aligning, yu2024rlaif, yu2024rlhf], which is significantly smaller than large-scale alignment datasets in the LLM field. This small dataset size makes it difficult to cover multiple modalities and diverse task types. Furthermore, machine-annotated data faces challenges related to quality assurance. Therefore, in this paper, we have invested considerable effort into constructing a dataset, , which surpasses existing works in both scale and annotation quality.
MLLM Evaluation. With the development of MLLMs, a number of benchmarks have been built [duan2024vlmevalkit,fu2024mme]. For instance, MME [fu2023mme] constructs a comprehensive evaluation benchmark that includes a total of 14 perception and cognition tasks. All QA pairs in MME are manually designed to avoid data leakage, and the binary choice format makes it easy to quantify. MMBench [liu2023mmbench] contains over \(3,000\) multiple-choice questions covering \(20\) different ability dimensions, such as object localization and social reasoning. It introduces GPT-4-based choice matching to address the MLLM's lack of instruction-following capability and a novel circular evaluation strategy to improve the evaluation robustness. Seed-Bench [li2023seed] is similar to MME and MMBench but consists of \(19,000\) multiple-choice questions. The larger sample size allows it to cover more ability aspects and achieve more robust results. SEED-Bench-2 [li2023seed2] expands the dataset size to \(24,371\) QA pairs, encompassing \(27\) evaluation dimensions and further supporting the evaluation of image generation. MMT-Bench [mmtbench] scales up the dataset even further, including \(31,325\) QA pairs from various scenarios such as autonomous driving and embodied AI. It encompasses evaluations of model capabilities such as visual recognition, localization, reasoning, and planning. Additionally, other benchmarks focus on real-world usage scenarios [fu2024blink,lu2024wildvision,bitton2023visit] and reasoning capabilities [yu2024mm,bai2023touchstone,han2023coremm,yan2024errorradar]. MME-RealWorld [zhang2024mme] places greater emphasis on quality and difficulty compared to its predecessor, containing the largest manually annotated QA pairs and the largest image resolution. These benchmarks reveal some common characteristics of MLLMs in task design and real-world applications. However, benchmarks specifically focused on reward models [li2024vlrewardbench] and those dedicated to evaluating safety and robustness remain relatively scarce. To further promote comprehensive evaluation of MLLM alignment, this paper contributes two benchmarks: one for reward models through self-construction and data cleaning, and another more comprehensive safety benchmark.
Annotation Guidelines for Evaluating MLLM Responses¶
Figure fig:web_ui: The user interface for data annotation, featuring image/video display, questions, outputs from each model, detailed scoring criteria, and a section for reviewers to verify the accuracy of the scores.
This document provides detailed annotation guidelines for evaluating responses generated by MLLMs. Annotators should rate and annotate each response according to four primary evaluation criteria: Visual Faithfulness, Helpfulness, Ethical Considerations (including safety, privacy, fairness, and harm), and Overall Performance. Annotators are expected to assess each response carefully based on these criteria to ensure high-quality feedback for model optimization.
I. Visual Faithfulness Evaluation¶
Definition: This criterion evaluates whether the generated response accurately reflects the objects and relationships in the image, ensuring consistency with the objects, relationships, and attributes of the true answer.
Guidelines: enumerate Object Description Accuracy: Ensure that the generated response accurately describes objects as in the true answer, avoiding references to non-existent objects and preventing errors in descriptions of existing objects. Object Relationship Accuracy: Evaluate whether the spatial, structural, or functional relationships between objects described in the response are correct. Minimize errors and misleading information in object relationship descriptions. Object Attribute Accuracy: Confirm that the response accurately describes the physical features, color, size, and state of objects, avoiding discrepancies with the true answer. Overall Accuracy: Assess the accuracy and consistency of the response in describing visual elements (objects, relationships, and attributes) in the image. enumerate
Rating Scale: itemize Severely Inaccurate: Major errors in object descriptions, relationships, or attributes, or references to non-existent objects. Partially Inaccurate: Object descriptions, relationships, or attributes are mostly correct, but some details are inaccurate. Moderately Accurate: Descriptions of objects, relationships, and attributes are generally accurate, with minor errors. Accurate: Descriptions of objects, relationships, and attributes are correct and match the true answer. Highly Accurate: Descriptions of objects, relationships, and attributes are exceptionally precise and include useful additional details. itemize
II. Helpfulness Evaluation¶
Definition: Carefully review the user prompt to ensure that the response provides valuable insights and relevant information that aid the user in understanding the image or task.
Guidelines: enumerate Assess whether the response provides useful context, insights, or relevant information to help the user better understand the image or task. Check if the model follows the specific instructions or guidelines provided by the user. Evaluate the overall contribution of the response in enhancing the user's understanding and experience. enumerate
Rating Scale: itemize Not Helpful: The response does not address the user's prompt, providing entirely irrelevant information. Partially Helpful: The response contains some useful information, but overall help is limited. Moderately Helpful: The response is somewhat helpful but may lack important information. Helpful: The response effectively addresses the user's prompt and provides valuable insights. Very Helpful: The response is highly useful, offering rich insights that significantly enhance the user’s understanding. itemize
III. Ethical Considerations Evaluation (Safety, Privacy, Fairness, and Harm)¶
Definition: This criterion evaluates whether the model adheres to ethical standards regarding safety, privacy protection, fairness, and avoiding harmful content.
Guidelines: enumerate Ensure the model provides appropriate warnings or avoids inappropriate recommendations when addressing sensitive topics (e.g., medical images). Confirm that the response does not disclose private information that could harm personal privacy. Assess whether the model shows fairness when addressing individuals or groups, avoiding any form of bias. Ensure the content does not incite violence, contain inappropriate workplace content (NSFW), or present any other unaddressed ethical issues. enumerate
Rating Scale: itemize Highly Unsafe/Privacy Infringing/Unfair/Harmful: The response contains severely inappropriate content that violates ethical standards. Unsafe/Privacy Issues/Unfair/Potentially Harmful: The response may pose safety risks, privacy issues, or show unfairness. Moderately Ethical: The response mostly adheres to ethical standards, with minor safety or fairness issues. Ethically Compliant: The response aligns with ethical standards, following safety, privacy protection, and fairness requirements. Highly Ethical/Safe/Privacy Protected/Fair/Non-Harmful: The response fully meets ethical standards, respecting privacy, fairness, and free from harmful content. itemize
Annotation Requirements¶
enumerate The labeling staff should carefully read the user's prompt and the model-generated response before scoring the response based on three criteria: visual Faithfulness, helpfulness, and ethical considerations. Each model should briefly record the reason for its score, for example, if the answer is incorrect, if it includes hallucinated content, or if there is an error in the description. The final evaluation of each response should comprehensively consider all criteria, followed by a manual ranking of all responses. Tie Status: Indicate whether the user perceives no significant difference between the outputs of each model. If a tie occurs, provide a negative example (for multiple-choice, offer an incorrect answer; for long text, modify the content to include erroneous information). Ranking Basis: Briefly explain the reasoning behind the ranking. enumerate
Safety and Trustworth Dataset and Benchmark Construction¶
Training Data Construction Details¶
The self-constructed content is divided into 850 safety samples and 500 adversarial samples. The safety data is sourced from the following datasets: Red Teaming VLM [li2024red], CelebA [liu2015deep], and VLSBench [hu2024vlsbench]. The adversarial data, on the other hand, is generated using the AnyAttack [zhanganyattack] method.
To ensure data diversity, the safety data is comprised of five categories: itemize 200 samples from Jailbreak, 200 samples from privacy and discrimination, 150 samples from hacking, 200 samples from violence, 100 samples from self-injury. itemize For the adversarial data, we randomly sampled 500 images from AnyAttack’s clean dataset. For each image, we then generate an adversarial image by pairing it with another, using \(\epsilon = 8/255\) and other parameters set to their original values. To ensure the effectiveness of the adversarial attacks, we manually verified that the generated adversarial images cause the LLaVA-OV-7B model to produce hallucinated outputs.
Questions of safety data are generated by using VLGuard’s question generation prompts to create queries. For adversarial data, to maintain prompt diversity, we use GPT-4o to generate 10 variations of the question "Please describe this image," and a random sentence from these variations is selected for each image to serve as the query.
Benchmark Construction Details¶
We constructed our benchmark by selecting a total of 9 tasks from the Multitrust [zhangmultitrust] benchmark, which includes adversarial evaluations (both targeted and non-targeted), risk identification, typographic jailbreak, multimodal jailbreak, and cross-modal jailbreak tasks. Additionally, we included 2 tasks from VLGuard that focus on evaluating the model's robustness against NSFW (Not Safe For Work) content. These tasks address high-risk scenarios such as harmful medical investment advice, self-harm, and explicit content. Specifically, we assess the model's ability to reject harmful outputs in situations where the image is dangerous or where the image is harmless but the accompanying instruction is harmful. Table tab:tasks presents a detailed summary of each task, including the sample size and evaluation metrics used to assess model performance in these critical safety and adversarial scenarios.
Table tab:tasks: -SafetyBench: summary of Task Data, Evaluation Metrics, and Comparison Methods for Safety and Adversarial Testing. This table provides an overview of various tasks used for evaluating multimodal models' safety and adversarial robustness. The tasks are categorized based on attack type (adversarial or safety), and the evaluation metrics include success rates of adversarial attacks or model rejection rates for harmful outputs. The arrows in the Comparison column indicate whether higher (()) or lower (()) values of the evaluation metric are preferred.
\begin{table}[ht]
\caption{\textbf{\abbr-SafetyBench: summary of Task Data, Evaluation Metrics, and Comparison Methods for Safety and Adversarial Testing}. This table provides an overview of various tasks used for evaluating multimodal models' safety and adversarial robustness. The tasks are categorized based on attack type (adversarial or safety), and the evaluation metrics include success rates of adversarial attacks or model rejection rates for harmful outputs. The arrows in the Comparison column indicate whether higher (\(\uparrow\)) or lower (\(\downarrow\)) values of the evaluation metric are preferred.}
\label{tab:tasks}
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{|l|c|p{5.5cm}|p{4.5cm}|c|}
\hline
\rowcolor{gray!20} \textbf{Task Name} & \textbf{Data Size} & \textbf{Task Category} & \textbf{Evaluation Metric} & \textbf{Comparison} \\
\hline
Adv target & 100 & Adversarial Attack: The ratio of model descriptions containing adversarial image content. & Ratio & \(\downarrow\) \\
Adv untarget & 200 & Adversarial Attack: The ratio of model descriptions containing original image content. & Ratio & \(\uparrow\) \\
Crossmodel ASR & 18 & Cross-modal Jailbreak: Inducing multimodal models to deviate from their expected behavior and security constraints by combining text and image contexts related to the jailbreak task. & ASR: Adversarial attack success rate (probability of harmful output) & \(\downarrow\) \\
Multimodel ASR & 100 & Multimodal Jailbreak: Using a combination of various modalities (e.g., text and images) to induce multimodal models to deviate from security mechanisms and expected behavior, performing malicious commands or unauthorized functions. & ASR: Adversarial attack success rate (probability of harmful output) & \(\downarrow\) \\
Typographic ASR & 600 & Typographic Jailbreak: Converting malicious text instructions into images to perform jailbreak attacks, inducing multimodal models to deviate from their expected behavior and security constraints. & ASR: Adversarial attack success rate (probability of harmful output) & \(\downarrow\) \\
Crossmodel RtA & 18 & Cross-modal Jailbreak: Inducing multimodal models to deviate from their expected behavior and security constraints by combining text and image contexts for the jailbreak task. & RtA: Model accuracy on adversarial samples (probability of rejecting output) & \(\uparrow\) \\
Multimodel RtA & 100 & Multimodal Jailbreak: Using a combination of various modalities (e.g., text and images) to induce multimodal models to deviate from security mechanisms and expected behavior, performing malicious commands or unauthorized functions. & RtA: Model accuracy on adversarial samples (probability of rejecting output) & \(\uparrow\) \\
Typographic RtA & 600 & Typographic Jailbreak: Converting malicious text instructions into image format for jailbreak attacks, inducing multimodal models to deviate from their expected behavior and security constraints. & RtA: Model accuracy on adversarial samples (probability of rejecting output) & \(\uparrow\) \\
Risk identification & 500 & Identification Ability: The model's ability to correctly identify dangerous items and assess risk. & Ratio & \(\uparrow\) \\
NSFW-Text & 558 & Visual-Linguistic Safety: Ratio of model not rejecting output when the image is harmless but the instruction contains harmful content. & ASR: Adversarial attack success rate (probability of model not rejecting output) & \(\downarrow\) \\
NSFW-Image & 442 & Visual-Linguistic Safety: Ratio of model not rejecting output when the image contains harmful information. & ASR: Adversarial attack success rate (probability of model not rejecting output) & \(\downarrow\) \\
\hline
\end{tabular}
}
\end{table}
Why We Need Large-Scale Human Annotation?¶
Manual annotation provides higher accuracy and adaptability than model-based annotation, especially in cases where the limitations of machine annotation become evident. In this section, we illustrate representative cases found in multi-modal data that are particularly challenging for models to annotate, highlighting the advantages of human intervention. All human annotations presented here come from our own dataset, while GPT-4o annotations were generated based on prompting GPT-4o by our ranking criteria.
Misleading and Incomplete Questions¶
Since training data is commonly annotated by models, maintaining perfect quality assurance is challenging, often resulting in some confusing or incomplete questions that cannot be answered accurately. In such cases, models struggle to provide effective annotations, whereas human annotators can identify and handle these issues with greater precision.
Figure fig:annotation_mcq: Example of a confusing question.
itemize Confusing Questions: As shown in Figure fig:annotation_mcq, conflicts between the question and the provided choices can lead to confusion and misinterpretation. Many models attempt to select a “preferred” choice, with models like GPT-4 assigning a rank and providing a seemingly logical rationale. However, our human annotators are able to identify these flaws, reject all model-generated answers, and instead offer a more accurate response, highlighting the strength of human review in recognizing and rectifying such issues. Incomplete Questions: Similar to confusing questions, issues with data quality often result in questions that lack essential information. In many cases, MLLMs fail to recognize these inconsistencies and instead attempt to generate an answer, as do annotation models, which tend to favor responses from models that provide answers. As shown in Figure fig:annotation_mcq2, this question requires calculating the length of side AF; however, the given conditions are insufficient to solve the problem. Both Qwen2-VL and Claude 35 make incorrect attempts at analysis, with Qwen2-VL ultimately ranking the highest despite providing an incorrect answer. In contrast, human annotators are adept at identifying such issues and can accurately indicate that the question lacks sufficient conditions, justifying this in their rankings. itemize
Figure fig:annotation_mcq2: Example of an incomplete question.
Difficult-to-Distinguish Answers¶
The limitations of model-based annotation extend beyond question quality issues. At times, the responses generated by models are themselves challenging to rank accurately. This difficulty arises in two primary scenarios:
Figure fig:short1: Example of a difficult question for model annotation.
itemize All Models Fail to Identify the Correct Answer. Certain questions, especially those involving complex reasoning, can be so challenging that no model can provide a correct response, indicating that the question surpasses the model's capabilities. Attempting to annotate such questions with model-based ranking often leads to further issues. For instance, in the high-resolution perception task shown in Figure fig:short1, the required information specified in the question does not actually appear in the image. However, multiple models still provide incorrect responses based on their interpretations. During scoring, the models tend to select the answer that aligns most closely with their understandingThe reason why GPT-4o annotator does not select its own response as the best may be due to the sampling strategy used in our API calls.. In contrast, human annotators excel in recognizing these limitations and can provide the truly correct answer, demonstrating the advantage of manual annotation in such complex cases. Model Responses Are Rich but May Contain Minor Errors at a Fine-Grained Level. In many datasets, especially in conversational data, when model responses are lengthy or involve specialized knowledge, it can be challenging—even for skilled multimodal annotators—to discern the subtle differences between outputs from various models. Our annotators take an average of 6 minutes to assess a single long-response question accurately, while models struggle even more with evaluating such extended replies. For instance, in Figure fig:annotation_long, the differences among models are confined to specific sections, where minor errors in visual perception or judgment occur (highlighted in red). These fine-grained details are often overlooked by the models themselves, resulting in scores that do not align with those given by human annotators. itemize
Figure fig:annotation_long: Example of subtle errors in model responses to a long question.
Table tab:prompt_example: Example of the Prompt Used for Augmenting Human Annotations.
\begin{table}[]
\caption{Example of the Prompt Used for Augmenting Human Annotations.}
\label{tab:prompt_example}
\resizebox{\textwidth}{!}{
\begin{tabular}{l}
\begin{tabular}[c]{@{}l@{}}You will receive an image-related question, an answer, and a comment provided by a human expert for the answer. \\ \\ Your task is to expand the human comment comprehensively while retaining its strengths and weaknesses, \\ making it more professional, and logically rigorous. Focus only on expanding the comment and do not answer the question. \\ \\ Ensure the expanded comment is strictly based on the provided human comment and avoids any speculation or uncertain content.\\ \\ {[}Question:{]} \{question\}\\ {[}Answer:{]} \{answer\}\\ {[}Human Comment for the answer:{]} \{reason\}\\ \\ Expanded Comment:\end{tabular}
\end{tabular}
}
\end{table}
Comparison to Existing Methods on Beta Adjustment in LLMs and MLLMs¶
Dynamic adjustment of the beta parameter is not a completely new concept, but its application in large multimodal language models has been relatively unexplored. In this section, we discuss the key differences between our approach and existing methods, particularly focusing on dynamic beta adjustment strategies in LLMs and MLLMs. Several studies have been conducted in the LLM domain, with many papers showing that common LLM DPO datasets contain a significant number of noisy samples [wu2024beta,chowdhury2024provably,amini2024direct]. In these works, the application of different beta values to samples of varying quality has been shown to significantly improve algorithm robustness and performance.
Our approach differs from the existing works in two primary ways:
First Exploration of Dynamic Beta Adjustment in MLLMs. To the best of our knowledge, we are the first to explore how MLLMs can dynamically adjust the beta parameter. We find that existing dynamic beta methods developed for LLMs cannot be directly adapted to the MLLM setting [wu2024beta]. This is mainly due to the increased complexity of the data in MLLM scenarios. Most existing methods [wu2024beta,amini2024direct] utilize implicit rewards during the training process of DPO algorithms to select higher-quality samples. However, in MLLMs, the signal discriminability of the model itself is weaker and cannot guide the selection of \(\beta\) (Figure fig:dpo_ablation (a)). Furthermore, as shown in our experiments, using MLLMs as reward models, especially with smaller models, results in suboptimal performance. This observation highlights a critical challenge in adapting existing methods to MLLMs.
Leveraging a High-Quality Reward Model for Beta Adjustment. Existing methods often rely on various tricks to ensure that the estimated beta value is reasonable and of high quality, such as batch-level normalization and other techniques. Instance-level beta adjustments, on the other hand, are generally considered unstable and typically result in suboptimal performance. However, our approach challenges this conventional wisdom. We demonstrate that when a high-quality external reward model is available, reasonable modeling can enable instance-level beta adjustments to yield significant improvements. By leveraging a robust reward model, we show that even fine-grained adjustments to the beta parameter at the instance level can effectively enhance the model's performance, contrary to the usual belief that such adjustments are unreliable.
Our work provides a fresh perspective on how dynamic beta adjustments can be effectively applied to MLLMs, improving their robustness and optimization stability. By incorporating a high-quality reward model and dynamically scaling beta based on the reward margin, we achieve notable improvements over existing methods, particularly in handling noisy data and improving algorithmic performance.
More Ablation and Analysis¶
Figure label:ablation_dpo_a: Ablation studies on our method and dataset. (a) Real-world tasks evaluation, where LLaVA-OV-7B serves as the baseline model, +MM-RLHF represents the use of our dataset combined with the traditional DPO algorithm. +Implicit Reward refers to using the dynamic beta strategy [wu2024beta] in LLMs. (b) Evaluation of the effect of the hyperparameters \(k\) and \(w\) on the MM-DPO model, demonstrating the effect of these variations on the leaderboard scores.
Improvement with Dataset and MM-DPO¶
With the help of our dataset, the baseline model demonstrates a general improvement across various benchmarks, with particularly significant gains observed in OCR and conversation tasks (Figure label:ablation_dpo_a)). To further exploit the observation that different samples have varying quality, we initially attempted methods from the LLM domain, specifically using Implicit Reward during training to decide whether to increase or decrease the beta of each sample. However, we found that this approach did not work. There are two possible reasons: 1) Our dataset is of relatively high quality, as it is ranked manually, so the noise is minimal and there is no need for too many penalty terms or a reduction in beta; 2) MLLM data is more complex, and Implicit Reward does not provide a reliable signal to adjust beta. Therefore, MM-DPO uses a high-quality reward model to directly provide the signal, and the value of beta is constrained using the function \([\beta_{\text{ori}}, (1 + w) \beta_{\text{ori}}]\) , preventing it from growing too excessively. This method overcomes the training instability caused by outliers, ultimately leading to a steady performance improvement.
Effect of Hyperparameters \(w\) and \(k\)¶
We experimented with various combinations of the hyperparameters \(w\) and \(k\), where \(k\) directly controls the mapping function from the reward margin to the scaling factor, and \(w\) governs the strength of the correction to () by the scaling factor. Figure label:ablation_dpo_b shows the impact of these hyperparameters on the final average performance (using the same benchmarks as Figure label:ablation_dpo_a). The results demonstrate that the method exhibits a certain level of robustness across different hyperparameter selections, generally leading to performance improvements. However, selecting the two hyperparameters requires some finesse; they cannot both be too large or too small simultaneously. The default values of \(w = 0.5\) and \(k = 0.5\) work well.