Despite emerging efforts to enhance the safety of Vision-Language Models (VLMs), current approaches face two main shortcomings. 1) Existing safety-tuning datasets and benchmarks only partially consider how image-text interactions can yield harmful content, often overlooking contextually unsafe outcomes from seemingly benign pairs. This narrow coverage leaves VLMs vulnerable to jailbreak attacks in unseen configurations. 2) Prior methods rely primarily on data-centric tuning, with limited architectural innovations to intrinsically strengthen safety. We address these gaps by introducing a holistic safety-tuning dataset and benchmark, HoliSafe, that spans all five safe/unsafe image-text combinations, providing a more robust basis for both training and evaluation. We further propose SafeLLaVA, a novel VLM augmented with a learnable safety meta token and a dedicated safety head. The meta token encodes harmful visual cues during training, intrinsically guiding the language model toward safer responses, while the safety head offers interpretable harmfulness classification aligned with refusal rationales. Experiments show that SafeLLaVA, trained on HoliSafe, achieves state-of-the-art safety performance across multiple VLM benchmarks. Additionally, our HoliSafe-Bench itself reveals critical vulnerabilities in existing models. We hope that HoliSafe and SafeLLaVA will spur further research into robust and interpretable VLM safety, expanding future avenues for multimodal alignment.

Unlike prior works that cover only a subset of image-text safeness combinations (e.g., unsafe image with safe text), we introduce a new holistic safety-tuning dataset and benchmark, called HoliSafe, that systematically covers all five image-text safeness combinations: (1) unsafe image + unsafe text (U_iU_t), (2) unsafe image + safe text (U_iS_t), (3) safe image + unsafe text (S_iU_t), (4) safe image + safe text yielding unsafe content (S_iS_t → U), and (5) safe image + safe text yielding safe content (S_iS_t → S).

HoliSafe defines a safety taxonomy with 7 main categories with 18 subcategories which are commonly encountered in real-world scenarios. We collect a total of 6,782 images and 15,114 instruction-response pairs. We split the dataset into a training set, 4,983 (74%) images, for safetytuning and a test set, 1,799 (26%) for Holisafe-Bench. Training and Test splits have 10,951 and 4,163 instruction-response pairs, respectively.

We propose a simple yet safety-effective VLM architecture. It incorporates a learnable image Safety Meta Token ([SMT]) and a safety head that not only classifies visually harmful content, acting as a visual safety guard model, but also provides the LLM with intrinsic safety cues for generating safer responses. To this end, The [SMT] is appended at the end of visual tokens, which allows for attention to visual tokens within both the vision encoder and decoder of the LLM.

We visualize average attention weights from generated tokens to [SMT] and visual tokens across layers of the LLM, revealing two key phenomena: i) [SMT] exhibits a bimodal attention pattern, strongest in early and late layers, consistent with similar phenomena observed for special tokens in prior language model research. This suggests [SMT] functions across decoding stages, from early context integration and safety constraint embedding to late-stage constraint enforcement and final safety checks. ii) Generated tokens attend more to [SMT] than to visual tokens across layers. This highlights [SMT]'s salience, suggesting its critical role in guiding the LLM's response from a safety perspective compared to raw visual input. Collectively, these attention patterns underscore the pivotal role of the safety meta token in influencing the LLM's decoding process for safety alignment.