Abstract
Medical vision-language systems are commonly evaluated on whether they answer image-based questions correctly, yet many real imaging consult queries are not purely image-answerable. Users may ask for diagnosis, prognosis, treatment, administrative interpretation, patient identity, legal responsibility, or causal explanation that exceeds what the image alone can support. A model that answers all such requests with the same style may appear useful while silently crossing clinical, informational, or institutional boundaries. This paper presents an empirical study of clinical intent boundary modeling for medical vision-language question answering. We created a dataset of 14,850 image-associated consult questions drawn from radiography, computed tomography, ultrasound, dermatology, and wound-care settings, with each question labeled by whether it was directly image-answerable, context-dependent, clinician-only, administrative, identity-related, prognostic, treatment-seeking, or unanswerable from the provided input. We trained a multimodal intent boundary model that jointly encodes the image, question, acquisition metadata, and expected evidence source before routing the query to one of four response policies. Compared with direct answering by six vision-language systems, boundary routing reduced unsupported clinical assertions from 22.6% to 9.4%, reduced treatment-seeking answer leakage from 18.1% to 5.7%, and preserved answer accuracy for image-answerable questions within 1.3 percentage points. The boundary model achieved macro F1 of 0.812 for eight-class intent recognition and detected non-image-answerable questions with sensitivity of 0.884. These findings suggest that medical vision-language evaluation should include intent-boundary competence rather than assuming every image-associated question deserves a direct visual answer.