Geospatially Grounded Satellite Image Captioning with Improved Factual Consistency using Directional Scene Graphs and Large Language Models: An Automated Pipeline for Semantic Summarization of Satellite Images

Abstract

Remote sensing image captioning has recently attracted significant attention for automated interpretation of satellite imagery; however, existing approaches often rely primarily on visual features and insufficiently model explicit geospatial relationships, resulting in factually inconsistent descriptions and limited spatial reasoning. To address this limitation, this paper proposes an automated end-to-end framework for geospatially grounded semantic summarization of remote sensing images by integrating object detection, geographic coordinate transformation, directional spatial reasoning, and large language models (LLMs). The proposed framework uses the YOLOv11 object detector on the DIOR dataset to identify objects and then the affine transformation is applied to convert pixel-level bounding boxes into WGS-84 geographic coordinates. A novel Directional Spatial Relationship Graph (DSRG) is then constructed to model adjacency, density-based clustering, transportation proximity, and object importance, thereby capturing both geometric and functional relationships among detected objects. The extracted spatial knowledge is organized using a Grounded Prompt Framework (GPF), which constrains LLMs to generate geographically grounded summaries based exclusively on detected objects and their spatial context. The framework is evaluated using object detection metrics (Precision, Recall, mAP), Geographic Intersection-over-Union (Geo-IoU), graph statistics, grounding score, hallucination rate, ROUGE, and BERTScore. Experimental results demonstrate that the proposed YOLOv11 model achieves a Precision of 0.893, Recall of 0.784, mAP@0.5 of 0.854, and mean Geo-IoU of 0.8685. The comparative analysis of different evaluated language models showed that the structured prompting strategy substantially improves factual consistency, with Gemini 2.5 Flash achieving a hallucination rate of 1.6% at the same time maintaining competitive grounding performance. A DIOR dataset does not provide real georeferenced object annotations; geospatial consistency is evaluated using synthetic reference polygons. It represented a current limitation of the study. Overall, the proposed DSRG–GPF framework provides an effective approach for generating geospatially grounded and factually consistent semantic summaries of remote sensing imagery, supporting applications in geospatial intelligence, infrastructure monitoring, and GIS-based decision-making.