Navegação autônoma local: estudo empírico sobre o ajuste de modelos de linguagem-visão compactos para o uso de interfaces gráficas

Abstract

The autonomous automation of graphical user interfaces (GUIs) by computer use agents (CUAs) is dominated by large models, whose dependence on large-scale infrastructure imposes cost, latency, and privacy barriers. These barriers motivate interest in compact agents executable on consumer hardware, but the appropriate training techniques for this scale range and their main bottlenecks remain poorly characterized. Given this, this work empirically investigates whether reinforcement learning fine-tuning improves the performance of a compact CUA, in the four-billion-parameter range, trained exclusively under commercial hardware constraints. To this end, a vision-language model (VLM) from the Qwen3-VL family is fine-tuned on the AgentNet dataset through reinforcement learning (RL), adopting the Group Relative Sequence Policy Optimization (GSPO) algorithm under an off-policy regime, utilizing single-step episodes and a multi-criteria reward function. The resulting model is evaluated on the ScreenSpot-Pro and OSWorld-Verified benchmarks in a controlled comparison with the equivalent base network. The results reveal a consistent gain in spatial grounding, with overall accuracy improving from 33.7% to 38.7%, primarily concentrated in the localization of textual elements. In contrast, a slight regression in long- horizon agentic performance is observed, dropping from 15.1% to 11.3%, predominantly attributed to an under-termination failure rather than a loss of perceptual or interface manipulation capabilities. An analysis of internal metrics associates this behavior with the scarce representation of the termination action and the short training duration, indications reinforced by a preliminary training phase that demonstrates the feasibility of a calibrated termination policy when this action receives sufficient representation. It is concluded that the reinforcement alignment strategy is structurally viable for compact architectures and highly amenable to optimization, providing a documented empirical baseline for future investigations into local CUAs.