Beyond the Browser: Building Desktop GUI Agents in 2026 with UI-TARS, Claude Computer Use, and OSWorld 2.0
Over the past three years, the AI ecosystem focused heavily on web-browser automation using tools like Browser-Use, Stagehand, and Playwright MCP to parse HTML DOM trees. However, over 70% of enterprise software workflows do not run inside a browser DOM. Legacy ERPs (SAP GUI), native Excel workbooks with embedded VBA macros, desktop CAD suites, terminal consoles, and native clients possess zero web DOM. When operating systems render directly via DirectX, Metal, Win32, or Qt, DOM-based agents are blind. To unlock true enterprise automation, 2026 marks the rise of Autonomous Desktop GUI Agents powered by native Vision-Language-Action (VLA) foundation models, pixel-coordinate grounding, and System-2 deliberate reasoning. This architectural deep dive analyzes UI-TARS 1.5, Claude 3.7 Computer Use, the OSWorld 2.0 benchmark, and production sandbox containment.
๐ Table of Contents
01. Quick Summary & The Desktop GUI Frontier
Web automation reached maturity, but desktop knowledge work remained untamed. In 2026, desktop GUI agents treat the entire computer screen as their interactive canvas:
- Beyond HTML Selectors: Desktop agents observe raw screen frames (1080p to 4K), recognize visual affordances directly through Multimodal Large Language Models (MLLMs), and output normalized coordinates (x, y) mapped to OS mouse and keyboard events.
- Open-Source Frontier vs. Proprietary APIs: ByteDance's open-source UI-TARS 1.5 introduced native System-2 reinforcement learning for step-by-step reflection and backtracking, while Anthropic's Claude 3.7 Sonnet provides high-level reasoning and native computer use tool calling.
- The OSWorld 2.0 Reality Check: On the OSWorld 2.0 benchmark spanning 369 complex real-world tasks across Ubuntu, Windows, and macOS, agents achieve 42%โ52% success on 100-step long-horizon tasks, revealing that long-term state drift remains the primary engineering hurdle.
- Mandatory Sandboxing: Direct OS control requires zero-trust isolation. Production architectures execute GUI agents inside disposable MicroVMs (e.g., E2B) or virtual display streams (VNC/RDP) backed by host-side action firewalls and human-in-the-loop kill switches.
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Autonomous Desktop GUI Agent Architecture (2026) |
| |
| [ User Goal: "Consolidate Q3 SAP exports into Excel macro, generate PDF" ] |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ HOST SUPERVISOR & ACTION FIREWALL (Safety Interceptor) โ |
| โ - Rate Limiting & Egress Filtering - Destructive Command Blocker โ |
| โ - Biometric Confirmation Gateway - Emergency Human Kill-Switch โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ Virtual Display / Peripherals |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ ISOLATED DESKTOP SANDBOX (E2B / Cloud VNC / KVM Virtual Machine) โ |
| โ โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ |
| โ โ Screenshot Buffer โ โ OS Input Controller โ โ |
| โ โ (1920x1080 RGB) โ โ (PyAutoGUI / uinput) โ โ |
| โ โโโโโโโโโโโโฌโโโโโโโโโโโ โโโโโโโโโโโโโโโฒโโโโโโโโโโโโ โ |
| โ โ Raw Frame โ (x, y) Click โ |
| โ โผ โ & Hotkeys โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ โ |
| โ โ AGENT RUNTIME: UI-TARS 1.5 / Claude 3.7 Sonnet โ โ |
| โ โ 1. Visual Grounding: Detect target UI elements via pixels โ โ |
| โ โ 2. System-2 Deliberation: Check milestones & reflect โ โ |
| โ โ 3. Action Plan: Emit precise mouse, click, and key sequences โ โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ |
| โ โ |
| โ [ Legacy SAP Client ] [ Native Excel ] [ Desktop CAD ] โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+02. The Death of DOM-Dependency: Why Desktop Enterprise Workflows Matter
Browser automation frameworks assume that applications expose structured accessibility trees, clean CSS selectors, and predictable element hierarchies. In high-value corporate operations, this assumption breaks down completely:
1. Canvas-Rendered Web Apps
Modern tools like Figma, Google Docs canvas rendering, and complex WebGL dashboards draw elements into blank canvas contexts without DOM nodes.
2. Legacy Enterprise Clients
Mission-critical systemsโSAP GUI, AS400 terminal emulators, Bloomberg Terminals, and healthcare EHRsโrun as native binaries with zero web interfaces.
3. Cross-Application Chaining
Workflows require downloading raw data, executing local Excel VBA macros, updating CRM desktop windows, and signing PDFs across operating system boundaries.
03. The Three Perception Architectures: Pixel vs. Accessibility Tree vs. Hybrid
How should an autonomous agent perceive what is on the desktop screen? Three architectural approaches dominate in 2026:
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Desktop Perception Paradigms |
| |
| [ Approach 1: Pure Visual Grounding (Pixel-to-Coordinate) ] |
| Screen Frame โโโถ High-Res VLM โโโถ Coordinates (x: 450, y: 720) |
| โข Pros: Universal, zero OS dependency, handles Canvas / Games / Legacy |
| โข Cons: Heavy token cost, coordinate distortion, resolution scaling drift|
| |
| [ Approach 2: OS Accessibility Tree Grounding (UIAutomation / AT-SPI) ] |
| Screen State โโโถ OS API Walk โโโถ Filtered Hierarchy โโโถ Element ID / Path|
| โข Pros: Deterministic, lightweight text tokens, 100% click precision |
| โข Cons: 40% of native apps have broken/missing a11y trees, slow tree walk|
| |
| [ Approach 3: Dual-Stream Hybrid Fusion (2026 Best Practice) ] |
| Visual Screenshot (VLM) โโโFused Decisionโโโบ OS A11y Tree (Cache) |
| โข Fast path: Use A11y node bounding box if recognized |
| โข Fallback: Use visual grounding when a11y nodes are obscured/custom |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+| Dimension | Pure Visual Grounding | OS Accessibility Tree | Dual-Stream Hybrid Fusion |
|---|---|---|---|
| Universality | 100% (Any pixel rendered on screen) | ~60% (Fails on custom UI/canvas) | 98% (Falls back gracefully) |
| Token Efficiency | Low (~1,200 to 2,000 tokens/frame) | High (~300 to 600 tokens/tree) | Medium (~1,500 tokens/decision) |
| Click Accuracy | 88% - 94% (Subject to drift) | 99% (When nodes exist) | 98.5% (Snaps to bounding box) |
| OS Independence | Complete (VNC/Streaming compatible) | Low (Requires platform API hooks) | High (Modular OS adapters) |
| Turn Latency | 1.8s - 3.5s (VLM inference) | 0.4s - 0.9s (Text LLM) | 1.9s - 3.2s |
04. UI-TARS: Native Vision-Language-Action & System-2 Reasoning
Developed by ByteDance and open-sourced in 2025โ2026, UI-TARS pioneered native **Vision-Language-Action (VLA)** modeling for graphical interfaces. Rather than hacking prompts onto general vision models, UI-TARS uses direct tokenization of motor actions and reinforcement learning for deliberate reflection:
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| UI-TARS 1.5 System-2 Reasoning Trajectory |
| |
| Observation โโโถ [ Reflection & Verification ] |
| โ "Did the previous action open the File dialog?" |
| โ State: YES. Detected 'Open File' window at (x: 200, y: 150)
| โผ |
| [ Sub-goal Decomposition ] |
| โ "Next sub-goal: Select 'Quarterly_Report.xlsx'" |
| โ Search Strategy: Visual scan of table rows |
| โผ |
| [ Milestone Recognition ] |
| โ Target element found at (x: 320, y: 410) |
| โผ |
| [ Action Emission ] |
| โ Action: click(point=[320, 410]) |
| โ Post-Action Expectation: File selected in input box |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+- Standardized Motor Tokens: Directly emits tokens for
click(point=[x, y]),double_click(),drag(start, end), andpress_hotkey()on a normalized 1000 × 1000 coordinate plane. - System-2 Deliberation: Evaluates previous state transitions before acting. If clicking a button failed to open a dialog, the model reflects, self-corrects, and retries rather than compounding errors.
- Local Deployment Capability: UI-TARS 7B runs locally on developer workstations (Apple Silicon 32GB or RTX 4090), providing sub-second turn latency with complete enterprise data privacy.
05. Claude 3.7 Computer Use vs. Open-Weights: Latency, Cost, and Accuracy Tradeoffs
Engineering teams must choose between proprietary frontier APIs (Claude 3.7 Sonnet) and self-hosted open-weights models (UI-TARS 7B/72B):
- The Vision Token Cost Curve: Streaming 1080p screenshots to a proprietary frontier model every 2 seconds costs to for a 100-step enterprise reconciliation task.
- Strategic Reasoning vs. Repetitive RPA: Claude 3.7 Sonnet excels at high-ambiguity tasks requiring semantic understanding of unstructured documents and spreadsheets. UI-TARS excels at high-throughput operational execution at a fraction of compute cost.
- Data Governance: Financial and healthcare organizations handling confidential client records mandate that screen pixels never leave the private enterprise perimeter, making self-hosted UI-TARS the architectural standard.
06. Production Implementation: Building a Safe Desktop GUI Agent in Python
Below is a complete, runnable reference implementation in Python demonstrating normalized coordinate scaling, safety gatekeeping, destructive command interception, and human-in-the-loop confirmation:
# Production Reference Implementation: Desktop GUI Agent Controller (2026)
# Demonstrates Vision-Language-Action Execution, Coordinate Normalization,
# Destructive Command Interception, and Human-in-the-Loop Safeguards.
import time
import math
from typing import Dict, Any, List, Tuple, Optional
from dataclasses import dataclass, field
from enum import Enum
# โโโ 1. CORE DATA STRUCTURES & ACTIONS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class ActionType(str, Enum):
MOUSE_CLICK = "mouse_click"
DOUBLE_CLICK = "double_click"
HOTKEY = "hotkey"
TYPE_TEXT = "type_text"
TERMINAL_COMMAND = "terminal_command"
WAIT = "wait"
@dataclass
class UIAction:
action_type: ActionType
coordinates: Optional[Tuple[int, int]] = None # (x, y) on 1000x1000 normalized grid
text_payload: Optional[str] = None
hotkey_sequence: Optional[List[str]] = None
thought_reasoning: str = ""
is_destructive: bool = False
@dataclass
class ScreenDimensions:
width: int
height: int
# โโโ 2. SAFETY INTERCEPTOR & GATEKEEPER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class DesktopSafetyGatekeeper:
"""
Host-side safety authority that validates actions before dispatching
to real or virtual OS input peripherals.
"""
DESTRUCTIVE_HOTKEYS = {("ctrl", "alt", "del"), ("cmd", "shift", "backspace")}
DESTRUCTIVE_COMMANDS = ["rm -rf", "format", "drop database", "mkfs", "dd if="]
def __init__(self, require_human_for_destructive: bool = True):
self.require_human = require_human_for_destructive
self.audit_log: List[Dict[str, Any]] = []
def inspect_action(self, action: UIAction, human_approved: bool = False) -> Tuple[bool, str]:
# 1. Inspect terminal/shell commands
if action.action_type == ActionType.TERMINAL_COMMAND and action.text_payload:
for pattern in self.DESTRUCTIVE_COMMANDS:
if pattern in action.text_payload.lower():
action.is_destructive = True
if not human_approved:
return False, f"Blocked destructive terminal command pattern: '{pattern}'"
# 2. Inspect high-risk hotkeys
if action.action_type == ActionType.HOTKEY and action.hotkey_sequence:
normalized_keys = tuple(sorted([k.lower() for k in action.hotkey_sequence]))
for risk_keys in self.DESTRUCTIVE_HOTKEYS:
if normalized_keys == tuple(sorted(risk_keys)):
action.is_destructive = True
if not human_approved:
return False, f"Blocked dangerous system hotkey: '{action.hotkey_sequence}'"
# Log action to immutable audit trail
self.audit_log.append({
"timestamp": time.time(),
"action": action.action_type.value,
"coordinates": action.coordinates,
"destructive": action.is_destructive,
"approved": True
})
return True, "Action approved."
# โโโ 3. PERIPHERAL ADAPTER & COORDINATE SCALER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class OSPeripheralController:
"""
Translates normalized model coordinates (1000x1000) to actual physical display pixels
and dispatches low-level OS input events.
"""
def __init__(self, display: ScreenDimensions):
self.display = display
def denormalize_coordinates(self, norm_x: int, norm_y: int) -> Tuple[int, int]:
"""Converts [0, 1000] model grid to [0, width] x [0, height]."""
real_x = math.floor((norm_x / 1000.0) * self.display.width)
real_y = math.floor((norm_y / 1000.0) * self.display.height)
return real_x, real_y
def execute_native_input(self, action: UIAction):
if action.coordinates:
rx, ry = self.denormalize_coordinates(*action.coordinates)
print(f"๐ฑ๏ธ [OS DRIVER] Moving cursor to ({rx}px, {ry}px) [Model: {action.coordinates}]")
if action.action_type == ActionType.MOUSE_CLICK:
print(f"โก [OS DRIVER] Emitting LEFT_BUTTON_CLICK at ({rx}, {ry})")
elif action.action_type == ActionType.DOUBLE_CLICK:
print(f"โก [OS DRIVER] Emitting DOUBLE_CLICK at ({rx}, {ry})")
elif action.action_type == ActionType.HOTKEY:
print(f"โจ๏ธ [OS DRIVER] Emitting KEY_COMBINATION: {' + '.join(action.hotkey_sequence or [])}")
elif action.action_type == ActionType.TYPE_TEXT:
print(f"โจ๏ธ [OS DRIVER] Typing string payload: \"{action.text_payload}\"")
elif action.action_type == ActionType.TERMINAL_COMMAND:
print(f"๐ฅ๏ธ [OS DRIVER] Executing Shell Command: '{action.text_payload}'")
# โโโ 4. AUTONOMOUS DESKTOP AGENT CONTROLLER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class DesktopGUIAgent:
"""
The orchestrator agent combining VLA decision logic, safety inspection,
and OS input driver dispatch.
"""
def __init__(self, display: ScreenDimensions, gatekeeper: DesktopSafetyGatekeeper):
self.driver = OSPeripheralController(display)
self.gatekeeper = gatekeeper
def step(self, action: UIAction, human_confirmed: bool = False) -> Dict[str, Any]:
print(f"\n๐ง [SYSTEM-2 REFLECTION] {action.thought_reasoning}")
# Safety Check
is_safe, reason = self.gatekeeper.inspect_action(action, human_approved=human_confirmed)
if not is_safe:
print(f"๐ [SAFETY INTERCEPT] Action Rejected: {reason}")
return {"success": False, "reason": reason}
# Dispatch
self.driver.execute_native_input(action)
return {"success": True, "reason": "Executed successfully"}
# โโโ 5. RUNTIME VERIFICATION HARNESS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
if __name__ == "__main__":
print("=" * 75)
print("DEMO: AUTONOMOUS DESKTOP GUI AGENT SAFETY & GROUNDING CONTROLLER (2026)")
print("=" * 75)
# Initialize 1080p display and safety gatekeeper
virtual_screen = ScreenDimensions(width=1920, height=1080)
safety_shield = DesktopSafetyGatekeeper(require_human_for_destructive=True)
agent = DesktopGUIAgent(display=virtual_screen, gatekeeper=safety_shield)
# STEP 1: Safe Action - Navigate SAP Menu
step1 = UIAction(
action_type=ActionType.MOUSE_CLICK,
coordinates=(180, 45), # 1000x1000 normalized grid
thought_reasoning="Milestone 1: Click 'Accounting' dropdown in SAP native menu bar."
)
agent.step(step1)
# STEP 2: Safe Action - Type Transaction Code
step2 = UIAction(
action_type=ActionType.TYPE_TEXT,
text_payload="FS10N",
thought_reasoning="Milestone 2: Enter general ledger balance inquiry transaction code."
)
agent.step(step2)
# STEP 3: Malicious / Accidental Destructive Step Intercepted
step3_destructive = UIAction(
action_type=ActionType.TERMINAL_COMMAND,
text_payload="rm -rf /var/sap/data/*",
thought_reasoning="Adversarial prompt injection attempt detected on unverified clipboard buffer."
)
print("\n[SCENARIO 1: Destructive Action Without Approval]")
agent.step(step3_destructive, human_confirmed=False)
# STEP 4: High-Risk Action With Human Biometric Approval
print("\n[SCENARIO 2: Authorized High-Risk Action via Supervisor Approval]")
agent.step(step3_destructive, human_confirmed=True)
print("\n" + "=" * 75)
print(f"Demonstration Complete. Total Audit Ledger Entries: {len(safety_shield.audit_log)}")
print("=" * 75)
07. OSWorld 2.0 Benchmark Analysis: Solving Long-Horizon Drift & Failure Modes
The OSWorld 2.0 benchmark evaluates agents across 369 realistic tasks in Ubuntu, Windows, and macOS. While human baseline performance sits at 88.3%, state-of-the-art agents achieve between 49% and 52% overall, revealing four systemic failure modes:
- Resolution & Coordinate Drift: Modal popups or window resizes shift visual targets by small pixel offsets, leading to misclicks on outdated coordinates.
- Context Window Fatigue: Retaining 60 full-resolution screenshots overflows model context. Production systems prune older screenshots into compact textual action histories.
- Silent Loading & Spinner Traps: Lacking browser DOM events like
networkidle, agents repeatedly click during application loading, triggering thread crashes. Visual delta variance checks solve this. - Low-Contrast UI Confusion: In dark-mode enterprise software or CAD wireframes, models frequently confuse visually similar tool icons (e.g., Save vs. Export).
08. Security & Sandboxing: VNC Isolation, eBPF Egress, and Kill-Switch Safeguards
Granting an autonomous agent direct control over physical peripherals on an employee's laptop creates severe security vulnerabilities. Production architectures enforce a three-layer zero-trust sandbox:
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Zero-Trust Desktop Sandbox Containment Stack |
| |
| [ Enterprise Gateway ] |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 1: VIRTUAL DISPLAY & PERIPHERAL ISOLATION โ |
| โ - Headless X11 / Wayland / RDP virtual server โ |
| โ - The agent NEVER touches physical user hardware โ |
| โ - Video frame streamed via WebRTC / VNC buffer โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 2: SYSTEM CALL INTERCEPTION & eBPF NETWORK EGRESS โ |
| โ - eBPF sensor intercepts dangerous POSIX syscalls (fork, execve, ptrace)โ
| โ - Network firewall restricts outbound traffic exclusively to whitelistโ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 3: SUPERVISOR KILL-SWITCH & USER TAKEOVER โ |
| โ - User moves physical mouse โโโถ Instant agent suspension (Kill-Switch)โ |
| โ - Live action stream mirrored to supervisor dashboard โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+- Layer 1: Virtual Display Containment: The agent interacts solely with a virtual X11/Wayland/RDP buffer inside an isolated MicroVM (e.g., E2B). Physical hardware is never exposed.
- Layer 2: eBPF Network & Syscall Filtering: Kernel-level eBPF sensors intercept destructive shell commands (
rm -rf, formatting, unauthorized exfiltration) and block non-whitelisted outbound IP connections. - Layer 3: Physical Motion Kill-Switch: On assisted human workstations, touching the physical mouse or pressing
Escinstantly revokes the agent's input driver privileges, returning immediate control to the user.
09. Architectural Comparison Matrix & Related Tools
How do the premier desktop and computer-use agent frameworks compare in 2026?
| Dimension | UI-TARS (ByteDance) | Claude 3.7 Computer Use | OpenHands | E2B Desktop Sandbox |
|---|---|---|---|---|
| Model Nature | Open-Weights (VLA 7B/72B) | Frontier Proprietary API | Model-Agnostic Orchestrator | Infrastructure Sandbox Runtime |
| Primary Perception | Raw Pixels (System-2 VLA) | Raw Pixels (Vision API) | Hybrid (DOM + Terminal + GUI) | Virtual Display Frame Buffer |
| Hosting Mode | Self-Hosted (On-Premises GPU) | Cloud API (Anthropic) | Self-Hosted / Cloud | Managed Cloud / MicroVM |
| OS Support | Windows, macOS, Linux, Android | Windows, macOS, Ubuntu | Linux, Docker, Browser | Linux (Firecracker MicroVM) |
| Target Use Case | High-Frequency RPA, QA, Batch | Complex Knowledge Tasks | Software Engineering Agents | Secure Agent Execution Sandboxing |
| Open Source | 100% Open Source | Proprietary | 100% Open Source | Open-Source SDK / Managed Cloud |
E2B
MicroVM SandboxThe premier MicroVM cloud sandbox runtime for executing untrusted desktop sessions, terminal commands, and headless environments with hardware-level isolation.
Explore E2B โClaude 3.7 Sonnet
Frontier ModelAnthropic's flagship reasoning model featuring native Computer Use capabilities for desktop navigation, browser automation, and multi-step tool execution.
Explore Claude 3.7 Sonnet โOpenHands
Open SourceThe leading open-source autonomous agent platform for software development, terminal operations, and desktop navigation, designed for full local deployment.
Explore OpenHands โDevin
Autonomous SoftwareCognition's flagship autonomous software engineering assistant equipped with integrated browser, terminal, and desktop workspace automation.
Explore Devin โ10. Frequently Asked Questions (FAQ)
Q1: Why not just build custom APIs instead of building desktop GUI agents?
In an ideal world, every application would expose rich, authenticated REST or GraphQL APIs. In practice, enterprise software ecosystems contain thousands of legacy applications (SAP, mainframe terminal emulators, desktop accounting tools) where adding APIs requires millions of dollars and years of refactoring. Desktop GUI agents enable zero-touch integration: automating legacy systems immediately without touching source code or altering underlying databases.
Q2: What screen resolution should I feed into a visual desktop agent?
Feeding raw 4K screenshots causes severe token bloat and latency degradation. Production systems typically capture the desktop at 1920x1080, normalize coordinates to an internal 1000x1000 grid for model inference, and project predicted coordinates back to physical screen pixels using mathematical scaling factors.
Q3: How do desktop agents handle dynamic UI loading and animations?
Unlike web browsers that fire DOM events like DOMContentLoaded or networkidle, desktop environments provide no native notification that an application has finished loading. Production agents utilize visual delta polling: capturing frames at 250ms intervals and verifying that pixel delta variance drops below 1% before concluding that the screen state is stable enough for the next action.
Q4: Can desktop agents operate on dual-monitor or multi-window setups?
Yes. Coordinate mapping engines can either treat multi-monitor setups as a single stitched canvas (e.g., 3840x1080) or utilize window-focus APIs to bring the active target application to the primary virtual display before running the perception step.
Q5: Is UI-TARS capable of running locally on consumer hardware?
The UI-TARS 7B model can run locally on modern workstation hardware (e.g., Apple Silicon Macs with 32GB+ Unified Memory or single NVIDIA RTX 4090 GPUs) using quantized GGUF or AWQ formats. The 72B model requires dual or quad A100/H100 enterprise GPUs for sub-second inference latency.
Mรกs Allรก del Navegador: Construcciรณn de Agentes GUI de Escritorio en 2026 con UI-TARS, Claude Computer Use y OSWorld 2.0
Durante los รบltimos tres aรฑos, el ecosistema de IA se centrรณ intensamente en la automatizaciรณn del navegador web mediante herramientas como Browser-Use, Stagehand y Playwright MCP para procesar รกrboles HTML DOM. Sin embargo, mรกs del 70% de los flujos de trabajo de software empresarial no se ejecutan dentro del DOM de un navegador. Los ERP heredados (SAP GUI), los libros de Excel nativos con macros VBA incrustadas, las suites de CAD de escritorio, las consolas de terminal y los clientes nativos carecen de DOM web. Cuando los sistemas operativos renderizan directamente a travรฉs de DirectX, Metal, Win32 o Qt, los agentes basados en DOM quedan ciegos e inoperantes. Para desbloquear la verdadera automatizaciรณn empresarial, 2026 marca el auge de los Agentes GUI de Escritorio Autรณnomos impulsados por modelos fundacionales nativos de Visiรณn-Lenguaje-Acciรณn (VLA), anclaje por coordenadas de pรญxeles y razonamiento deliberado Sistema 2. Este anรกlisis tรฉcnico en profundidad deconstruye UI-TARS 1.5, Claude 3.7 Computer Use, el benchmark OSWorld 2.0 y la contenciรณn en sandboxes de producciรณn.
๐ Tabla de Contenidos
01. Resumen Rรกpido y la Frontera de GUI de Escritorio
La automatizaciรณn web ha alcanzado su madurez, pero el trabajo de conocimiento en el escritorio seguรญa indomado. En 2026, los agentes GUI de escritorio tratan toda la pantalla del sistema operativo como su lienzo interactivo:
- Mรกs Allรก de Selectores HTML: Los agentes de escritorio observan fotogramas sin procesar (de 1080p a 4K), reconocen elementos visuales mediante Modelos de Lenguaje Multimodales (MLLM) y emiten coordenadas normalizadas (x, y) mapeadas a eventos nativos de ratรณn y teclado.
- Frontera de Cรณdigo Abierto vs APIs Propietarias: UI-TARS 1.5 de ByteDance introdujo aprendizaje por refuerzo Sistema 2 para reflexiรณn y retroceso algorรญtmico, mientras que Claude 3.7 Sonnet de Anthropic ofrece razonamiento de alto nivel y llamadas a herramientas nativas de Computer Use.
- La Realidad de OSWorld 2.0: En el benchmark OSWorld 2.0 que abarca 369 tareas del mundo real en Ubuntu, Windows y macOS, los agentes logran entre un 42% y un 52% de รฉxito en flujos de 100 pasos, demostrando que el desvรญo de estado a largo plazo es el principal desafรญo de ingenierรญa.
- Sandboxing Obligatorio: El control directo del SO exige aislamiento de confianza cero. Las arquitecturas de producciรณn ejecutan agentes dentro de MicroVMs desechables (como E2B) o flujos de pantalla virtual (VNC/RDP) respaldados por cortafuegos de acciones e interruptores de parada humana (Kill Switch).
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Autonomous Desktop GUI Agent Architecture (2026) |
| |
| [ User Goal: "Consolidate Q3 SAP exports into Excel macro, generate PDF" ] |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ HOST SUPERVISOR & ACTION FIREWALL (Safety Interceptor) โ |
| โ - Rate Limiting & Egress Filtering - Destructive Command Blocker โ |
| โ - Biometric Confirmation Gateway - Emergency Human Kill-Switch โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ Virtual Display / Peripherals |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ ISOLATED DESKTOP SANDBOX (E2B / Cloud VNC / KVM Virtual Machine) โ |
| โ โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ |
| โ โ Screenshot Buffer โ โ OS Input Controller โ โ |
| โ โ (1920x1080 RGB) โ โ (PyAutoGUI / uinput) โ โ |
| โ โโโโโโโโโโโโฌโโโโโโโโโโโ โโโโโโโโโโโโโโโฒโโโโโโโโโโโโ โ |
| โ โ Raw Frame โ (x, y) Click โ |
| โ โผ โ & Hotkeys โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ โ |
| โ โ AGENT RUNTIME: UI-TARS 1.5 / Claude 3.7 Sonnet โ โ |
| โ โ 1. Visual Grounding: Detect target UI elements via pixels โ โ |
| โ โ 2. System-2 Deliberation: Check milestones & reflect โ โ |
| โ โ 3. Action Plan: Emit precise mouse, click, and key sequences โ โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ |
| โ โ |
| โ [ Legacy SAP Client ] [ Native Excel ] [ Desktop CAD ] โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+02. El Fin de la Dependencia del DOM: Por Quรฉ Importan los Flujos Empresariales de Escritorio
Los marcos de automatizaciรณn web asumen que las aplicaciones exponen รกrboles de accesibilidad estructurados, selectores CSS limpios y jerarquรญas predecibles. En las operaciones corporativas de alto impacto, esta suposiciรณn falla por completo:
1. Aplicaciones Web en Canvas
Herramientas modernas como Figma, Google Docs con renderizado de canvas y paneles WebGL dibujan elementos directamente en lienzos sin nodos DOM accesibles.
2. Clientes Empresariales Heredados
Sistemas crรญticos (SAP GUI, emuladores AS400, terminales Bloomberg y registros mรฉdicos EHR) funcionan como binarios nativos sin interfaces web.
3. Encadenamiento Multi-Aplicaciรณn
Los flujos de trabajo requieren descargar datos, ejecutar macros VBA locales en Excel, actualizar ventanas de CRM y firmar PDFs cruzando fronteras de procesos.
03. Las Tres Arquitecturas de Percepciรณn: Pรญxeles vs รrbol de Accesibilidad vs Hรญbrido
ยฟCรณmo debe percibir un agente autรณnomo lo que hay en la pantalla del escritorio? En 2026 conviven tres enfoques arquitectรณnicos:
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Desktop Perception Paradigms |
| |
| [ Approach 1: Pure Visual Grounding (Pixel-to-Coordinate) ] |
| Screen Frame โโโถ High-Res VLM โโโถ Coordinates (x: 450, y: 720) |
| โข Pros: Universal, zero OS dependency, handles Canvas / Games / Legacy |
| โข Cons: Heavy token cost, coordinate distortion, resolution scaling drift|
| |
| [ Approach 2: OS Accessibility Tree Grounding (UIAutomation / AT-SPI) ] |
| Screen State โโโถ OS API Walk โโโถ Filtered Hierarchy โโโถ Element ID / Path|
| โข Pros: Deterministic, lightweight text tokens, 100% click precision |
| โข Cons: 40% of native apps have broken/missing a11y trees, slow tree walk|
| |
| [ Approach 3: Dual-Stream Hybrid Fusion (2026 Best Practice) ] |
| Visual Screenshot (VLM) โโโFused Decisionโโโบ OS A11y Tree (Cache) |
| โข Fast path: Use A11y node bounding box if recognized |
| โข Fallback: Use visual grounding when a11y nodes are obscured/custom |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+| Dimensiรณn | Anclaje Visual Puro | รrbol de Accesibilidad del SO | Fusiรณn Hรญbrida de Doble Flujo |
|---|---|---|---|
| Universalidad | 100% (Cualquier pรญxel visible) | ~60% (Falla en UI personalizada/canvas) | 98% (Recuperaciรณn elegante) |
| Eficiencia de Tokens | Baja (~1,200 a 2,000 tokens/frame) | Alta (~300 a 600 tokens/รกrbol) | Media (~1,500 tokens/decisiรณn) |
| Precisiรณn de Clic | 88% - 94% (Sujeto a desvรญo) | 99% (Cuando existen nodos) | 98.5% (Ajuste a caja delimitadora) |
| Independencia del SO | Total (Compatible con VNC/Streaming) | Baja (Requiere hooks de API del SO) | Alta (Adaptadores modulares) |
| Latencia por Turno | 1.8s - 3.5s (Inferencia VLM) | 0.4s - 0.9s (LLM de texto) | 1.9s - 3.2s |
04. UI-TARS: VLA Nativo y Razonamiento Sistema 2
Desarrollado por ByteDance y liberado como cรณdigo abierto, UI-TARS fue pionero en el modelado nativo de **Visiรณn-Lenguaje-Acciรณn (VLA)** para interfaces grรกficas. En lugar de adaptar modelos de visiรณn genรฉricos con ingenierรญa de prompts, utiliza tokenizaciรณn directa de acciones motoras y aprendizaje por refuerzo con reflexiรณn deliberada:
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| UI-TARS 1.5 System-2 Reasoning Trajectory |
| |
| Observation โโโถ [ Reflection & Verification ] |
| โ "Did the previous action open the File dialog?" |
| โ State: YES. Detected 'Open File' window at (x: 200, y: 150)
| โผ |
| [ Sub-goal Decomposition ] |
| โ "Next sub-goal: Select 'Quarterly_Report.xlsx'" |
| โ Search Strategy: Visual scan of table rows |
| โผ |
| [ Milestone Recognition ] |
| โ Target element found at (x: 320, y: 410) |
| โผ |
| [ Action Emission ] |
| โ Action: click(point=[320, 410]) |
| โ Post-Action Expectation: File selected in input box |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+- Tokens Motores Estandarizados: Emite directamente tokens para
click(point=[x, y]),double_click(),drag(start, end)ypress_hotkey()en un plano normalizado de 1000 × 1000. - Deliberaciรณn Sistema 2: Evalรบa las transiciones de estado anteriores antes de actuar. Si hacer clic en un botรณn no abriรณ el diรกlogo esperado, el modelo reflexiona, se autocorrige y reintenta en lugar de acumular fallos.
- Capacidad de Despliegue Local: UI-TARS 7B se ejecuta localmente en estaciones de trabajo de desarrolladores (Apple Silicon con 32GB o RTX 4090), brindando latencias inferiores a un segundo con privacidad de datos garantizada.
05. Claude 3.7 Computer Use vs Modelos de Cรณdigo Abierto: Compensaciones de Latencia, Coste y Precisiรณn
Los equipos de ingenierรญa deben elegir entre APIs comerciales de frontera (Claude 3.7 Sonnet) y modelos autohospedados de pesos abiertos (UI-TARS 7B/72B):
- La Curva de Coste de Tokens de Visiรณn: Transmitir capturas 1080p a un modelo propietario cada 2 segundos cuesta entre y para una tarea de conciliaciรณn de 100 pasos.
- Razonamiento Estratรฉgico vs RPA Repetitivo: Claude 3.7 Sonnet sobresale en tareas de alta ambigรผedad que requieren comprensiรณn semรกntica de documentos no estructurados. UI-TARS destaca en ejecuciรณn operativa de alto volumen con una fracciรณn del coste computacional.
- Gobernanza de Datos: Entidades bancarias y sanitarias exigen que los pรญxeles de pantalla jamรกs salgan del perรญmetro privado de la empresa, consolidando a UI-TARS autohospedado como el estรกndar arquitectรณnico.
06. Implementaciรณn en Producciรณn: Construcciรณn de un Controlador Seguro de Escritorio en Python
A continuaciรณn se presenta una implementaciรณn de referencia completa y ejecutable en Python que demuestra el escalado de coordenadas, la supervisiรณn de seguridad, la interceptaciรณn de comandos destructivos y la confirmaciรณn humana:
# Production Reference Implementation: Desktop GUI Agent Controller (2026)
# Demonstrates Vision-Language-Action Execution, Coordinate Normalization,
# Destructive Command Interception, and Human-in-the-Loop Safeguards.
import time
import math
from typing import Dict, Any, List, Tuple, Optional
from dataclasses import dataclass, field
from enum import Enum
# โโโ 1. CORE DATA STRUCTURES & ACTIONS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class ActionType(str, Enum):
MOUSE_CLICK = "mouse_click"
DOUBLE_CLICK = "double_click"
HOTKEY = "hotkey"
TYPE_TEXT = "type_text"
TERMINAL_COMMAND = "terminal_command"
WAIT = "wait"
@dataclass
class UIAction:
action_type: ActionType
coordinates: Optional[Tuple[int, int]] = None # (x, y) on 1000x1000 normalized grid
text_payload: Optional[str] = None
hotkey_sequence: Optional[List[str]] = None
thought_reasoning: str = ""
is_destructive: bool = False
@dataclass
class ScreenDimensions:
width: int
height: int
# โโโ 2. SAFETY INTERCEPTOR & GATEKEEPER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class DesktopSafetyGatekeeper:
"""
Host-side safety authority that validates actions before dispatching
to real or virtual OS input peripherals.
"""
DESTRUCTIVE_HOTKEYS = {("ctrl", "alt", "del"), ("cmd", "shift", "backspace")}
DESTRUCTIVE_COMMANDS = ["rm -rf", "format", "drop database", "mkfs", "dd if="]
def __init__(self, require_human_for_destructive: bool = True):
self.require_human = require_human_for_destructive
self.audit_log: List[Dict[str, Any]] = []
def inspect_action(self, action: UIAction, human_approved: bool = False) -> Tuple[bool, str]:
# 1. Inspect terminal/shell commands
if action.action_type == ActionType.TERMINAL_COMMAND and action.text_payload:
for pattern in self.DESTRUCTIVE_COMMANDS:
if pattern in action.text_payload.lower():
action.is_destructive = True
if not human_approved:
return False, f"Blocked destructive terminal command pattern: '{pattern}'"
# 2. Inspect high-risk hotkeys
if action.action_type == ActionType.HOTKEY and action.hotkey_sequence:
normalized_keys = tuple(sorted([k.lower() for k in action.hotkey_sequence]))
for risk_keys in self.DESTRUCTIVE_HOTKEYS:
if normalized_keys == tuple(sorted(risk_keys)):
action.is_destructive = True
if not human_approved:
return False, f"Blocked dangerous system hotkey: '{action.hotkey_sequence}'"
# Log action to immutable audit trail
self.audit_log.append({
"timestamp": time.time(),
"action": action.action_type.value,
"coordinates": action.coordinates,
"destructive": action.is_destructive,
"approved": True
})
return True, "Action approved."
# โโโ 3. PERIPHERAL ADAPTER & COORDINATE SCALER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class OSPeripheralController:
"""
Translates normalized model coordinates (1000x1000) to actual physical display pixels
and dispatches low-level OS input events.
"""
def __init__(self, display: ScreenDimensions):
self.display = display
def denormalize_coordinates(self, norm_x: int, norm_y: int) -> Tuple[int, int]:
"""Converts [0, 1000] model grid to [0, width] x [0, height]."""
real_x = math.floor((norm_x / 1000.0) * self.display.width)
real_y = math.floor((norm_y / 1000.0) * self.display.height)
return real_x, real_y
def execute_native_input(self, action: UIAction):
if action.coordinates:
rx, ry = self.denormalize_coordinates(*action.coordinates)
print(f"๐ฑ๏ธ [OS DRIVER] Moving cursor to ({rx}px, {ry}px) [Model: {action.coordinates}]")
if action.action_type == ActionType.MOUSE_CLICK:
print(f"โก [OS DRIVER] Emitting LEFT_BUTTON_CLICK at ({rx}, {ry})")
elif action.action_type == ActionType.DOUBLE_CLICK:
print(f"โก [OS DRIVER] Emitting DOUBLE_CLICK at ({rx}, {ry})")
elif action.action_type == ActionType.HOTKEY:
print(f"โจ๏ธ [OS DRIVER] Emitting KEY_COMBINATION: {' + '.join(action.hotkey_sequence or [])}")
elif action.action_type == ActionType.TYPE_TEXT:
print(f"โจ๏ธ [OS DRIVER] Typing string payload: \"{action.text_payload}\"")
elif action.action_type == ActionType.TERMINAL_COMMAND:
print(f"๐ฅ๏ธ [OS DRIVER] Executing Shell Command: '{action.text_payload}'")
# โโโ 4. AUTONOMOUS DESKTOP AGENT CONTROLLER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class DesktopGUIAgent:
"""
The orchestrator agent combining VLA decision logic, safety inspection,
and OS input driver dispatch.
"""
def __init__(self, display: ScreenDimensions, gatekeeper: DesktopSafetyGatekeeper):
self.driver = OSPeripheralController(display)
self.gatekeeper = gatekeeper
def step(self, action: UIAction, human_confirmed: bool = False) -> Dict[str, Any]:
print(f"\n๐ง [SYSTEM-2 REFLECTION] {action.thought_reasoning}")
# Safety Check
is_safe, reason = self.gatekeeper.inspect_action(action, human_approved=human_confirmed)
if not is_safe:
print(f"๐ [SAFETY INTERCEPT] Action Rejected: {reason}")
return {"success": False, "reason": reason}
# Dispatch
self.driver.execute_native_input(action)
return {"success": True, "reason": "Executed successfully"}
# โโโ 5. RUNTIME VERIFICATION HARNESS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
if __name__ == "__main__":
print("=" * 75)
print("DEMO: AUTONOMOUS DESKTOP GUI AGENT SAFETY & GROUNDING CONTROLLER (2026)")
print("=" * 75)
# Initialize 1080p display and safety gatekeeper
virtual_screen = ScreenDimensions(width=1920, height=1080)
safety_shield = DesktopSafetyGatekeeper(require_human_for_destructive=True)
agent = DesktopGUIAgent(display=virtual_screen, gatekeeper=safety_shield)
# STEP 1: Safe Action - Navigate SAP Menu
step1 = UIAction(
action_type=ActionType.MOUSE_CLICK,
coordinates=(180, 45), # 1000x1000 normalized grid
thought_reasoning="Milestone 1: Click 'Accounting' dropdown in SAP native menu bar."
)
agent.step(step1)
# STEP 2: Safe Action - Type Transaction Code
step2 = UIAction(
action_type=ActionType.TYPE_TEXT,
text_payload="FS10N",
thought_reasoning="Milestone 2: Enter general ledger balance inquiry transaction code."
)
agent.step(step2)
# STEP 3: Malicious / Accidental Destructive Step Intercepted
step3_destructive = UIAction(
action_type=ActionType.TERMINAL_COMMAND,
text_payload="rm -rf /var/sap/data/*",
thought_reasoning="Adversarial prompt injection attempt detected on unverified clipboard buffer."
)
print("\n[SCENARIO 1: Destructive Action Without Approval]")
agent.step(step3_destructive, human_confirmed=False)
# STEP 4: High-Risk Action With Human Biometric Approval
print("\n[SCENARIO 2: Authorized High-Risk Action via Supervisor Approval]")
agent.step(step3_destructive, human_confirmed=True)
print("\n" + "=" * 75)
print(f"Demonstration Complete. Total Audit Ledger Entries: {len(safety_shield.audit_log)}")
print("=" * 75)
07. Anรกlisis del Benchmark OSWorld 2.0: Resoluciรณn del Desvรญo a Largo Plazo y Modos de Fallo
El benchmark OSWorld 2.0 evalรบa agentes en 369 tareas realistas en Ubuntu, Windows y macOS. Mientras que la referencia humana alcanza el 88.3%, los modelos mรกs avanzados obtienen entre el 49% y el 52% global, revelando cuatro modos de fallo estructurales:
- Desvรญo de Resoluciรณn y Coordenadas: Los cuadros de diรกlogo emergentes o cambios de tamaรฑo de ventana desplazan los objetivos visuales por unos pocos pรญxeles, causando clics errรณneos sobre coordenadas obsoletas.
- Saturaciรณn de la Ventana de Contexto: Acumular 60 capturas a resoluciรณn completa satura la atenciรณn del modelo. Los sistemas de producciรณn podan las imรกgenes antiguas y las reemplazan por resรบmenes textuales de acciones.
- Trampas de Carga Silenciosa: Al carecer de eventos DOM como
networkidle, los agentes hacen clic reiterado durante la carga de aplicaciones, bloqueando subprocesos nativos. La verificaciรณn de varianza de pรญxeles soluciona esto. - Confusiรณn en Interfaces de Bajo Contraste: En modos oscuros corporativos o vistas de CAD, los modelos confunden con frecuencia iconos visualmente similares (ej. Guardar vs Exportar).
08. Seguridad y Sandbox: Aislamiento VNC, Filtrado eBPF e Interruptor de Parada (Kill-Switch)
Otorgar a un agente autรณnomo control directo sobre los perifรฉricos fรญsicos de un portรกtil corporativo genera graves riesgos de seguridad. Las arquitecturas de producciรณn aplican un sandbox de tres capas:
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Zero-Trust Desktop Sandbox Containment Stack |
| |
| [ Enterprise Gateway ] |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 1: VIRTUAL DISPLAY & PERIPHERAL ISOLATION โ |
| โ - Headless X11 / Wayland / RDP virtual server โ |
| โ - The agent NEVER touches physical user hardware โ |
| โ - Video frame streamed via WebRTC / VNC buffer โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 2: SYSTEM CALL INTERCEPTION & eBPF NETWORK EGRESS โ |
| โ - eBPF sensor intercepts dangerous POSIX syscalls (fork, execve, ptrace)โ
| โ - Network firewall restricts outbound traffic exclusively to whitelistโ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 3: SUPERVISOR KILL-SWITCH & USER TAKEOVER โ |
| โ - User moves physical mouse โโโถ Instant agent suspension (Kill-Switch)โ |
| โ - Live action stream mirrored to supervisor dashboard โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+- Capa 1: Contenciรณn en Pantalla Virtual: El agente interactรบa exclusivamente con un bรบfer virtual X11/Wayland/RDP dentro de una MicroVM aislada (ej. E2B). El hardware fรญsico jamรกs se expone.
- Capa 2: Filtrado eBPF de Llamadas al Sistema y Red: Sensores a nivel de kernel interceptan comandos de shell destructivos (
rm -rf, formateos, exfiltraciรณn no autorizada) y bloquean conexiones IP salientes no autorizadas. - Capa 3: Interruptor de Movimiento Fรญsico: En estaciones asistidas, mover el ratรณn fรญsico o pulsar
Escrevoca de inmediato los permisos del controlador del agente, devolviendo el mando al operador humano.
09. Matriz Comparativa de Arquitectura y Herramientas Relacionadas
ยฟCรณmo se comparan los principales marcos de agentes de escritorio en 2026?
| Dimensiรณn | UI-TARS (ByteDance) | Claude 3.7 Computer Use | OpenHands | E2B Desktop Sandbox |
|---|---|---|---|---|
| Naturaleza del Modelo | Pesos Abiertos (VLA 7B/72B) | API Propietaria de Frontera | Orquestador Agnรณstico | Entorno de Infraestructura Sandbox |
| Percepciรณn Principal | Pรญxeles sin Procesar (VLA Sistema 2) | Pรญxeles sin Procesar (API Visiรณn) | Hรญbrida (DOM + Terminal + GUI) | Bรบfer de Pantalla Virtual |
| Modalidad de Alojamiento | Autohospedado (GPU On-Premises) | API en la Nube (Anthropic) | Autohospedado / Nube | Nube Administrada / MicroVM |
| Sistemas Operativos | Windows, macOS, Linux, Android | Windows, macOS, Ubuntu | Linux, Docker, Navegador | Linux (Firecracker MicroVM) |
| Caso de Uso Objetivo | RPA de Alta Frecuencia, QA, Lotes | Tareas de Conocimiento Complejo | Agentes de Ingenierรญa de Software | Aislamiento Seguro de Ejecuciรณn |
| Cรณdigo Abierto | 100% Cรณdigo Abierto | Propietario | 100% Cรณdigo Abierto | SDK Abierto / Nube Gestionada |
E2B
Sandbox MicroVMEl entorno en la nube de MicroVMs lรญder para ejecutar sesiones de escritorio no confiables, comandos de terminal y entornos sin cabeza con aislamiento de hardware.
Explorar E2B โClaude 3.7 Sonnet
Modelo de FronteraEl modelo insignia de Anthropic con capacidades nativas de Computer Use para navegaciรณn de escritorio, automatizaciรณn web y ejecuciรณn de herramientas en mรบltiples pasos.
Explorar Claude 3.7 Sonnet โOpenHands
Cรณdigo AbiertoLa plataforma lรญder de agentes autรณnomos de cรณdigo abierto para desarrollo de software, operaciones de terminal y navegaciรณn de escritorio con despliegue local completo.
Explorar OpenHands โDevin
Software AutรณnomoEl asistente autรณnomo insignia de ingenierรญa de software de Cognition equipado con automatizaciรณn integrada de navegador, terminal y espacio de trabajo de escritorio.
Explorar Devin โ10. Preguntas Frecuentes (FAQ)
P1: ยฟPor quรฉ no crear APIs personalizadas en lugar de agentes GUI de escritorio?
En un mundo ideal, cada software expondrรญa APIs REST o GraphQL autenticadas. En la prรกctica empresarial existen miles de aplicaciones heredadas (SAP, emuladores de mainframe, software contable local) donde desarrollar APIs exigirรญa millones de dรณlares y aรฑos de refactorizaciรณn. Los agentes GUI de escritorio permiten integraciรณn sin modificaciones de cรณdigo.
P2: ยฟQuรฉ resoluciรณn de pantalla es ideal para alimentar al agente visual?
Enviar capturas 4K crudas provoca un gasto excesivo de tokens y aumenta drรกsticamente la latencia. Los sistemas de producciรณn capturan a 1920x1080, normalizan las coordenadas a una cuadrรญcula de 1000x1000 para la inferencia del modelo y luego proyectan los clics a los pรญxeles reales de la pantalla.
P3: ยฟCรณmo gestionan los agentes la carga dinรกmica y las animaciones de la interfaz?
A diferencia de los navegadores que emiten eventos DOM como DOMContentLoaded o networkidle, los escritorios carecen de notificaciones nativas de fin de carga. Los agentes de producciรณn emplean muestreo visual diferencial: toman fotogramas cada 250 ms y comprueban que la varianza entre pรญxeles sea inferior al 1% antes de actuar.
P4: ยฟPueden los agentes operar en entornos con mรบltiples pantallas o ventanas?
Sรญ. Los motores de mapeo de coordenadas pueden tratar configuraciones multipantalla como un รบnico lienzo combinado (ej. 3840x1080) o utilizar APIs de foco de ventanas para traer la aplicaciรณn objetivo al monitor virtual primario antes de la percepciรณn.
P5: ยฟSe puede ejecutar UI-TARS localmente en hardware de consumo?
El modelo UI-TARS 7B puede ejecutarse localmente en equipos modernos (Macs con Apple Silicon y 32GB+ de memoria unificada o GPUs NVIDIA RTX 4090) usando cuantizaciones GGUF o AWQ. La variante 72B requiere GPUs empresariales como NVIDIA A100/H100 para latencias inferiores al segundo.
Jenseits des Browsers: Aufbau autonomer Desktop-GUI-Agenten 2026 mit UI-TARS, Claude Computer Use und OSWorld 2.0
In den letzten drei Jahren konzentrierte sich das KI-รkosystem stark auf die Webbrowser-Automatisierung mit Tools wie Browser-Use, Stagehand und Playwright MCP zum Parsen von HTML-DOM-Bรคumen. Allerdings laufen รผber 70 % aller Unternehmenssoftware-Workflows auรerhalb des Browser-DOMs. Etablierte ERP-Systeme (SAP GUI), native Excel-Arbeitsmappen mit eingebetteten VBA-Makros, Desktop-CAD-Suiten, Terminal-Konsolen und proprietรคre On-Premises-Clients besitzen keinerlei Web-DOM. Wenn Betriebssysteme Oberflรคchen direkt รผber DirectX, Metal, Win32 oder Qt rendern, sind DOM-basierte Agenten vรถllig blind. Um echte Unternehmensautomatisierung zu erschlieรen, markiert das Jahr 2026 den Durchbruch autonomer Desktop-GUI-Agenten, angetrieben von nativen Vision-Language-Action (VLA)-Basismodellen, Pixelkoordinaten-Grounding und System-2-Deliberationslogik. Dieser Architektur-Deep-Dive dekonstruiert UI-TARS 1.5, Claude 3.7 Computer Use, den Benchmark OSWorld 2.0 und Sicherheits-Sandboxes fรผr den Produktivbetrieb.
๐ Inhaltsverzeichnis
01. Zusammenfassung & Die neue Desktop-GUI-Grenze
Die Webautomatisierung hat ihre Reife erreicht, doch die Wissensarbeit auf dem Desktop blieb unberรผhrt. 2026 behandeln Desktop-GUI-Agenten den gesamten Betriebssystem-Bildschirm als ihre interaktive Arbeitsflรคche:
- Jenseits von HTML-Selektoren: Desktop-Agenten erfassen rohe Bildschirmframes (1080p bis 4K), erkennen Benutzeroberflรคchenelemente direkt รผber multimodale Sprachmodelle (MLLMs) und generieren normalisierte Koordinaten (x, y), die in native Betriebssystem-Events รผbersetzt werden.
- Open-Source-Spitze vs. proprietรคre APIs: Das quelloffene UI-TARS 1.5 von ByteDance fรผhrte natives System-2-Reinforcement-Learning fรผr schrittweise Reflexion und Fehlerkorrektur ein, wรคhrend Claude 3.7 Sonnet von Anthropic herausragende Reasoning-Fรคhigkeiten und native Computer-Use-Tools bietet.
- Der Realitรคtscheck von OSWorld 2.0: Im OSWorld 2.0 Benchmark รผber 369 praxisnahe Aufgaben in Ubuntu, Windows und macOS erzielen fรผhrende Agenten 42 % bis 52 % Erfolgsquote bei 100-Schritt-Aufgaben โ was beweist, dass Zustandsabweichungen (Drift) die zentrale ingenieurtechnische Herausforderung bleiben.
- Zwingendes Sandboxing: Direkte Betriebssystemsteuerung erfordert Zero-Trust-Isolation. Produktivsysteme isolieren GUI-Agenten in flรผchtigen MicroVMs (z. B. E2B) oder virtuellen Bildschirm-Streams (VNC/RDP), abgesichert durch Aktions-Firewalls und Notfall-Kill-Switches.
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Autonomous Desktop GUI Agent Architecture (2026) |
| |
| [ User Goal: "Consolidate Q3 SAP exports into Excel macro, generate PDF" ] |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ HOST SUPERVISOR & ACTION FIREWALL (Safety Interceptor) โ |
| โ - Rate Limiting & Egress Filtering - Destructive Command Blocker โ |
| โ - Biometric Confirmation Gateway - Emergency Human Kill-Switch โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ Virtual Display / Peripherals |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ ISOLATED DESKTOP SANDBOX (E2B / Cloud VNC / KVM Virtual Machine) โ |
| โ โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ |
| โ โ Screenshot Buffer โ โ OS Input Controller โ โ |
| โ โ (1920x1080 RGB) โ โ (PyAutoGUI / uinput) โ โ |
| โ โโโโโโโโโโโโฌโโโโโโโโโโโ โโโโโโโโโโโโโโโฒโโโโโโโโโโโโ โ |
| โ โ Raw Frame โ (x, y) Click โ |
| โ โผ โ & Hotkeys โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ โ |
| โ โ AGENT RUNTIME: UI-TARS 1.5 / Claude 3.7 Sonnet โ โ |
| โ โ 1. Visual Grounding: Detect target UI elements via pixels โ โ |
| โ โ 2. System-2 Deliberation: Check milestones & reflect โ โ |
| โ โ 3. Action Plan: Emit precise mouse, click, and key sequences โ โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ |
| โ โ |
| โ [ Legacy SAP Client ] [ Native Excel ] [ Desktop CAD ] โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+02. Das Ende der DOM-Abhรคngigkeit: Warum Desktop-Workflows im Unternehmen zรคhlen
Web-Automatisierungs-Frameworks setzen voraus, dass Anwendungen strukturierte Barrierefreiheitsbรคume, saubere CSS-Selektoren und vorhersehbare Hierarchien bereitstellen. In unternehmenskritischen Prozessen greift diese Annahme ins Leere:
1. Canvas-gerenderte Web-Apps
Moderne Webanwendungen wie Figma, Google Docs mit Canvas-Rendering und WebGL-Dashboards zeichnen Inhalte direkt in leere Canvas-Elemente ohne DOM-Knoten.
2. Etablierte Enterprise-Clients
Kernsysteme wie SAP GUI, AS400-Terminal-Emulatoren, Bloomberg Terminals und Krankenhaus-KIS laufen als native Desktop-Binรคrdateien ohne Weboberflรคche.
3. Anwendungsรผbergreifende Ketten
Reale Workflows erfordern das Herunterladen von Daten, die Ausfรผhrung lokaler Excel-VBA-Makros, Aktualisierungen im Desktop-CRM und das Signieren von PDFs.
03. Die drei Wahrnehmungsarchitekturen: Pixel vs. Barrierefreiheitsbaum vs. Hybrid
Wie soll ein autonomer Agent den Bildschirminhalt erfassen? Im Jahr 2026 dominieren drei architektonische Ansรคtze:
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Desktop Perception Paradigms |
| |
| [ Approach 1: Pure Visual Grounding (Pixel-to-Coordinate) ] |
| Screen Frame โโโถ High-Res VLM โโโถ Coordinates (x: 450, y: 720) |
| โข Pros: Universal, zero OS dependency, handles Canvas / Games / Legacy |
| โข Cons: Heavy token cost, coordinate distortion, resolution scaling drift|
| |
| [ Approach 2: OS Accessibility Tree Grounding (UIAutomation / AT-SPI) ] |
| Screen State โโโถ OS API Walk โโโถ Filtered Hierarchy โโโถ Element ID / Path|
| โข Pros: Deterministic, lightweight text tokens, 100% click precision |
| โข Cons: 40% of native apps have broken/missing a11y trees, slow tree walk|
| |
| [ Approach 3: Dual-Stream Hybrid Fusion (2026 Best Practice) ] |
| Visual Screenshot (VLM) โโโFused Decisionโโโบ OS A11y Tree (Cache) |
| โข Fast path: Use A11y node bounding box if recognized |
| โข Fallback: Use visual grounding when a11y nodes are obscured/custom |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+| Dimension | Reines Pixel-Grounding | OS-Accessibility-Baum | Dual-Stream-Hybrid-Fusion |
|---|---|---|---|
| Universalitรคt | 100% (Jeder gerenderte Pixel) | ~60% (Scheitert bei Custom-UI/Canvas) | 98% (Eleganter Fallback) |
| Token-Effizienz | Niedrig (~1.200 bis 2.000 Tokens/Frame) | Hoch (~300 bis 600 Tokens/Baum) | Mittel (~1.500 Tokens/Entscheidung) |
| Klickgenauigkeit | 88% - 94% (Anfรคllig fรผr Drift) | 99% (Wenn Knoten existieren) | 98.5% (Einrasten auf Bounding-Box) |
| Betriebssystem-Unabhรคngigkeit | Vollstรคndig (VNC-/Streaming-fรคhig) | Gering (Benรถtigt OS-API-Hooks) | Hoch (Modulare OS-Adapter) |
| Latenz pro Schritt | 1.8s - 3.5s (VLM-Inferenz) | 0.4s - 0.9s (Text-LLM) | 1.9s - 3.2s |
04. UI-TARS: Natives Vision-Language-Action & System-2-Reasoning
Das von ByteDance entwickelte und quelloffene Modell UI-TARS etablierte natives **Vision-Language-Action (VLA)** fรผr grafische Benutzeroberflรคchen. Anstatt generische Bildmodelle mรผhsam zu prompten, setzt UI-TARS auf direkte Tokenisierung motorischer Aktionen und Reinforcement Learning fรผr deliberate Reflexion:
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| UI-TARS 1.5 System-2 Reasoning Trajectory |
| |
| Observation โโโถ [ Reflection & Verification ] |
| โ "Did the previous action open the File dialog?" |
| โ State: YES. Detected 'Open File' window at (x: 200, y: 150)
| โผ |
| [ Sub-goal Decomposition ] |
| โ "Next sub-goal: Select 'Quarterly_Report.xlsx'" |
| โ Search Strategy: Visual scan of table rows |
| โผ |
| [ Milestone Recognition ] |
| โ Target element found at (x: 320, y: 410) |
| โผ |
| [ Action Emission ] |
| โ Action: click(point=[320, 410]) |
| โ Post-Action Expectation: File selected in input box |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+- Standardisierte Motor-Tokens: Direkte Ausgabe von Tokens wie
click(point=[x, y]),double_click(),drag(start, end)undpress_hotkey()auf einem normalisierten 1000 × 1000 Koordinatensystem. - System-2-Deliberation: รberprรผfung vorheriger Zustandsรผbergรคnge vor der nรคchsten Aktion. Konnte ein Klick den Dialog nicht รถffnen, reflektiert das Modell den Fehler, korrigiert sich selbst und wiederholt den Schritt.
- Lokale Bereitstellung: UI-TARS 7B lรคuft lokal auf Workstations (Apple Silicon mit 32GB oder RTX 4090) und liefert Sub-Sekunden-Latenz bei vollstรคndiger Datensouverรคnitรคt.
05. Claude 3.7 Computer Use vs. Open-Weights: Latenz-, Kosten- und Genauigkeitsabwรคgungen
Entwicklungsteams mรผssen zwischen proprietรคren Spitzenmodellen (Claude 3.7 Sonnet) und selbst gehosteten Open-Weights-Modellen (UI-TARS 7B/72B) abwรคgen:
- Kostenverlauf fรผr Vision-Tokens: Das kontinuierliche Senden von 1080p-Screenshots an ein kommerzielles Modell alle 2 Sekunden verursacht 15 bis 30 US-Dollar pro 100-Schritt-Workflow.
- Strategisches Denken vs. repetitive RPA: Claude 3.7 Sonnet glรคnzt bei uneindeutigen Aufgaben, die semantisches Verstรคndnis unstrukturierter Dokumente erfordern. UI-TARS รผberzeugt bei hochfrequenten Ablรคufen mit minimalen Rechenkosten.
- Datenschutz & Compliance: Banken und Behรถrden fordern, dass Bildschirminhalte das interne Firmennetz niemals verlassen โ was selbst gehostete UI-TARS-Instanzen zum Standard macht.
06. Praktische Umsetzung: Aufbau eines sicheren Desktop-Agenten-Controllers in Python
Nachfolgend finden Sie eine vollstรคndige, lauffรคhige Referenzimplementierung in Python, die Koordinatenskalierung, Sicherheitsรผberwachung, das Abfangen destruktiver Befehle und menschliche Bestรคtigung demonstriert:
# Production Reference Implementation: Desktop GUI Agent Controller (2026)
# Demonstrates Vision-Language-Action Execution, Coordinate Normalization,
# Destructive Command Interception, and Human-in-the-Loop Safeguards.
import time
import math
from typing import Dict, Any, List, Tuple, Optional
from dataclasses import dataclass, field
from enum import Enum
# โโโ 1. CORE DATA STRUCTURES & ACTIONS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class ActionType(str, Enum):
MOUSE_CLICK = "mouse_click"
DOUBLE_CLICK = "double_click"
HOTKEY = "hotkey"
TYPE_TEXT = "type_text"
TERMINAL_COMMAND = "terminal_command"
WAIT = "wait"
@dataclass
class UIAction:
action_type: ActionType
coordinates: Optional[Tuple[int, int]] = None # (x, y) on 1000x1000 normalized grid
text_payload: Optional[str] = None
hotkey_sequence: Optional[List[str]] = None
thought_reasoning: str = ""
is_destructive: bool = False
@dataclass
class ScreenDimensions:
width: int
height: int
# โโโ 2. SAFETY INTERCEPTOR & GATEKEEPER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class DesktopSafetyGatekeeper:
"""
Host-side safety authority that validates actions before dispatching
to real or virtual OS input peripherals.
"""
DESTRUCTIVE_HOTKEYS = {("ctrl", "alt", "del"), ("cmd", "shift", "backspace")}
DESTRUCTIVE_COMMANDS = ["rm -rf", "format", "drop database", "mkfs", "dd if="]
def __init__(self, require_human_for_destructive: bool = True):
self.require_human = require_human_for_destructive
self.audit_log: List[Dict[str, Any]] = []
def inspect_action(self, action: UIAction, human_approved: bool = False) -> Tuple[bool, str]:
# 1. Inspect terminal/shell commands
if action.action_type == ActionType.TERMINAL_COMMAND and action.text_payload:
for pattern in self.DESTRUCTIVE_COMMANDS:
if pattern in action.text_payload.lower():
action.is_destructive = True
if not human_approved:
return False, f"Blocked destructive terminal command pattern: '{pattern}'"
# 2. Inspect high-risk hotkeys
if action.action_type == ActionType.HOTKEY and action.hotkey_sequence:
normalized_keys = tuple(sorted([k.lower() for k in action.hotkey_sequence]))
for risk_keys in self.DESTRUCTIVE_HOTKEYS:
if normalized_keys == tuple(sorted(risk_keys)):
action.is_destructive = True
if not human_approved:
return False, f"Blocked dangerous system hotkey: '{action.hotkey_sequence}'"
# Log action to immutable audit trail
self.audit_log.append({
"timestamp": time.time(),
"action": action.action_type.value,
"coordinates": action.coordinates,
"destructive": action.is_destructive,
"approved": True
})
return True, "Action approved."
# โโโ 3. PERIPHERAL ADAPTER & COORDINATE SCALER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class OSPeripheralController:
"""
Translates normalized model coordinates (1000x1000) to actual physical display pixels
and dispatches low-level OS input events.
"""
def __init__(self, display: ScreenDimensions):
self.display = display
def denormalize_coordinates(self, norm_x: int, norm_y: int) -> Tuple[int, int]:
"""Converts [0, 1000] model grid to [0, width] x [0, height]."""
real_x = math.floor((norm_x / 1000.0) * self.display.width)
real_y = math.floor((norm_y / 1000.0) * self.display.height)
return real_x, real_y
def execute_native_input(self, action: UIAction):
if action.coordinates:
rx, ry = self.denormalize_coordinates(*action.coordinates)
print(f"๐ฑ๏ธ [OS DRIVER] Moving cursor to ({rx}px, {ry}px) [Model: {action.coordinates}]")
if action.action_type == ActionType.MOUSE_CLICK:
print(f"โก [OS DRIVER] Emitting LEFT_BUTTON_CLICK at ({rx}, {ry})")
elif action.action_type == ActionType.DOUBLE_CLICK:
print(f"โก [OS DRIVER] Emitting DOUBLE_CLICK at ({rx}, {ry})")
elif action.action_type == ActionType.HOTKEY:
print(f"โจ๏ธ [OS DRIVER] Emitting KEY_COMBINATION: {' + '.join(action.hotkey_sequence or [])}")
elif action.action_type == ActionType.TYPE_TEXT:
print(f"โจ๏ธ [OS DRIVER] Typing string payload: \"{action.text_payload}\"")
elif action.action_type == ActionType.TERMINAL_COMMAND:
print(f"๐ฅ๏ธ [OS DRIVER] Executing Shell Command: '{action.text_payload}'")
# โโโ 4. AUTONOMOUS DESKTOP AGENT CONTROLLER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class DesktopGUIAgent:
"""
The orchestrator agent combining VLA decision logic, safety inspection,
and OS input driver dispatch.
"""
def __init__(self, display: ScreenDimensions, gatekeeper: DesktopSafetyGatekeeper):
self.driver = OSPeripheralController(display)
self.gatekeeper = gatekeeper
def step(self, action: UIAction, human_confirmed: bool = False) -> Dict[str, Any]:
print(f"\n๐ง [SYSTEM-2 REFLECTION] {action.thought_reasoning}")
# Safety Check
is_safe, reason = self.gatekeeper.inspect_action(action, human_approved=human_confirmed)
if not is_safe:
print(f"๐ [SAFETY INTERCEPT] Action Rejected: {reason}")
return {"success": False, "reason": reason}
# Dispatch
self.driver.execute_native_input(action)
return {"success": True, "reason": "Executed successfully"}
# โโโ 5. RUNTIME VERIFICATION HARNESS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
if __name__ == "__main__":
print("=" * 75)
print("DEMO: AUTONOMOUS DESKTOP GUI AGENT SAFETY & GROUNDING CONTROLLER (2026)")
print("=" * 75)
# Initialize 1080p display and safety gatekeeper
virtual_screen = ScreenDimensions(width=1920, height=1080)
safety_shield = DesktopSafetyGatekeeper(require_human_for_destructive=True)
agent = DesktopGUIAgent(display=virtual_screen, gatekeeper=safety_shield)
# STEP 1: Safe Action - Navigate SAP Menu
step1 = UIAction(
action_type=ActionType.MOUSE_CLICK,
coordinates=(180, 45), # 1000x1000 normalized grid
thought_reasoning="Milestone 1: Click 'Accounting' dropdown in SAP native menu bar."
)
agent.step(step1)
# STEP 2: Safe Action - Type Transaction Code
step2 = UIAction(
action_type=ActionType.TYPE_TEXT,
text_payload="FS10N",
thought_reasoning="Milestone 2: Enter general ledger balance inquiry transaction code."
)
agent.step(step2)
# STEP 3: Malicious / Accidental Destructive Step Intercepted
step3_destructive = UIAction(
action_type=ActionType.TERMINAL_COMMAND,
text_payload="rm -rf /var/sap/data/*",
thought_reasoning="Adversarial prompt injection attempt detected on unverified clipboard buffer."
)
print("\n[SCENARIO 1: Destructive Action Without Approval]")
agent.step(step3_destructive, human_confirmed=False)
# STEP 4: High-Risk Action With Human Biometric Approval
print("\n[SCENARIO 2: Authorized High-Risk Action via Supervisor Approval]")
agent.step(step3_destructive, human_confirmed=True)
print("\n" + "=" * 75)
print(f"Demonstration Complete. Total Audit Ledger Entries: {len(safety_shield.audit_log)}")
print("=" * 75)
07. OSWorld 2.0 Benchmark-Analyse: Bewรคltigung von Long-Horizon-Drift & Fehlermustern
Der Benchmark OSWorld 2.0 bewertet Agenten anhand von 369 praxisnahen Aufgaben in Ubuntu, Windows und macOS. Wรคhrend menschliche Experten 88.3 % Erfolgsquote erreichen, erzielen Spitzenagenten zwischen 49 % und 52 % Gesamtergebnis, was vier typische Fehlerursachen verdeutlicht:
- Auflรถsungs- und Koordinatendrift: Pop-up-Meldungen verschieben Oberflรคchenelemente um wenige Pixel, wodurch Klicks auf veraltete Koordinaten ins Leere gehen.
- Kontextfenster-Erschรถpfung: 60 hochauflรถsende Screenshots รผberfordern den Kontext des Modells. Produktionssysteme verwerfen รคltere Bilder und verdichten sie zu textuellen Aktionsprotokollen.
- Ladeanzeigen-Fallen: Da dem Desktop Events wie
networkidlefehlen, klicken Agenten wรคhrend Ladevorgรคngen mehrfach, was Anwendungsabstรผrze provoziert. Visuelle Deltaprรผfungen lรถsen dieses Problem. - Kontrastarme Benutzeroberflรคchen: Im Dark Mode oder in CAD-Programmen verwechseln Modelle hรคufig รคhnlich aussehende Icons (z. B. Speichern vs. Exportieren).
08. Sicherheit & Sandboxing: VNC-Isolation, eBPF-Netzwerkfilter und Kill-Switch-Schutz
Einem autonomen Agenten direkten Zugriff auf physische Peripheriegerรคte auf Mitarbeiterrechnern zu gewรคhren, birgt unkalkulierbare Risiken. Produktionsarchitekturen erzwingen eine dreistufige Sicherheits-Sandbox:
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Zero-Trust Desktop Sandbox Containment Stack |
| |
| [ Enterprise Gateway ] |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 1: VIRTUAL DISPLAY & PERIPHERAL ISOLATION โ |
| โ - Headless X11 / Wayland / RDP virtual server โ |
| โ - The agent NEVER touches physical user hardware โ |
| โ - Video frame streamed via WebRTC / VNC buffer โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 2: SYSTEM CALL INTERCEPTION & eBPF NETWORK EGRESS โ |
| โ - eBPF sensor intercepts dangerous POSIX syscalls (fork, execve, ptrace)โ
| โ - Network firewall restricts outbound traffic exclusively to whitelistโ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 3: SUPERVISOR KILL-SWITCH & USER TAKEOVER โ |
| โ - User moves physical mouse โโโถ Instant agent suspension (Kill-Switch)โ |
| โ - Live action stream mirrored to supervisor dashboard โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+- Stufe 1: Virtuelle Bildschirmoberflรคche: Der Agent interagiert ausschlieรlich mit einem virtuellen X11/Wayland/RDP-Puffer in einer isolierten MicroVM (z. B. E2B). Physische Hardware wird niemals berรผhrt.
- Stufe 2: eBPF-Syscall- & Egress-Filterung: Sensoren auf Kernelebene fangen gefรคhrliche Shell-Befehle (
rm -rf, Formatierungsbefehle, Datenexfiltration) ab und blockieren Verbindungen auรerhalb der Whitelist. - Stufe 3: Physischer Notfall-Kill-Switch: Auf assistierten Arbeitsplรคtzen entzieht das Bewegen der physischen Maus oder Drรผcken von
Escdem Agenten sofort alle Treiberrechte und gibt dem Menschen die volle Kontrolle zurรผck.
09. Architektur-Vergleichsmatrix & Relevante Tools
Wie positionieren sich fรผhrende Desktop-Agenten-Frameworks im Jahr 2026?
| Dimension | UI-TARS (ByteDance) | Claude 3.7 Computer Use | OpenHands | E2B Desktop Sandbox |
|---|---|---|---|---|
| Modellcharakter | Open-Weights (VLA 7B/72B) | Proprietรคre Spitzen-API | Modell-agnostischer Orchestrator | Infrastruktur-Sandbox-Laufzeit |
| Hauptwahrnehmung | Rohe Pixel (System-2 VLA) | Rohe Pixel (Vision-API) | Hybrid (DOM + Terminal + GUI) | Virtueller Bildschirm-Frame-Puffer |
| Hosting-Modell | Self-Hosted (Lokale GPU-Cluster) | Cloud-API (Anthropic) | Self-Hosted / Cloud | Managed Cloud / MicroVM |
| Betriebssystem-Support | Windows, macOS, Linux, Android | Windows, macOS, Ubuntu | Linux, Docker, Browser | Linux (Firecracker MicroVM) |
| Typischer Einsatzzweck | Hochfrequente RPA, QA, Batch | Komplexe Wissensarbeit | Software-Engineering-Agenten | Sichere Isolierung & Ausfรผhrung |
| Open Source | 100% Open Source | Proprietรคr | 100% Open Source | Open-Source-SDK / Managed Cloud |
E2B
MicroVM-SandboxDie fรผhrende MicroVM-Cloud-Laufzeitumgebung zur sicheren Ausfรผhrung von Desktop-Sitzungen, Terminalbefehlen und Headless-Workflows mit nativer Hardware-Isolation.
E2B entdecken โClaude 3.7 Sonnet
Frontier-ModellAnthropics Spitzenmodell mit nativen Computer Use-Fรคhigkeiten fรผr Desktop-Navigation, Browser-Automatisierung und mehrstufige Tool-Aufrufe.
Claude 3.7 Sonnet entdecken โOpenHands
Open SourceDie fรผhrende Open-Source-Plattform fรผr autonome Software-Entwicklungsagenten mit vollstรคndiger Unterstรผtzung von Terminal-, Browser- und Desktop-Aktionen.
OpenHands entdecken โDevin
Autonome SoftwareCognitions bahnbrechender autonomer Software-Engineering-Assistent mit integrierter Browser-, Terminal- und Desktop-Automatisierung.
Devin entdecken โ10. Hรคufig gestellte Fragen (FAQ)
F1: Warum baut man keine individuellen APIs statt Desktop-GUI-Agenten?
In einer idealen Welt bรถte jede Software REST- oder GraphQL-Schnittstellen. In der Praxis existieren in Unternehmen Tausende Legacy-Systeme (SAP, Mainframe-Terminals, Buchhaltungsprogramme), deren API-Nachrรผstung Millionen kosten und Jahre dauern wรผrde. Desktop-GUI-Agenten ermรถglichen Zero-Touch-Automatisierung ohne Eingriffe in Quellcode oder Datenbanken.
F2: Welche Bildschirmauflรถsung sollte an den visuellen Agenten รผbergeben werden?
4K-Rohbilder erzeugen extremes Token-Wachstum und hohe Latenzen. Produktionssysteme zeichnen รผblicherweise in 1920x1080 auf, normalisieren auf ein internes 1000x1000-Raster fรผr die Modellinferenz und rechnen vorhergesagte Koordinaten mathematisch auf physische Bildschirmpixel um.
F3: Wie bewรคltigen Desktop-Agenten dynamische Ladezeiten und Animationen?
Im Gegensatz zu Browsern mit Events wie DOMContentLoaded oder networkidle gibt es auf dem Desktop keine nativen Ladeend-Meldungen. Produktionsagenten erfassen Frames im 250ms-Intervall und prรผfen, ob die Pixelvarianz unter 1 % fรคllt, bevor der nรคchste Schritt ausgefรผhrt wird.
F4: Kรถnnen Desktop-Agenten mit mehreren Monitoren oder Fenstern arbeiten?
Ja. Koordinatentreiber behandeln Multi-Monitor-Setups entweder als zusammenhรคngende Arbeitsflรคche (z. B. 3840x1080) oder nutzen Fenster-Fokus-APIs, um die Zielanwendung vor der visuellen Erfassung auf das primรคre Display zu legen.
F5: Kann UI-TARS lokal auf Standard-Hardware betrieben werden?
Das Modell UI-TARS 7B lรคuft lokal auf moderner Hardware (z. B. Macs mit Apple Silicon und 32GB+ Unified Memory oder NVIDIA RTX 4090) mit GGUF- oder AWQ-Quantisierung. Fรผr das 72B-Modell sind Unternehmens-GPUs (A100/H100) fรผr Sub-Sekunden-Latenz erforderlich.
ใ2026ๅนด็ใใใฉใฆใถ่ชๅๅใฎๅ ใธ๏ผUI-TARSใClaude Computer UseใOSWorld 2.0ใงๆง็ฏใใใในใฏใใใGUI่ชๅพๆไฝใจใผใธใงใณใๅฎ่ทต
้ๅป3ๅนด้ใ็ๆAIใจใณใทในใใ ใฏBrowser-UseใStagehandใPlaywright MCPใชใฉใ็จใใWebใใฉใฆใถDOM่งฃๆใฎ่ชๅๅใซๆณจๅใใฆใใพใใใใใใใใจใณใฟใผใใฉใคใบ้ ๅใซใใใๅฎๆฅญๅใฝใใใฆใงใขใฎ70%ไปฅไธใฏใใฉใฆใถDOMใๆใกใพใใใใฌใฌใทใผERP๏ผSAP GUI๏ผใ่ค้ใชVBAใใฏใญใ็ตใฟ่พผใใ ใใคใใฃใExcelใใในใฏใใใCADใใฟใผใใใซใณใณใฝใผใซใใชใณใใฌใในใฎๅฐ็จๆฅญๅใฏใฉใคใขใณใใซใฏWeb DOMใๅญๅจใใใDirectXใMetalใWin32ใQtใง็ดๆฅๆ็ปใใใ็ป้ขใซๅฏพใใฆDOMใใผในใฎใจใผใธใงใณใใฏๅฎๅ จใซ็กๅใงใใใ็ใฎใจใณใฟใผใใฉใคใบ่ชๅๅใๅฎ็พใใใใใ2026ๅนดใฏใใคใใฃใใช่ฆ่ฆใป่จ่ชใป่กๅ๏ผVLA: Vision-Language-Action๏ผใขใใซใใใฏใปใซๅบงๆจใฐใฉใฆใณใใฃใณใฐใSystem-2้ ่ๆจ่ซใๅใใ่ชๅพๅใในใฏใใใGUIใจใผใธใงใณใใๅฐ้ ญใใฆใใพใใๆฌ็จฟใงใฏใUI-TARS 1.5ใClaude 3.7 Computer UseใOSWorld 2.0ใใณใใใผใฏใใใใฆๅฎๅ จใชใตใณใใใใฏใน้้ขๅบ็คใๅพนๅบ่งฃๅใใพใใ
๐ ็ฎๆฌก
01. ่ฆ็ดใจใในใฏใใใGUIใจใผใธใงใณใใฎๆๅ็ท
Web่ชๅๅใฏๅฎ็จๆฎต้ใซ้ใใพใใใใใในใฏใใใไธใฎ็ฅ็ๆฅญๅใฏ้ทใใๆไฝๆฅญใฎใพใพใงใใใ2026ๅนดใใในใฏใใใGUIใจใผใธใงใณใใฏOS็ป้ขๅ จไฝใๅฏพ่ฉฑๅใญใฃใณใในใจใใฆ็ดๆฅๆฑใใพใ๏ผ
- HTMLใปใฌใฏใฟใใใฎ่ฑๅด๏ผใจใผใธใงใณใใฏ็ใฎ็ป้ขใใฌใผใ ๏ผ1080pใ4K๏ผใ็ฃ่ฆใใใใซใใขใผใใซLLM๏ผMLLM๏ผใ้ใใฆUI่ฆ็ด ใใใฏใปใซๅไฝใง็ดๆฅ่ช่ญใๆญฃ่ฆๅๅบงๆจ (x, y) ใOSใฎใใฆในใปใญใผใใผใๅ ฅๅใคใใณใใซใใใใณใฐใใพใใ
- ใชใผใใณใฝใผในใฎๅ ้ ญ vs ๅ็จๆ้ซๅณฐAPI๏ผByteDanceใๅ ฌ้ใใใชใผใใณใฝใผในใขใใซ UI-TARS 1.5 ใฏๅผทๅๅญฆ็ฟใ็จใใSystem-2่ชๅทฑๆค่จผใปใใใฏใใฉใใญใณใฐ๏ผๅทปใๆปใ๏ผใๅฐๅ ฅใAnthropicใฎ Claude 3.7 Sonnet ใฏ้ซๅบฆใชๆจ่ซใจใใคใใฃใใชComputer Useใใผใซๅผใณๅบใใๆไพใใพใใ
- OSWorld 2.0ใฎ็พๅฎ่งฃ๏ผUbuntuใWindowsใmacOSใซใใใ369ใฎๅฎ็ฐๅขใฟในใฏใงๆงๆใใใOSWorld 2.0ใซใใใฆใใจใผใธใงใณใใฏ100ในใใใใฎ้ทๆใฟในใฏใง42%ใ52%ใฎๆๅ็ใ่จ้ฒใในใใใๆฐๅขๅ ใซไผดใ็ถๆ ใใชใใ๏ผ่ฑ็ท๏ผใฎ้ฒๆญขใๆๅคงใฎ็ฆ็นใจใชใฃใฆใใพใใ
- ๅฟ ้ ใจใชใใตใณใใใใฏใน้้ข๏ผOSใซๅฏพใใ็ดๆฅๆไฝใฏ็นๆจฉใชในใฏใไผดใใพใใๆฌ็ชใขใผใญใใฏใใฃใงใฏใไฝฟใๆจใฆMicroVM๏ผE2Bใชใฉ๏ผใไปฎๆณใใฃในใใฌใคในใใชใผใ ๏ผVNC/RDP๏ผๅ ใงใจใผใธใงใณใใๅฎ่กใใใในใๅดใใกใคใขใฆใฉใผใซใจ็ทๆฅใญใซในใคใใใไฝต็จใใพใใ
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Autonomous Desktop GUI Agent Architecture (2026) |
| |
| [ User Goal: "Consolidate Q3 SAP exports into Excel macro, generate PDF" ] |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ HOST SUPERVISOR & ACTION FIREWALL (Safety Interceptor) โ |
| โ - Rate Limiting & Egress Filtering - Destructive Command Blocker โ |
| โ - Biometric Confirmation Gateway - Emergency Human Kill-Switch โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ Virtual Display / Peripherals |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ ISOLATED DESKTOP SANDBOX (E2B / Cloud VNC / KVM Virtual Machine) โ |
| โ โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ |
| โ โ Screenshot Buffer โ โ OS Input Controller โ โ |
| โ โ (1920x1080 RGB) โ โ (PyAutoGUI / uinput) โ โ |
| โ โโโโโโโโโโโโฌโโโโโโโโโโโ โโโโโโโโโโโโโโโฒโโโโโโโโโโโโ โ |
| โ โ Raw Frame โ (x, y) Click โ |
| โ โผ โ & Hotkeys โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ โ |
| โ โ AGENT RUNTIME: UI-TARS 1.5 / Claude 3.7 Sonnet โ โ |
| โ โ 1. Visual Grounding: Detect target UI elements via pixels โ โ |
| โ โ 2. System-2 Deliberation: Check milestones & reflect โ โ |
| โ โ 3. Action Plan: Emit precise mouse, click, and key sequences โ โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ |
| โ โ |
| โ [ Legacy SAP Client ] [ Native Excel ] [ Desktop CAD ] โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+02. DOMไพๅญใฎ็ต็๏ผใชใใในใฏใใใๆฅญๅใ้่ฆใชใฎใ
ๅพๆฅใฎใใฉใฆใถ่ชๅๅใฏใใขใใชใฑใผใทใงใณใๆง้ ๅใใใใขใฏใปใทใใชใใฃใใชใผใไบๆธฌๅฏ่ฝใชCSSใฏใฉในใๆไพใใใใจใๅๆใจใใฆใใพใใใใใไผๆฅญใฎๅบๅนนๆฅญๅใงใฏใใฎๅๆใๅดฉๅฃใใพใ๏ผ
1. Canvasๆ็ปใฎWebใขใใช
FigmaใCanvasๆ็ปใขใผใใฎGoogle DocsใWebGLใใใทใฅใใผใใฏใDOMใใผใใๆใใชใ็ฉบใฎใญใฃใณใในไธใซ็ดๆฅUIใๆ็ปใใพใใ
2. ใฌใฌใทใผๅบๅนนใทในใใ
SAP GUIใAS400็ซฏๆซใจใใฅใฌใผใฟใBloomberg Terminalใๅป็้ปๅญใซใซใใชใฉใฎใใใทใงใณใฏใชใใฃใซใซใชใทในใใ ใฏใWebใคใณใฟใผใใงใผในใๆใใชใใใคใใฃใใใคใใชใงใใ
3. ่คๆฐใขใใชใฎๆจชๆญ้ฃๆบ
ใใผใฟใฎใใฆใณใญใผใใใญใผใซใซExcelใงใฎVBAใใฏใญๅฎ่กใ็คพๅ CRMใฎๆดๆฐใPDF็ฝฒๅใชใฉใOSใฎใใญใปในๅข็ใ่ทจใ่คๅๆฅญๅใๆฅๅธธใฎๅคงๅใๅ ใใพใใ
03. 3ใคใฎ็ฅ่ฆใขใผใญใใฏใใฃใฎๆฏ่ผ๏ผใใฏใปใซ vs ใขใฏใปใทใใชใใฃใใชใผ vs ใใคใใชใใ
ใจใผใธใงใณใใฏใในใฏใใใ็ป้ขใใฉใฎใใใซ่ช่ญใในใใงใใใใ๏ผ2026ๅนด็พๅจใฏ3ใคใฎ่จญ่จใขใใญใผใใๅญๅจใใพใ๏ผ
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Desktop Perception Paradigms |
| |
| [ Approach 1: Pure Visual Grounding (Pixel-to-Coordinate) ] |
| Screen Frame โโโถ High-Res VLM โโโถ Coordinates (x: 450, y: 720) |
| โข Pros: Universal, zero OS dependency, handles Canvas / Games / Legacy |
| โข Cons: Heavy token cost, coordinate distortion, resolution scaling drift|
| |
| [ Approach 2: OS Accessibility Tree Grounding (UIAutomation / AT-SPI) ] |
| Screen State โโโถ OS API Walk โโโถ Filtered Hierarchy โโโถ Element ID / Path|
| โข Pros: Deterministic, lightweight text tokens, 100% click precision |
| โข Cons: 40% of native apps have broken/missing a11y trees, slow tree walk|
| |
| [ Approach 3: Dual-Stream Hybrid Fusion (2026 Best Practice) ] |
| Visual Screenshot (VLM) โโโFused Decisionโโโบ OS A11y Tree (Cache) |
| โข Fast path: Use A11y node bounding box if recognized |
| โข Fallback: Use visual grounding when a11y nodes are obscured/custom |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+| ๆฏ่ผ่ปธ | ็ด็ฒใใฏใปใซใฐใฉใฆใณใใฃใณใฐ | OSใขใฏใปใทใใชใใฃใใชใผ | ใใฅใขใซในใใชใผใ ใปใใคใใชใใ่ๅ |
|---|---|---|---|
| ๆฑ็จๆง | 100%๏ผ็ป้ขไธใฎใใใใใใฏใปใซ๏ผ | ็ด60%๏ผใซในใฟใ ๆ็ปUIใงๅคฑๆ๏ผ | 98%๏ผๆฎต้็ใใฉใผใซใใใฏ๏ผ |
| ใใผใฏใณๅน็ | ไฝ๏ผ็ด1,200ใ2,000ใใผใฏใณ/็ป้ข๏ผ | ้ซ๏ผ็ด300ใ600ใใผใฏใณ/ใใชใผ๏ผ | ไธญ๏ผ็ด1,500ใใผใฏใณ/ๅคๆญ๏ผ |
| ใฏใชใใฏ็ฒพๅบฆ | 88% - 94%๏ผๅบงๆจใบใฌใฎใชในใฏใใ๏ผ | 99%๏ผใใผใใๅญๅจใใๅ ดๅ๏ผ | 98.5%๏ผ่ฟๅใใฆใณใใฃใณใฐใใใฏในใซๅธ็๏ผ |
| OS้ไพๅญๆง | ๅฎๅ จ๏ผVNC/็ป้ข้ ไฟกใซๅฏพๅฟ๏ผ | ไฝ๏ผOSๅบๆใฎAPIใใใฏใๅฟ ่ฆ๏ผ | ้ซ๏ผใขใธใฅใผใซๅผใขใใใฟใผ๏ผ |
| ๆจ่ซใฟใผใณใฌใคใใณใท | 1.8็ง - 3.5็ง๏ผVLMๆจ่ซ๏ผ | 0.4็ง - 0.9็ง๏ผใใญในใLLM๏ผ | 1.9็ง - 3.2็ง |
04. UI-TARS๏ผใใคใใฃใVLAใจSystem-2้ ่ๆจ่ซ
ByteDanceใ้็บใปใชใผใใณใฝใผในๅใใ UI-TARS ใฏใGUIๆไฝๅฐ็จใฎใใคใใฃใใช **Vision-Language-Action (VLA)** ใขใใซใฎ่ๅใใงใใๆฑ็จ่ฆ่ฆใขใใซใซๅพไปใใใญใณใใใๅฝใฆใใฎใงใฏใชใใๅไฝใใผใฏใณใฎ็ดๆฅ็ๆใจๅผทๅๅญฆ็ฟใซๅบใฅใๅ ็ใใญใปใน๏ผSystem-2๏ผใ็ตฑๅใใฆใใพใ๏ผ
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| UI-TARS 1.5 System-2 Reasoning Trajectory |
| |
| Observation โโโถ [ Reflection & Verification ] |
| โ "Did the previous action open the File dialog?" |
| โ State: YES. Detected 'Open File' window at (x: 200, y: 150)
| โผ |
| [ Sub-goal Decomposition ] |
| โ "Next sub-goal: Select 'Quarterly_Report.xlsx'" |
| โ Search Strategy: Visual scan of table rows |
| โผ |
| [ Milestone Recognition ] |
| โ Target element found at (x: 320, y: 410) |
| โผ |
| [ Action Emission ] |
| โ Action: click(point=[320, 410]) |
| โ Post-Action Expectation: File selected in input box |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+- ๆจๆบๅใใใ้ๅใใผใฏใณ๏ผๆญฃ่ฆๅใใใ 1000 × 1000 ๅบงๆจ็ณปใซใใใฆใ
click(point=[x, y])ใdouble_click()ใdrag(start, end)ใpress_hotkey()ใชใฉใฎใใผใฏใณใ็ดๆฅๅบๅใ - System-2่ชๅทฑๆค่จผ๏ผ็ดๅใฎๆไฝใๆๅพ ใใ็ป้ข้ท็งปใ่ตทใใใใใไบๅใซ็ขบ่ชใใใฟใณๆผไธใงใใคใขใญใฐใ้ใใชใใฃใๅ ดๅใ้ใกใ็นฐใ่ฟใใ่ชๅใงๅ่ฉฆ่กใพใใฏๅฅใซใผใใๆข็ดขใ
- ใญใผใซใซ้ ๅๆง๏ผUI-TARS 7Bใขใใซใฏใญใผใซใซใฏใผใฏในใใผใทใงใณ๏ผApple Silicon 32GBไปฅไธใใพใใฏRTX 4090๏ผใงๅฎ่กๅฏ่ฝใงใใใๆฉๅฏใใผใฟใๅค้จใซๅบใใ1็งๆชๆบใงๅฟ็ญใใพใใ
05. Claude 3.7 Computer Use vs ใชใผใใณใขใใซใฎใใฌใผใใชใ๏ผใฌใคใใณใทใใณในใใ็ฒพๅบฆ
้็บใใผใ ใฏๅ็จใฎๆๅ็ทAPI๏ผClaude 3.7 Sonnet๏ผใจ่ชๅใในใใฎใชใผใใณใฆใงใคใใขใใซ๏ผUI-TARS 7B/72B๏ผใฎใฉใกใใๆก็จใในใใงใใใใ๏ผ
- ่ฆ่ฆใใผใฏใณใณในใใฎ่ฉฆ็ฎ๏ผ1080pใฎในใฏใชใผใณใทใงใใใ2็งใใจใซๅ็จใขใใซใธ้ไฟกใ็ถใใใจใ100ในใใใใฎๆฅญๅๅฆ็ใง1ๅใใใ15ใ30ใใซใฎใณในใใ็บ็ใใพใใ
- ้ซๅบฆใชๆจ่ซ vs ๅฎๅๅคง้RPA๏ผๆๆงใชๆ็คบใ่ค้ใช้ๆง้ ๅๆๆธใ่ชญใฟ่งฃใๆฅญๅใซใฏClaude 3.7 Sonnetใๆ้ฉใงใใใๅฎๅ็ใชๅคง้ใใผใฟ่ปข้ใๅๅพฉๅ ฅๅใงใฏUI-TARSใใณในใ้ขใงๅงๅ็ๅชไฝใซ็ซใกใพใใ
- ใใผใฟใฌใใใณใน๏ผ้่ใปๅป็ๅ้ใชใฉใ็ป้ขไธใฎๅไบบๆ ๅ ฑใ่ฒกๅใใผใฟใๅค้จใฏใฉใฆใAPIใธ้ไฟกใงใใชใ็ฐๅขใงใฏใใชใณใใฌใในใง้็จๅฏ่ฝใชUI-TARSใๆจๆบ็ใช้ธๆ่ขใจใชใใพใใ
06. Pythonๅฎ่ทตๅฎ่ฃ ๏ผๅฎๅ จใชใในใฏใใใๆไฝใณใณใใญใผใฉใผใฎๆง็ฏ
ไปฅไธใฏใๅบงๆจๆญฃ่ฆๅในใฑใผใชใณใฐใๅฎๅ จใฒใผใใญใผใใผใ็ ดๅฃ็ใณใใณใใฎๆค็ฅ้ฎๆญใไบบ้ไปๅ ฅๆฟ่ชใๅซใใๅฎๅ จใซๅฎ่กๅฏ่ฝใชPythonใชใใกใฌใณในๅฎ่ฃ ใงใ๏ผ
# Production Reference Implementation: Desktop GUI Agent Controller (2026)
# Demonstrates Vision-Language-Action Execution, Coordinate Normalization,
# Destructive Command Interception, and Human-in-the-Loop Safeguards.
import time
import math
from typing import Dict, Any, List, Tuple, Optional
from dataclasses import dataclass, field
from enum import Enum
# โโโ 1. CORE DATA STRUCTURES & ACTIONS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class ActionType(str, Enum):
MOUSE_CLICK = "mouse_click"
DOUBLE_CLICK = "double_click"
HOTKEY = "hotkey"
TYPE_TEXT = "type_text"
TERMINAL_COMMAND = "terminal_command"
WAIT = "wait"
@dataclass
class UIAction:
action_type: ActionType
coordinates: Optional[Tuple[int, int]] = None # (x, y) on 1000x1000 normalized grid
text_payload: Optional[str] = None
hotkey_sequence: Optional[List[str]] = None
thought_reasoning: str = ""
is_destructive: bool = False
@dataclass
class ScreenDimensions:
width: int
height: int
# โโโ 2. SAFETY INTERCEPTOR & GATEKEEPER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class DesktopSafetyGatekeeper:
"""
Host-side safety authority that validates actions before dispatching
to real or virtual OS input peripherals.
"""
DESTRUCTIVE_HOTKEYS = {("ctrl", "alt", "del"), ("cmd", "shift", "backspace")}
DESTRUCTIVE_COMMANDS = ["rm -rf", "format", "drop database", "mkfs", "dd if="]
def __init__(self, require_human_for_destructive: bool = True):
self.require_human = require_human_for_destructive
self.audit_log: List[Dict[str, Any]] = []
def inspect_action(self, action: UIAction, human_approved: bool = False) -> Tuple[bool, str]:
# 1. Inspect terminal/shell commands
if action.action_type == ActionType.TERMINAL_COMMAND and action.text_payload:
for pattern in self.DESTRUCTIVE_COMMANDS:
if pattern in action.text_payload.lower():
action.is_destructive = True
if not human_approved:
return False, f"Blocked destructive terminal command pattern: '{pattern}'"
# 2. Inspect high-risk hotkeys
if action.action_type == ActionType.HOTKEY and action.hotkey_sequence:
normalized_keys = tuple(sorted([k.lower() for k in action.hotkey_sequence]))
for risk_keys in self.DESTRUCTIVE_HOTKEYS:
if normalized_keys == tuple(sorted(risk_keys)):
action.is_destructive = True
if not human_approved:
return False, f"Blocked dangerous system hotkey: '{action.hotkey_sequence}'"
# Log action to immutable audit trail
self.audit_log.append({
"timestamp": time.time(),
"action": action.action_type.value,
"coordinates": action.coordinates,
"destructive": action.is_destructive,
"approved": True
})
return True, "Action approved."
# โโโ 3. PERIPHERAL ADAPTER & COORDINATE SCALER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class OSPeripheralController:
"""
Translates normalized model coordinates (1000x1000) to actual physical display pixels
and dispatches low-level OS input events.
"""
def __init__(self, display: ScreenDimensions):
self.display = display
def denormalize_coordinates(self, norm_x: int, norm_y: int) -> Tuple[int, int]:
"""Converts [0, 1000] model grid to [0, width] x [0, height]."""
real_x = math.floor((norm_x / 1000.0) * self.display.width)
real_y = math.floor((norm_y / 1000.0) * self.display.height)
return real_x, real_y
def execute_native_input(self, action: UIAction):
if action.coordinates:
rx, ry = self.denormalize_coordinates(*action.coordinates)
print(f"๐ฑ๏ธ [OS DRIVER] Moving cursor to ({rx}px, {ry}px) [Model: {action.coordinates}]")
if action.action_type == ActionType.MOUSE_CLICK:
print(f"โก [OS DRIVER] Emitting LEFT_BUTTON_CLICK at ({rx}, {ry})")
elif action.action_type == ActionType.DOUBLE_CLICK:
print(f"โก [OS DRIVER] Emitting DOUBLE_CLICK at ({rx}, {ry})")
elif action.action_type == ActionType.HOTKEY:
print(f"โจ๏ธ [OS DRIVER] Emitting KEY_COMBINATION: {' + '.join(action.hotkey_sequence or [])}")
elif action.action_type == ActionType.TYPE_TEXT:
print(f"โจ๏ธ [OS DRIVER] Typing string payload: \"{action.text_payload}\"")
elif action.action_type == ActionType.TERMINAL_COMMAND:
print(f"๐ฅ๏ธ [OS DRIVER] Executing Shell Command: '{action.text_payload}'")
# โโโ 4. AUTONOMOUS DESKTOP AGENT CONTROLLER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class DesktopGUIAgent:
"""
The orchestrator agent combining VLA decision logic, safety inspection,
and OS input driver dispatch.
"""
def __init__(self, display: ScreenDimensions, gatekeeper: DesktopSafetyGatekeeper):
self.driver = OSPeripheralController(display)
self.gatekeeper = gatekeeper
def step(self, action: UIAction, human_confirmed: bool = False) -> Dict[str, Any]:
print(f"\n๐ง [SYSTEM-2 REFLECTION] {action.thought_reasoning}")
# Safety Check
is_safe, reason = self.gatekeeper.inspect_action(action, human_approved=human_confirmed)
if not is_safe:
print(f"๐ [SAFETY INTERCEPT] Action Rejected: {reason}")
return {"success": False, "reason": reason}
# Dispatch
self.driver.execute_native_input(action)
return {"success": True, "reason": "Executed successfully"}
# โโโ 5. RUNTIME VERIFICATION HARNESS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
if __name__ == "__main__":
print("=" * 75)
print("DEMO: AUTONOMOUS DESKTOP GUI AGENT SAFETY & GROUNDING CONTROLLER (2026)")
print("=" * 75)
# Initialize 1080p display and safety gatekeeper
virtual_screen = ScreenDimensions(width=1920, height=1080)
safety_shield = DesktopSafetyGatekeeper(require_human_for_destructive=True)
agent = DesktopGUIAgent(display=virtual_screen, gatekeeper=safety_shield)
# STEP 1: Safe Action - Navigate SAP Menu
step1 = UIAction(
action_type=ActionType.MOUSE_CLICK,
coordinates=(180, 45), # 1000x1000 normalized grid
thought_reasoning="Milestone 1: Click 'Accounting' dropdown in SAP native menu bar."
)
agent.step(step1)
# STEP 2: Safe Action - Type Transaction Code
step2 = UIAction(
action_type=ActionType.TYPE_TEXT,
text_payload="FS10N",
thought_reasoning="Milestone 2: Enter general ledger balance inquiry transaction code."
)
agent.step(step2)
# STEP 3: Malicious / Accidental Destructive Step Intercepted
step3_destructive = UIAction(
action_type=ActionType.TERMINAL_COMMAND,
text_payload="rm -rf /var/sap/data/*",
thought_reasoning="Adversarial prompt injection attempt detected on unverified clipboard buffer."
)
print("\n[SCENARIO 1: Destructive Action Without Approval]")
agent.step(step3_destructive, human_confirmed=False)
# STEP 4: High-Risk Action With Human Biometric Approval
print("\n[SCENARIO 2: Authorized High-Risk Action via Supervisor Approval]")
agent.step(step3_destructive, human_confirmed=True)
print("\n" + "=" * 75)
print(f"Demonstration Complete. Total Audit Ledger Entries: {len(safety_shield.audit_log)}")
print("=" * 75)
07. OSWorld 2.0ใใณใใใผใฏๅๆ๏ผ้ทๆใฟในใฏใฎใใชใใๅ ๆใจไธป่ฆใชๅคฑๆใใฟใผใณ
OSWorld 2.0 ใฏUbuntuใWindowsใmacOSใซใใใ369ใฎใชใขใซใชใฟในใฏใงใจใผใธใงใณใใ่ฉไพกใใพใใไบบ้ใฎใใผในใฉใคใณใ88.3%ใงใใใฎใซๅฏพใใๆๅ ็ซฏใขใใซใฏๅ จไฝใง49%ใ52%ใซใจใฉใพใฃใฆใใใ4ใคใฎๅ ธๅ็ใชๅคฑๆใใฟใผใณใๆใใใซใชใฃใฆใใพใ๏ผ
- ่งฃๅๅบฆใจๅบงๆจใฎใใชใใ๏ผใใใใขใใ้็ฅใใฆใฃใณใใฆใฎๆกๅคง็ธฎๅฐใซใใๅฏพ่ฑกใๆฐใใฏใปใซ็งปๅใใๅคใ่จๆถใฎๅบงๆจใใฏใชใใฏใใฆ่ชคๅไฝใใ็พ่ฑกใ
- ใณใณใใญในใ้ทใฎๅง่ฟซ๏ผ้ซ่งฃๅๅบฆ็ปๅใ60ๆ่็ฉใใใจๆณจๆๆฉๆงใ็ ด็ถปใใใใใๅคใ็ปๅใใใญในใ่ฆ็ดใธๅง็ธฎใใใใซใผใใณใฐใไธๅฏๆฌ ใงใใ
- ็กๅฟ็ญใญใผใใฃใณใฐๆใฎ้ฃๆใใฉใใ๏ผWebใฎใใใช
networkidleใใชใใใใๅฆ็ไธญใซใใฟใณใ้ฃๆใใฆใขใใชใใฏใฉใใทใฅใใใไบๆ ใ็ป้ขใใฏใปใซใฎๅทฎๅๆค็ฅใง้ฒใใพใใ - ไฝใณใณใใฉในใUIใงใฎ่ชค่ช๏ผใใผใฏใขใผใใฎๆฅญๅใฝใใใCAD็ป้ขใงใไฟๅญใจใจใฏในใใผใใชใฉไผผใใขใคใณใณใ่ฆ่ชคใๅ้กใ
08. ใปใญใฅใชใใฃใจใตใณใใใใฏใน๏ผVNC้้ขใeBPFๅถๅพกใ็ทๆฅๅๆญข๏ผKill-Switch๏ผ
็คพๅกใฎPCไธใง่ชๅพใจใผใธใงใณใใซ็ดๆฅใใฆในใจใญใผใใผใใฎๆจฉ้ใไธใใใใจใฏใๆฅตใใฆ้ๅคงใชใปใญใฅใชใใฃใชในใฏใๆใใพใใๆฌ็ช้็จใงใฏ3ๅฑคใฎใผใญใใฉในใใตใณใใใใฏในใๆง็ฏใใพใ๏ผ
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Zero-Trust Desktop Sandbox Containment Stack |
| |
| [ Enterprise Gateway ] |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 1: VIRTUAL DISPLAY & PERIPHERAL ISOLATION โ |
| โ - Headless X11 / Wayland / RDP virtual server โ |
| โ - The agent NEVER touches physical user hardware โ |
| โ - Video frame streamed via WebRTC / VNC buffer โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 2: SYSTEM CALL INTERCEPTION & eBPF NETWORK EGRESS โ |
| โ - eBPF sensor intercepts dangerous POSIX syscalls (fork, execve, ptrace)โ
| โ - Network firewall restricts outbound traffic exclusively to whitelistโ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 3: SUPERVISOR KILL-SWITCH & USER TAKEOVER โ |
| โ - User moves physical mouse โโโถ Instant agent suspension (Kill-Switch)โ |
| โ - Live action stream mirrored to supervisor dashboard โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+- ็ฌฌ1ๅฑค๏ผไปฎๆณใใฃในใใฌใค้้ข๏ผใจใผใธใงใณใใฏ้้ขใใใMicroVM๏ผE2Bใชใฉ๏ผๅ ใฎไปฎๆณX11/Wayland/RDPใใใใกใฎใฟใๆไฝใ็ฉ็ใใผใใฆใงใขใธใฎ็ดๆฅใขใฏใปในใฏๅฎๅ จใซๆ้คใ
- ็ฌฌ2ๅฑค๏ผeBPFใทในใใ ใณใผใซ๏ผ้ไฟก็ฃๆป๏ผใซใผใใซใฌใใซใฎeBPFใปใณใตใผใซใใใๅฑ้บใชใณใใณใ๏ผ
rm -rfใไธๆญฃใชใใผใฟๅค้จ้ไฟก๏ผใๆค็ฅ้ฎๆญใใ่จฑๅฏใชในใๅคใฎIP้ไฟกใๆๅฆใ - ็ฌฌ3ๅฑค๏ผ็ฉ็ๆไฝใซใใ็ทๆฅใญใซในใคใใ๏ผไบบ้ใ็ฉ็ใใฆในใๅใใใ
Escใญใผใๆผใใ็ฌ้ใซใใจใผใธใงใณใใฎใใฉใคใๆจฉ้ใๅณๆๅฅๅฅชใใฆไบบ้ใฎๆไฝใๆๅชๅ ใ
09. ใขใผใญใใฏใใฃๆฏ่ผใใใชใฏในใจ้ข้ฃ้็บใใผใซ
2026ๅนดใซใใใไธป่ฆใชใในใฏใใใใปใณใณใใฅใผใฟใผๆไฝใจใผใธใงใณใๅบ็คใฎๆฏ่ผใฏไปฅไธใฎ้ใใงใ๏ผ
| ๆฏ่ผ่ปธ | UI-TARS (ByteDance) | Claude 3.7 Computer Use | OpenHands | E2B Desktop Sandbox |
|---|---|---|---|---|
| ใขใใซๅฝขๆ | ใชใผใใณใฆใงใคใ๏ผVLA 7B/72B๏ผ | ๅ็จใใญใณใใฃใขAPI | ใขใใซ้ไพๅญใชใผใฑในใใฌใผใฟใผ | ใคใณใใฉใตใณใใใใฏใน็ฐๅข |
| ไธปใช็ฅ่ฆๆนๅผ | ็ใใฏใปใซ๏ผSystem-2 VLA๏ผ | ็ใใฏใปใซ๏ผVision API๏ผ | ใใคใใชใใ๏ผDOM + ็ซฏๆซ + GUI๏ผ | ไปฎๆณ็ป้ขใใฌใผใ ใใใใก |
| ใในใใฃใณใฐ็ฐๅข | ใชใณใใฌใใน๏ผ่ช็คพGPUๅบ็ค๏ผ | ใฏใฉใฆใAPI๏ผAnthropic๏ผ | ใปใซใใในใ / ใฏใฉใฆใ | ใใใผใธใใฏใฉใฆใ / MicroVM |
| ๅฏพๅฟOS | Windows, macOS, Linux, Android | Windows, macOS, Ubuntu | Linux, Docker, ใใฉใฆใถ | Linux๏ผFirecracker MicroVM๏ผ |
| ไธปใช็จ้ | ้ซ้ ปๅบฆRPAใQAใในใใไธๆฌๅ ฅๅ | ่ค้ใช็ฅ็ๆฅญๅใฎๆจ่ซๅฆ็ | ใฝใใใฆใงใข้็บ่ชๅพใจใผใธใงใณใ | ๅฎๅ จใชใจใผใธใงใณใ้้ขๅฎ่ก็ฐๅข |
| ใชใผใใณใฝใผในๆง | 100% ใชใผใใณใฝใผใน | ใใญใใฉใคใจใฟใช | 100% ใชใผใใณใฝใผใน | ใชใผใใณSDK / ใใใผใธใ |
E2B
MicroVMใตใณใใใใฏในไฟก้ ผใงใใชใใในใฏใใใใปใใทใงใณใใฟใผใใใซๆไฝใใใผใใฆใงใขใฌใใซใงๅฎๅ จใซ้้ขๅฎ่กใใใAIใจใผใธใงใณใๅใไธป่ฆMicroVMใฏใฉใฆใ็ฐๅขใ
E2Bใฎ่ฉณ็ดฐใ่ฆใ โClaude 3.7 Sonnet
ใใญใณใใฃใขใขใใซใในใฏใใใๆไฝใใใฉใฆใถ่ชๅๅใ้ซๅบฆใชใใผใซๅฎ่กใๅฏ่ฝใซใใใใคใใฃใใฎComputer Useๆฉ่ฝใๅใใAnthropicใฎๆ้ซๅณฐๆจ่ซใขใใซใ
Claude 3.7 Sonnetใฎ่ฉณ็ดฐใ่ฆใ โOpenHands
ใชใผใใณใฝใผในใฝใใใฆใงใข้็บใใฟใผใใใซๆไฝใใในใฏใใใ้ฃๆบใฎใใใฎๆ้ซๅณฐใชใผใใณใฝใผใน่ชๅพใจใผใธใงใณใใใฉใใใใฉใผใ ใ
OpenHandsใฎ่ฉณ็ดฐใ่ฆใ โDevin
่ชๅพๅใฝใใใฆใงใขAIใใฉใฆใถใใฟใผใใใซใใจใใฃใฟใ็ทๅๅถๅพกใใ่ชๅพ็ใซใณใผใ้็บใจใในใฏใใใ็ฐๅขไฝๆฅญใๅฎ้ใใCognitionใฎใใฉใใฐใทใใAIใ
Devinใฎ่ฉณ็ดฐใ่ฆใ โ10. ใใใใ่ณชๅ๏ผFAQ๏ผ
Q1: ๅๅฅAPIใ้็บใใไปฃใใใซใในใฏใใใGUIใจใผใธใงใณใใไฝฟใๅฉ็นใฏไฝใงใใ๏ผ
็ๆณ็ใชไธ็ใงใฏๅ จใฝใใใRESTใGraphQLใๆไพใในใใงใใใ็พๅฎใฎไผๆฅญๅ ใซใฏๆฐๅใฎใฌใฌใทใผใทในใใ ๏ผSAPใใชใใณใณ็ซฏๆซใๅฐ็จไผ่จใฝใใ๏ผใๅญๅจใใAPI้็บใซใฏ่จๅคงใช่ฒป็จใจๅนดๆใ่ฆใใพใใใในใฏใใใGUIใจใผใธใงใณใใฏๆขๅญใณใผใใDBใซไธๅๆใๅ ใใชใใผใญใฟใใ่ชๅๅใๅฎ็พใใพใใ
Q2: ่ฆ่ฆใจใผใธใงใณใใซๅ ฅๅใใๆ้ฉใช็ป้ข่งฃๅๅบฆใฏใฉใใใใใงใใ๏ผ
4Kใฎ็็ปๅใใใฎใพใพๅ ฅๅใใใจใใผใฏใณๆถ่ฒปใจๆจ่ซใฌใคใใณใทใๆฟๅขใใพใใๆฌ็ช็ฐๅขใงใฏ1920x1080ใงใญใฃใใใฃใใใขใใซๅ ้จใฎ1000x1000ๆญฃ่ฆๅใฐใชใใใงๅคๆญใใใๅพใ็ฉ็ใใฏใปใซใธใจๆฐๅญฆ็ใซในใฑใผใชใณใฐๅคๆใใฆใฏใชใใฏใ็บ่กใใใฎใไธ่ฌ็ใงใใ
Q3: ใขใใชใฎๅ็ใญใผใใฃใณใฐใใขใใกใผใทใงใณใฏใฉใฎใใใซๅพ ๆฉใใพใใ๏ผ
ใใฉใฆใถใจ็ฐใชใใในใฏใใใใซใฏใญใผใๅฎไบ้็ฅใคใใณใใใใใพใใใๆฌ็ชใจใผใธใงใณใใฏ่ฆ่ฆๅทฎๅใใผใชใณใฐ๏ผVisual Delta Polling๏ผใๆก็จใใ250ใใช็ง้้ใงใใฌใผใ ใๅๅพใใฆใใฏใปใซๅคๅ็ใ1%ๆชๆบใซๅฎๅฎใใใใจใ็ขบ่ชใใฆใใๆฌกใฎๆไฝใๅฎ่กใใพใใ
Q4: ใใซใใขใใฟใผใ่คๆฐใฆใฃใณใใฆ็ฐๅขใงใๅไฝใใพใใ๏ผ
ใฏใใๅไฝใใพใใ่คๆฐ็ป้ขใ1ใคใฎ็ตๅใญใฃใณใใน๏ผ3840x1080ใชใฉ๏ผใจใใฆๆฑใใใใฆใฃใณใใฆใใฉใผใซในAPIใๆดป็จใใฆๅฏพ่ฑกใขใใชใใกใคใณไปฎๆณใใฃในใใฌใคใซใขใฏใใฃใๅใใฆใใ่ฆ่ฆ่ช่ญใ่กใๆงๆใใจใใพใใ
Q5: UI-TARSใฏใญใผใซใซใฎไธ่ฌ็ใชPC็ฐๅขใงๅใใใพใใ๏ผ
UI-TARS 7BใขใใซใฏใGGUFใAWQใชใฉใฎ้ๅญๅใ่กใใใจใงใApple Siliconๆญ่ผMac๏ผใฆใใใกใคใใกใขใช32GBไปฅไธ๏ผใNVIDIA RTX 4090ๅไฝใงใใญใผใซใซๅไฝๅฏ่ฝใงใใ72BใขใใซใฏไผๆฅญๅใใฎA100/H100ใฏใฉในใฎGPU็ฐๅขใๆจๅฅจใใใพใใ
ู ุง ุจุนุฏ ุงูู ุชุตูุญ: ุจูุงุก ูููุงุก ูุงุฌูุงุช ุณุทุญ ุงูู ูุชุจ ุงูู ุณุชููุฉ ูู 2026 ุจุงุณุชุฎุฏุงู UI-TARS ู Claude Computer Use ู OSWorld 2.0
ุฎูุงู ุงูุณููุงุช ุงูุซูุงุซ ุงูู ุงุถูุฉุ ุฑููุฒุช ู ูุธูู ุฉ ุงูุฐูุงุก ุงูุงุตุทูุงุนู ุจุดูู ู ูุซู ุนูู ุฃุชู ุชุฉ ู ุชุตูุญุงุช ุงูููุจ ุจุงุณุชุฎุฏุงู ุฃุฏูุงุช ู ุซู Browser-Use ู Stagehand ู Playwright MCP ูุชุญููู ุดุฌุฑุฉ ูุงุฆูุงุช ุงูู ุชุตูุญ (DOM). ูู ุน ุฐููุ ูุฅู ุฃูุซุฑ ู ู 70% ู ู ุชุฏููุงุช ุงูุนู ู ูู ุจุฑู ุฌูุงุช ุงูู ุคุณุณุงุช ูุง ุชุนู ู ุฅุทูุงูุงู ุฏุงุฎู ู ุชุตูุญ ููุจ. ููุธู ุชุฎุทูุท ุงูู ูุงุฑุฏ ุงูู ุคุณุณูุฉ ุงููุฏูู ุฉ (ู ุซู ูุงุฌูุฉ SAP GUI)ุ ูู ุตููุงุช Excel ุงูุฃุตููุฉ ุงูู ุถู ูุฉ ุจู ุงูุฑู VBAุ ูุจุฑู ุฌูุงุช ุงูุชุตู ูู ุงูููุฏุณู CADุ ูููุงูุฐ ุงูุฃูุงู ุฑ ุงูุทุฑููุฉุ ูุงูุนู ูุงุก ุงูู ูุชุจููู ุงูุฃุตูููู ูุง ูู ุชูููู ุฃู ุจููุฉ DOM. ูุนูุฏู ุง ุชุฑุณู ูุธู ุงูุชุดุบูู ูุงุฌูุงุชูุง ู ุจุงุดุฑุฉ ุนุจุฑ DirectX ุฃู Metal ุฃู Win32 ุฃู Qtุ ูุตุจุญ ุงููููุงุก ุงูู ุนุชู ุฏูู ุนูู DOM ุนุงุฌุฒูู ุชู ุงู ุงู. ููุชุญููู ุฃุชู ุชุฉ ู ุคุณุณูุฉ ุญููููุฉุ ูุดูุฏ ุนุงู 2026 ุจุฒูุบ ูููุงุก ูุงุฌูุงุช ุณุทุญ ุงูู ูุชุจ ุงูู ุณุชูููู (Autonomous Desktop GUI Agents) ุงูู ุฏุนูู ูู ุจูู ุงุฐุฌ ุงูุนู ู ุงููุบูู ุงูุจุตุฑู ุงูุชุฃุณูุณูุฉ (VLA)ุ ูุชุญุฏูุฏ ุฅุญุฏุงุซูุงุช ุงูุจูุณูุ ูุงูุชูููุฑ ุงูุชุฃู ูู ุงูู ุชุฃูู (System-2). ูุญูู ูุฐุง ุงูุฏููู ุงูู ุนู ุงุฑู ุงูุนู ูู ูู ุงุฐุฌ UI-TARS 1.5ุ ู Claude 3.7 Computer Useุ ูู ุนูุงุฑ OSWorld 2.0ุ ูุจูุฆุงุช ุงูุนุฒู ุงูุขู ูุฉ ูู ุจูุฆุงุช ุงูุฅูุชุงุฌ.
๐ ุฌุฏูู ุงูู ุญุชููุงุช
01. ุงูู ูุฎุต ุงูุณุฑูุน ูุขูุงู ูุงุฌูุงุช ุณุทุญ ุงูู ูุชุจ
ุจูุบุช ุฃุชู ุชุฉ ุงูููุจ ู ุฑุญูุฉ ุงููุถุฌุ ููู ุงูุนู ู ุงูู ุนุฑูู ุนูู ุฃุฌูุฒุฉ ุณุทุญ ุงูู ูุชุจ ุธู ู ุนุชู ุฏุงู ุนูู ุงูุฌูุฏ ุงููุฏูู. ูู ุนุงู 2026ุ ูุชุนุงู ู ูููุงุก ูุงุฌูุงุช ุณุทุญ ุงูู ูุชุจ ู ุน ุดุงุดุฉ ุงูุญุงุณูุจ ุจุฃูู ููุง ูููุญุฉ ุชูุงุนููุฉ ู ุจุงุดุฑุฉ:
- ู ุง ูุฑุงุก ู ุญุฏุฏุงุช ุนูุงุตุฑ HTML: ูุฑุงูุจ ูููุงุก ุณุทุญ ุงูู ูุชุจ ุฅุทุงุฑุงุช ุงูุดุงุดุฉ ุงูู ุจุงุดุฑุฉ (ุจุฏูุฉ ุชุชุฑุงูุญ ู ู 1080p ุฅูู 4K)ุ ููุชุนุฑููู ุนูู ุงูุนูุงุตุฑ ุงูุชูุงุนููุฉ ู ุจุงุดุฑุฉ ุนุจุฑ ุงููู ุงุฐุฌ ุงููุบููุฉ ุงูุจุตุฑูุฉ ุงููุจูุฑุฉ (MLLMs)ุ ู ุน ุชุญููู ุงูุงุณุชุฏูุงู ุฅูู ุฅุญุฏุงุซูุงุช ููุงุณูุฉ ู ูุนุงูุฑุฉ (x, y) ุชูุชุฑุฌู ูุฃุญุฏุงุซ ูุฃุฑุฉ ูููุญุฉ ู ูุงุชูุญ ุญููููุฉ.
- ุฑูุงุฏุฉ ุงูู ุตุฏุฑ ุงูู ูุชูุญ ู ูุงุจู ูุงุฌูุงุช ุงูุจุฑู ุฌุฉ ุงูู ุบููุฉ: ูุฏู ูู ูุฐุฌ UI-TARS 1.5 ู ูุชูุญ ุงูุฃูุฒุงู ู ู ByteDance ุจููุฉ ุชูููุฑ System-2 ุฃุตููุฉ ู ุฏุนูู ุฉ ุจุงูุชุนูู ุงูุชุนุฒูุฒู ููุชุฃู ู ุงูุฐุงุชู ูุงูุชุฑุงุฌุน ุนู ุงูุฃุฎุทุงุกุ ูู ุญูู ููุฏู Claude 3.7 Sonnet ู ู Anthropic ูุฏุฑุงุช ุชูููุฑ ุนููุง ูุงุณุชุฏุนุงุกู ุฃุตููุงู ูุฃุฏูุงุช ุงูุชุญูู ุจุงูุญุงุณูุจ (Computer Use).
- ุงุฎุชุจุงุฑ ุงููุงูุน ุนุจุฑ ู ุนูุงุฑ OSWorld 2.0: ุนูู ู ุนูุงุฑ OSWorld 2.0 ุงูุฐู ูุถู 369 ู ูู ุฉ ูุงูุนูุฉ ู ุนูุฏุฉ ุนุจุฑ ุฃูุธู ุฉ Ubuntu ู Windows ู macOSุ ุชุญูู ุงููู ุงุฐุฌ ู ุนุฏู ูุฌุงุญ ุจูู 42% ุฅูู 52% ูู ุงูู ูุงู ุงูู ู ุชุฏุฉ ูุฃูุซุฑ ู ู 100 ุฎุทูุฉุ ู ู ุง ูุจุฑุฒ ุฃู "ุงูุญุฑุงู ุงูุญุงูุฉ ุงูุชุฑุงูู ู" ูู ุซู ุงูุนุงุฆู ุงูููุฏุณู ุงูุฃูุจุฑ.
- ุงูุนุฒู ุงูุฅูุฒุงู ู ูู ุจูุฆุงุช ุงูุชุดุบูู: ุชุชุทูุจ ุงูุณูุทุฑุฉ ุงูู ุจุงุดุฑุฉ ุนูู ูุธุงู ุงูุชุดุบูู ููุฏุณุฉ ุงูุนุฏุงู ุงูุซูุฉ (Zero-Trust). ูุชุนู ู ุจูู ุงูุฅูุชุงุฌ ุนูู ุชุดุบูู ุงููููุงุก ุฏุงุฎู ุฃุฌูุฒุฉ ุงูุชุฑุงุถูุฉ ุฎูููุฉ ู ุนุฒููุฉ (MicroVMs ู ุซู E2B) ุฃู ุนุจุฑ ุดุงุดุงุช ุงูุชุฑุงุถูุฉ (VNC/RDP) ู ุฏุนูู ุฉ ุจุฌุฏุฑุงู ุญู ุงูุฉ ูู ูุน ุงูุฃูุงู ุฑ ุงูุชุฎุฑูุจูุฉ ูุฒุฑ ุฅููุงู ุทุงุฑุฆ ููุจุดุฑ.
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Autonomous Desktop GUI Agent Architecture (2026) |
| |
| [ User Goal: "Consolidate Q3 SAP exports into Excel macro, generate PDF" ] |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ HOST SUPERVISOR & ACTION FIREWALL (Safety Interceptor) โ |
| โ - Rate Limiting & Egress Filtering - Destructive Command Blocker โ |
| โ - Biometric Confirmation Gateway - Emergency Human Kill-Switch โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ Virtual Display / Peripherals |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ ISOLATED DESKTOP SANDBOX (E2B / Cloud VNC / KVM Virtual Machine) โ |
| โ โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ |
| โ โ Screenshot Buffer โ โ OS Input Controller โ โ |
| โ โ (1920x1080 RGB) โ โ (PyAutoGUI / uinput) โ โ |
| โ โโโโโโโโโโโโฌโโโโโโโโโโโ โโโโโโโโโโโโโโโฒโโโโโโโโโโโโ โ |
| โ โ Raw Frame โ (x, y) Click โ |
| โ โผ โ & Hotkeys โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ โ |
| โ โ AGENT RUNTIME: UI-TARS 1.5 / Claude 3.7 Sonnet โ โ |
| โ โ 1. Visual Grounding: Detect target UI elements via pixels โ โ |
| โ โ 2. System-2 Deliberation: Check milestones & reflect โ โ |
| โ โ 3. Action Plan: Emit precise mouse, click, and key sequences โ โ |
| โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ |
| โ โ |
| โ [ Legacy SAP Client ] [ Native Excel ] [ Desktop CAD ] โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+02. ููุงูุฉ ุงูุชุจุนูุฉ ูู DOM: ุฃูู ูุฉ ุชุฏููุงุช ุงูุนู ู ุงูู ูุชุจูุฉ ูู ุงูู ุคุณุณุงุช
ุชูุชุฑุถ ุฃุทุฑ ุนู ู ุฃุชู ุชุฉ ุงูู ุชุตูุญุงุช ุฃู ุงูุชุทุจููุงุช ุชุชูุญ ุฏุงุฆู ุงู ุฃุดุฌุงุฑ ูุตูู ู ูุธู ุฉ (Accessibility Trees)ุ ูู ุญุฏุฏุงุช CSS ุฏูููุฉุ ููููููุฉ ู ุชููุนุฉ ููุนูุงุตุฑ. ููู ูุทุงุน ุงูุนู ููุงุช ุงูู ุคุณุณูุฉ ุงูุญุณุงุณุฉุ ูููุงุฑ ูุฐุง ุงูุงูุชุฑุงุถ ูููุงู ูุนุฏุฉ ุฃุณุจุงุจ:
1. ุชุทุจููุงุช ุงูููุจ ุงูู ุฑุณูู ุฉ ุนูู Canvas
ุชุฑุณู ุฃุฏูุงุช ู ุซู Figma ูู ุญุฑุฑ ู ุณุชูุฏุงุช Google ุงูู ุณุชูุฏ ุฅูู Canvas ูููุญุงุช ุชุญูู WebGL ุนูุงุตุฑูุง ุฏุงุฎู ุณูุงู ุฑุณูู ู ู ุณุทุญ ุฏูู ุฃู ุนูุงุตุฑ DOM ูุนููุฉ.
2. ุงูุจุฑู ุฌูุงุช ุงูู ุคุณุณูุฉ ุงููุฏูู ุฉ
ุงูุฃูุธู ุฉ ุงูุญูููุฉ ู ุซู SAP GUI ูู ุญุงููุงุช ุดุงุดุงุช AS400 ูู ุญุทุงุช ุจููู ุจุฑุบ ูุฃูุธู ุฉ ุงูุณุฌูุงุช ุงูุทุจูุฉ ุชุนู ู ูู ููุงุช ุชูููุฐูุฉ ุซูุงุฆูุฉ ุฏูู ุฃู ูุงุฌูุงุช ููุจ.
3. ุณูุงุณู ุงูุนู ู ุงูุนุงุจุฑุฉ ููุชุทุจููุงุช
ุชุชุทูุจ ุงูู ูุงู ุชูุฒูู ุจูุงูุงุชุ ูุชุดุบูู ู ุงูุฑู Excel ู ุญููุ ูุชุญุฏูุซ ุจุฑุงู ุฌ ุฅุฏุงุฑุฉ ุงูุนู ูุงุก ุงูู ูุชุจูุฉุ ูุชูููุน ู ููุงุช PDF ุนุจุฑ ุญุฏูุฏ ุงูุชุทุจููุงุช ููุธุงู ุงูุชุดุบูู.
03. ุงูุจูู ุงูุซูุงุซ ููุฅุฏุฑุงู ุงูุจุตุฑู ุนูู ุณุทุญ ุงูู ูุชุจ
ููู ูู ูู ูููููู ุงูู ุณุชูู ุฃู ูุฏุฑู ุจุฏูุฉ ู ุง ูู ู ุนุฑูุถ ุนูู ุดุงุดุฉ ุณุทุญ ุงูู ูุชุจุ ุชุชูุงูุณ ุซูุงุซ ู ุฏุงุฑุณ ู ุนู ุงุฑูุฉ ุฑุฆูุณูุฉ ูู ุนุงู 2026:
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Desktop Perception Paradigms |
| |
| [ Approach 1: Pure Visual Grounding (Pixel-to-Coordinate) ] |
| Screen Frame โโโถ High-Res VLM โโโถ Coordinates (x: 450, y: 720) |
| โข Pros: Universal, zero OS dependency, handles Canvas / Games / Legacy |
| โข Cons: Heavy token cost, coordinate distortion, resolution scaling drift|
| |
| [ Approach 2: OS Accessibility Tree Grounding (UIAutomation / AT-SPI) ] |
| Screen State โโโถ OS API Walk โโโถ Filtered Hierarchy โโโถ Element ID / Path|
| โข Pros: Deterministic, lightweight text tokens, 100% click precision |
| โข Cons: 40% of native apps have broken/missing a11y trees, slow tree walk|
| |
| [ Approach 3: Dual-Stream Hybrid Fusion (2026 Best Practice) ] |
| Visual Screenshot (VLM) โโโFused Decisionโโโบ OS A11y Tree (Cache) |
| โข Fast path: Use A11y node bounding box if recognized |
| โข Fallback: Use visual grounding when a11y nodes are obscured/custom |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+| ุงูู ุนูุงุฑ ุงูู ุนู ุงุฑู | ุงูุฅุฏุฑุงู ุงูุจุตุฑู ุงูุฎุงูุต (Pixel) | ุดุฌุฑุฉ ุฅู ูุงููุฉ ุงููุตูู (A11y Tree) | ุงูุฏู ุฌ ุงููุฌูู ุซูุงุฆู ุงูู ุณุงุฑ (Hybrid) |
|---|---|---|---|
| ุงูุดู ูููุฉ ูุชูุงูู ุงููุงุฌูุงุช | 100% (ุฃู ุจูุณู ููุนุฑุถ ุนูู ุงูุดุงุดุฉ) | ~60% (ููุดู ู ุน ุงููุงุฌูุงุช ุงูู ุฎุตุตุฉ ู Canvas) | 98% (ุชุญููู ุณูุณ ูุชููุงุฆู ุนูุฏ ุงูุชุนุฐุฑ) |
| ููุงุกุฉ ุงุณุชููุงู ุงูุฑู ูุฒ | ู ูุฎูุถุฉ (~1,200 ุฅูู 2,000 ุฑู ุฒ ููู ุฅุทุงุฑ) | ู ุฑุชูุนุฉ (~300 ุฅูู 600 ุฑู ุฒ ููู ุดุฌุฑุฉ) | ู ุชูุณุทุฉ (~1,500 ุฑู ุฒ ููู ูุฑุงุฑ) |
| ุฏูุฉ ุงูููุฑ ูุงูุงุณุชูุฏุงู | 88% - 94% (ู ุนุฑุถุฉ ููุงูุญุฑุงู) | 99% (ุนูุฏ ุชููุฑ ุนูุฏ ุงูุดุฌุฑุฉ) | 98.5% (ุชุซุจูุช ุฏููู ุฏุงุฎู ุงูุฅุทุงุฑ) |
| ุงุณุชููุงููุฉ ูุธุงู ุงูุชุดุบูู | ูุงู ูุฉ (ู ุชูุงูู ู ุน VNC ูุงูุจุซ ุงูุงูุชุฑุงุถู) | ู ูุฎูุถุฉ (ุชุชุทูุจ ุฎุทุงูุงุช ุจุฑู ุฌูุฉ ููู ูุธุงู ) | ุนุงููุฉ (ู ุญููุงุช ูุธุงู ุชุดุบูู ูู ุทูุฉ) |
| ุฒู ู ุงุณุชุฌุงุจุฉ ุงูุฎุทูุฉ | 1.8 ุซุงููุฉ - 3.5 ุซุงููุฉ (ุงุณุชุฏูุงู VLM) | 0.4 ุซุงููุฉ - 0.9 ุซุงููุฉ (ูู ูุฐุฌ ูุตู) | 1.9 ุซุงููุฉ - 3.2 ุซุงููุฉ |
04. ูู ูุฐุฌ UI-TARS: ูุฏุฑุงุช VLA ุงูุฃุตููุฉ ูุชูููุฑ System-2
ููุนุฏ ูู ูุฐุฌ UI-TARS 1.5 ุงูู ุทูุฑ ู ู ููุจู ByteDance ุฅูุฌุงุฒุงู ููุนูุงู ูู ู ุฌุงู ูู ุงุฐุฌ ุงูุนู ู ุงููุบูู ุงูุจุตุฑู ุงูุชุฃุณูุณูุฉ (VLA). ูุจููู ุง ูุงูุช ุงููู ุงุฐุฌ ุงูุณุงุจูุฉ ุชูุชูุท ุงูุดุงุดุฉ ูุชุทูู ููุฑุฉ ููุฑูุฉ ุจูู ุท ุญุฏุณู ุณุฑูุน (System-1)ุ ูุฏู ุฌ UI-TARS ุจููุฉ ุชูููุฑ ู ุชุฃููุฉ ุฎุงุถุนุฉ ูุชุฏุฑูุจ ุจุงูุชุนูู ุงูุชุนุฒูุฒู (Reinforcement Learning):
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| UI-TARS 1.5 System-2 Reasoning Trajectory |
| |
| Observation โโโถ [ Reflection & Verification ] |
| โ "Did the previous action open the File dialog?" |
| โ State: YES. Detected 'Open File' window at (x: 200, y: 150)
| โผ |
| [ Sub-goal Decomposition ] |
| โ "Next sub-goal: Select 'Quarterly_Report.xlsx'" |
| โ Search Strategy: Visual scan of table rows |
| โผ |
| [ Milestone Recognition ] |
| โ Target element found at (x: 320, y: 410) |
| โผ |
| [ Action Emission ] |
| โ Action: click(point=[320, 410]) |
| โ Post-Action Expectation: File selected in input box |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+- ุฑู
ูุฒ ุญุฑููุฉ ููุงุณูุฉ ู
ูุญุฏุฉ: ุฅุตุฏุงุฑ ู
ุจุงุดุฑ ูุฃูุงู
ุฑ ู
ุซู
click(point=[x, y])ูdouble_click()ูdrag(start, end)ูpress_hotkey()ุนุจุฑ ู ุณุชูู ุฅุญุฏุงุซูุงุช ู ูุนุงูุฑ 1000 × 1000. - ุชูููุฑ System-2 ุงูู ุชุฃูู: ุชูููู ุญุงูุฉ ุงูุดุงุดุฉ ุงูุณุงุจูุฉ ูุจู ุงุชุฎุงุฐ ุฃู ุฎุทูุฉ ุฌุฏูุฏุฉ. ูุฅุฐุง ูุดูุช ููุฑุฉ ุฒุฑ ูู ูุชุญ ู ุฑุจุน ุงูุญูุงุฑุ ูุชุฃู ู ุงููู ูุฐุฌ ุฎุทุฃู ููุตุญุญ ู ุณุงุฑู ุฐุงุชูุงู ุจุฏูุงู ู ู ู ุฑุงูู ุฉ ุงูุฃุฎุทุงุก.
- ุฅู ูุงููุฉ ุงููุดุฑ ูุงูุชุดุบูู ุงูู ุญูู: ูุนู ู ูู ูุฐุฌ UI-TARS 7B ู ุญููุงู ุนูู ู ุญุทุงุช ุงูุนู ู (ู ุซู ุฃุฌูุฒุฉ Mac ุจุฐุงูุฑุฉ 32GB ุฃู ุจุทุงูุฉ RTX 4090)ุ ู ููุฑุงู ุงุณุชุฌุงุจุฉ ุฏูู ุงูุซุงููุฉ ู ุน ุฎุตูุตูุฉ ุจูุงูุงุช ู ุทููุฉ.
05. ู ูุงุฒูุฉ Claude 3.7 Computer Use ู ูุงุจู ุงููู ุงุฐุฌ ู ูุชูุญุฉ ุงูุฃูุฒุงู
ุนูุฏ ุชุตู ูู ู ูุธูู ุฉ ุฃุชู ุชุฉ ู ูุชุจูุฉ ู ุคุณุณูุฉุ ููุงุฌู ู ููุฏุณู ุงูุฐูุงุก ุงูุงุตุทูุงุนู ุฎูุงุฑุงู ุงุณุชุฑุงุชูุฌูุงู ุจูู ูุงุฌูุฉ Claude 3.7 Computer Use ุงูุณุญุงุจูุฉ ู ู Anthropic ูุงูุญููู ุฐุงุช ุงูุฃูุฒุงู ุงูู ูุชูุญุฉ ู ุซู UI-TARS 7B/72B ู OS-Atlas:
- Claude 3.7 Computer Use: ูุชููู ุจูุฏุฑุงุช ุชูููุฑ ู ูุทูู ุงุณุชุซูุงุฆูุฉุ ูููู ุนู ูู ููุชุนููู ุงุช ุงูู ุนูุฏุฉ ุงูู ูููุฉ ู ู ุตูุญุงุช ู ุชุนุฏุฏุฉุ ูุฅุฏุงุฑุฉ ู ุชูุฏู ุฉ ููุณูุงู ุงููุบูู. ูู ุน ุฐููุ ููู ูุฑุณู ููุทุงุช ุดุงุดุฉ ูุงู ูุฉ ุนุจุฑ ุงูุณุญุงุจุฉ (ู ู ุง ูุซูุฑ ู ุฎุงูู ุงูุฎุตูุตูุฉ ูุงูุงู ุชุซุงู ูู ุงููุทุงุนุงุช ุงูู ุตุฑููุฉ ูุงูุตุญูุฉ)ุ ู ุน ุชูููุฉ ุฑู ุฒูุฉ ู ูุญูุธุฉ ูุฒู ู ูุตูู ุดุจูู ูุจูุบ 2 ุฅูู 4 ุซูุงูู ููู ุฅุฌุฑุงุก.
- UI-TARS ูุงููู ุงุฐุฌ ุงูู ูุชูุญุฉ: ูู ูู ูุดุฑูุง ุจุงููุงู ู ู ุญููุงู ุฃู ูู ุณุญุงุจุฉ ุฎุงุตุฉ ู ุนุฒููุฉ (VPC). ูุชููุฑ ุชุญูู ุงู ู ุทููุงู ูู ุญู ุงูุฉ ุงูุจูุงูุงุชุ ูุฏุนู ุงู ูุชุนุฏูู ุงูุฃูุฒุงู (Fine-tuning) ุนูู ุจุฑู ุฌูุงุช ุงูุดุฑูุฉ ุงูุฏุงุฎููุฉุ ูุฒู ู ุงุณุชุฌุงุจุฉ ูุงุฆู ุงูุณุฑุนุฉ ุนุจุฑ ุจุทุงูุงุช ุงูุญูุณุจุฉ ุงูู ุญููุฉ (0.8 ุฅูู 1.5 ุซุงููุฉ ููู ุฎุทูุฉ).
06. ุงูุชุทุจูู ุงูุนู ูู ุจูุบุฉ Python: ู ุชุญูู ุณุทุญ ู ูุชุจ ุขู ู
ููุถุญ ุงูููุฏ ุงูุชุงูู ุจููุฉ ููุฏุณูุฉ ู ุชูุงู ูุฉ ููุงุจูุฉ ููุชุดุบูู ุงูููุฑู ูู ุชุญูู ูููู ุณุทุญ ู ูุชุจุ ูุฏู ุฌ ูู ุฐุฌุฉ ุฅุญุฏุงุซูุงุช VLAุ ูู ุญุงุฐุงุฉ ุงูุดุงุดุฉุ ูุฌุฏุงุฑ ุญู ุงูุฉ ุงุนุชุฑุงุถู ูู ูุน ุงูุฅุฌุฑุงุกุงุช ุงูุชุฎุฑูุจูุฉ ุฃู ุงูุญุณุงุณุฉ ุฅูุง ุจู ูุงููุฉ ุจุดุฑูุฉ ุตุฑูุญุฉ:
# Production Reference Implementation: Desktop GUI Agent Controller (2026)
# Demonstrates Vision-Language-Action Execution, Coordinate Normalization,
# Destructive Command Interception, and Human-in-the-Loop Safeguards.
import time
import math
from typing import Dict, Any, List, Tuple, Optional
from dataclasses import dataclass, field
from enum import Enum
# โโโ 1. CORE DATA STRUCTURES & ACTIONS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class ActionType(str, Enum):
MOUSE_CLICK = "mouse_click"
DOUBLE_CLICK = "double_click"
HOTKEY = "hotkey"
TYPE_TEXT = "type_text"
TERMINAL_COMMAND = "terminal_command"
WAIT = "wait"
@dataclass
class UIAction:
action_type: ActionType
coordinates: Optional[Tuple[int, int]] = None # (x, y) on 1000x1000 normalized grid
text_payload: Optional[str] = None
hotkey_sequence: Optional[List[str]] = None
thought_reasoning: str = ""
is_destructive: bool = False
@dataclass
class ScreenDimensions:
width: int
height: int
# โโโ 2. SAFETY INTERCEPTOR & GATEKEEPER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class DesktopSafetyGatekeeper:
"""
Host-side safety authority that validates actions before dispatching
to real or virtual OS input peripherals.
"""
DESTRUCTIVE_HOTKEYS = {("ctrl", "alt", "del"), ("cmd", "shift", "backspace")}
DESTRUCTIVE_COMMANDS = ["rm -rf", "format", "drop database", "mkfs", "dd if="]
def __init__(self, require_human_for_destructive: bool = True):
self.require_human = require_human_for_destructive
self.audit_log: List[Dict[str, Any]] = []
def inspect_action(self, action: UIAction, human_approved: bool = False) -> Tuple[bool, str]:
# 1. Inspect terminal/shell commands
if action.action_type == ActionType.TERMINAL_COMMAND and action.text_payload:
for pattern in self.DESTRUCTIVE_COMMANDS:
if pattern in action.text_payload.lower():
action.is_destructive = True
if not human_approved:
return False, f"Blocked destructive terminal command pattern: '{pattern}'"
# 2. Inspect high-risk hotkeys
if action.action_type == ActionType.HOTKEY and action.hotkey_sequence:
normalized_keys = tuple(sorted([k.lower() for k in action.hotkey_sequence]))
for risk_keys in self.DESTRUCTIVE_HOTKEYS:
if normalized_keys == tuple(sorted(risk_keys)):
action.is_destructive = True
if not human_approved:
return False, f"Blocked dangerous system hotkey: '{action.hotkey_sequence}'"
# Log action to immutable audit trail
self.audit_log.append({
"timestamp": time.time(),
"action": action.action_type.value,
"coordinates": action.coordinates,
"destructive": action.is_destructive,
"approved": True
})
return True, "Action approved."
# โโโ 3. PERIPHERAL ADAPTER & COORDINATE SCALER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class OSPeripheralController:
"""
Translates normalized model coordinates (1000x1000) to actual physical display pixels
and dispatches low-level OS input events.
"""
def __init__(self, display: ScreenDimensions):
self.display = display
def denormalize_coordinates(self, norm_x: int, norm_y: int) -> Tuple[int, int]:
"""Converts [0, 1000] model grid to [0, width] x [0, height]."""
real_x = math.floor((norm_x / 1000.0) * self.display.width)
real_y = math.floor((norm_y / 1000.0) * self.display.height)
return real_x, real_y
def execute_native_input(self, action: UIAction):
if action.coordinates:
rx, ry = self.denormalize_coordinates(*action.coordinates)
print(f"๐ฑ๏ธ [OS DRIVER] Moving cursor to ({rx}px, {ry}px) [Model: {action.coordinates}]")
if action.action_type == ActionType.MOUSE_CLICK:
print(f"โก [OS DRIVER] Emitting LEFT_BUTTON_CLICK at ({rx}, {ry})")
elif action.action_type == ActionType.DOUBLE_CLICK:
print(f"โก [OS DRIVER] Emitting DOUBLE_CLICK at ({rx}, {ry})")
elif action.action_type == ActionType.HOTKEY:
print(f"โจ๏ธ [OS DRIVER] Emitting KEY_COMBINATION: {' + '.join(action.hotkey_sequence or [])}")
elif action.action_type == ActionType.TYPE_TEXT:
print(f"โจ๏ธ [OS DRIVER] Typing string payload: \"{action.text_payload}\"")
elif action.action_type == ActionType.TERMINAL_COMMAND:
print(f"๐ฅ๏ธ [OS DRIVER] Executing Shell Command: '{action.text_payload}'")
# โโโ 4. AUTONOMOUS DESKTOP AGENT CONTROLLER โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
class DesktopGUIAgent:
"""
The orchestrator agent combining VLA decision logic, safety inspection,
and OS input driver dispatch.
"""
def __init__(self, display: ScreenDimensions, gatekeeper: DesktopSafetyGatekeeper):
self.driver = OSPeripheralController(display)
self.gatekeeper = gatekeeper
def step(self, action: UIAction, human_confirmed: bool = False) -> Dict[str, Any]:
print(f"\n๐ง [SYSTEM-2 REFLECTION] {action.thought_reasoning}")
# Safety Check
is_safe, reason = self.gatekeeper.inspect_action(action, human_approved=human_confirmed)
if not is_safe:
print(f"๐ [SAFETY INTERCEPT] Action Rejected: {reason}")
return {"success": False, "reason": reason}
# Dispatch
self.driver.execute_native_input(action)
return {"success": True, "reason": "Executed successfully"}
# โโโ 5. RUNTIME VERIFICATION HARNESS โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
if __name__ == "__main__":
print("=" * 75)
print("DEMO: AUTONOMOUS DESKTOP GUI AGENT SAFETY & GROUNDING CONTROLLER (2026)")
print("=" * 75)
# Initialize 1080p display and safety gatekeeper
virtual_screen = ScreenDimensions(width=1920, height=1080)
safety_shield = DesktopSafetyGatekeeper(require_human_for_destructive=True)
agent = DesktopGUIAgent(display=virtual_screen, gatekeeper=safety_shield)
# STEP 1: Safe Action - Navigate SAP Menu
step1 = UIAction(
action_type=ActionType.MOUSE_CLICK,
coordinates=(180, 45), # 1000x1000 normalized grid
thought_reasoning="Milestone 1: Click 'Accounting' dropdown in SAP native menu bar."
)
agent.step(step1)
# STEP 2: Safe Action - Type Transaction Code
step2 = UIAction(
action_type=ActionType.TYPE_TEXT,
text_payload="FS10N",
thought_reasoning="Milestone 2: Enter general ledger balance inquiry transaction code."
)
agent.step(step2)
# STEP 3: Malicious / Accidental Destructive Step Intercepted
step3_destructive = UIAction(
action_type=ActionType.TERMINAL_COMMAND,
text_payload="rm -rf /var/sap/data/*",
thought_reasoning="Adversarial prompt injection attempt detected on unverified clipboard buffer."
)
print("\n[SCENARIO 1: Destructive Action Without Approval]")
agent.step(step3_destructive, human_confirmed=False)
# STEP 4: High-Risk Action With Human Biometric Approval
print("\n[SCENARIO 2: Authorized High-Risk Action via Supervisor Approval]")
agent.step(step3_destructive, human_confirmed=True)
print("\n" + "=" * 75)
print(f"Demonstration Complete. Total Audit Ledger Entries: {len(safety_shield.audit_log)}")
print("=" * 75)
07. ุชูููู OSWorld 2.0: ุญู ู ุดููุฉ ุงูุญุฑุงู ุงูู ูุงู ุงูุทูููุฉ
ููุนุฏ ู ุนูุงุฑ OSWorld 2.0 ุงูู ุฑุฌุน ุงูููุงุณู ุงูุนุงูู ู ุงูุฃูุซุฑ ู ูุซูููุฉ ูุงุฎุชุจุงุฑ ูููุงุก ุณุทุญ ุงูู ูุชุจ ุงูู ุณุชูููู ุนุจุฑ 369 ู ูู ุฉ ูุงูุนูุฉ ูู Ubuntu ู Windows ู macOS. ูุจููู ุง ูุญูู ุงูุฃุฏุงุก ุงูุจุดุฑู ูุณุจุฉ 88.3%ุ ุชุญูู ุฃุญุฏุซ ุงููู ุงุฐุฌ ู ุง ุจูู 49% ุฅูู 52% ุฅุฌู ุงูุงูุ ูุงุดูุฉ ุนู ุฃุฑุจุนุฉ ุฃูู ุงุท ูุดู ุจููููุฉ:
- ุงูุญุฑุงู ุงูุฏูุฉ ูุงูุฅุญุฏุงุซูุงุช (Resolution & Coordinate Drift): ุชุคุฏู ุงูููุงูุฐ ุงูู ูุจุซูุฉ ุฃู ุชุบููุฑ ุฃุญุฌุงู ุงูุดุงุดุงุช ุฅูู ุฅุฒุงุญุฉ ุงูุฃูุฏุงู ุจุจูุณูุงุช ููููุฉุ ู ู ุง ูุณุจุจ ููุฑุงุช ุฎุงุทุฆุฉ ุนูู ุฅุญุฏุงุซูุงุช ูุฏูู ุฉ.
- ุฅุฌูุงุฏ ูุงูุฐุฉ ุงูุณูุงู (Context Window Fatigue): ูุคุฏู ุงูุงุญุชูุงุธ ุจุฃูุซุฑ ู ู 60 ููุทุฉ ุดุงุดุฉ ูุงู ูุฉ ุฅูู ุชุฌุงูุฒ ุณูุงู ุงููู ูุฐุฌ. ูุชููู ุฃูุธู ุฉ ุงูุฅูุชุงุฌ ุจุงุฎุชุฒุงู ุงูููุทุงุช ุงููุฏูู ุฉ ูู ุณุฌู ูุตู ู ูุฌุฒ ููุฃูุนุงู.
- ู
ุตุงุฆุฏ ุงูุชุญู
ูู ุงูุตุงู
ุช ูู
ุคุดุฑุงุช ุงูุงูุชุธุงุฑ (Silent Loading Traps): ูุธุฑุงู ูุบูุงุจ ุฃุญุฏุงุซ ุงูู
ุชุตูุญ ู
ุซู
networkidleุ ูููุฑ ุงููููู ุจุดูู ู ุชูุฑุฑ ุฃุซูุงุก ุชุญู ูู ุงูุชุทุจูู ู ุณุจุจุงู ุชุนุทู ู ุณุงุฑุงุช ุงูุนู ู. ููุญู ุงุณุชุทูุงุน ุงููุฑูู ุงูุจุตุฑูุฉ ูุฐู ุงูู ุนุถูุฉ. - ุงูุงุฑุชุจุงู ูู ุงููุงุฌูุงุช ู ูุฎูุถุฉ ุงูุชุจุงูู (Low-Contrast UI Confusion): ูู ุงููุงุฌูุงุช ุงูุฏุงููุฉ ููุจุฑู ุฌูุงุช ุงูู ุคุณุณูุฉ ุฃู ู ุฎุทุทุงุช CAD ุงูุณูููุฉุ ุชุฎูุท ุงููู ุงุฐุฌ ุจูู ุฃููููุงุช ุงูุฃุฏูุงุช ุงูู ุชุดุงุจูุฉ (ู ุซู ุงูุญูุธ ู ูุงุจู ุงูุชุตุฏูุฑ).
08. ุงูุฃู ุงู ูุจูุฆุงุช ุงูุนุฒู: ุนุฒู VNC ูุฒุฑ ุงูุฅููุงู ุงูุทุงุฑุฆ
ุฅู ู ูุญ ูู ูุฐุฌ ุฐูุงุก ุงุตุทูุงุนู ุณูุทุฑุฉ ุญุฑููุฉ ูุงู ูุฉ ุนูู ููุญุฉ ุงูู ูุงุชูุญ ูุงููุฃุฑุฉ ูู ุซู ุฎุทุฑุงู ุฃู ููุงู ุฌุณูู ุงู ุฅู ูู ููุญุงุท ุจุจูุฆุฉ ุนุฒู ู ุญูู ุฉ. ูุชุนุชู ุฏ ุงูู ูุดุขุช ุงูู ุคุณุณูุฉ ุญุฒู ุฉ ุนุฒู ุตุงุฑู ุฉ ูุงุฆู ุฉ ุนูู ูู ูุฐุฌ ุงูุนุฏุงู ุงูุซูุฉ:
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+
| Zero-Trust Desktop Sandbox Containment Stack |
| |
| [ Enterprise Gateway ] |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 1: VIRTUAL DISPLAY & PERIPHERAL ISOLATION โ |
| โ - Headless X11 / Wayland / RDP virtual server โ |
| โ - The agent NEVER touches physical user hardware โ |
| โ - Video frame streamed via WebRTC / VNC buffer โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 2: SYSTEM CALL INTERCEPTION & eBPF NETWORK EGRESS โ |
| โ - eBPF sensor intercepts dangerous POSIX syscalls (fork, execve, ptrace)โ
| โ - Network firewall restricts outbound traffic exclusively to whitelistโ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ |
| โผ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| โ LAYER 3: SUPERVISOR KILL-SWITCH & USER TAKEOVER โ |
| โ - User moves physical mouse โโโถ Instant agent suspension (Kill-Switch)โ |
| โ - Live action stream mirrored to supervisor dashboard โ |
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
+โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+- ุนุฒู ุงูุนุฑุถ ูุงูู ูุญูุงุช ุงูุงูุชุฑุงุถูุฉ: ูุง ูุชุตู ุงููููู ุฅุทูุงูุงู ุจู ูููุงุช ุงูุญุงุณูุจ ุงููุนููุฉุ ุจู ูุนู ู ุฏุงุฎู ุฎุงุฏู ุนุฑุถ ุงูุชุฑุงุถู ู ุซู Xvfb ุฃู Wayland ุฃู ุญุงููุฉ VNC ู ุนุฒููุฉ.
- ุงุนุชุฑุงุถ ูุฏุงุกุงุช ุงููุธุงู ุนุจุฑ eBPF: ูุชู ุฑุตุฏ ูู ูุน ุฃู ุงุณุชุฏุนุงุกุงุช ููุธุงู ุงูุชุดุบูู ุชูุฏู ูุชุนุฏูู ู ููุงุช ุงููุธุงู ุงูุญุณุงุณุฉ ุฃู ุชุซุจูุช ุจุฑู ุฌูุงุช ุบูุฑ ู ุตุฑุญ ุจูุง.
- ุฒุฑ ุงูุฅููุงู ุงูุทุงุฑุฆ ุงูุจุดุฑู (Hardware Kill-Switch): ุจู ุฌุฑุฏ ุฃู ูุญุฑู ุงูู ุณุชุฎุฏู ุงููุฃุฑุฉ ุงูู ุงุฏูุฉ ุฃู ูุถุบุท ุนูู ู ูุชุงุญ ุงููุฑูุจุ ูุชู ุฅููุงู ุชุฏูู ุฅุดุงุฑุงุช ุงููููู ููุฑุงู ูุฅุนุงุฏุฉ ุงูุณูุทุฑุฉ ููุจุดุฑ.
09. ู ุตูููุฉ ุงูู ูุงุฑูุฉ ุงูู ุนู ุงุฑูุฉ ูุงูุฃุฏูุงุช ุงูู ุฑุชุจุทุฉ
ููู ุชุชู ุงูุฒ ุฃุจุฑุฒ ุฃุทุฑ ุนู ู ูููุงุก ุณุทุญ ุงูู ูุชุจ ูุงูุชุญูู ุจุงูุญุงุณูุจ ูู ุนุงู 2026ุ
| ุงูู ุนูุงุฑ ุงูู ุนู ุงุฑู | UI-TARS (ByteDance) | Claude 3.7 Computer Use | OpenHands | ุจูุฆุฉ ุนุฒู E2B Desktop |
|---|---|---|---|---|
| ุทุจูุนุฉ ุงููู ูุฐุฌ | ู ูุชูุญ ุงูุฃูุฒุงู (VLA 7B/72B) | ูุงุฌูุฉ ุจุฑู ุฌุฉ ุณุญุงุจูุฉ ุฎุงุตุฉ | ู ูุณู ู ุชุนุฏุฏ ุงููู ุงุฐุฌ | ุจูุฆุฉ ุชุดุบูู ุนุฒู ุจููููุฉ |
| ุงูุฅุฏุฑุงู ุงูุฑุฆูุณู | ุจูุณูุงุช ุฎุงู (System-2 VLA) | ุจูุณูุงุช ุฎุงู (Vision API) | ูุฌูู (DOM + ุทุฑููุฉ + GUI) | ู ุฎุฒู ุฅุทุงุฑุงุช ุดุงุดุฉ ุงูุชุฑุงุถูุฉ |
| ูู ุท ุงูุงุณุชุถุงูุฉ | ุงุณุชุถุงูุฉ ุฐุงุชูุฉ (ุจุทุงูุงุช GPU ู ุญููุฉ) | ุณุญุงุจู ู ุฏุงุฑ (Anthropic API) | ุงุณุชุถุงูุฉ ุฐุงุชูุฉ / ุณุญุงุจู | ุณุญุงุจู ู ุฏุงุฑ / MicroVM |
| ุฏุนู ูุธู ุงูุชุดุบูู | Windows, macOS, Linux, Android | Windows, macOS, Ubuntu | Linux, Docker, ุงูู ุชุตูุญ | Linux (Firecracker MicroVM) |
| ุญุงูุงุช ุงูุงุณุชุฎุฏุงู ุงูู ุณุชูุฏูุฉ | ุฃุชู ุชุฉ ุงูุนู ููุงุช RPAุ ูุงุฎุชุจุงุฑ ุงูุฌูุฏุฉ | ู ูุงู ุงูุนู ู ุงูู ุนุฑูู ุงูู ุนูุฏุฉ | ูููุงุก ููุฏุณุฉ ุงูุจุฑู ุฌูุงุช | ุนุฒู ูุญู ุงูุฉ ุชูููุฐ ุงููููุงุก |
| ุงูู ุตุฏุฑ ุงูู ูุชูุญ | ู ูุชูุญ ุงูู ุตุฏุฑ ุจุงููุงู ู (100%) | ู ุบูู ูุฎุงุต (Proprietary) | ู ูุชูุญ ุงูู ุตุฏุฑ ุจุงููุงู ู (100%) | ุญุฒู ุฉ SDK ู ูุชูุญุฉ / ุณุญุงุจุฉ ู ุฏุงุฑุฉ |
E2B
ุจูุฆุงุช ุนุฒู ุณุญุงุจูุฉุจูุฆุงุช ุชุดุบูู ู ุนุฒููุฉ ูุณุฑูุนุฉ ููุบุงูุฉ ุชุนุชู ุฏ ุนูู Firecracker MicroVMs ูุชุดุบูู ููุฏ ุงููููุงุก ูุชุตูุญ ุงูุฃุณุทุญ ุงูู ูุชุจูุฉ ุจุฃู ุงู ุชุงู ูุฒู ู ุจุฏุก ูุงุฆู.
ุงุณุชูุดู ุฃุฏุงุฉ E2B โClaude 3.7 Sonnet
ูู ูุฐุฌ ุฑุงุฆุฏุงููู ูุฐุฌ ุงูุงุณุชุฏูุงูู ุงูู ุชูุฏู ู ู Anthropic ุงูู ุฒูุฏ ุจูุฏุฑุงุช ุฃุตููุฉ ููุชุญูู ุจุงูุญุงุณูุจ ูุฃุชู ุชุฉ ูุงุฌูุงุช ุณุทุญ ุงูู ูุชุจ ูุงูู ุชุตูุญ ุนุจุฑ ุฎุทูุงุช ู ุชุนุฏุฏุฉ.
ุงุณุชูุดู Claude 3.7 Sonnet โOpenHands
ู ูุชูุญ ุงูู ุตุฏุฑุงูู ูุตุฉ ุงูุฑุงุฆุฏุฉ ู ูุชูุญุฉ ุงูู ุตุฏุฑ ูุจูุงุก ูุชุดุบูู ูููุงุก ุชุทููุฑ ุงูุจุฑู ุฌูุงุช ูุงูุชุญูู ูู ู ูุฌู ุงูุฃูุงู ุฑ ูุณุทุญ ุงูู ูุชุจ ู ุน ุฏุนู ุงููุดุฑ ุงูู ุญูู ุงููุงู ู.
ุงุณุชูุดู OpenHands โDevin
ููุฏุณุฉ ุจุฑู ุฌูุฉ ู ุณุชููุฉุงูู ุณุงุนุฏ ุงูู ุณุชูู ูููุฏุณุฉ ุงูุจุฑู ุฌูุงุช ู ู Cognition ุงูู ุฌูุฒ ุจุจูุฆุฉ ุนู ู ู ุชูุงู ูุฉ ุชุถู ุงูู ุชุตูุญ ูุงูุทุฑููุฉ ูู ุณุงุญุฉ ุณุทุญ ุงูู ูุชุจ ูุฅูุฌุงุฒ ุงูู ุดุงุฑูุน ุงูู ุนูุฏุฉ.
ุงุณุชูุดู Devin โ10. ุงูุฃุณุฆูุฉ ุงูุดุงุฆุนุฉ (FAQ)
ุณ1: ูู ุงุฐุง ูุง ูููู ุจุจูุงุก ูุงุฌูุงุช ุจุฑู ุฌูุฉ ู ุฎุตุตุฉ (APIs) ุจุฏูุงู ู ู ุจูุงุก ูููุงุก ูุงุฌูุงุช ุณุทุญ ุงูู ูุชุจุ
ูู ุงููุถุน ุงูู ุซุงููุ ุชููุฑ ุฌู ูุน ุงูุชุทุจููุงุช ูุงุฌูุงุช ุจุฑู ุฌุฉ ูุงุถุญุฉ ูู ูุซูุฉ. ูููู ุนู ููุงูุ ุชุญุชูู ุจูุฆุงุช ุงูู ุคุณุณุงุช ุนูู ุขูุงู ุงูุจุฑู ุฌูุงุช ุงููุฏูู ุฉ (SAPุ ุงูุฃูุธู ุฉ ุงูู ุฑูุฒูุฉุ ุฃุฏูุงุช ุงูู ุญุงุณุจุฉ ุงููุฏูู ุฉ) ุญูุซ ูุชุทูุจ ุจูุงุก ูุงุฌูุงุช API ู ูุงููู ุงูุฏููุงุฑุงุช ูุณููุงุช ู ู ุฅุนุงุฏุฉ ุงูููููุฉ. ูุชูุญ ูููุงุก ูุงุฌูุงุช ุณุทุญ ุงูู ูุชุจ ุชูุงู ูุงู ููุฑูุงู ุฏูู ูู ุณ ุงูุดูุฑุฉ ุงูู ุตุฏุฑูุฉ ุฃู ุชุนุฏูู ููุงุนุฏ ุงูุจูุงูุงุช ุงูุญุงููุฉ.
ุณ2: ู ุง ูู ุฏูุฉ ุงูุดุงุดุฉ ุงูู ูุงุณุจุฉ ูุชู ุฑูุฑ ููุทุงุช ุงูุดุงุดุฉ ููููู ุจุตุฑูุ
ูุคุฏู ุชู ุฑูุฑ ููุทุงุช ุจุฏูุฉ 4K ูุงู ูุฉ ุฅูู ุชุถุฎู ูุจูุฑ ูู ุงุณุชููุงู ุงูุฑู ูุฒ ูุฒูุงุฏุฉ ุฒู ู ุงูุงุณุชุฌุงุจุฉ. ุชุนุชู ุฏ ุงูุฃูุธู ุฉ ุงูุฅูุชุงุฌูุฉ ุนูู ุงูุชูุงุท ุงูุดุงุดุฉ ุจุฏูุฉ 1920x1080ุ ูู ุนุงูุฑุฉ ุงูุฅุญุฏุงุซูุงุช ุฅูู ุดุจูุฉ ููุงุณูุฉ ุฏุงุฎููุฉ 1000x1000 ููุงุณุชุฏูุงูุ ุซู ุฅุนุงุฏุฉ ุฅุณูุงุท ุงูุฅุญุฏุงุซูุงุช ุงููุงุชุฌุฉ ุฑูุงุถูุงู ุนูู ุงูุจูุณูุงุช ุงููุนููุฉ ููุดุงุดุฉ.
ุณ3: ููู ูุชุนุงู ู ูููุงุก ุณุทุญ ุงูู ูุชุจ ู ุน ุงูุชุญู ูู ุงูุฏููุงู ููู ูููุงุฌูุงุช ูุญุฑูุงุช ุงูุฑุณูู ุ
ุนูู ุนูุณ ู ุชุตูุญุงุช ุงูููุจ ุงูุชู ุชุตุฏุฑ ุฃุญุฏุงุซ ู ุซู DOMContentLoaded ุฃู networkidleุ ูุง ุชููุฑ ุจูุฆุฉ ุณุทุญ ุงูู ูุชุจ ุชูุจููุงุช ุฃุตููุฉ ุจุงูุชู ุงู ุชุญู ูู ุงูุชุทุจูู. ูุนุชู ุฏ ุงููููุงุก ุงูู ุชูุฏู ูู ุนูู ุงุณุชุทูุงุน ุงููุฑูู ุงูุจุตุฑูุฉ (Visual Delta Polling): ุงูุชูุงุท ุฅุทุงุฑุงุช ูู 250 ู ููู ุซุงููุฉ ูุงูุชุฃูุฏ ู ู ุงูุฎูุงุถ ู ุนุฏู ุชุบูุฑ ุงูุจูุณูุงุช ูุฃูู ู ู 1% ููุชุฃูุฏ ู ู ุงุณุชูุฑุงุฑ ุงูุดุงุดุฉ ูุจู ุงูุฎุทูุฉ ุงูุชุงููุฉ.
ุณ4: ูู ูู ูู ููููุงุก ุณุทุญ ุงูู ูุชุจ ุงูุนู ู ุนุจุฑ ุดุงุดุงุช ู ุชุนุฏุฏุฉ ุฃู ููุงูุฐ ู ุชุนุฏุฏุฉุ
ูุนู . ูู ูู ูู ุญุฑูุงุช ุชุฎุทูุท ุงูุฅุญุฏุงุซูุงุช ุฅู ุง ู ุนุงู ูุฉ ุงูุดุงุดุงุช ุงูู ุชุนุฏุฏุฉ ูููุญุฉ ูุงุญุฏุฉ ู ุฏู ุฌุฉ (ู ุซู 3840x1080) ุฃู ุงุณุชุฎุฏุงู ูุงุฌูุงุช ูุธุงู ุงูุชุดุบูู ูุชุฑููุฒ ุงููุงูุฐุฉ ุงูู ุณุชูุฏูุฉ ุนูู ุงูุดุงุดุฉ ุงูุงูุชุฑุงุถูุฉ ุงูุฑุฆูุณูุฉ ูุจู ุชูููุฐ ุฎุทูุฉ ุงูุฅุฏุฑุงู ูุงูุงุณุชุฏูุงู.
ุณ5: ูู ูุณุชุทูุน ูู ูุฐุฌ UI-TARS ุงูุนู ู ู ุญููุงู ุนูู ุฃุฌูุฒุฉ ุงูู ุณุชูููููุ
ูู ูู ููู ูุฐุฌ UI-TARS ุจุญุฌู 7B ุงูุนู ู ู ุญููุงู ุนูู ู ุญุทุงุช ุงูุนู ู ุงูุญุฏูุซุฉ (ู ุซู ุฃุฌูุฒุฉ Apple Silicon ุจุฐุงูุฑุฉ ู ูุญุฏุฉ 32GB+ ุฃู ุจุทุงูุฉ NVIDIA RTX 4090) ุจุงุณุชุฎุฏุงู ุชูุณููุงุช ุงูุชูู ูู ู ุซู GGUF ุฃู AWQ. ุฃู ุง ูู ูุฐุฌ 72B ููุชุทูุจ ุฎูุงุฏู ุจู ุณุฑุนุงุช A100/H100 ูุถู ุงู ุฒู ู ุงุณุชุฌุงุจุฉ ููู ุนู ุซุงููุฉ ูุงุญุฏุฉ.