Fij& UdZddlmZddlmZmZdddiddgd d d d dgd dd dddddddddddddddddddidddd d!dgd"d#d d$ddgd%d&d'd(d)dd*dd+dd,dd-dddiddd.d d/dddiddd0d d1dgd2d3d d4dd5dd6dd7dd8dd9dd:dd;ddd?dd>d@ddAdgdBdCZdDedE<dIdGZdHS)JuNSchema for the generic `computer_use` tool. Model-agnostic. Any tool-calling model can drive this. Vision-capable models should prefer `capture(mode='som')` then `click(element=N)` — much more reliable than pixel coordinates. Pixel coordinates remain supported for models that were trained on them (e.g. Claude's computer-use RL). ) annotations)AnyDict computer_useuDrive the macOS desktop in the background — screenshots, mouse, keyboard, scroll, drag — without stealing the user's cursor, keyboard focus, or Space. Preferred workflow: call with action='capture' (mode='som' gives numbered element overlays), then click by `element` index for reliability. Pixel coordinates are supported for models trained on them. Works on any window — hidden, minimized, on another Space, or behind another app. macOS only; requires cua-driver to be installed.objectactionstring) captureclick double_click right_click middle_clickdragscrolltypekey set_valuewait list_apps focus_appu Which action to perform. `capture` is free (no side effects). All other actions require approval unless auto-approved. Use `set_value` for select/popup elements and sliders — it selects the matching option directly without opening the native menu (no focus steal).)renum descriptionmode)somvisionaxuCapture mode. `som` (default) is a screenshot with numbered overlays on every interactable element plus the AX tree — best for vision models, lets you click by element index. `vision` is a plain screenshot. `ax` is the accessibility tree only (no image; useful for text-only models).appzOptional. Limit capture/action to a specific app (by name, e.g. 'Safari', or bundle ID, 'com.apple.Safari'). If omitted, operates on the frontmost app's window or the whole screen.)rr max_elementsintegeruOptional cap on the AX `elements` array returned by `action='capture'`. Default 100, hard maximum 1000. Dense UIs (Electron apps such as Obsidian or VS Code, JetBrains IDEs) can publish 500+ AX nodes — capping prevents a single capture from blowing session context. When the cap trims the response, `total_elements` and `truncated_elements` are surfaced in the result so you can re-call with `app=` to narrow scope or raise `max_elements` when the full tree is required. Has no effect on `mode='som'` / `mode='vision'` when a screenshot is included in the response; only the rare image-missing fallback returns an `elements` array and is subject to the cap.di)rrdefaultminimummaximumelementzoThe 1-based SOM index returned by the last `capture(mode='som')` call. Strongly preferred over raw coordinates. coordinatearrayrzPixel coordinates [x, y] in logical screen space (as returned by capture width/height). Only use this if no element index is available.)ritemsminItemsmaxItemsrbutton)leftrightmiddlezMouse button. Defaults to left. modifiers)cmdshiftoptionaltctrlfn)rrz%Modifier keys held during the action.)rr)r from_elementzSource element index (drag). to_elementzTarget element index (drag).from_coordinatez3Source [x,y] (drag; use when no element available). to_coordinatez3Target [x,y] (drag; use when no element available). direction)updownr-r.zScroll direction.amountzScroll wheel ticks. Default 3.valuezFor action='set_value': the value to set on the element. For AXPopUpButton / select dropdowns, pass the option's display label (e.g. 'Blue'). For sliders and other AXValue-settable elements, pass the numeric or string value.textz+Text to type (respects the current layout).keyszUKey combo, e.g. 'cmd+s', 'ctrl+alt+t', 'return', 'escape', 'tab'. Use '+' to combine.numberzSeconds to wait. Max 30.booleanuOnly for action='focus_app'. If true, brings the window to front (DISRUPTS the user). Default false — input is routed to the app without raising, matching the background co-work model.zIf true, take a follow-up capture after the action and include it in the response. Saves a round-trip when you need to verify an action's effect.)seconds raise_window capture_after)r propertiesrequired)namer parametersDict[str, Any]COMPUTER_USE_SCHEMAreturnctS)z2Return the generic OpenAI function-calling schema.)rL>/home/ubuntu/.hermes/hermes-agent/tools/computer_use/schema.pyget_computer_use_schemarRs rPN)rMrK) __doc__ __future__rtypingrrrL__annotations__rRrOrPrQrWsq#"""""  ;o   H%o 6  ///-  7o N  BOo ` ! *)ao N !'Oo ^  ),5  _o t  333@uo ~ $KKK G o P Y-KMMQo T 9+IKKUo X  ),1T   Yo d  ),1T eo r  7772so | !?}o F  SGo Z  L[o b  ;co r!9 "="BOo o o `Jess@'@'@@@@FrP