oq'jpRdZddlmZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z mZmZejeZehdZdZdd eDZejd ezd zejZejd ezd zejZd3dZehdZehdZd4dZd5dZ d6dZ!d7d Z" d8d9d!Z#d:d"Z$d;d%Z%d8dd1Z(gd2Z)dS)?uRouting helpers for inbound user-attached images. Two modes: native — attach images as OpenAI-style ``image_url`` content parts on the user turn. Provider adapters (Anthropic, Gemini, Bedrock, Codex, OpenAI chat.completions) already translate these into their vendor-specific multimodal formats. text — run ``vision_analyze`` on each image up-front and prepend the description to the user's text. The model never sees the pixels; it only sees a lossy text summary. This is the pre-existing behaviour and still the right choice for non-vision models. The decision is made once per message turn by :func:`decide_image_input_mode`. It reads ``agent.image_input_mode`` from config.yaml (``auto`` | ``native`` | ``text``, default ``auto``) and the active model's capability metadata. In ``auto`` mode: - If the user has explicitly configured ``auxiliary.vision.provider`` (i.e. not ``auto`` and not empty), we assume they want the text pipeline regardless of the main model — they've opted in to a specific vision backend for a reason (cost, quality, local-only, etc.). - Otherwise, if the active model reports ``supports_vision=True`` in its models.dev metadata, we attach natively. - Otherwise (non-vision model, no explicit override), we fall back to text. This keeps ``vision_analyze`` surfaced as a tool in every session — skills and agent flows that chain it (browser screenshots, deeper inspection of URL-referenced images, style-gating loops) keep working. The routing only affects *how user-attached images on the current turn* are presented to the main model. ) annotationsN)Path)AnyDictListOptionalTuple>autotextnative) .png.jpg.jpeg.gif.webp.bmpz.tiffz.tifz.heic|c#@K|]}|dVdS).N)lstrip).0es 8/home/ubuntu/.hermes/hermes-agent/agent/image_routing.py r;s,AAahhsmmAAAAAAz/(?\"']+?\.(?:z)(?:\?[^\s<>\"']*)?r strreturnTuple[List[str], List[str]]c t|tr|sggfSg tjd|tjD]=} ||f>tjd|D]=} ||f>d fd }g}t}t|D]}||r | d}tj |} tj |sun#t$rYwxYw||vr||||g}t} t"|D]y}||r | d} | d } | | vrO| | || z||fS) uScan free-form text for image references the model should see. Returns ``(local_paths, urls)``: * ``local_paths`` — absolute (``/``) or home-relative (``~/``) paths whose suffix is an image extension AND whose expanded form exists on disk as a file. Order-preserving, deduplicated. * ``urls`` — ``http(s)://…`` URLs whose path ends in an image extension (a ``?query`` is allowed after the extension). Order-preserving, deduplicated. Matches inside fenced code blocks (``` ``` ```) and inline backticks (`` `…` ``) are skipped so that snippets pasted into a task body for reference aren't mistaken for live attachments. This mirrors the behaviour of ``gateway.platforms.base.BaseAdapter.extract_local_files``. Local paths are validated against the filesystem; URLs are not (the provider fetches them at request time). z```[^\n]*\n.*?```z `[^`\n]+`posintrboolc<tfdDS)Nc3>K|]\}}|cxko|kncVdSN)rsrr s rrz7extract_image_refs.._in_code..ns;77DAq1<<<._in_codems'7777J777777rrz .,;:!?)]>)r r!rr") isinstancerrefinditerDOTALLappendstartendset_LOCAL_IMAGE_PATH_REgroupospath expanduserisfileOSErroradd _IMAGE_URL_RErstrip) r mr* local_paths seen_pathsmatchrawexpandedurls seen_urlsurlr)s @rextract_image_refsrFNss( dC 2v )+J [-tRY ? ?0017799aeegg.//// [t , ,0017799aeegg.////888888 K55J%..t44%% 8EKKMM " "  kk!nn7%%c** 7>>(++      H  z ! ! x   8$$$$D%%I''--   8EKKMM " "  kk!nnjj%% )    c C  sE66 FF>1onyestrue>0noofffalserArOptional[bool]c*t|tr|St|tr|dvrt|SdSt|tr<|}|t vrdS|tvrdSdS)z@Return True/False for recognised boolean values, None otherwise.)rNTF)r+r"r!rstriplower _TRUE_TOKENS _FALSE_TOKENS)rAr's r_coerce_capability_boolrVs#t #s &==99 t#s IIKK      4  5 4rcfgOptional[Dict[str, Any]]providermodelct|tsdS|d}t|tr|ni}t|d}||St |dpd}|d}t|tr|ni}ttd||fD]} || } t| tr| ni} | d} t| tr| ni} | |}t|tr|ni}t|d}||cS|d}t|trjt}td||fD]o} | | | d r+| | td dW| d | p|D]} t| tst | d pd}||vrS| d} t| tr| ni} | |}t|tr|ni}t|d}||cSdS) u`Resolve user-declared vision capability from config.yaml. Resolution order, first hit wins: 1. ``model.supports_vision`` (top-level shortcut for the active model) 2. ``providers..models..supports_vision`` (named custom providers — ``provider`` may be the runtime-resolved value ``"custom"`` and/or the user-declared name under ``model.provider``; both are tried) Returns None when no override is set, so the caller falls through to models.dev. Returns False explicitly only when the user wrote a recognised boolean false token. NrZsupports_visionrY providersmodelscustom_providerszcustom:name) r+dictgetrVrrRfromkeysfilterlistr2r: startswithlen)rWrYrZ model_cfg_raw model_cfgtopconfig_provider providers_raw providers_cfgp entry_rawentry models_raw models_cfg per_model_raw per_modelcoercedr`candidate_names entry_names r_supports_vision_overriderys7$ c4 tGGG$$M1;M41P1P X VXI !)--0A"B"B C CC  )-- 339r::@@BBOGGK((M5? t5T5T$\MMZ\M ]]6$?(CDD E E  !%%a(( -7 4-H-H P bYYx(( 3=j$3O3O%WZZUW "u-- 5? t5T5T$\MMZ\ ))--8I*J*JKK  NNN  ww122"D)) #uux9:: 3 3A    " " "||I&& 3##Ac)nnoo$67777##MaMM2222)  Ii.. Y]]6228b99??AAJ00"x00J'1*d'C'CKJ&NN511M)3M4)H)HP bI-immr]r T)r+rbrcrrRrS)rWauxrrYrZrs r_explicit_aux_vision_overriders  c4 u ''+   $"C c4 u WWX   $"F fd # #u6::j))/R006688>>@@H  7##)r * * 0 0 2 2E6::j))/R006688H<hu 4rct|||}||S|r|sdS ddlm}|||}n5#t$r(}td|||Yd}~dSd}~wwxYw|dSt |jS)aReturn True/False if we can resolve caps, None if unknown. Consults the user's ``supports_vision`` override in config.yaml first (so custom/local models declared as vision-capable don't fall through to text routing in ``auto`` mode), then falls back to models.dev. Nr)get_model_capabilitiesu2image_routing: caps lookup failed for %s:%s — %s)ryagent.models_devr Exceptionloggerdebugr"r\)rYrZrWoverridercapsexcs r_lookup_supports_visionr!s)h>>H 5t;;;;;;%%h66  I8UZ\_```ttttt |t $ % %%s0 A"AA"c@d}t|trN|dpi}t|tr"t|d}|dkrdS|dkrdSt |rdSt |||}|durdSdS)a1Return ``"native"`` or ``"text"`` for the given turn. Args: provider: active inference provider ID (e.g. ``"anthropic"``, ``"openrouter"``). model: active model slug as it would be sent to the provider. cfg: loaded config.yaml dict, or None. When None, behaves as auto. r agentimage_input_moder r T)r+rbrcr}rr)rYrZrWmode_cfg agent_cfgsupportss rdecide_image_input_moder<sH#tGGGG$$* i & & G#IMM2D$E$EFFH8x6v%S))v&x<GIF87aGIF89a image/gif sRIFFsWEBP image/webpsBM image/bmpsftyp>heicheimheisheixhevchevxmif1msf1z image/heic)rgrhrAs r_sniff_mime_from_bytesrls t ~~*++{ ~~o&&| 2A2w((({ 3xx2~~#bqb'W,,QrTg1E1E| ~~e{ 3xx2~~#ac(g--#ad)@33| 4rr6rOptional[bytes]c|t|}|r|Stjt|\}}|r|dr|S|j}ddddddd|dS) zReturn image MIME type for *path*. If *raw* bytes are provided, magic-byte sniffing wins (authoritative). Otherwise we fall back to ``mimetypes`` then suffix-based defaults. Nzimage/rrrrr)rrr rrr)r mimetypes guess_typerrgsuffixrSrc)r6rAsniffedmime_rs r _guess_mimers  (--  N"3t99--GD! )) [   F    c&, rc |}n4#t$r'}td||Yd}~dSd}~wwxYwt ||}t j|d}d|d|S)u!Encode a local image as a base64 data URL at its native size. Size limits are NOT enforced here — the agent retry loop (``run_agent._try_shrink_image_parts_in_messages``) shrinks on the provider's first rejection. Keeping this simple means providers that accept large images (OpenAI 49 MB+, Gemini 100 MB) don't pay a silent quality tax just because one other provider is stricter. Returns None only if the file can't be read (missing, permission denied, etc.); the caller reports those paths in ``skipped``. u'image_routing: failed to read %s — %sNrasciizdata:z;base64,) read_bytesrrwarningrbase64 b64encodedecode)r6rArrb64s r_file_to_data_urlrsoo @$LLLttttt t % % %D  3   & &w / /C &4 & & & &&s AAA user_text image_paths List[str] image_urlsOptional[List[str]]&Tuple[List[Dict[str, Any]], List[str]]cg}g}g}g}|D]}t|}|r|s#|t |\t |} | s#|t ||dd| id|t ||pgD]J} | pd} | s|dd| id|| K|pd} |s|r~| pd} g} | d|D| d|D| dd | z}d |d g}||||fSg}| r|d | d ||fS) uBuild an OpenAI-style ``content`` list for a user turn. Shape: [{"type": "text", "text": "...\n\n[Image attached at: /local/path]"}, {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}, {"type": "image_url", "image_url": {"url": "https://example.com/a.png"}}, ...] Local paths are read from disk and embedded as base64 ``data:`` URLs. Remote URLs (``http(s)://``) are passed through verbatim — the provider fetches them server-side. The model still sees the pixels either way. For each successfully attached image, a hint is appended to the text part: * local path → ``[Image attached at: ]`` * URL → ``[Image attached: ]`` The hint gives the model a string handle so MCP/skill tools that take an image path or URL argument can be invoked on the same image without an extra round-trip. This parallels the text-mode hint produced by ``Runner._enrich_message_with_vision`` (``vision_analyze using image_url: ``) so behaviour is consistent across both image input modes. Images are attached at their native size. If a provider rejects the request because an image is too large (e.g. Anthropic's 5 MB per-image ceiling), the agent's retry loop transparently shrinks and retries once — see ``run_agent._try_shrink_image_parts_in_messages``. Returns (content_parts, skipped). Skipped entries are local paths that couldn't be read from disk; URLs are never skipped (they're not validated here). image_urlrE)typerr]zWhat do you see in this image?c3"K|] }d|dV dS)z[Image attached at: ]Nr&)rros rrz-build_native_content_parts.. s.NN!5555NNNNNNrc3"K|] }d|dV dS)z[Image attached: rNr&)rus rrz-build_native_content_parts.. s.JJq2a222JJJJJJrz  r )rr ) rexistsis_filer/rrrRextendjoin)rrrskipped image_partsattached_paths attached_urlsraw_pathrodata_urlrEr base_text hint_lines combined_textpartss rbuild_native_content_partsrs\LG(*K "N!M - - NNxxzz   NN3x== ) ) ) $Q''  NN3x== ) ) ) *      c(mm,,,,R""yb!!        S!!!! O " " $ $D<< " NN~NNNNNNJJMJJJJJJ$***TYYz-B-BB 06 'N'N&O [!!!g~ E 5 fd33444 '>r)rrrF)r rrr)rArrrO)rWrXrYrrZrrrO)rArrr)rWrXrr"r%)rYrrZrrWrXrrO)rYrrZrrWrXrr)rArrr)r6rrArrr)r6rrr)rrrrrrrr)*__doc__ __future__rrloggingrr5r,pathlibrtypingrrrrr getLogger__name__r frozensetr{ _IMAGE_EXTSr_IMAGE_EXT_PATTERNcompile IGNORECASEr3r;rFrTrUrVryr}rrrrrrr__all__r&rrrs  D#"""""  33333333333333  8 $ $y33344  XXAA[AAAAA "rz69KKfTM !$669OOM BBBBZy33344  55566 "KKKK\8%)&&&&&6`    F     4''''2'+VVVVVr   r