,j-n UdZddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z ddl Z e rddl mZddlmZmZmZmZmZmZddlmZmZmZddlmZmZddlmZdZe e ed <dZ e e ed <dZ!e e ed <dZ"e e ed <dZ#e e ed <ddl$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z-m.Z/m0Z0ddl1m2Z2m3Z3m4Z4ddl5m6Z6m7Z7ddl8Z8ej9e:Z;defdZ?de@fdZAeBhdZCeBhdZDddefd!ZIde>fd"ZJdeKe<fd#ZLd$ZMd%e de>fd&ZNdd'e e<deOe e e e<eed8eZWdd?ZXdd@ZYddBed'e e<d1eSdefdJZ\de>fdKZ]d efdLZ^de>fdMZ_de>fdNZ`e:dOkr eadPeadQe_ZbeZce>ejddRd-eZfe>ejddSd-eZge`ZheQZiebreEZjeadTejejdUkr eadVnejdWkr eadXnejdYkr eadZnejd[kread\e=d]nejd^kr ead_nejd`kr eadanegrBeadbejddSekdcnTefr eaddnFecreadeen-eadfn!eadgeadheehs"eadieadjeadkneadleiebse8jldmeadnehreadoeieadpeMdqeRjmr0eadreRjneadseRjodteRjndun eadveadweadxeadyead-eadzead{ead-ead|ead}ead~eadehreadeadeadead-eadeadeadeadeadeadead-eadeadeadeadeadeadeadeadeadeadeadeadddlpmqZqmrZrdddddddddmddAdddBgddZsddddFdddiddAdidFgddZteqjuddesde\eLddeqjuddetde\eLdEdd dS)ae Standalone Web Tools Module This module provides generic web tools that work with multiple backend providers. Backend is selected during ``hermes tools`` setup (web.backend in config.yaml). When available, Hermes can route Firecrawl calls through a Nous-hosted tool-gateway for Nous Subscribers only. Available tools: - web_search_tool: Search the web for information - web_extract_tool: Extract content from specific web pages Backend compatibility: - Exa: https://exa.ai (search, extract) - Firecrawl: https://docs.firecrawl.dev/introduction (search, extract; direct or derived firecrawl-gateway. for Nous Subscribers) - Parallel: https://docs.parallel.ai (search, extract) - Tavily: https://tavily.com (search, extract) LLM Processing: - Uses OpenRouter API with Gemini 3 Flash Preview for intelligent content extraction - Extracts key excerpts and creates markdown summaries to reduce token usage Debug Mode: - Set WEB_TOOLS_DEBUG=true to enable detailed logging - Creates web_tools_debug_UUID.json in ./logs directory - Captures all tool calls, results, and compression metrics Usage: from web_tools import web_search_tool, web_extract_tool # Search the web results = web_search_tool("Python machine learning libraries", limit=3) # Extract content from URLs content = web_extract_tool(["https://example.com"], format="markdown") N)ListDictAnyOptional TYPE_CHECKING) Firecrawl)r_firecrawl_backend_help_suffix_get_firecrawl_client_get_firecrawl_gateway_url_is_tool_gateway_readycheck_firecrawl_api_key)_normalize_tavily_documents _normalize_tavily_search_results_tavily_request)_get_async_parallel_client_get_parallel_client)_get_exa_client_firecrawl_client_firecrawl_client_config_parallel_client_async_parallel_client _exa_client)async_call_llmextract_content_or_reasoningget_async_text_auxiliary_client) DebugSession)build_vendor_gateway_urlpeek_nous_access_tokenread_nous_access_tokenresolve_managed_tool_gateway)managed_nous_tools_enabled%nous_tool_gateway_unavailable_messageprefers_gateway)async_is_safe_urlnormalize_url_for_requestnamereturnc ddlm}||}n#t$rd}YnwxYw|tj|d}|pdS)uResolve ``name`` via Hermes config-aware env, falling back to process env. Mirrors the SearXNG provider's ``_searxng_url()`` so that values set through Hermes' config/.env layer (``hermes config set``, ``hermes tools``) are honored here too — not just raw process-env exports. Without this, a config-only ``SEARXNG_URL`` (or any provider key) leaves the backend auto-detect cascade and ``check_web_api_key()`` blind to it. See #34290. r) get_env_valueN)hermes_cli.configr) Exceptionosgetenvstrip)r&r)vals 4/home/ubuntu/.hermes/hermes-agent/tools/web_tools.py _env_valuer2qsz333333mD!!  {ib!! I2    s  ##c:tt|SN)boolr2)r&s r1_has_envr6s  4  ! !!c~ ddlm}|diS#ttf$ricYSwxYw)z5Load the ``web:`` section from ~/.hermes/config.yaml.r load_configweb)r+r:get ImportErrorr,r9s r1_load_web_configr>s^111111{}}  +++  # s #&<<>exaxaiddgstavilysearxngparallel firecrawl brave-free>r@rArCrFsearch capabilityc tdpd}|tvr|Sdt dfdt dfdt dfd t d pt d fd t fd t d fdt dfddtff }|D]\}}|s|dkr |tvr|cSdS)atDetermine which web backend to use (shared fallback). Reads ``web.backend`` from config.yaml (set by ``hermes tools``). Falls back to whichever API key is present for users who configured keys manually without running setup. ``capability`` ("search" | "extract") only affects auto-detect: for ``extract`` we skip search-only backends (``_SEARCH_ONLY_BACKENDS``) so a search-only credential never shadows the keyless Parallel free-MCP extract fallback. An explicit ``web.backend`` value is honored as-is (explicit wins, surfacing that backend's own search-only error rather than rerouting). backendr*rBTAVILY_API_KEYr? EXA_API_KEYrDPARALLEL_API_KEYrEFIRECRAWL_API_KEYFIRECRAWL_API_URLrC SEARXNG_URLrFBRAVE_SEARCH_API_KEY)rDTrAextract) r>r<lowerr/_KNOWN_WEB_BACKENDSr6r _ddgs_package_importable_SEARCH_ONLY_BACKENDS)rH configuredbackend_candidatesrJ availables r1 _get_backendrZs7#$$((339r@@BBHHJJJ((( 8,--. ''( X0112 h233Tx@S7T7TU ,../ H]++, x 6778  )++, 1    " "w2G'G'G  :r7c tdS)ujDetermine which backend to use for web_search specifically. Selection priority: 1. ``web.search_backend`` (per-capability override) 2. ``web.backend`` (shared fallback — existing behavior) 3. Auto-detect from env vars This enables using different providers for search vs extract (e.g. SearXNG for search + Firecrawl for extract). rG_get_capability_backendr7r1_get_search_backendr_s #8 , ,,r7c tdS)uDetermine which backend to use for web_extract specifically. Selection priority: 1. ``web.extract_backend`` (per-capability override) 2. ``web.backend`` (shared fallback — existing behavior) 3. Auto-detect from env vars rRr\r^r7r1_get_extract_backendras #9 - --r7ct}||dpd}|r|St |S)uPShared helper for per-capability backend selection. Reads ``web.{capability}_backend`` from config. Any explicit value is honored **regardless of availability** — including unrecognized typos like ``parrallel`` — so the dispatcher surfaces that backend's own setup/config error rather than silently rerouting to the keyless Parallel default (which would send user queries to a different provider and hide the misconfiguration). This matches ``web_search_registry``'s "explicit config wins" rule. Only an *unset* value falls through to ``_get_backend()``. _backendr*)r>r<rSr/rZ)rHcfgspecifics r1r]r]sa   C:///006B==??EEGGH  # ##r7rJcx|dkrtdS|dkrtdS|dkrtS|dkrtdS|dkrtd S|d krtd S|d krtS|d kr" ddlm}|S#t $rYdSwxYwdS)z:Return True when the selected backend is currently usable.r?rLrDrMrErBrKrCrPrFrQrAr@r)has_xai_credentialsF)r6r rUtools.xai_httprgr,)rJrgs r1_is_backend_availableris % &&&**++++&((((()))) &&&,.///&')))%   : : : : : :&&(( (   55  5sB)) B76B7c2 ddl}dS#t$rYdSwxYw)aJReturn True when the ``ddgs`` Python package can be imported. ddgs is the only backend whose availability is driven by a package presence rather than an env var / config entry. Wrapped in a helper so auto-detect and ``_is_backend_available`` share the same check (and tests can monkeypatch a single symbol). rNTF)rAr=)rAs r1rUrUs7 t uus  c gdS)uReturn tool metadata env vars for the currently enabled web backends. The gateway env vars are always reported — they're metadata strings used by the tool registry to light up the tool when the variable is set. Gating them on ``managed_nous_tools_enabled()`` only saved string noise in the metadata list, but cost a synchronous HTTP refresh against the Nous portal on every CLI startup (invoked at tool-registration time). The behavioral contract is: if the env var is set, the tool sees it; if not, it doesn't. Not-logged-in users simply don't have the vars set, so the extra entries are harmless. ) rLrMrKrNrOFIRECRAWL_GATEWAY_URLTOOL_GATEWAY_DOMAINTOOL_GATEWAY_SCHEMETOOL_GATEWAY_USER_TOKENr^r^r7r1_web_requires_envrp5s    r7clientcddlm}tt|ddpd}||jpd}|dkp|dS)z?Return True when the resolved auxiliary backend is Nous Portal.r)urlparsebase_urlr*znousresearch.comz.nousresearch.com) urllib.parsertstrgetattrhostnamerSendswith)rrrtruhosts r1_is_nous_auxiliary_clientr|]ss%%%%%%76:r228b99H HX   ' -2 4 4 6 6D % % K7J)K)KKr7modelctd\}}tjdd}|p|p|}i}|1t |r"ddlm}ddlm}|p d|i}|||fS) zHResolve the current web-extract auxiliary client, model, and extra body. web_extractAUXILIARY_WEB_EXTRACT_MODELr*Nr)get_auxiliary_extra_body)nous_portal_tagstags) rr-r.r/r|agent.auxiliary_clientragent.portal_tagsr)r}rr default_modelconfigured_modeleffective_model extra_bodyrrs r1_resolve_web_extract_auxiliaryrfs;MJJFMy!>CCIIKK@/@=O!#J 7??CCCCCC666666--//OF500k chars), uses chunked processing with synthesis. For extremely large content (>2M chars), refuses to process entirely. Args: content (str): The raw content to process url (str): The source URL (for context, optional) title (str): The page title (for context, optional) model (str): The model to use for processing (default: google/gemini-3-flash-preview) min_length (int): Minimum content length to trigger processing (default: 5000) Returns: Optional[str]: Processed markdown content, or None if content too short or processing fails ii 順rqi@Bz 2MB limit). Refusing to process.z[Content too large to process: z.1fz#MB. Try a more focused source URL.]z:Content too short (%d < %d chars), skipping LLM processingNzTitle: zSource:   r*z5Content large (%d chars). Using chunked processing...z+Processing content with LLM (%d characters)4 [... summary truncated for context management ...]r?z*Content processed: %d -> %d chars (%.1f%%)dzweb_extract LLM summarization failed (%s). Tip: increase auxiliary.web_extract.timeout in config.yaml or switch to a faster auxiliary model.xu' [Content truncated — showing first , of z chars. LLM summarization timed out. To fix: increase auxiliary.web_extract.timeout in config.yaml, or use a faster auxiliary model. Use browser_navigate for the full page.]) lenloggerwarningdebugappendjoininfo_process_large_content_chunked_call_summarizer_llmr,rw)rrrr}rMAX_CONTENT_SIZECHUNK_THRESHOLD CHUNK_SIZEMAX_OUTPUT_SIZE content_lensize_mb context_info context_strprocessed_contentprocessed_lengthcompression_ratioe truncateds r1process_content_with_llmr}s8!OJO@'ll  ) ) )!I-G NNY[b c c ceWeeee e  # # LLUWbdn o o o4  3    1% 1 1 2 2 2  2    03 0 0 1 1 1:FNdii --66B   ( ( KKOQ\ ] ] ]7eZ   A;OOO"6w U"S"SSSSSSS  ~$%%77$56F6F$GKC%C! ##455 BMPQ// 0; > >WZ  KKDkScevy|e| } } }   5 FF4C4L    ,_,- w< %d charszChunk %d/%d failed: %s2)rrrrr,rrw)rrrsummaryrchunksrr}s r1summarize_chunkz7_process_large_content_chunked..summarize_chunkusL #Oi!mOOVOOOJ0 % G D DiRSmUXY_U`U`befsbtbtvy{BwCwCDDDg% % # # # NN3Y]CKKQTUVQWQWX[Y[X[Q\ ] ] ]d? " " " " " " #sBB C!ACC!C!c.g|]\}}||Sr^r^).0ichunkrs r1 z2_process_large_content_chunked..s) I I I81e__Q & & I I Ir7return_exceptionsTNz#Chunk summarization task failed: %sc|dS)Nrr^)xs r1z0_process_large_content_chunked..s qtr7)keyz ## Section rrzAll chunk summarizations failedzB[Failed to process large content: all chunk summarizations failed]zGot %d/%d chunk summariesz [... truncated ...]zSynthesizing %d summaries...z --- a'You have been given summaries of different sections of a large document. Synthesize these into ONE cohesive, comprehensive summary that: 1. Removes redundancy between sections 2. Preserves all key facts, figures, and actionable information 3. Is well-organized with clear structure 4. Is under z characters zSECTION SUMMARIES: z, Create a single, unified markdown summary.z9No auxiliary model for synthesis, concatenating summariesrrrzdYou synthesize multiple summaries into one cohesive, comprehensive summary. Be thorough but concise.rrrrrrz3Synthesis LLM returned empty content, retrying onceu>Synthesis failed after retry — concatenating chunk summariesrrz+Synthesis complete: %d -> %d chars (%.2f%%)rzSynthesis failed: %sz. [... truncated due to synthesis failure ...]r^)rrrrrintrwtupler enumeratergather isinstance BaseExceptionrsortedrrrrrr,)rrr}rrrrtasksresultssuccessful_results result_item summariesrrresultcombined_summariessynthesis_promptrrrfallbackrr final_summary original_len final_len compressionrrrs `` @@r1rrWsr,F 1c'llJ / /!j.() e KK8#f++zRRR##S#U3PXY\P]K]E^########( J I I Iy7H7H I I IENEBTBBBBBBBBG// k= 1 1  NN@+ N N N !!+....I$%7^^LLLGG 7  G   E9q=EEGEE F F F T 6777SS KK+S^^S[[III 9~~1 v;; ( (,_,-0IIF  KK.I???&++I66 .  . .  . . . . .72PQV2W2W/ OZ  _  NNV W W W{{9--H8}}..#$4_$458QQO"$!/UVV,<==     3(2K %'66+666666664X>>  C NNP Q Q Q+::k::::::::H8BBM  NN[ \ \ \{{9--H8}}..#$4_$458QQO }   / /)*:?*:;>vvM7||  && 2>2B2Bi,..  A/__init__.py`` during plugin discovery. Tool dispatch can be reached from contexts that haven't already triggered discovery — subprocess agent runs, delegate children, standalone scripts, certain test paths — and without it the registry is empty and ``get_provider('firecrawl')`` returns ``None`` even when the user has ``web.extract_backend: firecrawl`` configured and ``FIRECRAWL_API_KEY`` set. The symptom is a misleading "No web extract provider configured" error (issue #27580). Mirrors :func:`tools.browser_tool._ensure_browser_plugins_loaded` exactly: the underlying discovery call is idempotent and cheap on subsequent invocations. Triggering discovery is necessary but not *sufficient*: the sweep can finish without registering the bundled web providers (its exception swallowed below as a warning, a packaged layout where discovery ran before the bundled tree was importable, or a stale empty-discovery cache). When that happens the registry is empty and *both* web_search and web_extract dead-end on "No web {search,extract} provider configured" — even though the keyless Parallel default is supposed to work with zero setup. So after discovery we verify the keyless default landed and, if not, register the bundled providers directly (see :func:`_register_bundled_web_providers_directly`). r)_ensure_plugins_discoveredz+Web plugin discovery failed (non-fatal): %sN get_providerrDz.Bundled web provider fallback check failed: %s) hermes_cli.pluginsr r,rragent.web_search_registryr (_register_bundled_web_providers_directlyr)r excr s r1_ensure_web_plugins_loadedrs8 KAAAAAA""$$$$ KKK DcJJJJJJJJ KL:::::: < # # + 4 6 6 6 6 6 , + LLL EsKKKKKKKKKLs* A=AA)) B3BBc^ ddlm}m}n3#t$r&}td|Yd}~dSd}~wwxYw|dz }|sdS|}ddlm ddl m G fdd }|}ddl }t| D]}|s|d z s|d z sFd |j|vs d |jdd|vrs |d|j} t%| dd} t'| r | |#t$r+}td|j|Yd}~d}~wwxYwdS)uIRegister the repo's bundled web providers without the plugin manager. The normal path is the general plugin sweep (:func:`hermes_cli.plugins._ensure_plugins_discovered`), which auto-loads every ``plugins/web/`` backend (they are ``kind: backend``). This fallback exists for the runtimes where that sweep does not leave the web registry populated — so the keyless Parallel default (and any bundled backend the user explicitly configured) keeps working instead of surfacing a misleading "No web provider configured" error. Imports each bundled ``plugins/web/`` package and calls its ``register()`` directly against :mod:`agent.web_search_registry`. Idempotent (re-register overwrites) and honors an explicit ``plugins.disabled`` entry so a backend the user turned off stays off. r)_get_disabled_pluginsget_bundled_plugins_dirz-Bundled web provider fallback unavailable: %sNr;)WebSearchProvider)register_providerc"eZdZdZdfd ZdS)H_register_bundled_web_providers_directly.._DirectRegistrationCtxz;Minimal plugin ctx exposing only web-provider registration.r'NcBt|r |dSdSr4)r)selfproviderrrs r1register_web_search_providerze_register_bundled_web_providers_directly.._DirectRegistrationCtx.register_web_search_providerws7($566 ,!!(+++++ , ,r7r'N)__name__ __module__ __qualname____doc__r)rrsr1_DirectRegistrationCtxrts>II , , , , , , , , ,r7r!z plugin.yamlz plugin.ymlzweb/zweb-r-z plugins.web.registerz;Direct registration of bundled web provider '%s' failed: %s)r rrr,rris_diragent.web_search_providerrr r importlibriterdirexistsr&replace import_modulerxcallable) rrrweb_dirdisabledr!ctxr&childmodule register_fnrrs @@r1rrRs            DcJJJ&%''%/G >>  $$&&H;;;;;;;;;;;;,,,,,,,,,,, ! " "C))**||~~   %--// 9M8U8U8W8W   5:  8 + +4ej((c2244@@  ,,-HEJ-H-HIIF!&*d;;K $$ ! C       LLM C         #s) =8=,AE55 F*?!F%%F*querylimitc\ t|}n#ttf$rd}YnwxYwtt |dd}||dddddd} ddlm}|rtd d Stdd l m }m }t}|r ||nd}|| s |}|d d d}n8td|j|||||}t%|didg|d<t)j|dd } t%| |d<t,d|t,| S#t2$r} dt5| } td| | |d<t,d|t,t| cYd} ~ Sd} ~ wwxYw)aE Search the web for information using available search API backend. This function provides a generic interface for web search that can work with multiple backends (Parallel or Firecrawl). Note: This function returns search result metadata only (URLs, titles, descriptions). Use web_extract_tool to get full content from specific URLs. Args: query (str): The search query to look up limit (int): Maximum number of results to return (default: 5) Returns: str: JSON string containing search results with the following structure: { "success": bool, "data": { "web": [ { "title": str, "url": str, "description": str, "position": int }, ... ] } } Raises: Exception: If search fails or API key is not set r2rrr3r4Nr) parameterserror results_countoriginal_response_sizefinal_response_size)is_interrupted InterruptedF)success)get_active_search_providerr zDNo web search provider configured. Run `hermes tools` to set one up.r>r8z#Web search via %s: '%s' (limit: %d)datar;r9rindent ensure_asciir;web_search_toolzError searching web: %sr8)r TypeError ValueErrorrmaxtools.interruptr< tool_errorrr r?r r_supports_searchrrr&rGrr<jsondumps_debuglog_callsaver,rwr) r3r4debug_call_datar<r?_wsp_get_providerrJr response_data result_jsonr error_msgs r1rErEsDE  z " E1 s # #E  "#   O5%222222 >   <mU;;; ; #$$$        &''18B$$W---d  8#;#;#=#= 2133H   8MM KK5 ue   %OOE599M+.}/@/@/L/L/P/PQVXZ/[/[+\+\(jquMMM 14[1A1A-.)?;;;  %%%4CFF44  T9%%%#, )?;;; )$$$$$$$$%s2(( F"6D+F"" H+,A4H& H+&H+Turlsformatuse_llm_processingc v &'Kddlm}ddlm}g}|D]}t |} ||sQ|||s3|| s||| rt jdddcS|| ||||ddddddggd } t d t|g} g} |D]G} t| d{Vs| | d d d d 2| | Hd}| sg}nGt}|dkotd }tddlm}m}|r ||nd}||s\|5|s!t jd|jdddS|}|t jddddSt d|jt| ddl}||jr|| |d{V}n"t3j|j| |d{V}| r| |z}d|i}t|dg}t d||| d<tt j|| d<|p t9&t;}|rq|rnt d| dd&fd'|dg}'fd|D}t3j|dd id{V}|D]}t?|t@rt!d!|3|\}}}|d"d#} |d$krG| d%|| d&xxd'z cc<t d(| |d)kr7| d%|t d*| t!d+| n|r7|s5t!d,| dd-|dgD]W}|d"d#} t|d.d }t d/| |Xd0|dgD} d| i}!|r d|!d1<d2|!d3<|!dgkrtEd4}"tG|"}#n&t j|!d5d6}"tG|"}#t|#| d7<| dd8tH%d9| tH&|#S#tN$r}$d:tQ|$}%t)d;|%|%| d<<tH%d9| tH&tE|%cYd}$~$Sd}$~$wwxYw)=a# Extract content from specific web pages using available extraction API backend. This function provides a generic interface for web content extraction that can work with multiple backends. Currently uses Firecrawl. Args: urls (List[str]): List of URLs to extract content from format (str): Desired output format ("markdown" or "html", optional) use_llm_processing (bool): Whether to process content with LLM for summarization (default: True) model (Optional[str]): The model to use for LLM processing (defaults to current auxiliary backend model) min_length (int): Minimum content length to trigger LLM processing (default: 5000) Security: URLs are checked for embedded secrets before fetching. Returns: str: JSON string containing extracted content. If LLM processing is enabled and successful, the 'content' field will contain the processed markdown summary instead of raw content. Raises: Exception: If extraction fails or API key is not set r) _PREFIX_RE)unquoteFz_Blocked: URL contains what appears to be an API key or token. Secrets must not be sent in URLs.r@)rWrXrYr}rN)r7r8pages_extractedpages_processed_with_llmr:r;compression_metricsprocessing_appliedz!Extracting content from %d URL(s)r*z:Blocked: URL targets a private or internal network addressrrrr8rDrM)get_active_extract_providerr zy is a search-only backend and cannot extract URL content. Set web.extract_backend to firecrawl, tavily, exa, or parallel.)rDzcNo web extract provider configured. Set web.extract_backend to firecrawl, tavily, exa, or parallel.zWeb extract via %s: %d URL(s))rXrzExtracted content from %d pagesr]r:z3Processing extracted content with LLM (parallel)...r`llm_processingcK|dd}|dd}|ddp|dd}|s|ddfSt|}t||| d{V}|r3t|}|d kr||z nd }||d<||d<|||| d }||d fS|||d dd d}||dfS)zHProcess a single result with LLM and return updated result with metrics.r Unknown URLrr* raw_contentrN no_contentrr)r original_sizeprocessed_sizer model_used processedcontent_too_short)rrhrirrjreason too_short)r<rr) rrrrfrhrkrirmetricsrrs r1process_single_resultz/web_extract_tool..process_single_resultsRjj 66 7B//$jj;;Xvzz)UW?X?X "6!455 #K 0 0 #;e_j## 8%(^^NJWZ[J[J[(F(Fad%)2F9%,7F=) #)6*8->&5 G"7K77 #)6*7-0&*"5 G"7K77r7c&g|] }|Sr^r^)rrrps r1rz$web_extract_tool..s%NNNv**622NNNr7rTz%Web result processing task failed: %srrerkr_r^rz%s (processed)rnz&%s (no processing - content too short)z%s (no content to process)zPLLM processing requested but no auxiliary model available, returning raw contentllm_processing_unavailablerfz%s (%d characters)cg|]h}|dd|dd|dd|ddd|vr d|diniiS)rr*rrr8rablocked_by_policy)r<)rrs r1rz$web_extract_tool..s     uuUB''w++55B//w    GZ]^F^F^)1-@+ABBdf      r7rzMExtraction powered by the free Parallel Web Search MCP (https://parallel.ai). attributionz%Content was inaccessible or not foundrrBr;base64_image_removalweb_extract_toolzError extracting content: rFr8)* agent.redactr[rvr\r%rGrMrNrrrrr$rar6rr rbr supports_extract display_namer&inspectiscoroutinefunctionrRr to_threadr<rcheck_auxiliary_modelrrrrrKrrOrPrQr,rwr)(rWrXrYr}rr[r\normalized_urls_urlnormalized_urlrR safe_urls ssrf_blockedr_free_parallel_extractrrJrbrSrr|rr]auxiliary_available results_listrprocessed_resultsrrrostatuscontent_lengthtrimmed_resultstrimmed_responserUcleaned_resultrrVrrps( ` @@r1rxrxs>('''''$$$$$$!#O / /2488   d # #   //   00   !8!899  : =    ~....$"4$   $%"# ! O"m% 7_9M9MNNN -/ " & &C*3//////// &##Y%%   %%%% "'F GG*,,G:%Jh7I.J.J*J # ' ( ( (         6=F((111$Hx'@'@'B'B'0I0I0K0K':',#+#8!<!<!<&+    7688#:',!<&+     KK/I    NNN**8+;<<  ( 0 06 0 J JJJJJJJ!( 1$i!!!  -"W,Gw'hll9b99:: 5GGG-<)*47 88L8L4M4M01B#@#B#B355 O G"5O G KKM N N N 0 1 8 89I J J J) 8) 8) 8) 8) 8) 8X$<< 266LNNNNNNNE'.ne&Tt&T&T T T T T T T  1 F F k=99NN#JKXXX*5'jj 66[((#$9:AA'JJJ#$>???1D???KK 0#6666{**#$9:AA'JJJKK H#NNNNNN#?EEEE F " [*= [qrrr 45<<=YZZZ",,y"55 G Gjj 66!$VZZ r%B%B!C!C 0#~FFFF   \\)R00    &7 ! ,6 Z () ] +    * *b 0 0$%LMMK0==NN*%5aeTTTK0==N14^1D1D-.,-445KLLL *O<<<  %%%9Q99  T9%%%#, *O<<< )$$$$$$$$%s-DX/"$X/P'X// Z89A4Z3-Z83Z8cdS)aWhether the web tools should be registered. Always True. Registration is decoupled from credential readiness: with no credentials, search/extract fall back to Parallel's free hosted Search MCP, and an explicitly configured-but-unavailable backend must stay registered so dispatch surfaces that backend's own setup error rather than the tool silently vanishing. For "is web actually configured?" use :func:`check_web_api_key`. Tr^r^r7r1web_tools_registeredr3s  4r7cht ddlm}|dduS#t$rYdSwxYw)zTrue when the bundled ``web-parallel`` provider is registered/enabled. Plugin discovery skips disabled plugins, so a disabled (``plugins.disabled``) or otherwise-unregistered parallel provider yields ``None`` here. rr rDNF)rr r r,r s r1_parallel_provider_registeredr@s^    ::::::|J''t33 uus # 11cf|dkrtdstSt|S)zTrue when *backend* can service calls. Keyless Parallel counts (free MCP). Unknown/typo'd backend names are not usable (so an explicit typo is reported as a config problem rather than masked by the keyless fallback). rDrM)r6rri)rJs r1_backend_usablerOs9 *X.@%A%A -...  ) ))r7cnttottS)uGUsability probe: True when the selected web backends can service calls. Probes the backends that :func:`_get_search_backend` / :func:`_get_extract_backend` actually select (not just shared ``web.backend``), so an explicit per-capability backend with missing credentials — or a typo'd name — reports unusable instead of being masked by the keyless Parallel fallback. Keyless Parallel itself genuinely services calls, so a zero-setup install reports usable. Distinct from :func:`web_tools_registered` (always True — whether the tool is offered). )rr_rar^r7r1check_web_api_keyr^s- .00 1 1 ]oFZF\F\6]6]]r7c.t\}}}|duS)zICheck if an auxiliary text model is available for LLM content processing.Nr)rrrs r1rrls133LFAq  r7__main__u 🌐 Standalone Web Tools Modulez(========================================rNrOu✅ Web backend: r?z! Using Exa API (https://exa.ai)rDz+ Using Parallel API (https://parallel.ai)rBz( Using Tavily API (https://tavily.com)rCz Using SearXNG (search only): rPrFz- Using Brave Search free tier (search only)rAz2 Using DuckDuckGo via ddgs package (search only)z Using self-hosted Firecrawl: /z# Using direct Firecrawl cloud APIz! Using Firecrawl tool-gateway: z0 Firecrawl backend selected but not configuredu$❌ No web search backend configuredzWSet EXA_API_KEY, PARALLEL_API_KEY, TAVILY_API_KEY, FIRECRAWL_API_KEY, FIRECRAWL_API_URLu;❌ No auxiliary model available for LLM content processingzVSet OPENROUTER_API_KEY, configure Nous Portal, or set OPENAI_BASE_URL + OPENAI_API_KEYuK⚠️ Without an auxiliary model, LLM content processing will be disabledu✅ Auxiliary model available: ru!🛠️ Web tools ready for use!u+🧠 LLM content processing available with z& Default min length for processing: z charsu&🐛 Debug mode ENABLED - Session ID: z Debug logs will be saved to: z/web_tools_debug_z.jsonu=🐛 Debug mode disabled (set WEB_TOOLS_DEBUG=true to enable)z Basic usage:z9 from web_tools import web_search_tool, web_extract_toolz import asyncioz # Search (synchronous)z/ results = web_search_tool('Python tutorials')z # Extract (asynchronous)z async def main():z? content = await web_extract_tool(['https://example.com'])z asyncio.run(main())z LLM-enhanced usage:zC # Content automatically processed for pages >5000 chars (default)zA content = await web_extract_tool(['https://python.org/about/'])z# # Customize processing parametersz# content = await web_extract_tool(z" ['https://docs.python.org'],z, model='google/gemini-3-flash-preview',z min_length=3000z )z # Disable LLM processingzY raw_content = await web_extract_tool(['https://example.com'], use_llm_processing=False)z Debug mode:z # Enable debug loggingz export WEB_TOOLS_DEBUG=truez # Debug logs capture:z$ # - All tool calls with parametersz # - Original API responsesz # - LLM compression metricsz # - Final processed resultsz3 # Logs saved to: ./logs/web_tools_debug_UUID.jsonuL 📝 Run 'python test_web_tools_llm.py' to test LLM processing capabilities)registryrK web_searchaSearch the web for information. Returns up to 5 results by default with titles, URLs, and descriptions. The query is passed through to the configured backend, so operators such as site:domain, filetype:pdf, intitle:word, -term, and "exact phrase" may work when the backend supports them.objectstringzThe search query to look up on the web. You may include backend-supported operators such as site:example.com, filetype:pdf, intitle:word, -term, or "exact phrase".)type descriptionintegerz3Maximum number of results to return. Defaults to 5.r)rrminimummaximumdefaultr6)r propertiesrequired)r&rr7ruExtract content from web page URLs. Returns page content in markdown format. Also works with PDF URLs (arxiv papers, documents, etc.) — pass the PDF link directly and it converts to markdown text. Pages under 5000 chars return full markdown; larger pages are LLM-summarized and capped at ~5000 chars per page. Pages over 2M chars are refused. If a URL fails or times out, use the browser tool to access it instead.arrayrz:List of URLs to extract content from (max 5 URLs per call))ritemsrmaxItemsr;c tt|dd|ddS)Nr3r*r4r2)r4)rEr<argskws r1rrs1txx/D/DDHHU\^_L`L`aaar7u🔍r)r&toolsetschemahandlercheck_fn requires_envemojimax_result_size_charsc tt|dtr|dgddngdS)NrWr2markdown)rxrr<listrs r1rrsP/$.txx/?/?$F$FNRaR  BPZ \ \r7u📄) r&rrrrris_asyncrr)rGr4)rFr*r)r2)vr rMloggingr-rrtypingrrrrrhttpxrErplugins.web.firecrawl.providerr r r r r plugins.web.tavily.providerrrrplugins.web.parallel.providerrrplugins.web.exa.providerrr__annotations__rrrrrrrrtools.debug_helpersrtools.managed_tool_gatewayrr_peek_nous_access_tokenr_read_nous_access_tokenr tools.tool_backend_helpersr!r"r#tools.url_safetyr$r%sys getLoggerrrrwr2r5r6dictr> frozensetrTrVrZr_rar]rirUrrp$DEFAULT_MIN_LENGTH_FOR_SUMMARIZATIONr|rrrrOrrrrrrrrErxrrrrrprint web_availabletool_gateway_availabler.r/firecrawl_key_availablefirecrawl_url_availablenous_availabledefault_summarizer_modelrJrstripexitactive session_idlog_dirtools.registryrrKWEB_SEARCH_SCHEMAWEB_EXTRACT_SCHEMAr#r^r7r1rs###J  ;;;;;;;;;;;;;; $######544444 $(8C='''*.(3-..."&(3-&&&(, ,,,! Xc]!!! -,,,,,  JIIIIIII  8 $ $ SS("3"4""""$ iVVV" "J"J"JKK22S22222j -S - - - -.c....$$$$$$$34> $    049L(,$LcLdLLLL / /(3- /5RUX`adXegkloqtltguIuCv / / / /x} k+< = = = : aa a a a C= a  a c] aaaaP ss ss C=s s  s  sc]sssslY YY C=Y Y  Y c] YYYYxccT2L2L2L2LjAAAAHh%h%3h%sh%3h%h%h%h%Z#: o%o% s)o% o%o% C= o%  o%  o%o%o%o%d d     t     *S *T * * * * ^4 ^ ^ ^ ^t z E ,--- E(OOO&%''M3355"d929-@"#E#E#K#K#M#MNN"d929-@"#E#E#K#K#M#MNN**,,N<<>> ,.. +'++,,, e   E5 6 6 6 6  " " E? @ @ @ @  E< = = = =  ! ! EPZZ 5N5NPP Q Q Q Q  $ $ EA B B B B    EF G G G G $ F EiYRY?R5S5S5Y5Y5[5[5b5bcf5g5gii j j j j $ F E7 8 8 8 8 # F ET6P6P6R6RTT U U U U ED E E E E 4555  2--// 2 2   L KLLL fggg [\\\\ J0HJJKKK   E -...e V