+ Vi@Rt^RIt^RIHtHtHtHtHtHt^RI H t H t H t H t HtHtHt^RIHtHtHtHt^RIHt^RI HtRRltR R ltR RR RR RRRRRRRRRRR/RRlltR.RRlltRR/RRlltR.RRlltRRltR/RRllt R RR RR RRRRRRRRRRRR ^R!R/ R"R#llt!R$R%lt"R RR RR RRRRRRRRR/R&R'llt#R0R(R)llt$R1R*R+llt%R,R-lt&R#)2z4 Batch scraping functionality for Firecrawl v2 API. N)OptionalListCallableDictAnyUnion)BatchScrapeRequestBatchScrapeResponseBatchScrapeJob ScrapeOptionsDocument WebhookConfigPaginationConfig) HttpClienthandle_response_errorvalidate_scrape_optionsprepare_scrape_options)normalize_document_input)CrawlErrorsResponsecrV^8dQhR\\\,,R\\,/#) data_listreturn)rrrr )formats"Y/home/ubuntu/hermes-agent/venv/lib/python3.14/site-packages/firecrawl/v2/methods/batch.py __annotate__rs'Xd3i-@T(^c.pT;'g.F@p\V\4'gK\V4pVP\ R/VB4KB V#)N) isinstancedictrappendr )r documentsdoc normalizeds& r_parse_batch_scrape_documentsr%sN "IB c4 1#6J   X3 3 4 rctV^8dQhR\\\3,R\\\3,/#)rbodyr)rstrr)rs"rrrs*  d38n c3h rcVPR4'g\VPRR44hRVPR4RVPR^4RVPR^4RVPR4RVPR 4R VPR 4R \VPR .4;'g.4/#) successerrorUnknown error occurredstatus completedtotal credits_used creditsUsed expires_at expiresAtnextdata)get Exceptionr%)r's&r#_parse_batch_scrape_status_responser8s 88I  *BCDD $((8$TXXk1-'1%/dhh{+ -dhhvr.B.H.HbI roptionswebhook append_to_idignore_invalid_urlsmax_concurrencyzero_data_retention integrationidempotency_keycV^8dQhR\R\\,R\\,R\\ \\ 3,,R\\,R\\,R\\,R\\,R \\,R \\,R \/ #) rclienturlsr9r:r;r<r=r>r?r@r) rrr(rr rr boolintr )rs"rrr-s77 7 s)7m $ 7 eC./ 0 7 3- 7"$7c]7"$7#7c]77rc \VVVVVVVVR7p VPV 4p VPRWR7p V P'g \ V R4V P 4p V P R4'g\V P RR44h\V P R4V P R 4V P R 4;'gR R 7#) a# Start a batch scrape job for multiple URLs. Args: client: HTTP client instance urls: List of URLs to scrape options: Scraping options Returns: BatchScrapeResponse containing job information Raises: FirecrawlError: If the batch scrape operation fails to start )r9r:r;r<r=r>r?z/v2/batch/scrape)headerszstart batch scraper*r+r,idurl invalidURLsN)rHrI invalid_urls) prepare_batch_scrape_request_prepare_headerspostokrjsonr6r7r )rBrCr9r:r;r<r=r>r?r@ request_datarGresponser's&&$$$$$$$$ rstart_batch_scraperS-s80 !/'/ L%%o6G{{-|{MH ;;;h(<= ==?D 88I  *BCDD  88D> HHUOXXm,44 rc^V^8dQhR\R\R\\,R\/#)rrBjob_idpagination_configr)rr(rrr )rs"rrrgs611 1 1 011 1rc VPRV 24pVP'g \VR4VP4p\ V4pVR,pV'd VP MRpV'd%VR,'d\ VVR,VV4p\VR,VR,VR,VR ,VR ,V'gVR,VR 7#R VR 7#) a7 Get the status of a batch scrape job. Args: client: HTTP client instance job_id: ID of the batch scrape job pagination_config: Optional configuration for pagination behavior Returns: BatchScrapeJob containing job status and data Raises: FirecrawlError: If the status check fails /v2/batch/scrape/zget batch scrape statusr5Tr4r-r.r/r0r2Nr-r.r/r0r2r4r5)r6rOrrPr8 auto_paginate_fetch_all_batch_pagesr )rBrUrVrRr'payloadr"rZs&&& rget_batch_scrape_statusr]gs(zz-fX67H ;;;h(AB ==?D1$7GI8I%33dM*  FO     x +&g^,<($1WV_   8<  rrequest_timeoutc^V^8dQhR\R\R\\,R\/#)rrBnext_urlr^r)rr(rfloatr )rs"rrrs4$$ $$e_ $  $rc VPWR7pVP'g \VR4VP4p\ V4p\ VR,VR,VR,VR,VR,VR,VR ,R 7#) a Fetch a single page of batch scrape results using the provided next URL. Args: client: HTTP client instance next_url: Opaque next URL from a prior batch scrape status response request_timeout: Timeout (in seconds) for the HTTP request Returns: BatchScrapeJob with the page data and next URL (if any) Raises: Exception: If the request fails or returns an error response )timeoutzget batch scrape status pager-r.r/r0r2r4r5rY)r6rOrrPr8r )rBr`r^rRr'r\s&&$ rget_batch_scrape_status_pagerds(zz(z H E  %DNN,cN d . +MMO  >yIL %V,,H&3y>[+H   X & -  "s9~'D  #6* a %  " % s F FFc<V^8dQhR\R\R\/#rrBrUr)rr(rD)rs"rrr s&-- - - -rcVPRV 24pVP'g \VR4VP4pVP R4R8H#)z Cancel a running batch scrape job. Args: client: HTTP client instance job_id: ID of the batch scrape job to cancel Returns: BatchScrapeStatusResponse with updated status Raises: FirecrawlError: If the cancellation fails rXzcancel batch scraper- cancelled)deleterOrrPr6)rBrUrRr's&& rcancel_batch_scraper sQ$}}09:H ;;;h(=> ==?D 88H  ,,rc jV^8dQhR\R\R\R\\,R\/#)rrBrU poll_intervalrcr)rr(rErr )rs"rrr&s>$"$" $" $"$"c] $"  $"rc\P!4p\W4pVPR9dV#V'd4\P!4V, V8d\ RV RV R24h\P !V4Kr)a Wait for a batch scrape job to complete, polling for status updates. Args: client: HTTP client instance job_id: ID of the batch scrape job poll_interval: Seconds between status checks timeout: Maximum seconds to wait (None for no timeout) Returns: BatchScrapeStatusResponse when job completes Raises: FirecrawlError: If the job fails or timeout is reached TimeoutError: If timeout is reached zBatch scrape job z did not complete within z seconds)r.failedr)rorpr]r- TimeoutErrorsleep)rBrUrrcrw status_jobs&&&& rwait_for_batch_completionr&sy,!J ,V<     D D  (:5@!26(:ST[S\\def f =!rrrccV^8dQhR\R\\,R\\,R\\ \\ 3,,R\\,R\\,R\\,R\\,R \\,R \\,R \R \\,R \/ #)rrBrCr9r:r;r<r=r>r?r@rrcr) rrr(rr rr rDrEr )rs"rrrMs33 3 s)3m $ 3 eC./ 0 3 3- 3"$3c]3"$3#3c]33c]33rc \\VVVVVVVVVV R7 p V Pp \W W4#)a Start a batch scrape job and wait for it to complete. Args: client: HTTP client instance urls: List of URLs to scrape options: Scraping options poll_interval: Seconds between status checks timeout: Maximum seconds to wait (None for no timeout) Returns: BatchScrapeStatusResponse when job completes Raises: FirecrawlError: If the batch scrape fails to start or complete TimeoutError: If timeout is reached )r9r:r;r<r=r>r?r@)rSrHr)rBrCr9r:r;r<r=r>r?r@rrcstartrUs&&$$$$$$$$$$ r batch_scraperMsLB  !/'/' EXXF %  rc\V^8dQhR\\,R\\,/#)rrCr)rr()rs"rrrs"d3iDIrcPV'g \R4h.pVFpV'd\V\4'g\RV 24hVPR4'g&VPR4'g\RV 24hVP VP 44K V#)z Validate and normalize a list of URLs for batch scraping. Args: urls: List of URLs to validate Returns: Validated list of URLs Raises: ValueError: If URLs are invalid zURLs list cannot be emptyz Invalid URL: zhttp://zhttps://z)URL must start with http:// or https://: ) ValueErrorrr( startswithr!strip)rCvalidated_urlsrIs& rvalidate_batch_urlsrs 455N*S#..}SE23 3y))S^^J-G-GHNO Ociik* rcVV^8dQhR\\,R\\,R\\\\ 3,,R\\,R\\ ,R\\,R\\ ,R\\,R \/ #) rrCr9r:r;r<r=r>r?r) rr(rr rr rDrEr )rs"rrrs// s)/m $/eC./ 0 / 3- / "$ /c]/"$/#/ /rcd\V4pRV/p V'd%\V4p V 'dV PV 4Ve1\V\4'dW)R&MVP RR7V R&VeW9R&VeWIR&VeWYR&VeWiR&Ve\ V4P 4V R &V #) z Prepare a batch scrape request payload. Args: urls: List of URLs to scrape options: Scraping options Returns: Request payload dictionary rCr:T) exclude_none appendToIdignoreInvalidURLsmaxConcurrencyzeroDataRetentionr?)rrupdaterr( model_dumpr) rCr9r:r;r<r=r>r?rrQ scrape_datas &$$$$$$$ rrLrLs*).N$*N#;L,W5     , gs # #&- #&-&8&8d&8&KL #%1\"&,?()")8%&&,?()&)+&6&<&<&> ]# rc~V^8dQhR\\,R\R\\\,,/#)rrC chunk_sizer)rr(rE)rs"rrrs-T#YC$tCy/rcv.p\^\V4V4FpVPWW1,4K V#)z Split a large list of URLs into smaller chunks for batch processing. Args: urls: List of URLs to chunk chunk_size: Maximum size of each chunk Returns: List of URL chunks )rangertr!)rCrchunksis&& r chunk_urlsrs8F 1c$i , dQ^,-- MrcV^8dQhR\R\\,R\\,R\ R\ R\\ ,R\\ ,/#)rrBrCr9rrrcr)rrr(rr rEr )rs"rrrs_-- - s)-m $- -  - c] - (^-rc \W4p.p^pVFIp \VV VVVR7p V P'dVPV P4V^, pKK V#)a Process a large batch of URLs by splitting into smaller chunks. Args: client: HTTP client instance urls: List of URLs to scrape options: Scraping options chunk_size: Size of each batch chunk poll_interval: Seconds between status checks timeout: Maximum seconds to wait per chunk Returns: List of all scraped documents Raises: FirecrawlError: If any chunk fails )r9rrc)rrr5extend) rBrCr9rrrc url_chunks all_documentscompleted_chunkschunkresults &&&&&& rprocess_large_batchrsl2D-JM  '   ;;;   -A rc<V^8dQhR\R\R\/#r})rr(r)rs"rrrs"--J--@S-rc .VPRV R24pVP'g \VR4VP4pVPRV4pRVPR.4RVPRVPR.44/p\ R/VB#) z Get errors for a batch scrape job. Args: client: HTTP client instance job_id: ID of the batch scrape job Returns: CrawlErrorsResponse with errors and robots-blocked URLs rXz/errorszget batch scrape errorsr5errorsrobots_blocked robotsBlockedr)r6rOrrPr)rBrUrRr'r\r$s&& rget_batch_scrape_errorsrszz-fXW=>H ;;;h(AB ==?Dhhvt$G'++h+'++ow{{CSUW7XYJ  , ,,r)N)rN)d)NrrN)'__doc__rotypingrrrrrrtypesrr r r r r rutilsrrrrutils.normalizerrr%r8rSr]rdr[rrrrrLrrrrrrrs ==gf6' 7(, 7 48 7 #' 7+/7&*7+/7"&7&*7t1h$(, $NDN-:$"N3(, 3 48 3 #' 3+/3&*3+/3"&3&*33"3l