ڰjK7SrSSKrSSKrSSKrSSKrSSKJrJrJr SSK J r J r J r SSK JrJrJrJr SSKJrJrJr SS KJrJr \"5rS rS \S \4S jr"SS5rg)zh runpod | serverless | rp_scale.py Provides the functionality for scaling the runpod serverless worker. N)AnyDictSet)AsyncClientSession ClientSessionTooManyRequests) _job_stop_urlget_jobget_stop_signals handle_job) RunPodLogger_reset_batch_id _set_batch_id) JobsProgress IS_LOCAL_TESTcb[R"XU5n[RSU35 g)NzUncaught exception | ) tracebackformat_exceptionlogerror)exc_type exc_value exc_tracebackexcs _/home/mande/repo/quber/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_scale.py_handle_uncaught_exceptionrs(  $ $X- HCII%cU+,current_concurrencyreturncU$)z Default concurrency modifier. This function returns the current concurrency without any modification. Args: current_concurrency (int): The current concurrency. Returns: int: The current concurrency. )r s r_default_concurrency_modifierr$s  rc\rSrSrSrS\\\44SjrSr Sr Sr Sr S r S rS \4S jrS \4SjrS \4SjrS \4SjrS\S \4SjrS \S\4SjrSrg) JobScaler(zV Job Scaler. This class is responsible for scaling the number of concurrent requests. configc[R"5UlSUlXl[ 5Ul0Ul[Ul SUl [R"URS9Ul [Ul[UlSUl[$UlUR)S5=n(aX l[*(dgURR)S5=n(aX0lURR)S5=n(aX@lURR)S5=n(aXPlURR)S5=n(aX`l URR)S 5=n(aXpl gg) Nr Zmaxsizeconcurrency_modifier jobs_fetcherjobs_fetcher_timeout jobs_handlerstop_signals_fetcherstop_signals_fetcher_timeout)asyncioEvent_shutdown_eventr r(r job_progress jobs_tasksr r1r2Queue jobs_queuer$r-r r.r/rr0getr)selfr(r-r.r/r0r1r2s r__init__JobScaler.__init__-s<&}}#$  (N46$4!,.)!--0H0HI$A!#$&!&#)::.D#E E  E(< %} ;;??>: :< : , #';;??3I#J J  J(< %;;??>: :< : , #';;??3I#J J  J(< %+/;;??;Y+Z Z ' Z0L - [rc# URUR5UlUR(a%URURR:XagUR 5S:a [ R "S5IShvN M4[ R"URS9Ul[RSUR35 gNL7f)Nrr r+z.JobScaler.set_scale | New concurrency set to: ) r-r r9r,current_occupancyr3sleepr8rdebugr;s r set_scaleJobScaler.set_scaleWs#'#<#*> ? MM&--)=)= >  DHHJ  N HHL M NsABB65B6cV[RSUS35 UR5 g)a[ Called when the worker is signalled to shut down. This function is called when the worker receives a signal to shut down, such as SIGTERM or SIGINT. It sets the shutdown event, which will cause the worker to exit its main loop and shut down gracefully. Args: signum: The signal number that was received. frame: The current stack frame. zReceived shutdown signal: .N)rrA kill_worker)r;signumframes rrJJobScaler.handle_shutdown{s& .vha89 rc# [5IShvN n[R"URU55n[R"UR U55n[R"UR U55nX#U/n[R "U6IShvN SSS5IShvN gNNN !,IShvN (df  g=f7f)N)rr3 create_taskget_jobsrun_jobsmonitor_stop_signalsgather)r;session jobtake_task jobrun_task jobstop_tasktaskss rrN JobScaler.runs%''7"..t}}W/EFL!--dmmG.DEK"..t/H/H/QRL! =E..%( ( (('' )('''sWCB5CB B;B7 B;$ C/B90C7B;9C;CC  CCc@URR5(+$)z, Return whether the worker is alive or not. )r5is_setrBs ris_aliveJobScaler.is_alives''..000rcb[RS5 URR5 g)z Whether to kill the worker. z Kill worker.N)rrAr5setrBs rrSJobScaler.kill_workers" .!   "rr!cURR5nURR5n[R SUR SUSU35 X!-$)Nz JobScaler.status | concurrency: z ; queue: z ; progress: )r9qsizer6 get_job_countrrAr )r;current_queue_countcurrent_progress_counts rr?JobScaler.current_occupancysn"oo335!%!2!2!@!@!B .t/G/G.H ReQffrtJsK L &;;rr]c :# UR5(GaUR5IShvN URUR5- nUS::a5[R S5 [ R"S5IShvN M[R S5 [ R"URX5URS9IShvN nU(d7[R S5 [ R"S5IShvN GMUHZnURRU5IShvN URRU5 [R SUS 5 M\ [RS URR!535 [ R"S5IShvN UR5(aGMggGNGN\GNNN!["a7 [R S 5 [ R"S 5IShvN N[ R$a [R S 5 e[ R&a [R S5 N[(a#n[R SUS35 SnANSnAf[*aBn[R-S[/U5R0S[3U535 SnAGNISnAff=fGN8![ R"S5IShvN f=f7f)z Retrieve multiple jobs from the server in batches using blocking requests. Runs the block in an infinite loop while the worker is alive. Adds jobs to the JobsQueue Nrz2JobScaler.get_jobs | Queue is full. Retrying soon.r z.JobScaler.get_jobs | Starting job acquisition.timeoutz&JobScaler.get_jobs | No jobs acquired.z Job QueuedidzJobs in queue: z?JobScaler.get_jobs | Too many requests. Debounce for 5 seconds.z+JobScaler.get_jobs | Request was cancelled.z9JobScaler.get_jobs | Job acquisition timed out. Retrying.z'JobScaler.get_jobs | Unexpected error: rRz!Failed to get job. | Error Type:  | Error Message: )rerCr r?rrAr3r@wait_forr.r/r9putr6addinforkr CancelledError TimeoutError TypeError Exceptionrtype__name__str)r;r] jobs_needed acquired_jobsjobrs rrYJobScaler.get_jobss[mmoo.." " "22T5K5K5MMKa NOmmA&&&& ' JK'.&6&6%%g; 55'! %IIFG6mmA&&&3)C//--c222%%))#.IIlCI6) ?4??+@+@+B*CDE(mmA&&&_mmoo " '!D'13 # ' UmmA&&&))  GH'' W UV N CE7!LMM  7U 8L8L7MM_`cdi`j_kl  'gmmA&&&s*LGALGL AG(G! G(2L G$ L#G(6G&7A,G(#L<K4=LLL!G($L&G((8K1 H#!K1&K7(AK19K7; K1J"K7" K1/7K,&K7,K11K74L7LLLLc# [5nSnUR5(d URR5(Gd[ U5UR :aURR5(dURR 5IShvN n[R"URX55nURU5 XPRUS'[ U5UR :a!URR5(dMU(aj[ U5nXc:wa[RSU35 Un[R"U[RSS9IShvN upxUR!U5 O[R""S5IShvN UR5(aGMURR5(dGM[R$"USS06IShvN n U HSn ['U [(5(dM['U [R*5(aM;[R-S U 35 MU gGNNNNe7f) z Retrieve jobs from the jobs queue and process them concurrently. Runs the block in an infinite loop while the worker is alive or jobs queue is not empty. rNrszJobs in progress: g?) return_whenrrreturn_exceptionsTz8JobScaler.run_jobs | Task failed during shutdown drain: )rhrer9emptylenr r:r3rXrrxr7rrywaitFIRST_COMPLETEDdifference_updater@r\ isinstancer}rzr) r;r]ralast_task_countrtaskcurrent_task_countdonependingresultsresults rrZJobScaler.run_jobss $'5mmooT__%:%:%<%<e*t777@U@U@W@W OO//11**4??7+HI $-1D * e*t777@U@U@W@W%(Z"%8HH12D1EFG&8O&-llw'>'>'!  ''-mmC(((3mmooT__%:%:%<%<> FFFF&),,ZH^H^5_5_ TU[T\]^;2!) GshBI)I BI)AI)7I#81I))I%*I)I)(I)I'I)"I)I)#I)%I)'I)c 6# UR[La![5c[R S5 gUR 5(a[ R"URU5URS9IShvN nUHnURU5IShvN M U(d[ R"S5IShvN [ R"S 5IShvN UR 5(aMggNNfN?![a" [ R"S5IShvN Nh[ Ra [RS5 e[ Ra [RS5 N[aan[R!S[#U5R$S ['U535 [ R"S5IShvN SnAGN SnAff=fGN![ R"S 5IShvN f=f7f) a0 Long-polls the dedicated stop channel and stops signalled jobs. Runs in an infinite loop while the worker is alive. The Runpod server signals a request to be stopped (for example when it is cancelled or times out) and this loop stops just that in-progress job, leaving the worker's other jobs running. NzJobScaler.monitor_stop_signals | Stop channel could not be derived from the job-take URL; per-job stop is disabled for this worker.rqr rtz7JobScaler.monitor_stop_signals | Request was cancelled.z?JobScaler.monitor_stop_signals | Stop poll timed out. Retrying.z-JobScaler.monitor_stop_signals | Error Type: rur)r1r r rrMrer3rvr2stop_jobr@r rzrAr{r}rr~rr)r;r]job_idsjob_idrs rr[JobScaler.monitor_stop_signalss  $ $(8 8]_=T HHS  mmoo '!( 0 0--g6 ==!&F--///&"--***mmA&&&9mmoo 0 +" 'mmA&&&))  ST'' ] [\ ' CDKDXDXCYYklopulvkwxmmA&&&  ' 'gmmA&&&sA H 1D>D?DD(DDD H#G2$H?HDDD#G/*D-+G/0G52AG/G5 G/AG*G!G*$G5*G//G52H5HHHHrc# URRU5nUc[RSUS35 g[R SU5 UR 5 g7f)z Stop a single in-progress job by cancelling its running task. Args: job_id: The id of the job to stop. Returns: True if a matching in-progress job was found and stopped, False otherwise. z,JobScaler.stop_job | No in-progress job for rRFz Stopping job.T)r7r:rrArycancel)r;rrs rrJobScaler.stop_jobAsS""6* < IIDVHAN O &) sA A"rc# [URS55n[RSUS5 UR XR U5IShvN UR RSS5(aUR 5 URR5 URRU5 UR R#USS5 [RS US5 [%U5 gN![Ra [RSUS5 e[a"n[RSU3US5 eSnAff=f!URR5 URRU5 UR R#USS5 [RS US5 [%U5 f=f7f) zQ Process an individual job. This function is run concurrently for multiple jobs. batchIdz Handling JobrsNrefresh_workerFz Job stopped.zError handling job: z Finished Job)rr:rrAr0r(rSr3rzryr}rr9 task_doner6remover7popr)r;r]rbatch_id_tokenerrs rrJobScaler.handle_jobUsx'swwy'9: , IInc$i 0##G[[#> > >{{/77  " OO % % '    $ $S ) OO  D 4 0 IInc$i 0 N +- ? %%  HH^SY /   II,SE2CI >   OO % % '    $ $S ) OO  D 4 0 IInc$i 0 N +sHG 8DD5D A9G D7E >EE  E##A:GG ) r5r-r(r r6r.r/r0r9r7r1r2N)r __module__ __qualname____firstlineno____doc__rrrr<rCrOrJrNrerSintr?rrYrZr[boolrdictr__static_attributes__r#rrr&r&(s(MtCH~(MT " &  )1 #<3<7'm7'r+_m+_Z,'-,'\ST(, ,D,rr&)rr3rHrFrtypingrrr http_clientrrr rp_jobr r r r rp_loggerrrr worker_staterrrrrr$r&r#rrrs^  !!MMHHCC5n- s s K,K,r