ڰj)%SrSSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSK J r SSK J r \ "5r SSSjjr/rS\S 'SS jrSS jrS S S .rS\S'SSjrSrSSjrSSjrSSjrSSjrg)am Fitness check system for worker startup validation. Fitness checks run before handler initialization on the actual RunPod serverless platform to validate the worker environment. Any check failure force-kills the worker via os._exit(1), signaling unhealthy state to the container orchestrator. Fitness checks do NOT run in local development mode or testing mode. ) annotationsN)Callable) RunPodLoggerc[R[R4H6n[R"[ 5 UR 5 SSS5 M8 [R"U5 g!,(df  M`=f)a Force-kill the worker after a fitness check failure. Uses os._exit rather than sys.exit because a fitness failure means the environment is broken and the worker must die immediately so the orchestrator can restart it. sys.exit only raises SystemExit, which triggers cooperative interpreter shutdown and blocks joining non-daemon threads. Workers routinely have such threads alive by the time checks run (e.g. vLLM's AsyncLLMEngine, constructed at import before the checks), so sys.exit can hang forever and the worker keeps serving jobs. os._exit bypasses thread joins, atexit handlers, and asyncgen cleanup. Args: code: Process exit code (default 1, signaling unhealthy). N) sysstdoutstderr contextlibsuppress Exceptionflushos_exit)codestreams a/home/mande/repo/quber/.venv/lib/python3.13/site-packages/runpod/serverless/modules/rp_fitness.py_terminate_unhealthyrsP&::szz*   + LLN, ++HHTN, +s A22 B zlist[Callable]_fitness_checksct[RU5 [RSUR35 U$)ab Decorator to register a fitness check function. Fitness checks validate worker health at startup before handler initialization. If any check fails, the worker is force-killed with os._exit(1). Supports both sync and async functions (auto-detected via inspect.iscoroutinefunction()). Example: @runpod.serverless.register_fitness_check def check_gpu(): import torch if not torch.cuda.is_available(): raise RuntimeError("GPU not available") @runpod.serverless.register_fitness_check async def check_model_files(): import aiofiles.os if not await aiofiles.os.path.exists("/models/model.safetensors"): raise RuntimeError("Model file not found") Args: func: Function to register as fitness check. Can be sync or async. Returns: Original function unchanged (allows decorator stacking). zRegistered fitness check: )rappendlogdebug__name__)funcs rregister_fitness_checkr6s/84 II*4==/:; Kc,[R5 g)z Clear all registered fitness checks. Used primarily for testing to reset global state between test cases. Not intended for production use. N)rclearrrclear_fitness_checksr!WsrF) gpu_check system_checkszdict[str, bool]_registration_statec(S[S'S[S'g)zY Reset global registration state. Used for testing to ensure clean state between tests. Fr"r#N)r$r rr_reset_registration_stater&gs (- $+0(rc[RRS5n[RRS5nU(a US:XdU(dgSSKJn SSKJn SSKJn URS U5nS UUSS US .nU"5nURRS U05 URX'[S9 UR5 g!UR5 f=f![a gf=f)u< Best-effort report of a fitness-check failure to the host before exit. Sends a single GET to the ping URL (same URL/credentials the heartbeat uses) with status=unhealthy plus the failing check name and reason, so the host can emit a queryable worker.fitness_failed event. Any failure — no ping URL, no API key, HTTP error, timeout — is swallowed, so this can never prevent the os._exit that follows. It is synchronous, so it may delay that exit by up to _REPORT_TIMEOUT_SECONDS (network phases only; it adds no delay when there is no ping URL/API key to report to). RUNPOD_WEBHOOK_PINGRUNPOD_AI_API_KEY PING_NOT_SETNr)SyncClientSession) WORKER_ID) __version__z$RUNPOD_POD_ID unhealthy)statuscheckreasonrunpod_version Authorization)paramstimeout)renvirongetrunpod.http_clientr,&runpod.serverless.modules.worker_stater-runpod.versionr.replaceheadersupdate_REPORT_TIMEOUT_SECONDScloser ) r2r3ping_urlapi_keyr,r-r4r6sessions r_report_unhealthyrEuszz~~34Hjjnn01G x>1 8D@##$4i@!Tcl,   $%  OO " "OW#= > KK9PK Q MMOGMMO    s*5C" 1C <C" CC"" C/.C/c[S(agS[S'SSKJn U"5 g![a [R S5 gf=f)z Ensure GPU fitness check is registered. Deferred until first run to avoid circular import issues during module initialization. Called from run_fitness_checks() on first invocation. r"NTrauto_register_gpu_checkz>GPU fitness check module not found, skipping auto-registration)r$rp_gpu_fitnessrH ImportErrorrrrGs r_ensure_gpu_check_registeredrKsG;''+ $T;! T RSTs (A  A c<SSKn[S(agURRSS5R 5S:Xa[ R S5 S[S'gS[S'S S KJn U"5 g![a [ R S 5 gf=f) z Ensure system resource fitness checks are registered. Deferred until first run to avoid circular import issues during module initialization. Called from run_fitness_checks() on first invocation. rNr#RUNPOD_SKIP_AUTO_SYSTEM_CHECKStruezOSystem fitness checks disabled via environment (RUNPOD_SKIP_AUTO_SYSTEM_CHECKS)Tr)auto_register_system_checkszASystem fitness check module not found, skipping auto-registration) rr$r8r9lowerrrrp_system_fitnessrPrJ)rrPs r _ensure_system_checks_registeredrSs?+ zz~~6;AACvM ] 04O,+/(WB#% W UVWs+ A99BBc H# [5 [5 [(d[R S5 g[R S[ [5S35 [R"5n[HnURn[R SU35 [R"5n[R"U5(aU"5IShvN OU"5 [R"5U- S-n[R SUSUS S 35 M [R"5U- S-n [R SU S S 35 gN~![an[U5Rn[U5n[R "5n[R#S US US U35 [R SU35 [%X&S U35 O![a Of=f[R#S5 ['S5 SnAGMSnAff=f7f)ai Execute all registered fitness checks sequentially at startup. Execution flow: 1. Auto-register GPU check on first run (deferred to avoid circular imports) 2. Check if registry is empty (early return if no checks) 3. Log start of fitness check phase 4. For each registered check: - Auto-detect sync vs async using inspect.iscoroutinefunction() - Execute check with timing instrumentation (await if async, call if sync) - Log success or failure with check name and execution time 5. On any exception: - Log detailed error with check name, exception type, and message - Log traceback at DEBUG level - Force-kill the worker via os._exit(1) immediately (fail-fast). This is a hard exit, not a cooperative sys.exit/SystemExit: it does not unwind the stack or run cleanup, so callers cannot catch it and it cannot be blocked by live non-daemon threads. 6. On successful completion of all checks: - Log completion message with total execution time Note: Checks run in registration order (list preserves order). Sequential execution (not parallel) ensures clear error reporting and handles checks with dependencies correctly. Timing uses high-precision perf_counter for accurate measurements. Note: A failing check terminates the process via os._exit(1); this function does not return in that case and does not raise SystemExit. z'No fitness checks registered, skipping.NzRunning z fitness check(s)...zExecuting fitness check: izFitness check passed: z (z.2fzms)zFitness check failed: z | z: z Traceback: zWorker is unhealthy, exiting.rzAll fitness checks passed. ()rKrSrrrinfolentime perf_counterrinspectiscoroutinefunctionr typestr traceback format_excerrorrEr) total_start_time check_func check_namecheck_start_timecheck_elapsed_msexc error_type error_messagefull_tracebacktotal_elapsed_mss rrun_fitness_checksrjsD!"%& ? ;<HHxO,--A BC((*% (( # $ II1*> ?#002 **:66 l"" $ 1 1 36F F$N  II.zl"=Mc>$FHH+,rss# $#n2#%$B(_ 1& RT(W