]jozSrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK J r SSK J r SSK JrJrJrJrJr SSKJr \ SSKrSSKJr SS KJr SS KJ r SS K!J"r" SS K#J$r$J%r%J&r& S r'\("S5r)\RT\RV-\RX-r-\RT\RV-r.\/"5r0Sr1Sr2Sr3\3\"RhS'"SS5r5Sr6S Sjr7Sr8Sr9Sr:Sr;"SS\<5r=S!Sjr>"SS\<5r?g!\a \"S5rNf=f!\a SrNf=f)"z/ Reducer using memory mapping for numpy arrays N)mmap)util)HIGHEST_PROTOCOL PicklingErrordumpsloads whichmodule)uuid4) as_strided) make_memmap) delete_folder)resource_tracker)dumploadload_temporary_memmapz/dev/shmgeAcSSKJn [R"SR [ R RU5[ R"555 URUS5 g)Nr )_resource_trackerzj[FINALIZER CALL] object mapping to {} about to be deleted, decrementing the refcount of the file (pid: {})file) 'externals.loky.backend.resource_trackerrrdebugformatospathbasenamegetpid maybe_unlink)filenamers W/home/mande/repo/quber/.venv/lib/python3.13/site-packages/joblib/_memmapping_reducer.py_log_and_unlinkr ;sLJJJ ;;A6 GG  X & <  ""8V4c 8[R"SR[U5[ U5[ R RUR5[ R"555 [R"U[UR5 g)NzD[FINALIZER ADD] adding finalizer to {} (id {}, filename {}, pid {})) rrrtypeidrrrrrweakreffinalizer )memmaps radd_maybe_unlink_finalizerr(Gs^JJNUU L"V*bgg&6&6v&G   V_foo>r!cSn[SUS-5Hn[R"U5 g g![aE [R "SR U55 X!:Xae[R"S5 Mm[a Myf=f)aWrapper around os.unlink with a retry mechanism. The retry mechanism has been implemented primarily to overcome a race condition happening during the finalizer of a np.memmap: when a process holding the last reference to a mmap-backed np.memmap/np.array is about to delete this array (and close the reference), it sends a maybe_unlink request to the resource_tracker. This request can be processed faster than it takes for the last reference of the memmap to be closed, yielding (on Windows) a PermissionError in the resource_tracker loop. r z9[ResourceTracker] tried to unlink {}, got PermissionErrorg?N) rangerunlinkPermissionErrorrrrtimesleepFileNotFoundError)r NUM_RETRIESretry_nos r unlink_filer3PsK![1_-  IIh  . JJKRR  & 3    s1A B ? B  B rc0\rSrSrSrSrSrSrSrSr g) _WeakArrayKeyMapszA variant of weakref.WeakKeyDictionary for unhashable numpy arrays. This datastructure will be used with numpy arrays as obj keys, therefore we do not use the __get__ / __set__ methods to avoid any conflict with the numpy fancy indexing syntax. c0UlgN_dataselfs r__init___WeakArrayKeyMap.__init__{s  r!cdUR[U5up#U"5ULa [U5eU$r8)r:r$KeyError)r<objrefvals rget_WeakArrayKeyMap.get~s1::bg& 5 3-  r!c^^[U5mTRTup4U"5ULa [U5eX24TRT'g![a UU4Sjn[R"X5nN:f=f)Nc >TRT gr8r9)_keyr<s r on_destroy(_WeakArrayKeyMap.set..on_destroysJJsOr!)r$r:r@r%rB)r<rAvaluerBrHrJrIs` @rset_WeakArrayKeyMap.setsrg /ZZ_FCuCsm# * 3 / $++c.C /s&A'A10A1c[S5e)Nz"_WeakArrayKeyMap is not pickleable)rr;s r __getstate___WeakArrayKeyMap.__getstate__s@AAr!r9N) __name__ __module__ __qualname____firstlineno____doc__r=rDrMrP__static_attributes__r!rr5r5ss%$Br!r5ch[USS5nUcg[U[5(aU$[U5$)z@Recursively look up the original np.memmap instance base if any.baseN)getattr isinstancer_get_backing_memmap)abs rr]r]s964 Ay At  #1%%r!c&SnUc [RRSS5nUc[RR [ 5(a[ [S5(a[R"[ 5nURUR-nU[:aa[ n[RRX5n[RR U5(d[R"U5 SnUc[R "5n[RR#[RR%U55n[RRX5nXR4$![[4a SnNf=f)aGet the full path to a subfolder inside the temporary folder. Parameters ---------- pool_folder_name : str Sub-folder name used for the serialization of a pool instance. temp_folder: str, optional Folder to be used by the pool for memmapping large arrays for sharing memory with worker processes. If None, this will try in order: - a folder pointed by the JOBLIB_TEMP_FOLDER environment variable, - /dev/shm if the folder exists and is writable: this is a RAMdisk filesystem available by default on modern Linux distributions, - the default system temporary folder that can be overridden with TMP, TMPDIR or TEMP environment variables, typically /tmp under Unix operating systems. Returns ------- pool_folder : str full path to the temporary folder use_shared_mem : bool whether the temporary folder is written to the system shared memory folder or some other temporary folder. FNJOBLIB_TEMP_FOLDERstatvfsT)renvironrDrexistsSYSTEM_SHARED_MEM_FShasattrrbf_bsizef_bavailSYSTEM_SHARED_MEM_FS_MIN_SIZEjoinmakedirsIOErrorOSErrortempfile gettempdirabspath expanduser)pool_folder_name temp_folderuse_shared_mem shm_statsavailable_nbytes pool_folders r _get_temp_dirrxs*<Njjnn%94@  77>>. / /GB 4J4J #JJ';< #,#4#4y7I7I#I #&CC#7K"$'',,{"MK77>>+66 K0%)N ))+ ''//"''"4"4["ABK'',,{=K  &&W% ##  #s'BE;;FFc[U5SL$)z?Return True if a is backed by some mmap buffer directly or not.N)r])r^s rhas_shareable_memoryrzs q ! --r!c hUS:XaSnUc[UUUUUUUS9$[UUUUUUUS9n [XUS9$)z2Reconstruct an array view on a memory mapped file.zw+zr+)dtypeshapemodeoffsetorderunlink_on_gc_collect)r|r}rr~rr)r}strides)r r ) rr|r~rrr}rtotal_buffer_lenrrZs r_strided_from_memmaprsg t| !5   "!5 $W==r!c t[R"SRUR[R "555 SSKJn U"U5up4U"U5SnX5- nXaR- nURS:aURS(aSnOSnURS(dURS(aS nS n OURnXC- UR-n [ UR"UR$UR&UUURUU S 4 4$![[4a SSK Jn Nf=f) zPickling reduction for memmap backed arrays. a is expected to be an instance of np.ndarray (or np.memmap) m is expected to be an instance of np.memmap on the top of the ``base`` attribute ancestry of a. ``m.base`` should be the real python mmap object. zC[MEMMAP REDUCE] reducing a memmap-backed array (shape, {}, pid: {})r) byte_boundsr F_CONTIGUOUSFC C_CONTIGUOUSNF)rrrr}rrnumpy.lib.array_utilsrModuleNotFoundError ImportErrornumpyrndimflagsritemsizerrr|r~) r^mra_starta_endm_startrrrrs r_reduce_memmap_backedrs  JJMTT GGRYY[  &5!^NG!nQG  F hhF vvzaggn-ww~!''."9))!O :  JJ GG FF   GG      ;  -&%&sDD76D7c[U5n[U[R5(aUR[ ;a [ X5$[[[R"U5[S944$)z5reduce a np.array or a np.memmap from a child processprotocol) r]r\npr'r JOBLIB_MMAPSrrrasarrayr)r^rs rreduce_array_memmap_backwardrWsUAA!RYYAJJl$B%Q**bjjm6FGIJJr!cB\rSrSrSrS Sjr\S5rSrSr Sr g) ArrayMemmapForwardReducerihaReducer callable to dump large arrays to memmap files. Parameters ---------- max_nbytes: int Threshold to trigger memmapping of large arrays to files created a folder. temp_folder_resolver: callable An callable in charge of resolving a temporary folder name where files for backing memmapped arrays are created. mmap_mode: 'r', 'r+' or 'c' Mode for the created memmap datastructure. See the documentation of numpy.memmap for more details. Note: 'w+' is coerced to 'r+' automatically to avoid zeroing the data on unpickling. verbose: int, optional, 0 by default If verbose > 0, memmap creations are logged. If verbose > 1, both memmap creations, reuse and array pickling are logged. prewarm: bool, optional, False by default. Force a read on newly memmapped array to make sure that OS pre-cache it memory. This can be useful to avoid concurrent disk access when the same data array is passed to different worker processes. cXlX lX0l[U5UlUS:Xa*UR R [5(+UlOX`lX`l[5Ul [5Ul X@l g)Nauto) _max_nbytes_temp_folder_resolver _mmap_modeintverbose _temp_folder startswithre_prewarmr5_memmaped_arraysrM_temporary_memmaped_filenames_unlink_on_gc_collect)r< max_nbytestemp_folder_resolver mmap_moderrprewarms rr="ArrayMemmapForwardReducer.__init__si&%9"#7| f  $ 1 1 < <=Q RRDM#M 0 2-0U*%9"r!c"UR5$r8)rr;s rr&ArrayMemmapForwardReducer._temp_folders))++r!cURSURUR4nURURS.n[ X4$)N)rr)rrrrrr)r<argskwargss r __reduce__$ArrayMemmapForwardReducer.__reduce__sD  $9S9ST||}} )$66r!c j[U5nUb*[U[R5(a [ X5$UR R (GdYURGbKURUR:Ga0[R"UR5 [R"UR[5 UR"R%U5n[R8R;URU5nXPR<;nUR<R?U5 UR@(a[BRD"US5 U(a[BRD"US5 [R8RGU5(d[HRJ"SR)URLUR U55 [OX5Hn[R"U[P5 M URR(a![UXPRVS9RY5 OX[HRJ"SR)URLUR [R8R[U555 [\XPRVUR@44$[HRJ"SR)URLUR 55 [^[aU[bS944$![a+nUR[R :waUeSnAGNiSnAff=f![&as SR)[R*"5[-[.R0"55[35R45nUR"R7X5 GNf=f)Nz {}-{}-{}.pklrzP[ARRAY DUMP] Pickling new array (shape={}, dtype={}) creating a new memmap at {})rzN[ARRAY DUMP] Pickling known array (shape={}, dtype={}) reusing memmap file: {}zB[ARRAY DUMP] Pickling array (NO MEMMAPPING) (shape={}, dtype={}).r)2r]r\rr'rr| hasobjectrnbytesrrkrchmodFOLDER_PERMISSIONSrmerrnoEEXISTrrDr@rrr$ threadingcurrent_threadr hexrMrrjraddrrregisterrdrrr}rFILE_PERMISSIONSrrrmaxrrrrr)r<r^rerr is_new_memmapdumped_filenames r__call__"ArrayMemmapForwardReducer.__call__s  " =Z29955(. .!!!  ,4+++  D--.**,>? 70044Q7ww||D$5$5x@H %,N,NNM  . . 2 28 <))!))(F; !))(F;77>>(++ 228&!''82T(,A'8OHH_.>?(9== __=AAC ..4f"''*:*:8*D/&??D,F,FG  JJ$fQWWagg6 E!.>?AB Bi 77ell*G+   7 *00IIKI$<$<$>!?%%))!6 7s,?AK=L5= L2 L--L25A9N21N2)rrrrrrrrN)rT) rRrSrTrUrVr=propertyrrrrWrXr!rrrhs4<:,,,7"dCr!rc Uc [5nUc [5n[b`[UUUUUUS9n X[R'X[R'[ U[R'[ U[R'X4$)zConstruct a pair of memmapping reducer linked to a tmpdir. This function manage the creation and the clean up of the temporary folders underlying the memory maps and should be use to get the reducers necessary to construct joblib pool or executor. )r)dictrrndarrayr'r) forward_reducersbackward_reducersrrrrrrrforward_reduce_ndarrays rget_memmapping_reducersrs"6  F ~";    " (>$&<# )E"**%'C"))$  ..r!cF\rSrSrSrS SjrSrSrSrSr S S jr S r g) TemporaryResourcesManageriBa!Stateful object able to manage temporary folder and pickles It exposes: - a per-context folder name resolving API that memmap-based reducers will rely on to know where to pickle the temporary memmaps - a temporary file/folder management API that internally uses the resource_tracker. NcSUlXlSUl[5Ul[ 5R Ul0UlUc[ 5R nURU5 gr8) _current_temp_folder_temp_folder_root_use_shared_memr_cached_temp_foldersr r_id _finalizersset_current_context)r<temp_folder_root context_ids rr="TemporaryResourcesManager.__init__LsX$(!!1#$(F!7;;   J   ,r!c2XlURU5 gr8)_current_context_idregister_new_context)r<rs rr-TemporaryResourcesManager.set_current_context[s#-  !!*-r!cXR;agSR[R"5URU5n[ X R 5up4URX15 X0RU'g)Nz!joblib_memmapping_folder_{}_{}_{})rrrrrrxrregister_folder_finalizer)r<rnew_folder_namenew_folder_pathrHs rr.TemporaryResourcesManager.register_new_context_sh 22 2 BHH TXXzO"/@V@V!W O  * *? G4C % %j 1r!c4URUR$)z@Return a folder name specific to the currently activated context)rrr;s rresolve_temp_folder_name2TemporaryResourcesManager.resolve_temp_folder_namevs(()A)ABBr!c^^[[S5m[R"TS5 UU4Sjn[R"U5UR U'g)Nrfolderc>[TS/S9RnU"TSS9 [R"TS5 g![a( [ R "SRT55 gf=f)Nr)fromlistTallow_non_emptyrz%Failed to delete temporary folder: {}) __import__rr unregisterrmwarningswarnr)rpool_module_namepool_subfolders r_cleanupETemporaryResourcesManager.register_folder_finalizer.._cleanupsg' O+<m  ndC ++NHE  ;BB>R s7/A)(A))r rrratexitr)r<rrrrs ` @rr3TemporaryResourcesManager.register_folder_finalizer|sB '}oF!!.(; ((.x'@$r!c"Uc-[UR5HnURXUS9 M gURRU5nU(Ga,[R R U5(Ga[R"U5HtnU(a6[R"[R RXE5S5 M@[R"[R RXE5S5 Mv X2-n[XCS9 URRUS5 [R"US5 URRUS5nUb[R"U5 gggg![ a gf=f)z9Clean temporary resources created by a process-based poolN)forcerrrr)listr_clean_temporary_resourcesrDrrrdlistdirrrrjrrpoprrrm)r<rrrrsr finalizers rr4TemporaryResourcesManager._clean_temporary_resourcessM  #4#<#<= //_0> 3377 CKrww~~k:: " ; 7H)33GGLL?)55GGLL?!8 (!+O--11*dC$// XF!% 0 0 4 4Z FI ,)))4-9 ;{> s A1F F F)rrrrrrr)NN)NFF) rRrSrTrUrVr=rrrrrrWrXr!rrrBs/ -.D.C A@=B0r!rr8)NNNg.ArrFT)@rVrrrstatrnrr.rr%rmultiprocessingrpicklerrrrr uuidr WindowsError NameErrorr#rrnumpy.lib.stride_tricksr r backportsr diskrexternals.loky.backendr numpy_picklerrrrerriS_IRUSRS_IWUSRS_IXUSRrrrMrr r(r3_CLEANUP_FUNCSr5r]rxrzrrrobjectrrrrXr!rrsi   MM2#4;;"!$C \\DLL04<<?<<$,,. u  5?@+6'&B&BZ&"8'v. '>T9xK"hChCX ,/^IIU:L  Bs$D D/D,+D,/D:9D: