Ei\ \1\/4r?\.\?S4r@\0\1\@4rA\0\1\ 4rB\\?\ 4rCS#\BS8\\@\ /S4S S4S9jrDS#\BS \.\B\A44S:jrES;\BS<\@S=\ S S4S>jrFS#\BS?\AS \B4S@jrGg)FN)CallableMappingMutableMapping)Anycast NamedTuple TYPE_CHECKINGUnion)AsyncCollectiveTensor)distributed_c10d) ShardedTensor)distribute_tensorDTensor Replicate)%compute_local_shape_and_global_offsetobjpgdevice companion_objreturncU$Nrrrrs `/home/mande/repo/quber/.venv/lib/python3.13/site-packages/torch/distributed/_state_dict_utils.py_identity_funcrs Jsharded_tensorr cxUc[R"5n[R"U5nUR 5nUR 5SnUR 5R 5n[R"XS- 5U-U-nUc[R"U5OUnU(aUSRR5n U RRUR:waU RU5n XyR 5- n U S:a[R "U SU /5n O["R$"XpR&US9n ["R("Xs-U R&US9n [R*"XUS9 U R-SSU5R/UR 55n U $)Nr)dtyper)group)r _get_default_groupdistget_world_size local_shardssizenumelmathceil_get_pg_default_devicetensorflattenrtypetoFpadtorchzerosr emptyall_gather_into_tensornarrowreshape) rrr world_sizeshards dim_0_size tensor_numel chunk_size pg_device local_tensor num_paddingr+s r_all_gather_sharded_tensorr? s}  z  0 0 2$$R(J  ( ( *F$$&q)J!&&(..0L:23lBjPJ7=~//36ay''//1    # #y~~ 5'??95L #5#5#77 ?55;/?@L{{ 229 [[  F  B? ]]1a . 6 6~7J7J7L MF Mrc\rSrSrSrg)CompanionMismatchFrN)__name__ __module__ __qualname____firstlineno____static_attributes__rrrrArAFsrrAFrTrr cpu_offloadr ranks_only type_check non_blocking iter_objectsharded_tensor_func dtensor_func tensor_funcrIrJ.rKrLc:[R"S5n [U[5(a U"XXW5n GOd[U[5(a U"XXW5n GOD[U[R 5(a U"XXW5n GO[U[ [[[[R45(dUcUn GO[U[5(aUb[U[5(a4[UR55[UR55:waZ[U[5(aSO7S[UR55<S[UR55<3n [U 5eUR!5VVs0sH upU[#UUUUUUUUbX~OSUU U S9 _M" n nnO[U[$[&45(aUb9[U[$[&45(a[)U5[)U5:wa[e[+U5VVs/sH!unn[#UUUUUUUUbUUOSUU U S9 PM# n nn[U[&5(a ['U 5n O5U (d[,R."U5n O[1S[3U535eU(a6"U5U;Ga9[U [R 5(GaU(aUcU R9U 5n Ub[U[5(aE[U [5(d [;S5eUR<R?U R<U S 9 O[U[5(ay[U [5(d [;S 5e[+URA55H;unnURBR?U RA5URBU S 9 M= OUR?XS 9 Un U $[U [5(a0OSn U $s snnfs snnf) aIterate through the state dict, applying the given functions to each tensor type. Args: iter_object (Any): the target state_dict. sharded_tensor_func (Callable): the function to apply to ShardedTensor dtensor_func (Callable): the function to apply to DTensor tensor_func (Callable): the function to apply to Tensor pg (Optional[dist.ProcessGroup]): process group passed to tensor functions device (Optional[torch.device]): device passed to tensor functions cpu_offload (bool): whether to offload the tensors to CPU memory. This option is ignored if a companion_obj is supplied. companion_obj (Any): A companion object to the state dict. If this object is supplied, we attempt to copy the tensor to the companion object. ranks_only (Tuple[int, ...]): if this tuple is empty, all ranks will have the same state_dicts. Otherwise only ranks that in ``ranks_only`` have the same state_dicts. Other ranks will get empty state_dicts. type_check (bool): check if the instance data type is a supported type that can be saved by DCP. The current supported data types are torch.Tensor, DTensor, int, float, str, list, dict, None. non_blocking (bool): whether to use non-blocking copy when copying to the companion object. cpuNzset(companion_obj.keys())=z set(iter_object.keys())=rHzUnexpected value type z5ret must be a DTensor when companion_obj is a DTensor)rLzAret must be a ShardedTensor when companion_obj is a ShardedTensor)"r1r isinstancer rTensorintfloatstrbytesioBytesIOdictsetkeysrAitems_iterate_state_dictlisttuplelen enumeratecopydeepcopy ValueErrorr-r#get_rankr.AssertionError _local_tensorcopy_r%r+)rMrNrOrPrrrIrrJrKrL cpu_deviceretmsgkeyvalueidxvshards rr`r`JsHe$J+}--!+6I K ) );FB K . .+6A;eS% DEE   K & &  $=$//=%%'(C 0@0@0B,CCmT222M..0133MS9I9I9K5L4NO  $C( ( *//1 2  $#'4A4Mm0SW%%)  2   K$ / /  $=4-88=!S%55# # $K0 1Q #'4A4MmC0SW%%) 1  k5 ) )*C mmK(1${2C1DEFF r*j8 c5<< ( (}4ffZ((mW55%c733,S"//55)) 6 }==%c=99,_'0 0J0J0L&M U **,,.s3::+'N "'''G# JsD))bt JY . s 'P(PrrrIrJrK state_dictc 8SnSn[UUU[UUUUUS9 $)a Given a state_dict, this API gathers all the ShardedTensors or DTensors in the state_dict. Args: state_dict (Dict[str, Any]): the target sharded state_dict. pg (Optional[dist.ProcessGroup]): the process group that is used to gather ShardedTensor. Note that gathering a DTensor will use the DeviceMesh. So this argument will be ignored when gathering a DTensor. device: (Optional[torch.device]): the device that is used to perform allgather for ShardedTensor. Note that gathering a DTensor will use the DeviceMesh. So this argument will be ignored when gathering a DTensor. cpu_offload (bool): whether to offload the tensors to CPU memory. The default value is False. ranks_only: (Tuple[int, ...]): if this tuple is empty, all ranks will have the same state_dicts. Otherwise only ranks that in ``ranks_only`` have the same state_dicts. Other ranks will get empty state_dicts. type_check: (bool): check if the instance data type is a supported type that can be saved by DCP. The current supported data types are torch.Tensor, DTensor, int, float, str, list, dict, None. Returns: The gathered state dictionary. c[R"S5n[XU5nUR5(a'UR5SRROUnURU:waUR U5nU$UnU$)NrRr)r1rr?r%r+r.)rprrrrl output_tensorlocal_shard_devices rrN/_gather_state_dict..sharded_tensor_funcs\\%( 25fE !!##     # * * 1 1    #5 5!$$%78E "E rcURURR:wa%URURR5nURVs/sH n[ 5PM nnUR URUS9nUR5n[U[5(aUR5nU$s snf)N) device_mesh placements) rr| device_typer.r}r redistributeto_localrTr wait)rprrr_r}s rrO(_gather_state_dict..dtensor_func s <<5,,88 8HHU..::;E ,1+;+;<+;aik+; <""))!#   e2 3 3JJLE =sB?rtr`r)rurrrIrJrKrNrOs r_gather_state_dictrs7J"*    r)rJrKc @[U[[[SSSUUS9 nU$)aS Given a state_dict, this API offload all the tensors to CPU memory. Args: state_dict (Dict[str, Any]): the target state_dict. pg (Optional[dist.ProcessGroup]): the process group that is used to gather ShardedTensor. Note that gathering a DTensor will use the DeviceMesh. So this argument will be ignored when gathering a DTensor. ranks_only: (Tuple[int, ...]): if this tuple is empty, all ranks will have the same state_dicts. Otherwise only ranks that in ``ranks_only`` have the same state_dicts. Other ranks will get empty state_dicts. type_check: (bool): check if the instance data type is a supported type that can be saved by DCP. The current supported data types are torch.Tensor, DTensor, int, float, str, list, dict, None. Returns: The gathered state dictionary. NTrtr)rurJrKrms r_offload_state_dict_to_cpur-s04   C Jrcopy_state_dictc@[U[[[SSSSUUUS9 $)aO Copies all tensors in a given state dict into a different state_dict with the same structure. Additionally, a copied state dict with the same value references is returned. Editing the keys on this state dict will not affect the passed in copy_state_dict (but the value references are the same). .. warning:: It is expected by this function that state_dict and copy_state_dict share the same structure and data types. .. warning:: The current supported data types are torch.Tensor, DTensor, int, float, str, list, dict, None. Args: state_dict (Dict[str, Any]): the target state_dict. copy_state_dict (Dict[str, Any]): The state dict we are copying into. This state_dict must have exactly the same structure as the source `state_dict`. non_blocking: (bool): Whether copy ops should be performed asynchronously type_check (bool): check if the instance data type is a supported type that can be saved by DCP. The current supported data types are torch.Tensor, DTensor, int, float, str, list, dict, None. Returns: State Dict copy NFr)rrrIrJrrKrLr)rurrLrKs r_copy_state_dictrUs3F  %!  r pin_memory share_memoryc ^^^S[RS[RS-S[RS-S[ S[R4 UU4SjjmS[ S[RS-S[RS-S[ S[ 4 U4SjjnS[S[RS-S[RS-S[ S[4 U4S jjn[UUUTSSS S S S 9 nU$) a Given a state_dict, create another state_dict with the same structure and elements. However, all tensors in the returned state_dict are new tensors on CPU. These tensors can be placed on pin_memory or share_memory based on the provided arguments. .. warning:: Setting both `pin_memory` and `share_memory` to True significantly increases the latency of this method because of the nuances which require us to register memory as pinned directly as opposed to relying on the pin_memory cache allocator. This option should only be used for long lived tensors which are required to be shared. This is not the case as long as at least one of `pin_memory` or `share_memory` is set to False. rrNrrrc>[UR55S:Xa[R"SURS9$UR 5S:XdUR 5S:Xa.[R"USS9nT(aUR5nU$T(a[R"[UR55SUR06nUR5nT(ax[R"UR 5UR 5UR5-5 [R"U[R UR 55 U$T(aE[R"[UR55SUR06R5$[R"[UR55SUR06$)Nr)r rRrr )rcr&r1r+r r'data_ptr zeros_like share_memory_r3rbpin_memory_utilsr element_sizeweakreffinalize unpin_memory)rrrrtrrs rrP+_create_cpu_state_dict..tensor_funcsA sxxz?a <<3 3 99;! s||~2  U3AOO%H  U388:.@cii@A!A ++AJJL!'')annFV:VW  $4$A$A1::<PH ;;chhj 1CCNNP P;;chhj 1CC Crc(>[UR55S:XaU$UR[R"S5:wa[ [ UR SS95nO[R"U5nT"URXS5Ul U$)NrrRr) rcr&rr1rrr.rerfrj)rrrrrmrPs rrO,_create_cpu_state_dict..dtensor_funcsr sxxz?a J ::e, ,we 45C--$C'(9(92tL rc*>UR5(dU$UR[R"S5:waURSS9nO[R "U5nUR5HnT"UR XS5UlM U$)NrRr)r%rr1r.rerfr+)rrrrrmr8rPs rrN3_create_cpu_state_dict..sharded_tensor_funcsz !!J ::e, ,&&&&C--$C&&(F' r4HFM) rFrrt) r1rUr# ProcessGrouprrrr r`)rurrrOrNrmrPs `` @r_create_cpu_state_dictrs&D \\D    $D t#D  D  DD>     $ t#          $ t#    &   C Jrcompared_state_dictc S[RS[RS-S[RS-S[ S[R4 Sjn[ U[[USSSS USS 9 g ![a gf=f) a Given two state_dicts, check if the structures are the same. And if a [key, tensor] pair exist in one state_dict there must be the a corresponding pait, [key, other_tensor], in the other state_dict, where tensor and other_tensor have the same size and dtype. Return the check result. rrNrrrcURUR:wd"UR5UR5:wa[eU$r)r r&rArs rrP1_check_state_dict_similarity..tensor_funcs7   #)) +}/A/A/Csxxz/Q# # rFr)rrrIrJrrKT) r1rUr#rrrr`rrA)rurrPs r_check_state_dict_similarityrs \\    $ t#       -   sA.. A;:A;cR\rSrSr%\R \S'\R\S'Srg) _TensorInfoir&r rN) rCrDrErFr1Size__annotations__r rGrrrrrs ** ;;rrfull_state_dictlocal_state_dictr^cDUc[RR5nURURVs1sHoURiM sn;aUOURSn/nUHn[R "5S:XaNXn[ U[R5(d [S5eUR5RU5n O.Xn [R"U RUU RS9n URU 5 UR!U5=n cM[ U ["5(aX4OU X'M [%U5S:a[R&"XFSS5 O[R("USSUS9 XS:wa[+X&5HqupyUR!U5=n cM[ U [,5(a.[ U S["5(aU SU RU54OU RU5X'Ms [/XX45 gs snf)Nrz!full_state must be a torch.Tensor)r&rr isrcr!)r#r r"r- _device_typesrhrTr1rUridetachr.r3r&r appendgetrrc_broadcast_coalesced broadcastziprb_distribute_tensors) rrr^rrr<tensorsro full_state full_tensor tensor_info local_states r_broadcast_tensorsrs z  " " 5 5 7 ;;2;K;KL;Ki>>;KL L    a  #%G ==?a (-Jj%,,77$%HII$++-00;K).K++ %% !''K {#+//4 4K = +w// & '2 7|a !!"sA6 wqzq3 #D 2 C/33C88 E#;66&{1~w??!^[^^F%;< %/ !%!3(;_MsHc Uc[RR5nUGHHnURU5nUb[R "U5(aM5USnUSn[ URURUR5up[X5V V s/sHup[XU -5PM n n n UR(aoU[U 5R5R5n [ R""U URURURUR%5S9nO-UnUR'5R)U[U 55 XU'GMK gs sn n f)Nrr)shapestride)r#r r"rr1 is_tensorrrr|r}rsliceis_metarbrcloner from_localrrrk)rr^rrro _local_staterrroffset cur_shape cur_offsetslicesr=rms rrrVsQ  z  " " 5 5 7'++C0  5??<#@#@ "1o "1o =   {66 8N8N  *-U); );%  *94 5);     &uV}5<<>DDFL$$''&&!''"))+ CC LLN U6]!; < #? sE2strictc40n[R"5S:XaUR5Htupx[R"U5(dXU'M&UR 5S:XaUR 5Xg'MN[UR5UR5Xg'Mv U/n [R"U SUS9 U Sn/n [UR55n [5n UR5HupxU RU5 [U[5(d Xq;aXU'M6[R"5S:XaXXg'U RU5 [!U 5S:dMw[#XaXU5 U(aU HnXR 5X'M U R%5 M U(a%X- =n (aU HnUR'U5 M U (a3[#XaXU5 U(aU HnXR 5X'M ggg)Nrrr)r#rhr_r1rdimrRrr&r broadcast_object_listr]r^addrTrrcrclearpop)rrrrrrIrmrorpbroadcast_listr^local_state_dict_keys global_keys missing_keyss r_broadcast_state_dictrs C }}!)//1JC??5)) C! 99;&uzz|U[[A 2UN~1B?  C D 0 5 5 78%Kiik %--&(-%  ==?a &+CH C t9> sdB GC,<,A,E,E,G$) JJL#"&1? @< @# $$S)$ 3$C (8(=(A(A(C %  rcJUR5GHupEX@;aM [R"U5(dXQU'M.UR5S:XaUR 5X'MV[ U[R 5(d [S5eURU5nUcM[ U[5(aB[UR5RU5URUR5X'MUR5RU5X'GM g)Nrzvalue must be a torch.Tensor)r_r1rrrRrTrUrirrrrr.r|r})rrrrrorprs r_distribute_state_dictrs&++-   % u%%$)S ! YY[A $)IIK  !eU\\22$%CDD*..s3K"K11(9LLN%%f-++**) % ). (9(9&(A %).rvisitorc^^S[S[SS4UU4SjjmUR5Hup#T"[U54U5 M g)z Invoke ``visitor`` for each value recursively in ``state_dict``. Mapping, list, and tuple will be flattened and other value types are treated as the terminal values and will invoke ``visitor``. pathrprNc>[U[5(a0UR5Hup#T"U[U54-U5 M g[U[[ 45(a![ U5HupCT"X4-U5 M gT"X5 gr)rTrr_rXrarbrd)rrpkrri _traverse_objrs rr+_traverse_state_dict.._traverse_objsq eW % % dc!fY.2& e} - -!%(dTk1-) D r)OBJ_PATHrr_rX)rurrorprs ` @r_traverse_state_dictrsI!H!S!T!!!&&( s3xk5))rcZ^^0m0mS[S[SS4UU4Sjjn[X5 TT4$)ao Flatten ``state_dict`` made of nested dicts and lists into a top level dictionary. Use ``unflatten_state_dict`` to revert this process. Returns: A tuple with the flatten state_dict and a mapping from original to new state_dict. N.B. The new keys are derived from the object paths, joined by dot. For example: ``{ 'a': {'b':...}}`` results in the key `a.b`. rrprNc>SR[[U55nUT;a[SU35eUTU'UTU'g)N.zduplicated flatten key )joinmaprXrg)rrpnew_fqn flattenedmappingss r flat_copy&_flatten_state_dict..flat_copy sF((3sD>* i 6wi@A A" ' r)rrr)rurrrs @@r_flatten_state_dictrsC"$I "H!!!!!/ h r root_dictrrpc[[U5nS[[S[SS4Sjn[ S[ U55HonXS- nXn[U5[La0O/n[U[5(a![[URXh55nMZU"X65 X6cXU'X6nMq USn[U5[LaU"[[[U5U5 X#U'g)z>Set ``value`` in ``root_dict`` along the ``path`` object path.lstrqrNch[U5U::a#URS5 [U5U::aM"ggr)rcr)rrqs r extend_list!_set_element..extend_lists&#h#o JJt #h#orr) rCONTAINER_TYPErarrVrangercr-rXrTr setdefault) rrrp cur_containerrrprev_keyrodef_vals r _set_elementrs3Mc1c$i A;g48I4Db" mW - -  8 8 KM  0&.*1h')3M! r(C CyCDcM2C8#rmappingcZ0nUR5Hup4[X!UU5 M U$)zaRestore the original nested state_dict according to ``mapping`` and the flattened ``state_dict``.)r_r)rurnestedrorps r_unflatten_state_dictr6s1!F &&( VS\51) Mr)NN)FT)FFr)NFF)HrerZr(rcollections.abcrrrtypingrrrr r r1torch.cuda._pin_memory_utilscuda_pin_memory_utilsrtorch.distributed distributedr#torch.nn.functionalnn functionalr/)torch.distributed._functional_collectivesr is_availabler 'torch.distributed._shard.sharded_tensorr torch.distributed.tensorrrrtorch.distributed.tensor._utilsrrUrrrr? ExceptionrAboolrbrVr\rXr`rrno_gradrrrrrarrrr PATH_ITEMrFLATTEN_MAPPINGSTATE_DICT_TYPErrrrrrrrrs ==>> 77 K-2ENNU  D  LL4    \\ $("&### D # LL4 # \\ #L  $("&"$HH!HH H D H LL4 HHHc3hHHH #s(^H\$("&"$US#XU D U LL4  U  U c3h UU #s(^Uv#% %S#X%c3h% %  #s(^ %P .S#X.#s(^.. .  #s(^ ..bOT_S#X_,0_HL_ #s(^__D'S#X'c3h' 'T*$( :<#s(^:<38n:< s):< LL :< D :<  :*9r