Ei_pSSKrSSKrSSKrSSKrSSKJrJrJr SSKJ r SSK J r J r SSK Jr SSKJrJrJrJrJr SSKrSSKJr SSKJs Js Jr SSKJr SSKJ r SSK!J"r" SS K#J$r$J%r%J&r&J'r'J(r(J)r)J*r* SS K+J,r, SS K-J.r.J/r/ SS K0J1r1J2r2 SS K3J4r4J5r5 SSK6J7r7J8r8J9r9 SSK:J;r;Jr> \(aSSK?J@r@ \R"\B5rC\ "SS55rDS\E\F\4S\\G\F\44SjrH\ "SS55rI"SS\5rJ"SS\5rKS\DS\E\F\4S\LS \LS!\LS\M\E\F\44 S"jrNS#\S\L4S$jrOS\DS\E\F\4S\I4S%jrPS\DS&\IS \LS\M\E\F\44S'jrQS(\%S)\E\F\4S*\RS-S\E\F\44S+jrSS(\%S&\S*\RS-S\4S,jrTS\DS-\FS)\E\F\4S\E\F\44S.jrUShS/\E\F\4S0\RS1\LS2\RRS-S3\LS*\RS-S\E\F\44S4jjrYS\DS5\E\F\E\F\44S6\M\FS\E\F\44S7jrZS8\FS9\M\RS6\M\FS:\\RS;\/S\R4 S<jr]S8\FS=\M\RS6\M\FS\R4S>jr^S8\FS?\M\S6\M\FS\4S@jr_SiSA\E\F\4S0\RS2\RRSB\M\E\F\4\\R-S-SC\LSD\LS\E\F\44SEjjraSjS0\RSB\M\E\F\4\\R-S-S\E\b\R44SFjjrcS0\R<RS\E\.\F44SGjrdSkS2\RRS0\RS-SD\LSH\E\R\M\F4S-SI\E\.\F4S-S\E\b\F-\R44 SJjjreSkS2\RRS0\RS-SD\LSH\E\R\M\F4S-SI\E\.\F4S-S\E\R\b\F-44 SKjjrfSjS0\RSB\M\E\F\4\\R-S-S\E\R\b44SLjjrgSM\M\KSN\E\K\F\b-4SO\E\F\b-\R4S*\RS-SS4 SPjrhSiS/\E\F\4S*\RS-SO\E\b\F-\R4SH\E\R\M\F4SQ\E\F\D4SR\LS\G\M\K\E\K\b\F-444SSjjriST\E\F\4SO\E\b\F-\R4SH\E\R\M\F4S\M\E\F\44SUjrjS/\E\F\4S\L4SVjrk\ "SWSX55rlS(\%SY\E\F\4S\M\E\F\l44SZjrmS\DS[\M\E\F\l4SY\E\F\4S\\E\F\E\F\44S\G\RS-\E\F\M\RS-444 S]jroS\DS\\E\F\E\F\44S8\FS \LS\LS!\LSS4S^jrpS\DS[\M\E\F\l4SY\E\F\4S \LS\LS!\LS\E\F\E\F\444S_jrqS\DSY\E\F\4S \LS\LS!\LS\E\F\44 S`jrrSlSa\M\KSN\E\K\b\F-4SQ\E\F\D4S/\E\F\b-\4S\LS \LS!\LS\E\F\44SbjjrsSlSa\M\KSN\E\K\b\F-4SQ\E\F\D4S/\E\F\b-\4S\LS \LS!\LS\E\F\44Scjjrt\R"5SmS0\RS2\RRS/\E\F\4SB\M\E\F\4\\R-S-S3\LS \LS*\RS-SC\LS1\LS!\LS\E\F\44Sdjj5rvS0\RS\E\F\D44Sejrw\S(\%Sf\8SS4Sgj5rxg)nN)IterableIteratorSequence) ExitStack) dataclassfield)chain)Anycast NamedTuple no_type_check TYPE_CHECKING)_gather_state_dict)_get_pg_default_device)_apply_to_modules _FSDPState._get_module_fsdp_state_if_fully_sharded_module_get_param_to_fqns_module_handle!_named_parameters_with_duplicatesclean_tensor_name)SimpleProfiler) FlatParameterFlatParamHandle)_ext_chunk_dtensor_ext_chunk_tensor) _lazy_init%_reset_flat_param_grad_info_if_needed)ShardingStrategyStateDictSettings StateDictType)DTensor Replicate) tree_map_only) ShardedTensorcN\rSrSr%\\S'\\S'\\\ 4\S'\ \ \S'Sr g) FSDPParamInfo5statehandle param_indicesparam_requires_gradN) __name__ __module__ __qualname____firstlineno__r__annotations__rdictstrintlistbool__static_attributes__r-`/home/mande/repo/quber/.venv/lib/python3.13/site-packages/torch/distributed/fsdp/_optim_utils.pyr'r'5s&  S>!d#r9r' dictionaryreturnc#`# [UR55nUH nX U4v M g7fN)sortedkeys)r;r@ks r: sorted_itemsrB=s- *//# $D As,.c\rSrSr%Sr\"\S9r\\\ R4\ S'\"\S9r \\\ R4\ S'\"\S9r \\\4\ S'Srg) _ConsolidatedOptimStateCa4 This holds the consolidated optimizer state on the target rank. Positive- dimension tensor state is communicated across ranks, while zero-dimension tensor state and non-tensor state is taken directly from the target rank. PyTorch version 1.12 moved to using zero-dimension tensors for scalar values, but user implemented optimizers may still use float (i.e. a non-tensor). Thus, we support both and handle them identically. Attributes: tensor_state (Dict[str, torch.Tensor]): Mapping from positive-dimension tensor state name to the unsharded flat tensor representing the state. zero_dim_tensor_state (Dict[str, torch.Tensor]): Mapping from zero- dimension tensor state name to its value. non_tensor_state (Dict[str, Any]): Mapping from non-tensor state name to its value. )default_factory tensor_statezero_dim_tensor_statenon_tensor_stater-N)r.r/r0r1__doc__rr3rGr4torchTensorr2rHrIr r8r-r9r:rDrDCs\&-2$,GL$sELL()G5:45P4U\\ 12P',T'Bd38nBr9rDcV\rSrSr%Sr\R \S'\R\S'Sr g)_PosDimTensorInfo]ac Metadata for positive-dimension tensors used internally for :meth:`scatter_full_optim_state_dict`. Attributes: shape (torch.Size): Sharded tensor shape (which is equal to the unsharded tensor shape if the tensor is optimizer state for a non-FSDP parameter and is hence not sharded). dtype (torch.dtype): Data type of the tensor. shapedtyper-N) r.r/r0r1rJrKSizer2rQr8r-r9r:rNrN]s  :: ;;r9rNc8\rSrSr%Sr\\S4\S'\\S'Sr g)_OptimStateKeymz This represents an optimizer state key that may be used commonly across ranks. It is based on the unflattened parameter names rather than parameter IDs to make it independent of each rank's own optimizer construction. .unflat_param_namesis_fsdp_managedr-N) r.r/r0r1rJtupler4r2r7r8r-r9r:rTrTms c3h'r9rTfsdp_param_infoflat_param_stateto_save shard_state cpu_offloadcNU(aU(d [S5e[UU5nU(ax[UUU5nUHcnU(dM [UR 55H:nXxn [ U [ R5(dM(U R5Xx'M< Me U$/$)a Unflattens the optimizer state, consisting of the "state" part and the "param_groups" part. Unflattening the "state" part involves consolidating the state on the target rank and remapping from flattened to unflattened parameter IDs, and the "param_groups" part only involves remapping from flattened to unflattened parameter IDs. Args: fsdp_param_info (FSDPParamInfo): The FSDP state, the handle, and a mapping from FQN to original parameter index. flat_param_state (Dict[str, Any]): Entry for the flat parameter in the "state" part of the optimizer state dict. to_save (bool): Whether to save the state on this rank. Returns: List[Dict[str, Any]]: A :class:`list` holding the entries in the "state" part of the optimizer state dict corresponding to the unflattened parameters comprising the flat parameter if on the target rank or an empty :class:`list` otherwise. The final optimizer state dict will need to map these entries using the proper unflattened parameter IDs. z7If ``shard_state`` is True, ``to_save`` has to be True.) AssertionError_communicate_optim_state#_unflatten_communicated_optim_stater6r@ isinstancerKrLcpu) rYrZr[r\r]consolidated_stateunflat_param_state optim_statekeyr)s r:_unflatten_optim_staterhxs:7VWW1@     .K{ 0 0 23C',E%eU\\:: ',yy{K$ 4."! r9xcb[R"U5=(a UR5S:H$Nr)rK is_tensordim)ris r:_is_zero_dim_tensorrns ??1  .!%%'Q,.r9cURnURRn[5nURUR UR pvn[U5GHup[R"U 5(GaHU R5S:Ga3URS:XdUR[R:XaXU'MkURc [!S5eU R"R$URR$:waU R'UR5n UR(R+5n U R,"U 6n [.R0"XUR2S9 UR4R75 [9[:R<UR>5RA5n U SU XX'GMj[CU 5(a#U RE5RG5Xh'GMXU'GM U$)a Communicates the optimizer state for a flat parameter across ranks. All ranks will hold the entire non-sharded optimizer state on GPU. If ``N`` is the number of tensor optimizer states in the optimizer state dict, then the communication complexity is 0 if ``N = 0`` and ``N + 1`` otherwise (where the plus 1 comes from all-gathering the padding per rank). Args: fsdp_param_info (FSDPParamInfo): The FSDP state, the handle, and a mapping from FQN to original parameter index. flat_param_state (Dict[str, Any]): The entry in the "state" part of the optimizer state dict corresponding to the flat parameter. Returns: ConsolidatedOptimState: Consolidated optimizer state for the target flat parameter. rNz'compute_device has not been initializedgroup)$r)r* flat_paramrDrGrHrIrBrKrlrm world_sizesharding_strategyrNO_SHARDcompute_devicer_devicetypeto_full_param_paddedsize new_zerosdistall_gather_into_tensor process_group_device_handle synchronizer nn Parameter_unpadded_unsharded_sizenumelrndetachclone) rYrZ fsdp_statersr)rGrHrI state_namevalue buffer_size tensor_bufferunpadded_numels r:r`r`s,!&&J ''22J # %E  ## *:L **:; ??5 ! !eiikAo %%*//3C3L3LL+0Z(((0$%NOO||  J$=$=$B$BB!:!:;%77<<>K!OO[9M  ' 'J,D,D   % % 1 1 3! jAAeg (5_n'EL $#5))49LLN4H4H4J%1/4,E>@"00"22 #K.9  *E(BJ !!,/D"EF!!,/?"@A!!"45W&X r9rrfrrcS/n[R"U5S:Xa[[RSU5US'[R "USUS9 [R"U5S:XaU$US$)NrcUR5S:XaUR5$[URUR5$rk)rmrcrNrPrQ)vs r:,_broadcast_processed_state..Qs.AaeegV3DQWWagg3VVr9srcrr)r~get_rankr$rKrLbroadcast_object_list)rrfrrobjectss r:_broadcast_processed_staterHsl G }}Uq " LL V    wAU; }}Uq qzr9c:[R"U5S:XaQ[U[R5(aUR 5S:XaU$UR UR5nO[U[R5(a/UR 5S:wa[SURS35eU$[U[5(dU$[R"URURURS9n[R"USUS9 U$)Nrz\For non-zero ranks, a tensor state should have zero dimension, but got the state with shape .rQrxr)r~rrbrKrLrmrzrwr_rPrNzerosrQ broadcast)rr)rrtensors r:_broadcast_stater[s }}Uq %..%))+2BL*334 eU\\ * *yy{a$449KK=CLE#455L KKu{{:3L3L  NN6q. Mr9fqncHU(d0$URnURRnURUnURUn[ X#R URS9nUR(d0$0nURnURn UR5Hyup[R"U 5(aUU R5S:aAUR[ R":wa#U R%5XS-R'5n XU 'M{ U$)z Shard the optimizer state for the original parameter with the name ``fqn``. This API should only be used when ``use_orig_params`` is True. pgrxrrp)r)r*rsr+_shard_param_infosrrrwin_shardintra_param_start_idxintra_param_end_idxitemsrKrlrmrurrvflattenr) rYrrfrrs param_idxshard_param_infonew_optim_staterrrrs r:_shard_orig_param_staterss   &&J ''22J--c2I!44Y?$009R9RK  $ $ &(O,BB*>>(..0 OOE " " a,,0@0I0IIMMO%)eg ', #1 r9optim_state_dictmodeluse_orig_paramsoptim rank0_onlyc Z[R"5 UnSU;aU(d [S5e[U5n[ U5n[ [ UR555Rn U(a [XUS9n0n USn [U R55n UR5GHFupUSnX;aMU RU5 U(a9UH.nX(dMXHn[XUUUS9XU'M M0 USnX;GaXnU(aG[R"[R R"5 [%UUX5nSSS5 O ['UU U5n[)[+U5S5nW(aUU U'OU(a[-U5S:wa[/SUS 35eUb_URR1U S5nUb[2R4"U5U U'O[6R8"S US U R:S 3S S 9 Oe[=SUS35e[-U5S:wa[/S[-U535e[)[+U5S5n[2R2"X5U U'U(dGMUH_nX(dM[?XR55H2unnU R:S:aXU MURA5XU'M4 Ma GMI U HYnU Un[CU[DRF5(aU(aU(a [U UUS9n[2R2"U5U U'M[ [RH"S5 SU;a[2R4"US5nU US.$SU 0$!,(df  GN/=f)a{ Flattens the full optimizer state dict, still keying by unflattened parameter names. If ``use_orig_params`` is True, each rank will have all FSDP-managed parameters but some of these parameters may be empty due to the sharding. For a regular optim.Optimizer, states for those empty parameters will not be initialized. So, when aggregating the FQNs across ranks, no assert will be raised on a rank even if it does not have all the states -- it is valid and FSDP know how to aggregate them. However, FSDP has to ignore handling those parameters that are not managed by FSDP and do not exist on the local rank -- it is managed by other parallelism and FSDP does not know ho to handle/aggregate them. Note that ``_flatten_tensor_optim_state`` does not need ``optim`` to flatten/shard the state. However, NamedOptimizer and KeyedOptimizer require all the states even if the corresponding parameters are empty. To this end, ``optim`` will be used to get the initial state of the empty parameters. ``optim`` should only be non-None if the ``optim` is KeyedOptimizer or NamedOptimizer. Returns: Dict[str, Any]: The flattened optimizer state dict. r)zO`optim_state_dict` must have the keys "state"to be a valid optimizer state dictrqrNTrpz5use_orig_params is True but there are multiple FQNs, rz optim_state[z] is not on rank stacklevelz The state of z8 is empty. This should happen when use_orig_params=True.zExpected len(fqns) == 1, got Fz,FSDP _flatten_optim_state_dict() profiling: param_groupsr)r)%rreset ValueErrorr_get_fqn_to_fsdp_param_inforitervaluesr)rsetr@rdifference_updaterprofileType RESHARDINGr_flatten_optim_staterTrXlenr_getcopydeepcopywarningswarnr RuntimeErrorr6rcrbrKrLdump_and_reset)rrrrrrr unflat_osd param_to_fqnsfqn_to_fsdp_param_inforflat_osd_stateunflat_osd_stateall_state_keysparamfqnsrrrY flat_statergr) param_state user_stateflat_osd_param_groupss r:_flatten_optim_state_dictrs@!Jj  1  'u-M8?d188:;<BBJ/ eT 79N!'*)..01N$**, 1g  & ((. ',"2"7J8H"S$9*$EU9$)*5#8q'C  (49O#++N,?,?,J,JK!8'(-"JLK2#$ !td3C&0s# t9>(OPTvUVW$!KKOOE48E(.2mmE.Bs+ *3%/? ?PPQR'( ##C5),, 4yA~$'DSYK%PQQ te4C"&)),<,A"BN3  :',/34D4I4O4O4Q/R+J !*,1*==HOOURnURnURn[U5nUS::a [ S5eUR n[U5nXh:wa[ SUSU35eUV s/sHn [ X;5PM n n [U 5(d0$UV s/sH+n X;a![XURURS9OSPM- n n Sn U HQn U cMU c[U R55n M&U [U R55:wdME[SU35e U c[ SU 35e0nU GHnU V s/sH n U bXOSPM nn UVs/sH nUcMUPM nnU(dSX'MAS =n=nnUHbnU[R"U5=(a UR!5S:-nU[#U5-nU[R"U5(+-nMd UVs1sHn[%U5iM nn[U5S :wdU(d"U(dU(d[S US US U35eU(av['UUUUU5nUR(S :waNUR*[,R.:wa0[0R2"UUR4UR(5unnOUnUX'GMU(a[7UUU5X'GMU(d[ SU35e[9UUU5X'GM U$s sn fs sn fs sn fs snfs snf)a3 Flattens the optimizer state in ``full_optim_state_dict`` for a single flat parameter in ``fsdp_param_info`` corresponding to the unflattened parameter names in ``unflat_param_names``. Args: fsdp_param_info (FSDPParamInfo): The FSDP state, the handle, and a mapping from FQN to original parameter index. unflat_osd_state (Dict[str, Dict[str, Any]]): The "state" part of the optimizer state dict corresponding to the unflattened parameters. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the flat parameter ``flat_param``. Returns: Dict[str, Any]: A :class:`dict` mapping state names to their values for a particular flat parameter. The sharded optimizer state dict's "state" part will map a key to this returned value. rzNExpects at least one unflattened parameter corresponding to the flat parameterzExpects z shapes but got rNz@Differing optimizer state names for the unflattened parameters: z)Expected state_names to be not None, got Trpz*Differing optimizer state types for state z , values z", and unflattened parameter names z)Expected are_non_tensors to be True, got )r)r*rsrr__shapesr7anyrrrwrr@rrKrlrmrnry_flatten_tensor_optim_statertrurrvr _get_shardr$_flatten_zero_dim_tensor_optim_state_flatten_non_tensor_optim_state)rYrrVrr*rsrunflat_param_shapesnum_unflat_param_shapesunflat_param_name has_stateunflat_param_states state_namesrerr state_valuesrnon_none_state_valuesare_pos_dim_tensorsare_zero_dim_tensorsare_non_tensorstypesrsharded_flat_tensorrs r:rr+s.!&&J  # #F""J./A \  %,,!"563())9:Q9R S  "4!3   23!3 y>> "4 "4   0   /'',,   "4 K1  %   05578Kc"4"9"9";<< ##5"682H VWW24J! ': &9"/A.L  *RV V&9  -9 JLqAL J$%)J " GKKK2_&A 5??1#5#E!%%'A+ E  $7$: : 5??1#55 5O'#88"7Qa"78 u:? #7? @  f E_B  >c%jAoUV V  FLLE$9$ll^1UG5  Ce$J#&o"K #L K  "  knnZ01 ]] KK!   #L (();=R=RSK!::,,";#4#4"55FGWFX Y   WE 0. sF/F/F4:AF9zero_dim_tensorsc UVs/sH o3cMUPM nnUVs1sHo3bUR5OSiM nnUVs1sHo3b UROSiM nn[U5[U5:wd[U5S:wd[U5S:wa[SUSUSUSU35e[ [ U55n[ [ U55n[ R"Xx[ R"S5S9$s snfs snfs snf) a, Flattens the zero-dimension tensor optimizer state given by the values ``zero_dim_tensors`` for the state ``state_name`` for a single flat parameter corresponding to the unflattened parameter names ``unflat_param_names`` by enforcing that all tensors are the same and using that common value. NOTE: The requirement that the tensors are the same across all unflattened parameters comprising the flat parameter is needed to maintain the invariant that FSDP performs the same computation as its non-sharded equivalent. This means that none of the unflattened parameters can be missing this state since imposing a value may differ from having no value. For example, for Adam's "step", no value means maximum bias correction, while having some positive value means less bias correction. Args: state_name (str): Optimizer state name. zero_dim_tensors (List[torch.Tensor]): Zero-dimension optimizer state for the unflattened parameters corresponding to the single flat parameter. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the single flat parameter. Returns: torch.Tensor: A zero-dimensional tensor giving the value of the state ``state_name`` for all unflattened parameters corresponding to the names ``unflat_param_names``. NrpAll unflattened parameters comprising a single flat parameter must have scalar state with the same value and dtype but got values z and dtypes rr rcr) itemrQrrrrrKrrx) rrrVrr values_setrrrQs r:rrsB$4E#3a#3E?OP?O!m!&&(5?OJP:J K:JQaggD0:JF K %5!66 z?a  v;!  (\fX[l;!"  $   j! "E f E <<5<<3F GG%FP KsC9C9C>D non_tensorscUVs/sH o3cMUPM nn[U5n[U5[U5:wd[U5S:wa[SUSUSU35e[[ U55nU$s snf)a@ Flattens the non-tensor optimizer state given by the values ``non_tensors`` for the state ``state_name`` for a single flat parameter corresponding to the unflattened parameter names ``unflat_param_names`` by enforcing that all values are the same and using that common value. See the note in :func:`_flatten_zero_dim_tensor_optim_state`. Args: state_name (str): Optimizer state name. non_tensors (List[Any]): Non-tensor optimizer state for the unflattened parameters corresponding to the single flat parameter. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the single flat parameter. Returns: Any: A non-tensor giving the value of the state ``state_name`` for all unflattened parameters corresponding to the names ``unflat_param_names``. rprrz" and unflattened parameter names )rrrrr)rrrVntnon_none_non_tensorsnon_tensor_set non_tensors r:rrFs2*5G2BG%N  C $44N8Kq8P ,-[ E++=*> @  d>*+J Hs A3A3 sharded_osd optim_inputusing_optim_inputis_named_optimizerc l[U5n[U5n[[[R [ [-4U(a [X5O [X!XVU55n[U5[U5:a#[S[U5S[U535e0n 0n UR5H*upX;aM Xn X[U 5'U HnXU'M M, USn0nUR5HMunn[U[5(aUUU'M"U RUR UR 5n UUU 'MO SU;a`/nUSHPn["R$"U5n['USVs1sHnXiM sn5nUUS'UR)U5 MR UUS.$SU0$s snf)a Rekeys the optimizer state dict from unflattened parameter names to flat parameter IDs according to the calling rank's ``optim``, which may be different across ranks. In particular, the unflattened parameter names are represented as :class:`_OptimStateKey` s. z r)rparamsr)r_get_flat_param_to_fqnr r3rrr5r4'_get_param_to_param_id_from_optim_input_get_param_to_param_keyrr_rrXrbrrVrrr?r)r rrr!r"r#rflat_param_to_fqnparam_to_param_key$unflat_param_names_to_flat_param_key#unflat_param_name_to_flat_param_keyrrVflat_param_keyrsharded_osd_staterekeyed_osd_statergrrekeyed_osd_param_groupsunflat_param_groupflat_param_groupflat_param_keyss r:_rekey_sharded_optim_state_dictr4ms'u-M.u58< R\\39 $%! 4E G(0AR 9 ]!33J3OaKbJccfgjkxgyfz {  )AC'%2%8%8%:!  * +2JXU3E-FG!3 ES0A B"4 &;$G,.0-335[ c3  %0 c " =AA  " "C$:$: -8.)6$9; "-n"= #}}-?@ $.@-I-I)8J-IO *9 X & $ + +,< =#>+??KIz%66 Zt,,  ySTTIf%&& B9+NOO,. ![0 `    X!67  +, --=  "m %  s*D;; EEEc xSnSn0n[UUU[U5VVs/sHupEUPM snnU5$s snnf)a Constructs a mapping from ``FlatParameter`` to a cleaned (devoid of prefixes from wrappers) fully qualified name (FQN). Note that this FQN is "non-canonical" because ``FlatParameter`` s do not come from the original module but are registered only after FSDP has been applied. This function returns the FSDP-given name for the ``FlatParameter`` (usually module._flat_param) as opposed to the canonical FQNs returned for ``FlatParameter`` s in ``_common_utils._get_param_to_fqns(...)``). Consequently, this function will only return a non-empty mapping if FSDP was applied with ``use_orig_params=False`` as, otherwise, the original parameters are used within the module and there would be no ``FlatParameter`` s in the module. cz[USS9H-upE[U[5(dM[X-5nXcU'M/ g)NF)recurse)rrbrr)moduleprefix tree_levelr) param_namerrs r: module_fn)_get_flat_param_to_fqn..module_fns@!B E" Je]33#F$78C'*e $ " r9cU$r>r-)r)s r: return_fn)_get_flat_param_to_fqn..return_fn  r9rr)rrHrKflat_param_to_fqn_retrrs r:r&r&sN+!79  rr:s r:_get_param_key_to_paramrV(s-/  $5$= V  = #Fug!NO O7>FC?$3..$?@B .9%%D@s ! E3E8r merge_keysc 0[R"U5n0n/nUR5HupXS;aMX:n [U [5n U (a4U SU;a+[ SU SS[ UR5535eU SU;n [[U 5U S9n US:XdU(aURU 5 XU 'M U(as[[R"U55Vs/sHn/PM nn[R"XUS9 /[R"U5Qn[![#U55nX4$US:XaU/OS/n[R$"USUS9 USc[ S US35eUSn['UUUU5 X4$s snf) a, Construct the local mapping between the ``_OptimStateKey`` and parameter keys and all the ``_OptimStateKey`` across ranks. If ``merge_keys`` is False, rank0 must contain all the ``_OptimStateKey``, an exception will be raised otherwise. Note that ``merge_keys`` should equal to ``use_orig_params``. r)rz Expected z, to be in fqn_to_fsdp_param_info, got keys: )rVrWrqNrz-Expected key_obj_list[0] to be not None, got )r~rrrbrr_r6r@rTrXrrr_r`r from_iterabler?rrrg)rrrrTrrrhrr[all_optim_state_keysrcrrrWoptim_state_keyrall_keysmerge_all_optim_state_keys key_obj_lists r:_map_param_key_to_optim_keysrps == DDF 13.446  W5 5 #$UM: Aw44$Qy(TUYZpZuZuZwUxTyzq'%;;($T{+  19 ' ' 88A_5'7*d11%890 91B9 0  xUK%Eu':':8'D%E"%c*D&EF$ =='+ai ! "dV  ""BcURS5nU(dg[[UR555n[ U[ 5$![an[ U5UeSnAff=f)z Returns whether the state_dict is from a NamedOptimizer. This function checks that the keys in the state_dict['state'] are strings (which usually are FQNs) versus integers (which usually refer to param_ids from a vanilla torch.optim.Optimizer). r)FN)rrrr@ Exceptionrbr4)rr)rgr:s r:_is_named_optimizerrxse   )E 14 %& c3  1()q01s"A A( A##A(cf\rSrSr%\\\4\S'\\\R4\S'\\\ 4\S'Sr g) StateInfoitensorsscalar_tensorsrr-N) r.r/r0r1r3r4rNr2rKrLr r8r-r9r:rzrzs7#(( ))ell*++c3hr9rz input_statesc00n[UR5Vs/sHn0PM nnUR5HupV[0005n[ U5Hup[ R "U 5(abU R5S:XaU R5URU'MS[U RU R5URU'MXRU'M XrU'M [R "UUUR"S9 U$s snf)z Given the ``input_states``, allgather StateInfo for each state. The function uses all_gather_object to gather StateInfo so no GPU tensors are sent. rrq)rrtrrzrBrKrlrmrcr|rNrPrQr{rr~r`r) rr}processed_state_dictrgathered_state_inforrfprocessed_staterrs r:_allgather_state_infors24*//070q07)..0#BB/!-k!: Ju%%99;!#AFO22:>:K U[[;O++J7;@++J7";%4S!1 && 17s Dr output_statesc 0nUR5GHupVUVs/sHowUPM nn[UV V s1sHoRHoiM M sn n 5n [5n Sn U GHan/n[5n[ U5HunnUR S5 URR US5nUb\URR5US'U (d URn O+U UR:wa[SU SUR35eUSS:XdMURU5 M U (aU U:Xd[SU SU35eUn X;aURVs/sHnSPM snXN'X%R US5nUb%URURR5nUXNURU'GMd [ U5HunnUU ;aMUR R5HAunnUR US5nUb!UU:Xd[SUS US US 3S U3-5eUUU'MC UR"R5HWunnUR US5nUb7[$R&"UU5(d[SUS US US 3S U3-5eUUU'MY M GM W U4$s snfs sn n fs snf) a Given the ``gathered_state_info`` and ``input_states``, the API converted the StateInfo into the original state if the state is not a non-scalar tensor. For a multi-dimensional tensor, the local state will be stored in ``state_buffer`` in a correct order for later allgather purpose. Nrz"Expected dtype == info.dtype, got  != z?Expected empty_ranks to be empty or equal to _empty_ranks, got z vs zRank z has different values for z: rz Other ranks: )rr?r{rr6rrrPrrQr_addr+rzr)rwrr|rKequal)rYrr}r state_buffersrgathered_states state_infor)nall_tensor_states empty_ranksrQrnumels _empty_ranksr object_stateinfor local_statenamenon_tensor_valuecurr_non_tensor_valuescalar_tensor_valuecurr_scalar_tensor_values r:_convert_all_state_inforBs7;=M,224&9:&9f&9 :":#U:%}}!A}A:#UV # $(,JF%(UL&/ &;"l a #++// DA#!%!1!1!3F2J $  DJJ."0"DUG4PTPZPZ| \#":? $$T*'<${l'B$UVaUbbfgsftu'K."1"?"?-"?QD"?- )'+// DAK&)nn_-B-B-Q-QR LWM %o&C&CC&H IC,L#,J"7 D,{"*6*B*B*H*H*J&&(6(:(:4(F%)1,0@@(v%?vRHXGYYZ[*+@*ABC(8t$+K.:-H-H-N-N-P))+9+=+=dD+I(,4{{#68PQQ(v%?vRH[G\\]^*+C*DEF(;t$.Q#8[ 5P - O;#U:-sK K + Kc U(dgURRnURnUR5GHupXn URUn [ U [ 5(aU RSn U [5:waU Rn U R[54S9 [URU 5nX==U RRS5-ss'[R "U5nU R#U5n O=U R#URU 5n OU R#URU 5n U(aUR$n['USS5(aRUR(c[+SUR(35e[-U UR.UR(UR05n OUR2c[+SUR235e[5U UR.UR6UR8R;5UR2UR05n ODU(d=[<R>"S5 U RA5RC5n SSS5 U(aG[<R>"[<RDRF5 U RI5n SSS5 XU'GM g!,(df  Nf=f!,(df  N)=f) a Given a output state dict, ``output_states``, which the keys are FQNs to the original parameters (not FlatParameters nor parameter ID), and the values are gathered states, unflatten the states to the original dimensions. This function performs the unflattening process in-place. Nr) placementsrFrrr)%r*rsr)rr+rbr"rr#rm redistributer6r device_meshr|rKrRreshaperrrr_rrrrrrtrrrrrrrD2Hrc)rYrrr\r[r]rsrrrrr placement placement_dim reshape_sizers r:_unflatten_orig_param_statesrsz  ''22J &&J,224*#11#6  eW % %((+IIK' ) ""y{n"=#J$6$6y$AB +u/@/@/E/Ea/HH+$zz,7  l3 j&8&8&CDMM*"4"4Y"?@E #<599**2(DZE\E\D]^+OO++.. ++3(EjF^F^E_`*OO))--::<,,.. ''0 ,,.1 ''(;(;(?(?@ A%*z"q 5d10A@sK2K& K# & K4 c URnURS:Xa_[R"5[RR :Xa.[ RSURR55 UVs0sHow0_M nn[XX(5up[U 5S:XaU$URR5VV s/sH up{Xx;PM n nn URRn [ R""[$R&XR(S9nU"U R*5nURR-5 U R5GHunn/nURU R.R15-nUU R.R15-S- nSunn[3U R4U R65GHunnU(+=(a% UR8U(+=(a U U(+nU(dU(aUUU-S- nnUUs=::aU::aO OUU:aUU- S-OUU- S-nOCUUs=::aU::aO OUU::aUU- S-OUU- S-nOUUs=:a Us=::aU:aO OUnOSnU(aUR;U"U55 U(d7UUb,UR;[=[$R>UU55 US- nUU- nGM U R.R15[ASU55- nU RBU:waJ[ESU RBS US U R.R15S U R4S US US3 5eUS:aUR;U"U55 [$RF"U5nUR15URH-UR15:wa [ES5eURR-5 [JRL"[JRNRP5 [RR"UUURTS9 URR-5 SSS5 USU RVR15nURn U RYU5n![U!5[UR5:wa [ES5eURR5H*up{UR8U (dXx;dM U!U XU'M, [[UUUUUU5 GM AU$s snfs sn nf!,(df  N=f)z Given the ``gathered_state_info`` and ``input_states``, the API allgathers all tensor states and restore non-tensor states from ``gathered_state_info``. rz@Memory Summary before calling to _allgather_orig_param_states %srrp)rrNc3@# UHoR5v M g7fr>)r).0rs r: /_allgather_orig_param_states..[s1=a =szLManually calculated _sharded_numel_padded is incorrect. _shard_numel_padded=z, shard_numel_padded=z, _sharded_size.numel=z, _numels_with_padding=z, begin=z, end=,zThe size of local shard times the world size should equal to the gathered tensor size. The inconsistency may be from a bug of FlatParameter's metadata or the reconstruction logic in optimizer state dict.rqzThe number of parameters from FlatParameter is not consistent to the number of states used by optimizer state dict reconstruction logic.).r)rr~get_debug_level DebugLevelDETAILloggerrrmemory_summaryrrr+rr*rs functoolspartialrKemptyrw_padded_unsharded_sizer _sharded_sizerr _numels_with_padding_is_padding_maskr,rr rLsum_shard_numel_paddedr_catrtrrr ALLGATHERrrr_get_unflat_views_alignedr)"rYrr}r\r[r]rrrrQridxhas_state_paramsrs empty_funcgathered_tensorrbuffers local_buffersbeginend mem_offsetrr is_paddingfrozen_and_no_state padding_begin padding_end padding_lenshard_numel_padded local_shardunpadded_tensorflat_param_handle orig_statess" r:_allgather_orig_param_statesrs2!&&J! 4 4 6$//:P:P P N  % % 4 4 6 DP/P!? @ii .    !6!6 6/:O:O:Q Q   !!--/  # #N$7$7$A$A B  ' 'J4L4L   % % 1 1 3 C**WJ,O,O,U,U,WX+22'AA/R { s?#@#@A A   (55;;=HC22373;O1 %        ] 5n  i0Q$ZC Bs T9 T>/;U U c URnURS:XdUR[R:Xa U(aU$0$[ R "[ RR5 [ R "[ RR5 [XQ5nSSS5 [UWUUUU5nSSS5 U(aURR5HmupUW;aM URU (dM#[US[!URR#55S[!UR#55S35e W$0$!,(df  N=f!,(df  N=f)a# Given a optimizer state dict, ``input_states``, which the keys are FQNs to the original parameters (not FlatParameters nor parameter ID), gather all the states and unflatten them to the original dimensions. Note that all the params referred by the ``input_states`` must be managed by FSDP. rpNzB is not in the output state. The FSDPParamInfo has the param keys z, while the output_states has the param keys r)r)rtrurrvrrrr ALLGATHER_OBJrrr+rr,rr?r@) rYr}r\r[r]rrrrgrs r:_gather_all_orig_param_statersU!&&J"  ' '+;+D+D D&|.B.    3 3 > > ?  # #N$7$7$E$E F"7 "Q G4         @'55;;=HCm#"66s;%8/77<<>?@A8-,,./0 3  > 5G F @ ?s$4/E6# E%/E6% E3 /E66 Frkc n0n0nUGHn URU 5n U cU R(dM+U R(a]U RSn URU 5n U cMaU c0OX:n [U 5U;a0U[U 5'X[U 5U 'MU(dM[ U R5S:wa![ S[ U R535eU RSn[ R"S5 [[[-U 5n [R"X:5X~'U(aI[X~5H8unn[R"U5(dM#UR5X~U'M: SSS5 GM UR!5Hn[#[%UR'555n X+n [ U R(5S::a [ S5eU R*R-5HnunnUU;aMU R(U(dM$[/US[1U R*R'55S[1UR'55S35e UR3[5U UUUU55 M U$!,(df  GM=f) Nrrp;Expected len(optim_state_key.unflat_param_names) == 1, got none_fsdp_managed_copyzgWith use_orig_params, FSDPParamInfo should have requires_grad information. However, the length is zero.zE is not in the optimizer state. The FSDPParamInfo has the param keys z( while the optimizer has the param keys r)rrWrVidrr_rrr r4r5rrBrKrlrcrrrr@r,r+rrr?rr)rkr[rrr[r\r]fsdp_osd_state all_statesrlrcrrYr)rrr _all_statesrgrs r:_convert_state_with_orig_paramsrs&(N -/J0&B&F&F&W  _%D%D   * *!44Q7C488=O&#+B1A1LE/"*424 2o./38r/* +C 0 W?556!;$QRUVeVxVxRyQz{!0 B B1 E ''(@A sI6 48II$/51-9&9.) E %u55$HM 9*E . BA90V"((* 4 ((*+,05 22 3q 8 < (55;;=HCk!"66s;%8/77<<>?@A4+**,-.a 1  >  (   )+< [BAs BJ$$ J4 c N0nUGHnURU5n U c[SUSU 35eUR(aURSn X*n [ U X9UUU5n U(av[ U 5[ UR5:wa-[S[ U 5S[ UR535eUR [URU 55 MMU(dM[ UR5S:wa![S[ UR535eURSn [R"X95X}'U(dGMS[X}5H7up[R"U5(dM"UR5X}U'M9 GM U$)NzQIf use_orig_params is False, we must be able to find the corresponding param id. r rzKExpected len(unflat_state) == len(optim_state_key.unflat_param_names), got rrpr) rr_rWrVrhrrr rrBrKrlrc)rkr[rrr[r\r]rrlrcrrY unflat_staterrrs r:_convert_state_with_flat_paramsr#s&(N0&B&F&F&W   ++:*;1YKI   * *"44Q7C49O1 + L|$O,N,N(OO("<01c/:\:\6]5^`%%'::$ W?556!;$QRUVeVxVxRyQz{!0 B B1 E 04 :J:U0VN -{)5"5*%J!??511 DIIIKN5jA *U0b r9c B[R"5 [5n U R[R"[R R 55 [[R"U55 U(+=(d" [R"U5S:H=(d Un [R"S5 [U5n [U5n [U5n[[ ["[$-[&R(4U(a [+X5O [-XXU 55n[/U5nSSS5 [R"S5 [1UUWW WUS9unnSSS5 [R"S5 U(a[2O[4nU"WWWUSU UU 5nSSS5 U (d0$SW0n[7W R955nUSR;5H<unnUU;aMUU;aMUW;aM[<R>"SUS 3S S 9 UUU'M> S U;a[AUWW 5US 'U RC5 [RD"S 5 U$!,(df  GN9=f!,(df  GN=f!,(df  N=f)av Consolidates the optimizer state and returns it as a :class:`dict` following the convention of :meth:`torch.optim.Optimizer.state_dict`, i.e. with keys ``"state"`` and ``"param_groups"``. The flat parameters in ``FSDP`` modules contained in ``model`` are mapped back to their unflattened parameters. Parameter keys are not well-defined. For a regular optimizer, the optimizer state_dict contains a mapping from parameter IDs to parameter states. Parameter IDs are the order of parameters in ``optim.param_groups()`` across all the groups. This API also allows user to pass ``optim_input`` for the mapping between parameters and parameter IDs. Using ``optim_input`` is being deprecated. If the optimizer is a ``NamedOptimizer``, the optimizer state_dict does not contain parameter IDs mapping but a mapping from parameter FQNs to parameter states. This API finds the mapping from FQNs to parameters if the optimizer is a ``NamedOptimizer``. If ``use_orig_params`` is True, each rank will have all FSDP-managed parameters but some of these parameters may be empty due to the sharding. For a regular optim.Optimizer, states for those empty parameters will not be initialized. So, when aggregating the FQNs across ranks, no assert will be raised on a rank even if it does not have all the states -- it is valid and FSDP knows how to aggregate them. However, FSDP has to ignore handling those parameters that are not managed by FSDP and do not exist on the local rank -- those are managed by other parallelisms and FSDP does not know how to handle/aggregate them. Args: model (nn.Module): Root module (which may or may not be a :class:`FullyShardedDataParallel` instance) whose parameters were passed into the optimizer ``optim``. optim (torch.optim.Optimizer): Optimizer for ``model`` 's parameters. rank0_only (bool): If ``True``, saves the populated :class:`dict` only on rank 0; if ``False``, saves it on all ranks. (Default: ``True``) shard_state (bool): If ``True``, shard and distribute all non-zero-dimension states. Returns: Dict[str, Any]: A :class:`dict` containing the optimizer state for ``model`` 's original unflattened parameters and including keys "state" and "param_groups" following the convention of :meth:`torch.optim.Optimizer.state_dict`. If ``rank0_only=False``, then nonzero ranks return an empty :class:`dict`. r preprocessingNpreprocessing_with_comm)rhstate_convertingr)zFound a optim state, aK, that FSDP cannot process. FSDP will directly copy everything to the returned state_dict. In most cases, this is a user-defined state that is not associated with any particular parameter. Another possible case is this state is managed by TorchRec. Otherwise, there may be a mismatched assumption of optim_state_dict of this mode.rrrz$FSDP _optim_state_dict() profiling: )#rrr enter_contextrrALLrtraversal_utils_get_fsdp_handlesr~rrr&rxr r3r5r4rrr@rVrrprrrrrrrrucloser)rrrr!rr\rrr"rr]cmr[rr)r#rTrrkr[ convert_fnrfsdp_osdflat_param_fqnsrgrs r:_optim_state_dictrcspz B^++N,?,?,C,CDE)/*K*KE*RSnH e 4 9H[G    0*51 25901AB! sBLL( )%8K,"4EV  "=U!C 1"    9 : )     "&   ( ;    2 3 ,0  $ ( " W %     4$  '8H+2245O&w/557 U .  / !  $ $   #C5)L L  $s'8*))#: 0-$  HHJ!!"HI Og 1 0" ; : 4 3s%A8I,I>%J, I;> J  Jc xSnSn0n[UUU[U5VVs/sHupEUPM snnU5$s snnf)a Construct the mapping from a param's fqn to its corresponding ``FSDPParamInfo`` if the param is managed by FSDP. Shared parameters, or original parameters that are shared across multiple nn.Modules, are required to belong to one and only one FSDP instance and thus correspond to one ``FlatParameter``. Within the one ``FlatParameter``, ``FlatParameter._fqns`` only stores the first FQN of a shared parameter. Thus, the keys in the mapping are guaranteed to map to unique parameters. c[U5nUcg[X@5 [X@5nU(dgURn[ XE0/5n[ UR 5Hup[X-5n X;a)X:RRULa[SU 35eXsU 'XRU 'URcMaURRURUR5 M g)NzDExpected fqn_to_param_info[fqn].handle.flat_param is flat_param for )rrrrsr'r6_fqnsrr*r_r+_paramsr,r requires_grad) rDrErFfqn_to_param_inforr*rsrYr local_fqnrs r:rH._get_fqn_to_fsdp_param_info..module_fnsCFK   :& 3 && ' BC( (8(89NC#F$67C'$)00;;:M(^_b^cd&5c "14 ) )# .!!-33::&&s+99:r9cU$r>r-)rs r:rK._get_fqn_to_fsdp_param_info..return_fn rMr9rN)rrHrKrrrs r:rrsP4!35  400-== m<< <9E]  HL  7 7 D1r9)FNFN)Fr>)NFNN)T)FT)yrrloggingrcollections.abcrrr contextlibr dataclassesrr itertoolsr typingr r r r rrKtorch.distributed distributedr~'torch.distributed.fsdp._traversal_utilsfsdp_traversal_utilsrtorch.nnr#torch.distributed._state_dict_utilsr"torch.distributed.distributed_c10dr$torch.distributed.fsdp._common_utilsrrrrrrr#torch.distributed.fsdp._debug_utilsr"torch.distributed.fsdp._flat_paramrr'torch.distributed.fsdp._fsdp_extensionsrr%torch.distributed.fsdp._runtime_utilsrrtorch.distributed.fsdp.apirr r!torch.distributed.tensorr"r#torch.utils._pytreer$'torch.distributed._shard.sharded_tensorr% getLoggerr.rr'r3r4rXrBrDrNrTr7r6rhrnr`ra ProcessGrouprrrModuler OptimizerrrrLrRrrrrr4r5r@r&rVr(r'rgrprurxrzrrQrrrrrrno_gradrrrr-r9r:rs  88 (FF AABE?M 8-E   8 $ $$ $T#s(^sCx0I  CC C2   Z3"338n33 3  3  $sCx. 3l/3/4/B"B38nBBJN"N "NN $sCx. Nbc3h   t # #s(^ &#&/3/@/@4/G0#"# #c3h# #s(^ #R"*.&* O)38nO) 99O)O) ;; 4 ' O)  O)   t # O) #s(^O)dM"M3S#X./MS M #s(^ M`RR%,,'RS R"%**- R  R  \\ Rj3H3H5<<(3HS 3H \\ 3Hl$$c$S $ $Z % J,c3hJ, 99J, ;; J,d38n%(>>E J,  J,  J, #s(^J,^IMF. 99F.d38n%(>>EF. #r|| F.R"%((//"d=#;M6N"N#$:>9= ; ;; ; 99t ;; d3i/047 ; M3./$6 ;  #)R\\ !" ;@#$:>9= N ;; N 99t NN d3i/047 N M3./$6 N  ",,c !" N(IMN 99Nd38n%(>>EN ",, N&&n-&&"&~sSy'@"A&&S3Y 45&&   t # &&  &&^ ?>38n?>   t #?>S3Y 45?> d3i/0 ?> !m!34 ?>  ?> 4 ncCi&?!@ @A?>DS#XS3Y 45 d3i/0 $sCx. , $sCx. T &     ""sCx." $sI~ "JW "W d3 >23W sCx.W T#s(^+, W  5;; tCellT.A)B$BC CD W tK+"K+T#s(^+,K+K+ K+  K+  K+ K+\g"gd3 >23gsCx.g g  g  g #tCH~ gT/"/sCx.// /  /  #s(^ /rZ~.Z"&~sSy'@"AZ!m!34Z39c>* Z  Z  ZZ #s(^ZH=~.="&~sSy'@"A=!m!34=39c>* =  =  == #s(^=@"U 99U ;; U38nUd38n%(>>E U  U  U   t #UUUU #s(^UUp1ryy1T#}:L5M1hLL*L LLr9