EiBY %SrSSKrSSKrSSKrSSKrSSKJrJrJr SSK J r J r SSK J r SSKJr SSKJrJrJrJrJr SSKrSSKJr SSKJs Js Jr SSKJr SSK J!r!J"r" SS K#J$r$ SS K%J&r& SS K'J(r( S S K)J*r*J+r+J,r,J-r-J.r.J/r/ \(aSSK0J1r1 SSK2J3r3 S SKJ4r4 Sr5\5S-r6Sr7\Rp"5r9\Rp\:S'"SS5r;"SS\;5r<"SS\"5r=S\R|S\=S-4Sjr?S\R|S\=S-4Sjr@"SS \ 5rA"S!S"\ 5rBS#\=4S$jrC\S#\=S\R|S\S%4S&j5rD\S#\=S\R|S\E4S'j5rFS(rGS)\HS\H4S*jrIS+\RSS4S,jrKS+\RS\E4S-jrLS\R|S.\S\M\N\H\R44S/jrPSGS0\R>R|S1\ES\Q\R\M\H44S2jjrR\S#\=S3S%S4\RSS4S5j5rT\S#\=S3S%S\M\HS-4S6j5rUSHS7\R>R|S8\S9\S:\M\HS-4S;jjrV\S#\=S<\M\ASS4S=j5rWS>\X\R|S\X\R|4S?jrYS@S04SA\R>R|SB\\Z\R|SC\Q\H\4S\X\Z\R|4SDjjr[S+\RSE\RSS4SFjr]g)Iz7 This file includes private common utilities for FSDP. N)Callable GeneratorIterable)autoEnum)partial)chain)Anycast no_type_checkOptional TYPE_CHECKING)_get_module_state_State)_CHECKPOINT_PREFIX)_apply_to_tensors) no_dispatch)FullOptimStateDictConfigFullStateDictConfigOptimStateDictConfigShardingStrategyStateDictConfig StateDictType) DeviceMesh)FSDPExtensions)FlatParamHandle_fsdp_wrapped_module._fsdp_flattened_MODULE_TO_INP_DTYPEc\rSrSrSrS S\R S\4Sjjr\ S\R SS4Sj5r S \ S\4S jr S r g) _FSDPDeviceHandle9z This is a simple abstraction for FSDP computing devices, which enables custom backends that implement CUDA-like semantics to be integrated with FSDP. NdevicebackendcUc'[[UR5UlXlgX lg![ a"n[ SUSURS35UeSnAff=f)NzDevice 'z=' does not have a corresponding backend registered as 'torch.z'.)getattrtorchtype_FSDPDeviceHandle__backend_FSDPDeviceHandle__deviceAttributeError)selfr%r&excs a/home/mande/repo/quber/.venv/lib/python3.13/site-packages/torch/distributed/fsdp/_common_utils.py__init___FSDPDeviceHandle.__init__@sh ? !( !< & %N " $vh&cdjdodocpprs s%2 AAAreturncURS:Xa[[[R5$URS:Xa[[[R 5$U"U5$)aC Return a device handle corresponding to the device, and through this handle, operations with the same semantics as CUDA can be performed on the device. Just return torch.cuda if the device is cuda to make attribute-access faster. Custom backend must first register a module with the same name with {device.type} on torch. cudamtia)r*r r#r)r5r6)clsr%s r0 from_device_FSDPDeviceHandle.from_deviceMsJ ;;& )5::6 6 [[F ")5::6 66{namec [URU5$![aCn[SURRSURRSUS35UeSnAff=f)NzCustom backend 'z' not implement 'torch.r')r(r+r-r,r*)r.r;r/s r0 __getattr___FSDPDeviceHandle.__getattr__[sk 4>>40 0  "4==#5#5"66MdmmN`N`Maabcgbhhij  s A%>A  A%) __backend__deviceN)__name__ __module__ __qualname____firstlineno____doc__r)r%r r1 classmethodr8strr>__static_attributes__r:r0r#r#9sX %u|| %c %  2E  3r:r#c0\rSrSrSSjrS\S\4SjrSrg) _UninitializedDeviceHandledr3NcgrBrKr.s r0r1#_UninitializedDeviceHandle.__init__es r:r;c[S5e)Nz-Trying to use an uninitialized device handle.) RuntimeError)r.r;s r0__getattribute__+_UninitializedDeviceHandle.__getattribute__hsJKKr:rKr3N) rCrDrErFr1rIr rTrJrKr:r0rMrMds LSLLr:rMc\rSrSrSSjrSrg) _FSDPStatelNc2[5Ul[5Ul[5UlSUlSUlSUlSUl[RUl SUl [RUl0Ul[ R"Ul['5Ul[+5UlSUlSUl0UlSUlSUlSUlSUlSUlSUl[A5Ul!/Ul"/Ul#SUl$g)NFr)%set_ignored_modules_ignored_params_ignored_buffer_names process_grouprank world_size _device_meshr FULL_SHARDsharding_strategy_use_orig_params TrainingStateIDLEtraining_state_unshard_params_ctxrFULL_STATE_DICT_state_dict_typer_state_dict_configr_optim_state_dict_config_is_root_handle_fully_sharded_module_to_handlecompute_device_gradient_predivide_factor_gradient_postdivide_factor _comm_hook_comm_hook_state_unshard_eventrM_device_handle_all_fsdp_states _all_handles_fsdp_extensionrPs r0r1_FSDPState.__init__ms1425%/2u"7; !/3!1!V>X%%) ?C   ,48/0'01(+/,0262L1M35CE6:r:)ryrzrurvrxrcr{rqrtrsrpr_r]r^rornrmrlrwrjrfrrr`rarerirbrV)rCrDrErFr1rJrKr:r0rXrXls$;r:rXmoduler3cN[U5nUb[U[5(dgU$rB)r isinstancerXr}states r0_get_module_fsdp_staters% f %E }Juj99 Lr:cR[U5nUcgX:XaU$XR;aU$grB)rrqrs r0._get_module_fsdp_state_if_fully_sharded_modulers4 #6 *E }   666 r:cB\rSrSrSr\"5r\"5r\"5rSr g)rgzM An enum that indicates the state of a ``FullyShardedDataParallel` instance. rKN) rCrDrErFrGrrhFORWARD_BACKWARDSUMMON_FULL_PARAMSrJrKr:r0rgrgs  6Dvr:rgc^\rSrSrSr\"5r\"5r\"5r\"5r \"5r Sr g)HandleTrainingStatez; An enum that indicates the state of a ``FlatParamHandle`. rKN) rCrDrErFrGrrhFORWARD BACKWARD_PRE BACKWARD_POSTrrJrKr:r0rrs- 6DfG6LFMr:rrc@[U[R5(+$rB)rnnModule)rs r0_is_composablers%+ ++r:rc[U5(aGURcgXR;a[SUSUR35eURU$UR$)z Returns the ``FlatParamHandle`` s corresponding to ``module``. This is the handle that contains some parameter in ``module``. Nz'Expects a fully sharded module but got z on rank )rrprqAssertionErrorrarr}s r0_module_handlersj e ==  >> > 9&5::,W 44V<<~~r:c[X5SL$)z5Returns if ``module`` has parameters managed by FSDP.N)rrs r0_has_fsdp_paramsrs % ( 44r:c,U(a UR$S$)z. Returns the sharding strategy of the handle. N)_sharding_strategy)handles r0_get_sharding_strategyrs)/6 $ $8D8r: tensor_namec^UR[S5nUR[S5nU$)zN Cleans the parameter or buffer name by removing any module wrapper prefixes. )replace FSDP_PREFIXr)rs r0clean_tensor_namers1 %%k26K%%&8"=K r:tensorc&[U[S5 g)z Sets an attribute on ``tensor`` to mark it as flattened by FSDP. This is to avoid re-flattening it during nested construction. TN)setattrFSDP_FLATTENEDrs r0_set_fsdp_flattenedrs  FND)r:c$[U[S5$)z;Returns if ``tensor`` has been marked as flattened by FSDP.F)r(rrs r0_is_fsdp_flattenedrs 6>5 11r:kwargsc SU;a [S5eSUS'[UR"S0UD65nU$![a0 URS5 [UR"S0UD65nU$f=f)zl This API is required as some modules overwrite `named_parameters()` but do not support `remove_duplicate`. remove_duplicatezR_named_parameters_with_duplicates cannot be used with `remove_duplicate` argument.FrK)rlistnamed_parameterspop)r}rrets r0!_named_parameters_with_duplicatesrsV# `  "'F 66**4V45 J 6 %&6**4V45 J6s56A/.A/modeldedup_shared_paramsc ^U4SjnSn0n[UUU[U5VVs/sHupVUPM snnU5$s snnf)a Constructs a mapping from parameter to a list of its "canonical" FQNs. Here, we use canonical to mean the fully-qualified name assigned to the parameter based on its position in the original nn.Module hierarchy before any wrapper or parallelism has been applied to it. This is in contrast to FQNs that may be generated after parallelisms or wrappers have been applied to the model. Each normal parameter maps to a singleton list containing its FQN, while each ``FlatParameter`` maps to a list of its original parameter FQNs, which may have length greater than one. All FQNs are prefixed starting from ``model``. In the case where FSDP was applied with ``use_orig_params=True``, there should be no ``FlatParameter`` s registered to the model's modules and this mapping will only contain mappings from ``nn.Parameter`` s to singleton FQN lists. It is only in the case where FSDP was applied with ``use_orig_params=False`` where a ``FlatParameter`` will be registered in place of the original parameters and there will be mappings from each ``FlatParameter`` to lists of FQNs corresponding to the original parameters. Args: model (torch.nn.Module): Root module (which may or may not be a :class:`FullyShardedDataParallel` instance). dedup_shared_params (bool): For shared parameters, if ``True``, only includes the FQNs corresponding to the first encounter of the shared parameter in the module traversal; if ``False``, then includes the FQNs across all encounters. (Default: ``True``) c>[USS9HupE[U[R5(a UROU/nUVs/sHn[ X-5PM nnXS;n U (dXU'Ma[U[R5(a[ R"SSS9 XU'MT (aMX5RU5 M gs snf)NF)recursezFlatParameter is being traversed more than once. This case should only happen when using DistributedModelParallel with FullyShardedDataParallel.) stacklevel) rrflat_param_file FlatParameter_fqnsrwarningswarnextend) r}prefix tree_level param_to_fqns param_nameparam local_fqnsr; global_fqnsis_shared_paramrs r0 module_fn%_get_param_to_fqns..module_fn4s!B E" J e_%B%BCC  \  >H=GT!&-0Z $4O"'2e$e_%B%BCCMMR$%  ,7%(,,!(// .return_fnZsr:)_apply_to_modulesr)rrrrparam_to_unflat_param_nameskey_s ` r0_get_param_to_fqnsrsSB$=LHJ   JJW#Nr:c^UcgURRmURRn/[R "U4SjU55QnU$)Nc3.># UH nTUv M g7frBrK).0p param_to_fqns r0 -_get_handle_fqns_from_root..{s&N 1|A s)_exec_order_datar flat_param_paramsr from_iterable)rr handle_paramsrrs @r0rrssP~))66L%%--MO5&&&N &NNOJ r: root_modulerr filter_fqnsc^^^^ Sm Ub`[5m UHPnURS5nUS:wdMT RUSUS-5 URSUS-5nUS:waM4MR S[RR S[ S[4UU U4SjjmT"US S /UQ70UD6 U"U0UD6$) a Performs a pre-order traversal of the modules in the hierarchy rooted at ``root_module``, applying ``module_fn`` at each module and finally returning a value using ``return_fn``. The traversal constructs the full module prefix name (e.g. "module.submodule." just like in model state dict) and makes that available to ``module_fn``. ``filter_fqns`` is used because some module may have its own prefix similar to ``FullyShardedDataParallel`` and the ``named_parameters()`` is overwritten to remove the prefix. Nrr[rr}rrc>T "XU/UQ70UD6 UR5HCupVUcM X-S-nUS-nT bUT ;aUS:XdUS:XaUnOUS:XaUnT "XgU/UQ70UD6 ME g)Nrrr_dmp_wrapped_moduler})named_children) r}rrargsrsubmodule_name submodule new_prefixnew_tree_levelffilter_prefixesrs r0r_apply_to_modules..fs&*>t>v>)/)>)>)@ %N 036J'!^N*_4'*@@)-BB%+ '83%+ i^ Ed Ef E)*Ar:rr)r\findaddr)rrrIint) rrrrrrfqnirrs ` @@r0rrs,(,O%C Ar'##C!a%L1HHS!a%(r' F%((//F3FCFF2k2q*4*6* d %f %%r:training_statescURU;a}SUSUR3nURS:XaP[U[R5(a[ SU35 [ SU35 [ R"5 [U5eg)z8Asserts that FSDP is in the states ``_training_states``.zexpected to be in states z but current state is rzAsserting FSDP instance is: zERROR: N) rirarrrprint traceback print_stack ValueError)rrmsgs r0_assert_in_training_statesrs ?2''88N##$ & ::?%++4UG<= GC5/ "  ! ! #o3r:modulesc$[5nUVs0sHo"[UR55_M nnUHQnSnUR5Hup&XBL=(a XF;nU(dMSn O U(dM@URU5 MS U$s snf)a  Returns: Set[nn.Module]: The subset of ``modules`` that are root modules (i.e. parent-less) with respect to the modules in the set itself. In other words, these are the modules in ``modules`` that are not the child of any other module in ``modules``. TF)r\ritemsr)r root_modulesr}module_to_submodulescandidate_moduleis_root_module submodulesis_child_modules r0_get_root_modulesr s$'5LHOPfC(8$99P#"6"<"<"> F .Q3C3Q !& #? >   - .$ Qs#B mixed_precisionrootmodule_classes_to_overridewrap_override_dictc ^[[U55n[5nUR5Hn[XA5(dMUR [ U55 X$lS[RS[RS[RS[R4SjmU4SjnU4SjnURUSS 9 URUSS 9 M U$) Ndtyper}xr3c[R"U5(aURU:XaU$UR[U'UR U5$rB)r)is_floating_pointrr!to)rr}rs r0cast_fn1_override_module_mixed_precision..cast_fnsB..q11QWW5EH/0ww$V,ttE{"r:cN>[[T[RU5U5$rB)rrr)float32)r}rrs r0forward_pre_hook:_override_module_mixed_precision..forward_pre_hooks(%--)PRVWWr:cX>U[;a[Un[[TX05U5$grB)r!rr)r}routput old_dtypers r0forward_post_hook;_override_module_mixed_precision..forward_post_hooks611 4V rPs99DD <<I6/8F,,!C' "3:2K2K2Mg//M((VL!2L%;%;P299d1B  II $  D   $  ,*, *biiHEV