EijjSSKrSSKrSSKrSSKrSSKJrJrJrJr SSK J r J r J r J r SSKrSSKJr SSKJs Js Jr SSKJs Js Jr SSKJs Js Jr SSKJr SSKJr SSK J!r! SSK"J#r# SSK$J%r%J&r&J'r'J(r(J)r)J*r*J+r+ SSK,J-r-J.r.J/r/J0r0 SS K1J2r2 SS K3J4r4J5r5J6r6J7r7J8r8J9r9J:r:J;r; SS KJ?r? SS K@JArA SSKBJCrC \ (aSSKDJErE SrFSSKGJHrHJIrI SrKSrL\M\R\R4rO\ \R\O-rP\9R\0R\9R\0R\9R\0R\9R\0R\9R\0R0rV\9R\9R/rW\9R\9R4rX\ S_S\&S\PS\9S\=S-S\!S-S\&4 Sjj5rY\ S\&S\PS\!S\&4Sj5rZ\ S\ S\[4Sj5r\\ S\!S\[4Sj5r]\ S\^S\R4S j5r_\ S!\RS\^S\R4S"j5r`S!\RS\^S\M\R\R44S#jra\ S_S\&S$\RS%\\R:RS-S&\\R:R\\R:R-S-S\&4 S'jj5rdS&\e\ S(\[SS4S)jrf\ S\&S$\RS*\g\RS+\^\R-S-S\&4 S,j5ri\ S\&S$\RS\&4S-j5rj\ S\&S\9S-S.\8S-S/\5S-S0\[S1\[S2\^S3\^S\&4S4j5rk\ S\&S\&4S5j5rl\ S\&S6\4S7\[S\&4S8j5rm\ S_S\&S\!S\&4S9jj5rn\ S\&S\&4S:j5roS$\RS;\e\RSS4S<jrp\ S\&S=\RS+\^\R-S-S>\\R/S4S-S?\[S\&4 S@j5rq\ S\&S;\e\RS=\R4SAj5rrSB\RSC\\R:RS-S\g\R4SDjrsS_SB\R:RS%\g\R:RSE\\R:RS-S\g\R:R4SFjjrtSB\R:RS%\g\R:RS\g\u4SGjrvSB\RS\g\u4SHjrwS$\RS*\g\RS+\^\R-S-SS4SIjrxS+\^\R-S-SJ\^SK\%S\RS-4SLjryS$\RS*\g\RS%\g\RS\M\[\[44SMjrzSB\RS>\\R/S4S%\g\RSS4SNjr{SB\RSO\RS-S%\g\RSK\%4SPjr|SB\RS%\g\RS\e\R4SQjr}S$\RS*\g\RSR\g\RSO\RS-SS4 SSjrS;\e\RST\e\RSO\RS-SS4SUjrSVrS$\RS*\g\RSO\RS-SJ\^SK\%S\R4 SWjrS$\RS;\e\RS\RSS4SXjrSY\e\RSS4SZjrS$\RS*\g\RS\\R4S[jrS*\g\RSS4S\jrS\94S]jrS\RS\GR4S^jrg!\Ja SrFGNf=f)`N)Callable GeneratorIterableIterator)Any no_type_checkOptional TYPE_CHECKING) default_hooks) DeviceMesh)_get_default_group)_FSDPDeviceHandle _FSDPState_get_module_fsdp_state_is_fsdp_flattened!_named_parameters_with_duplicatesclean_tensor_name TrainingState)_FSDP_USE_FULL_PREC_IN_EVAL FlatParameterFlatParamHandleHandleShardingStrategy)_FreeEventQueue)BackwardPrefetch CPUOffloadFullOptimStateDictConfigFullStateDictConfigMixedPrecisionShardingStrategyStateDictConfig StateDictType)_Policy)DTensorExtensions)_sync_params_and_buffers)is_traceable_wrapper_subclass)RemovableHandleT) deferred_initfakeFi _fsdp_syncedstate process_groupsharding_strategypolicy device_meshreturncFUbUb [S5eU[;nU(a%UcUcUc[SUS35e[XU5nO6U(aX@lUR SS9UlOUbUO [ 5UlUR R5UlUR R5Ul URnU(aX`RR5-n[RRU5UlX`R- UlU$)NzcCannot pass both process_group and device_mesh at the same time. Please just pass only one of them.zManual wrapping with zA requires explicit specification of process group or device_mesh.rmesh_dim) ValueErrorHYBRID_SHARDING_STRATEGIES*_init_process_group_state_for_hybrid_shard _device_mesh get_groupr+r ranksize world_size_inter_node_pgr DefaultState_get_gradient_predivide_factor_gradient_predivide_factor_gradient_postdivide_factor)r*r+r,r-r.is_hybrid_strategydata_parallel_world_sizes _/home/mande/repo/quber/.venv/lib/python3.13/site-packages/torch/distributed/fsdp/_init_utils.py_init_process_group_staterCYsA [%< <  +.HH  V^ 8K'(9':;SS  ?kE !, "-"7"7"7"CE "/!: @R@T  $$))+EJ**//1E$// $8$8$=$=$?? ""AA $  $ !#C#CC % LcU(aW[U5(a/X lURSS9UlURSS9UlO[ SUR 35eUc<[5n[X0RR55upEX@lXPlO7[U5(aUuUlUlO[ S[U535e[URS9UlU$)Nrr1z,Expected device_mesh to have ndim=2 but got zmExpected process_group to be passed in as either None or Tuple[dist.ProcessGroup, dist.ProcessGroup] but got r+)"_is_valid_hybrid_shard_device_meshr6r7r;r+r3ndimr !_init_intra_and_inter_node_groups_device_handle device_count_is_valid_hybrid_shard_pg_typetype_get_default_comm_hook_state_inter_node_state)r*r+r. default_groupintra_node_groupinter_node_groups rBr5r5s  -k : :!, $/#8#8!#8#DE "-"7"7"7"CE >{?O?O>PQ   *, -N //<<>. *// *- 8 89F 5E !5GGKMGZF[]  ;**E LrDc~[U[5=(a' [U5S:H=(a [SU55$)Nc3V# UHn[U[R5v M! g7fN) isinstancedist ProcessGroup).0pgs rB 1_is_valid_hybrid_shard_pg_type..s JMb 2t0011M'))rXtuplelenallrGs rBrMrMs: =%( K  ! # K JMJ JrDcN[U[5=(a URS:H$)NrU)rXr rI)r.s rBrHrHs k: . H;3C3Cq3HHrDnum_devices_per_nodec6[R"U5upU$)a5 Return a process group across the current node. For example, given each row is a distinct node: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 This API would return an intra-node subgroup across [0, 1, ..., 7] or [8, 9, ..., 15] depending on the process's rank. For example, rank 3 would get [0, 1, ..., 7]. )rY new_subgroups)rdintra_node_subgroup_s rB_init_intra_node_process_groupris"//0DE rDglobal_process_groupcdSn[R"U5n[R"U5nXA-n[R"U5U-n[ U5H?n[ U5Vs/sH oX--PM n n[R "XS9n Xv:XdM=U nMA Uc[ US35eU$s snf)al Return an inter-node process group where each contained rank has the same local rank. For example, given each row is a distinct node: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 This API would return inter-node process group [0, 8], [1, 9], [2, 10], and so forth depending on the process's rank. For example, rank 1 would get [1, 9], rank 5 would get [5, 13]. N)ranksbackendz. expected to assign inter-node pg, but did not)rY get_backendget_world_sizeget_rankrange new_groupAssertionError) rjrd inter_node_pgsharding_backendr: num_nodes my_local_rank local_rankiranks_for_inter_groupgrps rB_init_inter_node_process_groupr|s M''(<=$$%9:J2IMM"67:NNM01 =B9=M! =M!2 3=M ! nn#8S  &M2oK L   ! s'B-c.[U5[X54$)a Initialize intra and inter-node process groups and return the ones corresponding to this process's rank. This function can be used to initialize process groups for ``HYBRID_SHARD`` or ``_HYBRID_SHARD_ZERO2`` in FSDP. This function assumes each node has an equal number of CUDA-enabled devices. Returns: Tuple[dist.ProcessGroup, dist.ProcessGroup]: Intra and inter-node process group. )rir|)rjrds rBrJrJs  '';<&';R rDmoduleignored_modulesignored_statescUbUb [S5eSnUSLnU(a[U5n[US5 O/n[Ub [U5O/S5 [U5S:a'[ US[ R 5(aUnOUn[X5Ul[UURU5Ul [UUR5Ul U$)NzfCannot pass both ignored_modules and ignored_states at the same time. Please just pass ignored_states.TFr) r3list_check_ignored_statesrarXnn Parameter_get_ignored_modules_ignored_modules_get_ignored_params_ignored_params_get_ignored_buffer_names_ignored_buffer_names)r*r~rrignored_parameterspassed_as_ignored_statesignored_states_lists rB_init_ignored_module_statesrs"~'A :  -T9">2148 %4%@D !b%  !# )!,bll ; ;!4 1O1&JE/ E #< #E LrDrc[U5S:XagU(aj[SU55n[SU55nU(d>U(d6[UVs1sHn[U5iM sn[S9n[ SU35egg[SU55(d6[UVs1sHn[U5iM sn[S9n[ SU35egs snfs snf) z Check that the ignored states are uniformly parameters or uniformly modules. We may remove this check in the future if we permit mixing. rNc3V# UHn[U[R5v M! g7frW)rXrrr[r*s rBr](_check_ignored_states..JsUnUE2<<88nr_c3V# UHn[U[R5v M! g7frWrXrModulers rBr]rKsSN5*UBII66Nr_)keyzUignored_states expects all nn.Parameter or all nn.Module list elements but got types c3V# UHn[U[R5v M! g7frWrrs rBr]rTsL^E:eRYY//^r_z>ignored_modules expects nn.Module list elements but got types )rarbsortedrNreprr3)rr all_params all_modulesr* sorted_typess rBrr?s >aUnUU SNSS +!N"KN54;N"KQUVL**69 #.zL^LLL!N"KN54;N"KQUVL%( M #L#Ls C"Cignored_params device_idcbSnUb7[U[R5(aUO[R"U5nUc[X5HnURRS;aMUcURnM0URRUR:wdMV[ SURSURR35e U=(d [R R5nURS:Xa [ S5e[R"U5Ul U$)a Determine device handle used for initializing FSDP. If a device is specified by ``device_id``, then returns device handle corresponds to that device type. Otherwise, If the module is already on a non-CPU device, then the device type is that non-CPU device type. If the module is on CPU or meta, then the device type is the current accelerator device. See the :ref:`Accelerators` for details. This method will be called once ignored parameters was determined, as the device handle maybe needed for other initialization. N>cpumetazLFSDP does not support modules with different device types but got params on z and rzOFSDP needs a non-CPU accelerator device, but no accelerator device is detected.) rXtorchdevice_get_orig_paramsrN RuntimeError_C_get_acceleratorr from_devicerK)r*r~rrdetermined_deviceparams rB_init_device_handler\s()U\\22 i(   %f=E||  O3 ($)LL!<<$$(9(>(>>&-->-C-C,DE%,,J[J[I\^>.L1J1J1L  ! !U *a -889JKE LrDc[U5Ul0nUR5Hup4[U5nURX#'M X lU$rW)_get_buffer_names _buffer_names named_buffersrdtype_buffer_name_to_orig_dtype)r*r~r buffer_namebuffers rB_init_buffer_statersT ,F3E :<%335 ' 4 28,,"/ 6(B$ LrDmixed_precision cpu_offloadlimit_all_gathersuse_orig_paramsbackward_prefetch_limitforward_prefetch_limitcURS:XaUU[R:wa0[R"SU=(d [R S3SS9 [RnO.U[R:Xa[R"S[ SS9 U=(d [R UlU=(d [5Ul Ub5[RRS[UR535 [RR!["S 5S :HUlU=(d ['5UlX@lXPl[.R0UlSUl[75Ul[:R<"5Ul[@RB"UR>UU5Ul"SUl#0nXl$Sn Xl%/n Xl&U$) NrFz/FSDP is switching to use `NO_SHARD` instead of z since the world size is 1.rU stacklevelzoThe `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.z'torch.distributed.fsdp.mixed_precision.1)'r:rNO_SHARDwarningswarn FULL_SHARD FutureWarningr,rrrr_log_api_usage_oncestrosenvirongetr_use_full_prec_in_evalrrr_use_orig_paramsrIDLEtraining_state_is_rootr_free_event_queuerYget_debug_level _debug_levelexec_order_utils_ExecOrderData_exec_order_data_unshard_event_fully_sharded_module_to_handle_handleparams) r*r,rrrrrrrrrs rB_init_core_staters 1  0 9 9 9 MMA$C(8(C(CDE''   -55 .77 7  <   0N3C3N3NE+?~/?E" $$5c%:O:O6P5Q R  2B73>  $3z|E/,(--EEN-/E--/E-<< E  EIK#,K)'+GM"$FL LrDc`/nXl/nX l/nX0lSUlSUlSUlU$)NT)_root_pre_forward_handles_pre_forward_handles_post_forward_handles_sync_gradients _comm_hook_comm_hook_state)r*rrrs rB_init_runtime_statersE8:&?#24!535"7 EE!E LrDbackward_prefetchforward_prefetchcXlX lU$rW)rr)r*rrs rB_init_prefetching_staters 0- LrDcUbUR5OSnU(a+X R:wa[UR5UlU$SUlU$rW)_get_root_meshr6r#rK_fsdp_extension)r*r. root_meshs rB_init_extensionrsR 1<0G **,TIy$6$66 1%2F2F G L!% LrDc~[RUl[5n[ 5UlXl0nX lU$rW)r!FULL_STATE_DICT_state_dict_typerr_optim_state_dict_config_state_dict_config_unshard_params_ctx)r*state_dict_configunshard_params_ctxs rB_init_state_dict_staters;*::E)<)>%=%?E"057 2 LrDrcUH`n[UR5S:XdMSnUR5H upEX%LdM Un O U(d [S5e[ SUS35e g)z Verify if the parameters are accepted by FSDP. The only restriction now is that the parameter cannot be a scalar tensor (param.shape == []). rrzExpected param_name to be setz/FSDP doesn't support scalar parameters. Change z& to a 1D tensor with numel equal to 1.N)rashapenamed_parametersrsr3)r~rr param_namenameparam_s rB_verify_managed_paramsrsw  u{{ q J & 7 7 9 ?!%J!:$%DEE$%KM rDfully_sharded_module param_init_fnsync_module_statesc^[UTRU5 [UTRTR5n[ UTRTR 5upgU(dU(aUb[XTR 5 OJU(a#[UUTR TR5 O U(a[R"UU4SjS9 TR VV s1sHnUR5Hn U iM M n nn [UTRU U5 [UTRUTRTR5Tl[[!UTR55n [#X5 U(a@[%XTR&5 TR([*;a[%XTR,5 [/TX5 T$s sn nf)zHInitialize a ``FlatParamHandle`` from a module ``fully_sharded_module``.cJ>[U5SL=(a UTR;$rW)rr) submoduler*s rB0_init_param_handle_from_module..Ps('=i'HD'P(8!7!77(8rD)check_fn)_check_single_device_moduler_get_device_from_device_idr8rK_need_to_materialize_moduler_materialize_with_param_init_fn_materialize_meta_moduler'materialize_modulebuffers_move_module_to_device_get_compute_devicecompute_devicerrr_sync_module_params_and_buffersr+r,r4r;_init_param_handle_from_params) r*rrrrdevice_from_device_idis_meta_moduleis_torchdistX_deferred_initignored_modulerignored_buffersmanaged_paramss ` rB_init_param_handle_from_moduler1s 4e6K6KYW65::u333Ne33U5K5K3/N 5=;T' 1G1G     " "   %(( 8 $444N$,,.F . 4    /    E*+?AVAVWXN/@' %2E2E   " "&@ @ +$e6J6J #5.O L?s$Gc[U5S:Xag[UUUR[URUR R URRURRURRURURURS9 nUR5 UR(a [!S5eUR"R%UR&5 X0lX0R(UR*'[,R."S5nUR R (a-UR&R.U:waUR1U5 ggg)Nr)fsdp_extensionz!Expected state._handle to be Noner)rarrSHARDING_STRATEGY_MAPr,roffload_paramsr param_dtype reduce_dtypekeep_low_precision_gradsr+rrshardrrsrappend flat_paramr_fully_sharded_modulerr flat_param_to)r*rrhandle cpu_devices rBrrvs'  6{a  e556 (( )) ** 66  ,, F LLN }}@AA LL))*MJP))&*F*FGe$J ''F,=,=,D,D ,RZ(-S'rD root_modulerc SnUb [U5O [5nUHan[U[R R 5(d[US[U53-5e[U5(dMX[S5e UR5H1n[R"U5(aM URU5 M3 UVVs1sH>nUR5H&n[U[R5(aM$UiM( M@ nnnX;a[R "SW3SS9 UR5HJn[U5n U cM[#U S 5(d [%S 5eUR'U R(5 ML U$![a n[US[U53-5UeSnAff=fs snnf) aP Check that ``_ignored_modules`` is an iterable of ``nn.Module`` s without any FSDP instances. Return the modules contained in their module subtrees as a :class:`set`. Nested FSDP instances are excluded, but their already-computed ignored modules are included. ``_ignored_modules`` represents the argument passed by the user to FSDP. z>`ignored_modules` should be an iterable of `torch.nn.Module`s Nzbut got zbut got an iterable with z1`ignored_modules` should not include FSDP moduleszTrying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: rUrrz?Expected optional_fsdp_state to have _ignored_modules attribute)set TypeErrorrNrXrrrrr3modulestraversal_utils _composableadd fsdp_fileFullyShardedDataParallelrrhasattrrsupdater) r'r msg_prefixignored_root_moduleser~childrroptional_fsdp_states rBrrsRJQ%5%AC !su  '&%((//22J+DT&\N)SST T !& ) )PQ Q '%%'**622 $ $V ,(+*F^^%E%!C!CD % * %  228 ;  !((* 4Y?  *.0BCC$U  " "#6#G#G H+ Q Q x5E0F/G%HHIqPQ$s#F 7F? F? F<F77F<rc[5nUVVs1sH.oDR5Hn[U5(aMUiM M0 nnnURU5 Ub5UVs1sHn[U5(aMUiM nnURU5 UR 5HJn[ U5n U cM[ U S5(d [S5eURU R5 ML U$s snnfs snf)z Return the parameters of the modules in ``ignored_modules`` and the parameters in ``ignored_parameters``. :class:`FlatParameter` s are excluded from the result. rz>Expected optional_fsdp_state to have _ignored_params attribute) r) parametersrr2r+rr1rsr) r'rrall_ignored_paramsmpparams_in_ignored_modulesparams_in_ignored_parametersrr7s rBrrs36%#!"aLLNqBTUVBWN?!78%)( )!1CA1FA) %(  !!">?!((* 4Y?  *.0ABB$T  % %&9&I&I J+ -!( s'C1 C1 C78C7c [5nUVVs1sHo3R5HoDiM M nnnURUR5VVs1sHupdXE;dM [ U5iM snn5 UR 5HJn[ U5nUcM[US5(d [S5eURUR5 ML U$s snnfs snnf)z6Return the cleaned buffer FQNs in ``ignored_modules``.rzDExpected optional_fsdp_state to have _ignored_buffer_names attribute) r)r r2rrr+rr1rsr) r'rall_ignored_buffer_namesr;rbuffers_in_ignored_modulesrrr7s rBrrs *-("'1iikFk"##(3'@'@'B 'B# 3 + k *'B !((* 4Y?  *.0GHH$Z % + +,?,U,U V+ $#-"  s"C C "C cjUR5VVs1sHup[U5iM snn$s snnf)zrReturn the fully prefixed names of all buffers in the module hierarchy rooted at ``root_module`` as a class:`set`.)rr)r'rrhs rBrrs8>I=V=V=X =X>;+&=X  s/c[X5Vs1sHo3RiM nn[U5S:Xa)[R"S5U;aUc [ S5eg[U5S:a[ SU35egs snf)z Raise an error if ``module`` has original parameters on multiple devices, ignoring the parameters in ``ignored_params``. Thus, after this method, the module must be either fully on the CPU or fully on a non-CPU device. rUrNzTTo support a module with both CPU and GPU params, please pass in device_id argument.rFz;FSDP only supports single device modules but got params on )rrrarr)r~rrrdevicess rBrrs*:&)QR)Q||)QGR 7|qU\\%0G;  5   W I' S   SsA;r8 device_handlec rUcg[U[R5(aUO[R"U5nURS:wakURc^[ R "SUSUSUR5SURS3 SS 9 [R"UR55nU$) z Return a ``torch.device`` for the specified ``device_id``. Processes ``device_id`` and returns either the corresponding device or ``None`` if ``device_id`` is ``None``. Nrz"FSDP got the argument `device_id` z on rank zJ, which does not have an explicit index. FSDP will use the current device z6. If this is incorrect, please explicitly call `torch.zk.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.rUr)rXrrrNindexrrcurrent_device)rr8rErs rBrr=s 5<<88 ell9>U {{e 4 0 9f00=0L0L0N/OPCCI;;-P1 1  m::<= MrDc2[[X55n[SU55nUR5H.nXR;aM UR SS9HnXFR -nM M0 U(+=(a [ =(a [SU55nXG4$)z Return if ``module`` has parameters on meta device and if ``module`` is using torchdistX deferred initialization. At most of the returned bools can be ``True``. If either is ``True``, then ``module`` needs to be materialized. c38# UHoRv M g7frW)is_metar[rs rBr]._need_to_materialize_module..hsCN5NsFrecursec3N# UHn[R"U5v M g7frW)r(is_fakerLs rBr]rMts@ U##s#%)rranyr+r rK_TORCHDISTX_AVAIL)r~rrrrrbufrs rBrr[s*6BCNCNCCN^^%  ' $$U$3C kk )N4&  A  A @@ @  66rDc[U5(d[SUS[U535e[X5nUH nU"U5 M g)Nz Expected z to be callable but got )callabler3rN_get_modules_to_materialize)r'rrmodules_to_materializer~s rBr r ysR M " " &>tM?R>S T  9V(f)rDrc PU=(d$ [R"UR55n[X5nSn[R"5 UHtn[ R "URSS9URSS95n[[U55S:nU(dMTURUSS9 UR5 Mv SSS5 g!,(df  g=f![a5n [R"S[!U 5S[#U5S3SS 9 U eSn A ff=f) NFrNr)rrOzIUnable to call `reset_parameters()` for module on meta device with error z(. Please ensure that your module oftype z* implements a `reset_parameters()` method.rUr)rrrHrWno_grad itertoolschainr9r rarto_emptyreset_parameters BaseExceptionrrrrN) r'rrrEmaterialization_devicerXr~module_state_iterhas_module_statesr5s rBr r s3ell$$&79V F]]_0%.OO%%e%4NN5N1%! %(->(?$@1$D!$$OO+A5OQ++-1__   !!$Q)L>!K M  s<C&AC($C C& C#C&#C&& D%00D  D%cJ/n[R"U/5nU1nU(aUR5nURU5 UR 5HAnXd;dM [ U5bMXa;dMUR U5 URU5 MC U(aMU$rW) collectionsdequepopleftr!childrenrr.)r'rrXqueuevisited_modulesr~ child_modules rBrWrWs /1   {m ,E'2mO %%f-"OO-L3*<8@ 7##L1 \*. % "!rDrc^ [R"S5m UGb"[R"5nUR U5 /n/nU(aUR 5nUR U 4SjURSS955 UR U 4SjURSS955 UR5H5n[U[R5(aM$UR U5 M7 U(aMUV s/sH oU;dM U PM n n UV s/sH oU;dM U PM n n [XU5 g[[X5S5n U bU RT :Xa [!5 gggs sn fs sn f)a Move ``module`` depending on ``device_from_device_id`` and its current device. This includes moving ignored modules' parameters. - If ``device_from_device_id`` is not ``None``, then this moves ``module`` to the device. - If ``device_from_device_id`` is ``None``, then this does not move ``module`` but warns the user if it is on CPU. Precondition: ``_check_single_device_module()``. rNc3L># UHnURT:XdMUv M g7frWr)r[rr&s rBr])_move_module_to_device..s&BE<<:-B$ $FrNc3L># UHnURT:XdMUv M g7frWrm)r[rr&s rBr]rns&@F==J.@ro)rrrdrer!rfextendr9r rgrXr/r0_move_states_to_devicenextr_warn_cpu_init)r~rrrrhrr  curr_modulerr<params_to_move bufs_to_moverr&s @rBr r sV$e$J(/:.?.?.A V%'&(--/K MM(33E3B  NN)11%1@  )113 !)Y-O-OPPLL+4!e&&,GV/F!VG#*G7a.F7 G~=RS !&94 @E U\\Z78 HGs E3E3 E8)E8r c\[U5S:Xa[U5S:Xag[U5S:aUSRnO[U5S:aUSRn[R"S5nUbUHnn[R"5 UR U5UlUR b*UR R U5UR lSSS5 Mp UHnUR U5UlM gWU:Xa [5 gg!,(df  M=f)z Move states to the specified device. Precondition: ``_check_single_device_module()`` and module's parameters and buffers have been materialized if needed. rNr)rarrrZtodatagradrt)rr rrHr&rrs rBrrrrs 6{aCLA- 6{Q)) W  **e$J(E"XX&;< ::)&+jjmm4I&JEJJO! F ))$9:FK : % & !s AD D+ c.[R"SSS9 g)NamThe passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.rUr)rrrDrBrtrts MM 1 rDc[[X5S5nUb'URRS:wa URnO$[R"UR 55nUbXb:wa[ SUSUSU35eU$)a Determine and return this FSDP instance's compute device. If the module is already on a non-CPU device, then the compute device is that non-CPU device. If the module is on CPU, then the compute device is the current device. Since this method should be called after materializing the module, any non-CPU device should not be meta device. For now, the compute device is always a CUDA or CUDA-like device with its explicit index. Precondition: ``_check_single_device_module()`` and ``_move_module_to_device()``. Nrz4Inconsistent compute device and `device_id` on rank z: z vs )rsrrrNrrHr3)r~rrr8rErrs rBrr%s* !&94 @E U\\..%7m&B&B&DE(^-TB4&d#8"9 ;   rDc/nUR5Hn[U[S5(aM[U[S5 UR 5n[ U5(a@UR 5upgUVs/sHn[XX5PM n nURU 5 MURU5 M UHtn U R 5n [ U 5(a@U R 5upgUVs/sHn[X5PM n nURU 5 McURU 5 Mv [U5 [UU[SS9 gs snfs snf)z Synchronize module states (i.e. parameters ``params`` and all not-yet-synced buffers) by broadcasting from rank 0 to all ranks. Precondition: ``sync_module_states == True`` and ``self.process_group`` has been set. FTr)srcN) r getattr FSDP_SYNCEDsetattrdetachr%__tensor_flatten__rqr!+_check_module_states_for_sync_module_statesr$PARAM_BROADCAST_BUCKET_SIZE) r~rr+ module_statesrdetached_bufferattrsrhattr inner_buffersrdetached_param inner_paramss rBrrHs#)+M.."v{E22 FK .$mmoO,_==+==?LQ RED!?E R$$]3$$_5# ( 8 8%88:HEFKLedGN9eLL   .   00 ># !SMs 5E /ErcXU(a#[SU55(a [S5egg)Nc3f# UH'oR[R"S5:Hv M) g7f)rN)rr)r[tensors rBr]>_check_module_states_for_sync_module_states..ws#;H e,,=s/1zThe module has CPU parameters or buffers when `sync_module_states=True`, which requires them to be on GPU. Please specify the `device_id` argument or move the module to GPU before passing it to FSDP.)rRr3)rs rBrrts<;H C  }rDc## UR5n[U5nX1;a[U5(dUv M&![a gf=f7f)a, Return an iterator over the original parameters in ``module``. The iterator does not return the parameters in ``ignored_params``, any ``FlatParameter`` s (which may be present due to nested FSDP wrapping), or any original parameters already flattened (only relevant when ``use_orig_params=True``). N)r9rsr StopIteration)r~r param_genrs rBrrsT!!#I OE*3Ee3L3L     s A '; AA AA c [U5HHup#X1;dM [U5(aM[SUSUR5SUR35e g)a Check that original parameters in ``fsdp_module`` have been flattened. The flattened parameters are made invisible to ``named_parameters()`` for the module hierarchy rooted at ``fsdp_module``. This should be called as a sanity check after flattening the wrapped module's parameters. z Found an unflattened parameter: z;  N)rrrr9 __class__) fsdp_modulerrrs rB_check_orig_params_flattenedrsY?{K  &/A%/H/H2:,b::<.%//!24 LrDcjU[R:Xa[R$[R$rW)rrr allreduce_hookreduce_scatter_hook)r,s rB_get_default_comm_hookrs3  0 9 9 9 $$ . .rDc*[R"US9$)NrG)r r<rGs rBrOrOs  % %M BBrDrW)rdr[rrcollections.abcrrrrtypingrrr r rtorch.distributed distributedrY(torch.distributed.fsdp._exec_order_utilsfsdp_exec_order_utilsr'torch.distributed.fsdp._traversal_utils_traversal_utilsr,2torch.distributed.fsdp.fully_sharded_data_parallelfully_sharded_data_parallelr/torch.nnr(torch.distributed.algorithms._comm_hooksr torch.distributed.device_meshr "torch.distributed.distributed_c10dr $torch.distributed.fsdp._common_utilsrrrrrrr"torch.distributed.fsdp._flat_paramrrrr%torch.distributed.fsdp._limiter_utilsrtorch.distributed.fsdp.apirrrrrrr r!torch.distributed.fsdp.wrapr"&torch.distributed.tensor.parallel.fsdpr#torch.distributed.utilsr$torch.utils._python_dispatchr%torch.utils.hooksr&rS torchdistxr'r( ImportErrorrrr`rZHybridShardProcessGroupTypeProcessGroupTyperr SHARD_GRAD_OP HYBRID_SHARD_HYBRID_SHARD_ZERO2rr4#NO_RESHARD_AFTER_FORWARD_STRATEGIESrCr5boolrMrHintrir|rJrrrrrr)rrrrrrrrrrrrrrrrrrrr r rWTensorr rrrtrrrrrrr<rOr}rDrBrs  CC>> CCAAFFB4A B   0D<F1.0 #D$5$5t7H7H$HID--0KKL 5>>!7!B!B""$:$H$H!!#9#F#F((*@*T*T !!(( ""(('# &* 0 0#0(0 dN 0 d" 0  00f( (#(( ((V#$IJI4II  ARAR  "++"" ""J++ 4  d// /0& + + II+ehhoo.5+UXX//0uxx   +++\I9= :- - II- %-U\\!D( -  --`  II"? ?'$.?$d*?d" ?  ?  ?!? ???D         '     : J *  *299d2<<6HT(A A))AU\\!D(ARYYK-.5 A  A  AAH) )  )))))>::uxx/$6: ^:@?C"")"!!3!34t;"    "J$$)$ X$@299S II  % U\\!D(   <U\\!D( % \\D <7 II7 %7^7 4: 7<  RYYK-. ^  ## <<$.#^#% #L""-0^" "))_",3 II3 %3&3!<<$. 3  3l   %,, !<<$. @  II %!<<$.   %   \\ F) II)  )$$) )X   %      II  % bll , % (.>C$$CCw#s^&&^21^2