Ei* SSKrSSKrSSKrSSKrSSKJr SSKJr SSKJr SSK J r SSK J r J r SSKJrJrJrJrJrJrJr S\ R,S\\-S \\ R,S \\ R0S \\\4S \4 S jrS\ R,4SjrS\\\ R,4SjrS\ R,S\\ R,S \\ R0S\4Sjr S\ R,S\!\"\\ R,44Sjr#S\ R,S \\ R0S\\ R,S\S\\ R0\44 Sjr$g)N)Callable)partial)Any)_get_module_fsdp_state _override_module_mixed_precision)_construct_wrap_fn _or_policy_Policy_post_order_apply_recursive_wrap$_run_mixed_precision_override_policy_wrap_module_cls_individually root_modulepolicyignored_modulesignored_params root_kwargsfsdp_fnc USn[R"U5n[U5 [U[5(aSXG(aSOS'UR XU5nUb9[ UURUUU5n[XR5n [U 5 URSS5n [U[UR55UU 5 [XU5n [X 5 gUUUUUSS.n UbU[XR5n [ R""[$U[#[&URS 9/S 9nXS'[U 5 [)S 0U DUD6 g) a' Auto wraps modules in ``root_module`` 's tree according to ``policy`` following a post-order traversal. Precondition: ``root_kwargs`` should contain all arguments except ``module``. This function accepts the kwargs dict directly since it gets forwarded into the post-order traversal function. mixed_precisionNauto_wrap_policyruse_orig_paramsFT)moduler wrapper_clsrronly_wrap_children)module_classes)policies)inspectisclass_check_nested_wrapping isinstancer _run_policyr _module_classes_to_ignorer#_warn_on_overridden_mixed_precisionget_validate_frozen_paramssetkeysrr functoolsrr rr ) rrrrrrr is_wrappertarget_module_to_kwargsoverridden_module_classesrwrap_fnrecursive_wrap_kwargss _/home/mande/repo/quber/.venv/lib/python3.13/site-packages/torch/distributed/fsdp/_wrap_utils.py _auto_wrapr1s ""34O)J;'&'""FJ *&(C"("4"4 +#   &&J99' ' #)IFF) % 00I J%//*;UC  ',,. /    %[7S+/"*(" "%E BB% !"" 1#2#L#L  5;01+,EF;+;{;cpUR5H"up[U5cM[SUSU35e g)NzOFSDP auto wrapping requires modules to not already have FSDP applied but found z in ) named_modulesr ValueError)r module_namers r0r!r!isF*88: !& ) 5**5eK=J  ;r2r-cV[U5S:Xag[R"SUS3SS9 g)NrzvBoth mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type: zX These modules will be wrapped as separate FSDP instacnes with mixed precision disabled. stacklevel)lenwarningswarn)r-s r0r%r%rs< $%* MM ==VIf possible, wrap the frozen parameters with FSDP separately. z2The following parameters have requires_grad=True: z4 The following parameters have requires_grad=False: r8r9N) _get_post_order_named_modulesr(_get_managed_param_to_fqnitems requires_gradappendnumelr;r<r=r5)rr>rrpost_order_named_modulesvisited_modulesr6r param_to_fqnfrozen_param_fqnsfrozen_param_numelnonfrozen_param_fqnsnonfrozen_param_numelparamfqnmsgtotal_param_numels r0r'r'sg =[I&)eO7  $4L,. !" .0 $% !*002 &&(//4)[[]:)%,,S1&++-7& 3$%)c2F.G!.K$ %\]"(:(R%-..@AV@WX@@CffCXXIJ^I_`KK\J]_#MM#!4$S/)I 8r2returncVU1nSU4/n/nU(aUR5upEURXE45 UR5HBupgUcM Xq;dMURU5 US:waUS-U-nURXg45 MD U(aM[ [ U55nU$)a This returns the named modules following a post-order traversal, which is a valid reverse topological sort. We achieve this using the reverse of a stack-based DFS order instead of reversing ``root_module.named_modules()`` since the former gives the modules in registration order at each level in the module tree (as opposed to the reverse), which allows us to error/warn on the first registered module that violates the condition. For example, consider the following module structure: M( S1(), S2( SS1(), SS2(), ), S3(), ) The reverse DFS order is [S1, SS1, SS2, S2, S3, M], while the reverse ``named_modules()`` order is [S3, SS2, SS1, S2, S1, M]. .)poprDnamed_childrenaddlistreversed) rrGstack reverse_post_order_named_modulesr6rchild_module_name child_modulerFs r0r@r@s.#mO+  EDF$ #iik (//0EF/5/D/D/F + #2##L1"$(3c(9?0G % $H-M$NO ##r2module_to_wraprG root_prefixc0n[R"X4/5nURU5 U(aUR5upgUR SS9H upX;dM US:XaUOUS-U-n XU 'M" UR 5HDupU cM X;dMURU 5 US:XaU OUS-U -n UR X45 MF U(aMU$)a This returns a dict that maps managed parameter to its FQN for the given ``module_to_wrap``. The dict's keys are exactly the parameters that would be managed by the module, where this is achieved by calling this function on the modules to wrap in reverse topological order, destructively updating ``visited_modules``, and not traversing into those modules. The FQNs are prefixed from the root (via ``root_prefix``) to be more informative. NOTE: This function is meant to be called pre-wrapping and iteratively in reverse topological order to cover the full module tree. This differs from the ``_get_param_to_fqn()`` function meant to be called post-wrapping and on the full module tree in one shot. Given those differences, we do not try to unify the two. F)recurserSrT) collectionsdequerWpopleftnamed_parametersrVrD)r^rrGr_rHqueuerprefix param_namerMrNr\r] child_prefixs r0rArAs(-/L   <= >E' !'!8!8!8!G J*$*bLjfslZ6O&)U#"H06/D/D/F + #2##L1|&#(99  l9:0G %" r2)%rbr*rr<collections.abcrrtypingrtorch.nnnn$torch.distributed.fsdp._common_utilsrrtorch.distributed.fsdp.wraprr r r r r rModuler( Parameterdictstrr1r!typer%boolr'rXtupler@rArr2r0rws$L<L< w L<^L< % L< c3h L<  L<^  "4 ?3 2*2*^2* %2* 2*j'$'$ %RYY  '$T)II) %)^) )  ",,  )r2