EioSSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSKJ r SSK J r SSK J r JrJrJr SSKrSSKJr SSKJr SSKJr S S KJrJrJr S S KJr \(aS S KJrJr S S K J!r!J"r"J#r#J$r$ S SK%J&r&J'r'J(r(J)r)J*r*J+r+J,r, S SK-J.r. \R^"\05r1\RdRg\0S5r4\(aSSK5J6r6 SESjr7SFSjr8SFSjr9SFSjr:SFSjr;\ "SS55rSISjr?SJSKSjjr@SrASLSjrBSrCSMS jrDSNS!jrESOS"jrFS#rGS$rHS%rISPS&jrJSQS'jrKSRS(jrLSSS)jrMSTS*jrNSUS+jrOSVS,jrPSWS-jrQSXS.jrRSYS/jrSSZS0jrTS[S1jrUS\S2jrV\ "S3S455rWSXS5jrXS]S6jrYS^S7jrZSTS8jr[S_S9jr\SWS:jr]S`S;jr^SFS<jr_SaS=jr`S>raS?rbSFS@jrcSbSAjrdScSBjreSCrfSdSDjrgg)e) annotationsN) defaultdict) dataclass)AnyOptional TYPE_CHECKINGUnion)trace_structured)StorageWeakRef) OrderedSet)config config_commsir)WeakDep)IRNode Operation)estimate_peak_memory_allocfreeFreeableInputBufferget_freeable_input_buf SNodeMemory)contains_collective contains_waitfind_recursive_deps_of_nodefind_recursive_users_of_node is_collectiveis_fallback_opis_wait)Voverlap)BaseSchedulerNodecVSSKJn 0n0nUH&nUR5X$'U"U5cM X$X4'M( SSKJn SSKJn UR5nU"5n[U5V s/sHn /PM n n URU [UR55U5 [R"[R"U 5SS9RR5n [!UR#55H upXX4'M UHnXC;aX4X$'X$UlM gs sn f)Nr)_get_mm_like_fn)_get_default_group)dim)torch._inductor.schedulerr#get_estimated_runtimetorch.distributed distributed"torch.distributed.distributed_c10dr$get_world_sizerangeall_gather_objectlistvaluestorchmediantensortolist enumeratekeysoverride_estimated_runtime) snodesr#runtime_estimationsruntime_estimations_for_mmssnodedistr$ world_sizepg_$gathered_runtime_estimations_for_mms"median_runtime_estimations_for_mmsidxs R/home/mande/repo/quber/.venv/lib/python3.13/site-packages/torch/_inductor/comms.py6align_runtime_estimations_across_all_distributed_ranksrC4s3:"$%*%@%@%B" 5 ! -1D1K ' . %E$$&J  B*%?%q%)? , ( / / 12  */ 9:* fVVX' ; @ @ BC -O-T#*D  /)D)K  &+>+E(?s' D&c[USSSS9$)z/ Greedily schedules waits as late as possible. FT raise_comms sink_waitsreorder_for_overlap_schedule_for_commr7s rBrGrGZs Ed c[USSSS9$)z0 Greedily schedules comms as early as possible. TFrErIrKs rBrFrFcs DU rLc[USSSS9$)a This achieves the following overall scheduling procedure: Step 1: Given that we've currently scheduled comm N, we now schedule all compute nodes that are required for comm N + 1 but do not depend on comm N, to run at the same time with comm N. Step 2: If all those compute nodes are sufficient to overlap comm N, we're done. Otherwise, we now need to look elsewhere to find compute that overlaps with comm N. We prioritize compute nodes that are needed sooner. Step 3: We schedule the compute nodes dependent on comm N and required for comm N + 1. Step 4: We schedule comm N + 1. Repeat this for subsequent comm nodes. TrErIrKs rBreorder_compute_for_overlaprOls DTt rLc [U5upU$)a+ Reorders communication ops relative to computation ops to improve communication-compute overlapping and hide comm latency. Stops moving a particular op if it reaches a point that would have increased the peak memory footprint. Currently, follows these heuristics (subject to change or tune): - never reorders collectives relative to one another, for SPMD safety - has an option for per-collective prefetch limit, but does not enable it by default - limits the total number of reorder steps to some factor of the graph size to prevent worst-case quadratic performance Prerequisite: sink_comms_and_waits - ensure comm and wait nodes are scheduled as late as possible, respecting data dependencies. That allows reorder_communication_preserving_peak_memory to take a best case peak-memory snapshot, and then monotonically improve latency by moving collectives backward in time. Peak memory impact is computed in an iterative fashion. First, memory use at each timestep is computed, and global peak memory is computed as a max over timesteps. Then, when swapping any two adjacent nodes, only the curr-memory for the earlier of the nodes after the swap is affected. This enables checking step by step whether a swap is peak-memory-safe, and bailing out if not. Example: 0 n0 C0 1 n1 C0 + Allocs(n1) - Frees(n1) 2 n2 C0 + Allocs(n1) - Frees(n1) + Allocs(n2) - Frees(n2) 0 n0 C0 1 n2 C0 + Allocs(n2) - Frees(n2) <-- After moving n2 to Time 1, only time1 memory changes 2 n1 C0 + Allocs(n2) - Frees(n2) + Allocs(n1) - Frees(n1) )6_reorder_communication_preserving_peak_memory_internal)r7reordered_snodes node_statss rB,reorder_communication_preserving_peak_memoryrTs@ ?vF! rLc\rSrSr%SrSrS\S'SrS\S'SrS\S 'S r S\S 'S r S \S'S r S \S'S r S \S'S r S \S'SrS\S'\S5rSrg) ReorderInfoz= Debug info describing how an individual snode was reordered Nonestrlimiting_factorrintmovesgrouped grouped_infofloat comm_time comp_timeinitial_exposed final_exposed overlap_infoc4URUR- $Nrdreselfs rB improvementReorderInfo.improvement##d&8&888rLN)__name__ __module__ __qualname____firstlineno____doc__rZ__annotations__r\r]r_rbrcrdrerfpropertyrl__static_attributes__rorLrBrVrVst"OS!E3NGSL#IuIu!OU!M5L# 99rLrVcUcg[U[RRRR 5(ag[ USS5=n(aSU;agg)NFTpython_kernel_nameextern_kernels)rr0opsaten#_scaled_dot_product_flash_attentiondefaultgetattr)noderys rB is_gemm_likers] |  ::BB&d,@$GGG 0 0 rLcSSKJn [X5(a[SUR55$[ UR 5$)NrGroupedSchedulerNodec38# UHn[U5v M g7frh)contains_gemm_like).0xs rB %contains_gemm_like..s?,Q%a((,)r&r isinstanceanyr7rr)r:rs rBrrs4>%..?%,,???EJJ''rLcSSKJn [X5(a-UR(aURH nU"U5 M gU"U5 g)Nrr)r&rr temp_groupingr7)r:fnr_snodes rB_temp_group_visit_leavesrs6>%..53F3FllF vJ# 5 rLcSnURHGnU(aUS- nX#R5- nU(dM*U[UR55- nMI U$)Nr^r>)r7get_namer.get_buffer_names)r: with_bufsretns rB _group_namersV C \\  3JC zz| 9 d511345 6C  JrLcH[U[5=(a UR$rh)rris_fake)ds rB _is_fake_deprs a ! /aii/rLcjSRUVs/sHoR5PM sn5$s snf)N~)joinr)gnsgns rB _group_namesrs' 88S1Sr[[]S1 221s0c[X5n[XU5upEpg[[X55nSUS'[ U5n UUUUUU 4$)z*Initialize memory tracking data structures)rrN)rrdictzip_build_candidate_buffer_map) r7 graph_inputs graph_outputsname_to_freeable_input_buf peak_memorysnodes_curr_memorysnodes_allocfreebuf_to_snode_last_use _curr_memorycandidate_buffer_maps rB_initialize_memory_trackingrsl!7!M&   MK%5 F78LL77LM "  rLc0n0n[U5H3up4US:aXS- OSX'U[U5S- :aXS-OSX$'M5 USnXU4$)z/Create double-linked list structure from snodesrr N)r4len)r7_prev_nextir:_heads rB_initialize_double_linked_listr sh E Ef%()Av!e}4 ()CK!O(;v!e} & 1IE  rLc[[5nUH3nURRHnXR U5 M M5 [ U5$)ae Build inverted index: node -> set of buffers where node appears in successors. This optimization reduces buffer iteration from O(total_buffers) to O(buffers_per_node). Since buffer successors are immutable during reordering, this map doesn't need updates. Returns: dict mapping each node to the set of buffers that have this node in their successors )rr mpi_buffer succ_nodesaddr)rnode_to_candidate_bufsbuf succ_nodes rBrrsTCNC%22I " - 1 1# 63% & ''rLc nUVs0sH#o[SUR555_M% sn$s snf)z Pre-compute output name sets for all nodes. This optimization avoids creating OrderedSet objects repeatedly during exposed time calculations. Returns: dict mapping each node to a set of its output names c3@# UHoR5v M g7frhrros rBr/_precompute_node_output_sets..@sD0C1**,,0C)r get_outputs)r7r:s rB_precompute_node_output_setsr3s?SY RXzD0A0A0CDDDRX  s*2cb[U5(a[R$[R$rh)rr*reorder_sink_runtime_estimations_comm_mult.reorder_sink_runtime_estimations_non_comm_multr:s rB_op_runtime_estimate_multrDs'5!!FFF  F FFrLcN[URSS5=n(aSU;agg)z Filtering out ops that contain Collective and Wait inside and considered as Collectives. See contains_collective function. If the op contains Wait inside - consider as Synchronous compute. ryNz-torch.ops._dtensor.shard_dim_alltoall.defaultFT)rr)r:rys rBis_async_collectiverMs- %UZZ1EtLLL :>P P rLc"[U[5$rh)rrrs rBcontains_async_collectiverZs u&9 ::rLcP/nUnUbURU5 XA:XaU$X$nM")a! Traverse doubly-linked list from head to tail and return nodes as a list. Args: head: Starting node of the segment tail: Ending node of the segment (inclusive) next_dict: Dictionary mapping each node to its next node Returns: List of nodes from head to tail (inclusive) )append)headtail next_dictrrs rB_group_nodes_from_linked_listr^s< C A  = JJqM 9  J L rLc,X nX1n[XT-5$)zf Check if a wait node corresponds to a given collective node. Uses pre-computed sets for O(1) lookup. )bool)collective_snode wait_snodenode_output_sets node_dep_setscollective_outs unmet_depss rB!_is_corresponding_collective_waitrys"'8O*J  , --rLc^^TUnSmX0nSn/nXn U bXIn X-(aUSU R5S3- nO[U 5(a([U 5(dOURU 5 Xn M`[ U 5(a(Sn UHn [ X X45(dMSn O U (aO=Tn S UU4Sjjn[ X5 TnUSU R5S X- S3- nXn U bMUTU4$) z Calculate exposed communication time by iterating directly over linked list. Avoids O(N) list construction for each call. The collective_snode is the starting point, iteration continues via next_dict. r^z->W[]FTc>TTU- mgrhrorrcruntimess rBaccumulate_time9_coll_exposed_communication_time..accumulate_time &) )IrL+[rr!returnrX)rrrrrrr)rrrrrrbrrfcollectives_foundr:rhas_wait_for_collectives_found_collcomp_time_beforercomp_time_afterrcs ` @rB _coll_exposed_communication_timers;)*II&8OL13  'E  ")  ' d5>>#3"4A6 6L  u % %,U33!((/!(   -2 **4$-=6:2 + .$ * * !8#!ENN,-Q/Q.RRSTT  G  J i --rLc^^SnSmSn/nX n U b[U 5(aURU 5 [U 5(aj[XXE5(aTU nUSU R 5S3- nOy[ U 5(dSmX)n MzUHn [XXE5(dMSm O X)n MTn SUU4Sjjn [ X5 Tn USU R 5SX- S3- nX)n U bMUTU4$) z Calculate exposed communication time for a wait operation by iterating directly over linked list backwards. Avoids O(N) list construction. Iterates from wait_snode backwards using prev_dict to find corresponding collective. rr^z->C[rc>TTU- mgrhrors rBr9_wait_exposed_communication_time..accumulate_timerrLrrr)rrrrrrr)rr prev_dictrrrrbrf waits_foundr:wrrrrcs ` @rB _wait_exposed_communication_timers<IIL+-K  !E        u % u % %0#3%UO $u~~'7&8 :: ,U33 !($A8"2%(  % "($ * * !8#!ENN,-Q/Q.RRSTT  G  J i --rLcpX0nU(aXU'XcU'XBnU(aXU'XtU'X#U'XU'XP:XaU$U$)aQ Swap positions of candidate and group in doubly-linked list. Transforms: candidate_prev -> candidate -> group_head...group_tail -> group_tail_next Into: candidate_prev -> group_head...group_tail -> candidate -> group_tail_next Args: candidate: Node to swap with group group_head: First node of group group_tail: Last node of group prev_dict: Dictionary mapping nodes to their previous nodes next_dict: Dictionary mapping nodes to their next nodes head: Current head of the linked list Returns: New head of the linked list (may change if candidate was the head) ro) candidate group_head group_tailrrrcandidate_prevgroup_tail_nexts rB _perform_double_linked_list_swaprsa8)N$..!*j +O%./"*i&i%j  KrLcn0nSn U(d&[X4- XrSU- UR-5n X4$U*n UHSn X{SU -n XU '[X5n URU 5n U cM2U HnXRR- n M MU XrSU -UR-nXU'[X5n X4$)a Calculate potential peak memory after swapping candidate with group (reorder version). Computes new memory levels for all affected nodes and returns the potential peak memory along with cached post-allocation memory values for each node. Args: candidate: Node being moved gns: Group nodes group_tail: Last node of group group_peak_memory: Current peak memory within the group candidate_delta_mem: Net memory change from candidate (alloc - free) candidate_allocfree: Candidate's allocation/free info group_n_to_bufs_after_swap_dealloc_by_candidate: Buffers whose deallocation moves to candidate curr_memory: Current memory state dict Returns: Tuple of (potential_peak_memory, post_alloc_update_dict) rr )max size_allocgetr size_free)rrrgroup_peak_memorycandidate_delta_memcandidate_allocfree/group_n_to_bufs_after_swap_dealloc_by_candidate curr_memory_post_alloc_updatepotential_peakmem_after_reorder_deltargn_post_alloc_membufsrcandidate_mem_post_allocs rB(_calculate_potential_peak_memory_reorderr1s>8:N :  3  #A &! "!,, -  11%8#7'OA.1HH!22^?>BB2F  '>>+C+CC' " ! "  ( ( ) %=y!BN  --rLc U(dHUHn Xzn U SU- U SU- 4Xz'M XrSUR-n XR- n U U 4Xp'gUR5HnUHnXU'M M SnUHSnUUn[SUU55nUU- nU U=RU-slUU UR- nUU4UU'MU X`n X=RU- slXUR- nU U4Xp'g)a Update memory tracking structures after swap (reorder version). Updates curr_memory, buf_to_snode_last_use, and snodes_allocfree dictionaries to reflect the new memory state after swapping candidate with group. Args: candidate: Node that was moved gns: Group nodes group_tail: Last node of group candidate_delta_mem: Net memory change from candidate (alloc - free) candidate_allocfree: Candidate's allocation/free info group_n_to_bufs_after_swap_dealloc_by_candidate: Buffers whose deallocation moves to candidate post_alloc_update: Cached post-allocation memory values curr_memory: Current memory state dict (mutated) buf_to_snode_last_use: Buffer to last-use node mapping (mutated) snodes_allocfree: Node allocation/free info dict (mutated) rr Nc3N# UHnURRv M g7frhrrrrs rBr=_update_memory_tracking_after_swap_reorder..s!2 I NN $ $I#%)rrr/sum)rrrrr r post_alloc_updater rrrcm_candidate_post_alloc_mem_candidate_post_free_memrr"size_free_to_move_to_candidate_sumr_gn_post_alloc_memsize_free_to_move_to_candidategn_post_free_memcandidate_post_free_mems rB*_update_memory_tracking_after_swap_reorderr#tsy< ;BB1++1++KO  #A &)<)G)G G " &(E(E E ! & $"  @FFHC)2# &I/0& "3A"6.12 FqI2 / & +.LL*%%)GG% 25Ea5H5R5R R,.>? A!2 <))-OO)!Y$?$I$II "KrLc[[5nURU[55n[U5nUH!nX'nX;dMXHR U5 M# U$)a^ Find buffers whose last use will change after swapping candidate with group. When we swap [candidate [group]] to [[group] candidate], some buffers that were last used by a group node will now be last used by candidate instead. This affects memory deallocation timing. Args: candidate: The node being moved gns: Group nodes being swapped with candidate buf_to_snode_last_use: Mapping of buffers to their current last-use nodes candidate_buffer_map: Pre-computed map of node -> buffers using that node Returns: Dict mapping group nodes to buffers that will change their last-use node )rr.rr r) rrrrr candidate_bufsgns_setrsnode_last_uses rB#_find_buffers_with_changed_last_user(sb0 D4 *--iFNoG.3  $ ; K R RSV W ;:rLc[R(d%[U5(aSSUR534$[R(d[ U5(agg)aA Check if a candidate node can be grouped with collective during reordering. This pass processes collectives left to right, so we avoid grouping with already-processed collectives based on configuration. Args: candidate: Node to check for groupability Returns: Tuple of (is_groupable, reason_if_not_groupable) Fzcandidate contains_collective )FrTN)r(reorder_iterative_group_with_collectivesrr)reorder_iterative_use_runtime_estimationsrrs rB_is_node_groupable_for_reorderr.sU$  @ @ $Y / /01C1C1E0FG   A A i ( (. rLc^UnUVs0sHoXxR_M n n[U Vs/sHoUPM sn5n [UVs/sHoURPM sn5n SU SU S3m/SQn UR5VV s/sHup[ U5U R S- U R S- U RS- U RS- U RS- U RU RU RU RU R/ PM nnn [RRS5(aSSKJn TU"UU S 9- mO8TS - mT[#U 5S -- mTS R%['["U55- m[)US U5n[+U5U:Xde[-UXV5un nTS U3- mTSU3- m[.R1T5 [3SSU4SjS9 U$s snfs snfs snfs sn nf)a Format reordering statistics, log them, and return final node list. Computes improvement metrics, creates a formatted table (using tabulate if available), validates the reordered node count, recalculates peak memory, and logs all information. Args: stats: Per-node reordering statistics head: Head of the reordered linked list next_dict: Linked list next pointers original_snodes_num: Original number of nodes (for validation) peak_memory: Initial peak memory before reordering name_to_freeable_input_buf: Buffer memory tracking info graph_outputs: Graph output names Returns: Final reordered list of scheduler nodes zAreorder_communication_preserving_peak_memory improved overlap by z ns after z reorders. ) zCollective node comm_time(us) comp_time(us)initial exposed(us)final exposed(us)improvement(us)limiting factorr\r]r_rf@@tabulaterr7headers>Please `pip install tabulate` to nicely render overlap stats.  Nz peak_memory_before:z peak_memory_after:artifactc SSS.$)NrTstringnameencodingrororLrB2_format_and_log_reordering_stats..as B  rLc>T$rhro)reorder_log_strsrBrCrDes?rL metadata_fn payload_fn)rlrr\items node_summaryrbrcrdrerZr]r_rf importlibutil find_specr7rYrmaprrr overlap_loginfor )statsrroriginal_snodes_numrrrrSr:rltotal_improvement total_movesr: node_inforowsr7 new_snodesnew_peak_memoryr>rFs @rB _format_and_log_reordering_statsrZsL8JEOPZE*+777ZKP[I[E/[IJJGJ5%(..JGHK LL]K^_l , G6!+ 0 0 2 !3 E     # %   # %  % % +  # #c )  ! !C '  % % OO     " "  " " !3   ~~ ++%8     M  3w<$..499Sd^44.tT9EJ z?1 11 1=. OQ1/ }==O..?@@O_% +  KQIG& sG3G8 G=BHc  SnUHn[U5(dMSn O U(dU04$[U5n[[RR R 55n[[RR55n[XU5unnnn n n UVs0sHnU[U5[U5-_M n nUVs0sH#o"[SUR555_M% n nUVs0sH nU[SUR55_M" nn0nSn[U5unnn[R nSnUn[R"nSnUGbUUGbUUnU(aGO[%U5(dUnM-UbUU:aGOUS- n['5=nUU'[)UUXU5unnnUUlUUlUU- =UlUlUUlUUnUnUn 0n!Sn"UUSn#[UU5n$[U U5n%UGbY[R4(a5UR0[R6*UR*-:a S UlGO[R:(d[U5(a S UlGOU$U%- n&U&(dSn'OU Un([=U(U&-5n'U'(Ga[?U5un)n*U)(aUnU$RAUU5 U%RAU U5 [R4(aC[CU5(aUUUU U U5unnn+UU4U!U'[%U5(dU"U U- n"[GU#UUS5n#U=RHS- sl$UUnGMlS URK5S UR5V,s/sHn,U,RK5PM sn,S U*3n-U-UlGO[R4(Gab[%U5(dU Un.U.S:a[U!5S:a[GSUR*UR,- 5n/[GSUR*UR,- U.- 5n0U0U/- n1U!RM5H/un2n3[GSU2U3- 5n4[GSU2U3- U.-5n5U1U5U4- - n1M1 U1S:aSU.SU13UlGOU!RO5Hun6un2n3U2U3U.- 4U!U6'M OjU"S:ad[)UUXU5unnn+[GSUU- 5n/[GSUU- U"-5n0U0U/- n1U1S:a&SURK5SU"SU1SUSU3 UlGO*[QUU U5n7UUn8U8RRU8RT- n9[WUU7X5n:[YUU7U U#U9U8U:U5un;nU>U4$[kUUUUUU U5n>U>U4$s snfs snfs snfs sn,f)z| Internal testing helper that also returns debug info. Returns: - reordered snodes list - dict {snode: ReorderInfo} FTc3@# UHoR5v M g7frhrrs rBrI_reorder_communication_preserving_peak_memory_internal..C/B!/Brc3^# UH#n[U5(aMURv M% g7frhrrArrs rBrr]  4qLOFAFF4--rNr r unexposed by runtime estimationszcollective ordering$data dependency detected candidate:z(outs:z) non_group_reason:candidate has compute z,, group contains waits, total_exposed_delta z candidate z is collective, group_runtime:z, exposed_delta:z c_comm_time:z c_comp_time:peak memory new: vs base:!_debug_iterative_memory_recomputerT)6rrr rgraphrr5get_output_namesrestimate_op_runtimer frozensetrunmet_dependenciesrr(reorder_iterative_debug_limit_to_reorder(reorder_iterative_debug_memory_recomputerrVrrbrcrdrerfr,)reorder_iterative_extra_comm_comp_overlaprZ,reorder_iterative_unsafe_collectives_reorderrr.updaterrrr]rr/rJrrrr(r$reorder_iterative_peak_memory_budgetr\rr# comms_debugrjrreorder_sink_verbose_loggingrZ)?r7has_collectivesr:rSrrrrrrrrrrrrRrUrrr debug_num_collectives_to_reordernum_processed_collectivescurr debug_iterative_memory_recomputeiterative_recompute_error _next_currrQrbrcrfrrr group_waits group_runtimergroup_unmet_deps_namesgroup_output_namesdata_deps_namesdata_depcandidate_out_namesis_groupable_resultgrouping_reasonr>rmsg c_runtimeexposed_before exposed_after exposed_delta gw_comm_time gw_comp_timegw_exposed_beforegw_exposed_aftergwrr rr r r rjrXs? rBrQrQksO u % %"O  rzf+$.qww/C/C/H/H/J$KL%/0H0H0J%KM $F-H" 0E "5),Ee,LLL 0RXAQWyCu/@/@/BCCCQWA >E y !44    >35EK8@E5% ==%&'(-D==%!&  uT{64[ $ (..D  + 7 %)I I !Q&!(]*uT{-M %]. * 9l#"4= 4IIt1($K     (.q1",M$,?!@'(8(>?#EE""III..!!(J$!MM' 22'<$57IIO" '7y&A# 3o EF7U84#_'!*J+11- 2JK&--.>y.IJ#MM(336V ) % % ( 0 - 73Iy!6? 5IK 28CC)Xi-@@M(+)< +B1+E)%LLA%L %i 0I((1(:(:(<'=V[d[p[p[rDs[rVWQZZ\[rDsCtu//>.?A ,/D(EEE0;; ( 3I 1}[)9A)=),Q0O)P(+t~~>J) )6(F :E:L:L:N6L,03A|l7R0S-/2 !<,#>#J0,*-=@Q-QQM;O)1,"8 DNN[_!^!0" "-!2!2!4!% , ,%1$09$<3" B"5%q(2R%uh-3/ 9a*-Q I0E)F(+Ay9/D}/T(U (5(F (1,",Y-?-?-A,BC22?A22? i[Xefoep!r!0 ",IJ,C0@ /J #..1D1N1NN 4s$9 <9%''C   /N., Q QQR'~&6i }M$ JJ!OJ 1 K4:z5%E2ReX2 .Iy,'DN&DN ,D !*Y!6D  6##?"%  0K,M %1%uE.! $BC -)-j)IC#DK  uT{6N  4 425$F 5  1   "J u o 0 A>pEts%"Z: *Z?='[$[ c^^^^^^^^^^^^0n0m000smmm[U5H|upVUR5HnXdU'M UR5HnUTU'M UTUR5'UR5n [R TU 'STU 'UTU 'M~ Sn UHnU(ab[ U5(aRU TUR5'URH(n TU R5n [TU U 5TU 'M* U S- n MlU(dMu[U5(dMSTUR5'M "UUUU4SjS5mUVs0sH nU[SUR55_M" snm/m[[5mUVs0sHof[U5_M snmTR5HNupm[U 5S:Xa[ R""TT"U55 U HnTUR%U5 M MP /mUUUUU4SjmU4SjmUUUU4SjnT(aQ[ R&"T5R(nU(a[ U5(a U"U5 OT"U5 T(aMQTR+5Hn [U 5S:XaMS T35e T$s snfs snf) a Schedule `snodes` for various comm optimization objectives. Args: snodes: the nodes to be scheduled. raise_comms: whether to greedily schedule collectives as early as possible sink_wait: whether to greedily schedule waits as late as possible reorder_compute_for_overlap: whether to reorder compute nodes to optimize for compute/communication overlapping. Returns: The new schedule order. Some notes on the synergy between different options: - `raise_comms` provides more overlapping oppurtunies for `reorder_compute_for_overlap`. - When both `raise_comms` and `sink_waits` is `True`, `raise_comms` is prioritized. rr c2>\rSrSrSUUUU4SjjrSrSrg)$_schedule_for_comm..Runnableic>Xl[[UR555nTUR 5nTUTUTU4Ulgrh)r:nextiterget_operation_namesrscore)rkr:rA fused_namename_to_fused_nodescores_0scores_1scores_2s rB__init__-_schedule_for_comm..Runnable.__init__sTJU6689:D+D1::.Runnable.__lt__s:: + +rL)rr:N)rrX)rprqrrrsrrrw)rrrrsrBRunnablers   ,rLrc38# UHoRv M g7frh)rA)rdeps rBr%_schedule_for_comm..sG.Fs((.Frc>TRU5 UR5HTnTUHHnTURU5 [TU5S:XdM+[R "TT"U55 MJ MV g)zE Schedules `snode` and put all unblocked nodes onto the ready queue. rN)rrremoverheapqheappush)r:buf_namer buffer_usersready scheduledrs rBschedule$_schedule_for_comm..schedule#sl ..0H%h/5!((2z%()Q.NN5(5/:01rLc>TVs/sH=n[UR5(aM[UR5(aM;UPM? nn[U5S:Xag[ USS9$s snf)zP Return the next node in the ready queue that's neither a collective or a wait. rNcUR$rhrrs rBrCG_schedule_for_comm..get_overlapping_candidate..:sQWWrLkey)rr:rrmin)r candidatesrs rBget_overlapping_candidate5_schedule_for_comm..get_overlapping_candidate.sf &qww/ 8Eagg8N   z?a :#455 sA'A'A'c>[U5(deT"U5 TUnUS:aQT"5=nbGTRU5 T"UR5 UTUR-nUS:a T"5=nbMG[R"T5 g)z Schedules collective node `snode`, along with one or more compute nodes to overlap with it. The strategy is described in the comment of `reorder_compute_for_overlap`. rN)rrr:rheapify)r:collective_costrrrr snode_to_costs rBschedule_collective_for_overlap;_schedule_for_comm..schedule_collective_for_overlap<s #5))))'. a 799F LL # Y__ % }Y__= =O a 799F  erLz;Detected unscheduled nodes. Nodes with unmet dependencies: )r4rrrsysmaxsizer ancestorsrrr rorrmrJrrrrheappopr:r/)r7rFrGrHbuf_name_to_snoderAr:rop_name node_namecomm_idxancestoranc_fused_namedepsrrrrrrrrrrrrrrs @@@@@@@@@@@@rBrJrJsL#%r2 Hh' ..0H*/h '1002G*/ w '3/45>>+,NN$ !kk!(H .u55)1HU^^% &!OO!3H!=!F!F!H+.x/G+R(, MH ZM%00)*HU^^% & , , <E zGe.F.FGGG<J E=H=TLDJKF5/66FKM!'')  t9> NN5(5/ 2C   ! !% (* I ; ; 6*  e$** #6u#=#= +E 2 UO %!!#4yA~ I* V ~$ U<Ls 'J7J<c [RR5(dU$UVs/sHn[U5(dMUPM nn[ S[ U55H^n[ [XER555nXES- R5HnXER[XvSS95 M M` U$s snf)z Decide global ordering of comms, by just enforcing the ordering that's in the input graph (might not be the same ordering as the eager mode program). TODO: Come up with a better approach r T mutating_bufr) r0r) is_availablerr,rrrr add_fake_depr)nodes name_to_bufrr comm_nodesrrrs rBdecide_global_ordering_of_commsr_s    ) ) + + "=U&9!&s CCc\rSrSr%SrS\S'SrS\S'SrS\S'SrS\S 'S r S\S 'S r S \S'S r S \S'S r S \S'S r S \S'S rS\S'\S5rSrg) SinkWaitInfoiwrr[r]r^rYr_r\ moves_inforXrZr`rarbrcrdrerfc4URUR- $rhrirjs rBrlSinkWaitInfo.improvementrnrLroN)rprqrrrsr]rur_r\rrZrbrcrdrerfrvrlrwrorLrBrrwswGSL#E3NJ!OS!IuIu!OU!M5L# 99rLrcV[U5(aSSUR534$[U5(aSSUR534$[R(dJ[ U5(aSSUR534$[ U5(aSSUR534$g)a0 Check if a candidate node can be grouped during sink_waits pass. Sink Waits traverses waits right to left, so we don't group with processed waits on the right or with async collectives. Args: candidate: Node to check for groupability Returns: Tuple of (is_groupable, reason_if_not_groupable) Fzcandidate contains wait $candidate contains_async_collective zcandidate contains collective zcandidate contains gemm_like r*)rrrr&sink_iterative_use_runtime_estimationsrrr-s rB!_is_node_groupable_for_sink_waitsrs"Y01C1C1E0FGGG ++ 293E3E3G2H I   > > y ) )01C1C1E0FG  i ( (/ 0B0B0D/EF  rLc `USn XySXR- n U(d@XR-n U XR- 4Xp'UHn X|n U SU-U SU-4X|'M gU/UQHCnX^nX=RURUS5- slUXUR- 4X~'ME g)a Update memory tracking structures after swap (sink_waits version). Updates curr_memory and snodes_allocfree dictionaries to reflect the new memory state after swapping candidate with group. Args: candidate: Node that was moved gns: Group nodes candidate_delta_mem: Net memory change from candidate (alloc - free) candidate_allocfree: Candidate's allocation/free info group_n_to_bufs_after_swap_dealloc_instead_of_candidate: Buffers whose deallocation moves from candidate to group post_alloc_update: Cached post-allocation memory values size_free_delta_update: Cached size-free delta values curr_memory: Current memory state dict (mutated) snodes_allocfree: Node allocation/free info dict (mutated) rr N)rrr)rrrr 7group_n_to_bufs_after_swap_dealloc_instead_of_candidatersize_free_delta_updater rr pre_group_memcandidate_post_allocrrr post_allocs rB-_update_memory_tracking_after_swap_sink_waitsrs8QJ+A.1A1M1X1XXM B,/M/MM #@#@ @" BB1++1++KO  # &) %%)?)C)CAq)II%  !,66 6 rLc XrSXR- n 0n 0n Sn U(d[X4-XR-5n XU 4$XR-n XU'U n [S[RR UR 5555nU*X'XN-nUHXnUUSU-nUU U'[U U5n SnUU;a,UUnUHnUURR- nM UU U'UU-nMZ XU 4$)aO Calculate potential peak memory after swapping candidate with group (sink_waits version). Computes new memory levels for all affected nodes and returns the potential peak memory along with cached post-allocation and size-free delta values. Args: candidate: Node being moved gns: Group nodes group_head: First node of group group_peak_memory: Current peak memory within the group candidate_delta_mem: Net memory change from candidate (alloc - free) candidate_allocfree: Candidate's allocation/free info group_n_to_bufs_after_swap_dealloc_instead_of_candidate: Buffers whose deallocation moves from candidate to group curr_memory: Current memory state dict snodes_allocfree: Allocation/free info for all nodes Returns: Tuple of (potential_peak_memory, post_alloc_update_dict, size_free_delta_update_dict) rc3N# UHnURRv M g7frhrrs rBr>_calculate_potential_peak_memory_sink_waits..s%& C    r) rrr itertoolschain from_iterabler/rr)rrrrrr rr rrr _size_free_delta_updater rcandidate_size_free_to_move delta_memr gn_post_allocgn_size_free_to_addrrs rB+_calculate_potential_peak_memory_sink_waitsrsT> +A.1A1M1X1XXM79<>N B  3 :: : 3JJJ(+I+II$8y!)N"%&??00 C J J L &# +F)E&#AI#B*Y6 !.2^]; H HJ2ND#s~~'?'??#*= #B '((  /F FFrLcpX1nU(aXU'XcU'X@nU(aX#U'XtU'XU'XU'X:XaU$U$)a Swap positions of candidate and group in doubly-linked list (sink_waits version). Transforms (moves candidate to the left): group_head_prev -> group_head...group_tail -> candidate -> candidate_next Into: group_head_prev -> candidate -> group_head...group_tail -> candidate_next Args: candidate: Node to swap with group group_head: First node of group group_tail: Last node of group prev_dict: Dictionary mapping nodes to their previous nodes next_dict: Dictionary mapping nodes to their next nodes head: Current head of the linked list Returns: New head of the linked list (may change if group_head was the head) ro)rrrrrrgroup_head_prevcandidate_nexts rB+_perform_double_linked_list_swap_sink_waitsr/sa8 +O%./"*i)N$..!*j&j%i KrLc.^/SQnUR5VV s/sHup[U5U RS- U RS- U RS- U R S- U R S- U RU RU RU RU RU R/ PM n nn Sm[RRS5(aSSKJn TU "U US9- mO8TS- mT[#U5S -- mTS R%['["U 55- m[(R+T5 [-US U5n [/U 5U:Xde[1XU5un nTS U3- mTS U 3- m[3S SU4SjS9 U $s sn nf)a Format sink_waits statistics, log them, and return final node list. Computes improvement metrics, creates a formatted table (using tabulate if available), validates the reordered node count, recalculates peak memory, and logs all information. Args: stats: Per-node sink_waits statistics head: Head of the reordered linked list next_dict: Linked list next pointers original_snodes_num: Original number of nodes (for validation) peak_memory: Initial peak memory before reordering name_to_freeable_input_buf: Buffer memory tracking info graph_outputs: Graph output names Returns: Final reordered list of scheduler nodes ) z Wait noder0r1r2r3r4r5r]r_r\rrfr6r^r7rr8r9r;r<Nz* sink_waits_iterative peak_memory_before:z) sink_waits_iterative peak_memory_after:r=c SSS.$)Nsink_waits_iterative_infor?r@rororLrBrC2_format_and_log_sink_waits_stats..s /  rLc>T$rhro)log_strsrBrCrs7rLrG)rJrKrbrcrdrerlrZr]r_r\rrfrLrMrNr7rYrrOrPrQrrrr )rRrrrSrrrr:r:rQrWr7rXrYr>rs @rB _format_and_log_sink_waits_statsr`s8 G:!;;= )KE   NNS NNS  3 &    $   s "  LL    JJ OO    )  "G~~ ++%8    TT3w<$&&499Sd^,,W.tT9EJ z?1 11 1=  OQ1 <[MJJG ;O;LMMG #  [ sB Fc[[5nURU[55nUHPnX&nXp:waMURR nSn UH n X;dM U n M U cM=XIR U5 MR U$)a} Find buffers whose last use will change after swapping in sink_waits pass. When we swap [group] candidate to candidate [group], some buffers that were last used by candidate will now be last used by a group node instead. This is the opposite direction from the reorder version. Args: candidate: The node being moved (currently last use) gns: Group nodes being swapped with candidate buf_to_snode_last_use: Mapping of buffers to their current last-use nodes candidate_buffer_map: Pre-computed map of node -> buffers using that node Returns: Dict mapping group nodes to buffers that will change their last-use node N)rr.rr rrr) rrrrrr%rr'r last_succ_gnrs rB._find_buffers_with_changed_last_use_sink_waitsrs0 D< *--iFN.3  & ^^..  B!     @MTT  %, CBrLc [U5nUS:XaU04$[[RRR 55n[[RR 55n[XU5unnnnnn [U5upn 0n UVs0sHnU[U5[U5-_M nnUVs0sH#o[SUR555_M% nnUVs0sH nU[SUR55_M" nnUSn[5n[Rn[R nSnUGbyU UGbrU UnU(aGOdUb[U5U:aGOP[#U5(aUU;dUnMGUR%U5 ['5=nU U'[)UXUUU5unnnUU- =UlUlUUlUUlUUlU UnUnUn0nSn UUSn![UU5n"[5U5n#UGb[R6(a5UR,[R8*UR.-:a SUlGOSU"(dSn$OUUn%[=U%U"-5n$[R>(d*[AU5(aSURC53UlGOU#=(a [5U5n&U$(dU&(Ga[EU5un'n(U'(aUnU"RGUU5 U#=(d [5U5n#[R6(aA[5U5(a1[IUXUU5unnn)UU4UU'[AU5(dU UU- n [KU!UUS5n!U=RLS - sl&U UnGMU$(d7[RN(d!U&(aS URC53UlGOOS URC5S U(3UlGO[R6(GaS[QURR5(aJ[)UU U UUU5unnn)[KSUU- 5n*[KSUU- U -5n+U+U*:aS U*SU+3UlGO;[AU5(dUUn,U,S:a[U5S:a[KSUR.UR0- 5n-[KSUR.UR0- U,- 5* URU5H+un.n/U-[KSU.U/- 5[KSU.U/- U,-5- - n-M- U-S:aSU,SU-3UlGOsURW5Hun0un.n/U.U/U,- 4UU0'M [YUUU 5n1UUn2U2RZU2R\- n3[_UU1Xy5n4[aUU1UU!U3U2U4UU5 un5n6n7U5U- U[Rb-:aSU5SU3UlOU=RdS - sl2U=RfSURC53- sl3[iUUUXU 5n [)UXUUU5unnnUUlUUlUU- UlUUl[kUU1U3U2U4U6U7UU5 U(a4S SK6J7n8 U8"UU1[qU15[YU SU 5UUUUUSU45 nU(aO U UnUbGMUnUb U UbGMr[Rr(d[YU SU 5n9U9U 4$[uU U U UUUU5n9U9U 4$s snfs snfs snf)Nrc3@# UHoR5v M g7frhrrs rBr1_sink_waits_iterative_internal.. r^rc3^# UH#n[U5(aMURv M% g7frhr`ras rBrrrbrcFrrdrr z$collective ordering with candidate:rez non_group_reason:z"candidate is wait, exposed_before:z vs exposed_after:rfz2, group contains collectives, total_exposed_delta rgrhrrisink_waits_iterative);rr rrkrr5rlrrrmrrnrrorrq(sink_waits_iterative_debug_limit_to_sinkrrrrrdrerbrcrfrr&sink_iterative_extra_comm_comp_overlaprZr*sink_waits_iterative_swap_with_collectivesrrrrtrrr]/sink_waits_iterative_unsafe_collectives_reorderrrr/rJrrrrr!sink_iterative_peak_memory_budgetr\rrrrvrjrrwr):r7rSrrrrrrrrrrrrRr:rrrr{processed_waitsr|debug_num_sink_waits_to_reorderr} _prev_currrQrbrcrfrrr group_collsrrrgroup_contains_collectivercandidate_dep_namesboth_contain_comms _is_groupablegroupable_reasonr>rrrr gc_comm_time gc_comp_timegcrr rrr r rrjrXs: rB_sink_waits_iterative_internalrsf+arz$.qww/C/C/H/H/J$KL%/0H0H0J%KM $F-H"9@E%35E0E "5),Ee,LLL 0RXAQWyCu/@/@/BCCCQWA >E y !44    >)/r D lO==% ==$!&  uT{64[ $  + 7O$(GG d##O(CD D!)^+uT{-M %*:M. * 9l5> 4IIt1""($K     (.q1((8(>?$7$=!#BB""FFFWX(J$& '4I&># 36H HI  JJ,Y77>y?Q?Q?S>TU(";"?R@ -2S3/ /!!*J'--.>y.IJ1S5H5S. %KK/ ::2R%u8H-3/ 9a3>**/O! (% /+Iy!&)I ,A%BN$'9y+@=+P$QM%~5//=.>>PQ^P_a,1;;!) 3I 1}[)9A)=), NNT^^;) Q ?) KLL:E:L:L:N6L,)SL<4O-PSV !<,#>#JT.M;O )1,"8 D88E!H!0" "-!2!2!4!% , ,%1$09$<3" B"5,IJ,C0@ /J #..1D1N1NN ?s$9 D<%''K $  HN.0G, N NNO'~&6i }M$ JJ!OJ OO9#5#5#7"89 9O?:z5E2ReH.> 2 .Iy,'DN&DN!*Y!6D  ,D  9##G"'  0J,M %1%uE.! $*K -)-j)IQ#RY  uT{6\  4 425$F 5  1   "J u o 0 A>s"[?=*\-'\ c[U5S$)a_ Similarly to reorder_communication_preserving_peak_memory this pass will try to iteratively push Wait nodes later, recomputing estimated peak memory before each swap, and preventing peak memory regressions. Pass will be applied to every Wait node. If there are immediate dependencies with next node, pass will try to group them together and on the next step to swap the group with next candidate. If _inductor.config_comms.sink_iterative_use_runtime_estimations is set True, pass will stop reordering of Wait once corresponding Collective is unexposed, based on runtime estimations. inductor.config_comms.sink_iterative_peak_memory_budget allows to tune how much pass can regress initial peak memory. E.g.: sink_iterative_peak_memory_budget == 0.0 - No regression of initial peak memory is allowed sink_iterative_peak_memory_budget == 0.2 - Pass can improve comm-compute overlap, sacrificing 20% of initial peak memory value. inductor.config_comms.sink_iterative_extra_comm_comp_overlap config allows to more aggressively sink waits, stopping only when overlap_compute >= (1 + extra_comm_comp_overlap) * comm_time r)rrKs rBrr@s. *& 1! 44rLc[RS:XaUR5nU$[[R5(de[R"U5nU$)z3 Returns estimated op runtime in milliseconds (ms) r~)rrmr'callable)r:runtimes rBrmrmZsU!!Y.--/ N223333,,U3 NrLc UR5n[U5S:XGaSn[UR[R [R 45(aSUR5Vs/sHo3R5PM sn3nSURVs/sHoURPM sn3nSUR5SURRSUSUS 3 nUR5Vs/sHowRR5PM nnS RUV s/sHAn [U [R5(aS U RS U R S 3OSPMC sn 5n URR#5n URR&R(UU SU SUR+5S S3$/n UHn U R-[/U 55 M UR&R(SSRU 53$s snfs snfs snfs sn f![$a Sn Nf=f)Nr r^zouts:zins: z (z) (),z (size=z , stride=z.0fz ns): z, ) get_nodesrrrrExternKernelOut_CollectiveKernelrrrorAryget_output_specrLayoutsizestridemaybe_get_nameAttributeError __class__rpr'rrK)r:r7detailrouts_strrins_strchildlayoutslayoutout_tensor_infor summaries child_snodes rBrKrKfs& __ F 6{a ejj2#5#5r7K7K"L M Me6G6G6IJ6I 6IJKLHe.F.FG.Fff.FGHIG)*"UZZ-J-J,K4PXzYZ[bZccdeF=B__=NO=NE::--/=NO(( &  &Ffbii00&++i a@&     113I**&&//08II;VXY^YtYtYvwzX{{A AI k23oo&& 'r$))I*>)? @@/ KGO  I s+/H"H'9#H,-AH1=H66 IIcXSnSnSn[U5HupEUci[U5(aU[U5- nURnO)[ UR5(aOU[U5- nU"U[ U55 Mq[U5(a/U[U5- nURnU"U[ U55 M[ UR5(aU"U[ U55 SnMU"US[ U535 M [ RSUS- S- 35 g)Nrc:[RUSSU35 g)Nz>6r#)rPdebug)steprs rBstep_log#visualize_overlap..step_logsT"IRu-.rLz| zEst. runtime (ms): i)r4rrmrrrKrPr9)ordertotal_est_runtime cur_comm_noder;r:r:s rBvisualize_overlapr@s# #M/!'   "5))!%8%??! % $$!%8%??! Tl512 4"5))!%8%??! % ,u"5!68$$,u"5!68 $ L$7#89:-(. /$6=>?rLcUn[RHVn[U[5(aU[ 5;a [ 5Un[ U5(d SUS35eU"U5nMX U$)Nz3Invalid reorder_compute_and_comm_for_overlap pass: z is not callable)r'reorder_for_compute_comm_overlap_passesrrYglobalsr)r7r=ps rB$reorder_compute_and_comm_for_overlaprEsm E  ; ; a  !wy. ! A{{ A!DT U {% < LrLc < ^^^^^^[UR5m[[5m[[5m[T5HupURS:XdMUR [ RRRRLdMTURSRS:XdSUSURSS35eURSnURSnUS:aTURU5 MTURU5 M UUU4Sjn[[5n[T5HupURS:XdMUR [ RRRRLdMTUnURSmTRS:XdS TS US35eU"T5(dMUTRU5 M S nS mTHnURS:XdM[UR [ R R"5(dMJUR R$R&(dMqU"U5(aMT"X&R)55(dMS US35e UR+5GH3umn [U 5GHupTU nURSTLdeURun mU S-n U [-U 5S- :aXS-O [-T5S- nTXn[/UU4SjU55(aSTSUSUS35eUHnURS:XdMTUR;dM'UR [ RRRR:wdMe[1UU4SjUR55nUUl M GM GM6 UR35H"n U Hn TU nUR5U5 M M$ THxnURS:XdMUR [ RRRRLdMRURSU;dMgUR5U5 Mz g)ab This FX graph pass replaces uses of FSDP2 unsharded params with their corresponding graph intermediates that were fsdp.copy_ into the unsharded params in the original graph. NOTE: Can only apply this pass to any of the FSDP2 unsharded params that have this pattern (or repetition of): `resize_(full) -> copy_ -> resize_(0)`. Because of this, for partial-graph case where `resize_(full) -> copy_` is in one graph and `resize_(0)` is in another graph, we can't remove these resize and copy ops and thus we will have worse performance there. In other words, "do we try to remove all the resize_(full) -> copy_ -> resize_(0) nodes for this unsharded param" is actually a per-unsharded-param decision, since for each unsharded param, we look at its resize sequence pattern (in `check_resize_pattern()`) to determine if its set of resize and copy nodes can be removed. call_functionr placeholderz1Resize can only operate on graph inputs, but got z# which is resizing non-graph-input r<r cn>TRU/5nTRU/5n[U5[U5:wa2[RSUS[U5S[U5S35 g[ X5H7up4X4:dM [RSUSTUSUS TUSUS 3 5 g g ) NzH Unequal number of resize-to-full and resize-to-0 nodes for graph input z: z vs. zK. Skipping `remove_fsdp2_unsharded_param_graph_input_usage` FX graph pass. Fz For graph input z: resize-to-full node z at index z happens after resize-to-0 node zd. Skipping `remove_fsdp2_unsharded_param_graph_input_usage` FX graph pass for that unsharded param. T)rrlogwarningr) graph_inputresized_to_full_idxesresized_to_0_idxesresize_to_full_idxresize_to_0_idx&graph_input_to_resized_to_0_node_idxes)graph_input_to_resized_to_full_node_idxes node_lists rBcheck_resize_patternLremove_fsdp2_unsharded_param_graph_input_usage..check_resize_patterns !J M M ! DGG UWX $ %-?)@ @ KKHHS}UE#&8"9!:; 47 !4 / "4  3I>P4Q3RR\]o\pq )/ :;:oEVW4 rLz\ Assumed all FSDP2 `unsharded_param`s to be graph input, but it's not true! Offending node: z . Graph: cUR[RRRR L=(d: UR[RR RR L$rh)targetr0r{fsdpcopy_r~inductorresize_storage_bytes_)rs rBis_allowed_mutationKremove_fsdp2_unsharded_param_graph_input_usage..is_allowed_mutation sO KK599>>//77 7 O{{eii00FFNNN rLc [UR[RR5(aj[ URR R5VVs/sH3up#URcMURR(dM1UPM5 snnO/n[UVs/sH6n[URURSR55PM8 sn5n[UVs/sH)n[URSR55PM+ sn5n[XW-5S:$s snnfs snfs snf)Nvalr)rrWr0_ops OpOverloadr4_schema arguments alias_infois_writer r argsmetauntyped_storager)runsharded_paramsrrmutated_arg_idxesmutated_node_arg_storagesunsharded_paramstorages_of_unsharded_paramss rB-is_node_mutating_unsharded_param_or_its_aliaseremove_fsdp2_unsharded_param_graph_input_usage..is_node_mutating_unsharded_param_or_its_alias s)$++uzz'<'<==&dkk&9&9&C&CD DDA<<01 0E0ED   %/+ *Atyy|007GGIJ* % ! (2(8 '7O33E:JJLM'7 ( $ ,KLqPP)    s D=7D=D=)=E60EzdUser mutation on FSDP2 unsharded param is not allowed when Traceable FSDP2 is used. Violating node: c38># UHnT"UT/5v M g7frhro)rrrnrls rBrAremove_fsdp2_unsharded_param_graph_input_usage..Y s%*D>d_DUVV*sz(Assumed no ops mutating unsharded param z in subgraph z, but it's not true! Graph: c36># UHnUTLaTOUv M g7frhro)rarg replacementrls rBrrqf s$%#,C(+o'= 3F#,sN)r.rrr4oprWr0r{rZr[r~rfrrXrYrr`rarb is_mutabler5rJrrtupler/ erase_node)rkrArrLnew_sizerT'unsharded_param_to_fsdp_copy_node_idxesfsdp_copy_noder\fsdp_copy_node_idxesrfsdp_copy_node_idxr>subgraph_start_idxsubgraph_end_idxsubgraph_nodesnew_argsrQrRrnrSrtrls @@@@@@rB.remove_fsdp2_unsharded_param_graph_input_usagersSU[[!I1*y)  GG & uyy11GGOOO99Q> ##...'--DBBD eeidjk : 1 6 6 8 %./C%D !A&'9:N!&&q)_< <<+00NA{!3a!7 s/0144%U+^a'  ''9KN* ))8(9~FVW w 'GG.'4994 uyy'9'9'O'O'W'WW$%#'99% H!)DI')&E 9D!H N N P"6 &'9:N   ^ ,#7!Q  GG & uyy11GGOOO ! GG   T " rLc^ SSKm T RR5(deT RRR (a%T RRR (deSSK J nJ nJ nJnJn U 4SjnU"5nU"U"T RRR R U"["R$U"T RR&R(R U"S5U"S5U"S5U"S5U"S 55U"S 55U"S 5U"S 55US S9SU 4Sjj5nU"U5 UR+U5 g![[[4a gf=f)Nrr ) CallFunction KeywordArgMatchPatternMatcherPassregister_graph_patterncX>[UR5nUHnUR[RLdM"UR SRTR RRRLdMhUR SS:XdM}URU5 M g)Nrr ) r.rrWoperatorgetitemrfr{rXall_gather_copy_inr~rx)grSrr0s rBremove_unused_getitem8reinplace_fsdp_all_gather..remove_unused_getitem spM AH,,,FF1I$$ (I(I(Q(QQFF1IN Q rLall_gather_inputsall_gather_outputinp_split_sizesall_gather_input_numelrankitem_idx group_size group_namec&URSS:H$)Nrr)kwargs)matchs rBrC+reinplace_fsdp_all_gather.. s%,,z":a"?rL) pass_dict extra_checkc n>U4SjnURUUSUSUSUSUSUSUS/5 g) Nc>USSnUSnUSnTRRRR"U6nUSnUSnTRRR RXRX6S9nU$)Nrrr )out)r{rXrr~_c10d_functionalall_gather_into_tensor_out) rf copy_in_argsrrrr getitem_1all_gather_into_tensorr0s rBreplEreinplace_fsdp_all_gather..reinplace_all_gather..repl s 9LbJbJ!&!B!B!J!J" )+G*1-I **EEMMN # * )rLrrrrrrr)replace_by_example)rrfrrr0s rBreinplace_all_gather7reinplace_fsdp_all_gather..reinplace_all_gather s[. *$   *+*+()/0v|$|$  rL)rr)5torch.distributed.fsdp._fully_shard._fsdp_collectivesr)rr{rrr ImportErrorr,AssertionErrorpattern_matcherrrrrrr~rrrXrapply) rkrrrrrr graph_passrr0s @rBreinplace_fsdp_all_gatherr| sg D  --//// II & & = = **EE  FE   $%J II & & = = E E   IINN55==23230178v& :&  | $ | $ $?), -, B% UA  8sA1E))FFc[U[RRR[RRR 45(ae[ UR5SS5$)N)rr0 _inductor schedulerFusedSchedulerNoderr[rrs rB get_op_idxr sb OO % % 8 8 OO % % : :   u~~# $$rLc  ^^^ ^!SSKJm /n[["5nSnSn0n0n0m!U U!4Sjn UGH"n [ U R [ RRRRS9(Ga[U4SjU R55(GaSnU n [5n [U U UT5 [[ RRRR[ RRRR[ RRR R/5m[#U U UTUU 4SjS 9 [%U S S 9n ['U 5nS n[)['U 55H^nU Un[+UR [ RRR R5(aUS- nUS:dM\Un O U SUn Sn[)['U 5S- 5H9n[-U US-R [.R05(dM4US-n O UceU "U SU5nU "U US5nUUU'GM;[+U R [ RRR2R5(dGMSnU n[5n[#UUUT5 [%US S 9nSn[)['U5S- 5H9n[-UUS-R [.R05(dM4US-n O UceU "USU5nU "UUS5nUUU'GM% ['T!5S :deU(a['U5S :deU(a['U5S :deUHSn U R55T!;aT!U R55n X;aM1UR7U 5 UR9U 5 MU SnUR;5HlunnUba[=[?URA555nURC5H+nURE[GUR55USS95 M- UnMn SnUR;5HlunnUba[=[?URA555nURC5H+nURE[GUR55USS95 M- UnMn U$)Nr )rFc>TRRU5nUHnUTUR5'M UTUR5'U$rh)rcreater)snodes_to_group group_noder:rsnode_name_to_final_snodes rB_create_group_node:enforce_comm_ordering_for_fsdp.._create_group_node sO33::?K $E:D %enn&6 7%;E!*"5"5"78rL)ruc3># UHJn[TUR[RRR R 5v ML g7frh)rrr0r{rXrr~)rrrs rBr1enforce_comm_ordering_for_fsdp.. sJ % "1%**EIINN,M,M,U,U  %sAATc>[UTR5=(d6 [UTR5=(a URRT;(+$rh)rNopKernelSchedulerNodeExternKernelSchedulerNoder op_overload)r allowed_opsrs rBrC0enforce_comm_ordering_for_fsdp... sDq)"B"BC"1i&I&IJ>FF..+= 'rL) criteria_cbc[U5$rhrrs rBrCr9 JqMrLrrc[U5$rhrrs rBrCrj rrLr)$r^rr rrrr0r{rrr~rrr wait_tensorrXsplit_with_sizes_copyrsortedrr,rrr _WaitKernel chunk_catrrrrJrrrrrr)"r7rr new_orderr ag_exists rs_exists$ag_grouped_node_to_wait_grouped_node$rs_grouped_node_to_wait_grouped_noderr:ag_snodeag_related_snode_setag_related_snodesend_idx_of_current_ag_blockcopy_out_countr cur_snode wait_node_idx ag_group_nodeag_wait_group_noders_snoders_related_snode_setrs_related_snodes rs_group_noders_wait_group_node prev_ag_waitwait_group_noderr prev_rs_waitrrrs" ` @@@rBenforce_comm_ordering_for_fsdpr s )+I3!III+-(+-( "  JJ59955PPXX   __    IHLVL  ($"  %II..IIQQII..::BBIINN88@@K )$" !'$*A!  +..?*@ 'N3012-a0 !NNEIINN$H$H$P$P#a'N!A%23/3!22N3N O !M301A56/A6;;R^^LL$%EM7!, ,,./@-/PQM"44Emn4U!V BT 0 ?EJJ (@(@(H(H I IIHMWL ($"  !'$*A!  !M301A56/A6;;R^^LL$%EM7!, ,,./@-/PQM"44Emn4U!V BT 0 ?]` ( )A -- -781<<<781<<< >> 8 8-enn.>?E    e L*N*T*T*V&   #]%C%C%E FGL!--/**AJJL|TR0' +WL*N*T*T*V&   #]%C%C%E FGL!--/**AJJL|TR0' +W rL)r7list[BaseSchedulerNode])r7rrr)rz"Optional[Union[IRNode, Operation]]rr)r:r!rr)r:r!)F)rrY)rrrrY)r7rrztuple[dict[BaseSchedulerNode, Optional[BaseSchedulerNode]], dict[BaseSchedulerNode, Optional[BaseSchedulerNode]], BaseSchedulerNode])rrr#dict[BaseSchedulerNode, OrderedSet])r7rrz(dict[BaseSchedulerNode, OrderedSet[str]])rOptional[BaseSchedulerNode]rrr4dict[BaseSchedulerNode, Optional[BaseSchedulerNode]]rr) rr!rr!r'dict[BaseSchedulerNode, frozenset[str]]rrrr) rr!rrrdict[BaseSchedulerNode, float]rrrrrtuple[float, float, str])rr!rr!rrrrrrrrrr)rr!rr!rr!rrrrrr!rr!)rr!rrrr!rr[rr[r rr rr rrz(tuple[int, dict[BaseSchedulerNode, int]])rr!rrrr!rr[r rr rrdict[BaseSchedulerNode, int]r rrrrrrrX) rr!rrrrrrrz>dict[BaseSchedulerNode, list[Union[FreeableInputBuffer, Any]]])rr!rztuple[bool, Optional[str]])rRz$dict[BaseSchedulerNode, ReorderInfo]rr!rrrSr[rr[rrrOrderedSet[str]rr)r7rrzDtuple[list[BaseSchedulerNode], dict[BaseSchedulerNode, ReorderInfo]]) r7rrFrrGrrHrrr)rrrr)rr!rrrr[r rrrrrrrr rrrrrX)rr!rrrr!rr[rr[r rrrr rrrrzFtuple[int, dict[BaseSchedulerNode, int], dict[BaseSchedulerNode, int]])rRz%dict[BaseSchedulerNode, SinkWaitInfo]rr!rrrSr[rr[rrrrrr)r7rrzEtuple[list[BaseSchedulerNode], dict[BaseSchedulerNode, SinkWaitInfo]])r:r!rra)rktorch.fx.Graph)rkrrrX)r71list[torch._inductor.scheduler.BaseSchedulerNode]rz4dict[str, torch._inductor.scheduler.SchedulerBuffer]rzdict[str, BaseSchedulerNode]rr)h __future__rrrLrloggingrr collectionsr dataclassesrtypingrrrr r0torch._loggingr torch.multiprocessing.reductionsr torch.utils._ordered_setr r^rrr dependenciesrrrmemoryrrrrutilsrrrrrrr virtualizedr getLoggerrprJ_logginggetArtifactLoggerrPr&r!rCrGrFrOrTrVrrrrrrrrrrrrrrrrrrrr#r(r.rZrQrJrrrrrrrrrrrmrKr@rErrrrrorLrBrs#  #!66 +;/&&!% !nn..xC ;#F ##FL #&# ###L 99 9("(032 #"((((0 #-"G ; % %D 6 .' .! .> .; .  .9.'9.C9.-9.> 9. ; 9.  9.x9.!9. 9.D9.- 9. > 9. ; 9.9.x. .!.".D . D .  ..b@. @. @."@. @.  @. % @.6:@.@..@.FJ J J"J J % J 6: J4JJ JJ JZ$; $; $; $;> $; D $;N B 2 4 2 92 2 2  2 jCG CG CG"CG CG  CG % CG>BCGCGCGLCGL. .!.".D . D .  ..bW 0W WDW W  W !% W#WWt4C 4C 4C 4C> 4C D 4CnL #LJL^ 54 A>#L  #   ~#BjZ%n =nEn5n7 nrL