Eii ~%SSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK r SSK r SSK r SSK r SSKJrJr SSKJrJr SSKJr SSKJrJrJrJrJrJrJrJr SSKJrJr SSK r SSK!r!SSK"r!SSK#J$r$ SS K%J&r' SS K(J)r) SS K*J+r+ SS K,J-r- SS K.J/r0 SSK1J2r2J3r3J4r4 SSK5J6r6J7r7 SSK8J9r9J:r: SSK;Jr>J?r? SSK@JArA SSKBJCrCJDrDJErEJFrFJGrGJHrHJIrIJJrJJKrKJLrLJMrMJNrN SSKOJPrPJQrQJRrRJSrSJTrTJUrUJVrV \(akSSKWJXrXJYrYJZrZJ[r[ SSK"J\r\ SSK]J^r^ SSK_J`r`JaraJbrbJcrc SSKdJere SSKfJgrgJhrhJiri SSK@Jjrj SSKkJlrl \"S 5rm\X\\h/\g4rn\o\lrp\\q\ R4rs\qrt\!RR\wS!5rx\R"\w5rzGSS"jr{\R"S#S$55r}"S%S&\R$5r~"S'S(\5r\H"S)S*9"S+S,\55r"S-S.5r\R"S/S055r\R"S1S255r\R"S3S455r\R"S5S655r\R"S7S855r\\\\\\4r0rS9\S:'"S;S<5r0rS=\S>'0rS?\S@'0rSA\SB'GSGSSCjjr"SDSE\5rGSSFjrGSSGjrGSSHjrGSGSSIjjrGSSJjrGSSKjr\GR,GSSLj5rGSSMjrGSSNjrGSSOjr\!GR6\!GR8\!GR:\!GR80\!GR<\!GR>\!GR@\!GRB\!GRD\!GRF\!GRH\!GRJ\!GRL\!GRN\!GRP4 Vs0sHnX_M snErSP\SQ'GSSRjrGSSTjrGSSUjrGSSVjr"SWSX5r"SYSZ\05r/"S[S\5r\ GR`"S]\ GRbS^9rGS S_jr"S`Sa\>\\R\5r\R"SbSc55r\"GS!0Sd\"\$GRnSeSfSg9_Sh\"\$GRnSiSjSkSl9_Sm\"\$GRnSnSoSpSl9_Sq\"\$GRnSrSsStSl9_Su\"\$GRnSvSwSxSl9_Sy\"\$GRnSzS{SyS|9_S}\"\$GRnS~SSSl9_S\"\$GRnSSSSS9_S\"\$GRpSSSSl9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSSS9_S\"\$GRnSSSS|9_S\"\$GRnSSSSl9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSSl9_S\"\$GRnSSSSl9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_S\"\$GRnSSSg9_6rS\S'GS"Sjr"SS\D5r"SS\G5r"SS\5r\R"SS55r"SS5r\"5r"SS5r"SS5r\"S\qS9r\"SS\\S9r\(a\\!GR\T\\\\GS444r"GSGS\\\45r"GSGS5r"GSGS\\\5r\R"GSGS55r\GR,GS#GS j5r"GS GS 5r"GS GS \?5rgs snf($) annotationsN)ABCabstractmethod)autoEnum)chain)AnycastClassVarGeneric NamedTupleOptional TYPE_CHECKINGUnion)SelfTypeVar)ELEMENTWISE_TYPE_PROMOTION_KIND)_pytree) ConfigModule) OrderedSet)int_oo) PythonPrinter)free_symbol_is_typesymbol_is_typeSymT) bound_sympy ValueRanges)configmetrics)DtypePropagationOpsHandler)BasicMathOpsMixinDefaultHandler)ShapePropagationOpsHandler) boolean_opsDeferredLineBasegenerate_assertget_current_backendIndentedBuffer ir_dataclass ScopedDict sympy_dotsympy_index_symbol sympy_subs triton_typeunique) NullHandlerops OpsHandlerOpsValue ReductionType StoreModeV)CallableIteratorMutableMappingSequence) GraphModule)CustomGraphModulePass)Buffer ChoiceCaller FixedLayoutIRNodeLoopBody)BaseScheduling Scheduler SchedulerNode)BlockShapeTypePythonWrapperCodegen_Tschedulec[R[R5(a[R SU5 gg)NzData type propagation: %s) schedule_log isEnabledForloggingDEBUGdebug)msgs [/home/mande/repo/quber/.venv/lib/python3.13/site-packages/torch/_inductor/codegen/common.pydata_type_loggerrUas-  //6<0cj\rSrSr%SrS\S'S\S'S Sjr\SSj5rSS jr \SS j5r S r g )FileBackedGraphModulefz Output of FX wrapper codegen. Exposes the same methods as ModuleType, but these map back to a GraphModule instead of Python source. r<gmzCallable[..., Any] compiled_fnc.[R"SSSS9Ul[R"[R URR 5 URnURUR5 SSS5 g!,(df  g=f)Nzw+z.pyF)modesuffixdelete) tempfileNamedTemporaryFileatexitregisterosremovenamewritevalue)selffs rT __post_init__#FileBackedGraphModule.__post_init__ps^!33eE    4==#5#56 ]]a GGDJJ ]]s !B Bc.URR$N)r`rfris rT__file__FileBackedGraphModule.__file__zs}}!!!rVc UR"U6$rn)r[riargss rTcallFileBackedGraphModule.call~s&&rVc.URR$rn)rZcoderos rTrhFileBackedGraphModule.valuesww||rV)r`NreturnNoner{str)rt list[Any]r{r ) __name__ __module__ __qualname____firstlineno____doc____annotations__rkpropertyrprurh__static_attributes__rVrTrXrXfsF O## ""'rVrXcH\rSrSrSrSrSr\S Sj5r\S Sj5r Sr g) WorkspaceZeroModerrHrcX:XdU[R:XaU$U[R:XaU$[SU<SU<S35e)NzWorkspaceZeroMode.combine(, ))r UNINITIALIZEDNotImplementedErrorabs rTcombineWorkspaceZeroMode.combinesI 6Q+999H !// /H!$>qe2aU!"LMMrVcPU(a[R$[R$rn)r ZERO_ON_CALLr) zero_fills rT from_boolWorkspaceZeroMode.from_bools $11 1 ...rVrN)rrrrr{r)rboolr{r) rrrrrrZERO_PER_GRAPH staticmethodrrrrrVrTrrs9MLNNN//rVrc@\rSrSrSr\SSj5r\SSj5rSrg) CodegenSymbolzH An IR object possibly corresponding to a variable in the wrapper code. cgrnrros rTget_nameCodegenSymbol.get_name rVcgrnrros rT get_exampleCodegenSymbol.get_examplerrVrNr}r{z!Union[torch.Tensor, sympy.Symbol]) rrrrrrrrrrrVrTrrs/    rVrT)frozencN\rSrSr%SrS\S'S\S'S\S'S \S 'S rS \S '\Rr S \S'\ S S!Sjj5r \ S"Sj5r \ S#Sj5r \ S#Sj5rS$Sjr\rS%SjrS&SjrS'Sjr\S'Sj5r\r\r\rS(SjrS)SjrS)SjrS*SjrS+SjrS,SjrSrg)- WorkspaceArgaA temporary buffer used for a single kernel, then discarded. Not registered as a traditional buffer since there are no users, so it would be dead code eliminated. Args: nbytes: The size of the buffer in bytes. zero_fill: Whether the buffer should be initialized to zero. sympy.Exprcountr zero_mode torch.devicedevicer~ outer_namews_ptr inner_name torch.dtypedtypecPU[[RR53$rn)nextr7graph workspace_id)prefixs rT unique_nameWorkspaceArg.unique_names!$qww334566rVcURUR:H=(a9 URUR:H=(a URUR:H$rn)rrrrs rTcan_joinWorkspaceArg.can_joins@ LLALL ( XQWW-? XAHHPQPXPXDX rVc [URUR-[RURUR5UR UR URURS9$N)rrrrrr) rrrrrrrrrrs rTjoinWorkspaceArg.joinsS''AGG#'// Q[[I''88||||   rVc URUR:Xa4URUR:XaURUR:Xde[[R "UR UR 5[RURUR5URURURURS9$r) rrrrsympyMaxrrrrrrs rTmaximumWorkspaceArg.maximums GGqww 188qxx#7ALLALLUR5R5$rn) get_layoutrros rTrWorkspaceArg.get_examples ,,..rVc`SSKJn U"URURUR/S/S9$)Nr)r@rH)rrsizestride)irr@rrr)rir@s rTrWorkspaceArg.get_layouts.$;;****3   rVc"UR5$rn)rros rTlayoutWorkspaceArg.layouts  rVc6[RR$rn)rSZeroros rT get_offsetWorkspaceArg.get_offsetsww||rVcUR/$rn)rros rTget_sizeWorkspaceArg.get_sizes |rVc8[RR/$rn)rrOneros rT get_strideWorkspaceArg.get_strides }rVcUR$rn)rros rTrWorkspaceArg.get_names rVcg)NFrros rT get_is_pinnedWorkspaceArg.get_is_pinned srVc/$rnrros rTget_inputs_that_alias_output)WorkspaceArg.get_inputs_that_alias_output s rVrN) workspace_)rr~r{r~)rrrrr{r)rrrrr{r)r{r)r{rr)r{r@)r{r)r{list[sympy.Expr]r})r{r)r{ list[str]) rrrrrrrtorchuint8rrrrrrrget_device_or_errorrrrrrget_output_specmaybe_get_output_specmaybe_get_layoutrrrrrrrrrVrTrrs    OJE;$77        %/ !!!O&!rVrc(\rSrSrSSjrSSjrSrg)TritonScratchWorkspaceicXlX lgrn)r_generate_dtype_str)rirgenerate_dtype_strs rT__init__TritonScratchWorkspace.__init__s  #5 rVc"UR5$rn)rros rTr)TritonScratchWorkspace.generate_dtype_strs''))rV)rrN)rintrCallable[..., str]r})rrrrrrrrrVrTrrs 6*rVrcx\rSrSr%S\S'S\S'S\S'\R RrS\S'S r S \S 'S r g ) TensorArgir~rfbufferrrroffsetN Optional[str]alias_ofr) rrrrrrrrrr rrrVrTrrs. I K FJ%"Hm"rVrc>\rSrSr%S\S'S\S'\S Sj5rSrg) SizeArgi"r~rfrexprcgrnrros rTr SizeArg.alias_of'srVrNr{r )rrrrrrr rrrVrTr r "s I  rVr c \rSrSr%S\S'Srg) ConstexprArgi,r~rfrNrrrrrrrrVrTrr,s IrVrc>\rSrSr%S\S'S\S'S\S'S\S'S rg ) TMADescriptorArgi1r~rfapi_typezOptional[list[sympy.Expr]] block_shapeOptional[torch.dtype]rrNrrrVrTrr1s IM++  rVrcF\rSrSr%S\S'S\S'SrS\S'SrS\S 'S rg) DeviceCodegeni9SchedulingConstructor schedulingWrapperConstructorwrapper_codegenNOptional[WrapperConstructor]cpp_wrapper_codegenfx_wrapper_codegenr)rrrrrr r!rrrVrTrr9s&%%''8<5<7;4;rVrzdict[str, DeviceCodegen]device_codegensc\rSrSrSSjrSSjrSSjrSSjrSSjrSSjr SSjr SS jr SS jr SS jr SS jrSS jrSSjrSSjrSSjrSSjrSSSjjrSrg)DeviceOpOverridesiFc[ernrrirfs rTimport_get_raw_stream_as*DeviceOpOverrides.import_get_raw_stream_asG!!rVc[ernr&ri device_idxs rT set_deviceDeviceOpOverrides.set_deviceJr*rVc[ernr&ros rT synchronizeDeviceOpOverrides.synchronizeMr*rVc[ernr&r,s rT device_guardDeviceOpOverrides.device_guardPr*rVc[ernr&ros rTcpp_device_guard"DeviceOpOverrides.cpp_device_guardSr*rVc[ernr&ros rTcpp_aoti_device_guard'DeviceOpOverrides.cpp_aoti_device_guardVr*rVc[ernr&ros rTcpp_stream_guard"DeviceOpOverrides.cpp_stream_guardYr*rVc[ernr&ros rTcpp_aoti_stream_guard'DeviceOpOverrides.cpp_aoti_stream_guard\r*rVc[ernr&ros rTcpp_getStreamFromExternal+DeviceOpOverrides.cpp_getStreamFromExternal_r*rVc[ernr&ros rT kernel_headerDeviceOpOverrides.kernel_headerbr*rVc[ernr&ros rT kernel_driverDeviceOpOverrides.kernel_driverer*rVc[ernr&ros rTcpp_stream_type!DeviceOpOverrides.cpp_stream_typehr*rVc[ernr&ros rTaoti_get_stream!DeviceOpOverrides.aoti_get_streamkr*rVc[ernr&ros rTcpp_kernel_type!DeviceOpOverrides.cpp_kernel_typenr*rVc[ernr&ros rTcpp_device_ptr DeviceOpOverrides.cpp_device_ptrqr*rVc[ernr&ros rTtma_descriptor_helpers(DeviceOpOverrides.tma_descriptor_helperstr*rVNc[ernr&)riidx workspacers rT cpp_scratchDeviceOpOverrides.cpp_scratchws "!rVrrfr~r{r~)r-rr{r~r}rn)r[rr\rrr r{zOptional[tuple[list[str], str]])rrrrr(r.r1r4r7r:r=r@rCrFrIrLrOrRrUrXr]rrrVrTr$r$Fs""""""""""""""""TX""#9"CP" (""rVr$zdict[str, DeviceOpOverrides]device_op_overrides_dictz*dict[str, Optional[CustomGraphModulePass]]custom_backend_passesz!dict[str, Optional[ConfigModule]]custom_backend_codegen_configsc[UUUU5[U'U[U'U(a1[U[5(a U[ LdSU<S[ <35eU[ U'g)Nzdevice_custom_config=z: cannot be the same as the default inductor config config=)rr"ra isinstancerrrb)rdevice_schedulingdevice_wrapper_codegendevice_cpp_wrapper_codegendevice_fx_wrapper_codegendevice_custom_passdevice_custom_configs rTregister_backend_for_devicerksy,"! OF %7&! +\ : :$F2 %#%%`Y_Xa b   3 .B"6*rVc\rSrSr\"5r\"5r\"5r\"5r\"5r \"5r \"5r \"5r \"5r \"5rSrg)BackendFeatureirN)rrrrrFOREACH BUCKETIZEINPLACE_BUFFERSMASKED_SCATTER_WITH_INDEXSCANSORTTUPLE_REDUCTIONPREFER_STORE_LOOP_ORDERTRITON_TEMPLATESREDUCE_TO_SINGLE_ELEMENTrrrVrTrmrmsLfGIfO $ 6D 6DfO"fv#vrVrmc\Uc [5$[5 [U[R5(a UR nO=[U[ 5(d[ U55eUn[R"U5n[U5nU(deU"S5nURU5$rn) rinit_backend_registrationrdrrtyper~get_scheduling_for_deviceget_backend_features)r device_typescheduling_ctorrs rTr|r|s~|&%,,''kk &#&&4V 4& k*/ R@RC5n U S:wa\[5U 5cOSSK"J#n U"S5nU"S 5nU"S!5nU"S"5nU(a U(aU(a[7U UUUU5 gggggg![Ha gf=f)#z Register the backend for different devices, including the scheduling for kernel code generation and the host side wrapper code generation. rH) CppScheduling) CppWrapperCpu)CppWrapperCpuArrayRef) CppWrapperGpu) CppWrapperMps)CUDACombinedScheduling)HalideScheduling)MetalScheduling)PallasScheduling)PythonWrapperMtia)TritonSchedulingrI)WrapperFxCodegencpuN)cpphalidetritonpallasc6>T[R"U5$rn)r cpu_backend)r cpu_backendss rT+init_backend_registration..|F,>,>? KrVcuda)rrrc6>T[R"U5$rn)r cuda_backend)r cuda_backendss rTrr!s}V-@-@A*MrVxputpurc6>T[R"U5$rn)r tpu_backend)r tpu_backendss rTrr6rrVmpsmtia privateuseoner)_get_custom_mod_func SchedulingrJCppWrapperCodegenr)%rrcpp_wrapper_cpurcpp_wrapper_cpu_array_refrcpp_wrapper_gpurcpp_wrapper_mpsrcuda_combined_schedulingrrrrrrrpython_wrapper_mtiarrrwrapperrJ wrapper_fxirrr{rkr aot_inductorallow_stack_allocationr_C_get_privateuse1_backend_name torch.utils.backend_registrationr RuntimeError)rrrrrrrrrrrrJrprivate_backendrrerr r!rrrs @@@rTryrys #.@..@($(6(-. '/ &&&  $  K ""99 "  !(0-&& $  M    !'/#      !'/ &  $  K  !'/#      !(0#       hh<<>O?* %o 6 >I  4\ B 23IJO"67J"K !56H!I  _9L+#%#'& :M_  ? +$   s'AF11 F>=F>cJSSKJn /UQ[XRU55P$)Nr)FlexibleLayout)rrr,contiguous_strides)index index_varssizesrs rTindex_prevent_reorderingrds* $ UU TIj*K*KE*RS TTrVcU[U'grn)r`)rdevice_op_overridess rTregister_device_op_overridesros(;V$rVc[U[5(d[U55e[(dSSKJnJn SSKJn SSK Jn SSK Jn U[;aSSKJ n [X"55 [U$)NrH)cpu_device_op_overridesmps_device_op_overrides)r)CpuDeviceOpOverrides) rdr~rzr`rrrrrrrr)rrrrmtia_op_overridesxpu_op_overridesrs rTget_device_op_overridesrusU fc " "0DL0 " # #F-B@ --A$V-A-CD #F ++rVzdict[torch.dtype, torch.dtype]DTYPE_TO_COMPUTATION_DTYPEcU[5;a[R$US;aSU;aUS$US$US;a[R$US;a[R$US:XaSU;aUS$US$US:XaSU;aUS$US$US ;a$USn[ R RU5$US :XaSU;aUS$US $g ) zC Given op name and a list of input dtypes, deduce the output dtype )to_dtype index_exprr)randrandn) get_index randint64 load_seed reductionrHconstant)loadstorestore_reductionto_dtype_bitcastN)r%rrfloatint64r7rr)op_namertkwargsbuf_names rTdeduce_output_dtype_by_namers +-zz  #*V"3vgAbA  {{   {{ K ")V"3vg@a@ J ")V"3vgAbA   7ww  ** & &")V"3vgAbA rVCSEVariableTypec*[5n[RR(a(US:Xa"UR SUS[ U5S35 g[RR (aUS:XaSSKJnJ n [X5(d[U55eU[R:Xa"UR(aSUS 3nO2S US US 3nO(S US3nUR(aSUS3nSUSXRS3nUR SUS35 ggg)Nrtl.static_assert(z .dtype == rrrH)CppCSEVariable DTYPE_TO_CPPzIsVecMaskType::valuezstd::is_same_v || std::is_same_vz decltype(z typename z ::value_typezstd::is_same_vzstatic_assert(z);)r(r test_configsruntime_triton_dtype_assert writeliner/static_cpp_dtype_assert cpp_utilsrrrdrzrris_vec)rvarrbackendrr is_same_dt c_var_types rT check_dtypers"#G 667h;N,SEK*R89!:J 4rVc[5nUce[RR(aOUS:XaH[ U5S:waSR SU55OUSS3nUR SUSUS 35 ggg) NrrHrc38# UHn[U5v M g7frnr~).0ds rT check_shape..s,ec!ffesr,rz .shape == ()))r(rrruntime_triton_shape_assertlenrr)rrshaper shape_strs rT check_shaper s"#G    667h;N03E aDII,e, ,azQR^  ,SEYKrJK .sR) ! # #qvv - B*../55TA B sAA)all_input_nodesrdrfxNodeoprall functoolsreduce promote_typesr rrr)rinodeinputsr! input_nodesall_input_nodes_propagateds rTdeduce_node_dtype_by_inputs/DataTypePropagation.deduce_node_dtype_by_inputss%% !Auxx}}!=A!$$-BWAv   { q %() !) & " *   T$++->>+ ;; ! !"3 4 455d; ;8 ++ L    //55rVcUR(deSnURHn[RUR;aUR[RnO [5nUR U5UlXCR[R'UR S:XdMUR nM U$)Nr8)nodesrrr r<rr2)rir graph_dtyper+opt_ctxs rTr3#DataTypePropagation.propagate_graph6s{{{-1 KKD"&&$))3))$7$;$;<-/ 2248GM18II)-- .{{h&%mm  rVc>URURS5$r)r3rros rT propagateDataTypePropagation.propagateHs##DKK$788rVc.U"U5R5$rn)rG)clsrs rTpropagate_loopbody&DataTypePropagation.propagate_loopbodyKs4y""$$rVcSSKJn SSKJn [ X5(d[ U55e[ UR U5(d[ UR 55e[RUR 5$)NrrB)rF) loop_bodyrC schedulerrFrdrz_bodyrrK)rJr+rCrFs rTpropagate_scheduler_node,DataTypePropagation.propagate_scheduler_nodeOs\(-$..:T :.$**h//Adjj1AA/"55djjAArV)rrN)rrCr{r|)r+ torch.fx.Noder{r)r+rSr{r)rztorch.fx.Graphr{r)r{r)rrCr{r)r+rFr{r)rrrrrr/r5r<r3rG classmethodrKrQrrrVrTrrsJ% * 6:$9%%BBrVrcX^\rSrSrSSS.SU4SjjjrSS U4SjjjrSrU=r$) riYT)simplifypc>U(ag[U[R5(aH[[R S5(a)[R R RU5n[TU]%U5$)Nsizevars) rdrExprhasattrr7rrYrVsuperdoprint)rir rVrW __class__s rTr]PythonPrinter.doprintZsQ  444*9U9U77##,,T2Dwt$$rVc>[U[R5(aSURU5S3$[TU]XU5$N(r)rdrMod_printr\ parenthesize)riitemlevelstrictr^s rTrePythonPrinter.parenthesizebsA dEII & &t{{4()+ +7'V< ?@@rVc [R"S[R5n[R"U[R "U[R "[R"U5555$rr)r2rrrtrsrrneg)rvones rTsigmoidOpDecompositions.sigmoidsCll1ekk*{{3SWWSWWQZ-@ ABBrVcv[R"U[R"S[R55$Nr)r2rrrrtrus rTreluOpDecompositions.relus"{{1cll1ekk:;;rVcX[R"[R"X5U5$rn)r2rr{rvyzs rTfmaOpDecompositions.fmaswwswwq}a((rVc.[R"X5$rnrzrvrs rTmul_rnOpDecompositions.mul_rnswwq}rVcX[R"[R"U5U5$rn)r2rfloorrrs rT floor_to_intOpDecompositions.floor_to_int||CIIaL%00rVcX[R"[R"U5U5$rn)r2rceilrs rT ceil_to_intOpDecompositions.ceil_to_ints||CHHQK//rVcX[R"[R"U5U5$rn)r2rtruncrs rT trunc_to_intOpDecompositions.trunc_to_intrrVc [R"X5n[R"[R"U[R"S[ R 55[R"[R"U5[R"U555n[R"U[R"X!5U5$r) r2modand_nerrrtsignbitwherer)rrrconds rT remainderOpDecompositions.remaindersu GGAMxx FF1cll1ekk2 3 FF3;;q>3;;q> 2 yyswwq}a00rVcX[R"[R"U5U5$rn)r2rroundrs rT round_to_intOpDecompositions.round_to_intrrVrN)rhOpVarTr{rrvrr{r)rvrrrrrr{rrvrrrr{r)rrrrr{rrrrrr{r)rrrrrrrorwr|rrrrrrrrrrrrrrrrrrrVrTrmrmls<<CC<<CCRRQQMMAACC<<))1100111111rVrmz[a-z0-9_.]+|\([^)]*\)|)flagscUSS:wd[U5S:agSn[USS5H8up#US:XaUS- nO US:XaUS-nUS:XdM$U[U5S- :wdM8 g US:Xdeg)NrrbrFrHrT)r enumerate)stringrichars rT_all_in_parensrs ayC3v;? EVABZ( 3; QJE S[ QJE A:!s6{Q. ) A:: rVc\rSrSr\S$Sj5r\S%Sj5r\S&Sj5r\S'Sj5r\S(Sj5r \S(Sj5r \S(Sj5r \S(S j5r \S(S j5r \S)S j5r\S*S j5rS+S,SjjrS-SjrS.SjrS/S0SjjrS1SjrS2SjrS3SjrS4SjrS5SjrS6S7SjjrS8SjrS(SjrS\R:S SS.S9SjjrS:SjrS;Sjr \SS"j5r$S#r%g)? OpOverridesic[U[5(d*[RU5(d[ U5(aU$SUS3$ra)rd CSEVariable_RE_PAREN_NOT_NEEDED fullmatchr)rs rTparenOpOverrides.parensB v{ + +#--f55f%%M6(!}rVc[U5$rn)repr)rhrs rTrOpOverrides.constants E{rVc2S[RU53$)N~rrrus rT bitwise_notOpOverrides.bitwise_nots;$$Q'())rVc2[RU5S3$)Nz == 0r)rs rT logical_notOpOverrides.logical_nots##A&'u--rVc\[RU5S[RU53$)Nz & rrs rT bitwise_andOpOverrides.bitwise_and+##A&'s;+<+> rrs rTbitwise_right_shiftOpOverrides.bitwise_right_shiftrrVc.[R"X5$rn)r2rsrs rT int_truedivOpOverrides.int_truedivs {{1  rVcX[R"U[R"U55$rn)r2rrInteger)rfrs rTrOpOverrides.load_seedsxxemmF344rVTc*[[U55$rn)r-r~)rirrcheckwrap_negs rTindirect_indexingOpOverrides.indirect_indexings"#c(++rVcD[[U5RS35e)Nz,: check_bounds should be handled by CSEProxyrrzrrir rloweruppers rT check_boundsOpOverrides.check_boundss'"Dz""##O P  rVcD[[U5RS35e)Nz$: load should be handled by CSEProxyrrirfrs rTrOpOverrides.load"s%!Dz""##G H  rVNcD[[U5RS35e)Nz%: store should be handled by CSEProxyrrirfrrhr]s rTrOpOverrides.store's'"Dz""##H I  rVcD[[U5RS35eNz3: device_assert_async should be handled by CSEProxyrrirrSs rTdevice_assert_asyncOpOverrides.device_assert_async.%!Dz""##V W  rVcD[[U5RS35e)Nz/: store_reduction should be handled by CSEProxyrrirfrrhs rTrOpOverrides.store_reduction3s%!Dz""##R S  rVcD[[U5RS35e)Nz): reduction should be handled by CSEProxyrrir src_dtypereduction_typerhs rTrOpOverrides.reduction8s'"Dz""##L M  rVcD[[U5RS35e)Nz$: scan should be handled by CSEProxyrridtypes combine_fnvaluess rTscanOpOverrides.scanCs'"Dz""##G H  rVcD[[U5RS35e)Nz$: sort should be handled by CSEProxyrrir"r$stable descendings rTsortOpOverrides.sortPs'"Dz""##G H  rVcD[[U5RS35e)Nz): bucketize should be handled by CSEProxyrrir$ boundariesboundary_indicesindexing_dtyperightsortersorter_indicess rT bucketizeOpOverrides.bucketize[s'"Dz""##L M  rVcD[[U5RS35e)Nz2: halide_clamp only implemented for Halide backendr)rirhrrs rT halide_clampOpOverrides.halide_clampis%!Dz""##U V  rVcD[[U5RS35e)Nz): dot only implemented for Triton backendr)rirvrs rTdotOpOverrides.dotns%!Dz""##L M  rVrH) constraintsris_purepackcD[[U5RS35e)Nz<: inline_asm_elementwise only implemented for Triton backendr)riasmr=rr>r?r,s rTinline_asm_elementwise"OpOverrides.inline_asm_elementwisess'"Dz""##_ `  rVcD[[U5RS35e)Nz.: ops.output should not appear at codegen timeAssertionErrorrzrrss rTr8OpOverrides.outputs%Dz""##Q R  rVcD[[U5RS35e)Nz3: ops.placeholder should not appear at codegen timerErirs rTrOpOverrides.placeholders%Dz""##V W  rVc4^SU4SjjnTUlSUlU$)NcJ>[[U5RST35e)Nz does not implement ops.r)rirtrrfs rT unimplemented1OpOverrides._unimplemented..unimplementeds*%:&&''?vF rVT)rirrtr rr r{r)ris_unimplemented)rfrMs` rT_unimplementedOpOverrides._unimplementeds   "& )- &rVc[XS5n[[US5nU(+=(d X#:H=(d [USS5$)NrOF)getattrr3)rJrffn default_fns rT_is_unimplementedOpOverrides._is_unimplementeds; S %Zt4 vS)SWR9KU-SSrVc`US;dU5e[R5Hup#[X15nUc5URU5(a[ XUR U55 MFMHX R ;dSUSUR35eX$l[ X[U55 M g)N)rrcppvecrrzmultiple definitions of z on ) pointwise_overrides_datarrSrVsetattrrP__dict__rr)rJr2funcnamedataimpls rT_initialize_pointwise_overrides+OpOverrides._initialize_pointwise_overridessEEMvME6<<>NH4(D|((22C3+=+=h+GH3 ||3.xjS\\NK3!) |D'9:?rVr)rrr{r)rhzUnion[bool, float, int]rrr{rr)rrr{rrr)rfr~rrr{rTT) rrrUnion[sympy.Expr, int]rrrrr{ sympy.Symbol r rrrrrr rr{r|)rfr~rrr{rrn) rfr~rrrhrr]r6r{r|rrrSr~r{r|)rfr~rrrhrr{r|) rrrrrr5rh!Union[OpVarT, tuple[OpVarT, ...]]r{rg)r"tuple[torch.dtype, ...]r#zFCallable[[tuple[OpVarT, ...], tuple[OpVarT, ...]], tuple[OpVarT, ...]]r$tuple[OpVarT, ...]r{ri) r"rhr$rir)rr*rr{riNN)r$rr/.tuple[str, sympy.Expr, sympy.Expr, sympy.Expr]r0rr1rr2rr3 Optional[tuple[str, sympy.Expr]]r4zOptional[OpVarT]r{r)rhrrrrrr{r)r,rrAr~r=r rrr>rr?rr{r)rtrr{r|)rrr{r)rfr~r{zCallable[..., OpVarT]rfr~r{r)r2r~r{r|)&rrrrrrrrrrrrrrrrrr rrrrrr%r+r5r8r;rrrBr8rrPrTrVr`rrrVrTrrs1  **..BBBBBBCCCC!! 55 , ,%, ,  ,  ,  &0 9= FJ   NR  * 39 AJ         &  1  +    '     #       '  #        $48+/    C  !  $    1  )       &*"]]      #             TT  ; ;rVrc\rSrSr%S\S'S\S'SrS\S'SrS\S '\Rr S \S 'Sr S\S 'Sr S\S 'Sr g) OverridesDatair~rfrrNzOptional[Callable[..., str]]rrYrtype_promotion_kindrrr) rrrrrrrYrDEFAULTrprrrrrVrTrorosQ I +/F (/+/F (/'//8,0F (/(,C %,rVroairy_aicSUS3$)Nzairy_ai_forward(rrrus rTrrs(1-rVspecial_airy_ai)rprrf bessel_j0cSUS3$)Nzbessel_j0_forward(rrrus rTrr*1#Q/rVcSUS3$)Nz libdevice.j0(rrrus rTrr =1-rVspecial_bessel_j0)rprrrf bessel_j1cSUS3$)Nzbessel_j1_forward(rrrus rTrrrwrVcSUS3$)Nz libdevice.j1(rrrus rTrrryrVspecial_bessel_j1 bessel_y0cSUS3$)Nzbessel_y0_forward(rrrus rTrrrwrVcSUS3$)Nz libdevice.y0(rrrus rTrrryrVspecial_bessel_y0 bessel_y1cSUS3$)Nzbessel_y1_forward(rrrus rTrrrwrVcSUS3$)Nz libdevice.y1(rrrus rTrrryrVspecial_bessel_y1digammacSUS3$)Nz calc_digamma(rrrus rTrrs aS*rVc US3$)Nz .digamma()rrus rTrrs A3j)rV)rprrYrfrcSUS3$)Nz calc_erfcx(rrrus rTrr A3a(rVcSUS3$)Nzlibdevice.erfcx(rrrus rTrrs+A3a0rV special_erfcxrcSUSUSUS3$)Nz std::fma(rrrrs rTrrsis"QCr!A6rVcSUSUSUS3$)Nzfmadd(rrrrs rTrrss"QCr!A6rVcSUSUSUS3$)Nztl.fma(rrrrs rTrrs2aS1#Q7rV)rprrYrrfrcSUSUS3$)Nrb) * (rrrs rTrrs1QCuQCq)rVcd[RR(a SUSUS3$SUSUS3$)Nrbrrzlibdevice.mul_rn(r)rversionhiprs rTrrs? ==   s%s!,+ 2aS *+rVigammacSUSUS3$Nz calc_igamma(rrrrs rTrr<s"QCq1rVigammaccSUSUS3$Nz calc_igammac(rrrrs rTrr =2aS2rVgammainccSUSUS3$rrrs rTrrrrVspecial_gammainc gammaincccSUSUS3$rrrs rTrrrrVspecial_gammaincci0cSUS3$)Nzcalc_i0(rrrus rTrr 1orVcSUS3$Nzlibdevice.cyl_bessel_i0(rrrus rTrr3A3a8rVc US3$)Nz.i0()rrus rTrrs A3erV)rprrrYrfi0ecSUS3$)Nz calc_i0e(rrrus rTrr  !A&rVc US3$)Nz.i0e()rrus rTrr!s A3frV special_i0ei1cSUS3$)Nzcalc_i1(rrrus rTrr&rrVcSUS3$Nzlibdevice.cyl_bessel_i1(rrrus rTrr'rrV special_i1i1ecSUS3$)Nz calc_i1e(rrrus rTrr,rrV special_i1elog_ndtrcSUS3$)Nzcalc_log_ndtr(rrrus rTrr1s qc+rVspecial_log_ndtrmodified_bessel_i0cSUS3$)Nzmodified_bessel_i0_forward(rrrus rTrr73A3a8rVcSUS3$rrrus rTrr8rrVspecial_modified_bessel_i0modified_bessel_i1cSUS3$)Nzmodified_bessel_i1_forward(rrrus rTrr=rrVcSUS3$rrrus rTrr>rrVspecial_modified_bessel_i1modified_bessel_k0cSUS3$)Nzmodified_bessel_k0_forward(rrrus rTrrCrrVspecial_modified_bessel_k0modified_bessel_k1cSUS3$)Nzmodified_bessel_k1_forward(rrrus rTrrHrrVspecial_modified_bessel_k1ndtrcSUS3$)Nz calc_ndtr(rrrus rTrrNs 1#Q'rV special_ndtrndtricSUS3$)Nz calc_ndtri(rrrus rTrrSrrV special_ndtri polygammac *USUSUSUSUSUS3 $)Nz == 0 ? calc_digamma(z) : (z == 1 ? trigamma(z) : calc_polygamma(rrrrs rTrrXs3S%aSaS0A!DWXYWZZ\]^\__a brVscaled_modified_bessel_k0cSUS3$)Nz"scaled_modified_bessel_k0_forward(rrrus rTrr`:1#Q?rV!special_scaled_modified_bessel_k0scaled_modified_bessel_k1cSUS3$)Nz"scaled_modified_bessel_k1_forward(rrrus rTrrerrV!special_scaled_modified_bessel_k1spherical_bessel_j0cSUS3$)Nzspherical_bessel_j0_forward(rrrus rTrrks4QCq9rVspecial_spherical_bessel_j0zetacSUSUS3$)Nzzeta(rrrrs rTrrps52aS*rV special_zetachebyshev_polynomial_tcSUSUS3$)Nzchebyshev_polynomial_t_forward(rrrrs rTrru:1#Rs!DrVspecial_chebyshev_polynomial_tchebyshev_polynomial_ucSUSUS3$)Nzchebyshev_polynomial_u_forward(rrrrs rTrrzrrVspecial_chebyshev_polynomial_uchebyshev_polynomial_vcSUSUS3$)Nzchebyshev_polynomial_v_forward(rrrrs rTrrrrVspecial_chebyshev_polynomial_vchebyshev_polynomial_wcSUSUS3$)Nzchebyshev_polynomial_w_forward(rrrrs rTrrrrVspecial_chebyshev_polynomial_wlegendre_polynomial_pcSUSUS3$)Nzlegendre_polynomial_p_forward(rrrrs rTrr9!BqcCrVspecial_legendre_polynomial_pshifted_chebyshev_polynomial_tcSUSUS3$)Nz'shifted_chebyshev_polynomial_t_forward(rrrrs rTrrB1#Rs!LrV&special_shifted_chebyshev_polynomial_tshifted_chebyshev_polynomial_ucSUSUS3$)Nz'shifted_chebyshev_polynomial_u_forward(rrrrs rTrrrrV&special_shifted_chebyshev_polynomial_ushifted_chebyshev_polynomial_vcSUSUS3$)Nz'shifted_chebyshev_polynomial_v_forward(rrrrs rTrrrrV&special_shifted_chebyshev_polynomial_vshifted_chebyshev_polynomial_wcSUSUS3$)Nz'shifted_chebyshev_polynomial_w_forward(rrrrs rTrrrrV&special_shifted_chebyshev_polynomial_whermite_polynomial_hcSUSUS3$)Nzhermite_polynomial_h_forward(rrrrs rTrrs82aSBrVspecial_hermite_polynomial_hhermite_polynomial_hecSUSUS3$)Nzhermite_polynomial_he_forward(rrrrs rTrrrrVspecial_hermite_polynomial_helaguerre_polynomial_lcSUSUS3$)Nzlaguerre_polynomial_l_forward(rrrrs rTrrrrVspecial_laguerre_polynomial_lzdict[str, OverridesData]rZc^[U4Sj[RR[RR[RR [RR 455$)Nc3.># UH nTU;v M g7frnr)rrvrfs rTr$is_buffer_removed..s  A   s)anyr7rremoved_bufferskernelinplaced_to_removerfs`rTis_buffer_removedrsU  GG # # HH $ $ GG & & HH ' '   rVcD^\rSrSrSrSU4SjjrSSjrS SjrSrU=r $) DeferredLineizHA line that can be 'unwritten' by adding name to V.graph.removed_buffersc^>[TU]U5 Xl[U[5(aegrn)r\rrfrdr&)rirfliner^s rTrDeferredLine.__init__s,  d$455555rVcP[UR5(d UR$grn)rrfrros rT__call__DeferredLine.__call__s ++99 rVc.[URU5$rn)rrf)rirs rT _new_lineDeferredLine._new_linesDIIt,,rVr )rfr~rr~r)rr~r{r) rrrrrrrrrrjrks@rTrrsR6  --rVrc"\rSrSrSSSjjrSrg) BracesBuffericL^^[RSUU4Sjj5nU"5$)Nc3># [T5H)nTRS5 T=RS- slM+ [T*5H)nT=RS-slTRS5 M+ Sv [T*5H)nTRS5 T=RS- slM+ [T5H)nT=RS-slTRS5 M+ g7f)N{rH})ranger_indent)_rris rTctx BracesBuffer.indent..ctxs6]s# ! #F7^ ! s#$ F7^s# ! $6] ! s##sC(C+)r{Iterator[None]) contextlibcontextmanager)rirr#s`` rTindentBracesBuffer.indents$  " " $ # $u rVrN)rH)rrr{z'contextlib.AbstractContextManager[None])rrrrr(rrrVrTrrs rVrc*\rSrSr%S\S'S\S'Srg)InplacedBufferir~rr other_namesrNrrrVrTr+r+s OrVr+c8\rSrSr%S\S'SrS\S'S SjrSrg ) ArgNameir~rfFr is_constexprcLURUR(aS3$S3$)Nz : tl.constexprr)rfr/ros rT full_nameArgName.full_names*))$2C2C.LMMLMMrVrNr})rrrrrr/r1rrrVrTr.r.s IL$NrVr.c\rSrSrSSjrSrg) RemovedArgicg)NREMOVEDrros rT__str__RemovedArg.__str__srVrNr})rrrrr7rrrVrTr4r4srVr4c8\rSrSr\SSj5rSSjrSSjr\SSj5rSSjr SSjr SSjr \ R4SS jjrSS jrS S jrS!S jrS"S jrS#SjrS$SjrS%SjrS&S'SjjrS(SjrS)SjrS*SjrS+SjrSrg), KernelArgsicURU[5n[U[5(aU[ U53=X'nU$U$rn)rr6rdr4r)rodictrfresult new_results rT_lookupKernelArgs._lookupsE */4)A fj ) )*0#e*'> >EK*  rVcJ0Ul0Ul0Ul0Ul/Ulgrn) input_buffersoutput_buffersinplace_buffersrYworkspace_argsros rTrKernelArgs.__init__s)-/ACMO/1 24rVc SRSR[[URUR UR UR/555$)NzKernelArgs({})r)formatrmaprrBrCrDrYros rT__repr__KernelArgs.__repr__sS&& II**++,,    rVc"[U[5$rn)rdr4r s rT_buffer_is_marked_removed$KernelArgs._buffer_is_marked_removeds$ ++rVcT[RR(a3[RRRR X5nU[RR ;dU5eXR ;a[[UR U5$XR;a'[[URU5R$URS5(aURSURU5$URSURU5$)Nseedin_ptr)r7rrOmutation_real_namerr rCr r~rDr+rr=r?rBr's rTinputKernelArgs.input#s 77  77$$77;;DGD1772228D82 && &T0067 7 '' '(<(R!R  /01)C03  ,03  -! sF 6F F7Fc~[U[RU5[RR 5[R 5US9n[UR5HupV[RXd5(aIURn[RXd5URU'URURU4s $URUR:waURUR:waMU5e URRU5 URURS4$)a Allocate or extend a workspace buffer of nelem elements. This function manages the allocation of a workspace buffer. It either creates a new WorkspaceArg or extends an existing one. Note: - Calling this function will in-place mutate the args by adding or updating a WorkspaceArg. - The codegen for generating the Python argdefs and call_defs will check this field and allocate the buffer accordingly. - A new argument "ws_ptr" will be present in the generated code. Args: nelem (sympy.Expr): The number of elements to allocate. zero_fill (bool): Whether to initialize the buffer to zero. dtype (torch.dtype): the dtype of the workspace tensor Returns: Tuple[str, str, int]: A tuple containing: - "ws_ptr": A string identifier for the workspace pointer. - "workspace_{i}": agraph level unique identifier for the workspace tensor. - offset: An integer representing the item offset in the workspace. )rrrrrr)rrrr7rget_current_device_or_throwrrrErrrrrr[)rinelemrrargr existing_argrs rTr\KernelArgs.workspaceSs 8'11)<77668#//1   ))<)<=OA$$\77%++)5):):<)M##A&#.. 0G0GOO''3>>9 ++s~~=  > > ""3'~~s~~q00rVc [RR5n[U[R [ RSSURSUR3US9nURH,nURUR:XdMX4:XaM&X445e URRU5 UR$)a Lazily allocate a graph-wide semaphores buffer with at least min_size. This is a single buffer shared by all kernels and zero initialized once at graph start. Each kernel must leave the buffer zeroed on exit. Warning: multiple calls to this function will return the same buffer. Args: min_size: the number of int32 semaphores required Returns: name of the semaphores buffer sem_ptr semaphores_r")rrrrrr) r7rrerrrruint32rzrrErr[)rimin_sizecurrent_devicergrhs rT semaphoresKernelArgs.semaphoress<<>'66,, $^%8%8$9>;O;O:PQ!  !//L&&#..8*?S,??*0 ""3'~~rVcz^[U[5(d[U5U45e[R"U5nX R ;aUR U$TUR R 5;a1T[U4SjUR R 5553mTUR U'T$)Nc3V># UHoRT5(dMSv M g7f)rHN)r=)rrrfs rTr)KernelArgs.seed_offset..sU(>1,,tBTQQ(>s) ))rdrrzrrrYr$sum)rirfrhs ` rT seed_offsetKernelArgs.seed_offsets%%%;U U';;% e$ MM !==' ' 4=='') )&U (<(<(>UUVW  $ e rVc[U[R5(d[U5U45eURS:XaSUR U'gUR SUR U5$)NrPks)rdrSymbolrzrfrYr?r's rTrKernelArgs.sizes[$ --AT D/AA- 99 "(DMM$ ||D$--66rVc[URR5URR5URR55$rn)rrBkeysrCrYros rT call_namesKernelArgs.call_namessA    # # %t':':'?'?'A4==CUCUCW  rVc&URRUS5nUb![U[5(d UR$UR RUS5nUb[U[5(dU$UR RUS5$)z+ Returns inner name of a given outer name. N)rDrrdr4rrCrB)rirfinplacedr]s rTarg_nameKernelArgs.arg_names''++D$7   8Z(H(H&& &))--dD9  ":k:+N+N !!%%dD11rVcU$rnr)rir^rs rT wrap_ptr_argKernelArgs.wrap_ptr_args rVc[U5$rnr)rirs rT wrap_size_argKernelArgs.wrap_size_args 4yrVNcSSKJn UcSSKJn Un/n/n/n[URR 55Hn[ U[5(aMURSnURn [RRU5n Xn URU SU 35 URURX55 URU S35 M URR!5HupXR;aM[RRU5n Xn URSU SU 35 URURX55 URSU S35 M UR"R!5HupXR;d[ U [5(aM+[RRU5n Xn URU SU 35 URURX55 URU S35 M UR$R!5Hup[ U[&R(5(aE[+U[,R.5(a&URSU 35 URS 5 O+URSUS U 35 URSU35 URUR1U55 [RR2(dM[RR2R5U5 GM UR6(aS 5eXTU4$) NrH) INDEX_TYPE)rrz* *zconst z const float z const float zWorkspace not supported on CPU )rrrr0rDr$rdr4r,rr7rrr[rrBrrCrYrrzrrUNBACKED_FLOATr wrapper_codeensure_size_computedrE) ridtype_to_cpp_typerr call_argsarg_defs arg_typesrouterinnerr cpp_dtype maybe_inners rT cpp_argdefsKernelArgs.cpp_argdefss *  $ / ,   t33::<=H(J//((,E''EGG%%e,E)0I OOykE73 4   T..u< =    {!_ ->!..446LE,,,GG%%e,E)0I OOfYKr%9 :   T..u< =   vi[2 37#'"5"5";";"= E,,, ; 0S0SGG%%e,E)0I OOykK=9 :   T..u< =    {!_ -#>!MM//1LE%..>++44,ug 67  /& AeW =>  6*!67   T//6 7ww###$$99%@2&&I(II&I--rVc /n/n/n/n[URR55Hn[U[5(aMUR [ UR55 UR URS5 UR [RRURS55 UR [URURS[RRURS5S95 M [URR5UR R55HupgX`R;d[U[5(aM+UR [ U55 UR U5 UR [RRU55 UR [UU[RRU5S95 M UR"R5HupgUR [ U55 UR U5 UR [%U55 UR ['Xv55 [RR((dM[RR(R+U5 M UR,HnnUR [ UR55 UR UR.5 UR U5 UR UR05 Mp XXC4$)Nr)rfrr)r0rDr$rdr4r[r.rr,r7rrrrrBrrCrYrzr rrrErr) rirrrprecompile_argsrrrrgs rTpython_argdefsKernelArgs.python_argdefss#%! ! /1t33::<=H(J// OOGH$7$78 9   X11"5 6   QWW..x/C/CB/GH I  " "!,,#//3''++H,@,@,DE  >"    $ $ &    % % ' LE ,,, 5*0M0M OOGEN +   U #   QWW..u5 6  " " ''++E2  "!MM//1LE OOGEN +   U #   T%[ )  " "75#8 9ww###$$99%@ 2&&C OOGCNN3 4   S^^ ,  " "3 '   SYY ' ' O>>rVc## [URR55Hn[U[5(aMUR HnU[ RR;dU[ RR;aMAX R;aURUUR4v X R;dM~[[URU5UR4v M M g7frn)r0rDr$rdr4r,r7rr r rBrrCr r~)rirothers rTaliasesKernelArgs.aliases2st33::<=H(J//!--QWW777 ; ;;...,,U3X5H5HHH///sD$7$7$>?ATATTT.>s CC> 4C>c[URRU[5[5=(a. [UR RU[5[5$rn)rdrCrr6r4rDr's rT is_removedKernelArgs.is_removedAsK    # #D' 2J N--11$@*M NrVc[5n[URR55H8n[ U[ 5(aMUR URS5 M: URR5H<up4X0R;d[ U[ 5(aM+UR U5 M> U$)Nr) rr0rDr$rdr4rr,rCr)ri live_outsrrrs rTlive_output_buffersKernelArgs.live_output_buffersIs%/\ t33::<=H(J// MM(..r2 3>!//557LE,,, 5*0M0M MM% 8rV)rDrBrCrYrE)rr~r<z6Union[dict[_T, Union[str, RemovedArg]], dict[_T, str]]rfrKr{r~rzr})rfr r{rr_)r\r~r]r~r{r|)rfrrrrrr{ztuple[str, str, int])rnrr{r~)rfr~rhrr{r~)rfrdr{r~)r{z Iterator[str])rfr~r{r )r^r~rrr{r~)r SymbolLiker{r~rn)rz Optional[dict[torch.dtype, str]]r{z&tuple[list[str], list[str], list[str]])r{z?tuple[list[ArgName], list[str], list[KernelArgType], list[Any]])r{zIterator[tuple[str, str]]rm)r{zOrderedSet[str])rrrrrr?rrJrMrSr8rbrrr\rprvrr~rrrrrrrrrrrVrTr:r:s  E    5  ,, @B4:HM{{-1-1,0-19D-1 -1^8 7 2EI4.!A4. /4.l1? H1?f UN rVr:cx^\rSrSrSrS S U4SjjjrS SjrS SjrSSjrSSjr S Sjr S r U=r $)riVa4A CSEVariable is just a name for an expression but it is useful to be able to annotate them on a backend dependent basis. To do so, the backends can simply overload `Kernel.create_cse_var` The "CSEVariable.update_on_args" method gives you a hook for annotations See example of TritonCSEVariable in triton.py c>[TU]5 [U[5(d[ U55eXlX lSUlX0lX@l grr) r\rrdrrzrfbounds use_countrr)rirfrrrr^s rTrCSEVariable.__init__]sG &+..<V <.    rVcUR$rnr ros rTr7CSEVariable.__str__ls yyrVc,[UR5$rn)hashrfros rT__hash__CSEVariable.__hash__osDIIrVcb[U[5=(a URUR:H$rn)rdrrf)rirs rT__eq__CSEVariable.__eq__rs!%-I%** 2IIrVcgrnr)rirfrtrs rTupdate_on_argsCSEVariable.update_on_argsus rVcPURRSUR<S3$ra)r^rrfros rTrJCSEVariable.__repr__xs$..))*!DII=::rV)rrrfrrrj)rfr~rValueRanges[Any]rrrrGr})r{r)robjectr{r)rfr~rtr rr r{r|) rrrrrrr7rrrrJrrjrks@rTrrVs_(, $  ! %    J ;;rVr AugmentedKeyT)default)boundr.cx\rSrSrSrSSSjjrSSjrSSjrSSjrSSjr SS jr SS jr SS jr SS jr \R"5S S SSS.SSjjr\R"5SS4SSjjr\R"5SS4SSjjrSrg)CSEiz Common subexpression eliminationNcXlX l0UlX0lU=(d 0UlU=(d 0UlU=(d [ R"5Ul[5Ul U=(d 0Ul grn) rr^_cache name_prefix store_cachereduction_cache itertoolsriter_buffer_idsrinvalidated_stores varname_map)rirr^r iter_buffersrrrs rTr CSE.__init__sg  FH &ALARPR  !r 6B5VY__EV3=<7B7HbrVcL/URR5QH4up#X1;dM URU URRU5 M6 U(a<URR5VVs0sHupEXQ;dM XE_M snnUlg0Ulgs snnfrn)rrrrr)ri keep_varsrftmprrs rT invalidateCSE.invalidates44++1134ID#$$T*''++D15 ,0KK,=,=,?R,?DA1>414,?RDKDKSs 9 B B c [U5"URURURURUR UR URS9$)N)rr^rrrrr)rzrr^rrrrrros rTclone CSE.clonesNDz;;;;((--(((( 00  rVcUR5n[UR5Ul[UR5Ul[UR5UlU$)zNReturn a copy of using ScopedDict so changes to *_cache aren't visible in self)rr+rrr)rinew_cses rT scoped_copyCSE.scoped_copysH**,#DKK0",T-A-A"B()9)9:rVc"[[U5$)z@Override this method to augment cache key with backend specifics)r rri cache_keys rT augment_keyCSE.augment_keysM9--rVc>X RURU5'grnrr)rirr_s rTputCSE.puts36 D$$Y/0rVc>URU5UR;$rn)rrrs rTcontains CSE.containss *dkk99rVcXURRURU5S5$rn)rrrrs rTtry_get CSE.try_gets"{{t// :DAArVc>URURU5$rnrrs rTrCSE.gets{{4++I677rVT)rrg assignmentrrc p[U[5(a URnU(d U(de[U[5(aEURR U5UlU=R S- sl[[U5$[U[5(aUR5nO;[U[5(a URnO[U[5(deUnURU5n Uc U(dSnU (GdURX6U5n UR!X5 U(Ga["R$R&(a(["R$R&R)USS9 [U[5(aUU(a UR+UR,U S35 UR/U5 UR+UR05 U $[U[5(aRU(deUR+UR3UR,U SURUR0355 U $U(a UR,U SUUR03n OUUR03n UR+U 5 U(a[[4R6R8(d[4R6R:(aUb[=5S:wa [?XU5 U $U RR U5U lU =R S- slU $)NrHrT) only_oncez =z = r) rdr4rhrrtightenrr rr)getvaluer&rr~rnewvarrr7r  current_nodecodegen_originating_inforrsplicer^rrrrrr(r) rirr rrgrrrrrrs rTgenerate CSE.generates{ dH % %::D "" dK ( (++--f5DK NNa N. . n - - I . / / IdC(( ((Ill9% =E++fU3C HHY $88((HH))BB$CdN33!((DKK=R)@AMM$'$$T[[1: 9 &677%%:$$$++se3tyyk$++'WX4 -""&++se3tfT[[MJ"& }5$$T*#"//KK%22JJ!-/1U:#F7  ++F3CJ MMQ M rVcUR[UR53n[RR XAX#5nXPR U'U$rn)rrrr7r create_cse_varr)rirrrvar_namers rTr CSE.newvarsN &&'T-A-A(B'CDhh%%hE%(" rVc^[R"TUR;U4Sj5 [RR TX#U5nXPRT'U$)Nc>ST3$)Nzduplicate name: rr srTrCSE.namedvar../s 4DTF2KrV)r _check_valuerr7r r)rirfrrrrs ` rTnamedvar CSE.namedvar'sQ  (( (*K hh%%dF5A!$ rV) rrrrrrrr^r)rrrNNNN)rr~r^r~rr~rzOptional[itertools.count[int]]rz.Optional[MutableMapping[str, CSEVariableType]]rz[TU]5 [R"5Ulgrn)r\rr& ExitStack exit_stackrir^s rTrCodeGen.__init__7s $..0rVc:URR5 U$rn)r __enter__ros rTr CodeGen.__enter__;s !!# rVc<URRXU5 grn)r__exit__)riexc_typeexc_valexc_tbs rTrCodeGen.__exit__?s   F;rV)rrzrrr rr rr r{r|) rrrrrr rrrjrks@rTrr6s1<S!jr#S?S"jr$S@S#jr%S$r&U=r'$)AKerneliCrr~ newvar_prefixr^Nz'Optional[Callable[[], OpsHandler[Any]]] overridescH>[TU]5 U(a[=RS- slU=(d [ 5Ul[ 5Ul[ 5Ul[ 5Ul SUl SUl SUl SUl [URUR 5Ul[%5Ul[%5UlSUlSUlSUlSUl[%5Ul[%5Ul0UlSUlSUlg)NrHFr)r\rr generated_kernel_countr:rtr)loadscomputestoresatomic_add_foundnum_load num_store num_reductionrrr^csermust_keep_buffersstore_buffer_names _load_mask _load_otherrnode_to_boundsr r inplace_update_buffersmin_elem_per_thread kernel_name)rirtincrease_kernel_countr^s rTrKernel.__init__Hs   * *a / *(JL #% %' $& % .1$2D2Ddkk.R2<,3=<)-4859OS0: 3=< 79##$ *.rVc## URnXlURR5R5UlSv X lg!X lf=f7frn)rrPr get_boundsr')rir+priors rTset_current_nodeKernel.set_current_nodemsH!! "jj//1<<> &  %  sAAAAAAc## UcUnUSL=n(a [5nURnURnURnURnXlX lX0lUR 5UlSv XPlX`lXplXlU(aU(aS5egg!XPlX`lXplXlU(aU(aS5eff=f7f)Nz$unexpected store inside swap_buffers)r)rrrr"r) rilbcbsbdisallow_storesrrrr"s rT swap_buffersKernel.swap_buffersws :B Dj (? (!B ,,hh   ??$ F J"L KHEEEv2 J"L KHEEEv2sA2C5B(9/C(0CCcg)zOverride kernel emission. Return True if overridden, False to use default. External template handlers (e.g. Helion) can override this method to implement custom kernel emission to the wrapper. Fr)rirsrc_coder* node_schedule kernel_pathget_kernel_metadatas rTemit_kernel_overrideKernel.emit_kernel_overridesrVc[ernr&r s rTr Kernel.loadr*rVcURnURUlURX5X0l$!X0lf=f)z+A load the depends on an index we have read)rrr)rirfrr/s rT indirect_loadKernel.indirect_loads2  DJ99T)JJs!6>c[ernr&rs rTrKernel.store_reductionr*rVc[ernr&rs rTr Kernel.store "!rVcD[[U5RS35errrs rTrKernel.device_assert_asyncrrVc[ernr&rs rTrKernel.reduction "!rVc[ernr&)rirfrrh extra_metas rTpartial_accumulateKernel.partial_accumulaterNrVc[ernr&r!s rTr% Kernel.scans "!rVc[ernr&r(s rTr+ Kernel.sortrNrVc[ernr&ros rT var_rangesKernel.var_rangesr*rVc[e)z# See [Note: Inductor bucketize op] r&r.s rTr5Kernel.bucketizes "!rVc[ernr&ros rTassert_functionKernel.assert_functions!!rVc [U[5(a [U5n[U[5(d[U55eUb[U[5(deUb[U[5(deU(a!U(aSUSUSUSUS3 nUSUSU3nO#U(a USU3nUnOU(deUSU3nUnU(a SUSUS3nURSUSUS3$) Nrbz <= r z < rz) | ~(z, "index out of bounds: z"))rdrr~rzr])rirrr maskr cond_prints rTindirect_assertKernel.indirect_asserts c; ' 'c(C#s##.T#Y.#} 5# 6 666} 5# 6 666 UugT#eC5E7![TU]5 UR(deURR [ R "[XR5555 URR [ R"U55 U$rn) r\r rr enter_contextr7set_ops_handlerCSEProxyset_kernel_handlerr s rTr Kernel.__enter__ sj ~~~ %%  ht^^-=> ?  %%a&:&:4&@A rVcF>UR5 [TU] XU5 grn)remove_kernel_local_buffersr\r)rirrrr^s rTrKernel.__exit__) s ((* F3rVc^^[RRmT(dg[U4SjUR55n[5mURHmnX R ;dMX R R;dM/TRX!5(dMGU=RS-sl TRU5 Mo THnX R R;aUR RUn[U[5(aML[U4SjUR55nU(aUR!U5 UR"RU5 MUR%U5 M g)z Any buffers that are both created and have a last use in the same kernel can be removed. Note that V.graph.scheduler can be None when codegening triton template kernels. Nc3># UH4nUTR;dMTRUR5v M6 g7frn) name_to_bufdefining_op_name)rr^rOs rTr5Kernel.remove_kernel_local_buffers..8 s?& .i+++ :I ! !# & 7 7 9 9.s?$?rHc3,># UH oT;v M g7frnr)rr!names_to_removes rTrrvN sK?a/1?s)r7rrOrr$r#rtrB$can_buffer_be_removed_through_fusionr rrDrdr4r'r,remove_inplace_bufferr  remove_buffer)rifused_node_namesrfr^rerxrOs @@rTrp"Kernel.remove_kernel_local_buffers- s(GG%%  %& ..&   ,6<++D222 7 77BB!###D),$Dyy000ii//5c:..K3??KK..t4''++D1""4($rVc[RSU5 [URRU'UR R U5 g)Nzremove_buffer(%r))rrRr6rtrCr rr's rTr{Kernel.remove_bufferU s; %t,)0   &   &rVc[RSU5 [URRU'UR R U5 g)Nzremoving_inplace_buffer(%r))rrRr6rtrDr rr's rTrzKernel.remove_inplace_buffer] s9 /6*1 !!$'   &rVc [U[[45(a!UVs/sHo RU5PM sn$[R R RU5n[URSS9nUVs0sHmn[U[R[R[R[R45(dMRX R R#U5_Mo nn[%X5$s snfs snf)NcUR$rnr )ss rTr(Kernel.rename_indexing..j s!&&rV)r)rdlisttuplerename_indexingr7rrYrVsorted free_symbolsrr UNBACKED_INTSIZEPRECOMPUTED_SIZErrtrr.)rirrvsorted_symbols replacementss rTrKernel.rename_indexingb s edE] + +5:;U((+U; ;  ))%0 2 28HI$  #%%II))''  !Ayy~~a #   %..!<  sD?ADDc[U0UD6$rn)r)rirtrs rTrKernel.create_cse_varz sD+F++rVc\UcgURRUR55$)z3 Returns arg name of a given input or output node. N)rtrr)rir+s rTrKernel.arg_name} s' <yy!!$--/22rV)r%r&rtrrr"rr(r r*rr)r#r'rr!r r r$r)NT)rtzOptional[KernelArgs]r+rr{r|)r+rFr{r%rj)r3r)r4Optional[IndentedBuffer]r5rr{r%)r:r~r*r~r<r~r{rrfr~rrr{rrfr~rrrhrr{r|rn rfr~rrrhrr]r6r{r|rf rrrrrr5rh+Union[CSEVariable, tuple[CSEVariable, ...]]r{r) rfr~rr5rhrrPdict[str, Any]r{r|r"rhr#zUCallable[[tuple[CSEVariable, ...], tuple[CSEVariable, ...]], tuple[CSEVariable, ...]]r$tuple[CSEVariable, ...]r{r r"rhr$rr)rr*rr{r)r{zdict[sympy.Symbol, sympy.Expr]r$rr/rkr0rr1rr2rr3rlr4zOptional[CSEVariable]r{rr}) rzUnion[CSEVariable, str]rr r r r`z!Optional[Union[CSEVariable, str]]r{r~re)rrr{r~rrrz)rfr~r{r|)rz;Union[list[sympy.Expr], tuple[sympy.Expr, ...], sympy.Expr]r{r)rtr rr r{r)r+rAr{r )(rrrrrrr^rrr&r'r0r7r>rrCrrrrrQr%r+rXr5rr]rbr rgr rrpr{rzrrrrrjrks@rTrrCsM3FC9=I6=PT#/(#/HL#/ #/#/J&&(,'+ F F %F % F  FF:     ""SW"" *"3>"FO" "  """& " ; " 5 """&" " # "  ""'" " ( " !""'"(" "  " ! ""4804 " "C "& " $ "  "1 ". "  """37 V $VV V 0 V V<""&0"9="FJ" " "4&)P'' /P/ /0,33rVrc@\rSrSr%SrS\S'SrS\S'SrS \S 'S rg) ri rDz ClassVar[str]rNrrrr~ops_namer) rrrrrrrrrrrVrTrr s!"C"#'E 'HcrVrc^SSKnURURS9$![a gf=f)Nr) undefined)jinja2 EnvironmentStrictUndefined ImportError)rs rT jinja2_envr s?!!,,"   s  ,,c\rSrSrSr\SSSjj5r\SSj5r\SSj5rSSSjjr \ SSj5r \ SS j5r SS jr SS jrSS jrS rg)KernelTemplatei z^ Base class for defining kernel templates. Children classes: TritonTemplate, CUTLASSTemplate cURS5n[U5S:a"USSVs/sHnSU-U-U-PM snUSS&SRU5$s snf)NTrHrr) splitlinesrr)source num_indentsindents_spacinglinesrs rTindent_except_first"KernelTemplate.indent_except_first sh!!$' u:>INqrIR&4<E!"Iwwu~sAc[5nUcg[RURS'SSKJn UR U5$!Uan"SSU5nU"U5UeSnAff=f)Nrr)TemplateSyntaxErrorc6^\rSrSrSU4SjjrSSjrSrU=r$)IKernelTemplate._template_from_string..DetailedTemplateSyntaxErrori c>[TU]URURURUR 5 Xlgrn)r\rmessagelinenorffilenameoriginal_error)rirr^s rTrRKernelTemplate._template_from_string..DetailedTemplateSyntaxError.__init__ s<G$&..&--&++&//  +9'rVcXSURS3nUSURS3- n[URS5(aURRR S5nUS- n[ SURS- 5n[[U5URS-5n[X45HwnXPRS- :XaSXS-S X%S3- n[URS 5(a'US S URRS- --S -- nMfMhXS-SX%S3- nMy U$)NzError in template at line  zError message: rz Context: rrrHz: --> columnz rz^ z: ) rrr[rrsplitmaxminrr r)ri error_inforstartendrs rTr7QKernelTemplate._template_from_string..DetailedTemplateSyntaxError.__str__ s:#=dkk]"!MJODLL>"DDJt22H== $ 3 3 : : @ @ F"l2 #At{{Q 7!#e*dkkAo>!&u!2A KK!O3 *QvehZr.J J #*4+>+>#I#I$.(/*-1D1D1K1Ka1O*P)Q*/)0%&J$J!+QvehZr.J J "3&%rV)r)rrr{r|r})rrrrrr7rrjrks@rTDetailedTemplateSyntaxErrorr s9&&rVr)rrrfiltersrr from_string)renvrers rT_template_from_string$KernelTemplate._template_from_string sjl ;-;-O-O )*.# 8??6* *"! 8 &.A &>.a0a 7C! 8sAA# AA#c0^^[RRm[U[[ 45(a0UVs0sH!oR 5UR5_M# snmO UR 5UR50mSUU4SjjnU$s snf)Nc@>TRU5nUbU$T"U5$rn)r)rfr=_get_dtype_reallookups rTr1KernelTemplate._fake_get_dtype..get_dtype s'ZZ%F! "4( (rV)rfr~r{r)r7rrrdrrr) fake_outsr^rrrs @@rT_fake_get_dtypeKernelTemplate._fake_get_dtype s|''++ i$ / /AJK#llncmmo5KF((*I,?,?,ABF ) ) Ls(BNcXlX lgrn)rf_hash)rirfrs rTrKernelTemplate.__init__ s   rVcUR$)z entry point to override for templates to ensure a uid e.g. through a prefix the purpose of this is that every KernelTemplate/ExternKernelChoice is unique in the system, but reproducible e.g. restarting pytorch should yield the same id r ros rTuidKernelTemplate.uid syyrVcUR$)z source hash for a Template. Templates can optionally provide a src hash to make it easier to cache/validate that a template has not changed from one version to another. Override this if that detection is different for your specific Template )rros rTsrc_hashKernelTemplate.src_hash szzrVc \/nUR"U40UD6nUc[U5S:XaUS$g)z Maybe generates a new ChoiceCaller and returns it, or None if generation fails. kwargs: Additional kwargs to be passed to self.generate() to generate a new ChoiceCaller. NrHr)maybe_append_choicer)rir temp_choicesr=s rTchoice_or_noneKernelTemplate.choice_or_none s< #% )),A&A >c,/14? "rVc URUR"S0UD65 g![aNn[R SU[ U5[R 5[R:S9 UsSnA$SnAff=f)z Maybe generates a new ChoiceCaller and appends it into existing choices. Returns None if success, otherwise returns the error. choices: A list of ChoiceCallers. kwargs: Additional kwargs to be passed to self.generate() to generate a new ChoiceCaller. Nz3Cannot Append Choice: %s. KernelTemplate type is %s) stack_infor) r[rrrinforzgetEffectiveLevelrPINFO)richoicesrrs rTr"KernelTemplate.maybe_append_choice sl  NN4==262 3"  HHET 002W\\A   H s!$ A<AA71A<7A<c [e)z= Generates a ChoiceCaller instance from the given arguments. r&)rirs rTrKernelTemplate.generate( s "!rV)rrf))rr~rrrrr{r~)rr~r{r )rzUnion[list[Buffer], Buffer]r{zCallable[[str], torch.dtype]rn)rfr~rr r{r|r})r{zUnion[str, None])rr r{zOptional[ChoiceCaller])rrrr r{zOptional[NotImplementedError])rr r{r?)rrrrrrrrrrrrrrrrrrrVrTrr s >?"%8; *8*8X. %"  ,/ &."rVrcd^\rSrSrSrSrSU4SjjrSSjrSSjrSSSjjr SSjr SSjr SS jr SSS jjr SS jrSS jrSS jrS SjrS!SjrS"SjrS#S$SjjrSrU=r$)%rli0 zvA ops handler that proxies calls to `kernel` and its handler and returns `CSEVariable`s with correct shape and dtype. c^>[TU]5 SSKJn U"5UlXlX lg)NrValueRangeAnalysis)r\rrr vr_analysisr parent_handler)rir rrr^s rTrCSEProxy.__init__7 s' /-/ ,rVc z^^^^ ^ ^ ^ ^UR"T/TQ70TD6m [URT5"T0TD6n[5n[ 5n[ 5m [UT5nSm SmTS:XaT S:XaUR m URmO}TS:XaUT S:XaO[RRRR[RS5R m SmO"T S;a[UT5nU"T0TD6m U"T0TD6mT S;aT ceSm SUU U UUU U U4Sjjn [R "X5$) Nmaskedrr)rrr)rrrc>[T [[45(aT T OT n[T [[45(a2[T 5S:a#[T S[[45(aT T OT nT S- m [U[5(a,TS:XaUR cXlUR cX l[RRR[RRUTT T S9nURTTT5 [RR(d[RR (a)Uce[#[RRX15 [RR$(a*T ce['[RRUT 5 [R((a$[+[RRU5 U$)NrrHrrrr)rdrrrrrrr7r r"rrrrrrrrrr runtime_triton_nan_assertsr ) r var_dtype var_shapecsevarrtrrrrfr? output_idx output_shapes rTdo_cse!CSEProxy._default..do_cse` s lT5M::Z(! lT5M:: %)|Au >>Z("  !OJ![))e#'G77?'GXX\\**  "" +F  ! !$f 5##??&&>> ,,,AHH,,f@"">>#///AHH,,flC00!((**F3MrV)rzUnion[str, CSEVariable]r{r)_bound_variablerSrr!r$r(rrr7 interpreterrr rrrpytreetree_map)rirfrtrrh dtype_handler shape_handlershape_opdtype_oprrrr?rrs ``` @@@@@rT_defaultCSEProxy._default? s:%%d>#''e  L 0 0}d3H#T4V4L#T4V4L ' '+ ++ 0 0 dv--rVc>^ SSKJn SSKJn SSKJn [ [RU5(a[R"5$[ [RU5(a[R"5$[ [R[5(a[R"5$[RRm T RU:XaURRb[ URR[ 5(d$[#URR55eURRR%T [R"55$[&R((az[+XA5(aj[-U 4SjS55(a[R"5$U(aeS Sjn[/[1Xr55n[3UR4U5"U6$[R"5$) z If the variable comes from an FX node, we forward the bound we have already computed Else, if the variable when codegen'ing another op, we try to compute its bounds rr)TritonTemplateKernelrH)CUTLASSTemplateKernelc3@># UHoTR;v M g7frn)r2)rrfx_nodes rTr+CSEProxy._bound_variable.. sV0U1&0Us) set_indirectrr%c[U[5(a UR$[U[R5(a [ U5$U$rn)rdrrrrZrrus rT arg_to_bound.CSEProxy._bound_variable..arg_to_bound s8a--88O5::..&q>)HrV)rvr r{r )rrselect_algorithmr cutlass.kernelr rdr7r rrrr1rr2r'dictrzrrcompute_all_boundsr[r rrIrSr) rirfrtrrr r r arg_boundsrs @rTrCSEProxy._bound_variable sy 0;9 ahh 4 5 5&&( ( ahh 5 6 6&&( ( amm[ 1 1&&( (--,, >>T !dkk&@&@&Ldkk88$??  **B ?;;--11';;N;N;PQ Q  & &73E+L+LV0UVVV"**,,  : c,56J4++T2J? ?""$$rVc[U[5(a[R"U5n[U[R5(d[ U5U45eUR RS:GaU(a[R"U[R"U[R55nUR RS:a.[R"US5n[R"XeU5nOUn[ R""5nUR [ R""5:wa[U[R$5(aUR [![&*S5-n[!URU-URU-5nUR RS:a!UR [!S[&5-n Xy-nUR(R*R-UR(R.UUUR0UR2S9nUR4R7XU5n [9U5(avUR RS:+n [U[R$5(+=(d UR RU:+n UR(R;XX5 U $)Nrrr)rdrrrrZrzrrr2rrrlongr ltrrrNumberrr r"rrrrrrr'r ) rirrrrstmr new_bounds neg_boundspos sympy_var assert_lower assert_uppers rTrCSEProxy.indirect_indexing s dC ==&D$ ++?d4j$-??+ ::  a ggc3>>$ #CD::##q(QB))BS1C%,,.Jzz[0022z$ 7U7U!ZZ+vgr*BB ($$t+Z-=-=-D ::##q(**{1f'==C!+!1J++//** ##!iiii +C''99#UK 5 ! ! # 0 0A6L)$ ==   4BL KK $ $Yl QrVc:URRXX45$rn)r r rs rTr CSEProxy.check_bounds s{{''EAArVcXRRR;a)[RRR U5 [ U[R5(aURRX5$URRRnX;aX1$URRX5nURS:XaUR=RS- sl U$rr)r r"rr7r#rrrTMPrCrrrr)rirfrrouts rTr CSEProxy.load s ;;??55 5 HH & & * *4 0 udhh / /;;,,T9 9kkoo11  $ $kkt+ ==A  KK A %  rVczX RRRU'URR(a~U[R R ;a_URRRU5nUR5H%nX RRRU'M' gggrn) r r"rrr7rname_to_buffer get_output get_mutations)rirfrhr^ other_names rT_update_store_cacheCSEProxy._update_store_cache s,1 ##D) ;; # #0F0F(F++**55d;C!//1 :? ++J72)G #rVc&URRRU5 UcURX5 U[R R ;a:URRXX4S9 UR=RS- slgg)N)r]rH) r r$rr2r7rr rr rs rTrCSEProxy.store sr &&**40 <  $ $T 1 qww.. . KK  d5  < KK ! !Q & ! /rVc:URRX5 grn)r rrs rTrCSEProxy.device_assert_async" s ''2rVc6URR"U6 grn)r rQrss rTrQCSEProxy.partial_accumulate& s &&-rVc"URRRU5 URX5 U[R R ;a;UR=RS- slURRXU5$grr) r r$rr2r7rr r rrs rTrCSEProxy.store_reduction) sk &&**40   - qww.. . KK ! !Q & !;;..tEB B /rVcxUR=RS- slURRXX45$rr)r r!rrs rTrCSEProxy.reduction1 s0 !!Q&!{{$$U~MMrVc:URRXU5$rn)r r%r!s rTr% CSEProxy.scan; s{{F;;rVc:URRXX45$rn)r r+r(s rTr+ CSEProxy.sortF s{{CCrVc DURRUUUUUUU5$)a [Note: Inductor bucketize op] Inputs: ------- values: the values to be bucketized. boundaries: a tuple containing (a) the name of the boundaries tensor (which must be sorted, unless the sorting tensor is present), (b) the length of the tensor in the last dimension (i.e. the length of one set of boundaries), (c) the number of elements in the underlying storage (i.e. the length of the flattened tensor, ignoring striding), and (d) the stride of the tensor in the last dimension. boundary_indices: indices into a flattened version of the boundaries tensor, of the same size and shape as "values". Each index points to the first element in the set of boundaries to be used for the corresponding value. indexing_dtype: the dtype to use when indexing into the boundaries tensor. This must be int64 or int32. This additionally specifies the dtype of the return value. right: see "Details" below. sorter: an optional tuple containing (a) the name of an optional sorting tensor, used to access unsorted boundaries without reordering the boundaries tensor, and (b) the stride of the tensor in the last dimension. The values in the sorting tensor are used as indices into the *last* dimension of the boundaries tensor, with all other indices matching. The size of the sorting and boundaries tensors must be equivalent. sorter_indices: must be present if the sorting array is present; see "boundary_indices" for the equivalent definition for the boundaries tensor. Output: ------- The buckets each value belongs in, within a given set of boundaries. 0 indicates a position before the first boundary, and len(boundaries_set) represents a position after the last boundary. Details: -------- Given a value and a set of boundaries, calculate the bucket that each value belongs to. This works differently in 1-D and N-D cases. for values [[-1, 0, 1, 2], [3, 4, 5, 9]], boundaries [0, 4, 4, 8], right=True return = [[ 0, 1, 1, 1], [1, 3, 3, 4]]. for values [[-1, 0, 1, 2], [3, 4, 5, 9]], boundaries [[0, 4], [4, 8]], right=True return = [[ 0, 1, 1, 1], [0, 1, 1, 2]] Note that in the N-D boundaries case, the shape of "values" and "boundaries" must match in every dimension _except_ the last. When right == False, bucket i refers to range (boundaries[i], boundaries[i+1]]. When right == True, bucket i refers to range [boundaries[i], boundaries[i+1]). Boundaries must be non-decreasing, or a sorter must be provided which would re-index offsets in a non-decreasing order (e.g. the second output of torch.sort(offsets)). Otherwise, the result is undefined. )r r5r.s rTr5CSEProxy.bucketizeO s1L{{$$         rV)r rr)r z Kernel[Any]rzOpsHandler[Any])rfr~rtztuple[Any, ...]rrr{r )rfr~rtr rr r{rrb) rrrrcrrrrr{rdrer)rfr~rhrr{r|rnrrf)rtr r{r|rrrrrjr)rrrrrrfrr rrr rr2rrrQrrr%r+r5rrjrks@rTrlrl0 s D-S.j.%h 5 5%5 5  5  5nBB&0B9=BFJB B "@SW'' *'3>'FO' '3.CNNN& N ; N 5 N <' <  <( < ! <D'D(D D  D ! D 4804N N CN & N $ N  N 1N .N  N N rVrl)rSr~r{r|)NNNN)rr~rerrfrrgrrhrriOptional[CustomGraphModulePass]rjOptional[ConfigModule]r{r|)rUnion[torch.device, str, None]r{zOrderedSet[BackendFeature])rrFrrmr{r)rr~r{zOptional[SchedulingConstructor])FF)rr~rrrrr{r)rr~r{rD)rr~r{rErz)rSequence[sympy.Expr]rrGrrGr{r)rr~rr$r{r|)rr~r{r$)rr~rtr rr r{r)rr)rrrrr{r|)rr)rrrrGr{r|)rr)rrr{r|)rr~r{rrrm)r{r ) __future__rrbr& dataclassesenumr(rrPrr:rdrer`abcrrrrrtypingr r r r r rrrtyping_extensionsrrrrtorch.fxtorch._prims_commonr torch.utilsrrtorch.utils._config_modulertorch.utils._ordered_setrtorch.utils._sympy.numbersrtorch.utils._sympy.printersr_PythonPrintertorch.utils._sympy.symbolrrrtorch.utils._sympy.value_rangesrrrrr dtype_propagationr! ops_handlerr"r#shape_propagationr$utilsr%r&r'r(r)r*r+r,r-r.r/r0 virtualizedr1r2r3r4r5r6r7collections.abcr8r9r:r;r<custom_graph_passr=rr>r?r@rArNrCrOrDrErFrGrrJrKrrzrr~rzrr_logginggetArtifactLoggerrrN getLoggerrrU dataclassrXrrrrrr rrr KernelArgTyper"rr$r`rarbrkrmr|rr{rrrcacheryrrrbfloat16rfloat16rrfloat64int8int16rtrruint16rmuint64rrrr r rrmcompile IGNORECASErrrror INT_TO_FLOATrqrZrrrr+r.r4r6r:rrrrrReductionCacheKeyrrrrrrrlrs0rTrqs[ "    #   , ?)3/-GOOD:;:    LL$9>>$DD2- B$hy&9%:N%JK23sELL()JF~~//*E  !=  >/ /(  C  Td=ddN** ###   !!! <<<lIw8H,VW ,.).5"5"p:<6;DFAFDF AF8@D>B:>37B B,B/B!= B < B 8 B1B B4 &T &3 *33$3 *35C3 3U @E   " 8< ! -6 mm`U U$U U U; ;&7; ; ,& NNEKK MM5;;> JJ MM MM JJ KK KK KK KK LL LL LL    E    >:,' ' '' 'T: :!0:9D: :2 L  L!0 L9G L  L aBaBH=N=&X1X1vzz";2==Q  O;#%5z#O;d  - - - 6:r6 ;HH -  r6 ;HH /-  r6;HH /- r6$;HH /- %r60;HH /- 1r6< ;HH *)   =r6L ;HH (0   Mr6X ;HH 66 8  Yr6z ;CC )+  {r6L ;HH 1  Mr6V ;HH 2  Wr6`;HH 1 ar6j;HH 2 kr6t;HH %8$  ur6B ;HH &%   Cr6N;HH %8  Or6Z ;HH &  [r6d;HH + er6p%;HH 88 ) qr6|%;HH 88 ) }r6H%;HH 8 )Ir6R%;HH 8 )Sr6^ ;HH '  _r6h ;HH (  ir6r;HH c  sr6B,;HH ? 0Cr6L,;HH ? 0Mr6X&;HH 9 *Yr6b ;HH *  cr6l);HH D -mr6v);HH D -wr6@);HH D -Ar6J);HH D -Kr6T(;HH C ,Ur6^$1;HH L 5$_r6h$1;HH L 5$ir6r$1;HH L 5$sr6|$1;HH L 5$}r6F';HH B +Gr6P(;HH C ,Qr6Z(;HH C ,[r62rj -#-">*Z  NNN ,XXv #;#;L5 +; T  k5c!1223 5l'/=0 1l^ < <@3Wgo.@3F   S"S"lm ~m M=s/ d: