Ei %SSKrSSKrSSKJr SSKJr SSKrSSKJs J s J r SSK J r SSK J s Jr SSKJr SSKJrJrJr SSKJrJr SSKJr SSKJr S S KJrJrJ r /r!\"\\#S 'S \RHS \%S\RHS\RH4Sjr&\RN\RP\RR\RT0r+"SS\ RX5r-"SS\ RX5r.S\-S\.S\RH4Sjr/"SS\"SSS/55r0\-Rc\Rd\RRS9r3\.Rc\Rh\RTS9r5\0"\3\5S9r6S\74Sjr8S\74Sjr9S\74S jr:S!\S\74S"jr;S#\ RXS\74S$jr\ RX4S\?\S-\.S-44S(jr@S)\S'\=\>\ RX4S\S-4S*jrAS!\S'\=\>\ RX4S\-S-4S+jrBS!\S'\=\>\ RX4S\RHS-4S,jrCS!\S'\=\>\ RX4SS4S-jrDS!\S'\=\>\ RX4S.\RHS/\RHS-SS4 S0jrES)\S&\S'\=\>\ RX4S.\RHS/\RHS-SS4 S1jrFS)\S'\=\>\ RX4SS4S2jrGS&\S!\S3\4S4jrHS&\S'\=\>\ RX4S\=\>\.44S5jrIS&\S'\=\>\ RX4S6\=\>\.4SS4S7jrJS&\4S8jrKS9\ RXS:\ RXS;\RHS\=\>\L44S<jrMS=\=\>\L4S>\%S\4S?jrNg)@N) namedtuple)Any)_get_observed_graph_module_attr) _with_args ObserverBasePerChannelMinMaxObserver) _parent_namecheck_min_max_valid) GraphModule)Node)get_new_attr_name_with_prefixmaybe_get_next_modulenode_arg_is_weightCUSTOM_MODULE_SUPP_LISTscaleaxisinputreturncjS/UR-nURU5X1'URU5$)zMReshapes the scale so that we can multiply it to the input by the given axis.r )ndimsizeview)rrr new_shapes _/home/mande/repo/quber/.venv/lib/python3.13/site-packages/torch/ao/quantization/fx/_equalize.py reshape_scalers1ejj Ijj&IO ::i  c^\rSrSrSr\R \RSSS4S U4SjjjrSr Sr Sr Sr \ "\5rS rU=r$) _InputEqualizationObserver,aObserver for tracking the running min/max values of input columns, and computing the quantization parameters for the overall min/max input values. Args: dtype: Quantized data type qscheme: Quantization scheme quant_min: Minimum quantization value. If unspecified, it will follow the 8-bit setup. quant_max: Maximum quantization value. If unspecified, it will follow the 8-bit setup. The running minimum/maximum :math:`x_\text{min/max}` are computed in the same way as :class:`~torch.ao.quantization.observer.PerChannelMinMaxObserver`, with the difference that the running min/max values are stored per column. This observer is intended to be used along with a WeightEqualizationObserver to calculate the equalization scale. Nc >[TU]5 U[R[R1;a [ S5eXlX l[Un[SUUUUUS9Ul [R"S5Ul /Ul g)Nz Input qscheme must be per-tensorr ch_axisdtypeqscheme quant_min quant_maxfactory_kwargs)super__init__torchper_tensor_affineper_tensor_symmetric TypeErrorr$r%(qsheme_mapping_per_tensor_to_per_channelr input_obstensorequalization_scaleequalization_shapeselfr$r%r&r'r(per_channel_qscheme __class__s rr*#_InputEqualizationObserver.__init__?s  522E4N4NO O>? ?  FwO1')  #(,,q/-/rcURS:dURS:a [S5eS/UR-UlURS5URS'UR U5$)N>InputEqualizationObserver only supports Linear and Conv layersr )r ValueErrorr3rr0)r5x_origs rforward"_InputEqualizationObserver.forward\sc ;;?fkkAoP  $%# "3%+[[^"~~f%%rcZURRURR4$N)r0min_valmax_valr5s rget_input_minmax+_InputEqualizationObserver.get_input_minmaxhs!&&(>(>??rcUR5S:XaU[R"S5:Xag[R"XR5Ulg)Nr )nelementr+r1reshaper3r2r5r2s rset_equalization_scale1_InputEqualizationObserver.set_equalization_scaleksC  & & (A -2D UV2W "'--  7 7# rcURR5S:Xa:UR[R"S5:Xa[R "SSS9 gUR 5up[URSU5n[R"[R"X55n[R"[R"X#55nXE4$)z!Returns the scaled min/max inputsr z}Must call calculate_equalization_scale before calling calculate_scaled_minmax. Will not scale the next quantization observer.r: stacklevelNNr) r2rIr+r1warningswarnrFrminmulmax)r5 min_inputs max_inputsequalization_scale_reshapedmin_input_scaledmax_input_scaleds rcalculate_scaled_minmax2_InputEqualizationObserver.calculate_scaled_minmaxts  # # , , .! 3''5<<?: MMC   $(#8#8#: &3  # #Q ' #!99UYYz%WX 99UYYz%WX11r)r$r2r3r0r%rN)__name__ __module__ __qualname____firstlineno____doc__r+quint8r,r*r?rFrLr\ classmethodr with_args__static_attributes__ __classcell__r7s@rrr,sX(ll'' 0 00: &@ 22J'Irrc^\rSrSrSr\R \RSSS4S U4SjjjrSr Sr Sr \ "\ 5rSrU=r$) _WeightEqualizationObserveraCObserver for tracking the running min/max values of weight columns and rows, and computing the quantization parameters for the weight rows. Args: dtype: Quantized data type qscheme: Quantization scheme quant_min: Minimum quantization value. If unspecified, it will follow the 8-bit setup. quant_max: Maximum quantization value. If unspecified, it will follow the 8-bit setup. This observer is made up of 1 PerChannelMinMaxObserver `weight_col_obs` used to record the running minimum and maximum of columns of incoming weight tensors. This observer is intended to be used along with an InputEqualizationObserver to calculate the equalization scale. The running minimum/maximum :math:`w_\text{min/max}` are computed in the same way as :class:`~torch.ao.quantization.observer.PerChannelMinMaxObserver`. Nc >[TU]5 XlX lSUlUnU[ R [ R1;a [Un[SUUUUUS9Ul [ R"S5Ul g)Nr r") r)r*r$r%r#r+r,r-r/rweight_col_obsr1r2r4s rr*$_WeightEqualizationObserver.__init__s|    % u..0J0JK K"J7"S 6')  #(,,q/rczURS:dURS:a [S5eURU5$)Nr:r;r<)rr=rn)r5w_origs rr?#_WeightEqualizationObserver.forwards: ;;?fkkAoP ""6**rcZURRURR4$rB)rnrCrDrEs rget_weight_col_minmax1_WeightEqualizationObserver.get_weight_col_minmaxs%##++T-@-@-H-HIIrcXlgrB)r2rKs rrL2_WeightEqualizationObserver.set_equalization_scales"4r)r#r$r2r%rnr^)r_r`rarbrcr+qint8r,r*r?rtrLrerrfrgrhris@rrkrksS,kk'' 2 228+J5J'Irrkr0 weight_obscUR5up#UR5upE[X#5(a[XE5(d+[R"SSS9 [ R "S5$URUR:wa)[SSURSURS3-5e[ R"XT- X2- - 5nSXfS :H'[ R"USSSS 9nU$) zCalculates the equalization scale and sets the equalization_scale value in the observers. Args: input_obs: Observer that tracks the ranges for the input columns weight_obs: Observer that tracks the ranges for the weight columns ztMust run observer before calling calculate_equalization_scale. Returning default equalization scale torch.tensor(1).r:rOr z6Input and Weight must have the same column dimension. zFound z and z shapes instead.g)nanposinfneginf) rFrtr rRrSr+r1shaper=sqrt nan_to_num)r0ryrWrX min_weights max_weightsr2s rcalculate_equalization_scalers )99;Z!+!A!A!C[ J33  9 9  F ||A;,,, Dz''(k.?.?-@@PQ R    "z'>?56S01))*[U[R5(d[U[R5(a [S5e[TU]XU5nU$)NzEqualizationQConfig received observer instance, please pass observer class instead. Use MyObserver.with_args(x=1) to override arguments to constructor if needed) isinstancennModuler=r)__new__)clsinput_activationweightr5r7s rrEqualizationQConfig.__new__sO & 2 2j6S6Sa wsf= r) r_r`rarbrc __slots__r+rIdentityrrgrhris@rrrs2$I&+hh&7&7@Q@Qrrrr)r$r%)rrc[U5[R[R[R[R 4;$)z/Checks if the fused node supports equalization.)typenni LinearReLU ConvReLU1d ConvReLU2d ConvReLU3dmodules r"fused_module_supports_equalizationr's4 <      rc[U5[R[R[R[R 4;$)z2Checks if the torch.nn node supports equalization.)rrLinearConv1dConv2dConv3drs rnn_module_supports_equalizationr1s*  F [ U[ UR5[5(dMkUs $ g)z8Gets the weight equalization observer node if it exists.rz inp_obs1 -> eq_obs1 -> linear1 -> out_obs1 -> eq_obs2 -> linear2 -> out_obs2 If the node being passed in is the linear1 node, then we want to return eq_obs2, the following equalization observer for linear2. However, if there are no connecting layers: x -> inp_obs1 -> eq_obs1 -> linear1 -> out_obs1 -> add Then we want to return None. In the case of an unfused linear-relu layer with a connecting linear layer: linear1 -> relu -> out_obs1 -> eq_obs2 -> linear2 -> out_obs2 Since it is unfused, we want to skip over the relu layer and return eq_obs2, the following equalization observer for linear2. z"Node does not support equalizationN)target_functional_typezPExpected the following equalization observer to be an _InputEqualizationObserver) rrrrReLUrrelurrrr)rrmaybe_relu_nodemaybe_obs_nodemaybe_eq_obs_node maybe_eq_obss rmaybe_get_next_input_eq_obsrs( &d 4 4ABB,D277CO/ !&&   " d\: "?\ J -!; 3012L l$> ? ? ^   rc[X5nU(aOURR5S:Xa%UR[R"S5:XagUR$g)aAIf the next next node is an InputEqualizationObserver then we want to return its equalization scale, else we return 1 This is used in the case where there are two connecting linear layers: linear1 -> LinearOutObs -> InputEqObs -> linear2 In this case, the node given is linear1 and we want to locate the InputEqObs. r N)rr2rIr+r1)rrnext_inp_eq_obss r!maybe_get_next_equalization_scalersP2$@O  . . 7 7 9Q >22ell1oE111 rcxU[UR5n[U[5(d [ S5eUR Sn[U[ 5(d [ S5eU[UR5n[U[5(dgUR5upVUcUcgXTl Xdl g)zScales the following input quantization observer's min/max values by updating the values with the scaled min/max values calculated by the input equalization observer zFExpected the module at node.target to be an _InputEqualizationObserverrz:Expected the input quantization observer node to be a NodeN) rrrrrrr rr\rCrD)rr input_eq_obsinput_quant_obs_nodeinput_quant_obsrZr[s rscale_input_observerrs 3t{{+,L l$> ? ? T   99Q< *D 1 1 H  c"6"="=>?O o| 4 4)5)M)M)O&$4$<..rr2next_equalization_scalecUcg[U[UR55(aU[UR5SnOU[UR5n[U5(d[ U5(d [ S5eUR n[U[R5(d [ S5e[USU5n[R"U[R"U55nUc[R"U5Ulg[USU5n[R"Xx5n[R"U5UlURn U cg[U [R5(d [ S5e[USU 5n[R"X5n [R"U 5Ulg)aScale the weights for input-weight equalization by multiplying the weight by 1/equalization_scale and next_equalization_scale Args: node: Current node whose weights we want to scale equalization_scale: Current node's calculated equalization scale next_equalization_scale: Next node's calculated equalization scale if the following node needs to be equalized, 1 otherwise Nrz@Expected operation module to support equalization (nn or custom)z.Expected op_module.weight to be a torch.Tensorr z,Expected op_module.bias to be a torch.Tensor)rrrrrrrrr+TensorrrU reciprocalr Parameterbias) rrr2r op_modulerrY scaled_weight next_equalization_scale_reshapedr scaled_biass rscale_weight_noder s|!)'#dkk2B*CDDC ,-a0 C ,- ' 22 .y 9 9 N    F fell + +MNN#00BAv"NIIfe&6&67R&STM&<< 6 (55LaQW'X$IImNM||M2I >>D | dELL ) )KLL(55LaQU'V$))DCK\\+.INrcUcg[X5nUcgURSnUcg[U[5(a+[U[ UR 5[ 5(d [S5eURSnUcg[U[5(aURS:Xd [S5e[UR 5up[X(U 5n [USU 5n [R"U [R"U 55n Uc[X(X5 g[USU 5n [R"X5n [X(X5 [R "UR#[ UR 55U 5(d [S5eSnURH@n[U[5(dMURS:XdM,SUR$;dM>Un O Ucg[UR 5unn[UUU5n[USU5n [R"UU 5n[UUUU5 g) z-Scales the weight value for functional layersNrzKExpected weight_quant_obs_node to be a Node whose module is an ObserverBaseget_attrz,Expected weight node to be a 'get_attr' Noder z8Model buffer for weight does not match the scaled weightr)rrrr rrrrrr getattrrr+rUrsetattrallclose get_bufferr)rrrr2rweight_eq_obs_nodeweight_quant_obs_noderweight_parent_name weight_namerrYrr bias_noderbias_parent_name bias_namerrs rscale_weight_functionalrNs.!6gG!/33A6$($// ws#8#?#?@A< P P Y  (,,Q/K {D ) )knn .JKLL&2;3E3E&F# W0+ >F #00BAv"NIIfe&6&67R&STM&+[H(5M($IImNM G 'D >>%**3{/A/A+BC] S SWXXI  dD ! !dgg&;$))@SI   ".y/?/?"@i 7+,i 8D(55LaQU'V$))D"BCK G$ %y+>rcD[X5nUcgURSnUcg[U[5(d [ S5eU[ UR 5n[U[ UR 5[5(d [ S5eUR5 g)zlGiven the operation node, we want find the corresponding quantization observer and reset its min/max values Nrz+Expected weight_quant_obs_node to be a NodezBExpected the module at weight_quant_obs_node to be an ObserverBase) rrrr rrrrreset_min_max_vals)rrrrweight_quant_obss rclear_weight_quant_obs_noders6gG!.33A6$ +T 2 2JKKs#8#?#?@A gc"7">">?@, O O P  '')r prev_nodec[URR55nUHnURX5 M URR U5 g)zaRemoves the given node from the model by replacing all of its users with the given previous node N)listrkeysreplace_input_withgraph erase_node)rrr orig_users user_nodes r remove_nodersE djjoo'(J $$T5  KK4 rc0nURRGH_nURS:XdM[XR[ 5(dM9XRn[U[ 5(d [ S5e[X0U5upVUbUcM}URS:Xa[U[UR55(aIU[UR5Sn[U5(d [ S5eU"UR5 O(U"U[UR5R5 [XF5nURU5 URU5 XbUR'GMb U$)agUpdate all of the observer's equalization scale. For each InputEqualizationObserver, we will find the location of the next WeightEqualizationObserver, create it, and calculate the equalization scale based on the two observers. We will then return a dictionary mapping operation node names to the corresponding WeightEqualizationObservers for that operation. rzBExpected module at node.target to be an _InputEqualizationObserverrz-Expected fused module to support equalization)rnodesrrrrrrrrrrrrLr) rrweight_eq_obs_dictrrrrrr2s rupdate_obs_for_equalizationrsY !! 77m # KK "<) ) #;;/Ll,FGG$X&C4PW%X "G-"7zz]*6gc'..>Q6RSS$S%89!"E"EF">"   / /0B C  0 01C D/ InpQuantObs -> InpEqObs -> linear -> OutQuantObs After this function: scaled weight values | equalization scale WeightQuantObs | | x -> mul -> InpQuantObs (scaled min/max) -> linear -> OutQuantObs After convert: equalization scale scaled weight values | | x -> mul -> quantize_per_tensor -> quantized::linear Note that although the equalization observer appeared after the quantization observer after prepare_fx, the mul node appears before the quantization node after convert_fx. This is because placing the equalization observer after the quantization observer in prepare_fx would allow us to keep the invariant that the graph before the current node inserts its observers is not modified. Having the equalization observer before the quantization observer would also cause some inconsistences between the ordering of the quantization and equalization observers. For example, a single linear layer would look like: x -> InpEqObs1 -> InpQuantObs1 -> linear1 -> OutQuantObs1 But between two connected linear layers, it would look like: linear1 -> OutQuantObs1 -> InpEqObs2 -> linear2 -> OutQuantObs2 rrr_equalization_scalerNrrr z=Expected operation node to be 'call_module' or 'call_functionzInstead got node z as 'z'.) rrrrrrrrrrrinserting_beforerrr2 create_nodeinserting_afterr+rUrrrkrrIr1rrrrrrr=)rrrrinp_quant_obs_noderget_new_eq_scale_namer eq_scale_nodeinputsmul_noderr2maybe_next_equalization_scalers rconvert_eq_obsrsf !! 77m # KK "<) ) "&1 *//2I+9>>Y^^+E);< ! /--.@A(ENN%::)%-W5W[[%9%L%LM % 7 7 D I B,,];#3 ;;22?EIIvV<  1 1)X F %7 8  # #DII . :.22499=Mm-HII$_"/!A!A #++-2&%,,q/9%)",M- ) ww-'!&1 O+'&1 &B$%P"%-!C 2 9 9:;=X)c ,D:/33A6 EyA S)$))E$''"EFK":BA<;sAL/L( L% ( L7 c[URSS95n[X5n[XU5 [ XR 5$)zbReference function which applies changes needed for equalization, but does not quantize the nodes F)remove_duplicate)dict named_modulesrrr r)rrrs r_convert_equalization_refrsC5&&&>?G5UD5#56 ukk **rmodel_amodel_bxc SSKJs Js Jn SSKJn U"5nUSR [R5 URSUSUURUS9upgU"U5 U"U5 URXgURS5nURUSS[RRRRRS5 0n UH*n XS SSS n XS SSSSn XU 'M, U $) aRuns the Numeric Suite on model_a and model_b and returns a dictionary containing the SQNR between layers in model_a and model_b. Note: In order to support equalized models, this function has a hacky fix in which we do not match any torch.mul operators. This is because equalized models contain extra mul operators to scale the input by the equalization scale, but this edge case has not been resolved yet within the numeric suite code. Args: model_a: A float model model_b: A quantized model x: Inputs to use during calibration rN)get_unmatchable_types_mapfuns_unmatchablefp32int8)unmatchable_types_mapsqnr node_outputfqn)torch.ao.ns._numeric_suite_fxaons_numeric_suite_fxtorch.ao.ns.fx.mappingsraddr+rU add_loggers OutputLoggerextract_logger_info%extend_logger_results_with_comparisonfxutils compute_sqnr) rrrr$rr model_a_ns model_b_nsactivation_comparison_dictlayer_sqnr_dictkeylayerrs rget_layer_sqnr_dictr5s /.A57,-11%))<^^ 3 ,JqMqM!#!7!7",,"  )) O)*/ >vFqI%P).}=fEaHPQRS!%* rr2num_layers_to_equalizec[UR5[R"S5S9nUSUnUVs/sH oDS[4PM nnSU0nU$s snf)aGiven the layer to SQNR dictionary, find the layers with the highest quantization errors, and return an equalization_qconfig_dict specifying to only equalize those top layers. Args: layer_sqnr_dict: Dictionary mapping layer names to SQNR values (found when comparing an equalized model against a float model) num_layers_to_equalize: Number of layers with the highest quantization errors to equalize r )r3Nr module_name)sorteditemsoperator itemgetterdefault_equalization_qconfig)r2r6layer_sqnr_sortedlayers_to_equalizeitemmodule_to_qconfig_listequalization_qconfig_dicts rget_equalization_qconfig_dictrCsv 446HrYs" ##Q J &(c'!!S!!%,,! U55  ; ;,( a(a(H?("))?(D%)%7R% \\%P$'98&DED9BB ,, : :C ;DD ++u:: E  309U  $HtH 343 T t ryyT88$/8:>sBII~:N8 4$;3d: :;8v  bii0 D[*3 3c299n-3$&3l c299n- \\D,/t/d3 >.B/t/8>/ >/ #ryy. !>/ >/#\\D0 >/  >/BQ? Q? Q?#ryy. !Q? Q? #\\D0 Q?  Q?h**S"))^8L*QU*, !{ !$ !4 !0 0!%c299n!50 #* *+0f[ [ #ryy. ![S"==>[ [| +[ +&4 YY4!#4/4||4 #u*4n%#u*%%?B%%r