qyif)SSKJr SSKJrJrJrJr SSKJr SSK J r \"5(aSSK r \(a SSK J r SS KJr \R "\5r"S S \5rg) ) TYPE_CHECKING)is_accelerate_availableis_torch_availableis_torch_xpu_availablelogging) HfQuantizer)get_module_from_nameN)PreTrainedModel)FineGrainedFP8Configc^\rSrSr%SrSrS\S'U4SjrSrSS S \ S \ 4S jr SS S \ S SS \ 4U4Sjjr SSjrSrSr\S \ 4Sj5r\S \ 4Sj5rSrSrSrSrU=r$)FineGrainedFP8HfQuantizerzz FP8 quantization implementation supporting both standard and MoE models. Supports both e4m3fn formats based on platform. Fr quantization_configc (>[TU]"U40UD6 g)N)super__init__)selfrkwargs __class__s n/home/mande/repo/quber/.venv/lib/python3.13/site-packages/transformers/quantizers/quantizer_finegrained_fp8.pyr"FineGrainedFP8HfQuantizer.__init__s ,77c[5(d [S5eURR(ag[R R 5(dR[5(dCUR(a'[RS5 SURlg[S5e[R R 5(ab[R R5nUupEUS:d US:Xa4US:a.[RSUSUS 35 SURlgURS 5nUc[RS 5 g[U[5(aTUR(d#[!U5S :aS UR#5;dSUR#5;a [%S5egg)NzMLoading an FP8 quantized model requires accelerate (`pip install accelerate`)zUsing FP8 quantized models requires a GPU or XPU, we will default to dequantizing the model to bf16 since no GPU or XPU is availableTzANo GPU or XPU found. A GPU or XPU is needed for FP8 quantization. ziFP8 quantized models is only supported on GPUs with compute capability >= 8.9 (e.g 4090/H100), actual = `.z`. We will default to dequantizing the model to bf16. Feel free to use a different quantization method like bitsandbytes or torchao device_mapzYou have loaded an FP8 model on CPU and have a CUDA or XPU device available, make sure to set your model on a GPU or XPU device in order to run your model. To remove this warning, pass device_map = 'cuda' or 'xpu'. r cpudiskzYou are attempting to load an FP8 model with a device_map that contains a cpu/disk device.This is not supported when the model is quantized on the fly. Please use a quantized checkpoint or remove the cpu/disk device from the device_map.)r ImportErrorr dequantizetorchcuda is_availabler pre_quantizedlogger warning_once RuntimeErrorget_device_capabilityget isinstancedictlenvalues ValueError)rargsrcompute_capabilitymajorminorrs rvalidate_environment.FineGrainedFP8HfQuantizer.validate_environments&((mn n  # # . . zz&&((1G1I1I!!##[7;((3"#fgg :: " " $ $!&!A!A!C -LE uzeai####('5'2Z[ 7;((3ZZ -     6   D ) )&& Oa'Z..00Z..00 k1 *rmodelr param_namereturnc SSKJnJn [X5upg[ XeU45(aUR (dUS:Xaggg)Nr) FP8Experts FP8LinearbiasFT)integrations.finegrained_fp8r<r=r r-r')rr8r9rr<r=module tensor_names rparam_needs_quantization2FineGrainedFP8HfQuantizer.param_needs_quantizationOs;H25E f*5 6 6!![F%:rparamz torch.TensorcR>URX5(ag[TU] XU5$)z4Return the element size (in bytes) for `param_name`.r )rBrparam_element_size)rr8r9rDrs rrF,FineGrainedFP8HfQuantizer.param_element_sizeZs)  ( ( ; ;w)%UCCrc SSKJn URXRRUR 5UlU"UURURUR S9ng)Nr)replace_with_fp8_linear)modules_to_not_convertrr')r?rIget_modules_to_not_convertrrJ_keep_in_fp32_modulesr')rr8rrIs r$_process_model_before_weight_loading>FineGrainedFP8HfQuantizer._process_model_before_weight_loadingas\ K&*&E&E ++BBED_D_' #( #'#>#> $ 8 8,,  rchSURR;aSSSSSSSSSSSSSSS.nX!lU$)NQwen3colwiserowwise)z layers.*.self_attn.q_proj.weightz*layers.*.self_attn.q_proj.weight_scale_invz layers.*.self_attn.k_proj.weightz*layers.*.self_attn.k_proj.weight_scale_invz layers.*.self_attn.v_proj.weightz*layers.*.self_attn.v_proj.weight_scale_invz layers.*.self_attn.o_proj.weightz*layers.*.self_attn.o_proj.weight_scale_invzlayers.*.mlp.gate_proj.weightz'layers.*.mlp.gate_proj.weight_scale_invzlayers.*.mlp.up_proj.weightz%layers.*.mlp.up_proj.weight_scale_invzlayers.*.mlp.down_proj.weightz'layers.*.mlp.down_proj.weight_scale_inv)r__name__base_model_tp_plan)rconfig text_plans rupdate_tp_plan(FineGrainedFP8HfQuantizer.update_tp_planssT f&&// /4=>G4=>G4=>G4=>G1:;D/89B1:;DI")2 % rcgNTrs ris_serializable)FineGrainedFP8HfQuantizer.is_serializablesrcg)NFr[r\s r is_trainable&FineGrainedFP8HfQuantizer.is_trainablesrcgrZr[r\s ris_compileable(FineGrainedFP8HfQuantizer.is_compileablesrcSSKJn U"U5$)Nr) Fp8Quantize)r?rf)rrfs rget_quantize_ops*FineGrainedFP8HfQuantizer.get_quantize_opss>4  rcSSKJn SSKJn UR(a-UR R (aU"/SQSU"U5/S9/$/$)Nr)WeightConverter Fp8Dequantize)zweight$weight_scale_invactivation_scaleweightsource_patternstarget_patterns operations)core_model_loadingrjr?rlr'rr#)rrjrls rget_weight_conversions0FineGrainedFP8HfQuantizer.get_weight_conversionssK8@   $":":"E"E $W$, -d 34  rc fUR(aURR(dXR5-$SSKJnJn SSKJn U"SSS9nU/[U5-n/nUGHn[Xr5(dURU5 M'URVs/sHoRS5(dMUPM n nU (aU Vs/sHoS-PM n nU Vs/sHoS [S5*S -PM n nURVs/sHoRS5(aMUPM n nX-U -n U"U5/[UR5-nU"U UR US 9nURU5 GM UR#UR55 U$s snfs snfs snfs snf) u]When loading with ``dequantize=True``, attach an :class:`Fp8Dequantize` op to every existing :class:`WeightConverter` so that per-block scales are folded into the weight *before* any later merge/concat ops collapse the per-expert structure. For each model-supplied converter that has a ``.weight`` source, we: 1. anchor the existing weight patterns with ``$`` so they don't accidentally also match the ``.weight_scale_inv`` keys (the regex is searched, so the unanchored prefix would match both, sending scales to the wrong bucket); 2. add anchored ``*.weight_scale_inv`` sources next to each weight pattern so the loader collects scale tensors alongside the weight tensors into the *same* converter bucket (both keys rewrite to the same target); 3. prepend a fresh :class:`Fp8Dequantize` op so dequant runs first, before any merge/concat collapses the per-expert structure. The generic ``weight$ + weight_scale_inv → weight`` converter from :meth:`get_weight_conversions` is still appended at the end as a fallback for plain ``nn.Linear`` weights with no model-specific converter. r)rjWeightRenamingrkz ^(.+)\.scale$z\1.weight_scale_inv)rqrrz.weight$Nz.weight_scale_inv$rp)r'rr#rurtrjrxr?rllistr-appendrqendswithr/rs_original_target_patternsextend)rweight_conversionsrjrxrl scale_renameupdatedconvpweight_sourcesanchored_weight scale_sourcesother new_sourcesnew_opss rupdate_weight_conversions3FineGrainedFP8HfQuantizer.update_weight_conversionss&""t'?'?'J'J%(C(C(EE EH@&6FXno *^d3E.FF&Dd44t$)-)=)=W)=AIAVa)=NW4B"CNqs7N"CVd eVdQR#4c)n_!58L!LVd e$($8$8V$8q 9@U$8V-=E (./$t2GG&$/$($B$B& NN4 %'( t2245X"C eVs$F9F F$ F) F.(F.)rJ)r8r )rS __module__ __qualname____firstlineno____doc__requires_calibration__annotations__rr6strboolrBfloatrFrMrWr]propertyr`rcrgrur__static_attributes__ __classcell__)rs@rrrs !//8/b .? S _c D(9DsDSaDfkD   $.d!  88rr)typingrutilsrrrrbaser quantizers_utilsr r$modeling_utilsr utils.quantization_configr get_loggerrSr(rr[rrrsI ``20@   H %P Pr