qyi.%SSKJr SSKJr SSKJrJr SSKJr SSK J r SSK J r \"5(aSS K r \(aSS KJr \R "\5r"S S \ 5rg ) ) annotations) TYPE_CHECKING)is_torch_availablelogging) SinqConfig) HfQuantizer)get_module_from_nameN)PreTrainedModelc^\rSrSr%SrSrS\S'S\S'SU4SjjrSS jr\ SS j5r S r SS jr SS jr SSjrSSjrSrSrSSSjjrSSjrSrU=r$)SinqHfQuantizer!a| HF v5 quantizer for SINQ. Modes: - method="sinq" (default): * weight-only SINQ * param-level ConversionOps (`SinqQuantize`) during load for pure language models (each Linear.weight is turned into a SINQLinear module) * module-level quantization after load for multimodal models - method="asinq": * A-SINQ (activation-aware) SINQ quantization Tbool requires_parameters_quantizationrquantization_configc D>[TU]"U40UD6 SUlSUlg)NF)super__init___normalized_device_str_do_param_level_sinq)selfrkwargs __class__s c/home/mande/repo/quber/.venv/lib/python3.13/site-packages/transformers/quantizers/quantizer_sinq.pyrSinqHfQuantizer.__init__2s& ,7726#*/!cgNTrs ris_serializableSinqHfQuantizer.is_serializable8srcgrr r!s r is_trainableSinqHfQuantizer.is_trainable;srcUca[RR5(a!S[RR50nOSS0n[R SUS35 U$)Ncpuz:The device_map was not initialized. Setting device_map to zJ. If you want to use the model for inference, please set device_map='auto')torchcuda is_availablecurrent_deviceloggerinfo)r device_maps rupdate_device_map!SinqHfQuantizer.update_device_map?se  zz&&(( %**";";"=>  %[ KK))3 5[[  rc8Uc[RnXlU$N)r*bfloat16dtype)rr6s r update_dtypeSinqHfQuantizer.update_dtypeLs =NNE  rcSSKJn U"5(d [S5e[RR 5(d[ RS5 URS5n[U[5(a@[UR55n[U5S:a[S[U5S35eUR R"S :XaUR$(d ['S 5egg) Nr)is_sinq_availablezMThe 'sinq' package is not installed. Please install it with: pip install sinqzNo CUDA device is available. Quantization and inference will run on the CPU. Please note that this will significantly slow down inference speed and increase quantization time.r0r zkSinqHfQuantizer: multi-GPU device_map detected, but SINQ currently supports only a single CUDA device. Got z. Please use device_map=None.asinqzYou are using `method='asinq'` in the quantization config. Right now the calibrated version of SINQ is not supported in Hugging Face, please refer and use the official SINQ repository `to quantize a model with this method. )utilsr: ImportErrorr*r+r,r.warningget isinstancedictsetvalueslen RuntimeErrorsortedrmethod pre_quantized ValueError)rargsrr:r0device_map_valuess rvalidate_environment$SinqHfQuantizer.validate_environmentRs- ""mn nzz&&(( NNB ZZ - j$ ' ' #J$5$5$7 8 $%)"##)*;#<"==Z\  # # * *g 5d>P>P: ?Q 5rc SSKJn URnU"[UR5UR b[UR 5OSSSSS[ UR5US9$)z9 Build the dict that SINQLinear expects as quant_config. r)sinq_base_quant_configNFr )nbits group_size quant_zero quant_scale view_as_floataxis tiling_moderG)sinq.sinqlinear_hfrOrGintrPrQstrrV)rcfgsinq_base_quant_config_fnrGs r_build_sinq_quant_dict&SinqHfQuantizer._build_sinq_quant_dictns[ [(cii..1nn.Hs3>>*dCOO,  rc SSKJn UR(agURRS:XagUR (dg[ X5upVUS:wag[XT5n[USS5nU=(a U(+n U $)a Called per-parameter to decide whether to run `SinqQuantize` on it. - If `self.pre_quantized`, we do *not* quantize again (handled by SinqDeserialize instead). - For method="asinq": return False (ASINQ is not supported in Hugging Face). - For method="sinq": True only for SINQLinear.weight not in modules_to_not_convert. Note: After _process_model_before_weight_loading(), the modules are already SINQLinear, not nn.Linear. We check for SINQLinear modules that are not yet quantized (ready=False). r) SINQLinearFr;weightreadyT) rWr_rHrrGrr r@getattr) rmodel param_namerr_module tensor_nameis_sinqis_readyresults rparam_needs_quantization(SinqHfQuantizer.param_needs_quantizations{ 2     # # * *g 5((25E ( "V067D1)\ rcSSKJn U"U5$)z Return the ConversionOps used for param-level quantization (Sinq). The actual SINQLinear construction is in integrations/sinq.py. r) SinqQuantize)integrations.sinqrm)rrms rget_quantize_ops SinqHfQuantizer.get_quantize_opss 5D!!rcfSSKJn UR(aSSKJn U"/SQS/U"U5/S9/$/$)z If `pre_quantized=True`, interpret a checkpoint produced by SINQLinear.state_dict: .W_q .bias .meta via a WeightConverter + SinqDeserialize so that we reconstruct a SINQLinear module instead of a plain nn.Linear. r)WeightConverter)SinqDeserialize)z.W_qz.metaz.biasz.weight)source_patternstarget_patterns operations)core_model_loadingrrrHrnrs)rrrrss rget_weight_conversions&SinqHfQuantizer.get_weight_conversionssE 9    ; % &/K / 56   rc SSKJn URXRR=(d /U5UlURR S:H=(a UR (+UlUR (aSOURUR5n[U[5(aJ[[UR55S5n[U[5(aSU3nO3[U5nO'[ R"R%5(aSOSnU"UURUUR&UUR S 9ng) z Called on meta-initialized model, before loading any weights. For SINQ, we replace nn.Linear modules with empty SINQLinear modules here. The actual quantization happens later in SinqQuantize.convert() when weights are loaded. r)replace_with_sinq_linearsinqNrzcuda:zcuda:0r))modules_to_not_convert quant_config compute_dtypedevicerH)rnr{get_modules_to_not_convertrr}rGrHrr\r@rAnextiterrCrXrYr*r+r,r6) rrcr0keep_in_fp32_modulesrr{sinq_quant_dict first_device device_strs r$_process_model_before_weight_loading4SinqHfQuantizer._process_model_before_weight_loadings A&*&E&E ,,CCIrL`' # %)$<$<$C$Cv$M$hVZVhVhRh!"&"4"4$$:U:UVZVnVn:o j$ ' 'Z%6%6%8 91=L,,,$\N3  . %*ZZ%<%<%>%>EJ( #'#>#>(**,,  rc  SSKJn U"5 U$)a9 Called after *all* weights have been loaded. For SINQ: 1. Move non-SINQLinear modules to GPU (embeddings, norms, lm_head, etc.) - SINQLinear modules already have GemLite buffers on GPU - We skip moving SINQLinear's W_q/meta to avoid memory duplication 2. Patch HF save/load methods for SINQ serialization r)patch_hf_pretrained_io) sinq.hf_ior)rrcrrs r#_process_model_after_weight_loading3SinqHfQuantizer._process_model_after_weight_loadings 6   r)rrr6r})rr)returnr)r6 torch.dtyperr)rNone)rZrrrA)rcr rdrYrrr4)rcr rzlist[str] | None)rcr )__name__ __module__ __qualname____firstlineno____doc__r__annotations__rr"propertyr%r1r7rLr\rjrorxrr__static_attributes__ __classcell__)rs@rrr!s .2$d1##0   8 $ D"B26 ) ) / ) Vrr) __future__rtypingrr<rrutils.quantization_configrbaser quantizers_utilsr r*modeling_utilsr get_loggerrr.rr rrrsK# /220   H %eker