>:j LSSKrSSKJr SSKJrJr SSKrSSKJr SSK J r SSK J r SSK Jr S\RS\ 4S jrS\RS \4S jr\R&"5SS\RS \ S \/S4S\\4Sjj5rS\RS \ S \/S44Sjrg)N)Callable)AnyOptional)Conv1D) LoraConfig) LoraModelmodelconfigc## UR5HNup#[R"X5(dM"[U[R [ 45(dMIX#4v MP g7f)z Iterate over LoRA-GA target name and modules of a model. A module is a target if its name is in `config.target_modules` and is `nn.Linear` or `Conv1D`. N) named_modulesr_check_target_module_exists isinstancennLinearr)r r namemodules T/home/mande/repo/quber/.venv/lib/python3.13/site-packages/peft/tuners/lora/loraga.pyget_target_modulesrsN ++-   0 0 > >:fWYW`W`bhViCjCj, .s/A&#A& A&returnc[US5(a URn[[UR 555R $)Nr)hasattrrnextiter parametersdevice)r s rget_model_devicer(s7uh  U%%'( ) 0 00 lora_config train_step cache_filecrURc [S5eURcV[R"U5n[R R US5nUc [S5e[U5Ul[X5H%upg[US5(dM[SUS35e Ub[RRU5(ab[RRU5S:a?[R"U[!U5S 9n[X5HupgXS 3UlM g[%XU5 Ubp0n[X5HupgUR"XS 3'M [R&"[RR)U5S S 9 [R*"X5 gg) a/ Build necessary LoRA-GA fields for a model by estimating gradients. For each linear layer, gradients will be estimated by running the provided train_step callback. These gradients are then attached to the modules and used during initialization. Args: model (`nn.Module`): Model to preprocess. lora_config (`LoraConfig`): Lora configuration of the model. `lora_config.lora_ga_config` should be set. train_step (`Callable[[], None]`): Callback to run gradient estimation. Typically you should run model forward and backward passes in this callback. The gradients will be accumulated across all calls within this callback. cache_file (`Optional[str]`): Optional path to cache file for saving/loading gradients. If provided and the file exists, gradients will be loaded from cache. Otherwise, gradients will be estimated and saved to this path. Upon completion, the following fields are set for each target module: _peft_loraga_grad (`torch.Tensor`): Accumulated gradient for the weight matrix. NzIf you want to use LoRA-GA, please initialize the LoraConfig with init_lora_weights='lora_ga' and lora_ga_config=LoraGAConfig(...). model_typez0Please specify `target_modules` in `peft_config` quant_statezDLoRA-GA does not support quantized models. Found quantized module: 'zB'. LoRA-GA requires full-precision gradients during preprocessing.r) map_locationz._peft_loraga_gradT)exist_ok)lora_ga_config ValueErrortarget_modulesrget_model_configtarget_module_mappinggetsetrrospathexistsgetsizetorchloadr_peft_loraga_gradestimate_gradientsmakedirsdirnamesave) r rrr model_configr(rrcaches rpreprocess_loragar:.s:!!) P  !!) 11%8 "88<<\,=WX  !OP P%(%8 "+5>  6= ) )VW[V\]RR ?"''.."<"<Q[A\_`A` :4DU4KL.uBLD',v5G-H'IF $C 5z:  !$&E 25 F 5;5M5M012!G KK 3d C JJu ) "rcURnUR5 [[X55nU(d [ S5eUH upVSUlM 0nUR 5HupXURXu'SUlM UHupVSURlM /n Sn UH'upVURU 5n U RU 5 M) [R"5 U"5 SSS5 U Hn U R5 M UR 5HupXXW;dM XuUlM UH^upVUR S:a6URRR5UR - UlSURlU?M` U(dUR#5 gg!,(df  N=f)z Estimate gradients for LoRA-GA initialization. This function enables gradient computation ONLY on target module weights and runs the train_step callback. This is more memory-efficient than enabling gradients globally. zNo supported layers found for LoRA-GA initialization. LoRA-GA only supports nn.Linear and Conv1D layers. Please ensure your model contains at least one of these layer types in target_modules.rFTc.U=RS- slg)N)_peft_loraga_grad_count)r grad_input grad_outputs r backward_hook)estimate_gradients..backward_hooks&&!+&rN)trainingtrainlistrr'r>named_parameters requires_gradweightregister_full_backward_hookappendr1 enable_gradremovegraddetachr3eval) r rr was_trainingtarget_module_listrroriginal_requires_gradparamhooksrAhooks rr4r4vs>>L KKM0DE  e  + )*&+  --/ ',':':$#0 + &* #+ E,+ 11-@ T+       --/   )"8">E 0 +   ) )A -'-}}'9'9'@'@'BVEcEc'cF $!   * +   )  s (F77 G)N)r-collections.abcrtypingrrr1torch.nnrtransformers.pytorch_utilsrpeft.tuners.lora.configrpeft.tuners.lora.modelrModulerstrrno_gradr:r4rrr`s$ $ -.,bii1BII1#1  !% D* 99D*D*T"D* D*D*NI 99IIT"Ir