"Ei`o&SSKJrJr SSKrSSKJr SSKJrJrJrJ r J r J r J r J r SS/r"SS\ 5rS S \ S \3-S -\lS \\S\\S\\S-S\\S-S\\S-S\\S\S-S\S-S\S\\-S\S\S\S-S\S\S\SS4"SjrS\ S\\\R.S-\R0S-\4\\\S-44S jrS \\S\\S\\S-S\\S-S\\S-S\\S\S-S\S-S\S\\-S\S\S\S-S\S\S\SS4"S!jr\"\S"9S%S \\S\\S\\S-S\\S-S\\S-S\\S#\S-S\S-S\S-S\S\S\\-S\S\S\S\S\SS4$S$jj5rg)&)cast TYPE_CHECKINGN)Tensor)_disable_dynamo_if_unsupported_get_scalar_dtype _maximize_doc _params_doc _to_scalar OptimizerParamsTTensorListList Adafactor adafactorc^\rSrSrSSSS.S\S\\-S\S\\S-\4S \S \S \S-S \S S4U4Sjjjjr U4Sjr S \4Sjr \ R"5SSj5rSrU=r$)rNF)foreachmaximizeparamslr beta2_decayepsd weight_decayrrreturnc>[U[5(aUR5S:wa [S5eSU::d[SU35eSU:d[SU35eUSbSUS::d[SUS35eSUS::d[SUS35eS U::d[S U35eSU::d[S U35eUUUUUUUS .n [T U]X5 g) NrzTensor lr must be 1-elementz%Learning rate should be >= 0 but is: z#beta2_decay should be <= 0 but is: rz epsilon1 should be >= 0 but is: z epsilon2 should be >= 0 but is: ?z,Clipping threshold d should be >= 1 but is: z$weight_decay should be >= 0 but is: )rrrrrrr) isinstancernumel ValueErrorsuper__init__) selfrrrrrrrrdefaults __class__s S/home/mande/repo/quber/.venv/lib/python3.13/site-packages/torch/optim/_adafactor.pyr#Adafactor.__init__s  b& ! !bhhjAo:; ;byDRDIJ Jk!B;-PQ Q q6 cSVm?AxHI Ic!f}?AxHI IaxKA3OP Pl"CL>RS S&(   *c|>[TU]U5 URHnURSS5 USH~nURR U/5n[ U5S:wdM0[R"US5(aMP[US5n[R"U[5S9US'M M g)Nrrrstepdtype) r" __setstate__ param_groups setdefaultstategetlentorch is_tensorfloattensorr)r$r1grouppp_statestep_valr&s r'r.Adafactor.__setstate__=s U#&&E   Y -8_**..B/w<1$U__WV_-M-M$WV_5H&+ll8CTCV&WGFO %'r)c>USGHnURcM[R"U5(a [S5eURR(a [S5eUR U5 UR UR5 UR Un [U 5S:Xa[R"S[5S9U S'URR5S:a[URR5n SU S 'URRU 5U S '[URR5n SU S 'URRU 5U S 'O0[R"UR[RS 9U S'UR U R!S S55 UR U R!S S55 UR U R!SS55 UR U S5 GM g)Nrz-Adafactor does not support complex parametersz+Adafactor does not support sparse gradientsrrr,r+rrow_varcol_var) memory_formatvarianceF)gradr4 is_complex RuntimeError is_sparseappendr1r3r7rdimlistshape new_zeros zeros_likepreserve_formatr2) r$r8params_with_gradgradsrow_varscol_vars variances state_stepsr9r1 row_shape col_shapes r' _init_groupAdafactor._init_groupGsxAvv~"""#RSSvv"#PQQ  # #A & LL JJqME5zQ!& S8I8K Lf 66::2i(z*"4t<!$ T:'# ''N U%$s B;; C )g{Gz?g)NgMbP?rrN)__name__ __module__ __qualname____firstlineno__r r6rtupleboolr#r.rWr4no_gradr+__static_attributes__ __classcell__)r&s@r'rrs"!*6!#+ $#+#+ FN#+ #+ 54<& ' #+  #+#+#+#+ #+#+JX0 0d ]]_55r)aImplements Adafactor algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{(lr)}, \: \tau \text{(}\beta_2\text{ decay)}, \: \theta_0 \text{(params)}, \: f(\theta) \text{(objective)}, \\ &\hspace{15mm} \: \epsilon_1, \epsilon_2 \text{ (epsilons)}, \: d \text{(clipping threshold)}, \\ &\hspace{15mm} \: \lambda \text{(weight decay)}, \: \textit{maximize} \\ &\textbf{initialize} : \: R_0 \leftarrow 0 \text{ (second moment row factor)}, \\ &\hspace{23mm} \: C_0 \leftarrow 0 \text{ (second moment col factor)}, \\ &\hspace{23mm} \: \widehat{V}_0 \leftarrow 0 \text{ (second moment for vectors)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}\textbf{if} \: \textit{maximize}: \\ &\hspace{10mm}G_t \leftarrow -\nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}G_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\widehat{\beta}_{2_t} \leftarrow 1 - t^{\tau} \\ &\hspace{5mm}\rho_t \leftarrow min(lr, \frac{1}{\sqrt{t}}) \\ &\hspace{5mm}\alpha_t \leftarrow max(\epsilon_2, \text{RMS}(\theta_{t-1}))\rho_t \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \gamma \lambda \theta_{t-1} \\ &\hspace{5mm}\textbf{if} \: \text{dim}(G_t) > 1: \\ &\hspace{10mm}R_t \leftarrow \widehat{\beta}_{2_t}R_{t-1}+ (1-\widehat{\beta}_{2_t})(G_t \odot G_t) \cdot 1_m \\ &\hspace{10mm}C_t \leftarrow \widehat{\beta}_{2_t}C_{t-1}+ (1-\widehat{\beta}_{2_t}) 1^\top_n \cdot (G_t \odot G_t) \\ &\hspace{10mm}\widehat{V}_t \leftarrow \frac{R_t \cdot C_t}{max(1^\top_n \cdot R_t, \epsilon_1)} \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}\widehat{V}_t \leftarrow \widehat{\beta}_{2_t}\widehat{V}_{t-1}+ (1-\widehat{\beta}_{2_t}) \cdot (G_t \odot G_t) \\ &\hspace{5mm}U_t \leftarrow \frac{G_t}{max(\sqrt{\widehat{V}_t}, \epsilon_1)} \\ &\hspace{5mm}\widehat{U}_t \leftarrow \frac{U_t}{max(1, \frac{\text{RMS}(U_t)}{d})} \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \alpha_t \widehat{U}_t \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adafactor: Adaptive Learning Rates with Sublinear Memory Cost`_. z Args: a lr (float, Tensor, optional): unlike other optimizers, Adafactor does not require a learning rate, and Noam Shazeer and Mitchell Stern do not use lr at all. Deviating from the paper, this implementation uses lr for applying weight decay and as the maximum value for relative step size rho_t. Note that in the paper, a constant of 0.01 is used as the maximum value for relative step size, and so we set 0.01 as the default value. (default: 1e-2) beta2_decay (float, optional): the decay rate of beta2. beta2 standardly refers to the coefficient used for computing the running average of the gradient squared. (default: -0.8) eps (Tuple[float, float], optional): epsilon1 is the term added to the denominator of the update calculation to improve numerical stability. This use of epsilon1 deviates from the algorithm written in the paper! See note below for more details. epsilon2 is the term used to avoid having too small a weight update when applying parameter scaling. (default: (None, 1e-3)) d (float, optional): the clipping threshold, used to avoid larger-than-desired updates. weight_decay (float, optional): weight decay coefficient (default: 1e-2) foreach (bool, optional): whether foreach implementation of optimizer is used. Note that the foreach implementation uses ~ sizeof(params) more peak memory than the for-loop version due to the intermediates being a tensorlist vs just one tensor. As Adafactor is commonly used when memory is prohibitive, Adafactor will default to the slower single tensor for-loop implementation unless this flag is explicitly True. This behavior is contrary to other optimizers, which will attempt defaulting to foreach on CUDA for faster runtime. (default: None) a4 .. Note:: The implementation of Adafactor subtly differs from Noam Shazeer and Mitchell Stern and implementations in some other frameworks with its use of learning rate and :math:`\epsilon_1`. Regarding the learning rate hyperparameter: Noam Shazeer and Mitchell Stern do not use lr at all, as the stated algorithm uses :math:`\rho_t` and update clipping to affect the step size. This implementation allows `lr` to influence the maximum value for :math:`\rho_t`: .. math:: \begin{aligned} &\hspace{5mm}\rho_t \leftarrow min(lr, \frac{1}{\sqrt{t}}) \end{aligned} This differs from Noam Shazeer and Mitchell Stern, who use a constant of 0.01 as the maximum value of :math:`\rho_t` .. math:: \begin{aligned} &\hspace{5mm}\rho_t \leftarrow min(0.01, \frac{1}{\sqrt{t}}) \end{aligned} Noam Shazeer and Mitchell Stern do not enforce an opinion on how weight decay should be computed, and so we use the learning rate as a coefficient for decoupled weight decay, similar to what is suggested in `Decoupled Weight Decay Regularization`_. Regarding the use of :math:`\epsilon_1`: The implementation attempts to replicate the presumed intention of Noam Shazeer and Mitchell Stern to use :math:`\epsilon_1` as a stabilizing term when the squared gradient becomes small. This stabilization can be written as .. math:: \begin{aligned} &\hspace{5mm}R_t \leftarrow \widehat{\beta}_{2_t}R_{t-1}+ (1-\widehat{\beta}_{2_t})(G_t \odot G_t + 1_n \cdot 1^\top_m) \cdot 1_m \\ &\hspace{5mm}C_t \leftarrow \widehat{\beta}_{2_t}C_{t-1}+ (1-\widehat{\beta}_{2_t}) 1^\top_n \cdot (G_t \odot G_t + 1_n \cdot 1^\top_m) \\ &\hspace{5mm}\widehat{V}_t \leftarrow \frac{R_t \cdot C_t}{max(1^\top_n \cdot R_t, \epsilon_1)} \\ &\hspace{5mm}U_t \leftarrow \frac{G_t}{max(\sqrt{\widehat{V}_t}, \epsilon_1)} \\ \end{aligned} where the row and column factors of gradient squared :math:`R_t` and :math:`C_t` are left alone, and we apply :math:`\epsilon_1` at the final calculation of the variance estimate :math:`\widehat{V}_t` and for the update :math:`U_t`. This is in contrast to Noam Shazeer and Mitchell Stern and other frameworks which apply :math:`\epsilon_1` to both row and column factors of the squared gradient, but not in the calculations after: .. math:: \begin{aligned} &\hspace{5mm}R_t \leftarrow \widehat{\beta}_{2_t}R_{t-1}+ (1-\widehat{\beta}_{2_t})(G_t \odot G_t + \epsilon_1 1_n \cdot 1^\top_m) \cdot 1_m \\ &\hspace{5mm}C_t \leftarrow \widehat{\beta}_{2_t}C_{t-1}+ (1-\widehat{\beta}_{2_t}) 1^\top_n \cdot (G_t \odot G_t + \epsilon_1 1_n \cdot 1^\top_m) \\ &\hspace{5mm}\widehat{V}_t \leftarrow \frac{R_t \cdot C_t}{1^\top_n \cdot R_t} \\ &\hspace{5mm}U_t \leftarrow \frac{G_t}{\sqrt{\widehat{V}_t}} \\ \end{aligned} You may note that Noam Shazeer and Mitchell Stern describe using the sum of squared gradients, while this implementation uses the mean instead. This choice is mathematically equivalent and allows for greater numerical stability for large sums. .. _Adafactor\: Adaptive Learning Rates with Sublinear Memory Cost: https://arxiv.org/pdf/1804.04235 .. _Decoupled Weight Decay Regularization: https://arxiv.org/abs/1711.05101 rrPrQrRrSrTrZr[rrrrr\r]rr^rc0UcUb [S5e[RR5(a-[ U [ 5(d[S[ U 535eO [U 5n [U5GHunnU(dUUOUU*nUUnUUnUUnUUnU c*[R"UR5Rn US- nUR5nUU -n[U SUS-- 5n[U URS5R5UR!5S-- 5U-nU S:waUR#SX-- 5 UR%5S:aUbUc [S5e[R"USS S 9R'5R)UR+S55nUR-UU5 [R"US S S 9R'5R)UR+S 55nUR-UU5 UU-nUR)UR/S S S 9R1U S 95 O5Uc [S 5eUU-nUR-UU5 UR35nUR1X-S 9R55nUR#U5 [SURS5R5UR!5S-U-- 5n UR7UU*U - S9 GM g)N5Grad scaling should occur outside of optimizer.step()z#Expected lr to be a float, but got r?rCrow_var and col_var should be defined when grad is multidimensionalr>TrIkeepdimr@)min0variance should be defined when grad is a vectorralpha)AssertionErrorr4jit is_scriptingrr6typer enumeratefinfor-ritemrwmaxnormr mul_rIsquare_div_sizelerp_meanclamp_clonersqrt_add_)!rrPrQrRrSrTrZr[rrrrr\r]rr^iparamrDstep_tr?rArC step_floatone_minus_beta2_trho_trzrow_meancol_mean var_estimate grad_squaredupdatedenoms! r'_single_tensor_adafactorrJs.!6TUU yy"e$$ #FtBxj!QR R%^f%5'uQxeAhYQ1+1+Q< <;;u{{+//D ! [[] & 3BZ_-.D%**Q-,,.%++-32FGH5P 1  JJq2,, - 88:>'/$Y  4R6>>@EEdiiPRmT  MM($5 6 4R6>>@EEdiiPRmT  MM($5 6"W,L   gllr4l@GGDGQ R$%WXX$;L NN<): ;#>>+L$$$5<<> DCQ,,.6<<>S3HA2MNO 6%% 0e&r) tensorlistscL[R"U5n0nUR5Huup4upX4S4nX4S4n[US5HupU c [ S5eU R 5S:aRXb;aUVs/sHn/PM snX&'[ [U55Hn X&U RX U5 M MyXr;aUVs/sHn/PM snX''[ [U55Hn X'U RX U5 M M M U$s snfs snf)zGroups tensors by device, dtype, AND multidimensionality -- whether the tensor has multiple dims or just one dim (is a vector). This allows the foreach impl of Adafactor to assume that every group of params will either be factored or not.TFrzgrad should not be None) r "_group_tensors_by_device_and_dtypeitemsrr{rIranger3rH) rgrouped_tensorsultra_grouped_tensorsdevicer-_ matrix_key vector_keyjr7rs r'._group_tensors_by_device_dtype_and_is_multidimrs< BB;OO .=-B-B-D))+T* U+ #;q>2IA~$%>??zz|a:EP8Q[[8Q)5s;/0A)5a8?? q@QR1:EP8Q[[8Q)5s;/0A)5a8?? q@QR13 .E$ ! 9R 9Rs 9 D D!c  [U5S:XagUcUb [S5e[U 5n [XX#XE/5nUR 5GH_uunnnunnnnnn[ [ [U5n[ [ [U5n[ [ [U5nU c.Uc [S5e[R"U5Rn [(aUSceU(a[R"U5n[RR5(d>USR(a*[R "U[R""SSS9SS9 O[R "US5 /n/n/nUHyn UR%U R'5U -5 UR%SU R'5U -- 5 UR%[)U SU R'5S -- 55 M{ [+UUS S 9V!V"s/sHFun!n"[-U U!R/S 5R'5U!R15S -- 5U"-PMH n#n!n"U S:wa[R2"USX-- 5 U(Ga[ [ [U5n$[ [ [U5n%U$SbU%Sc [S 5eUV&s/sHn&[R."U&SS S9PM n'n&[R2"U'U'5 [R4"U'UV&s/sHn&U&R7S5PM sn&5 [R8"U$U'U5 A'UV&s/sHn&[R."U&SS S9PM n(n&[R2"U(U(5 [R4"U(UV&s/sHn&U&R7S5PM sn&5 [R8"U%U(U5 A([+U$U%S S 9V)V*s/sH un)n*U)U*-PM n+n)n*U$V)s/sHn)U)R;SS S9PM n,n)[R<"U,U 5 [R4"U+U,5 A,Ox[ [ [U5n-U-Sc [S5e[R>"UU5n.[R8"U-U.U5 A.U-V/s/sHn/U/RA5PM n+n/[R<"U+X-5 [RB"U+5 [R2"U+U5 U+n0[+U#U0S S 9V1V2s/sHJun1n2U1*[-SU2R/S 5R'5U2R15S -U-- 5- PML n#n1n2[R2"U0U#5 [R "UU05 GMb gs sn"n!fs sn&fs sn&fs sn&fs sn&fs sn*n)fs sn)fs sn/fs sn2n1f)Nrrqz2dtype is needed to compute eps1 when eps1 is unsetrcpu)rryrrrT)strictrsrtr>rur@rx)"r3r{r rrrrJrr4rrr _foreach_negcompiler is_compilingis_cpu _foreach_add_r7rHrrwziprrr _foreach_mul_ _foreach_div_r_foreach_lerp_r_foreach_clamp_min_ _foreach_mulr_foreach_rsqrt_)3rrPrQrRrSrTrZr[rrrrr\r]rr^rrr- is_multidimdevice_params_ device_grads_device_row_vars_device_col_vars_device_variances_device_state_steps_ device_params device_gradsdevice_state_stepsone_minus_beta2_tsbeta2_tsrho_tssr9ralphasdevice_row_varsdevice_col_varsrD row_means col_meansr?rA var_estimates row_var_meansdevice_variances grads_squaredvupdatesars3 r'_multi_tensor_adafactorrs~. 6{a!6TUU BBD ICO     E;      T&\>: DL-8 !$v,0CD <}$H;;u%))D =%a(4 44  --l;L ~~**,,1CA1F1M1M   "ELLU$C3     2C 8#A  % %affh+&= > OOAK 77 8 MM#b!qvvx3"78 9$M6$? ?1 affQinn&!'')s*:;  > 6B#c6;;q>..0V\\^s5Ja4OPQ R>   GV, M73S !H ",T ,T  B s8A V0 V6V;  W,W 5W W6W(AW)single_tensor_fnrc [RR5(d"[SU55(d [ S5eU(a[ nO[ nU"UUUUUUU U U U UUUUUU S9 g)zpFunctional API that performs Adafactor algorithm computation. See :class:`~torch.optim.Adafactor` for details. c3V# UHn[U[R5v M! g7frf)rr4r).0ts r' adafactor..xs!5-8 1ell##[s')z?`state_steps` argument must contain a list of singleton tensors) rrrrr\r]rrZr[r^N)r4rrallrFrr)rrPrQrRrSrTrrZr[r^rrrrr\r]rfuncs r'rr]s6 >> & & ( (5-8522 M  &'  !  !r))NNNF)typingrrr4r optimizerrrr r r r r r__all__r__doc__rJr6rlrdictrkrr-rrrrer)r'rs'      $X Xx.^   2 7_JVHWS pV1 LV1 <V16D=!V16D=!V1FTM"V1fV1 V1}V1 V1 V1 !V1"#V1$ $,%V1& 'V1()V1*+V1, -V1r ! !  %,, u{{T14 78ftm  !FT4 LT4 <T46D=!T46D=!T4FTM"T4fT4 T4}T4 T4 T4 !T4"#T4$ $,%T4& 'T4()T4*+T4, -T4n 1IJ $#7 L7 <76D=!76D=! 7 FTM" 7 f 7D[7 7}77 7 7 !7"#7$ %7& '7()7* +7K7r)