!EiafSrSSKrSSKrSSKJr SSKJrJr SSKJ r SSK r SSK J r /SQr \"S5r \ "S 5r\S r\S rS2S \ S \S\S\ R$S-S\ 4 SjjrS2S \ S\S\S\ R$S-S\ 4 SjjrS2S \ S\S\S \S\S\ R$S-S\ 4SjjrS \ S\S\ 4SjrS \ S\ 4SjrS2S\S\\-S-S\4SjjrS3S \ S \S\S\ R$S-S\ 4 SjjrS3S \ S\S\S\ R$S-S\ 4 SjjrS4S \ S\S\S \S\S\ R$S-S\ 4SjjrS \ S\S\ 4SjrS \ S\ 4S jrS \ S\ 4S!jrS \ S\ 4S"jr S5S \ S#\S\ 4S$jjr!S \ S\"\\44S%jr#S6S \ S&\S\ R$S-S\ 4S'jjr$S6S \ S&\S\ R$S-S\ 4S(jjr%S \ S)\S\4S*jr&S7S \ S \S)\S\S\ R$S-S\ 4 S+jjr'S7S \ S \S)\S\S\ R$S-S\ 4 S,jjr(S8S \ S&\S\ R$S-S\ 4S-jjr)S9S \ S.\S\S\ R$S-S\ 4 S/jjr*S0\\\ 4S\\\ 44S1jr+\+"\5r,\+"\5r-\+"\5r.\+"\ 5r/\+"\!5r0\+"\$5r1\+"\%5r2\+"\'5r3\+"\(5r4\+"\)5r5\+"\*5r6g):zHThis file contains utilities for initializing neural network parameters.N)Callable)LiteralTypeVar) ParamSpec)Tensor)calculate_gainuniform_normal_ trunc_normal_ constant_ones_zeros_eye_dirac_xavier_uniform_xavier_normal_kaiming_uniform_kaiming_normal_ orthogonal_sparse_uniformnormalconstanteyediracxavier_uniform xavier_normalkaiming_uniformkaiming_normal orthogonalsparse_R_P) linearconv1dconv2dconv3dconv_transpose1dconv_transpose2dconv_transpose3dsigmoidtanhrelu leaky_reluselu)fan_infan_outtensorab generatorreturnc[R"5 URXUS9sSSS5 $!,(df  g=fNr5)torchno_gradr r2r3r4r5s J/home/mande/repo/quber/.venv/lib/python3.13/site-packages/torch/nn/init.py_no_grad_uniform_r>Es' qy9 0 >meanstdc[R"5 URXUS9sSSS5 $!,(df  g=fr8)r:r;r r2r@rAr5s r=_no_grad_normal_rDLs' ~~d9~= r?cS[S[4SjnXSU-- :d XSU--:a[R"SSS9 [R"5 U"X1- U- 5nU"XA- U- 5nUR SU-S- SU-S- US9 UR 5 URU[R"S 5-5 URU5 URX4S 9 UsSSS5 $!,(df  g=f) Nxr6chS[R"U[R"S5- 5-S- $)N?@)matherfsqrt)rFs r=norm_cdf(_no_grad_trunc_normal_..norm_cdf_s(dhhq499S>122c99zjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect. stacklevelr9rI)minmax) floatwarningswarnr:r;r erfinv_mul_rJrLadd_clamp_) r2r@rAr3r4r5rMlus r=_no_grad_trunc_normal_r_Vs:E:e: 1s7{1s7{ 2  ;  ah#% & ah#% & A 1q519 B   C$))C.() D  ! #+ s BC-- C;valc[R"5 URU5sSSS5 $!,(df  g=fN)r:r;fill_r2r`s r=_no_grad_fill_res! ||C  s1 ?c[R"5 UR5sSSS5 $!,(df  g=frb)r:r;zero_r2s r=_no_grad_zero_ris ||~ r? nonlinearityparamc/SQnX;dUS:XagUS:XagUS:Xa[R"S5$US:XavUcS nOQ[U[5(d[U[5(d[U[ 5(aUnO[ S US 35e[R"SSUS --- 5$US :Xag[ SU35e)aReturn the recommended gain value for the given nonlinearity function. The values are as follows: ================= ==================================================== nonlinearity gain ================= ==================================================== Linear / Identity :math:`1` Conv{1,2,3}D :math:`1` Sigmoid :math:`1` Tanh :math:`\frac{5}{3}` ReLU :math:`\sqrt{2}` Leaky Relu :math:`\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}` SELU :math:`\frac{3}{4}` ================= ==================================================== .. warning:: In order to implement `Self-Normalizing Neural Networks`_ , you should use ``nonlinearity='linear'`` instead of ``nonlinearity='selu'``. This gives the initial weights a variance of ``1 / N``, which is necessary to induce a stable fixed point in the forward pass. In contrast, the default gain for ``SELU`` sacrifices the normalization effect for more stable gradient flow in rectangular layers. Args: nonlinearity: the non-linear function (`nn.functional` name) param: optional parameter for the non-linear function Examples: >>> gain = nn.init.calculate_gain( ... "leaky_relu", 0.2 ... ) # leaky_relu with negative_slope=0.2 .. _Self-Normalizing Neural Networks: https://papers.nips.cc/paper/2017/hash/5d44ee6f2c3f71b73125876103c8f6c4-Abstract.html )r$r%r&r'r(r)r*r+rSr,g?r-rIr.{Gz?znegative_slope z not a valid numberrPr/g?zUnsupported nonlinearity )rJrL isinstanceboolintrV ValueError)rjrk linear_fnsnegative_slopes r=rrsLJ!\Y%>    yy~  % =!N5$''5#&&%''#Nug5HIJ JyyNA$5 5677    4\NCDDrOc [RRU5(a%[RR[U4XX#S9$[ XX#5$)aFill the input Tensor with values drawn from the uniform distribution. :math:`\mathcal{U}(a, b)`. Args: tensor: an n-dimensional `torch.Tensor` a: the lower bound of the uniform distribution b: the upper bound of the uniform distribution generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.uniform_(w) r<)r: overrideshas_torch_function_variadichandle_torch_functionr r>r<s r=r r sO( 226::44 viq5   V 55rOc [RRU5(a%[RR[U4XX#S9$[ XX#5$)aFill the input Tensor with values drawn from the normal distribution. :math:`\mathcal{N}(\text{mean}, \text{std}^2)`. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.normal_(w) rC)r:rurvrwr rDrCs r=r r sO( 226::44 fYvc5   F# 99rOc [XX#XES9$)aFill the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq \text{mean} \leq b`. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.trunc_normal_(w) r9)r_)r2r@rAr3r4r5s r=r r s8 "& OOrOc[RRU5(a$[RR[U4XS9$[ X5$)zFill the input Tensor with the value :math:`\text{val}`. Args: tensor: an n-dimensional `torch.Tensor` val: the value to fill the tensor with Examples: >>> w = torch.empty(3, 5) >>> nn.init.constant_(w, 0.3) rd)r:rurvrwr rerds r=r r +sK 226::44 y5   & &&rOc[US5$)zFill the input Tensor with the scalar value `1`. Args: tensor: an n-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.ones_(w) rH)rerhs r=r r =s &# &&rOc[U5$)zFill the input Tensor with the scalar value `0`. Args: tensor: an n-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.zeros_(w) )rirhs r=rrJs & !!rOcUR5S:wa [S5e[R"5 [R"UR XR S.6 SSS5 U$!,(df  U$=f)aFill the 2-dimensional input `Tensor` with the identity matrix. Preserves the identity of the inputs in `Linear` layers, where as many inputs are preserved as possible. Args: tensor: a 2-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.eye_(w) rP,Only tensors with 2 dimensions are supported)out requires_gradN) ndimensionrqr:r;rshaperrhs r=rrWsYaGHH  6<1, each group of channels preserves identity Args: tensor: a {3, 4, 5}-dimensional `torch.Tensor` groups (int, optional): number of groups in the conv layer (default: 1) Examples: >>> w = torch.empty(3, 16, 5, 5) >>> nn.init.dirac_(w) >>> w = torch.empty(3, 24, 5, 5) >>> nn.init.dirac_(w, 3) )z5Only tensors with 3, 4, or 5 dimensions are supportedrz!dim 0 must be divisible by groupsrSrrPrN)rrqsizerTr:r;rgrange)r2r dimensionssizesout_chans_per_grpmin_dimgds r=rrls| ""$J"PQQ KKME Qx&A<==aF*#1X.G  vA7^?PQF0014aQ19LLM1_  -1 A!+ A!+- -1 A!+ A!+ A!+ -$ , M- , Ms 4CE E'cUR5nUS:a [S5eURS5nURS5nSnUR5S:aURSSHnXE-nM X$-nX4-nXg4$)NrPzNFan in and fan out can not be computed for tensor with fewer than 2 dimensionsrSr)dimrqrr)r2rnum_input_fmapsnum_output_fmapsreceptive_field_sizesr0r1s r=_calculate_fan_in_and_fan_outrsJA~ \  kk!nO{{1~ zz|aab!A % "  3F5G ?rOgainc[U5up4U[R"S[X4-5- 5-n[R"S5U-n[ X*Xb5$)aFill the input `Tensor` with values using a Xavier uniform distribution. The method is described in `Understanding the difficulty of training deep feedforward neural networks` - Glorot, X. & Bengio, Y. (2010). The resulting tensor will have values sampled from :math:`\mathcal{U}(-a, a)` where .. math:: a = \text{gain} \times \sqrt{\frac{6}{\text{fan\_in} + \text{fan\_out}}} Also known as Glorot initialization. Args: tensor: an n-dimensional `torch.Tensor` gain: an optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain("relu")) rI@)rrJrLrVr>)r2rr5r0r1rAr3s r=rrsQ44F;OF 3v'7!889 9C #A VR 66rOc[U5up4U[R"S[X4-5- 5-n[ USXR5$)aFill the input `Tensor` with values using a Xavier normal distribution. The method is described in `Understanding the difficulty of training deep feedforward neural networks` - Glorot, X. & Bengio, Y. (2010). The resulting tensor will have values sampled from :math:`\mathcal{N}(0, \text{std}^2)` where .. math:: \text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan\_in} + \text{fan\_out}}} Also known as Glorot initialization. Args: tensor: an n-dimensional `torch.Tensor` gain: an optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.xavier_normal_(w) rI)rrJrLrVrD)r2rr5r0r1rAs r=rrs?24F;OF 3v'7!889 9C FC 88rOmodecUR5nSS/nX;a[SUSU35e[U5up4US:XaU$U$)Nr0r1zMode z" not supported, please use one of )lowerrqr)r2r valid_modesr0r1s r=_calculate_correct_fanrsT ::>> w = torch.empty(3, 5) >>> nn.init.kaiming_uniform_(w, mode="fan_in", nonlinearity="relu") Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.kaiming_uniform_(w.T, ...)``. )r2r3rrjr5r,Initializing zero-element tensors is a no-oprPrQrr9N)r:rurvrwrrrWrXrrrJrLr;r ) r2r3rrjr5fanrrAbounds r=rrsV 226::44  I%5   FLL DQRS  .C , *D 3 C IIcNS E vuB s C,, C:c.SUR;a[R"SSS9 U$[X5n[ X15nU[ R "U5- n[R"5 URSXtS9sSSS5 $!,(df  g=f)a+Fill the input `Tensor` with values using a Kaiming normal distribution. The method is described in `Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification` - He, K. et al. (2015). The resulting tensor will have values sampled from :math:`\mathcal{N}(0, \text{std}^2)` where .. math:: \text{std} = \frac{\text{gain}}{\sqrt{\text{fan\_mode}}} Also known as He initialization. Args: tensor: an n-dimensional `torch.Tensor` a: the negative slope of the rectifier used after this layer (only used with ``'leaky_relu'``) mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'`` preserves the magnitude of the variance of the weights in the forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the backwards pass. nonlinearity: the non-linear function (`nn.functional` name), recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default). generator: the torch Generator to sample from (default: None) Examples: >>> w = torch.empty(3, 5) >>> nn.init.kaiming_normal_(w, mode="fan_out", nonlinearity="relu") Note: Be aware that ``fan_in`` and ``fan_out`` are calculated assuming that the weight matrix is used in a transposed manner, (i.e., ``x @ w.T`` in ``Linear`` layers, where ``w.shape = [fan_out, fan_in]``). This is important for correct initialization. If you plan to use ``x @ w``, where ``w.shape = [fan_in, fan_out]``, pass in a transposed weight matrix, i.e. ``nn.init.kaiming_normal_(w.T, ...)``. rrrPrQr9N) rrWrXrrrJrLr:r;r )r2r3rrjr5rrrAs r=rrBsoV FLL DQRS  .C , *D 3 C ~~a~: s ,B BcUR5S:a [S5eUR5S:XaU$URS5nUR5U-nUR X445R SSUS9nX4:aUR 5 [RRU5upg[R"US5nUR5n Xi-nX4:aUR 5 [R"5 URU5RU5 URU5 SSS5 U$!,(df  U$=f)alFill the input `Tensor` with a (semi) orthogonal matrix. Described in `Exact solutions to the nonlinear dynamics of learning in deep linear neural networks` - Saxe, A. et al. (2013). The input tensor must have at least 2 dimensions, and for tensors with more than 2 dimensions the trailing dimensions are flattened. Args: tensor: an n-dimensional `torch.Tensor`, where :math:`n \geq 2` gain: optional scaling factor generator: the torch Generator to sample from (default: None) Examples: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_LAPACK) >>> w = torch.empty(3, 5) >>> nn.init.orthogonal_(w) rPz4Only tensors with 2 or more dimensions are supportedrrSr9N)rrqnumelr new_emptyr t_r:linalgqrdiagsignr;view_ascopy_rZ) r2rr5rowscols flattenedqrrphs r=rrws,QOPP ||~ ;;q>D <<>T !D  $.66q!y6QI {  <>> w = torch.empty(3, 5) >>> nn.init.sparse_(w, sparsity=0.1) rPr~rr9N) rrqrrJceilr:r;r rrandperm) r2rrAr5rr num_zeroscol_idx row_indices zero_indicess r=rrs.aGHHJD (/*I q#3T{G...K&z 2L,-F( )#  M  Ms AB)) B8methc^^^TRmTSSmS[RS[RS[4UUU4SjjnSTSTSTS 3UlTUlU$) Nargskwargsr6cV>[R"STSTS3[SS9 T"U0UD6$)Nz `nn.init.z)` is now deprecated in favor of `nn.init.z`.rPrQ)rWrX FutureWarning)rrrnew_nameold_names r=deprecated_init(_make_deprecate..deprecated_inits; z!J8*TV W  T$V$$rOz z_(...) .. warning:: This method is now deprecated in favor of :func:`torch.nn.init.z"`. See :func:`~torch.nn.init.z` for details.)__name__r#rrr"__doc__)rrrrs` @@r=_make_deprecaters~}}H}H%rww%"))%%%$ JHIQzR'j :O (O rOrb)rrHN)rrHgrIN)rS)rHN)rr0r.N)rSN)rmN)7rrJrWcollections.abcrtypingrrtyping_extensionsrr:r__all__r"r#_NonlinearityType_FanModerV Generatorr>rDr_rerirprr r r r r rrrtuplerrrrrrrrrrrrrrrrrrr r!rOr=rsbN $#'  > T]t_    & 'MQ: ::!&:38??T3I: :)- > > > >% >  > )- ) ) ) ) )  ) % ) )X!6!!&! 6f BFGE#GE,/%K$,>GE GEX(, 6 6 6 6% 6  6:(, : : : :% :  ::(, P P P P P  P % P P>'f'5'V'$ '& 'V ' "6 "f "F*26232v2j&U38_.(,7 7 7%7 7F(,9 9 9%9 9>3633c3&2(, >C >C >C >C$ >C % >C  >CF&2(, 2; 2; 2; 2;$ 2; % 2;  2;n(,0 0 0%0 0l(, # ## #% #  #N(2r6*xB/?. ( #  ! 9 %d 1/ !"23 1 [ )  !rO