Ei|}J%SSKrSSKJr SSKJr SSKJrJrJrJ r SSK r SSK J r SSK JrJr SSKJrJr SSKJr S S KJrJrJr S S KJr S S KJrJr /S Qr"SS\5r"SS\5r "SS\ RB5r""SS\ RB5r#"SS\ RB5r$S\%S\%S\%S\%S\%S\ \S\&S\S \$4S!jr'S"\0r(\)\*\4\+S#'0\(ES$S%S&.Er,"S'S(\5r-"S)S*\5r."S+S,\5r/"S-S.\5r0"S/S0\5r1\"5\"S1\-Rd4S29SS3S4.S\ \-S\&S\S \$4S5jj55r3\"5\"S1\.Rd4S29SS3S4.S\ \.S\&S\S \$4S6jj55r4\"5\"S1\/Rd4S29SS3S4.S\ \/S\&S\S \$4S7jj55r5\"5\"S1\0Rd4S29SS3S4.S\ \0S\&S\S \$4S8jj55r6\"5\"S9S29SS3S4.S\ \1S\&S\S \$4S:jj55r7SAS;\%S\%S\*S?\&S S=4 S@jjr8g)BN) OrderedDict)partial)AnyCallable NamedTupleOptional)Conv2dNormActivationMLP)ImageClassificationInterpolationMode)_log_api_usage_once)register_modelWeights WeightsEnum)_IMAGENET_CATEGORIES)_ovewrite_named_paramhandle_legacy_interface) VisionTransformerViT_B_16_WeightsViT_B_32_WeightsViT_L_16_WeightsViT_L_32_WeightsViT_H_14_Weightsvit_b_16vit_b_32vit_l_16vit_l_32vit_h_14c\rSrSr%\\S'\\S'\\S'\Rr\ S\R4\S'\Rr \ S\R4\S'Sr g ) ConvStemConfig out_channels kernel_sizestride. norm_layeractivation_layerN)__name__ __module__ __qualname____firstlineno__int__annotations__nn BatchNorm2dr'rModuleReLUr(__static_attributes__r)b/home/mande/repo/quber/.venv/lib/python3.13/site-packages/torchvision/models/vision_transformer.pyr"r" sJ K+->>Jbii(913hsBII~.8r5r"cL^\rSrSrSrSrS\S\S\4U4SjjrU4Sjr S r U=r $) MLPBlock(zTransformer MLP block.r in_dimmlp_dimdropoutcv>[TU]XU/[RSUS9 UR 5Hn[ U[R 5(dM$[RRUR5 URcM\[RRURSS9 M g)N)r(inplacer<ư>std) super__init__r0GELUmodules isinstanceLinearinitxavier_uniform_weightbiasnormal_)selfr:r;r<m __class__s r6rCMLPBlock.__init__-s} 6!2RWWVZdklA!RYY''''166%GGOOAFFO5  r5c >URSS5nUbUS:aN[S5H?n SH6n USU S-SU 3n USU -SU 3n X;dM#URU 5X'M8 MA [T U]UUUUUUU5 g)Nversionr )rJrKlinear_r.)getrangepoprB_load_from_state_dict)rM state_dictprefixlocal_metadatastrict missing_keysunexpected_keys error_msgsrRitypeold_keynew_keyrOs r6rYMLPBlock._load_from_state_dict6s!$$Y5 ?gk1X.D!'!uAdV[TU]5 XlU"U5Ul[R "X!USS9Ul[R"U5UlU"U5Ul [X#U5Ul g)NT)r< batch_first) rBrCrpln_1r0MultiheadAttentionself_attentionDropoutr<ln_2r8mlp)rMrprqr;r<rrr'rOs r6rCEncoderBlock.__init__Ysh "z*  33JSdrvwzz'* z* J9r5inputc*[R"UR5S:HSUR35 UR U5nUR X"USS9up#UR U5nX!-nURU5nURU5nX$-$)NrU2Expected (batch_size, seq_length, hidden_dim) got F) need_weights) torch_assertdimshaperurwr<ryrz)rMr|x_ys r6forwardEncoderBlock.forwardns eiikQ&*\]b]h]h\i(jk IIe ""1"? LLO I IIaL HHQKu r5)r<ruryrzrprwr*r+r,r-rfrr0 LayerNormr.rhrrr2rCTensorrr4rirjs@r6rlrlVs$6=R\\t5T::: :  : ! :S%((//12::* U\\  r5rlc^\rSrSrSr\"\RSS94S\S\S\S\S \S \ S \ S \ S \ R R44U4Sjjjr S\ R4SjrSrU=r$)Encoderzz?Transformer Model Encoder for sequence to sequence translation.r?rn seq_length num_layersrprqr;r<rrr'.c >[T U]5 [R"[R "SX5R SS95Ul[R"U5Ul [5n [U5Hn [UUUUUU5U SU 3'M [R"U 5UlU"U5Ulg)Nrg{Gz?r@encoder_layer_)rBrCr0 ParameterremptyrL pos_embeddingrxr<rrWrl Sequentiallayersln) rMrrrprqr;r<rrr'rrarOs r6rCEncoder.__init__}s  \\%++a*P*X*X]a*X*bczz'* .9mz"A+7! ,F^A3' (#mmF+ Z(r5r|c[R"UR5S:HSUR35 XR-nUR UR URU555$)NrUr~)rrrrrrrr<)rMr|s r6rEncoder.forwardsZ eiikQ&*\]b]h]h\i(jk***wwt{{4<<#6788r5)r<rrrrrjs@r6rrzsI6=R\\t5T))) )  )  ))!)S%((//12)):9U\\99r5rc.^\rSrSrSrSSSS\"\RSS9S4S\S \S \S \S \S \S\ S\ S\S\ \S\ S\ R R4S\ \\4U4SjjjrS\ R"S\ R"4SjrS\ R"4SjrSrU=r$)rz;Vision Transformer as per https://arxiv.org/abs/2010.11929.iNr?rn image_size patch_sizerrprqr;r<rr num_classesrepresentation_sizer'.conv_stem_configsc  >[TU]5 [U5 [R"X-S:HS5 XlX lXPlX`lXl Xpl Xl Xl Xl U b[R"5n Sn[!U 5HeunnU R#SU3[%UUR&UR(UR*URUR,S95 UR&nMg U R#S[R."XSS95 XlO[R."SXRUS 9UlX-S -n[R2"[R4"SSU55UlUS- n[9UUUUUUUU 5UlUUl[?5nU c[R@"XY5US 'OJ[R@"XZ5US '[RB"5US '[R@"X5US '[R"U5Ul"[GUR0[R.5(aUR0RHUR0R(S-UR0R(S-n[RJRMUR0RN[PRR"SU- 5S9 UR0RTb3[RJRWUR0RT5 GOUR0RXGb[GUR0RX[R.5(a[RJR[UR0RXRNS[PRR"SUR0RXR&- 5S9 UR0RXRTb=[RJRWUR0RXRT5 []URDS 5(a[GURDR^[R@5(aURDR^R`n[RJRMURDR^RN[PRR"SU- 5S9 [RJRWURDR^RT5 [GURDRb[R@5(a{[RJRWURDRbRN5 [RJRWURDRbRT5 gg)Nrz&Input shape indivisible by patch size!rU conv_bn_relu_) in_channelsr$r%r&r'r( conv_lastr)rr$r%)rr$r%r&r head pre_logitsactr@rg@)meanrA)2rBrCrrrrrrqr;rrr<rrr'r0r enumerate add_moduler r$r%r&r(Conv2d conv_projrzeros class_tokenrencoderrrrGTanhheadsrFrrH trunc_normal_rJmathsqrtrKzeros_rrLhasattrr in_featuresr)rMrrrrprqr;r<rrrrr'rseq_proj prev_channelsraconv_stem_layer_configr heads_layersfan_inrOs r6rCVisionTransformer.__init__s? D! j-24\]$$$ !2 &#6 $  (}}HM-67H-I))###A3'($1%;%H%H$:$F$F5<<#9#D#D)?)P)P  !7 C C .J   RYY=ghi )1NYYJWaDN!.14 << Aq*(EFa            %4?M  &#%99Z#EL )+:)SL &"$'')L #%99-@#NL ]]<0 dnnbii 0 0^^//$..2L2LQ2OORVR`R`RlRlmnRooF GG ! !$.."7"7TYYq6z=R ! S~~"".t~~223 ^^ % % 1jAYAY[][d[d6e6e GGOO((//ctyyt~~OgOgOtOtIt?u  ~~'',,8t~~77<<= 4::| , ,DJJ">DIIaRXjDY ! Z GGNN4::0055 6 djjooryy 1 1 GGNN4::??11 2 GGNN4::??// 0 2r5rreturncURup#pEURn[R"X@R:HSURSUS35 [R"XPR:HSURSUS35 XF-nXV-nUR U5nUR X RXx-5nURSSS5nU$)NzWrong image height! Expected z but got !zWrong image width! Expected rr r) rrrrrrreshaperqpermute) rMrnchwpn_hn_ws r6_process_input VisionTransformer._process_input sWW a OO a??*.KDOOK\\efgehhi,jk a??*.J4??J[[defdggh,ijff NN1  IIa#) 4 IIaA r5cURU5nURSnURRUSS5n[R "X1/SS9nUR U5nUSS2S4nURU5nU$)Nrrr)rrrexpandrcatrr)rMrrbatch_class_tokens r6rVisionTransformer.forward!s~    " GGAJ!,,33Ar2> II(,! 4 LLO adG JJqMr5)rrrrr<rrrqrr;r'rrrr)r*r+r,r-rfrr0rr.rhrrrr2listr"rCrrrr4rirjs@r6rrsE#&-15`_'s training recipe.  num_paramsrr_metrics_ops _file_size_docsurl transformsrz>https://download.pytorch.org/models/vit_b_16_swag-9ac1b537.pthr resize_size interpolationi^-)rrg~jtSU@giX@gˡEK@g|?5^t@ These weights are learnt via transfer learning by end-to-end fine-tuning the original `SWAG `_ weights on ImageNet-1K data. rrrrrrzAhttps://download.pytorch.org/models/vit_b_16_lc_swag-4e70ced5.pth+https://github.com/pytorch/vision/pull/5793gbX9xT@gQ X@ These weights are composed of the original frozen `SWAG `_ trunk weights and a linear classifier learnt on top of them trained on ImageNet-1K data. rrrrrrrr)Nr*r+r,r-rrr r IMAGENET1K_V1r BICUBIC_COMMON_SWAG_METAIMAGENET1K_SWAG_E2E_V1IMAGENET1K_SWAG_LINEAR_V1DEFAULTr4r)r5r6rr_s# G.#>  ""f##  ! M,% L +33    ""##  ! 4!( O +33    C""##  ! !6Gr5rc T\rSrSr\"S\"\SS90\ESSSSS S S .0S S SS.ES9r\r Sr g)riz9https://download.pytorch.org/models/vit_b_32-d86f8d99.pthrri1BrzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_b_32rg|?5^R@gW@rgA`Т@gl u@rrrr)N r*r+r,r-rrr rrrr4r)r5r6rrs\ G.#>  ""f##  ! M,Gr5rc \rSrSr\"S\"\SSS90\ESSSS S S S .0S SSS.ES9r\"S\"\SS\ RS90\ ESSS SSS .0SSSS.ES9r \"S\"\SS\ RS90\ ESSSS SS S .0S SS!S".ES9r \rS#rg$)%riz9https://download.pytorch.org/models/vit_l_16-852ce7e3.pthr)rri#rzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_l_16rg|?5^S@gFԨW@rgףp= N@g;O$@a These weights were trained from scratch by using a modified version of TorchVision's `new training recipe `_. rrz>https://download.pytorch.org/models/vit_l_16_swag-4f3808c9.pthri0)r r gjtV@gT㥛ĠX@gƟv@gy&11@rrzAhttps://download.pytorch.org/models/vit_l_16_lc_swag-4d563306.pthrgMbXIU@g^I [X@rrr)Nrr)r5r6rrs% G.#3O  #"f##  " M.% L +33    #"##  " 4!( O +33    C#"##  " !6Gr5rc T\rSrSr\"S\"\SS90\ESSSSS S S .0S S SS.ES9r\r Sr g)riz9https://download.pytorch.org/models/vit_l_32-c7638314.pthrri[ErzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_l_32rg|?5>S@gGzDW@rgK7.@gE@rrrr)Nr r)r5r6rrs\ G.#>  #"f#"  " M,Gr5rc \rSrSr\"S\"\SS\RS90\ ESSSSS S .0S S S S.ES9r \"S\"\SS\RS90\ ESSSSSSS .0SSSS.ES9r \ r Sr g)ri2z>https://download.pytorch.org/models/vit_h_14_swag-80465313.pthri%)rrrgS#V@g#~jX@rg~jŏ@gK7I@rrrzAhttps://download.pytorch.org/models/vit_h_14_lc_swag-c1eb923e.pthrri@%rgZd;OmU@gQnX@g= ףpd@gI k֢@rrr)N)r*r+r,r-rrr r rrrrrr4r)r5r6rr2s$ L +33    #"##  " 4!( O +33    C#"##  " !6%Gr5r pretrained)rT)rrc R[RU5n[SSSSSSUUS.UD6$)au Constructs a vit_b_16 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_B_16_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_B_16_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_B_16_Weights :members:   rrrprqr;rrr))rverifyrrrrs r6rrkE(%%g.G      r5c R[RU5n[SSSSSSUUS.UD6$)au Constructs a vit_b_32 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_B_32_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_B_32_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_B_32_Weights :members: r#rrrrr))rrrrs r6rrrr5c R[RU5n[SSSSSSUUS.UD6$)au Constructs a vit_l_16 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_L_16_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_L_16_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_L_16_Weights :members: rrr))rrrrs r6rrE(%%g.G      r5c R[RU5n[SSSSSSUUS.UD6$)au Constructs a vit_l_32 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_L_32_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_L_32_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_L_32_Weights :members: r#rrrrrr))rrrrs r6rrrr5)rNc R[RU5n[SSSSSSUUS.UD6$)au Constructs a vit_h_14 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_H_14_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_H_14_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_H_14_Weights :members: r#riirr))rrrrs r6r r rr5r model_statezOrderedDict[str, torch.Tensor]interpolation_mode reset_headscUSnURupgnUS:wa[SUR35eX-S-S-n X:wGa3US-nU S-n USS2SS2SS24n USS2SS2SS24n U RSSS5n [[R "U55n X-U:wa[SX-SU35eU R SXU 5n X-n [RRU U US S 9nUR SX5nURSSS5n[R"X/SS 9nXS'U(aC[5nUR5H#unnURS 5(aMUUU'M% UnU$) awThis function helps interpolate positional embeddings during checkpoint loading, especially when you want to apply a pre-trained model on images with different resolution. Args: image_size (int): Image size of the new model. patch_size (int): Patch size of the new model. model_state (OrderedDict[str, torch.Tensor]): State dict of the pre-trained model. interpolation_mode (str): The algorithm used for upsampling. Default: bicubic. reset_heads (bool): If true, not copying the state of heads. Default: False. Returns: OrderedDict[str, torch.Tensor]: A state dict which can be loaded into the new model. zencoder.pos_embeddingrz%Unexpected position embedding shape: r NrzPseq_length is not a perfect square! Instead got seq_length_1d * seq_length_1d = z and seq_length = T)sizemode align_cornersrr)r ValueErrorrr.rrrr0 functional interpolaterrritems startswith)rrr#r$r%rrrrqnew_seq_lengthpos_embedding_tokenpos_embedding_img seq_length_1dnew_seq_length_1dnew_pos_embedding_imgnew_pos_embeddingmodel_state_copykvs r6interpolate_embeddingsr9s* 78M - 3 3A:Av@ATAT@UVWW .14q8N #a !+Arr1H5)!QR(3.55aA>DIIj12  (J 6bcpdAcCCUV`Uab  .55aTab&4!# 9 9 "# !:! !6 = =a \!6 = =aA F!II':&RXYZ/@+, AL #))+1||G,,*+$Q',+K r5)bicubicF)9r collectionsr functoolsrtypingrrrrrtorch.nnr0ops.miscr r transforms._presetsr r utilsr_apirrr_metar_utilsrr__all__r"r8r2rlrrr.boolrrdictstrr/rrrrrrrrrrrr r9r)r5r6rIsh #66 0H'66'B 9Z9+ s+ \!299!H#9bii#9LQ Qh    k " B&  d38n8KL{L^{4M{M`{46%{6%r,0@0N0N!OP6:T"23d]`evQ@,0@0N0N!OP6:T"23d]`evQ@,0@0N0N!OP6:T"23d]`evQ@,0@0N0N!OP6:T"23d]`evQ@!566:T"23d]`ev7H( KKK2K K  K & Kr5