qyiXTSSKJr SSKJr SSKrSSKJr SSKJr SSKJ r J r SSK J r SS K JrJrJr SS KJr SS KJr SS KJrJr SS KJrJr SSKJrJr SSKJr SSK J!r!J"r"J#r# SSK$J%r%J&r& SSK'J(r( SSK)J*r* Sr+\"S5S7Sj5r,S\RZS\.S\RZ4Sjr/S8S\R`S\RZS\RZS\RZS \RZS-S!\1S"\1S#\\!4S$jjr2\"\,5"S%S&\R`55r3\"S'5"S(S)\R`55r4"S*S+\R`5r5"S,S-\5r6\""S.S/\55r7"S0S1\R`5r8\""S2S3\755r9\""S4S5\7\ 55r:/S6Qr;g)9)Callable)OptionalN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)maybe_autocastmerge_with_config_defaults)capture_outputs) GraniteConfigcUSSURSS-24nUSURSS-S24n[R"U*U4SS9$)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)xx1x2s i/home/mande/repo/quber/.venv/lib/python3.13/site-packages/transformers/models/granite/modeling_granite.py rotate_halfr+,sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''rotary_pos_embcURU5nURU5nX-[U5U--nX-[U5U--nXV4$)aIApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer+)qkcossin unsqueeze_dimq_embedk_embeds r*apply_rotary_pos_embr73sS& -- &C -- &Cw;q>C/0Gw;q>C/0G  r, hidden_statesn_repreturncURup#pEUS:XaU$USS2SS2SSS2SS24RX#XU5nURX#U-XE5$)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r$expandreshape)r8r9batchnum_key_value_headsslenhead_dims r* repeat_kvrBMs_ 2?1D1D.E z!!Qa"23::5W\dlmM  e(CT TTr,modulequerykeyvalueattention_maskscalingdropoutkwargsc [X R5n[X0R5n [R"XR SS55U-n UbX-n [ R RU S[RS9RUR5n [ R RXURS9n [R"X5n U R SS5R5n X4$)Nr!rr )r#dtype)ptrainingr)rBnum_key_value_groupsr%matmul transposer functionalsoftmaxfloat32torLrIrN contiguous) rCrDrErFrGrHrIrJ key_states value_states attn_weights attn_outputs r*eager_attention_forwardr[Ys3 ; ;[TU]5 XlX l[ USUR UR -5UlUR UR-Ul URUl URUl SUl [R"UR UR UR-UR S9Ul[R"UR URUR-UR S9Ul[R"UR URUR-UR S9Ul[R"UR UR-UR UR S9Ulg)NrATbias)super__init__r_r`getattr hidden_sizenum_attention_headsrAr?rOattention_multiplierrHattention_dropout is_causalrLinearattention_biasq_projk_projv_projo_projselfr_r` __class__s r*reGraniteAttention.__init__vsF  " F4F4F&JdJd4de $*$>$>&B\B\$\!22 !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  r,r8position_embeddingsrGpast_key_valuesrJr:c URSSn/UQSPURP7nURU5RU5R SS5nUR U5RU5R SS5n UR U5RU5R SS5n Uup[XX5upUbURXUR5up[R"URR[5n U "UUU U U4UR(dSO UR UR"S.UD6upUR$"/UQSP76R'5nUR)U5nX4$)Nr rr!)rIrH)r$rArnviewrQrorpr7updater`r get_interfacer__attn_implementationr[rNrjrHr=rVrq)rsr8rvrGrwrJ input_shape hidden_shape query_statesrWrXr2r3attention_interfacerZrYs r*forwardGraniteAttention.forwards~$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &#7 RU#[  &'6'='=jX\XfXf'g $J(?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! "));;;;FFH kk+. ((r,) rjr_rArkror`rOrqrnrHrpNNNN)__name__ __module__ __qualname____firstlineno____doc__rintrer%Tensortuplerrrr__static_attributes__ __classcell__rts@r*r]r]rsG } t  4IM.2(, &)||&)#5<<#=>E&) t+ &)  &) +, &) u||U\\) *&)&)r,r]RMSNormcx^\rSrSrS S\SS4U4SjjjrS\RS\R4SjrSr S r U=r $) GraniteRMSNormepsr:Nc>[TU]5 [R"[R "U55UlX lg)z- GraniteRMSNorm is equivalent to T5LayerNorm N)rdrer Parameterr%onesweightvariance_epsilon)rsrgrrts r*reGraniteRMSNorm.__init__s/ ll5::k#:; #r,r8cURnUR[R5nUR S5R SSS9nU[R "X0R-5-nURURU5-$)Nr!r T)keepdim) rLrUr%rTpowmeanrsqrtrr)rsr8 input_dtypevariances r*rGraniteRMSNorm.forwardsw#)) %((7  $$Q',,R,>% H?T?T4T(UU {{]--k:::r,c^[URR5SUR3$)Nz, eps=)rrr$r)rss r* extra_reprGraniteRMSNorm.extra_reprs* ))*+6$2G2G1HIIr,)rr)gư>) rrrrfloatrer%rrrrrrs@r*rrsB$$$$$;U\\;ell;JJr,rc.^\rSrSrU4SjrSrSrU=r$) GraniteMLPc>[TU]5 XlURUlURUl[ R "URURURS9Ul[ R "URURURS9Ul [ R "URURURS9Ul [URUl g)Nrb)rdrer_rgintermediate_sizerrlmlp_bias gate_projup_proj down_projr hidden_actact_fnrsr_rts r*reGraniteMLP.__init__s  !--!'!9!94#3#3T5K5KRXRaRabyy!1!143I3IPVP_P_` 4#9#94;K;KRXRaRabV../ r,cURURURU55URU5-5nU$r)rrrr)rsr'rs r*rGraniteMLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r,)rr_rrrgrr)rrrrrerrrrs@r*rrs0r,rc^\rSrSrS\S\4U4SjjrSS\RS\RS-S\RS-S \ S-S \ S-S \ \R\R4S-S \ \S \R4SjjrSrU=r$)GraniteDecoderLayerr_r`c.>[TU]5 URUl[XS9Ul[ U5Ul[URURS9Ul [URURS9Ul URUl g)N)r_r`r) rdrergr] self_attnrmlpr rms_norm_epsinput_layernormpost_attention_layernormresidual_multiplierrrs r*reGraniteDecoderLayer.__init__sx !--)Mf%-f.@.@fFYFYZ(6v7I7IvObOb(c%#)#=#= r,Nr8rG position_idsrw use_cachervrJr:c UnURU5nUR"SUUUUUUS.UD6upXUR--nUnURU5nUR U5nXUR--nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Cache`, *optional*): cached past key and value projection states position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r8rGrrwrrv)rrrrr) rsr8rGrrwrrvrJresidual_s r*rGraniteDecoderLayer.forwards<!,,]; >> ')%+ 3    !43K3K#KK  55mD /  43K3K#KK r,)rgrrrrr)NNNFN)rrrrrrrer%r LongTensorrboolrrrrrrrs@r*rrs>}>>/304(,!&HL2||2 t+2&&- 2  2 $; 2#5<<#=>E2+,2 22r,rcR\rSrSr%\\S'SrSrS/rS/r Sr Sr Sr Sr Sr\\S.rSrg ) GranitePreTrainedModelir_modelTrrw)r8 attentionsrN)rrrrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr]_can_record_outputsrrr,r*rrsQ &*#./#4"5N!"&,&r,rc ^\rSrSr%\R \S'SS\4U4Sjjjr\ SS\S-S\ SS\ S-S \ S \ 44S jj5r\R"5\S 55rS rU=r$)GraniteRotaryEmbeddingi.inv_freqNr_c>[TU]5 URUlURUlXlUR R SUlURnURS:wa[URnU"UR U5uo@l URSUSS9 URSUR5SS9 g)N rope_typedefaultrF) persistentoriginal_inv_freq) rdremax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr_rope_parametersrcompute_default_rope_parametersrattention_scalingregister_bufferclone)rsr_device rope_init_fnrrts r*reGraniteRotaryEmbedding.__init__1s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L(( ZeD 0(..2BuUr,rz torch.deviceseq_lenr:z torch.Tensorc URSn[USS5=(d URUR-nSnSU[R "SUS[R S9RU[RS9U- -- nXe4$) aH Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetarANg?rr!rL)rrL) rrfrgrhr%arangeint64rUr)r_rrbaser#attention_factorrs r*r6GraniteRotaryEmbedding.compute_default_rope_parametersAs&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k ))r,cLURSSS2S4R5RURSSS5R UR 5nUSS2SSS24R5n[ UR R[5(a0UR RS:waUR ROSn[USS9 UR5UR5-RSS5n[R"Xf4SS 9nUR5UR-nUR5UR-n SSS5 WR UR S 9W R UR S 94$!,(df  N@=f) Nrr rmpscpuF) device_typeenabledr!r"r)rrr<r$rUr isinstancetypestrrrQr%r&r2rr3rL) rsr'rinv_freq_expandedposition_ids_expandedrfreqsembr2r3s r*rGraniteRotaryEmbedding.forward_sN!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E'E!((--[`J`ahhmmfk  U C&,,.1F1L1L1NNYYZ[]^_E))UN3C'')d444C'')d444C D vvAGGv$cff177f&;;; D Cs BF F#)rr_rrrrr)rrrrr%rrrre staticmethodrrrrrno_gradrrrrrs@r*rr.sllV}VV '++/"*$*(*t* ~u$ % **: ]]_ < [TU]U5 URUlURUl[ R "URURUR5Ul[ R"[UR5Vs/sHn[X5PM sn5Ul [URURS9Ul[#US9UlSUlUR(UlUR+5 gs snf)Nrr_F)rdre pad_token_id padding_idx vocab_sizer Embeddingrg embed_tokens ModuleListrangenum_hidden_layersrlayersrrnormr rotary_embgradient_checkpointingembedding_multiplier post_initrrs r*reGraniteModel.__init__qs  !.. ++LL):):FO  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L(;;')+%  & "oomoW![[)H4;;+H+HIM)*) /#$7 MJ -0 &++  r,)rrrrrr rr )NNNNNN)rrrrrrerrrr%rrr FloatTensorrrrrrrrrs@r*rros}" .2.204(,26!%5 ##d*5  t+5 &&- 5  5 ((4/ 5 $;5 +,5  !5  5 r,rcP^\rSrSrSS0rSS0rSS/S/40rU4Sjr\\ SS \ RS -S \ RS -S \ RS -S \ S -S\ RS -S\ RS -S\S -S\\ R-S\\S\4Sjj55rSrU=r$)GraniteForCausalLMizlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputr8logitsc>[TU]U5 [U5UlURUl[ R "URURSS9UlUR5 g)NFrb) rdrerrr rrlrgr&rrs r*reGraniteForCausalLM.__init__sU  !&)  ++yy!3!3V5F5FUS  r,NrrGrrwrlabelsrlogits_to_keeprJr:c UR"SUUUUUUS.U D6n U Rn [U[5(a [ U*S5OUn UR U SS2U SS245n XR R- n SnUb)UR"SXUR RS.U D6n[UU U RU RU RS9$)a{ Example: ```python >>> from transformers import AutoTokenizer, GraniteForCausalLM >>> model = GraniteForCausalLM.from_pretrained("meta-granite/Granite-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-granite/Granite-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)rrGrrwrrN)r(r+r )lossr(rwr8rr)rrrrslicer&r_logits_scaling loss_functionr rrwr8r)rsrrGrrwrr+rr,rJoutputsr8 slice_indicesr(r.s r*rGraniteForCausalLM.forwards>,0::, )%+' , ,  11 8B>SV8W8W~ot4]k mA}a,?@A++444  %%pVt{{OeOepiopD%#33!//))   r,)r&rr )NNNNNNNr)rrrr_tied_weights_keys_tp_plan_pp_planrerrr%rrrr#rrrrrrrrrs@r*r%r%s*,GH23H_-z:;H.2.204(,26*.!%-.6 ##d*6  t+6 &&- 6  6 ((4/ 6   4'6 $;6 ell*6 +,6  6 6 r,r%)r%rr)r)ry)<collections.abcrtypingrr%r activationsr cache_utilsrr generationr integrationsr r r masking_utilsrmodeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrrutils.output_capturingrconfiguration_graniterr+r7rrrBModulerr[r]rrrrrrr%__all__rr,r*rJs,% !.)ff/9OKF&IIG50(*+,2 UU\\ U# U%,, U&% II% <<% % << % LL4' %  %%'(%2)*@)ryy@)+@)FY'JRYYJ(J(   =4=@_$>