pyi, SSKrSSKrSSKJr SSKJrJrJrJr SSK r SSK r SSK J r SSKJr SSKJrJr SSKJr SS KJrJrJr \"5(aSS KJr \(aSS KJr SS KJr SSKJr "S S5r"SS\5r "SS\ 5r!"SS\ RD5r#"SS\ RD5r$"SS5r%"SS5r&"SS\!5r'"SS\5r("SS \ 5r)"S!S"\ 5r*S#\+\,\4S$\-S%\.S&\+\,\44S'jr/S#\+\,\4S$\-S%\.S&\+\,\44S(jr0S#\+\,\4S$\-S&\+\,\44S)jr1g)*N)Iterable) TYPE_CHECKINGAnyOptionalcast)prune_linear_layer) ModelOutputis_sklearn_available)GenerationConfig)LogitsProcessorListMinLengthLogitsProcessorSuppressTokensLogitsProcessor) roc_curve)PreTrainedModel)PreTrainedTokenizerBasec\rSrSr%SrSr\\S'S\RS\ \R\R44Sjr S\RS\RS \ 4S jrS rg ) CandidateGenerator'z`Abstract base class for all candidate generators that can be applied during assisted generation.Frequires_model_outputs input_idsreturnc 2[URS35e)a Fetches the candidates to be tried for the current input. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) Return: `torch.LongTensor` of shape `(batch_size, candidate_length)` containing the candidate sequences to be assessed by the model and, optionally, a `torch.FloatTensor` of shape `(batch_size, candidate_length, vocabulary_size)` containing the logits associated to each candidate. zT is an abstract class. Only classes inheriting this class can call `get_candidates`.NotImplementedError __class__)selfrkwargss h/home/mande/repo/quber/.venv/lib/python3.13/site-packages/transformers/generation/candidate_generator.pyget_candidates!CandidateGenerator.get_candidates,s!"~~r s  scores num_matchesc2[URS35e) Updates the candidate generation strategy based on the outcomes. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) scores (`torch.FloatTensor` of shape `(batch_size, candidate_length, config.vocab_size)`): Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam search or log softmax for each vocabulary token when using beam search num_matches (`int`): The number of matches between the candidate sequences and the model predictions. z_ is an abstract class. Only classes inheriting this class can call `update_candidate_strategy`.rrrr$r%s r update_candidate_strategy,CandidateGenerator.update_candidate_strategy=s%"~~+ +  r#N)__name__ __module__ __qualname____firstlineno____doc__rbool__annotations__torch LongTensortuple FloatTensorr!intr)__static_attributes__r+r#r rr'sgj#(D( (8(8 uUM]M]_d_p_pMpGq " 53C3C UM^M^ mp r#rc\rSrSrSrSS\R SSSSS \S \RS-S \ S 4 S jjr S\R S\ \R \R44Sjr S\R S\RS\4SjrS\R S\ \\44SjrSS\R S\S\S\4SjjrS\R S\S\S\4SjrS\S\ \R \RS-44SjrSrg)AssistedCandidateGeneratorPa& `CandidateGenerator` class to be used for assisted generation and speculative decoding. This class generates candidates through the use of a smaller model. Read the following blog post for more information: https://huggingface.co/blog/assisted-generation Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) assistant_model (`PreTrainedModel`): The model to be used for generating candidates. This model should be smaller than the main model. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. model_kwargs (`Dict`): The keyword arguments that will be passed to the main model, and are used as base inputs for the assistant model as well. inputs_tensor (`torch.Tensor`, *optional*): The model input tensor. In encoder-decoder models, this is the encoder input. Nrassistant_modelrgeneration_configr model_kwargs inputs_tensorlogits_processorrc &URnURU5nUbURU5nX l[R"UR 5UlUR R5nUR R"S0UDSS0D6 UR RUl UR RUl URUR l 0n UR5HcupU S;dM [U [R5(aU R!5RU5O[R"U 5X'Me SU ;aUR#5(dU S UR$R&(aFUR)XPR R*U 5up\n UR-XYXR 5n OSU;aUSU S'XlUR$R&(aSUlOySU ;alSUlUR.R3S[R4"UR6S S 4UR[R8S 95UR.S 'OSUlUbUO [;5Ul[R"U5UlSUR lSUR l URUR l SUR l!UR RDUl#SUR l"SUR l$UR RJUl&SUR l%UR<V s/sHn [U [N5(aMU PM sn UlS UR l([S5(a>UR R(a"[UU5[VLa/Ul,/Ul-ggggs sn f)N defaults_onlyT)encoder_outputspast_key_valueslogits_to_keeprCdecoder_input_idsrdecoder_attention_maskrr devicedtypeattention_mask dynamic_fullr+).rItor<copydeepcopyr=assistant_generation_config_get_default_generation_paramsupdatenum_assistant_tokensassistant_confidence_threshold eos_token_iditems isinstancer3Tensordetach_supports_logits_to_keepconfigis_encoder_decoder_prepare_model_inputs bos_token_id._prepare_encoder_decoder_kwargs_for_generationassistant_kwargs input_ids_keygetonesshapelongrr@return_dict_in_generate output_scores is_assistant min_lengthmain_model_min_lengthmin_new_tokens max_lengthmain_model_max_lengthrcache_implementationr typer:probsmatches)rrr<r=r>r?r@rIglobal_defaultsr`keyvaluemodel_input_name processors r __init__#AssistedCandidateGenerator.__init__gs!''LL(  $),,V4M /,0==9Z9Z+[(::YY[ ((//V/VQUV$($D$D$Y$Y!.2.N.N.m.m+9J8V8V((5&,,.JC@@1;E5<<1P1PELLN%%f-VZVcVcdiVj!%/ / /8`8`8b8b !12  ! ! 4 4@O@e@e??LLN^A =M-= /]]1ACcCc  , .2>?P2Q . / 0  ! ! 4 4!4D  "2 2!,D 6:6K6K6O6O( IOOA.29;K;KSXS]S]^7D ! !"2 3 "-D 5E4P 0ViVk!%/@!A9=6/3,@D@c@c=.2+&*%;%;%F%F",0)04-%)%;%;%F%F",0)'+'<'<! '<)JyZrDsI'<!  7E3 ! " "00OOT 88DJDL9P #! s P!Prc URURR5nURU5up4US:XaUS4$UR U5 UR XU5nUR U5upgXg4$) Fetches the candidates to be tried for the current input. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) Return: `torch.LongTensor` of shape `(batch_size, candidate_length)` containing the candidate sequences to be assessed by the model and a `torch.FloatTensor` of shape `(batch_size, candidate_length, vocabulary_size)` containing the logits associated to each candidate. rN)rMr<rI_calculate_new_tokens_update_past_and_masks_prepare_generation_args_generate_candidates)rrrrkmax_new_tokensgeneration_args candidate_idscandidate_logitss r r!)AssistedCandidateGenerator.get_candidatessLL!5!5! heuristicheuristic_transientrr rN)rPnum_assistant_tokens_schedulelenrSmaxr rTror:rqextendrpappendissubsetrnpargmin) rrr$r% excess_lengthfprtpr thresholdsfnrcostsoptimal_threshold_indexbest_thresholds r r)4AssistedCandidateGenerator.update_candidate_strategys  + + I IN  c&)nq00))Q.),/43L3Lq3P,Q) ! " "00OOOT 88 LL  k 1 24::T\\!22 ##A&  Oc$,,.??Mq JJ ~/DJJ!#A (E(E'0tzz'J$*#ga#g +-))E*:'!+!DR`00O)F#9P #r#cURSn[[UR5URU- S- 5n[ [X0R U- 5S5nXC4$)zCCalculate the minimum and maximum number of new tokens to generate.r r)rdminr7rSrmrrj)rr new_cur_lenrrks r r{0AssistedCandidateGenerator._calculate_new_tokenss_oob) S!:!:;T=W=WZe=ehi=ijS1K1Kk1YZ\]^--r#remove_from_pkvnum_added_tokenscHURRSS5SLnU(aURSS- U- nURSRXS- 5 [ URURSUR R R5Ul[URURSUR R R5Ul[URURS5UlSURl U$)zLUpdate past key values and attention masks for subsequent generation rounds.rDNrr ) r`rbrdcrop_prepare_attention_maskr<r[r\_prepare_position_ids_prepare_token_type_idsr=rn)rrrrhas_past_key_valuesnew_cache_sizes r r|1AssistedCandidateGenerator._update_past_and_masks$s#33778I4PX\\ &__R014FN  ! !"3 4 9 9.:[ \$;%%yr':D'>AF(22 ..r#) rTrPr`r<r=rar@rmrjrqrSrpNN)rr )r,r-r.r/r0r3r4dictrXrrwr5r6r!r7r)r{r1r|r}r~r8r+r#r r:r:Pst8.2<@a##a+a. a  a ||d* a##89aF/(8(8/uUM]M]_d_p_pMpGq/28a53C3C8aUM^M^8amp8at.u/?/?.E#s(O.^_#))#r?r@rc >[T U]XXVXx5 X0lX@lSUlSUlUR RUlUR RUlgN) superrwrrprev_target_ids_lenprev_assistant_idsrPtarget_lookbehindassistant_lookbehind) rrr<rrr=r>r?r@rs r rw6AssistedCandidateGeneratorDifferentTokenizers.__init__xs\ 5FVcv 0#6 /3 ;?!%!A!A!S!S$($D$D$Y$Y!r#c[5n0nUHn[R"U5n[UR 55nXb;aM9UR U5 SnUS- nUSUR S:aUSUR S:at[UR 55nUR U5 XSUS4S:Xa US- nUS- nOO.USUR S:aUSUR S:aMtXsU'M U$)a Calculates the length of the longest diagonal sequence in a given matrix. Args: input_matrix (torch.Tensor): The input matrix. nonzero_idx (torch.Tensor): The indices of the non-zero elements in the matrix. Returns: dict: A dictionary where the keys are the indices of the non-zero elements and the values are the lengths of the longest diagonal sequences starting from those indices. r r)setr3cloner5raddrd) input_matrix nonzero_idxvisiteddiagsidx start_idxtuple_start_idx cur_diag_lens r _get_longest_diag_dictDAssistedCandidateGeneratorDifferentTokenizers._get_longest_diag_dicts%C C(I#I$4$4$67O) KK (L NIA,!3!3A!669Q<,J\J\]^J_;_"' (8(8(:"; O,! il :;q@ A%LNIA,!3!3A!669Q<,J\J\]^J_;_&#J)* r#c[RXR55n[UR 55n[UR 55n[ R"U5nX4nX$nXV4$)z Returns the start index and length of the longest diagonal in the given input. Args: input_matrix (numpy.ndarray): The input matrix. Returns: tuple: A tuple containing the start index and length of the longest diagonal. )rrnonzerolistvalueskeysrargmax)rr diags_values diags_keys best_diagdiag_start_indexdiag_start_lengths r _get_longest_diag_indexEAssistedCandidateGeneratorDifferentTokenizers._get_longest_diag_indexsk>TT ..0 ELLN+ %**,' IIl+ %0(322r#cXR:Hn[R"U5(d[R"U5nUR [ 5nUR 5R5(dg[RU5upEUSU-nUSU-nURSU- R5nUSS2Xh-S24n USS2XfU-24n XU 4$)ay Input: prompt: 2D array of shape (batch_size, prompt_length), represents the original prompt tokens prompt_plus_new_tokens: 2D array of shape (batch_size, prompt_length), represents the suffix of the original prompt, with additional new tokens. Output: discrepancy_length: int, represents the number of tokens that need to be replaced from prompt new_tokens_only: 2D array of shape (batch_size, new_token_length), represents the new tokens that are not in prompt discrepancy_only: 2D array of shape (batch_size, discrepancy_length), represents the new tokens that are in prompt but not in prompt_plus_new_tokens NNNrr N) Tr3 is_tensortensorrMr7anyitemrrrd) promptprompt_plus_new_tokens compare_matcompare_mat_intlongest_locationlongest_diag_lengthnew_token_start_indexdiscrepancy_with_olddiscrepancy_lengthnew_tokens_onlydiscrepancy_onlys r _get_tokens_diag>AssistedCandidateGeneratorDifferentTokenizers._get_tokens_diags 8 88 {++,,{3K%..-""$))++#0]0u0u 1 -!1 36I I/25HH$ll1o0DDJJL04I4^4`1`a1 $?Q'QQ Q "4DDDr#cpURUSSS9nU"USSS9SnURUR5$)z Convert token IDs from one tokenizer to another. Args: input_ids: The input token IDs. source_tokenizer: The source tokenizer. destination_tokenizer: The destination tokenizer. Returns: The converted token IDs. Tskip_special_tokensclean_up_tokenization_spacesptadd_special_tokensreturn_tensorsr)decoderMrI)rrsource_tokenizerdestination_tokenizertextdest_idss r &convert_source_tokens_to_target_tokensTAssistedCandidateGeneratorDifferentTokenizers.convert_source_tokens_to_target_tokenssF &&ydim&n($W[\]hi{{9++,,r#rc [UR5nUS:XaUS4$URURR5nSnUR U5upTXPl[[X0RURS- 5S5nURXT5 URXVU5nURRSS5 URR"S0UDURD6nUR!XR"5n URSUlUR&URS'UR"UlUR$U RS:aUS4$U S4$)rzrNrrKr rDr+)r7rSrMr<rI_prepare_assistant_input_idsrrrrjrdr|r}r`popr_process_assistant_outputsrrrD) rrrrrassistant_input_idsrkrrnew_target_idss r r! >d? "t##r#cdURURS.nSnURGb^URUR:GaCURUR- nUR "USS2US2440UD6nUR SnURSS2U*S24nURXu5upn URn U bUS:azU R SS:agXR S:Xa XSS2U*S24'OGXR S:a5XR S- n U SS2SU *24n XSS2U R S*S24'UnU R SS:a[R"X/SS9n X4$[R"X/SS9n X4$UR "U40UD6n UR SUlX4$)zIConverts target input IDs to assistant input IDs, handling discrepancies.rrrNr rr) rrrrrrrdrr3r) rrconvert_kwargsrstart_index_in_target_windownew_assistant_idsprompt_use_length prompt_userrrr discrepancy_length_diffs r rJAssistedCandidateGeneratorDifferentTokenizers._prepare_assistant_input_ids"s!% 5 5%)%=%=   " " .43K3KdNdNd3d+/+C+CdF\F\+\ ( $ K K!9::;!?M! !2 7 7 : 005F4F4G1GHJDHDYDYE A 1A#'"9"9 *%).>.D.DQ.G!.K)-C-CA-FFGWA0B/B/C,CD+.D.DQ.GG2DG]G]^_G`2`/.A!E^G^F^E^B^._+O_A0@0F0Fq0I/I/K,KL&8O"((+a/*/))5H4Z`b*c'#33 ',ii1D0X^`&a# #33#'"M"Mi"j[i"j '0q'9D $"33r#assistant_sequencesc[[RUR5nURSnX@R - nUR USS2US24URURS9nURSnUSS2U*S24nURX5upn Un U b*U RSS:a[R"X/SS9n O[R"X/SS9n URbU SS2SUR24n U $)z7Processes assistant outputs to obtain target input IDs.r Nrrrr) rr3r4rrdrrrrrrrm) rrrrnum_prev_assistantstart_assistant_look_indexnew_target_ids_from_windowtarget_prompt_use_lengthtarget_prompt_use_target_new_tokens_onlyr s r r HAssistedCandidateGeneratorDifferentTokenizers._process_assistant_outputsPs-"%"2"2D4K4KL/55a8%7:S:S%S"%)%P%P #=#> > ?!55"&"7"7&Q& " $>#C#CA#F %a*B)B)C&CD'+'<'<=N'k$1" ! -%++A.2!&N+SY[!\#YY'SY[\N  % % 1+A/K1K1K/K,KLNr#)rrrrrrr)r,r-r.r/r0r3r4rrXrrw staticmethodrrrrr5r6r!r7rr r8 __classcell__rs@r rrWs^P.2<@Z##Z+Z4 Z 7 Z . ZZ||d*Z##89ZZ(!!F33&EE@-&($(8(8($uUM]M]_d_p_pMpGq($T,4e6F6F,45QVQaQacfQfKg,4\))@E@P@P   r#rc2^\rSrSrSrU4SjrSrSrU=r$)_PruneReindexingLMHeadiraP A class to prune and reindex the language model head. This class prunes the language model head to only include the specified token IDs and reindexes the logits to map back to the original vocabulary. Args: original_lm_head (nn.Module): The original language model head. token_ids (list[int]): The list of token IDs to keep. c>[TU]5 [X5RURR 5Ulgr)rrwr rMweightrJpruned_lm_head)roriginal_lm_headassistant_overlap_token_idsrs r rw_PruneReindexingLMHead.__init__~s6 01A_bb  # # ) ) r#c(URU5nU$rr()r hidden_states pruned_logitss r forward_PruneReindexingLMHead.forwards++M: r#r-) r,r-r.r/r0rwr0r8r"r#s@r r%r%rs  r#r%c~^\rSrSrS\R 4U4SjjrS\RS\R4Sjr Sr U=r $)_MapInputEmbeddingioriginal_embeddingcj>[TU]5 XlURUlX lSUlg)a$ Wraps an existing embedding layer and remaps token IDs before lookup. Args: original_embedding (nn.Embedding): Pre-trained or existing embedding layer. assistant_overlap_token_ids (dict): Mapping from original token IDs to new token IDs. Example: {old_id: new_id} FN)rrwr4r'r*map)rr4r*rs r rw_MapInputEmbedding.__init__s0 "4(// +F(r#rrcUR(a1URUSRS5RS5nO SUlUnURU5$)z Args: input_ids (torch.LongTensor): Tensor of token IDs (batch_size, seq_len). Returns: torch.FloatTensor: Corresponding input embeddings. )rrrT)r6r* unsqueezer4)rr my_input_idss r r0_MapInputEmbedding.forwardsU 88;;Ie'^?Z^C^$ t'' (1 ,++0K38<<;;BBDE**7740 $3#H#H#J !78HJjJj!k$55nE,;,P,P,R)'9:SUuUu'v$-445IJ,@))<243O3O3QSWSoSopq*&' -r#c~UR(a,[UR5S:aSURlggg)a[ Disables the mapping of input ids despite the assistant pruning for the language model head being enabled. This method is required for the first forward pass of `_MapInputEmbedding` where input ids are already in the assistant vocabulary space. By disabling the mapping, it ensures that the input ids are processed correctly without remapping. rFN)rDrrOrUr6rs r unmap_input_ids+AssistantToTargetTranslator.unmap_input_idss6  ' 'C0H0H,IA,M,1D % % )-N 'r#c XURR5nURR5nSnURUSS9Sn[U5S:aURR U5SSnURUSS9Sn[U5S:asURR U5SSnXW:waMUR 5VV s0sH0upUR U5(aURXuS5OUU _M2 nnn [UR55n [R"U S-4UR[S9n 0n UR 5H#upURU5nUcMXU 'XU'M% U RUR 5U 4$s sn nf)N F)rrrr rJ)rH get_vocabrIrconvert_ids_to_tokensrV startswithreplacerrr3fullrBr7rbrMrJ)r target_vocabassistant_vocab space_strtarget_space_idstarget_space_signassistant_space_idsassistant_space_signtokrmax_assistant_indexassistant_to_target_input_idsrM assistant_id target_ids r rK>AssistantToTargetTranslator._get_assistant_to_target_input_idss--779 33==? 11)PU1VWbc  1 $ $ 6 6 L LM] ^_` abc d "&";";IZ_";"`al"m &'!+'+'@'@'V'VWj'klm'nop'q$$<)8(=(=(? ' )@HC #~~.BCC KK(Gl;;Gi8 "9 -//0L0LMOlll#'s7F&rcb[R"URUR:H5S$)zP Get the input ids that are in the assistant vocab but not in the target vocab. r)r3whererLrBrYs r rN3AssistantToTargetTranslator._get_suppress_input_idss*{{4>>$BXBXXYZ[\\r#assistant_candidate_idsc [US5URS- nUS:XaU$USU*S24nUR(aURUnURUn[ R "X&RS54SS9$)a3 Return the target candidate ids that correspond to the assistant candidate ids. Note that we have already the target ids for the prompt and we only need to find the target ids for the new tokens. Moreover, assistant ids of the original prompt does not necessarily appear in _assistant_to_target_input_ids. rr Nr)rrdrDr*rLr3rr9)rr target_input_idsrtnum_new_tokenslast_candidate_idstransformed_slices r get_target_ids*AssistantToTargetTranslator.get_target_ids#s4Q78;N;T;TUV;WW Q # #"9^O> ?&6c;iT=`=`=f=fgh=i;i6i%j "BXY\^tYtBuM#>> ?r#) rJrLrIrOrHr*rDrPrUrMrCNF)r,r-r.r/r0floatrAr2rBr7rr1rwrZrKrrNr3r4rzr6rr8r+r#r r?r?s(!<-L%'s8<(- '3'7' ' ""34 ' "& 'R 2!mF]c] XNSN^N^X   X*%2C2CHYHYr#r?c\rSrSrSr\R "5r\SSSSSS\ S\ S S \ S \ 4 S jj5r \S 5rSrg)AssistantVocabTranslatorCacheiNz Cache for `AssistantToTargetTranslator` instances. The instances are computed at pre-processing time, and this cache allows us to avoid recomputing them. NrrrrCr<rrDrcURRU5nUc#[R"5nX`RU'URU5nUc[ UUUUU5nXvU'U$r)_cacherbweakrefWeakKeyDictionaryr?)clsrrrCr<rDassistant_dictmappings r get_translator,AssistantVocabTranslatorCache.get_translatorVsw(89  !$668N+9JJ' ( $$%89 ?1 #!' G3:. /r#cURVs/sH obMUPM nnUHnURU M URR5H$nUVs/sH obMUPM nnUHnX1 M M& gs snfs snf)z Clean up dead references in the cache. This removes entries where either the target_tokenizer or assistant_tokenizer has been garbage collected. N)rr)rrs dead_keysrs r cleanup%AssistantVocabTranslatorCache.cleanupqsy%(JJ>JSSJ >C 3"jj//1N(6FIF "'!2 ? GsA:A:A?"A?r+r)r,r-r.r/r0rrr classmethodr7rr1r?rrr8r+r#r rrNs  & & (F 8<(- 37  ""34  "&  %4((r#rcV^\rSrSrSrSS\R SSSSS SS S S \S \S\RS-S\ S4U4Sjjjr S\R S\ \R \R44SjrSS\R S\S\4U4SjjjrS\R S\R 4SjrSrU=r$)%UniversalSpeculativeDecodingGeneratoriz `CandidateGenerator` class to be used for Universal Speculative Decoding (USD): speculative decoding with different tokenizers for the assistant and main models. This class generates candidates through the use of a smaller model. Nrr<rrrrr=r r>atm_translatorr?r@rc Z>Xpl[T U] UUUUUUUU 5 SUlSUlg)Nr)_atm_translatorrrw_target_seq_len_with_candidates_prev_assistant_ids) rrr<rrr=r>rr?r@rs r rw.UniversalSpeculativeDecodingGenerator.__init__sB .          56,<@ r#rc XURURR5nURU5upEUR U5upgUS:XaUS4$UR XES9 UR XFU5nSUSlSUSlURRbURRUS'URU5uUl n URRXCUR5n U RSUlURR!U 5n X4$)z[ Simplified version of get_candidates that uses the translator cache for token conversion. rNrTr=r@r)rMr<rIrr{r|r}rgrfrrPr~rrzrdrr) rrrrvr rrkrrassistant_candidate_logitstarget_candidate_idstarget_candidate_logitss r r!4UniversalSpeculativeDecodingGenerator.get_candidatess?%<<(<(<(C(CD040Q0QRb0c-)-)C)CDT)U& Q d? " ##$7#[778K]kl>B+,:GK+,D    1 1 =262F2F2X2XO. /?C?X?XYh?i< "< $33BB 43K3K 0D/I/I"/M,"&"6"6"H"HIc"d#<URc[URURSURR R 5Ul[URURSURR R 5Ul[TU]%XS9$)Nrr) rrr`rdr<r[r\rrr|)rr rrs r r|*>?!  / /! 3 $ 4 4 R R V VWefgWhWmWmWo p   $"33::DW[;O!% 8 8E$!9!!2d2299: !& /@.A-B4K_K_KfKf g   # # +"3 #CCaG.X !#+/+C+CAGYIYHYGYDY+Z("'))T-E-EGX,Y_a"b 1445::4F ,,."C(9!(<$===r#)rrrr`r)r )r,r-r.r/r0r3r4rr?rXrrwr5r6r!r7r1r|rr8r"r#s@r rrs.2<@A##A+A4 A 7 A . AA4A||d*A##89AA8=(8(8=uUM]M]_d_p_pMpGq=@ f%:J:J f^a fjn f f&>U=M=M&>RWRbRb&>&>r#rc\rSrSrSrSS\R S-S\S\S\S\S S \S-4 S jjr S \RS \ \R\R44Sjr S \RS\RS\4SjrSrg)PromptLookupCandidateGeneratoria `CandidateGenerator` class to be used for prompt lookup generation. This class generates candidates by looking up likely continuations in the provided prompt (input_ids) itself. Read the following blog post for more information: https://github.com/apoorvumang/prompt-lookup-decoding Args: eos_token_id (`torch.Tensor`, *optional*): The token id of the end of sequence token. num_output_tokens (`int`, *optional*, defaults to 10): The number of tokens to be output as candidate tokens. max_matching_ngram_size (`int`, *optional*, defaults to 2): The maximum ngram size to be considered for matching in the prompt max_length (`int`, *optional*, defaults to 20): The number of total maximum tokens that can be generated. For decoder-only models that includes the prompt length. Defaults to 20, which is the max length used as default in generation config. logits_processor (`LogitsProcessorList`, *optional*): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. In prompt lookup assisted generation, they are not used to manipulate probabilities, but rather to find forbidden tokens (p = -inf) and block them from being valid candidates. vocab_size (`int`, *optional*): The size of the vocabulary. Required if `logits_processor` is provided. NrUnum_output_tokensmax_matching_ngram_sizerlr@r vocab_sizecX lX0lX@lXlXPlX`lURS::dURS::a [ S5eg)Nrz4Invalid max_matching_ngram_size or num_output_tokens)rrrlrUr@r ValueError)rrUrrrlr@rs r rw'PromptLookupCandidateGenerator.__init__sS"3'>$$( 0$  ' '1 ,0F0F!0KST T1Lr#rrc URup4URUS-:XaUS4$SnSn[[URUS- 5SS5GHnUR SUSS9nUSU*S24n X:HR SS9n U RS S 9Sn U GHn X-n XR-n[XUR5nX:dM3USX24nURbUn[R"X0R4UR[RS 9n[U5HunnURUU5nUSU4nU[!S 5*[R""UR$5R4;aUSUn O/[R&"XSUS-R)S54SS9nM URSS:XaGM3S n[R*"XPR,5n[R"U5nUR/5S:aUSR15nUSUn O U(dGM O U(aUb[3U5S:XaUS4$UR)S5n[R&"X4SS9nUS4$) al Fetches the candidates to be tried for the current input. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) Return: `torch.LongTensor` of shape `(num_candidates, candidate_length)`: The candidate sequences to be tried. r NFrr) dimensionsizesteprrT)as_tuplerHr@)rdrlrrrunfoldallrrr@r3rcrrIfloat32 enumeraterfinforJrr9isinrUnumelrr)rrrbsz input_length chosen_ids match_found ngram_sizewindows ngram_tensorrq match_indicesrrend_idxsequence_with_candidatefake_input_logits candidate_idxnew_candidate_tokenfake_output_logitsfake_candidate_logitsmaskmatch_indices_eosfirst_eos_indexcandidate_input_idss r r!-PromptLookupCandidateGenerator.get_candidates's&OO ??lQ. .d? "  D$@$@,QRBR SUVXZ[J&&!&LG%Q _5L.333:G$OOTO:1=M %, #&<&<<gT__E&!*1i.?+?!@J,,82;/,1JJ //29;K;KSXS`S`-)CLJBW>M+>151F1FG^`q1r.4FqJ]G]4^14%, TiToToHpHtHt8uu-7-G %:?))%.;N]Q=N0O0Y0YZ[0\$]cd;" 7CX&++A.!3$"&K !::j2C2CDD(- d(;%(..014*;A*>*C*C*E%/0@%A [%\{\Dj0C Oq4Hd? " ))!, #ii(?QG"D((r#r$r%cg)r'Nr+r(s r r)8PromptLookupCandidateGenerator.update_candidate_strategys r#)rUr@rlrrr)N rNN)r,r-r.r/r0r3rXr7rrwr4r5r6r!r)r8r+r#r rrs4-1!#'(<@!%UllT)UU"% U  U ##89 U$JU&\)(8(8\)uUM]M]_d_p_pMpGq\)|53C3CUM^M^mpr#rc^\rSrSrSrSS\R SSSSS \S \RS-S \ S 4 U4S jjjr S\R S\ \R \R44U4Sjjr SrU=r$)EarlyExitCandidateGeneratoriaI `CandidateGenerator` class to be used for assisted generation and speculative decoding. This class generates candidates through the use of **the model itself**, exiting early. Can only be used with models that support early exit, e.g., `facebook/layerskip-llama3.2-1B`. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) assistant_model (`PreTrainedModel`): The original model. This model must support early exit (i.e. is trained to compute logits in earlier layers). generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. model_kwargs (`Dict`): The keyword arguments that will be passed to the main model, and are used as base inputs for the assistant model as well. inputs_tensor (`torch.Tensor`, *optional*): The model input tensor. In encoder-decoder models, this is the encoder input. Nrr<rr=r r>r?r@rc >[TU]UUUUUUS9 URRUlSURlg)N)rr<r=r>r?r@)rrwr=assistant_early_exit)rrr<r=r>r?r@rs r rw$EarlyExitCandidateGenerator.__init__sM +/%'-  %)$:$:$O$O!6:3r#rc >[URURR5nURRnUR URl[ TU]U5upVXCRlXV4$r)getattrr<base_model_prefixr[num_hidden_layersrrr!)rrr base_modeloriginal_num_hidden_layersrrrs r r!*EarlyExitCandidateGenerator.get_candidatessnT1143G3G3Y3YZ %/%6%6%H%H".2.G.G +*/'*@*K' .H+..r#)rr)r,r-r.r/r0r3r4rrXrrwr5r6r!r8r"r#s@r rrs:.2<@;##;+;. ;  ; ||d* ;##89;;,/(8(8/uUM]M]_d_p_pMpGq//r#rct^\rSrSr%SrSr\\S'SSS.r\ \ \ 4\S'SS\ RS S S \RS S S\ S\ R S-S\SS\\\-\ R -S-4U4SjjjrS\ RS\ \ \ 4S\S\S\S\\ R\ R.S-44 SjrSrU=r$)*SinglePositionMultiTokenCandidateGeneratoria5 Candidate generator for predicting multiple draft tokens from a single token position using the MTP method. The Multi-Token Prediction (MTP) method defines an assisted candidate generator with an assistant model that autoregressively drafts multiple candidate tokens from a constant `position_ids` value, the last "seen" token, with the goals of reducing assistant model compute, increasing drafted token acceptance rates, and increasing target model throughput. This capability is enabled by three concrete requirements for assistant models: * **Use KV cache sharing techniques for the entire assistant model** (see Gemma 4) to reduce attention computation in the assistant model. This method of cache sharing passes the `key_states` and `value_states` from the main model in a dictionary, allowing the assistant model to skip pre-fill entirely, and reducing attention computations (e.g., `k_proj`, `k_norm`, `v_proj`, and `v_norm` in the Gemma 4 architecture) as applicable to the main and assistant models' architectures. This architectural feature effectively locks the assistant into a constant `position_ids` value. * **Concatenate the embedding and hidden states for the seen last token** from the target model as the input to the assistant. This adaptation allows the assistant to retain rich context about the preceding (possibly drafted) tokens while working from a constant `position_ids` value. For the first token drafted after pre-fill, the last seen token will be the last token from the prompt. For subsequent drafting steps, the last seen token will be the last token generated by the assistant (within a drafting round) or the last token accepted by the target model (between drafting rounds). * **Use cross-attention** to allow Q from the assistant to attend to KV from the main model. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) assistant_model (`PreTrainedModel`): The model to be used for generating candidates. This model should be smaller than the main model. target_model_input_embeddings (`torch.nn.Embedding`): The input embedding table from main model, used to get the embeddings from the last seen token. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. model_kwargs (`dict`): The keyword arguments that will be passed to the main model, and are used as base inputs for the assistant model as well. inputs_tensor (`torch.Tensor`, *optional*): The model input tensor. In encoder-decoder models, this is the encoder input. logits_processor (`LogitsProcessorList`, *optional*): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. eos_token_id (`int` or `list[int]` or `torch.Tensor`, *optional*): The token id of the end of sequence token. If not `None`, only the provided values will be used. If None, values will be inferred from the available `generation_config` and `assistant_generation_config`. Tr)output_hidden_statesreturn_shared_kv_statesmodel_kwargs_overridesNrr<rtarget_model_input_embeddingsr=r r>r?r@rrUc >SURR;a#[SURRS35e[T U]XXEXg5 X0lUGc[ 5n[URR[5(a&URURR5 ON[URR[5(a%URURR5 [URR[5(a&URURR5 ON[URR[5(a%URURR5 U(a,[R "[#U5[R$S9OSUl g[U[R&5(dA[U[5(aU/n[R "U[R$S9Ul gUR%5Ul g)NGemma4AssistantzAExpected assistant_model to be a Gemma4AssistantForCausalLM. Got z This candidate generator requires that the assistant model is able to work from a shared_kv_states dictionary. Currently, only the Gemma4AssistantForCausalLM supports this.r^)rr,rrrwrrrWr=rUrrRr7rrPr3rrrerX) rrr<rr=r>r?r@rUrs r rw3SinglePositionMultiTokenCandidateGenerator.__init__s O$=$=$F$F FSTcTmTmTvTvSw]]  5FVcv-J*   #L$00==xHH##D$:$:$G$GHD22??EE  !7!7!D!DE$::GGRR##D$D$D$Q$QRD<<II3OO  !A!A!N!NOVb T,-?uzz RhlD L%,,77,,, ,~ % \ LD  , 1 1 3D r# model_outputsis_first_iterationn_last_matchesrc U(aUS4$[[UR5URURS- S- 5nUS::aUS4$Ub"[ US5(a[ US5(d [ S5eURSnURn URSn U R5V V s0sH-upXSSS2SS2SU 2SS24U SSS2SS2SU 2SS244_M/ n n n USS2XUS-24nUSS2SS24n [R"URSS- //[RURRS9n[R"URS[R URS9n/n/n[#U5GHnUR%U 5n[R&"UU/SS 9n[R("5 URUUR+S 5UU S S 9nSSS5 WR,R/SS 9n UR0nUR35(aUR5S5n[R6"UUR8R:U 5n UR=[R6"UR5S5[R>"UR,5UR,55 OUR=UR,5 UR=U 5 UR@cGM~[RB"U[RD"U RGS5UR@RIU R555nURK5(dGM O [R&"U[R&"USS 9/SS 9n[R&"USS 9nUU4$s sn n f!,(df  GN=f) z2Generate draft token candidates using the drafter.Nr rr.shared_kv_statesz^`model_outputs` cannot be None, and they need to contain `hiden_states` and `shared_kv_states`rrGrrKF) inputs_embedsrK position_idsr use_cache)&rr7rSrmrdhasattrrr.rrVr3rrer<rIzerosr1rrrno_gradrblogitsrlast_hidden_staterr9rrr= pad_token_idr zeros_likerU logical_orrsqueezerMr)rrr>rrrrrrrcurrent_lengthkv last_token_idrsequence_stoppeddrafted_logitsdrafted_tokensrlast_token_embeddingroutputsstoppedrrs r r!9SinglePositionMultiTokenCandidateGenerator.get_candidates.s d? "S!:!:;T=W=WZcZiZijkZl=lop=pq Q d? "  !=/::=*<==p +8*E*Eb*IIVIgIg#+\l\r\r\t \tTXTUA!Q?N?A-.!Q?N?A5M0NO O\t  .aSTBT1T.TU!!RS&) ||iooa&81&<%=$>ejjY]YmYmYtYtu  ;;yq'9T]TdTde~&A#'#E#Em#T !II';=N&OUWXM.."/#/#3#34D#E!-%5# /!$NN11b19M ' 9 9  ##%%*44Q7 % GT5K5K5X5XZg h %%KK 1 1" 5u7G7G7WY`YgYgh%%gnn5  ! !- 0  ,#(#3#3$JJ}44Q79J9J9M9MmNbNb9cd$ $''))G'L 9eiiA.N"OUVW  99^;...m $!s?4O##O## O2 )rUrr)r,r-r.r/r0rr1r2rrstrrr3r4r<r=rXrr7rrwr r5r6r!r8r"r#s@r rrs=+Z$(D'!%#'.DcN.2<@>B(4##(4+(4(*|| (4 . (4  (4||d*(4##89(4DIo 4t;(4(4T[/##[/38n[/# [/ ! [/  [/ u!2!2T!99 :[/[/r#rr> new_lengthr\rc|U(aSOSnX0;aU$XnXRS- nUS:aUSS2SU24X'Orr\mask_keyrmask_length_diff cross_masknew_masks r rrs,>'CSH#  !D!JJqM1!!%a):*:):&:!;  A !&D--AP`@a2b+cik!l -!"89 a 3=aARBRAR>R3SL/ 0  !!!RS!Q,/66q:JAqQH3899j=SYZ3[L/ 0  < /!"89 a 3=aARBRAR>R3SL/ 0   !!!RS!),33A7GKH3899j=SYZ3[L/ 0 r#cU(aSOSnURU5cU$XnXRS- nUS:aUSS2SU24X'U$US:axS/UR5S- -S/-n[R"XTR UR S9R"U6USSS24-S-n[R"XG/SS 9nXpU'U$) zVExpands or crops the model's position ids for decoding purposes, to the defined lengthdecoder_position_idsrNrrr rG.r) rbrdrr3arangerJrIviewr)r>rr\ position_key positionsposition_length_diff required_dimnext_position_idss r rrs.@)^L %-*I%(;;a%.q2G3G2G/G%H "   !simmo12bT9 LL-__YM]M] ^ c ceq rRS! "   "IIy&D"M%6\" r#c URS5cU$USnUSS2S4RS5nXRS- nUS:aUSS2SU24US'U$US:a/URSU5n[R "USU/SS9US'U$)zXExpands or crops the model's token_type_ids for decoding purposes, to the defined lengthtoken_type_idsNrr rr)rbr9rdrr3r)r>rr"final_token_typetype_length_difftoken_type_copiess r rrs()1""23N%ae,66r:!$8$8$;;!)7;Lr2s $55 .51hh)0A5& & RD/!3D/NX4NXvRYY.55D``F3(3(lt>,Yt>nX%7Xv5/"<5/p{/1K{/|!$sCx.!c!_c!hlmprumuhv!HS#XC]afjknpsksft4$sCx.cdSVX[S[nr#