pyi,SrSSKrSSKrSSKrSSKJr SSKJr SSKJ r SSK J r SSK J r SSKJr SSKJrJr SS KJr SS KJr SS KJr SS KJrJr SS KJrJ r J!r!J"r" SSK#J$r$ SSK%J&r& SSK'J(r( SSK)J*r* SSK+J,r,J-r-J.r.J/r/J0r0J1r1J2r2 SSK3J4r4J5r5J6r6 \6Rn"\85r9Sr:Sr;Sr\,S- r,\\ \!\"S.r?\:\=S.r@\5"\,5"SS\/55rA\ArBg)z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)Iterable)copyfile)Any)is_offline_mode) AddedToken processors)Encoding) Tokenizer)Decoder)BPEUnigram) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer cached_file) SpmConverter)convert_gguf_tokenizer)load_gguf_checkpoint)INIT_TOKENIZER_DOCSTRING BatchEncodingPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategygenerate_merges)PaddingStrategyadd_end_docstringsloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonztokenizer.modelzadded_tokens.jsonu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. )r r WordLevel WordPiece)tokenizer_file vocab_filec)^^\rSrSrSr\rSrSr\ SNSj5r U4Sjr \ S\ 4Sj5r\ S\ 4S j5rSOS \S \S-S\\4S jjrS r\ S5r\ S5r\R,S5r\R,S5rSr\ S\4Sj5rS\\\44Sjr\ S\\\44Sj5r\ S\\\44Sj5r\ S\\\44Sj5r\r \r!S\\\44Sjr"S\ 4Sjr#S\4Sjr$\ S\%4Sj5r&\ S\'4Sj5r(SPS\)S\ S-S \ S-S!\ S"\ S#\ S$\ S%\ S\\\\*4\+\)44S&jjr,S'\S\4S(jr-S)\S\S-4S*jr.SNS+\+\\-S\4S,jjr/SNS-\ S\4S.jjr0SNS/\\+\-S0\ S\\+\-4S1jjr1SQS2\S-\S-S3\ S\+\4S4jjr2S5\3S6\4S7\S8\S9\S-S:\S-4 S;jr5SS\3Rl\4RnSS\ S9\S-S:\S-S?\ S-S\ S-S \ S-S!\ S"\ S#\ S$\ S%\ S@\ S-S\:4(SAjjr;SB\+\S\4SCjr<SRSD\\+\-S0\ SE\ S-S\4SFjjr=SSS \\>R~-SG\\SH4SI\ S-S \S-S\\SH44 SJjjr@STSKjrA\ SUSLj5rBSMrCU=rD$)VTokenizersBackendSa5 Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). NFc "^([U5nURSS5nUb_[RR U5(a;U[ LdSUR ;dU(a[R"U5US'U$UGb[RR U5(Ga[USS9n[R"U5nSSS5 WRS05RS5nUS ;a][U5n[US5n 0U S 'US :Xa/U S 'XS'/US '[R"[R"U55n O[R"U5n U RUS'U R US'U R"US'U R"bU R"US'U R bU R US'URS5n U (awU RSS5S:XaU Sn O[%U [&5(dU /n U H=n U RS5S:XdMSU ;dM"SSKn U R+U S5US' O URS05RS S5nUR,c0[%U[&5(a['[/[0U55nGO0UR,R2S:XaV[%U[&5(a@U(a9[%US[&[045(aUVs/sHn[1U5PM nnOUR,R2S:Xa"[5U5VVs0sH unnUU_M nnnOUR,R2S :XdUR,R2S:XaP[%U[&5(a;[5U5VVs0sH#unn[%U[&5(aUSOUU_M% nnnXS '[7USS5nS URS05;anU(agUR2S :XaWUSS nUVs/sH=n[%U[85(a[1UR;S55O [1U5PM? nnUUS 'U$URS5nURS5nURS 5nURS 5n[%U[85(a`UR=S 5(aJ[RR U5(a&S!S"KJ n U"US#9RCU5uUS 'US 'U$[%U[85(Ga[RR U5(GaUR=S$5(GaxS!S%KJ"n U"U5nURF"UR,40UD6nS!S&KJ$n URUR25nUb#[KUS'5(aURL"S?0UD6n[KUS+5(aURT"S?0UD6nSU;GaU[ LdSUR ;GaURS S5nURS S5nURS,5=(d 0nUbU(aURW5VVs0sH unnUU_M nnnURW5H_unn[YU5n[9U5nURU5n U (dM6U U:wdM>UU;dMFURU 5UU'UUU'Ma [ZR\"UR^UUS-9n!U!bU!US'UR^R`n"U"RbS:a%UReS.U"Rf=(d S/5 U"RhS:a%UReS0U"Rj=(d S15 U"RlS:a%UReS2U"Rn=(d S35 U$UcC[%U[85(a.[RR U5(a UUS 'US nUcC[%U[85(a.[RR U5(a UUS 'US nUcUR,bUR,R2S :Xa[%U[5(akS:[t[vS;[&[84U(4S<jjm(/S=Qn%[y5n&U%H&n'U'U;dM U&R{T("UU'/55 M( [}UU&S>9nUUS 'U$!,(df  GN~=fs snfs snnfs snnfs snf![Na1nRSS(UR2S)US*35 SnAGNSnAff=fs snnf![NaVnRSS4US5US635 S!S7KJ8n# U#"UURS85S99n$U$Rs5US'SnAU$SnAff=f)@z Build a `tokenizers.Tokenizer` backend from the available serialization files (tokenizer.json, sentencepiece models, tekken.json, vocab/merges). r%N__init__tokenizer_objectutf-8encodingmodeltype)Nrvocabr merges added_tokenspost_processortokenizer_paddingtokenizer_truncation_json_truncation _json_padding normalizerSequence normalizers Precompiledprecompiled_charsmapr_spm_precompiled_charsmaprr#r$ r& merges_filez tekken.jsonr)MistralConverter)r&.model)SentencePieceExtractor)SLOW_TO_FAST_CONVERTERSconvert_from_spmz,Could not reorder vocab using converter for z due to z/. Falling back to raw SentencePiece extraction.convert_from_spm_modeladded_tokens_decoder)protor2r3 bos_token eos_token unk_tokenzz+Could not extract SentencePiece model from z$ using sentencepiece library due to z%. Falling back to TikToken extractor.)TikTokenConverterextra_special_tokens)r&rPvaluesreturnc>/nUHVnUcM[U[[45(aURT"U55 M<UR [ U55 MX U$N) isinstancelisttupleextendappendstr)rQ collectedval_iter_special_tokenss g/home/mande/repo/quber/.venv/lib/python3.13/site-packages/transformers/tokenization_utils_tokenizers.pyr]HTokenizersBackend.convert_to_native_format.._iter_special_tokens)sY') !C{ !#e}55!(()=c)BC!((S2 "! ) pad_tokenrNrJrL sep_token cls_token mask_tokenadditional_special_tokensrP) skip_tokens)?dictpopospathisfiler(__dict__ TokenizerFast from_fileopenjsonloadgetfrom_strdumpsr5padding truncationrUrVbase64 b64decoder0maprW__name__ enumerategetattrrZsplitendswithconvert_slow_tokenizerrBextract_vocab_merges_from_modelrDextractrEhasattrrF ExceptionloggerwarningrGitemsintrbuild_tokenizer_from_spm_protorI trainer_specbos_id setdefault bos_pieceeos_id eos_pieceunk_id unk_piecerO convertedrrsetupdater))clstrust_remote_codekwargs local_kwargsfast_tokenizer_filetokenizer_handletokenizer_json model_typeminimal_tokenizer_json minimal_model tok_from_filenormalizer_configr:rxr2itemitokenr3merger&rArBrD extractorrEconverter_classerHtoken_id id_to_token new_token current_tokenr, proto_specrO converterspecial_tokens_keysrfkeyr]s) @r^convert_to_native_format*TokenizersBackend.convert_to_native_formatesH F| *../?F +233))Zs||-KO`/CY]gCg%DJDTDT&'=>E %@A#4#&&w377FEyy eT** UE!23E##y0eT**uE!HtUZm9\9\5:;UTU4[UE;##{22;E2BC2Bha2BC##u, 0B0Bk0QeT**_hin_op_oS[STV[E4)@)@U1XeQN_oEp$) ! gt4J>--gr:: zObObfkOk'0:kqrkqbgZs5K5K% C 01QVW\Q]]kqr)/ X& !%%l3 "&&}5   )!!(+ j# & &:+>+>}+M+MRTRYRYR`R`akRlRl @>*+E+E*J]J]^fJgJgF IJ3:> (00KlK  O&=&A&A#,,&OO&2wPb7c7c'6'G'G'W,'W 3 899#&#=#=#M #ML &\9,, #,,0N(,,Wd;E)--h=F,8+;+;jc::rww~~k?Z?Z%0L "!(+F >cii3 8J8Je8SXbchjnXoXo !Xc] !tCy ! # %(EK*,&&&';\#=Nf.f.C:f. e* f%4&f f. f%% f.. h8A h  hc <>URSS5nURSS5nURSS5 URSS5nURSS5nURSS5nURS05nURSS 5n URS 5n URS 5n URS 5n Sn Ub[R"U5n GOUb<[R R U5(a[R"U5n GOUb|[URS S5U40UD6n[U5nUSSnUSnUSn[UU5un nURU5 [U5S:aURU5 GOURcU bU bT[U [ 5(aU O&[#U 5VVVs0sH ununnUU_M snnnn[[%UU SSS95n O[U [ 5(a[[%U /SSS95n Ou[U [&5(aGU (a@[U S[([&45(a"[[+XRSS5S95n OURc [-S5eUc4Uc1URc$UR/SS5 UR/SS5 U bXlURc [-S5eURSS5=(d URR0=(d UnUbqURR2"S80UD6 UR/SUS5 UR/S US!5 UR/S"US"5 UR/S#US$5 OURR55 URS%S5=(d URR6=(d UnUbURR8"S80UD6 UR/S&US&5 UR/S'US(5 UR/S)US!5 UR/SUS*5 UR/S+US+5 S,U;aS-US,'S.U;=(d S/U;nURS.S 5UlURS/S 5UlURS0S5=n(aUURlU=(d URR>SLUl [BT&U]"S80UD6 U bXl#Xl$URJURl&URNVs1sHn[Q[SU55iM nn[UURW5S1S29VVs/sH"unn[Q[SU55U;dM UPM$ nnn['URXR[55UVs/sHn[]U5PM sn-n UR^Ra5H2n!U!cM[]U!5U ;dMU!U;dM!URcU!5 M4 URdH-n[]U5U ;dMUU;dMURcU5 M/ [U5S:a/n"UR^Ra5V#s/sHn#U#(dM []U#5PM n$n#UHpn[U[\5(a [gUSS39nO<[U[f5(a'URh(d[]U5U$;aSUl4U"RcU5 Mr U"(aURkU"5 URRm5n%U%S4:a[qURS5S5blURSS5 URr"URURtRS S54URtURS6S5S7.UD6UlUR@=(d URR>SLUl UR@(aURw5 ggs snnnfs snfs snnfs snfs sn#f![na Sn%GNf=f)9Nr8r9r?r, gguf_filer%rHadd_prefix_spaceFr&r2r3 name_or_pathconfigr tokenizertokenizer_configrT)r2r3fuse_unkdropoutr)r2ra9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one.rJrKrLrMz3The backend tokenizer is not correctly initialized.r7 max_lengthtruncation_side directionstridetruncation_strategystrategyr6rapad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofbackend tokenizers add_bos_token add_eos_tokenr5c US$Nrrg)xs r^,TokenizersBackend.__init__..s STUVSWr`r)speciali pre_tokenizerfix_mistral_regex) init_kwargsrrg)p$//B\B\p`p  " OO - - < <   lK ,E F   / J   h H(= >   3[5L M OO ) ) +::148dDOO ?   2H=Q4R S F " ,F9 %4%>%[/U[B["$jj%@$jj%@#ZZ(8$? ?> ?-;DOO *-G-q4??KiKimqKq* "6"  !(O 0040I0I-DHD]D]$^D]5T$u+%6D]!$^!'';'A'A'C X X uDK (AA  X  t005578Ta;bTa5CJTa;bb$(#;#;#B#B#D "*&'w6;NVc;c$$%89 $E//E5z(U--G$$U+0 }  !F040H0H0O0O0QW0Q1UVA0Q W&eS))&ud;Ez22 ==SZ;K-K(,  e$'' 779J  74??OT#R#^ JJ{D )"77  $$^T:!,,"(**-@$"G   DO  . . X$//2P2PTX2X *  - -  & & ( .oDl~%_ .j dhijdkr`rrrHrcontentrvks r^r&TokenizersBackend.added_tokens_encoderds? *00I0I0O0O0QWk)lm)l 1 )lmmmAc6URR5$)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `dict[str, int]`: The added tokens. )rget_added_tokens_decoderr s r^rH&TokenizersBackend.added_tokens_decoderls7799r`c[URR5SS9VVs0sHupURU_M snn$s snnf)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `dict[str, int]`: The added tokens. c US$rrgrEs r^r3TokenizersBackend.get_added_vocab..rGr`rrHrJs r^get_added_vocab!TokenizersBackend.get_added_vocab{s?*00I0I0O0O0QWk)lm)l 1 )lmmmrNcg)z> Returns True, to avoid expensive `assert tokenizer` gotchas. Trgr s r^__bool__TokenizersBackend.__bool__sr`c4URRSS9$)z4 Size of the full vocabulary with the added tokens. Tr:r<r s r^__len__TokenizersBackend.__len__s---EEr`cUR$)zS `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rr s r^backend_tokenizer#TokenizersBackend.backend_tokenizers r`c.URR$)zE `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rdecoderr s r^raTokenizersBackend.decoders &&&r`Tr/return_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec UcSUR;nUcSUR;nU(aURbU/UR-n OU/n [[5n U Hn U SR U R 5 U(aU SR U R 5 U(aU SR U R5 U(aU SR U R5 U(aU SR U R5 U(dMU SR [U R 55 M X4$)ar Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) token_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingr) model_input_names overflowingrrVrYidstype_idsrlrnoffsetsr) rr/rcrdrerfrgrhri encodings encoding_dictrs r^_convert_encoding#TokenizersBackend._convert_encodings( ! ($48N8N$N ! ($48N8N$N ! $)=)=)I! X%9%99I! I#D) A + & - -aee 4$./66qzzB$./66q7G7GH)34;;A This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rnum_special_tokens_to_add)rrs r^r+TokenizersBackend.num_special_tokens_to_adds&88>>r`rrskip_special_tokensc@[U[5(aURRU5$/nU(a[ UR 5O [ 5nUH?n[U5nXT;aMUR URRU55 MA U$)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `list[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `list[str]`: The decoded token(s). )rUrrrrall_special_idsrY)rrrrr ids_to_skiprs r^convert_ids_to_tokens'TokenizersBackend.convert_ids_to_tokenss c3  ??..s3 33Fc$../CE EJE# MM$//55e< =   r`textrc HUR"SXUS.UD6R5$)N)r text_pairrrg) _encode_plusr)rrrrrs r^tokenizeTokenizersBackend.tokenize s(  ldOaleklssuur`padding_strategyrrrrrcURRnURRnU[R:XaUbURR 5 OdUUUR URS.n UcSn O"U V s0sHoURU S5_M n n X:waURR"S0U D6 U[R:XaUbURR5 ggU[R:XaUOSn U UbUO URURUR UR"US.n X:waURR$"S0U D6 ggs sn f)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)rrrr)rrpad_idrarrrg)rrwrvrDO_NOT_TRUNCATErr1rrsrr DO_NOT_PAD no_padding MAX_LENGTHr pad_token_idrarr) rrrrrrrrrtargetcurrentrLrs r^set_truncation_and_padding,TokenizersBackend.set_truncation_and_paddingsRBoo00 ??** "4"D"D D&--/) /55!11 F"@FG1kooa66G 11;F; 99 9#**,$$47Q7Q#QZW[F -9-E\4K\K\++!^^#55&8 F!..88"%HsE'rris_split_into_wordsreturn_tensorsrc vSnU"U5(d [S5eUbU"U5(d [S5eU(a@[U[[45=(a" U=(a [US[[45nO[U[[45nU(ay[U[5(a [ S5eUb<[ U5[ U5:wa$[S[ U5S[ U5S35eUb[[X55OUnOU(aX4/OU/n[U[[45(d[ S[U5S 35eURUUUUU U S 9 Uc URnURRU:waUURl URRUUUS 9nUVs/sHnURUU U UUUUUS 9PM nn0nUSSH.nUVVVs/sHunnUUHnUPM M n nnnU UU'M0 UVVVs/sHunnUHnUPM M n!nnnU(a4/n"[U5Hun#un$nU"U#/[ U$S 5-- n"M U"US'US Hn%UR!U%UU5 M [#UU!U S9n&U(dqU cnU(dg[#U&R%5VV's0sH5unn'U[ U'5S:a[U'S[5(aU'SOU'_M7 sn'nU&R&5n&U&$s snfs snnnfs snnnfs sn'nf)Nc[U[5(ag[U[[45(a[ U5S:Xag[US[5(ag[US[[45(a[ US5S:Xd[USS[5(ag[USS[[45(a4[ USS5S:H=(d [USSS[5$ggg)NTrF)rUrZrVrWr)rs r^_is_valid_text_input._is_valid_text_inputqs!S!!Ae}--q6Q;!c**!tUm441Q4yA~AaDGS)A)A##AaDGdE];;"1Q47|q0OJqtAwqz34OO$ r`ztext input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) or `list[list[str]]` (batch of pretokenized examples) or `list[tuple[list[str], list[str]]]` (batch of pretokenized sequence pairs).rzdwhen tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`.zbatch length of `text`: z- does not match batch length of `text_pair`: .z:batch_text_or_text_pairs has to be a list or a tuple (got ))rrrrrr)ris_pretokenized)r/rcrdrerfrgrhrirmoverflow_to_sample_mapping) tensor_type)rrUrVrWrZ TypeErrorrzipr1rrrr encode_batchrwr|&_eventual_warn_about_too_long_sequencerrru)(rrrrrrrrrrrrrcrdrerfrgrhrirrr is_batchedbatch_text_or_text_pairsrur/tokens_and_encodingssanitized_tokensrrrrstacksanitized_encodingsrrtoksrmbatched_outputr1s( r^rTokenizersBackend._encode_plusYs0 ($D))W   )=i)H)HW  #D4-8hThjQUVWQX[_afZgFhJ#D4-8J )S))$Tc)n)D .s4yk:I'q*FOEZtC,@'A`d $?H(9':dV $2UDMBBLTRjMkLllmn  ''- 3!1% (  '#'#<#< ?? 0 04H H4HDOO 1OO00 $1/1 $& &  " "!&;&;*C+E'=+ # &  '*1-C&:N&:74DIqQIQ&:EN$) S !.1ES0DWQdqdq0DS %)+ & )*> ? 9D!*qcC[8I4J.JJ*!@=W 9 :)+6I  7 7 :w W7''79LZhin4=V*'5&:&:&<&< Uc%j1nE!Hd9S9S%(Y^^&<(( NW "OS"s L"7L' &L.[SUSS55(a![RR5US'[ USSnU"S(X-S.UD6nU R#XUS9 U Gb@[R"U R55nSU ;aU SHnU SUSnUb"UVs/sHnUR%UU5PM nnUU SUS'UH"nU R'U5nUbM[ S 5e UVs/sHnU R'U5PM snU SUS!'M S"HGnUU ;dM U UunnUb UU;aUUnU R'U5nUc [ S 5eUU/U U'MI U US'[ R"[R"U55n UR(R+5n[,R.Hn[1UU5cM[1UU5nUb UU;aUUnUR2R%US5n[5U[5(a;[UUR6UR8UR:UR<S#S$9UU'MUUU'M UR>(aUR>R+5O/nUbURU5 [AU5S :aUUS%'XS&'URB"S(0UD6$s snfs snf![DaGnS'[GU5;a2UR S&S5 URB"S(0UD6nU UlUsSnA$eSnAff=f))u Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `list[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. r4r5Nr0r1r r2r3rrrg)r#r$z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece.rNridrIcontinuing_subword_prefixend_of_word_suffixr ByteLevelr; pretokenizersc32# UH nUSS:Hv M g7f)r1rNrg).0 pretokenizers r^ .s!(X !(K7(Xsinitial_alphabet)rr)rtrainerrrzQAttempted to set a token in the post processor that does not exist in the mappingrr)rsepT) single_wordlstriprstripr5rrPr,z7multiple values for keyword argument 'tokenizer_object'rg)$rqloadsrto_strrirrnrtrurYrrXanypre_tokenizers_fastralphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrsrzrrrSPECIAL_TOKENS_ATTRIBUTESr}rrUrrrr5rPrrrrZ)r text_iteratorrrnew_special_tokensspecial_tokens_maprrr4r5rNrrr added_tokenrr trainer_classrtrained_tokenizer_jsonrrrr special_tokenspecial_token_fullrPr new_tokenizers r^train_new_from_iterator)TokenizersBackend.train_new_from_iterator6sDDOO$:$:$<=%)).9 '++,<= ' "6 *e 3/1N7 #G ,02N7 #H - G $V , 9g&x0<'0:*73G4D3Ew'0 G $V ,0J J/1N7 #G ,Mn]dNeflNmMno>>   *~g66w' 48JJ3EU\F]^iFj3kN7 #K 0!**4::n+EF 'K!ooi6Gd+Ag&v.);G!-+i2HL^2^); 1)*:;C+,<=cB8LF)5TZ![TZ5"4"8"8"FTZ![FLN#34S9(C!'#,#8#8#?#+", s#"(ouCuntejIDYDYZ_D`ntCuN#34S9%@<"0  N2-m>+F+ +{"\Dvj HCPQFR -t4 $ 8 8 +4 ($$ s04T' T,T11 V;;U=6V<U==Vc ~^^SSKmSSKJn SSKJm SSKJn SSKJn U "SS9S [S [4UU4S jj5nU(d[5(aS nUGbRU(dU(GdCU"U5(Ga5U "US UUUSSUS9nSnUb[USS9n[R"U5nSSS5 WRS5nURS5nU(a7U R!U5U R!S5:aU(a UbUS;aU$O-U(a&U R!U5U R!S5:aU$S nU(dU(Gd\U"U5(GaNU(aSU;a[#USUS5 U c:[%USS5(d([#USS5 [&R)SUS35 U$U S Ld[%USS5(a[#USS 5 SSKnUR,R/UR1S5SS9nUR2n[5UUR,R65(aUUR2S'U$[5UUR,R85(aUR,R;SSS9nUR,R7UU/5UlU$!,(df  GN =f)a6 Patches mistral related tokenizers with incorrect regex if detected 1) Local file with an associated config saved next to it >> Model type one of the mistral models (on older versions) 2) Remote models on the hub from official mistral models >> Tags including `base_model:.*mistralai` rN) lru_cache) model_info)versionr)maxsizemodel_idrRc>T"U5nURb1TRSSRUR55(agg![a gf=f)NFzbase_model:.*mistralairT)rtagssearchr)rr0rres r^is_base_mistral?TokenizersBackend._patch_mistral_regex..is_base_mistralsW "8,zz%995rwwuzz7JKK   sA AATz config.jsonF) cache_dirrlocal_files_only%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errors _commit_hashr-r.transformers_versionrz5.0.0)mistralmistral3voxtral ministralpixtralrz$The tokenizer you are loading from 'a' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.z[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+isolated)patternbehavior)r use_regex)r functoolsrhuggingface_hubr packagingrtransformers.utils.hubrrZboolrrprqrrrsparsesetattrr}rrrpre_tokenizersSplitRegexrrUr; Metaspacer)rrpretrained_model_name_or_pathrrrris_localrrrrrrr _config_filemistral_config_detectedf_configrtransformers_model_typersplit_pretokenizercurrent_pretokenizerrrs @@r^r&TokenizersBackend._patch_mistral_regexs* '.%6 3  c d    00H ( 4 X/:W*X*X&-#!16;8=) L', #',9Q"iilG:'.{{3I'J$*1++l*C' (GMM:N,ORYR_R_`gRh,h 3?3  )()gmm?\>]^eeH?'$.')EXZ_2`2`I':DA%)3)B)B)H)H * 0 0s!", *I*& ,5+B+B(!"6 8Q8Q8Z8Z[[5G //2"&&:JA tCH~  nd38nnn:d3 ?&;::10nc3hn$ FF = '''.2-1*/+0',#-(-( $d{-( $d{ -( $( -( %) -(!%-(-(-( tCH~tL11 2-(^ 7#7#*76d3+;&<6WZ6 ?d?s?*tCyt`cfjknfo`o4vSvd vtvjnorjsvI9)I90I9 I9  I9  $J I9DjI9\gk#',;,F,F2D2T2T!%$))-#'&*-1-1*/+0',#,0)X++d9o=EV@WWX004 ?BTJ[E\\_ccX! X * X 0 X$JXX"X $JXDjXt X $d{X $d{X$(X %)!X"!%#X$%X&'X(#Tk)X, -Xt tCy S %*48 )c?)")'+Tk ) )^&*&* bkk)#s(Od{  t  sCx * CJ CCr`r()Cr%rrqrj collectionsrcollections.abcrshutilrtypingrtokenizers.pre_tokenizersrrrrrrr r r,r rntokenizers.decodersr r+tokenizers.modelsr rtokenizers.trainersrrrrrrrrintegrations.ggmlrmodeling_gguf_pytorch_utilsrtokenization_utils_baserrrrrrrutilsr r!r" get_loggerr{rrSPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILETIKTOKEN_VOCAB_FILEADDED_TOKENS_FILErrr(PreTrainedTokenizerFastrgr`r^rBs  #$7+-/16*^^.05=@?   H %"3/'( !! (6EXY,-k/k.k^),r`