pyi:SSKrSSKrSSKJrJr SSKJr SSKJ r SSK J r SSK r SSKJr SSKJr \ r\\\/\\\ 44r"S S 5rS rS*S \\S \\\ 44Sjjr\"SS\55rS \\S \\\ 44SjrS \\S \\\ 44Sjr\"SS55r\"SS\55rS+S\S-4Sjjr S+S\S-4Sjjr!\"SS\55r"\"SS55r#\"SS\55r$\"S S!\$55r%S \\ 4S"jr&S \ RN\ 4S#jr(\"S$S%\$55r)\"S&S'\55r*\"S(S)\55r+g),N)CallableMapping) dataclass)randint)Any)PreTrainedTokenizerBase)PaddingStrategyc,\rSrSrSS\S-4SjjrSrg)DataCollatorMixin%Nreturn_tensorscUc URnUS:XaURU5$US:XaURU5$[SUS35e)Nptnpz Framework 'z' not recognized!)r torch_call numpy_call ValueErrorselffeaturesrs \/home/mande/repo/quber/.venv/lib/python3.13/site-packages/transformers/data/data_collator.py__call__DataCollatorMixin.__call__&sW  !!00N T !??8, , t #??8, ,{>*::KLM MN)__name__ __module__ __qualname____firstlineno__strr__static_attributes__rrrr r %sNtNNrr c [US5(dUR"U0UD6$URRSS5nSURS'UR"U0UD6nX0RS'U$!X0RS'f=f)zr Pads without triggering the warning about how using the pad function is sub-optimal when using a fast tokenizer. deprecation_warningszAsking-to-pad-a-fast-tokenizerFT)hasattrpadr%get) tokenizerpad_args pad_kwargs warning_statepaddeds r"pad_without_fast_tokenizer_warningr.1s 94 5 5}}h5*5522667WY^_MGKI""#CDY7J7LY&&'GH MLY&&'GHs A22BrreturncHUS:Xa [U5$US:Xa [U5$g)a Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. rrN)torch_default_data_collatornumpy_default_data_collator)rrs rdefault_data_collatorr3Gs/"*844 4 *844 rcX\rSrSr%SrSr\\S'S S\\ \\ 4S\ \\ 44Sjjr S r g) DefaultDataCollator^a Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. This is an object (like other data collators) rather than a pure function like default_data_collator. This can be helpful if you need to set a return_tensors value at initialization. Args: return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". rrNrr/c6Uc URn[X5$r)rr3rs rrDefaultDataCollator.__call__ts  !!00N$X>>rrr) rrr r!__doc__rr"__annotations__listdictrrr#rrrr5r5^sC$NC?d38n!5?tTWY\T\~??rr5c JSSKn[US[5(dUVs/sHn[U5PM nnUSn0nSU;aUSb[USUR5(aUSR 5OUSn[U[ 5(a URO URnURUVs/sHo"SPM snUS9US'OSU;aUSb[USUR5(a(URUVs/sHo"SPM sn5US'OY[USS[ 5(a URO URnURUVs/sHo"SPM snUS9US'UR5HupxUS;dM UcM[U[5(aM)[XR5(a(URUVs/sHo"UPM sn5XG'Mk[U[R5(a<UR[R"UVs/sHo"UPM sn55XG'MURUVs/sHo"UPM sn5XG'M U$s snfs snfs snfs snfs snfs snfs snfNrlabeldtypelabels label_ids)r?rC)torch isinstancervarsTensoritemintlongfloattensorstackitemsr"rndarray from_numpy) rrDffirstbatchr?rAkvs rr1r1zs? hqk7 + +%-.XDGX. QKE E %E'N6)3E'NELL)Q)Qg##%W\]dWe(44 %++,,H'EHq' H'EU,Sh  % "4"@ eK(%,, 7 7#kk8*L8a[>8*LME(O",U;-?-BC"H"HEJJekkE#llH+MHqkNH+MUZl[E(O  * *q}ZPQSVEWEW!\\** ;;h'?h!h'?@Arzz** ++BHHH5MHqdH5M,NO <) rErrFrrOrHrIint64float32arrayrMrNr")rrQrRrSr?rArTrUs rr2r2s hqk7 + +%-.XDGX. QKE E %E'N6)3E'NBJJ)O)Og##%UZ[bUc&uc22 ((#A1gJ#AOh  % "4"@ eK("** 5 5 hh'I1+'IJE(O *5+=a+@# F FBHHBJJE hh'I1+'IQVWE(O  * *q}ZPQSVEWEW!RZZ((888$<8aqT8$<=888$<8aqT8$<=  L7/$B(J(J%=$= 7.0 (Volta). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". r)TpaddingN max_lengthpad_to_multiple_ofrrrr/c [URUURURURUR S9nSU;a USUS'US SU;a USUS'US U$)Nr]r^r_rr?rBrC)r.r)r]r^r_r)rrrSs rr DataCollatorWithPadding.__call__sw2 NN LL#66..   e #GnE(Og % #K0E(Ok" rr)rrr r!r9r r:r]boolr"r r^rIr_rr;r<rrr#rrrr[r[sq4'&,0GTCZ/ )0!Jd !%)d )NCd38n!5$sCx.rr[c\rSrSr%Sr\\S'Sr\\ -\ -\S'Sr \ S-\S'Sr \ S-\S'S r\ \S 'S r\ \S 'S rSrSrg)"DataCollatorForTokenClassificationa! Data collator that will dynamically pad the inputs received, as well as the labels. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.0 (Volta). label_pad_token_id (`int`, *optional*, defaults to -100): The id to use when padding the labels (-100 will be automatically ignore by PyTorch loss functions). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". r)Tr]Nr^r_label_pad_token_idrrc ^ SSKm SUS;aSOSnX!S;aUVs/sHo3UPM snOSnUVVVs/sH.o3R5VVs0sHupVXR:wdM XV_M snnPM0 nnnn[URUURUR UR SS9nUcU$USRSn URRn U 4Sjn U S :Xa8UV s/sH'o"U 5UR/U [U 5- --PM) sn X'O7UV s/sH'oR/U [U 5- -U "U 5-PM) sn X'T RXT RS 9X'U$s snfs snnfs snnnfs sn fs sn f) Nrr?rBrra input_idscp>[UTR5(aUR5$[U5$r)rErGtolistr;)tensor_or_iterablerDs rto_list>DataCollatorForTokenClassification.torch_call..to_list.s0,ell;;)0022*+ +rrightr@) rDrNr.r)r]r^r_shape padding_siderhlenrLrW)rr label_namefeaturerBrTrUno_labels_featuresrSsequence_lengthrsror?rDs @rr-DataCollatorForTokenClassification.torch_calls '8A; 6WH BLYZP[B[X>X'*%X>aecklckX_ S1?tqtSckl2 NN LL#66   >L ,2215~~22  , 7 "io!io`e$"9"9!:oPSTYPZ>Z![[io!E  jp!io`e(()_s5z-IJWUZ^[io!E "LL):%++LN E?Sl.!!s.E/E: E4E4%E:$.F.F4E:c SUS;aSOSnX!S;aUVs/sHo3UPM snOSn[URUURURURUcSOSS9nUcU$[ R "US5RSnURRnUS:Xa=UVs/sH+n[U5UR/U[U5- --PM- snUS'O;UVs/sH*oR/U[U5- -[U5-PM, snUS'UR5V V s0sH)upU [ R "U [ RS 9_M+ nn n U$s snfs snfs snfs sn n f) Nr?rrBrrarjrkrqr@)r.r)r]r^r_rrYrrrsr;rhrtrNrW) rrrurvrBrSrxrsr?rTrUs rr-DataCollatorForTokenClassification.numpy_call?s} '8A; 6WH BLYZP[B[X>X'*%X>ae2 NN LL#66#)>4t  >L((5#56<r)rrr r!r9r r:r]rcr"r r^rIr_rhrrrr#rrrreres_8'&,0GTCZ/ )0!Jd !%)d )""NC&Prrer_c2^ SSKn[US[[[R 45(a(UVs/sHoCR XCRS9PM nnUSRS5m [U 4SjU55nU(a6Ub T U-S:Xa*[XR5(dURUSS9$URc#[SURRS35e[!SU55nUbXb-S:wa Xb-S -U-nUSR#[%U5U/UR&5n[)U5HAupUR*S :XaXUSU R,S24'M,XXR,S*S24'MC U$s snf) _Collate `examples` into a batch, using the information in `tokenizer` for padding if necessary.rNr@c3J># UHoRS5T:Hv M g7frNsize.0xlength_of_firsts r '_torch_collate_batch..js!Q1&&)">s #)dimCYou are attempting to pad samples but the tokenizer you are using () does not have a pad token.c3B# UHoRS5v M g7frrrrs rrrws11VVAYYsrkrq)rDrEr;tuplerrOrLrJrallrGrM pad_tokenr __class__rmaxnew_fullrt pad_token_id enumeratersrr) examplesr)r_rDeare_tensors_same_lengthr^resultiexamplers @r_torch_collate_batchr^s(1+eRZZ899?GHx!LL**L5xHqk&&q)O"!Q!QQ$6$>/TfBfjkBk(LL11;;xQ;/ /" $$--..J L  111J%:+Ja+O!71<@RR a[ ! !3x=*"=y?U?U VF)   ! !W ,,31( a((( )-41}}Q'')) * * M9Is"Fc^ [US[[45(a3UVs/sH&n[R"U[R S9PM( nn[ US5m [U 4SjU55nU(a!Ub T U-S:Xa[R"USS9$URc#[SURRS35e[SU55nUbXR-S:wa XR-S -U-n[R"[ U5U4URUSR S 9n[#U5HAupxUR$S :XaXUSUR&S24'M,XXxR&S*S24'MC U$s snf) r}rr@c3@># UHn[U5T:Hv M g7frrtrs rr'_numpy_collate_batch..s!NX#a&O";XsNaxisrrc38# UHn[U5v M g7frrrs rrrs.XSVVXrk)rr fill_valuerArq)rEr;rrrYrWrtrrMrrrrrfullrrArrsrr) rr)r_rrr^rrrrs @r_numpy_collate_batchrs(1+e}--9ABABHHQbhh/B(1+&O!!NX!NN$6$>/TfBfjkBkxxq))" $$--..J L  .X..J%:+Ja+O!71<@RR WWCM:69CYCYaijkalarar sF)   ! !W ,,31( a((( )-41}}Q'')) * * M3Cs-E?c\rSrSr%Sr\\S'Sr\\ -\ -\S'Sr \ S-\S'Sr \ S-\S'S r\ \S 'S \\\ \44S jrS rg)DataCollatorForMultipleChoiceia Data collator that dynamically pads a batch of nested examples for multiple choice, so that all choices of all examples have the same length. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences according to the model's padding side and padding index among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): Pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". r)Tr]Nr^r_rrrc ^ SSKnSUS;aSOSnUVs/sHoDRU5PM nn[U5n[USS5m [U 4SjU5/S9nURR UUR URURSS9nUR5V V s0sHupXRUT S 5_M nn n URXRRS 9US'U$s snfs sn n f) Nrr?rBrjc 3># UHJn[T5VVVs/sH*o!R5VVs0sH up4X4U_M snnPM, snnnv ML gs snnfs snnnf7fr)rangerN)rrrrTrU num_choicess rr;DataCollatorForMultipleChoice.torch_call..sN i`hU\U;=O P=O==?3?41qA$w?3=O P P`h3 Ps'A"A AA A"A A")startrrar@) rDpoprtsumr)r'r]r^r_rNviewrLrW) rrrDrurrB batch_size flat_examplesrSrTrUrs @rr(DataCollatorForMultipleChoice.torch_calls!(8A; 6WH 9ABg++j)B] (1+k23  i`h iqs "" LL#66 # EJKKMRMDAFF:{B77MR,,v[[,Ah /C*Ss C11C6r)rrr r!r9r r:r]rcr"r r^rIr_rr;r<rrr#rrrrrsb8'&,0GTCZ/ )0!Jd !%)d )NC4S#X#7rrc\rSrSr%Sr\\S'Sr\S-\S'Sr \ \ -\ -\S'Sr \S-\S'Sr\S-\S 'S r\\S 'S r\ \S 'SSjrSrg)DataCollatorForSeq2Seqia< Data collator that will dynamically pad the inputs received, as well as the labels. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. model ([`PreTrainedModel`], *optional*): The model that is being trained. If set and has the *prepare_decoder_input_ids_from_labels*, use it to prepare the *decoder_input_ids* This is useful when using *label_smoothing* to avoid calculating loss twice. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.0 (Volta). label_pad_token_id (`int`, *optional*, defaults to -100): The id to use when padding the labels (-100 will be automatically ignored by PyTorch loss functions). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". r)NmodelTr]r^r_rgrhrrc .Uc URnSUS;aSOSnX1S;aUVs/sHoDUPM snOSnUb[SU55(aSnUVVVs/sH.oDR5VVs0sHupgXc:wdM Xg_M snnPM0 nnnn[URUUR UR URUS9n UR SL=(d UR [R:Hn UGb8U (aY[USU[5(a[U5U S'GOUV s/sHn [R"U //5PM sn U S'GOUR [R:H=(a UR SLn U (d[SU55O UR n URb+XR-S- UR-UR-n URR n[USU[5(aXUV s/sHFn US :XaXR"/U [%U 5- --OUR"/U [%U 5- -U -PMH sn U S'OUV s/sHn US :XaS[R"U [R&"UR"/U [%U 5- -[R(S 9/5OR[R"[R&"UR"/U [%U 5- -[R(S 9U /5PM sn U S'U R+SS5bUUS :Xa%SSKnUR/U SUR(S 9U S'O/[R&"U S[R(S 9U S'OSU S'UbIUR0b<[3UR0S 5(a!UR0R5U SS 9nUU S'U $s snfs snnfs snnnfs sn fs sn fs sn f)Nr?rrBc3(# UHoSLv M g7frr)rr?s rr2DataCollatorForSeq2Seq.__call__..s%HtmsraFc38# UHn[U5v M g7frr)rls rrr1s&>v!s1vvvrrkrqr@r%prepare_decoder_input_ids_from_labels)rBdecoder_input_ids)rrrNr.r)r]r^r_r DO_NOT_PADrEr;r concatenate MAX_LENGTHrrsrhrtrYrWr(rDrLrr&r)rrrrurvrBrTrUnon_labels_featuresrS no_paddingr? max_paddingmax_label_lengthrsrDrs rrDataCollatorForSeq2Seq.__call__s  !!00N '8A; 6WH BLYZP[B[X>X'*%X>ae  #%H%H"H"HFdlmdlY`TAOTdlm3 NN LL#66)  \\U*Xdllo>X>X.X  hqk*5t<<&*6lE(OPV&WPVur~~ubk'BPV&WE(O"llo.H.HHhT__dhMh FQ3&>v&>#>W[WfWf **6),C,CCaG223112% $~~:: hqk*5t<< &, '&,E(72!8!8 9=MPSTYPZ=Z [["556:JSQVZ:WX[``a&, 'E(O*&,'&,E(72  % "$*A*A)BFVY\]bYcFc)dlnltlt u ^^ "$*A*A)BFVY\]bYcFc)dlnltlt u % &,'E(O$ 99Xt $ 0%"',,uXekk,"Rh"$((5?"(("Kh"E(O   & $KLL $ P PX]^fXg P h ):E% & c? Um&'X''s7O6P6 O;O; P9"P'A P ?B6P;Prr)rrr r!r9r r:rrr]rcr"r r^rIr_rhrrr#rrrrrsiB'&E3:,0GTCZ/ )0!Jd !%)d )""NCVrrc \rSrSr%Sr\\S'Sr\\S'Sr \\S'Sr \ S -\S 'S r \ \S 'S r \ \S'S r\S -\S'Sr\\S'S r\S -\S'SrSrSrS\\\\-\\\4-S\\\44SjrS%S\S\S -S\S -S\\\44SjjrS\\\\-\\\4-S\\\44SjrS%S\S\S -S\S -S\\\44Sjjr\S\R@\R@\\\4S\R@\R@\S\\R@\R@\\R@\R@\44S j5r!\S!\R@\R@\S"\S\4S#j5r"S$r#g )&DataCollatorForLanguageModelingija4 Data collator used for language modeling. Inputs are dynamically padded to the maximum length of a batch if they are not all of the same length. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. mlm (`bool`, *optional*, defaults to `True`): Whether or not to use masked language modeling. If set to `False`, the labels are the same as the inputs with the padding tokens ignored (by setting them to -100). Otherwise, the labels are -100 for non-masked tokens and the value to predict for the masked token. whole_word_mask (`bool`, *optional*, defaults to `False`): Whether or not to mask whole words instead of individual tokens. mlm_probability (`float`, *optional*, defaults to 0.15): The probability with which to (randomly) mask tokens in the input, when `mlm` is set to `True`. mask_replace_prob (`float`, *optional*, defaults to 0.8): The probability with which masked tokens are replaced by the tokenizer's mask token (e.g., `[MASK]`). Defaults to 0.8, meaning 80% of the masked tokens will be replaced with `[MASK]`. Only works when `mlm` is set to `True`. random_replace_prob (`float`, *optional*, defaults to 0.1): The probability with which masked tokens are replaced by random tokens from the tokenizer's vocabulary. Defaults to 0.1, meaning 10% of the masked tokens will be replaced with random tokens. The remaining masked tokens (1 - mask_replace_prob - random_replace_prob) are left unchanged. Only works when `mlm` is set to `True`. pad_to_multiple_of (`int`, *optional*): If set, will pad the sequence to a multiple of the provided value. return_tensors (`str`): The type of Tensor to return. Allowable values are "np", or "pt". seed (`int`, *optional*): The seed to use for the random number generator for masking. If not provided, the global RNG will be used. For best performance, this data collator should be used with a dataset having items that are dictionaries or BatchEncoding, with the `"special_tokens_mask"` key, as returned by a [`PreTrainedTokenizer`] or a [`PreTrainedTokenizerFast`] with the argument `return_special_tokens_mask=True`. 1. Default Behavior: - `mask_replace_prob=0.8`, `random_replace_prob=0.1`. - Expect 80% of masked tokens replaced with `[MASK]`, 10% replaced with random tokens, and 10% left unchanged. 2. All masked tokens replaced by `[MASK]`: - `mask_replace_prob=1.0`, `random_replace_prob=0.0`. - Expect all masked tokens to be replaced with `[MASK]`. No tokens are left unchanged or replaced with random tokens. 3. No `[MASK]` replacement, only random tokens: - `mask_replace_prob=0.0`, `random_replace_prob=1.0`. - Expect all masked tokens to be replaced with random tokens. No `[MASK]` replacements or unchanged tokens. 4. Balanced replacement: - `mask_replace_prob=0.5`, `random_replace_prob=0.4`. - Expect 50% of masked tokens replaced with `[MASK]`, 40% replaced with random tokens, and 10% left unchanged. Note: The sum of `mask_replace_prob` and `random_replace_prob` must not exceed 1. If their sum is less than 1, the remaining proportion will consist of masked tokens left unchanged. r)TmlmFwhole_word_maskg333333?Nmlm_probability皙?mask_replace_probg?random_replace_probr_rrseedcUR(auURRc [S5eURb URS:dURS:a [S5e[ UR5UlOUR (a [S5eURUR-S:a [S5eURS:dURS:a [S5eURS:dURS:a [S5eUR (ajURR(d[R"S [5 URS:a$[R"S 5 SUlSUl[ UR5Ul[ UR5UlSUl g) NzThis tokenizer does not have a mask token which is necessary for masked language modeling. You should pass `mlm=False` to train on causal language modeling instead.rrkz*mlm_probability should be between 0 and 1.zmWhole word masking can only be used with mlm=True.If you want to use whole word masking, please set mlm=True.zHThe sum of mask_replace_prob and random_replace_prob should not exceed 1z,mask_replace_prob should be between 0 and 1.z.random_replace_prob should be between 0 and 1.zcWhole word masking depends on offset mapping which is only natively available with fast tokenizers.zbRandom token replacement is not supported with whole word masking. Setting mask_replace_prob to 1.)rr) mask_tokenrrrKrrris_fastwarningswarn UserWarning generator)rs r __post_init__-DataCollatorForLanguageModeling.__post_init__s 88~~((0 `##+t/C/Ca/G4K_K_bcKc !MNN#()=)=#>D  ! !N   ! !D$<$< >)) y %%) 6*+&+,(!&t'='=!>#()A)A#B rcURS:Xa#SSKnUR5RU5$[R R U5$)Nrr)rrD Generator manual_seedrrandom default_rng)rrrDs r get_generator-DataCollatorForLanguageModeling.get_generators@   $ & ??$006 699((. .rcV[R"5RS:Xa!URUR5UlgSSKnURRR5nUc Sn[U5eURURUR-5Ulg)N MainProcessr)zZWorker process information is not available for seeding the generator. This may be becausezZyou are using multiprocessing without using a PyTorch DataLoader. The `seed` parameter canzVonly be used when using multiprocessing with a PyTorch DataLoader. Please either use azAsingle process or use a PyTorch DataLoader with multiple workers.) mpcurrent_processnamerrrrDutilsdataget_worker_inforid)rrD worker_info error_strings r create_rng*DataCollatorForLanguageModeling.create_rngs    $ $ 5!// :DN ++**::>#G#GHaHa#b  ! !Q &$*B*Ba*G> !T333&*%=%=%N" OOEJJv||5OP\`\j\jO k p p r    }}S%8&,,ejjdhdrdr}s !-!=~c#s#K cXUR(aURcUR5 [US[5(a![ UR USURS9nO S[XR URS90nURSS5nURSS5nUR(aURUSX4S9uUS'US 'U$[R"US5nUR RbS XUUR R:H'XRS 'U$) NrrrrjrrrrrBrg)rrrrErr.r)r_rrrnumpy_mask_tokensrcopyrrs rr*DataCollatorForLanguageModeling.numpy_call[s 99/ OO  hqk7 + +6RVRiRiE 1(NN_c_v_vwE $ii(=tD#3T: 88262H2Hk"8K3I3 /E+ h WWU;/0F~~**6@D!>$**40XX1>  ,-( >>!^^44Q8JQiQi4jqqrvwNYY//3ELdLd/ellmqrN   !228^LN"& >>''4+A+A 'U\\]abess   ""1d&<&<6<<"PWWX\]`nn $(>>#?#?  ! !Q &$*B*Ba*G> !T333&*%=%=%N" >>''+ELL'Y``aef !##$   >>22C/b6F6F~6V^`^f^f3L  ""1&@||"T[[\`a !##$  99,,C/b6F6F~6V^`^f^f-L".~~G#s#O)offsetsc2USS2SS2S4nUSS2SS2S4n[R"USSS9nSUSS2S4'[R"USSS9nSUSS2S4'UR[5nU)X$:gUS:H--n[R"USS9nSXq'U)nXx4$)z Map tokens to word ids and create mask of tokens to not mask. Tokens that are part of the same word will have the same word id and we will only set a mask probability for the first token of each word. Nrrkrr)rrollrrccumsum) r r token_starts token_endsprev_token_endsprev_token_special is_new_wordr  prob_masks rr++1PUgklUl0mn 99[q1(*% L ""rr maskc[U5nUS:gnUSS2SS2S4USS2SSS24:HUSS2SS2S4-USS2SSS24-n[R"X1SS2SSS24-SS9$)z. Mask whole words based on word ids and mask. rNrr)rrany)r r+ valid_ids same_words rr0DataCollatorForLanguageModeling._whole_word_masks ~N aDj)Xaqj-AAYqRSUYzEZZ]fghjnpqgq]rr vviq$z"22;;r)rrrrNN)$rrr r!r9r r:rrcrrrKrrr_rIrr"rrrrr;rr<rrrrr staticmethodrrOrrr#rrrrrjs<<|'&C!OT!$(OUT\("u"!$$%)d )NCD#*(T/L.4S C$sCx.(H#IdSVX[S[n@ae==03d =SVY]S]= sCx=~4S C$sCx.(H#IdSVX[S[nD+/%) PP!4ZPd P sCx Pd#BJJuS#X78#OQzzZ\ZdZdehZiOj# rzz"**S/*BJJrzz#,GG H##: <2::bjjo#> [R"S[5 [TU]"U0UD6 SUlSUlg)NzDataCollatorForWholeWordMask is deprecated and will be removed in a future version, you can now use DataCollatorForLanguageModeling with whole_word_mask=True instead.T)rr FutureWarningsuper__init__rr)rargskwargsrs rr8%DataCollatorForWholeWordMask.__init__s;  Q  $)&)#r)rr)rrr r!r9r8r# __classcell__rs@rr4r4s$$rr4c[U[5(aU$[US5(aUR5nUR 5$)Nnumpy)rEr;r&r?rmrs rrmrms8!T G   GGI 88:rc[U[R5(aU$[US5(a,UR 5R 5R 5$[R"U5$)Ndetach)rErrOr&rBcpur?rYr@s rrrsO!RZZ   H  xxz~~%%''xx{rch\rSrSrSrSrS\\\\ 4S\\\ 44Sjr S\ S\ \ \ \ 44Sjr S r g ) DataCollatorForSOPiz Data collator used for sentence order prediction task. - collates batches of tensors, honoring their tokenizer's pad_token - preprocesses batches for both masked language modeling and sentence order prediction c:[R"S[5 g)NzDataCollatorForSOP is deprecated and will be removed in a future version, you can now use DataCollatorForLanguageModeling instead.)rrr6)rr9r:s rr8DataCollatorForSOP.__init__(s  7  rrr/chSSKnSSKJn UVs/sHoDSPM nn[XPR5nUR U5upVnUVs/sHoDSPM nnU"USURR S9nUVs/sHoDSPM n nURU 5n UUUUU S.$s snfs snfs snf) Nr) pad_sequencerjtoken_type_idsT) batch_first padding_valuesentence_order_label)rjrBattention_maskrJrM)rDtorch.nn.utils.rnnrIrr) mask_tokensrrM) rrrDrIrrjrBrNrJsop_label_listrMs rrDataCollatorForSOP.__call__/s39ABg[) B(NNC ,0,<,CKL8"238L%n$VZVdVdVqVqrIQRg"89R${{>:#,,$8   CMSsB% B*<B/rcSSKnURRc [S5eUR 5nUR UR UR5nUR5Vs/sHoPRRUSS9PM nnURURXbRS9SS9 URRb5URURR5nURUSS9 UR!U5R5nU)R#5n URRb5URURR5n U RU S S9 S X8)'UR!UR UR S 55R5U-n URR%URR5X'UR!UR UR S 55R5U-U )-n UR'[)UR5UR UR*S9n XX'XU 4$s snf) z Prepare masked tokens inputs/labels/attention_mask for masked language modeling: 80% MASK, 10% random, 10% original. N-gram not applied yet. rNzThis tokenizer does not have a mask token which is necessary for masked language modeling. Remove the --mlm flag if you want to use this tokenizer.Trr@rrg?rgrg?)rDr)rrrrrrrrmrrrLrcreqrrrKrrrtrJ)rrrDrBrr r padding_maskr rNattention_padding_maskr rrs rrPDataCollatorForSOP.mask_tokensFs'  >> $ $ ,A  "ZZ d6J6JKdjdqdqds ds]`NN 2 23SW 2 Xds   '' 5HPZPZ ([cf'g >> # # /!99T^^%@%@AL  + +L + D);<AAC)/002 >> # # /%+YYt~~/J/J%K "  ' '(>c ' J"&!??5::fllC+HINNPSaa#'>>#G#GHaHa#b FLL#)FGLLNQ__csbss}}S%8&,,ejj}Y !-!=~--3 s/#I7rN)rrr r!r9r8r;r<r"rrrrPr#rrrrErEsR  d38n!5 $sCx. .).#).%S# *>).rrEc \rSrSr%Sr\\S'Sr\\S'Sr \ \S'Sr \ \S 'S \ \ \ \-\\ \4-S \\ \44S jrS \ \ \ \-\\ \4-S \\ \44S jrS\S \\\\\44SjrS\S \\\\\44SjrSrg)*DataCollatorForPermutationLanguageModelingirz Data collator used for permutation language modeling. - collates batches of tensors, honoring their tokenizer's pad_token - preprocesses batches for permutation language modeling with procedures specific to XLNet r)gUUUUUU?plm_probabilitymax_span_lengthrrrr/c[US[5(aUVs/sHo"SPM nn[XR5nUR U5upEpgXEXgS.$s snfNrrj)rj perm_masktarget_mappingrB)rErrr)rrrrrSrr_r`rBs rr5DataCollatorForPermutationLanguageModeling.torch_call\ hqk7 + +0891+H9$X~~>484J4J54Q1>#~pp:Ac[US[5(aUVs/sHo"SPM nn[XR5nUR U5upEpgXEXgS.$s snfr^)rErrr)rras rr5DataCollatorForPermutationLanguageModeling.numpy_callrcrdrc SSKnURRc [S5eUR S5S-S:wa [S5eUR 5nUR URSURS9nURUR S5UR S5UR S54URS9n[UR S55HnSnUR S5nXx:aURSURS-S5R5n [XR - 5n XrRX- S-S5R5-n SXFXU -24'Xz- nXx:aMUR#UR S55XV'M UR%UR'5V s/sHoRR)U S S 9PM sn URS9n UR+U S S 9 URR,b5UR/URR05nUR+US S 9 WU -)nURR2X'S X4)'URUR S5UR S5UR S54URS9n[UR S55GHnUR5UR S55nUR7SUR S5S-45R9SS5nUUR;UR S5S-5nUR=UR9SS55nUR+XF)X-S5 UR7UR S5S45UR7SUR S545:*XF-UU'GM UR?5UXSR?54$s sn f) The masked tokens to be predicted for a particular sequence are determined by the following algorithm: 0. Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far). 1. Sample a `span_length` from the interval `[1, max_span_length]` (length of span of tokens to be masked) 2. Reserve a context of length `context_length = span_length / plm_probability` to surround span to be masked 3. Sample a starting point `start_index` from the interval `[cur_len, cur_len + context_length - span_length]` and mask tokens `start_index:start_index + span_length` 4. Set `cur_len = cur_len + context_length`. If `cur_len < max_len` (i.e. there are tokens remaining in the sequence to be processed), repeat from Step 1. rNThis tokenizer does not have a mask token which is necessary for permutation language modeling. Please add a mask token if you want to use this tokenizer.rkrThis collator requires that sequence lengths be even to create a leakage-free perm_mask. Please see relevant comments in source code for details.r@)rkTrrrrgr) rDr)rrrrrrrrczerosrXrrr\rHrIrZeyerLrmrrrrTrrarangereshape transposerandpermflattenrJ)rrrDrBr r`rcur_lenmax_len span_lengthcontext_length start_indexr rrU non_func_maskr_ perm_indexs rr> $ $ ,N  ;;q>A  "A  FLL!5::Ffkk!nfkk!nfkkRSn%U]b]j]jkv{{1~&AGkk!nG##mmAt/C/Ca/GNSSU !$[3G3G%G!H% n6RUV6VX\(](b(b(dd MN+k0I"IIJ)#!& &++a. 9N %'($llekereret uet^a^^ 3 3CTX 3 Yet u**+  ##$7s#C >> # # /!99T^^%@%@AL  ' ' C ' @')<<= !%!=!="&KKQQQ PX]XeXeKf v{{1~&Afkk!n5J#++RQ11D,EFPPQRTUVJ#ENN6;;q>Q3F$GHJz';';Aq'ABJ  # #^%6$69I$I2 N ""FKKNA#67:;M;MqRXR]R]^_R`Na;bb!"IaL1'8{{}iFFY vs#Pc jURRc [S5eURSS-S:wa [S5e[R "U5n[R "URS[S9n[R"URSURSURS4[RS9n[URS5HnSnURSnXg:aY[SURS-5n[XR- 5n U[SX- S-5-n SX5XU-24'Xi- nXg:aMY[R"URS5XE'M [R "UR#5V s/sHoRR%U SS9PM sn [S9n SX<'URR&bX RR(:Hn S X='W U -)nURR*X'S X#)'[R"URSURSURS4[RS9n[URS5Hn[R,"URS5nUR/S URSS-45R0n[R2R5U5 UR0R75nS UX5)X-'UR/URSS45UR/SURS45:*X5-X'M UR9[R:5XUR9[R:54$s sn f) rhrirkrrrjr@Trrrgr)r)rrrrrrrrcrkrXrrr\rIrZrlrYrmrrrrrmrnTrshufflerqrrW)rrrBr r`rrrrsrtrurvr rrUrwr_rxs rr> $ $ ,N  <<?Q ! #A  q=6<<?FLLOV\\RS_"U]_]g]ghv||A'AGll1oG#%a)=)=)AB !$[3G3G%G!H%>3ORS3S(TT MN+k0I"IIJ)#!#v||A 7N %((!hhekereret uet^a^^ 3 3CTX 3 Yet u /0+ >> # # /!^^%@%@@L+.N (')<<= !%!=!="&HHfll1ov||A QPXZXbXbc v||A'A6<<?3J#++RaA1E,FGIIJ II  j )#--/J ACJ))M,<< = ""FLLOQ#78J Using `DataCollatorWithFlattening` will flatten the entire mini batch into single long sequence. Make sure your attention computation is able to handle it! TrgF)return_position_ids separator_idreturn_flash_attn_kwargsreturn_seq_idxc>[TU]"U0UD6 XlX lX0lX@l1SkUl1SkUlSS1Ulg)N>rBrj position_ids>rBseq_idxrjr max_length_q max_length_k) r7r8rrrr _int_64_keys_batch_dim_keys _py_int_keys)rrrrrr9r:rs rr8#DataCollatorWithFlattening.__init__fsL $)&)#6 ((@%,CQ+^<rc Uc URnUc URnSUS;n//S.nUR(aURS/05 UR(aURS/05 UR (aS/nSn[ U5GHQupU Sn [U S5(aU R5n US==U - ss'U(a;U Sn [U S5(aU R5n US==U/U SS-- ss'OUS==U/U SS-- ss'UR(a(US==[[[U 555- ss'UR(a/US==[[U 55V s/sHoPM sn - ss'UR (dGMWRUS [U 5-5 [W[U 55nGMT UR (aW=US 'US 'W=US 'US 'US:Xa)SSKn U RnU R nU R"nOGUS:Xa1[$R&n[$R n[$R"nO[)SU<S35eUR+5HGunnUUR,;aU/nUUR.;dM+U"UUUR0;aUOUS9UU'MI U$s sn f)NrBr)rjrBrrrjrmrkr cu_seq_lens_q cu_seq_lens_krrrrz;return_tensors must be one of ("pt", "np"), return_tensors=z not supportedr@)rrrupdaterrrr&rmr;rrtappendrrDrLrWint32rrYrrNrrr)rrrris_labels_providedrS cu_seq_lensr^rsamplerjrB_rDdata_clsdtype_64dtype_32rTrUs rr#DataCollatorWithFlattening.__call__xs  !!00N  ,,L%!4 B/  # # LL."- .    LL)R )  ( (#KJ(2OG{+Iy(++%,,. + ) + !)68,,#]]_FhL>F12J#>>hL>IabM#AA''n%eC N.C)DD%""i eC N6K$L6KW6K$LL ,,,"";r?S^#CD S^< + 3.  ( (>I IE/ "U?%;O>O9OXU]^a " =%Ms K# )rrrrrrrr1) rrr r!r9r8rr#r<r=s@rrrSs*(!!& ==$>>rr)rr),multiprocessingrrcollections.abcrr dataclassesrrrtypingrr?rtokenization_utils_baser rr InputDataClassr;r<r" DataCollatorr r.r3r5r1r2r[rerIrrrrrr4rmrOrrErYrrrrrs;-!=#n-.S#X>?  N N,5D$85RVWZ\_W_R`5. ?+? ?6!$~*>!4S>!H$~*>4S>@ 00 0f h):h hV"#*"J#*@ ?$5? ?D @@ @F L<&7L< L<^  $#B$ $&c2::c? O.8O. O.d ][1B][ ][@ b!4b br