>:jwjSrSSKJr SSKrSSKrSSKJr SSKJrJ r SSK J r SSK r SSKrSSKJrJrJrJr SSKJr SS KJrJr \R8"\5rS S S S S .rSS jr SS jr!SSjr"SSjr#SSjr$SSjr%"SS5r&SSSjjr'SSSjjr(SSjr)g!\a SrNtf=f!\\4a SrSrN}f=f)zAModality detection, input parsing, and message format conversion.) annotationsN) defaultdict)AnyLiteral)urlparse) MessageFormatModality PairInput SingleInput)Image) AudioDecoder VideoDecoderflat)apertus deepseek_v3gpt_ossseed_ossc<URS5=(a SU;$)zQCheck if a string looks like a valid URL (starts with http(s) and has no spaces).)zhttp://zhttps:// ) startswithtexts `/home/mande/repo/quber/.venv/lib/python3.13/site-packages/sentence_transformers/base/modality.py_looks_like_urlr,s ??2 3 G4Gc[U5(a4[U5RR5nUR U5$UR5R U5=(a [ RR U5$)zOCheck if a string is a URL or local file path with one of the given extensions.)rrpathlowerendswithosisfile)r extensionsrs r_is_media_url_or_pathr#1s]t~""((*}}Z(( ::<  , E1EErcHURS5(ag[US5$)z:Check if a string is an image URL, file path, or data URI.z data:image/T)z.jpgz.jpegz.pngz.gifz.bmpz.tiffz.webp)rr#rs ris_image_url_or_pathr%9s" }%% 'b ccrcz[US5(ag[U5=(a [U5RS;$)z.Check if a string is a video URL or file path.)z.mp4z.aviz.movz.wmvz.flvz.mkvT)zwww.youtube.comz youtube.comzyoutu.bez m.youtube.com)r#rrnetlocrs ris_video_url_or_pathr(@s;T#STT 4  Xd^%:%:?&rc[US5$)z/Check if a string is an audio URL or file path.)z.mp3z.wavz.oggz.flacz.aac)r#rs ris_audio_url_or_pathr*Ls 'P QQrc[U[[45(a[U5S:wag[US[5(a[US[5(agUH`n[U[ 5(aSU;aSU;a g[U[5(dM=U(dMF[US[ 5(dM` g g)ayCheck if a sample is a non-text pair (2-element tuple/list with at least one non-string element). Text pairs ``(str, str)`` are handled natively by tokenizers and detected as ``"text"`` modality by :func:`infer_modality`. This helper detects pairs that contain at least one non-string element (e.g. an image, audio array, or dict), which require conversion to message format. FrrolecontentT) isinstancetuplelistlenstrdict)sampleelems r_is_non_text_pairr8Qs fudm , ,F q0@&)S!!jC&@&@ dD ! !fnd9J dD ! !ddz$q'4/H/H  rc\rSrSrSrSSSjjrSSjrSSjrSSjrSSSjjr SS jr \ SS j5r SS jr SS jrSS jrSrg)InputFormatterfaHHandles input parsing, modality detection, and message format conversion. This class manages the complete input preprocessing pipeline: 1. Parsing raw inputs to detect their modality (text, image, audio, video, message) 2. Converting inputs to different chat template formats 3. Normalizing mixed-modality inputs Different models require different message/chat template formats: - **Structured format**: Content is a list of dicts with type annotations [{"role": "user", "content": [{"type": "text", "text": "hello"}]}] - **Flat format**: Content is the direct value [{"role": "user", "content": "hello"}] Args: model_type: The model type string (e.g. from ``config.model_type``). message_format: Message format to use. Options: - ``"structured"``: Content is a list of dicts with type/modality keys - ``"flat"``: Content is the direct value - ``"auto"``: Automatically infer from processor (default) processor: Optional processor to infer format from when ``message_format="auto"``. supported_modalities: Optional list of modalities supported by the model. When provided, string inputs that look like media URLs/paths are only classified as non-text if the model actually supports that modality. This prevents text-only models from misclassifying text containing media URLs. NcXlX0lX@lUS:Xa U(aURU5OSUlgX lg)Nauto structured) model_type processorsupported_modalities _infer_formatmessage_format)selfr?rCr@rAs r__init__InputFormatter.__init__s:%"$8! V #CL$"4"4Y"?R^D "0 rc^UR[;a[UR$[USS5m[T[5(aT(dg/SQn[ U4SjU55(agg)acInfer the message format expected by the processor. Checks known model types first, then inspects the processor's chat template for patterns indicating structured format. Defaults to ``"structured"`` if neither approach is conclusive. Args: processor: The processor/tokenizer to inspect. Returns: ``"structured"`` or ``"flat"`` message format. chat_templateNr>)z content[0]z.typez'type'z"type"z item.typezmessage.content[c3,># UH oT;v M g7fN).0patterntemplates r /InputFormatter._infer_format..sF2Ew("2Esr)r?!KNOWN_MODEL_TYPES_MESSAGE_FORMATSgetattrr0r4any)rDr@structured_patternsrNs @rrBInputFormatter._infer_formatsc ??? ?4T__E E9ot<(C((  F2EF F Frc tU(dSS/0[[54$/n[[5nSnUGHn[U5(aURSU45 SnM+[ XPR S9nUS:Xa'[ U[5(aUSnUSUSS'GOUS:Xag[b`[ U[5(aKUR5nURRS S 9R5nURUSS'GO.US :XaB[ U[5(a-USnUS RS /5RUS 5 OUS :Xa[b[ U[5(a[U5n UR!S U 5n U RnUS RS /5RUR"R$UR"R&[)[+U RR,S 55S .5 O!US:Xa[ U[5(aU/nOUnURXg45 GM U(a/n UHupU S:Xa"U RUR/U55 M-U S:XaL[ U[0[(45(a1[U5S:Xa"U RUR/U55 MU S:XaU RU5 M[ U [05(aUOX0n U RUR3U 55 M SSU 0U4$[5U6up[)U5n[7U5n[U5S:XaeUR95n[ U[:5(aXo0nOUS R=5nUV Vs0sHoUVs/sHnUU PM sn_M nn nO[[>RASUS35 SUVVs/sH.upgUR3[ U[05(aUOXg05PM0 snn0nSnXoU4$s snfs snn fs snnf)aParse inputs and group by modality. Analyzes a list of inputs to detect their modality (text, image, audio, video, message) and groups them appropriately for the processor. Handles mixed modalities by converting to message format when necessary. Non-text pairs (e.g. ``(image, text)`` or ``(image, image)``) are detected and converted to message format with ``"query"``/``"document"`` roles via :meth:`pair_to_messages`. Args: inputs: List of inputs to parse. Can be: - str: Text inputs - tuple/list of str: Text pairs (for cross-encoders) - tuple/list of mixed types: Non-text pairs (e.g. image + text) - dict: Chat messages, audio data, or multimodal inputs - PIL.Image.Image: Image inputs - np.ndarray/torch.Tensor: Audio (1-2D) or video (3-5D) inputs Returns: Tuple of (modality, processor_inputs_dict, extra_modality_kwargs) where: - modality: Detected modality string (``"text"``, ``"image"``, etc.) or tuple of modalities - processor_inputs_dict: Dictionary mapping modality names to input lists - extra_modality_kwargs: Extra kwargs per modality (e.g. ``sampling_rate`` for audio) rFpairTrAaudioarray sampling_rater)dimvideovideo_metadata)fpstotal_num_framesframes_indicesmessager,r-zMixed modalities detected: z!. Converting to 'message' format.)!rr5r8appendinfer_modalityrAr0r get_all_samplesdatameannumpy sample_rate setdefaultrr3get_frames_in_rangemetadata average_fps num_framesr2rangeshapepair_to_messagesr1 to_messagezipsetpopr4keysloggerdebug)rDinputs typed_inputsextra_modality_kwargs has_pairsitemmodalityvaluesamplesrn frame_batchmessagesmodtyped modalitiesprocessed_inputsunique_modalities ordered_keysentrys r parse_inputsInputFormatter.parse_inputss8FB<T):: :EG +D 1 D!&&##VTN3 %dAZAZ[H7"z$'='=W BFBW%g.?W$)AjQUWcFdFd..0 ))a)0668BIBUBU%g.?W$D$)?)?W %g.99:JBOVVW[\lWmnW$)AjQUWcFdFd Y "66q*E #((%g.99:JBOVV#}}88,0MM,D,D*.u[5E5E5K5KA5N/O*PY&:dD+A+A    1 2QZ H* &=OOD$9$9%$@AF]z%%'G'GCPUJZ[OOOD$9$9%$@AI%OOE*%/U%;%;E#EOODOOE$:;+y(35JJ J'*L'9$  01 O  !Q &(,,.H(C(($,#?  02779 _k#l_kX[BR)SBR%*BR)S$S_k #l LL67H6IIjk l+7+7OOZ%-H-HExN_`+7   !H+@@@*T#ls& P.0P)?P.)5P4)P.cUup#[U5n[U5nURS:Xa SUS.SUS./$SnSU"XB5S.SU"XS5S./$)aConvert a pair of inputs to query/document message format. Each element of the pair is wrapped in a message with role ``"query"`` (first element) or ``"document"`` (second element). The modality of each element is inferred individually via :func:`infer_modality`. Args: pair: A 2-element tuple or list of inputs (e.g. ``(image, text)``). Returns: List of two message dictionaries with ``"query"`` and ``"document"`` roles. rqueryr.r/documentc[U[5(a4[U[5(aUVs/sHo"U;dM SX"X0PM sn$SXU0/$s snf)Ntype)r0r1r5)r~r}rs r _to_content4InputFormatter.pair_to_messages.._to_content@sY(E**z$/E/EAIY#TX[5495YYX67 7Zs A A)rdrC)rDrW query_itemdoc_itemquery_modality doc_modalityrs rrqInputFormatter.pair_to_messages)ss $ ' 3%h/   & ( Z8#9   8^)P QK ,O P  rc  URS:XaM[U5S:Xa)[[UR 555up4X$S./$[ R S5 UUR 5VVs/sH upTSXUU0PM snnS./$s snnf)aConvert a typed input dictionary to message format. Produces a single message with the given ``role``. For pair/multi-value inputs, use :meth:`pair_to_messages` instead (which is called automatically by :meth:`parse_inputs`). Args: typed_input: Dictionary mapping modality to input value (single value per modality). role: Role for the message (default: ``"user"``). Returns: List of message dictionaries (single message). rr-rzbFlat message format requested but multiple modalities detected. Falling back to structured format.r)rCr3nextiteritemsrwwarning)rD typed_inputr._rr~s rrrInputFormatter.to_messageLs   & (;1$[%6%6%8 9:!%8999WbWhWhWjkWjOHVX?Wjk   ls2B cU(dSS/04$[U[5(aU4OUn[[[ UR 5555n/n[ U5HnUVs0sHowU;dM XrUU_M nn[U5S:Xa[[ UR 555n [U [[45(aH[U 5S:Xa9[SU 55(a"URURU 55 MURURU55 M SSU04$s snf)a3Convert a batch of modality-specific inputs into the unified message format. Args: modality: The modality key (string) or tuple of modality keys. processor_inputs: Dictionary mapping modality names to lists of inputs. Returns: Tuple of ``("message", {"message": [messages_per_sample, ...]})`` rbr-r,c3B# UHn[U[5v M g7frJ)r0r4)rLvs rrO2InputFormatter.batch_to_message..s OrlqghPZ[\^aPbPblqs) r0r4r3rrvaluesror1r2allrcrqrr) rDr~processor_inputsr batch_sizerirrrs rbatch_to_messageInputFormatter.batch_to_messagejs! y"o- -$.x$=$=h[8 d#3#:#:#<=>? z"ADTjDTS_iXi83 5a 88DTKj;1$T+"4"4"678eeT]33E aCOrlqOrLrLrOOD$9$9%$@A OODOOK8 9#9h///ks ' E4 EcUHfnUH]nURS5n[U[5(aM+[U[5(a[ SU55(a gM\ g Mh g)ahCheck whether all messages in a batch contain only text content. Works with both flat format (``{"content": "hello"}``) and structured format (``{"content": [{"type": "text", "text": "hello"}]}``). Args: messages_batch: List of message lists, one per sample. Returns: True if every message contains only text, False if any contain non-text content. r/c3J# UHoRSS5S:gv M g7f)rrN)get)rLr}s rrO7InputFormatter.is_text_only_messages..sR'$88FF3v='s!#FT)rr0r4r2rS)messages_batchrrbr/s ris_text_only_messages$InputFormatter.is_text_only_messagessg'H#!++i0gs++gt,,R'RRR$S!$'rc/nUGHUnSU;dSU;aSUS35 M+USn[U[5=(a U=(a [US[5nUR S:XasU(al[ U5S:Xa(SUS;aUR0UESUSS0E5 M[RS [ U5S 35 URU5 MUR S :XaNU(dG[U[5(aUR0UESSUS ./0E5 GM0URU5 GMDURU5 GMX U$) a Normalize messages to the target format (``self.message_format``). Extra keys beyond ``"role"`` and ``"content"`` are preserved during conversion. Args: messages: List of message dictionaries to normalize. Returns: Normalized list of message dictionaries. r.r/zInvalid message format: z . Skipping.rrr-rz;Cannot convert structured message to flat format: contains z# content items. Keeping structured.r>rr) rwrr0r2r5rCr3rcr4)rDr normalizedrbr/is_currently_structureds rnormalize_messages!InputFormatter.normalize_messagessQ GW$ (@!9'+NOi(G&0$&?&lG&lPZ[bcd[egkPl #""f,1Hw<1$71:)=%%&P&P)WQZ=O&PQNN$$'L>1TV%%g.$$ 4=Tgs++%%&a&a)vW^>_=`&ab%%g.!!'*/ 2rcURS:XaUVs/sH nSUS./U-PM sn$UVs/sHnSSUS./S./U-PM sn$s snfs snf)zPrepend a system prompt to message format inputs. Args: messages: List of message lists (each message list represents one input). prompt: System prompt to prepend. Returns: Messages with system prompt prepended to each message list. rsystemrrr)rC)rDrprompt message_lists rprepend_prompt_to_messages)InputFormatter.prepend_prompt_to_messagessy   & (_gh_g|h6:;lJ_gh h!) ( VV,L+M N OR^ ^ (  i s A Ac /nUHSn[U[5(aURX$-5 M-URX$S-/[USS5-5 MU U$)a"Prepend a prompt to text format inputs. For single texts, prepends the prompt directly. For text pairs (cross-encoder inputs), prepends only to the first text. Args: texts: List of text inputs (strings or pairs) prompt: Prompt to prepend Returns: Texts with prompt prepended rr-N)r0r4rcr2)rDtextsrresultrs rprepend_prompt_to_texts&InputFormatter.prepend_prompt_to_textssZD$$$ fm, vQ/04QR>AB   r)rCr?r@rA)r=NN)r?r4rCrrAlist[Modality] | NonereturnNone)rzLiteral['structured', 'flat'])rylist[SingleInput | PairInput]rzBtuple[Modality, dict[str, list], defaultdict[str, dict[str, Any]]])rWz tuple | listrlist[dict[str, Any]])user)rzdict[Modality, Any]r.r4rr)r~r rr5rz*tuple[Literal['message'], dict[str, list]])rlist[list[dict[str, Any]]]rbool)rrrr)rrrr4rr)rz'list[str | tuple[str, str] | list[str]]rr4rzlist[str | list[str]])__name__ __module__ __qualname____firstlineno____doc__rErBrrqrrr staticmethodrrrr__static_attributes__rKrrr:r:fs<)/6: 1 1& 1 4 1  1 DtA-tA LtAl! F <0 0480 30:0%N 2 rrYrr]z;Multimodal dict input contains unrecognized modality keys: z. Expected keys from: z-Empty dict input is not a valid input sample.)r-r,)z#Unsupported tensor dimensionality: z> BBksAcR[U[5(aSRU5$U$)zSFormat a modality for display, e.g. ``("text", "image")`` becomes ``"text+image"``.+)r0r1join)r~s rformat_modalityres#(E""xx!! Or)rr4rr)rr4r"ztuple[str, ...]rr)r6rrrrJ)r6zSingleInput | PairInput | AnyrArrr )rrrArrr )r~r rr4)*r __future__rloggingr collectionsrtypingrr urllib.parserrhrr)sentence_transformers.base.modality_typesrr r r PIL.Imager r ImportErrortorchcodec.decodersr rOSError getLoggerrrwrQrr#r%r(r*r8r:rdrrrKrrrsG" #! +>   8 $  %!H Fd R *II\ 37X )X/XXz37C *C/CC0]H WLLs#B*B8*B54B58 CC