>:jC SSKJr SSKrSSKrSSKrSSKJr SSKJr SSK r SSK r SSK J r SSK Jr SSKJr SS KJr \R&"\5r\(aSS KJr S S S SSS\SSS4 SSjjrS SSS\4SSjjrSSjrSSS\4SSjjrSSSjjrg)) annotationsN)Callable) TYPE_CHECKING)Tensor)tqdm)cos_sim)normalize_embeddings)SentenceTransformerF iii dc HURUUUSU U U S9n [U UUUUUS9$)a Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and returns a list with the pairs that have the highest cosine similarity score. Args: model (SentenceTransformer): SentenceTransformer model for embedding computation sentences (List[str]): A list of strings (texts or sentences) show_progress_bar (bool, optional): Plotting of a progress bar. Defaults to False. batch_size (int, optional): Number of texts that are encoded simultaneously by the model. Defaults to 32. query_chunk_size (int, optional): Search for most similar pairs for #query_chunk_size at the same time. Decrease, to lower memory footprint (increases run-time). Defaults to 5000. corpus_chunk_size (int, optional): Compare a sentence simultaneously against #corpus_chunk_size other sentences. Decrease, to lower memory footprint (increases run-time). Defaults to 100000. max_pairs (int, optional): Maximal number of text pairs returned. Defaults to 500000. top_k (int, optional): For each sentence, we retrieve up to top_k other sentences. Defaults to 100. score_function (Callable[[Tensor, Tensor], Tensor], optional): Function for computing scores. By default, cosine similarity. Defaults to cos_sim. truncate_dim (int, optional): The dimension to truncate sentence embeddings to. If None, uses the model's ones. Defaults to None. prompt_name (Optional[str], optional): The name of a predefined prompt to use when encoding the sentence. It must match a key in the model `prompts` dictionary, which can be set during model initialization or loaded from the model configuration. Ignored if `prompt` is provided. Defaults to None. prompt (Optional[str], optional): A raw prompt string to prepend directly to the input sentence during encoding. For instance, `prompt="query: "` transforms the sentence "What is the capital of France?" into: "query: What is the capital of France?". Use this to override the prompt logic entirely and supply your own prefix. This takes precedence over `prompt_name`. Defaults to None. Returns: List[List[Union[float, int]]]: Returns a list of triplets with the format [score, id1, id2] T)show_progress_bar batch_sizeconvert_to_tensor truncate_dim prompt_nameprompt)query_chunk_sizecorpus_chunk_size max_pairstop_kscore_function)encodeparaphrase_mining_embeddings) model sentencesrrrrrrrrrr embeddingss a/home/mande/repo/quber/.venv/lib/python3.13/site-packages/sentence_transformers/util/retrieval.pyparaphrase_miningr sN\+!J ()+%  c US- n[R"5nSnSn[S[U5U5GH*n [S[U5U5GH n U"X X-X X-5n [R "U [ U[U S55SSSS9upU R5R5n U R5R5n [[U 55Hrn[X5H^unnX-nU U-nUU:wdMXUU:dM$URXUUU45 US- nX:dMIUR5nUSnM` Mt GM GM- [5n/nUR5(dpUR5unnn[UU/5unnUU:wa/UU4U;a'URUU45 UR!UUU/5 UR5(dMp[USSS9nU$) a Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and returns a list with the pairs that have the highest cosine similarity score. Args: embeddings (Tensor): A tensor with the embeddings query_chunk_size (int): Search for most similar pairs for #query_chunk_size at the same time. Decrease, to lower memory footprint (increases run-time). corpus_chunk_size (int): Compare a sentence simultaneously against #corpus_chunk_size other sentences. Decrease, to lower memory footprint (increases run-time). max_pairs (int): Maximal number of text pairs returned. top_k (int): For each sentence, we retrieve up to top_k other sentences score_function (Callable[[Tensor, Tensor], Tensor]): Function for computing scores. By default, cosine similarity. Returns: List[List[Union[float, int]]]: Returns a list of triplets with the format [score, id1, id2] rrTFdimlargestsortedc US$)Nrxs r.paraphrase_mining_embeddings..s!A$r!keyreverse)queue PriorityQueuerangelentorchtopkmincputolist enumerateputgetsetemptyr'addappend)rrrrrrpairs min_score num_addedcorpus_start_idxquery_start_idxscoresscores_top_k_valuesscores_top_k_idx query_itr top_k_idx corpus_itrijentry added_pairs pairs_listscoresorted_isorted_js rrrYs 0 QJE    !EII!!S_6GH$QJ9IJO#_-OP.>.RSF 5:JJE3vay>24PU5 1 #6"9"9";"B"B"D /335<<> "3v;/ -67G7R-S)Iz'3A(:5AAv"5"@"Ki"W #6#A)#LaQR"ST!Q $1$)IIKE(-aI.T0 KI6%KJkkmmiik q!#QF^( x Xx$8 $K OOXx0 1   uh9 : kkmm EJ r!c[U0UD6$)z8This function is deprecated. Use semantic_search instead)semantic_search)argskwargss rinformation_retrievalrXs D +F ++r! c [U[R[R45(a[R "U5nO+[U[ 5(a[R"U5n[UR5S:XaURS5n[U[R[R45(a[R "U5nO+[U[ 5(a[R"U5nURUR:waURUR5n[[U55Vs/sHn/PM nn[S[U5U5GHn[X-[U55n UR(a2[R "XURS9n UR#SU 5n OXU n [S[U5U5GHdn [X-[U55n UR(a2[R "XURS9n UR#SU 5nOXU nU"X5n[R$"U[U[US55SSSS9unnUR'5R)5nUR'5R)5n[[U55Hun[+UUUU5H\unnU U-nUU-n[UU5U:a[,R."UUUU45 M@[,R0"UUUU45 M^ Mw GMg GM [[U55HIn[[UU55HnUUUunnUUS.UUU'M [3UUSSS 9UU'MK U$s snf) a This function performs by default a cosine similarity search between a list of query embeddings and a list of corpus embeddings. It can be used for Information Retrieval / Semantic Search for corpora up to about 1 Million entries. Args: query_embeddings (:class:`~torch.Tensor`): A 2 dimensional tensor with the query embeddings. Can be a sparse tensor. corpus_embeddings (:class:`~torch.Tensor`): A 2 dimensional tensor with the corpus embeddings. Can be a sparse tensor. query_chunk_size (int, optional): Process 100 queries simultaneously. Increasing that value increases the speed, but requires more memory. Defaults to 100. corpus_chunk_size (int, optional): Scans the corpus 100k entries at a time. Increasing that value increases the speed, but requires more memory. Defaults to 500000. top_k (int, optional): Retrieve top k matching entries. Defaults to 10. score_function (Callable[[:class:`~torch.Tensor`, :class:`~torch.Tensor`], :class:`~torch.Tensor`], optional): Function for computing scores. By default, cosine similarity. Returns: List[List[Dict[str, Union[int, float]]]]: A list with one entry for each query. Each entry is a list of dictionaries with the keys 'corpus_id' and 'score', sorted by decreasing cosine similarity scores. rrdeviceTFr$) corpus_idrQc US$)NrQr)r*s rr,!semantic_search..s \]^e\fr!r.) isinstancenpndarraygenericr5 from_numpyliststackr4shape unsqueezer\tor3r7 is_sparsearange index_selectr6r8r9zipheapqheappush heappushpopr')query_embeddingscorpus_embeddingsrrrr_queries_result_listrE query_end_idxindices query_chunkrDcorpus_end_idx corpus_chunk cos_scorescos_scores_top_k_valuescos_scores_top_k_idxrI sub_corpus_idrQr]query_iddoc_itrs rrUrUs0"RZZ$<== ++,<= $d + + ;;'78  ! !"a'+55a8#bjj"**%=>>!,,->? %t , ,!KK(9:#3#:#::+../@/G/GH',S1A-B'CD'C!2'CD C(8$9;KLO>DT@UV  % %ll?JZJaJabG*777CK*=IK!&a->)?AR S  !1!EsK\G]^N **,,'7PaPhPhi0==aI 0.Q ( BJ=BJJCs:a='9:4X]= 9 #%9'>&A&A&C&J&J&L ##7#;#;#=#D#D#F "3z?3 ,/0DY0OQhirQs,t(M5 0= @I.:H.x89EA/9E9;M))*=h*G%QZI[\-u4%!TMN#123S!4X!>?@G28.SsAr!r.c[U5$rrr*s rr,rcs#a&r!)r`r5rtensorr\r r7r4maxrr3Ttypesumanyr6rmr@r9r'r=r:update)r thresholdmin_community_sizerrextracted_communities sort_max_size start_idxrzthreshold_maskrow_wise_countlarge_enough_maskrrs top_k_indicescountrv top_k_valuesrL top_val_large top_idx_largeunique_communities extracted_ids cluster_id communitynon_overlapped_communityidxs rcommunity_detectionrs0 j%,, / /\\*-  Y/@/@AI%j1J/ZAA 22B7ZIM aZ*-4FTePe  I,BCjllR     ! !_ 4'4N+//2N!/ D $((**+>N#6J""$A)DA A#&n"Dw%,,WVe_-C-C-EF#E)oo0BDoQOL!3|,-?2&)33=a=3E3E _c3E3d0M=(+i7MCPZO<[(+A ,=s:(O 7A!}7I7IMcg7I7h4 }(+i7MCPZO<[*00}PY?Y1Z1a1a1cd.AZ##8>NX\]EM!*+@!A I#% C-'(//4 ' (,> >  % %&> ?  !9 :"B 28HRVW r!)rr rz list[str]rboolrintrrrrrrrrr"Callable[[Tensor, Tensor], Tensor]rz int | Noner str | Nonerrreturnlist[list[float | int]])rrrrrrrrrrrrrr)r"list[list[dict[str, int | float]]])rqrrrrrrrrrrrrrr)g?rYiF) rztorch.Tensor | np.ndarrayrfloatrrrrrrrzlist[list[int]]) __future__rrnloggingr1collections.abcrtypingrnumpyrar5rtqdm.autonotebookr similarityr rr getLogger__name__logger0sentence_transformers.sentence_transformer.modelr r rrXrUrr)r!rrs"  $  "(   8 $T $ #9@#"? ??? ?  ?  ?? ?7??? ??H!#9@ FFFF F  F 7 FFR, #9@ XXXX X  X 7 X(Xz # c)ccc c  c  cr!