j F S SK r S SKrS SKrS SKrS SKJr S SKJr S SKJ r J
r
J r \ ( a S SK J
r
\R " \5 r " S S5 rSSSSSSSS.S S
S \\\
4 S- S \S- S
\S- S\S- S\S- S\\\ - S- S\\\ - S- SS4S jjr " S S\5 rg) N)abstractmethod)deepcopy)
TYPE_CHECKINGAnyListGenerationConfigc B \ rS rSrSr\S\S\4S j5 rS\ 4S jr
Sr g) GenerationStopper z
Base interface for stopping logic.
- should_stop(s): True to stop given the current decoded text window.
- lookback_tokens(): how many tokens should be considered (default: sys.maxsize).
sreturnc g N )selfr
s b/home/mande/repo/quber/.venv/lib/python3.13/site-packages/docling/models/utils/generation_utils.py should_stop GenerationStopper.should_stop s c " [ R $ r )sysmaxsize)r s r lookback_tokens!GenerationStopper.lookback_tokens s {{r r N) __name__
__module__ __qualname____firstlineno____doc__r strboolr intr __static_attributes__r r r r r s5
S
T
r r ) overridesmax_new_tokens use_cache do_sample temperature pad_token_id eos_token_id base_configzGenerationConfig | Noner% r& r' r( r) r* r+ r r c SSK Jn U b [ U 5 OU" 5 n Ub X)l Ub X9l Ub Xil OU R
c Ub Xyl U( a U R " SSS0UD6 Ub XIl Ub XYl U $ )Nr r allow_custom_entriesTr ) transformersr r r& r' r* updater( r) )
r, r% r& r' r( r) r* r+ r configs
r build_generation_configr2 ! s .&1&=Xk
"CSCUF
! . $
* $ )A *
=4=9= $ ( Mr c ^ \ rS rSrSrSSS.S\S\4S jjrS \\ S
\4S jr S \
S
\4S
jr Sr g)DocTagsRepetitionStopperP a&
Detects repetitive ...text blocks,
but only when repeats are **consecutive** and both tag & inner text are identical.
Performance:
- Heavy check runs every N calls (default 32).
- Only decodes the last LOOKBACK_TOKENS tokens per sequence (default 200).
)Nr r8 r c [ S[ U5 5 U l [ S[ U5 5 U l SU l [
R " S[
R [
R - 5 U l g )N r z
<(?P[a-zA-Z0-9_]+)>\s*
(?P.*?)?
\d+)>\d+)>\d+)>\d+)>
(?P.*?)
)
maxr# r8 _lookback_tokens _call_countrecompileDOTALLVERBOSE_PATTERN)r r8 r s r __init__!DocTagsRepetitionStopper.__init__Z sX QA #As?'; <
II
"
r valsr c ^^ [ U5 S: a g[ R " U5 VVs/ s H u p#X2-
PM nnn[ S U 5 5 ( a g[ U5 [ U5 - mST- m[ UU4S jU 5 5 $ s snnf )u< 3+ strictly increasing values with ~regular spacing (±20%). Fc 3 * # U H oS :* v M g7fr Nr ).0ds r 4DocTagsRepetitionStopper._regular..q s %u!Avus g?c 3 F ># U H n[ UT-
5 T:* v M g 7fr )abs)rJ rK meantols r rL rM u s 7A3q4x=C's !)len itertoolspairwiseanysumall)r rE abdiffsrP rQ s @@r _regular!DocTagsRepetitionStopper._regularl s{ t9q=#,#5#5d#;<#;41#;< %u% % %5zCJ&Dj7777 =s B r
c ^ S=p#/ nS[ [ S[ 4U 4S jjnT R R U5 H nUR S5 UR S5 p[
UR S5 5 [
UR S5 5 [
UR S 5 5 [
UR S
5 5 4n X':X a X8:X a UR U 5 M U" U5 ( a g Xxp2U /nM U" U5 $ ) u
Trip only on **consecutive** runs (no other matched blocks between) of ≥3 items
with the same and identical inner text, where within that run we see:
- any exact duplicate (x,y,w,h), or
- stable X/W with regular Y progression, or
- stable Y/H with regular X progression.
Nboxesr c >^^^^ [ U 5 S: a g[ [ U 5 5 [ U 5 : a g[ U 6 u mmmm[ U4S jT 5 5 n[ U4S jT 5 5 n[ U4S jT 5 5 n[ U4S jT 5 5 nU( d U( a T R [ T5 5 ( a gU( d U( a T R [ T5 5 ( a gg)NrG FTc 3 2 ># U H oTS :H v M g7frI r )rJ xxss r rL ODocTagsRepetitionStopper.should_stop..run_repetitive.. 2r!1:r c 3 2 ># U H oTS :H v M g7frI r )rJ yyss r rL rc rd re c 3 2 ># U H oTS :H v M g7frI r )rJ wwss r rL rc rd re c 3 2 ># U H oTS :H v M g7frI r )rJ hhss r rL rc rd re )rR setziprW r[ list)
r^ x_stabley_stablew_stableh_stablern rk rb rh r s
@@@@r run_repetitive.run_repetitive s 5zA~ 3u:U+ %[NBB2r22H2r22H2r22H2r22HH$--R*A*AH$--R*A*Ar tagtextra rg rj rm T)r tupler" rB finditergroupr# append)
r r
prev_tag prev_textrunrv mrx ry boxs
` r r $DocTagsRepetitionStopper.should_stopw s $# $u+ $ & ''*AAGGCL!AGGCL!AGGCL!AGGCL! C 9#4
3 "#&& &))e! +& c""r )r8 rB r= r< N)
r r r r r r# rC r r" r[ r! r r$ r r r r4 r4 P sK $&c
S
$ 8T#Y 84 82#S 2#T 2#r r4 )rS loggingr> r abcr copyr typingr r r r/ r getLoggerr _logr dictr! r# r" floatrq r2 r4 r r r r s
+ +-" $ (,!% ! ! $+/+/,*, CH~$, $J ,
d{ , d{
, , S /D(, S /D(, ,^Y# 0 Y#r