pyiz%SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK J r SSK J r SSKJ r SSK JrJr SSKJr SSKJrJrJr SS KJrJrJrJr SS KJrJr SS KJrJ r J!r!J"r" SS K#J$r$ SS K%J&r& \(aSSK'J(r( \RR"\*5r+Sr,Sr-/SQr.\/\0\1S'Sr2Sr3Sr4Sr5Sr6Sr7S\/\SS4Sjr8SSSSSSSS.SSS \9S!\9\ -S"\9S#\9S-S$\:S-S%\/\9\9-S-S&\/\9\9-S-S'\0S-S(\:S)\04S*jjr;"S+S,\Rx5r=\>\\4r?"S-S.5r@S/\@SSS \9S"\9S#\94 S0jrAS/\@S\>\=\/\?4S-4S1jrBS2\?SS4S3jrCS4\/\?SSS \9S"\9S#\9SS4 S5jrDS4\/\?SSS \9S"\9S#\9SS4 S6jrES4\/\?SSS \9S"\9S#\9SS4 S7jrF"S8S9\5rGS2\?S\G4S:jrHS;S\0S\/\?4S?jrJS@\9SS4SAjrKg)BN)datetime)Path)Lock) TYPE_CHECKINGAny)quote)CommitOperationAdd UploadInfo_fetch_upload_modes)LocalUploadFileMetadataLocalUploadFilePathsget_local_upload_pathsread_upload_metadata)DEFAULT_REVISION REPO_TYPES)DEFAULT_IGNORE_PATTERNS _format_sizefilter_repo_objectstqdm)is_xet_available) sha_fileobj)HfApi d) 2Kr}iiXiCOMMIT_SIZE_SCALEii'r r paths_listreturnc[RS5 [U5[:a+[R S[U5SS[SS35 SSKJn U"S5nUHn[UR5nURn[UR5S :wa[UR5OS nX&S ==S - ss'[US S 5H7upxUS:XaS OSRUS U5nX&SRU5 M9 M UR5H_upU S n [U S5n X-n U [ :dM(U S :XaSOU n[R SUSU SSU SSU SS[ SS3 5 Ma /n/nUH~nUR"R%5R&nUS- nU[(:aUR+URU45 MUU[,:dMaUR+URU45 M U(alSRSUS S55n[U5S:aS[U5S- S3OSn[R S[U5S[(S UUS!35 U(alSRS"US S55n[U5S:aS[U5S- S3OSn[R S[U5S#[,S$UUS%35 [RS&5 g )'aS Validate upload against repository limits and warn about potential issues. Args: paths_list: List of file paths to be uploaded Warns about: - Too many files in the repository (>100k) - Too many entries (files or subdirectories) in a single folder (>10k) - Files exceeding size limits (>20GB recommended, >200GB maximum) z/Running validation checks on files to upload...zYou are about to upload ,z. files. This exceeds the recommended limit of z files per repository. Consider: - Splitting your data into multiple repositories - Using fewer, larger files (e.g., parquet files) - See: https://huggingface.co/docs/hub/repositories-recommendationsr) defaultdictcS[5S.$)Nr)filessubdirs)seta/home/mande/repo/quber/.venv/lib/python3.13/site-packages/huggingface_hub/_upload_large_folder.py)_validate_upload_limits..Us qUXUZ=[r..r*r N/r+rootzFolder 'z ' contains z entries (z files and z/ subdirectories). This exceeds the recommended z< entries per folder. Consider reorganising into sub-folders.iʚ;z - c38# UHupUSUSS3v M g7fz: z.1fGBNr-.0pathsizes r/ *_validate_upload_limits..s#!aL`jdTF"T#Jb"9L`z ... and z more filesFound z files exceeding the zGB recommended maximum: - z4 Consider splitting these files into smaller chunks.c38# UHupUSUSS3v M g7fr7r-r9s r/r=r>s!!\OjdTF"T#Jb"9Or?z files larger than zGB (recommended limit): - z2 Large files may slow down loading and processing.zValidation checks complete.)loggerinfolenMAX_FILES_PER_REPOwarning collectionsr(r path_in_repopartsstrparent enumeratejoinadditemsMAX_FILES_PER_FOLDER file_pathstatst_sizeMAX_FILE_SIZE_GBappendRECOMMENDED_FILE_SIZE_GB)r$r(entries_per_folderpathsr;rKrMichildfolderdata file_count subdir_count total_entriesfolder_display large_filesvery_large_filesr<size_gb files_strmore_strs r/_validate_upload_limitsrh8s> KKAB :++&s:q&9:55G4JKT U ()45[)\E&&' &)%5%<T[[!#"7+q0+"%*-HAFSr(;F  &y 1 5 5e <.+002 '] 4 ?+ "1 / /'-}V&N NN>*+mA5FGq>\!,<=00DQ/GH:: 3K##%--& % %  # #U%7%7$A B / /    2 2G< =MM!aL\]_^_L`!aa LOP`LadeLe\#&6"7!";!54PM nn[AUU5n[CU5Vs/sH"n[DRF"[HUUUUUS.S9PM$ nnUHnURK5 M U (a[MSURO5-5 [PRP"5n[PRR"S5 [PRP"5U- U :a5U (a[UURO55 [PRP"5nURW5(a[RS5 OMUHnURY5 M [RURO55 [RS 5 g![(a#n[R+SU35 SnAGNqSnAff=fs snfs snfs snf)!zUpload a large folder to the Hub in the most resilient way possible. See [`HfApi.upload_large_folder`] for the full documentation. NzFor large uploads, `repo_type` is explicitly required. Please set it to `model`, `dataset` or `space`. If you are using the CLI, pass it as `--repo-type=model`.z"Invalid repo type, must be one of zProvided path: 'z' is not a directoryr T)rrrtrkexist_okzRepo created: )rrrtrji zP================================================================================z WARNING: This repository has aR commits. Repositories with a large number of commits can experience performance issues. Consider squashing your commit history using `super_squash_history()`. To do so, you need to stop this process, run the snippet below and restart the upload command. from huggingface_hub import super_squash_history super_squash_history(repo_id='z', repo_type='z') Note: This is a non-revertible operation. See the documentation for more details: https://huggingface.co/docs/huggingface_hub/main/en/package_reference/hf_api#huggingface_hub.HfApi.super_squash_history zCould not check commit count: c3># UH:oR5(dMURT5R5v M< g7fN)is_file relative_toas_posix)r:r;rss r/r=/upload_large_folder_internal..s6i>VdZfZfZh 1  + & / / 1 1>Vs A&Az**/*)rlrmrBz candidate files to uploadzStarting upload...zRecovering from metadata files)desc)statusrqrrrtrj)targetkwargsz zIs done: exiting main loopzUpload is complete!)- ValueErrorrrr expanduserresolveis_dir isinstancerLros cpu_countmax create_reporDrErrlist_repo_commitsrFrH ExceptiondebugrUPLOAD_BATCH_SIZE_XETUPLOAD_BATCH_SIZE_LFSrglobrrhrrrJLargeUploadStatusrange threadingThread _worker_jobstartprintcurrent_reporttimesleep_print_overwriteis_donerO)rqrrrsrtrjrkrlrmrnrorpnb_coresrepo_urlcommits commit_counteupload_batch_sizefiltered_paths_listrelpathr$rZrQr_threadsthreadlast_report_tss ` r/upload_large_folder_internalrs& I   "=j\JKK#{#..088:K     +K=8LMNN OS ) )*+..O<<>&Q(a-+ wW_cdH KK. +,G;''W_'`7| #  NNXJ00<~>3 4;)>)UL*B   &2B1C1C-I^.ik>N>Nv>Vi%' ObbNa7(g>NaJb KK&Z))CDEJ' KK$% *+KL LE $[%2D2DEFL   u&7 8F{# $A  "&$  $   fv,,../YY[N  1 99; '+= = !6!6!89!YY[N >>   KK4 5    KK%%'( KK%&A ; 5aS9::;c  s+ ANO <"O9)O O &OO c\rSrSr\R "5r\R "5r\R "5r\R "5r \R "5r Sr g) WorkerJobi#r-N) __name__ __module__ __qualname____firstlineno__enumautoSHA256GET_UPLOAD_MODE PREUPLOAD_LFSCOMMITWAIT__static_attributes__r-r.r/rr#s9 YY[FiikOIIKM YY[F 99;Dr.rcx\rSrSrSrSS\\S\4SjjrS\4Sjr S\ S \S \ SS 4S jr S\ 4S jrS\ 4SjrSrg )ri.zBContains information, queues and tasks for a large upload process.rQrcXl[R"5Ul[R"5Ul[R"5Ul[R"5Ul[5UlSUl SUl SUl X l SUl SUlSUl[ R""5UlSUl[5UlURHnUupEUR*cURR-U5 M1UR.cURR-U5 M[UR.S:Xa.UR0(dUR R-U5 MUR2(dUR R-U5 M[4R7SUR8S35 M g)Nrr lfszSkipping file z! (already uploaded and committed))rQqueueQueue queue_sha256queue_get_upload_modequeue_preupload_lfs queue_commitrlocknb_workers_sha256nb_workers_get_upload_modenb_workers_preupload_lfsrnb_workers_commitnb_workers_waitinglast_commit_attemptrnow _started_at _chunk_idx _chunk_locksha256put upload_mode is_uploaded is_committedrDrrJ)selfrQritemrZmetadatas r/__init__LargeUploadStatus.__init__1sV 7<{{}@E ">Ckkm 7<{{}F &'/0'-.%&7&''(15 #<<> !%JJD"OE&!!%%d+%%-**..t4%%.x7K7K((,,T2**!!%%d+ ~e.@.@-AAbcdr.r%cvUR [URsSSS5 $!,(df  g=frz)rr"rrs r/ target_chunkLargeUploadStatus.target_chunkSs#   $T__5  s* 8successnb_itemsdurationNc UR U(d/[RSUS35 U=RS-slOKU[UR:a4US:a.[R SUS35 U=RS- sl[ S[UR[[5S- 55UlSSS5 g!,(df  g=f)NzFailed to commit z9 files at once. Will retry with less files in next batch.r (zSuccessfully committed z. at once. Increasing the limit for next batch.r) rrDrHrr"rErminrF)rrrrs r/ update_chunkLargeUploadStatus.update_chunkWs   !28*O:PST:T%UVDO  s B7C  Cc SnSnSnSnSnSnSnSnSn Sn Sn UR URHupU R(aU S- n MXR- n U S- n U RbUS- nX-R- nU R S:XaUS- nU R cUS- nU R (aUS- nXmR- nU R(dMUS- nXR- nM [U 5n[R"5nURS5nXR- n[U5RS5SnSnUSUS US 3- nUS - nUS - nUS USU S [U5SUS3 - nUSUSUS [U5SUS3 - nUS:a USUS3- nUSUSU S [U5SUS3 - nUSU S3- nUS- nUSURS3- nUSUR S3- nUSUR"S3- nUSUR$S3- nUSUR&S3- nUS- nUsSSS5 $!,(df  g=f)z >G *4+J+J*K3O OG )F)F(GsK KG d&<&<%=SA AG 4#:#:";2> >G x G_YYsB7IEI IcUR [SUR55sSSS5 $!,(df  g=f)Nc3d# UH&upUR=(d URv M( g7frzrrr:rrs r/r=,LargeUploadStatus.is_done..s(eZd;1x,,F0F0FFZd.0)rallrQrs r/rLargeUploadStatus.is_dones( YYeZ^ZdZdeeYYs 3 A)rrrrQrrrrrrrrrrrr)r )rrrr__doc__list JOB_ITEM_TintrrboolfloatrrLrrrr-r.r/rr.snL ed:. e3 eD6c6 WD WC W5 WT W==~ffr.rrc Sn[U5nUcgUupgU=[R:XaY USn[U5 URR U5 UR U=RS-slSSS5 GOS=[R:Xa [!XqX#US9 UHnUupU R"(aMU R$=S:Xa UR&R U5 MIS :XaUR(R U5 MkURR U5 M UR U=R*S-slSSS5 GOy=[R,:Xa] [/XqX#US9 UHnUR(R U5 M UR U=R0S-slSSS5 GO=[R2:Xa [4R4"5n Sn [7XqX#US9 [4R4"5U - nUR9U [;U5U5 UR [4R4"5UlU=R>S-slSSS5 OW[R@:XaD[4RB"[D5 UR U=RFS-sl#SSS5 GM![ a e[aSn [RSU 35 [R"5 URR U5 Sn A GNSn A ff=f!,(df  M|=f![ a e[a8n [RSU 35 [R"5 Sn A GNSn A ff=f!,(df  M=f![ a e[a\n [RS U 35 [R"5 UHnUR&R U5 M Sn A GNSn A ff=f!,(df  GMc=f![ a e[a^n [RS U 35 [R"5 UHnUR(R U5 M S n Sn A GNSn A ff=f!,(df  GM=f!,(df  GM=f) a Main process for a worker. The worker will perform tasks based on the priority list until all files are uploaded and committed. If no tasks are available, the worker will wait for 10 seconds before checking again. If a task fails for any reason, the item(s) are put back in the queue for another worker to pick up. Read `upload_large_folder` docstring for more information on how tasks are prioritized. TNrzFailed to compute sha256: r )rqrrrtrjzFailed to get upload mode: rregularzFailed to preupload LFS: zFailed to commit: F)$_determine_next_jobrr_compute_sha256rrKeyboardInterruptrrDerror traceback format_excrrrr_get_upload_moderrrrrr_preupload_lfsrrr_commitrrFrrrrWAITING_TIME_IF_NO_TASKSr)rrqrrrtrjnext_jobjobrQrrrrstart_tsrrs r/rrs >B'v.    !!!Qx2#D)0044T:[[,,1,![+**+$UWdlm"D"&KA-- "..""66::4@&"//33D9"88<qc!BC((**+*![) =LL#0R3:SM/AL==M MN#+-NN#& N58P( AP##P(+ P;>R0AR++R03 S ScBUR URS:XaURR5S:aURb[ R "5UR- S:agU=RS- sl[ RS5 [R[URUR554sSSS5 $URS:XaURR5S:agU=RS- sl[ RS5 [R[URUR554sSSS5 $URR5[:aeU=RS- sl[ RS[S35 [R[UR[54sSSS5 $UR R5UR":asUR$S:XacU=R$S- sl[ RS 5 [R&[UR UR"54sSSS5 $UR(R5S:ahUR*S:XaXU=R*S- sl[ RS 5 [R,[/UR(54sSSS5 $URR5S:amURS:Xa]U=RS- sl[ RS 5 [R[UR[54sSSS5 $UR R5UR":acU=R$S- sl[ RS 5 [R&[UR UR"54sSSS5 $UR(R5S:aXU=R*S- sl[ RS 5 [R,[/UR(54sSSS5 $URR5S:a]U=RS- sl[ RS5 [R[UR[54sSSS5 $UR R5S:acU=R$S- sl[ RS 5 [R&[UR UR"54sSSS5 $URS:XaURR5S:aURb[ R "5UR- S:agU=RS- sl[ RS5 [R[URUR554sSSS5 $URS:XGaURR5S:aUR(R5S:XaURR5S:XaUR R5S:XaUR*S:XaURS:XawUR$S:XagU=RS- sl[ RS5 [R[URUR554sSSS5 $[1SUR255(a[ R5S5 SSS5 gU=R6S- sl[ RS[8S35 [R:/4sSSS5 $!,(df  g=f)Nri,r z;Job: commit (more than 5 minutes since last commit attempt)zJob: commit (>100 files ready)zJob: get upload mode (>z files ready)z5Job: preupload LFS (no other worker preuploading LFS)z.Job: sha256 (no other worker computing sha256)z:Job: get upload mode (no other worker getting upload mode)zJob: preupload LFSz Job: sha256zJob: get upload moderiz-Job: commit (1 min since last commit attempt)z Job: commitc3d# UH&upUR=(d URv M( g7frzrrs r/r=&_determine_next_job..ts(aT`[Q&&@(*@*@@T`rz.All files have been processed! Exiting worker.zNo task available, waiting... (zs))rrrqsizerrrDrrr_get_nrrMAX_NB_FILES_FETCH_UPLOAD_MODErrrrrrrrr_get_onerrQrErrr)rs r/r r s$   $ $ )##))+a/**6 f8886A  $ $ ) $ LLV W$$fV-@-@&BUBUBW&XY  % % *v/B/B/H/H/Jc/Q  $ $ ) $ LL9 :$$fV-@-@&BUBUBW&XY! & ) ) / / 15S S  - - 2 - LL23Q2RR_` a--vf6R6RTr/st- 2 ' ' - - /63K3K KPVPoPostPt  + +q 0 + LLP Q++VF4N4NPVPhPh-ij9 > & & (1 ,1I1IQ1N  $ $ ) $ LLI J$$hv/B/B&CDE J ) ) / / 1A 5&:[:[_`:`  - - 2 - LLU V--vf6R6RTr/stQ V ' ' - - /63K3K K  + +q 0 + LL- .++VF4N4NPVPhPh-ij] b & & (1 ,  $ $ ) $ LL '$$hv/B/B&CDi n ) ) / / 1A 5  - - 2 - LL/ 0--vf6R6RTr/stu z ' ' - - /! 3  + +q 0 + LL- .++VF4N4NPVPhPh-ijA H  $ $ )##))+a/**6 f8886A  $ $ ) $ LLH I$$fV-@-@&BUBUBW&XYW `  $ $ )##))+a/##))+q0,,2249**002a7((A-11Q6//14  $ $ ) $ LL '$$fV-@-@&BUBUBW&XYw |aTZT`T`a a a KKH IA H  % % * % LL:;S:TTVW XNNB'M sfB?`B `+A=`2B` A<`B`B`)A,`A1`A7`B>`#D`7`A` `rcUupURcBURRS5n[U5R 5UlSSS5 UR U5 g!,(df  N =f)z1Compute sha256 of a file and save it in metadata.Nrb)rrSopenrhexsave)rrZrfs r/r r sUOE __ ! !$ '1)!n002HO( MM%( 's A&& A4rQc XUVs/sHn[U5PM nn[UUUUR5[USS9URS9 [ X5HMupWUupUR U lURU l URU l U RU5 MO gs snf)zeGet upload mode for each file and update metadata. Also receive info if the file should be ignored. rA)safe) additionsrtrrheadersrjendpointN) _build_hacky_operationr _build_hf_headersrr)zip _upload_moder_should_ignorer _remote_oid remote_oidr#) rQrqrrrtrjrr'additionrZrs r/rrs ;@@%$'-%I@%%'xb) e/'44!)!8!8&22 e 0AsB'cUVs/sHn[U5PM nnURUUUUS9 UHupxSUlURU5 M gs snf)z(Preupload LFS files and update metadata.)rrrtrjr'TN)r*preupload_lfs_filesrr# rQrqrrrtrjrr'rZrs r/rrsc:?@%$'-%I@ !# e!AsAcUVs/sHn[U5PM nnURUUUUSS9 UHupxSUlURU5 M gs snf)zCommit files to the repo.z(Add files using upload-large-folder tool)rrrtrj operationscommit_messageTN)r* create_commitrr#r4s r/rrsf:?@%$'-%I@A ! $ e!AsAc\rSrSrSSjrSrg)HackyCommitOperationAddiNcx[UR[5(a[UR5Ulggrz)rpath_or_fileobjrrLrs r/ __post_init__%HackyCommitOperationAdd.__post_init__s- d**D 1 1#&t';';#!=AIIK!= >> >sAreportc,^US- n[R"5Rm[U4SjUR 555n[ U5HAn[ RRS5 [ RRS5 MC [ RRU5 [ RRST[UR 5S5- -5 [ RR5 g)zPrint a report, overwriting the previous lines. Since tqdm in using `sys.stderr` to (re-)write progress bars, we need to use `sys.stdout` to print the report. Note: works well only if no other process is writing to `sys.stdout`! rxc3F># UHn[U5T-S-v M g7f)r N)rF)r:lineterminal_widths r/r=#_print_overwrite..s!S?Rt3t9.2?Rs!z zrr3N) shutilget_terminal_sizecolumnssum splitlinesrsysstdoutwriterFflush)rLnb_linesrrPs @r/rrs dNF--/77NSv?P?P?RSSH8_ $ " JJVJJSNS1B1B1DR1H-IIJKJJr.)LrloggingrrrRrWrrrrpathlibrrtypingrr urllib.parser _commit_apir r r _local_folderr rrr constantsrrutilsrrrrutils._runtimer utils.sharhf_apir getLoggerrrDrrr"rr__annotations__rrrGrRrVrXrhrLrrEnumrtuplerrrr r rrrr:r*rrrr-r.r/rks<   %LLvv3SS,"   8 $!$O49O]/-A(B]/t]/L -1.2" C' C' C'tC'  C' Dj C'D[C'IOd*C'#Y_t+C'tC'C'C'V ')@@ A ufufpl3 l3 l3l3 l3  l3^g( 1g(eItJGW