jSrSSKJr SSKrSSKJr SSKJr SSKJ r J r J r SSK J r SSKJr SSS jjrSSS jjrSSS jjrSS jrSSS jjrg)uoTTL-aware S3 ingestion cache. Registers a persistent-cache ``cloudpathlib.S3Client`` as the process-global default for ``s3://`` URIs, so any ``CloudPath("s3://...")`` / ``AnyPath("s3://...")`` materializes through a project-local cache dir instead of re-downloading every run. Source-change correctness is cloudpathlib's built-in ETag check; disk hygiene is a wall-clock TTL enforced by a single startup sweep. No subclass, no per-access hook. Two functions set up the cache. The cache dir, TTL, and (optional) region come from ``get_settings().s3_cache``, and both functions accept explicit overrides for tests: - ``evict_stale_s3_cache()`` — startup sweep. Walks the cache dir and unlinks any file whose access time is older than the TTL. This is the *only* TTL reclamation path: cloudpathlib's ETag check handles source correctness; the sweep handles "downloaded once, never touched again" files that would otherwise live in the cache forever. - ``init_s3_cache()`` — package-import entrypoint. Runs the sweep, then constructs and registers the default S3 client. Called once from ``quber/__init__.py``. Three more functions resolve inputs against it. ``resolve_document`` and ``resolve_artifacts`` read ``s3://`` sources through the cache. ``fetch_file`` copies a source to a caller-named path and bypasses the cache. TTL is keyed off ``st_atime``, so re-processing a cached document resets its window: actively used files stay warm, abandoned ones age out after the TTL. On ``noatime``-mounted volumes atime is frozen and TTL degrades to "time since download" — accepted rather than adding mount-detection complexity. ) annotationsN)Path)Sequence)AnyPathS3ClientS3Path)logger) get_settingsc[5RnUbUO URnUbUO URnUR 5(dg[ R "5U- nSnUR S5HOnUR5(dMUR5RU:dM:UR5 US- nMQ U$)aDelete cached files whose atime is older than ``ttl_seconds``. ``cache_dir`` / ``ttl_seconds`` default to ``get_settings().s3_cache`` when not passed explicitly. Walks ``cache_dir`` once with ``rglob('*')`` and unlinks regular files past the TTL. Returns the count evicted. Idempotent and safe to call when the cache directory does not exist yet (returns 0). Linear in the cached-file count; sub-second even at thousands. r*) r s3_cache cache_dir ttl_secondsexiststimerglobis_filestatst_atimeunlink)rrs3cutoffevictedps //home/mande/repo/quber/src/quber/files/cache.pyevict_stale_s3_cacher-s  B&2  I!,!8+bnnK      YY[; &FG __S ! 99;;1668,,v5 HHJ qLG" Ncj[5RnUbUO URnUbUO URn[ X5nU(a[ R "SX05 UR(a)SSKnURURS9n[USUS9nO [USS9nUR5 g)uaPrune the stale cache, then register the default S3 client. Called once at package import (see ``quber/__init__.py``). ``cache_dir`` / ``ttl_seconds`` default to ``get_settings().s3_cache``. Two ordered steps: 1. ``evict_stale_s3_cache()`` — bound disk usage for keys not re-accessed between runs. 2. Construct a persistent-cache ``S3Client`` rooted at ``cache_dir`` and publish it via ``set_as_default_client()`` so plain ``CloudPath("s3://...")`` / ``AnyPath("s3://...")`` picks it up. When ``s3_cache.region`` is set, the client is bound to a region-scoped boto3 session; otherwise region resolves from boto3's own chain. Constructing the client requires neither AWS credentials nor network access (boto3 resolves both lazily on first GET), so this is safe to run at import on any host, including CPU-only / credential-less dev environments. Nz,Evicted {} stale file(s) from S3 cache at {}r) region_name persistent)local_cache_dirfile_cache_mode boto3_session)r"r#) r rrrrr debugregionboto3Sessionrset_as_default_client)rrrrr'sessionclients r init_s3_cacher,Gs*  B&2  I!,!8+bnnK"9:G CWX yy--BII-6)\ahi)\R   "rc\[U5n[U[5(aUbUO[5RR nXR - UR- nUR5(a[R"SX#5 O[R"SX#5 [U5$)aCoerce a user-supplied path string to a local filesystem Path. Local inputs pass straight through. For ``s3://`` inputs, log an INFO line when the document is already present in the local cache (cache hit), then materialize it (cloudpathlib downloads on miss) and return the local Path. ``cache_dir`` defaults to ``get_settings().s3_cache.cache_dir``. The cache-hit check mirrors cloudpathlib's on-disk layout (``//``) using only public ``S3Path`` attributes. Note: cloudpathlib still revalidates the S3 ETag on access, so a logged hit means "present locally"; if the source object changed upstream, cloudpathlib transparently re-downloads the fresh copy. zS3 cache hit: {} served from {}z#S3 cache miss: downloading {} to {}) r isinstancerr rrbucketkeyrr infor%r)rawrdoccacheds rresolve_documentr5ls| #,C#v!*!6ILN L 9rcURRSSS9 [U5n[U[5(a([ R "SXS9 URU5 U$UR[U5R55 U$)uCopy a local path or s3:// URI to `dest`, bypassing the cache. Returns `dest`. Unlike `resolve_document`, the caller names the destination — used where a file must land at a specific staged path rather than wherever the cache keeps it. T)parentsexist_okzDownloading {src} -> {dest})srcdest) parentmkdirrr.rr r1 download_to write_bytesr read_bytes)r9r:src_paths r fetch_filerAsw KKdT2s|H(F## 1sFT" K c--/0 Krc U(d [S5e[U5n[U[5(d [ U5$UVs/sHn[ [ X4- 5U5PM nnUSR$s snf)ucCoerce an artifacts-directory string to a local directory Path. A local directory passes straight through. For an ``s3://`` prefix, each named artifact file is materialized through the persistent cache — exactly ``resolve_document`` per file, with the same hit/miss logging and ETag revalidation — and the returned Path is the local cache directory that mirrors the prefix, so a caller written against a directory of files (``ParseResult.load``) reads the cached copies in place. ``filenames`` must name every file the caller will read: only the named files are downloaded, nothing else under the prefix. zAresolve_artifacts: filenames must name at least one artifact filer) ValueErrorrr.rrr5strr;)r2 filenamesr directorynamefiless rresolve_artifactsrIsk \]] I i ( (CyLU VID c)"23Y ?IE V 8?? Ws!A4)NN)r Path | Noner float | Nonereturnint)rrJrrKrLNone)N)r2rDrrJrLr)r9rDr:rrLr)r2rDrEz Sequence[str]rrJrLr)__doc__ __future__rrpathlibrtypingr cloudpathlibrrrlogurur quber.settingsr rr,r5rArIrrrWs@# 22'" $ 6" $"#"#"# "#J2"r