o ^v(j@sdZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl m Z   ddddZ  ddddZddddZdS)uTTL-aware S3 ingestion cache (QUE-223). Registers a persistent-cache ``cloudpathlib.S3Client`` as the process-global default for ``s3://`` URIs, so any ``CloudPath("s3://...")`` / ``AnyPath("s3://...")`` materializes through a project-local cache dir instead of re-downloading every run. Source-change correctness is cloudpathlib's built-in ETag check; disk hygiene is a wall-clock TTL enforced by a single startup sweep. No subclass, no per-access hook — one module, two functions. The cache dir, TTL, and (optional) region come from ``get_settings().s3_cache`` (QUE-228); each function still accepts explicit overrides for tests: - ``evict_stale_s3_cache()`` — startup sweep. Walks the cache dir and unlinks any file whose access time is older than the TTL. This is the *only* TTL reclamation path: cloudpathlib's ETag check handles source correctness; the sweep handles "downloaded once, never touched again" files that would otherwise live in the cache forever. - ``init_s3_cache()`` — package-import entrypoint. Runs the sweep, then constructs and registers the default S3 client. Called once from ``quber/__init__.py``. TTL is keyed off ``st_atime``, so re-processing a cached document resets its window: actively used files stay warm, abandoned ones age out after the TTL. On ``noatime``-mounted volumes atime is frozen and TTL degrades to "time since download" — accepted rather than adding mount-detection complexity. ) annotationsN)Path)AnyPathS3ClientS3Path)logger) get_settings cache_dir Path | None ttl_seconds float | NonereturnintcCstj}|dur |n|j}|dur|n|j}|sdSt|}d}|dD]}|r>|j |kr>| |d7}q)|S)aDelete cached files whose atime is older than ``ttl_seconds``. ``cache_dir`` / ``ttl_seconds`` default to ``get_settings().s3_cache`` when not passed explicitly. Walks ``cache_dir`` once with ``rglob('*')`` and unlinks regular files past the TTL. Returns the count evicted. Idempotent and safe to call when the cache directory does not exist yet (returns 0). Linear in the cached-file count; sub-second even at thousands. Nr*) rs3_cacher r existstimerglobis_filestatst_atimeunlink)r r s3cutoffevictedpr//home/mande/repo/quber/src/quber/files/cache.pyevict_stale_s3_cache(s  rNonecCstj}|dur |n|j}|dur|n|j}t||}|r$td|||jr:ddl}|j |jd}t |d|d}nt |dd}| dS)uPrune the stale cache, then register the default S3 client. Called once at package import (see ``quber/__init__.py``). ``cache_dir`` / ``ttl_seconds`` default to ``get_settings().s3_cache``. Two ordered steps: 1. ``evict_stale_s3_cache()`` — bound disk usage for keys not re-accessed between runs. 2. Construct a persistent-cache ``S3Client`` rooted at ``cache_dir`` and publish it via ``set_as_default_client()`` so plain ``CloudPath("s3://...")`` / ``AnyPath("s3://...")`` picks it up. When ``s3_cache.region`` is set, the client is bound to a region-scoped boto3 session; otherwise region resolves from boto3's own chain. Constructing the client requires neither AWS credentials nor network access (boto3 resolves both lazily on first GET), so this is safe to run at import on any host, including CPU-only / credential-less dev environments. Nz,Evicted {} stale file(s) from S3 cache at {}r) region_name persistent)local_cache_dirfile_cache_mode boto3_session)r#r$) rrr r rrdebugregionboto3Sessionrset_as_default_client)r r rrr(sessionclientrrr init_s3_cacheBs   r-rawstrrcCslt|}t|tr2|dur|ntjj}||j|j}|r+t d||t |St d||t |S)aCoerce a user-supplied path string to a local filesystem Path. Local inputs pass straight through. For ``s3://`` inputs, log an INFO line when the document is already present in the local cache (cache hit), then materialize it (cloudpathlib downloads on miss) and return the local Path. ``cache_dir`` defaults to ``get_settings().s3_cache.cache_dir``. The cache-hit check mirrors cloudpathlib's on-disk layout (``//``) using only public ``S3Path`` attributes. Note: cloudpathlib still revalidates the S3 ETag on access, so a logged hit means "present locally"; if the source object changed upstream, cloudpathlib transparently re-downloads the fresh copy. NzS3 cache hit: {} served from {}z#S3 cache miss: downloading {} to {}) r isinstancerrrr bucketkeyrrinfor&r)r.r doccachedrrrresolve_documentgs r6)NN)r r r r r r)r r r r r r )N)r.r/r r r r)__doc__ __future__rrpathlibr cloudpathlibrrrlogururquber.settingsrrr-r6rrrrs      %