# Scripts Directory

Utility scripts for PDF preprocessing, docling VLM serving, and database import.

> **Status: transitional.** These scripts are slated to be either folded into the main `quber` package as proper subcommands or deprecated outright. New entrypoints should be exposed as `quber` CLI commands and wired up as `just` recipes — not added here. Run everything through `uv run`.

## Pipeline overview

The scripts compose into two operational lanes:

1. **PDF preprocessing → docling VLM extraction**
   `repair_pdf.py` → `split_pdf.py` → either `batch_process_pages.py` (single-shot, in-process) **or** `docling_fastapi_server.py` + `docling_client.sh` (long-running daemon).

2. **Loading analysis results into the RAG store**
   Main `quber analyze` produces `*_analysis.json` → `import_json_to_db.py` writes to Postgres+pgvector.

## Inventory

| Script | Purpose | Invocation |
|---|---|---|
| `docling_fastapi_server.py` | Long-running FastAPI daemon. Loads the docling VLM model **once** (~150 s init) and keeps it in VRAM; serves concurrent PDF→Markdown jobs via a thread pool. Endpoints: `POST /convert`, `POST /convert_batch`, `GET /health`, `GET /stats`. Default port **8900** (kept above the review-suite devserver's auto-increment range — see Port assignments below). | `uv run python scripts/docling_fastapi_server.py [--host 0.0.0.0] [--port 8900] [--workers 4]` |
| `docling_client.sh` | Curl wrapper that POSTs a PDF to the docling server and writes the returned markdown to stdout or a file. Reads `DOCLING_SERVER_URL` (default `http://localhost:8900`). | `./scripts/docling_client.sh <input.pdf> [output.md]` |
| `batch_process_pages.py` | Standalone batch runner — no server. Loads the VLM model once and processes a directory of PDFs in groups of 8 via `converter.convert_all()`. Pipeline enables `do_table_structure` and `do_ocr`. Tuned for 24 GB VRAM. | `uv run python scripts/batch_process_pages.py <input_dir> [output_dir]` |
| `split_pdf.py` | Splits a PDF into per-page PDFs (`page_001.pdf`, …) using `pypdf`. Feeds `batch_process_pages.py`. | `uv run python scripts/split_pdf.py <input_pdf> [output_dir]` |
| `repair_pdf.py` | Re-saves a PDF through PyMuPDF (`garbage=4, deflate=True, clean=True`) to fix malformed coordinates / bboxes that crash docling. Use as a pre-processor when docling fails on a source PDF. Output: `<stem>_repaired.pdf`. | `uv run python scripts/repair_pdf.py <input.pdf> [output.pdf]` |
| `import_json_to_db.py` | Imports `*_analysis.json` files (output of `quber analyze`) into Postgres+pgvector via `quber.db.importer`. Reads DB target from env (`get_engine()` — there is no `--db-url` flag). Optionally bootstraps the schema and prints stats. | `uv run python scripts/import_json_to_db.py <path> [--pattern '*_analysis.json'] [--init-db] [--drop] [--stats] [-v]` |

### `import_json_to_db.py` flags

| Flag | Effect |
|---|---|
| `<path>` (positional) | JSON file or directory of JSON files. |
| `--pattern` | Glob pattern for directory imports. Default `*_analysis.json`. |
| `--init-db` | Create the schema before importing. |
| `--drop` | Drop existing tables first. **Destructive — use with care.** |
| `--stats` | Print database statistics after import. |
| `--verbose` / `-v` | Enable DEBUG logging. |

## Port assignments

| Port range | Service |
|---|---|
| **8765–8800** | review-suite devserver — picks the **first free port in this range** at startup; the chosen port is written to `.plan-review/.devserver-port` so subsequent `/devserver` invocations reuse it. Don't hardcode anything else into this range. |
| **8900** | `docling_fastapi_server.py` default (configurable via `--port`). Kept above the review-suite range. |

> The review-suite devserver's start port is `8765` but it auto-increments on collision; co-located projects naturally drift to 8766, 8767, … Check each project's `.plan-review/.devserver-port` to see what it landed on.

## Prerequisites

- Python 3.13+ and `uv`
- For docling scripts: GPU with sufficient VRAM (≥24 GB recommended), `fastapi[standard]` (`uv pip install "fastapi[standard]"`)
- For `repair_pdf.py`: `pymupdf` (project dep)
- For `split_pdf.py`: `pypdf` (project dep)
- For `import_json_to_db.py`: PostgreSQL with the `pgvector` extension; schema per `docs/DATABASE_SETUP.md`; `DATABASE_URL` env var set

## Adding new functionality

Don't add new scripts here. Instead:

1. Add the entrypoint as a subcommand on the `quber` CLI under `src/quber/`.
2. Expose the recipe in the project `justfile` (e.g. `just <recipe>`).
3. Run via `uv run quber <subcommand>` or `just <recipe>`.

If a one-off script is genuinely needed (e.g. a migration helper), document it in this table and link the Jira ticket that scheduled its removal.