Add intelligent PDF processing that detects whether a PDF is text-based or scanned before choosing an extraction method. Text-based PDFs (with ≥80% confidence) are now extracted locally using the Rust pdf-inspector library. Optimize the flow by deferring base64 encoding until MinerU is actually needed, and add structured logging for monitoring extraction methods.

Summary by cubic

Adds smart PDF routing that inspects each file and picks the fastest extractor. Text PDFs (≥80% confidence) are extracted locally in Rust; scanned/image PDFs fall back to MinerU (RunPod) or pdf-parse.

• New Features

  • Added detectPdfType and extractPdfToMarkdown via Rust pdf-inspector with N-API bindings and structured logging.
  • Routed extraction in the scraper: local (text, ≥0.8) → RunPod MU → pdf-parse, with Sentry capturing failures and a final summary log of the chosen method.

• Performance

  • Skips GPU/OCR for text PDFs with fast local markdown extraction.
  • Defers base64 encoding until MU is needed and checks size before MU to avoid oversized uploads; added per-stage timing for monitoring.

^{Written for commit7ad32b3. Summary will update on new commits.}