Make PDF text extraction a standard archive step

Browse Source

Request:
- Add extracted PDF text generation to the archivist workflow as a standard step.

Changes:
- Run PDF text extraction automatically for newly archived HKEX PDF sources.
- Make the PDF text extractor incremental and manifest-preserving.
- Document extracted-text handling in the archivist skill and README.
- Mark generated extracted text as no-diff data evidence.
- Backfill extracted text for all archived PDF source references.

Verification:
- Ran git diff --cached --check.
- Ran .venv/bin/python -m py_compile scripts/extract_pdf_text.py scripts/archive_hkex_documents.py.
- Ran full PDF extraction, then confirmed an incremental rerun skips unchanged files.
- Verified 557 PDF source_refs, 557 manifest rows, all status ok, and zero missing text/hash/path issues.

Next useful context:
- HKEX HTML notices and Yahoo JSON market data remain under data/raw and are not expected in data/extracted_text.

...

This commit is contained in:

geometrybase

2026-06-15 13:27:41 +00:00

parent 48b89552fe

commit 8a0dfd88f0

557 changed files with 8721479 additions and 28 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

data/extracted_text/02652/allotment_results_2025-10-06_2025100601430.txt

+2253

File diff suppressed because it is too large Load Diff

data/extracted_text/02652/prospectus_2025-09-26_2025092600041.txt

+30273

File diff suppressed because it is too large Load Diff