geometrybase/hk-ipo
Private
Public Access
0
0
Fork 0
Code Activity

Run full HKEX document backfill

Browse Source 
Request:
- Remove small-batch guidance and execute the HKEX document archiver across all open T0/T1 sync tasks in one run.

Changes:
- Make archive_hkex_documents.py process every open T0/T1 ticker by default when --limit is omitted.
- Add per-ticker progress output and keep full refreshes moving if one ticker fails.
- Suppress noisy pypdf warnings during large official document extraction.
- Update archivist and README instructions to show the full-run command without batch notes.
- Archive official HKEXnews prospectus and allotment-results PDFs under repo-relative data/raw paths.
- Refresh hk_ipo.sqlite and CSV snapshots for parsed T0/T1 fields, source_refs, sync_runs, sync_tasks, and ticker_sync_state.

Execution:
- Ran .venv/bin/python scripts/archive_hkex_documents.py --as-of 2026-06-15T09:00:00Z.
- Selected 284 open T0/T1 tickers, processed 210 tickers, and archived 398 source files.
- Left 74 tickers as missing target docs because title search did not return target prospectus/allotment documents for this pass.

Verification:
- Parsed archivist scripts with Python ast.
- Confirmed README, archivist skill, and archiver script no longer contain batch guidance.
- Ran git diff --check.
- Checked SQLite integrity and DB/snapshot row counts.
- Verified 706 source_refs use relative local paths, all files exist, and SHA256 hashes match.

Next useful context:
- Current source_refs count is 706 and ipo_demand count is 134.
- Sync ledger now reports 414 complete, 1595 pending_due, and 42 pending_not_due states.
This commit is contained in:
geometrybase
2026-06-15 07:57:33 +00:00
parent 993d7b26fa
commit 9aab267f80
409 changed files with 1675530 additions and 4537 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.codex/skills/archivist/SKILL.md
+2 -4
View File
@@ -117,16 +117,14 @@ By default, exclude report rows without a numeric IPO offer price because transf
## HKEX Document Backfill
Use the document archiver to fill detailed T0/T1 facts from official HKEXnews PDFs in small batches:
Use the document archiver to fill detailed T0/T1 facts from official HKEXnews PDFs:
```bash
.venv/bin/python scripts/archive_hkex_documents.py --limit 5 --as-of YYYY-MM-DDTHH:MM:SSZ
.venv/bin/python scripts/archive_hkex_documents.py --as-of YYYY-MM-DDTHH:MM:SSZ
```
The script resolves HKEXnews stock IDs, archives prospectus and allotment-results PDFs under `data/raw/{ticker}/`, updates `source_refs`, parses high-confidence fields into `ipo_master`, `offering_terms`, and `ipo_demand`, exports snapshots, and refreshes sync state.
Prefer small batches. After each batch, inspect a few tickers for date/source mismatches before continuing, because HKEXnews title search can include historical, postponed, or supplemental offering documents for the same stock code.
## Quality Checks
Before finishing, confirm: