geometrybase/hk-ipo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Run full HKEX document backfill

Browse Source 
Request:
- Remove small-batch guidance and execute the HKEX document archiver across all open T0/T1 sync tasks in one run.

Changes:
- Make archive_hkex_documents.py process every open T0/T1 ticker by default when --limit is omitted.
- Add per-ticker progress output and keep full refreshes moving if one ticker fails.
- Suppress noisy pypdf warnings during large official document extraction.
- Update archivist and README instructions to show the full-run command without batch notes.
- Archive official HKEXnews prospectus and allotment-results PDFs under repo-relative data/raw paths.
- Refresh hk_ipo.sqlite and CSV snapshots for parsed T0/T1 fields, source_refs, sync_runs, sync_tasks, and ticker_sync_state.

Execution:
- Ran .venv/bin/python scripts/archive_hkex_documents.py --as-of 2026-06-15T09:00:00Z.
- Selected 284 open T0/T1 tickers, processed 210 tickers, and archived 398 source files.
- Left 74 tickers as missing target docs because title search did not return target prospectus/allotment documents for this pass.

Verification:
- Parsed archivist scripts with Python ast.
- Confirmed README, archivist skill, and archiver script no longer contain batch guidance.
- Ran git diff --check.
- Checked SQLite integrity and DB/snapshot row counts.
- Verified 706 source_refs use relative local paths, all files exist, and SHA256 hashes match.

Next useful context:
- Current source_refs count is 706 and ipo_demand count is 134.
- Sync ledger now reports 414 complete, 1595 pending_due, and 42 pending_not_due states.
This commit is contained in:
geometrybase
2026-06-15 07:57:33 +00:00
parent 993d7b26fa
commit 9aab267f80
409 changed files with 1675530 additions and 4537 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data/snapshots/sync_runs.csv
+1
View File
@@ -7,4 +7,5 @@ sync_state_20260615T083000Z,hkex_document_archive,2026-06-15T08:30:00Z,2026-06-1
sync_state_20260615T083500Z,hkex_document_archive,2026-06-15T08:35:00Z,2026-06-15T08:35:00Z,2026-06-15T08:35:00Z,complete,Derived ticker sync state refreshed.
sync_state_20260615T084500Z,hkex_document_archive,2026-06-15T08:45:00Z,2026-06-15T08:45:00Z,2026-06-15T08:45:00Z,complete,Derived ticker sync state refreshed.
sync_state_20260615T085000Z,hkex_document_archive,2026-06-15T08:50:00Z,2026-06-15T08:50:00Z,2026-06-15T08:50:00Z,complete,Derived ticker sync state refreshed.
sync_state_20260615T090000Z,hkex_document_archive,2026-06-15T09:00:00Z,2026-06-15T09:00:00Z,2026-06-15T09:00:00Z,complete,Derived ticker sync state refreshed.
sync_state_seed_2026_06_15,bootstrap_state_refresh,2026-06-15T06:30:00Z,2026-06-15T06:30:00Z,2026-06-15T06:30:00Z,complete,Derived ticker sync state refreshed.
1 sync_run_id mode as_of started_at finished_at status notes
7 sync_state_20260615T083500Z hkex_document_archive 2026-06-15T08:35:00Z 2026-06-15T08:35:00Z 2026-06-15T08:35:00Z complete Derived ticker sync state refreshed.
8 sync_state_20260615T084500Z hkex_document_archive 2026-06-15T08:45:00Z 2026-06-15T08:45:00Z 2026-06-15T08:45:00Z complete Derived ticker sync state refreshed.
9 sync_state_20260615T085000Z hkex_document_archive 2026-06-15T08:50:00Z 2026-06-15T08:50:00Z 2026-06-15T08:50:00Z complete Derived ticker sync state refreshed.
10 sync_state_20260615T090000Z hkex_document_archive 2026-06-15T09:00:00Z 2026-06-15T09:00:00Z 2026-06-15T09:00:00Z complete Derived ticker sync state refreshed.
11 sync_state_seed_2026_06_15 bootstrap_state_refresh 2026-06-15T06:30:00Z 2026-06-15T06:30:00Z 2026-06-15T06:30:00Z complete Derived ticker sync state refreshed.