Request: Add archivist support for remembering which IPO archive stages have already been synced and which stages should be updated next. Changes: - Add sync_runs, ticker_sync_state, sync_tasks, and price_performance tables to the archive schema. - Add scripts/update_sync_state.py to derive per-ticker stage status and rebuild the next-sync task queue. - Export the new sync-state tables as Git-friendly CSV snapshots. - Document the incremental archive flow in the archivist skill and README. Verification: - Ran scripts/bootstrap_historical_data.py. - Ran scripts/update_sync_state.py with a deterministic as-of timestamp. - Checked SQLite integrity and DB-to-snapshot row counts with Python sqlite3. - Parsed Python scripts with ast.parse. - Ran git diff --check and checked for temporary SQLite/cache files. Next useful context: - Current derived queue has 2 open tasks for 06658 and 15 waiting_until_due tasks for future stages.
4.0 KiB
name: archivist description: Use for Hong Kong IPO fact archiving in this project: downloading or recording prospectuses, allotment results, listing facts, market data, source references, file hashes, SQLite updates, and CSV snapshots. Do not use for investment conclusions, subscription recommendations, score interpretation, or research memos.
HK IPO Archivist
Purpose
Maintain the project-local Hong Kong IPO evidence archive and structured fact database. This skill owns facts, sources, database updates, path hygiene, and reproducible snapshots.
It does not decide whether an IPO is worth subscribing for. Route judgment, scoring, prediction cards, review cards, and reports to analyst.
Project Storage Contract
Use repo-relative paths everywhere. Never store machine-specific absolute paths.
- Resolve the repo root at runtime, for example with
git rev-parse --show-toplevel. - Store paths without a leading
./. - Store paths with POSIX separators, such as
data/raw/06658/prospectus.pdf. - Store
path_base = "repo_root"when a table needs an explicit base. - Store
file_sha256for archived source files whenever practical.
Expected project layout:
data/
hk_ipo.sqlite
raw/
snapshots/
memos/
reports/
rules/
schema/
scripts/
references/
Responsibilities
- Archive primary source files under
data/raw/{ticker}/. - Record source references, URLs, as-of timestamps, relative paths, and hashes.
- Update embedded SQLite tables for IPO facts.
- Export Git-friendly CSV snapshots after database updates.
- Maintain
sync_runs,ticker_sync_state, andsync_tasksso repeated syncs know what is already archived and what remains pending. - Preserve raw source files; do not overwrite without first checking whether the contents changed.
- Label missing, stale, inconsistent, or estimated fields explicitly.
Boundaries
Do not write:
- Subscription decisions.
- Investment ratings.
- Scoring interpretations.
- Prediction cards.
- Review conclusions.
- Rule-change recommendations.
If a user asks for both data update and analysis, complete the archive/update step first, then hand the frozen as-of dataset to analyst.
Workflow
- Inspect current repo state and recent commits before changing files.
- Identify the IPO ticker, company, stage, and source documents needed.
- Save raw source files under
data/raw/{ticker}/using descriptive names. - Compute hashes for archived files.
- Insert or update structured facts in
data/hk_ipo.sqlite. - Record every source in the source reference table using repo-relative paths.
- Refresh sync state with
scripts/update_sync_state.pyafter fact updates. - Export key tables to
data/snapshots/for readable Git diffs. - Verify path rules, required fields, hash checks, sync state, and snapshot generation.
- Commit only the related archive/database/snapshot changes.
Incremental Sync State
Use ticker_sync_state as the per-ticker stage ledger and sync_tasks as the next-sync queue.
Stages:
T0_prospectusT1_allotmentT2_grey_marketD1D5D20D60
Status values:
complete: required facts or source files are archived.pending_not_due: the stage is expected in the future.pending_due: the stage is due and should be updated on the next sync.blocked: the missing data has no known resolution date or needs manual intervention.not_applicable: the stage does not apply.
Default incremental flow:
python3 scripts/update_sync_state.py
Then update only rows in sync_tasks whose task_status is open or blocked. Do not re-download existing source files unless the upstream source changed or the stored hash no longer matches.
Quality Checks
Before finishing, confirm:
- No stored local path is absolute.
- No stored local path starts with
./. - Raw files referenced by the database exist.
- Source hashes match current file contents.
- CSV snapshots reflect the database update.
sync_tasksreflects only missing or future work, not completed stages.- Any unavailable field is marked as a data gap rather than invented.