Add archivist incremental sync state
Request: Add archivist support for remembering which IPO archive stages have already been synced and which stages should be updated next. Changes: - Add sync_runs, ticker_sync_state, sync_tasks, and price_performance tables to the archive schema. - Add scripts/update_sync_state.py to derive per-ticker stage status and rebuild the next-sync task queue. - Export the new sync-state tables as Git-friendly CSV snapshots. - Document the incremental archive flow in the archivist skill and README. Verification: - Ran scripts/bootstrap_historical_data.py. - Ran scripts/update_sync_state.py with a deterministic as-of timestamp. - Checked SQLite integrity and DB-to-snapshot row counts with Python sqlite3. - Parsed Python scripts with ast.parse. - Ran git diff --check and checked for temporary SQLite/cache files. Next useful context: - Current derived queue has 2 open tasks for 06658 and 15 waiting_until_due tasks for future stages.
This commit is contained in:
Binary file not shown.
@@ -0,0 +1 @@
|
||||
performance_id,ticker,stage,source_id,as_of_date,open_price_hkd,high_price_hkd,low_price_hkd,close_price_hkd,return_pct,turnover_hkd_m,data_as_of,notes
|
||||
|
@@ -0,0 +1,2 @@
|
||||
sync_run_id,mode,as_of,started_at,finished_at,status,notes
|
||||
sync_state_seed_2026_06_15,bootstrap_state_refresh,2026-06-15T06:30:00Z,2026-06-15T06:30:00Z,2026-06-15T06:30:00Z,complete,Derived ticker sync state refreshed.
|
||||
|
@@ -0,0 +1,18 @@
|
||||
task_id,ticker,stage,task_type,task_status,due_date,data_gap_id,last_sync_run_id,updated_at,notes
|
||||
06658_D1,06658,D1,archive_price_performance,open,2026-06-15,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet.
|
||||
06658_T2_grey_market,06658,T2_grey_market,archive_grey_market_result,open,2026-06-15,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet.
|
||||
06675_T1_allotment,06675,T1_allotment,archive_allotment_results,waiting_until_due,2026-06-16,06675_allotment_results_pending_2026_06_15,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Allotment result facts are not archived yet.
|
||||
06675_D1,06675,D1,archive_price_performance,waiting_until_due,2026-06-17,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet.
|
||||
06675_T2_grey_market,06675,T2_grey_market,archive_grey_market_result,waiting_until_due,2026-06-17,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet.
|
||||
06658_D5,06658,D5,archive_price_performance,waiting_until_due,2026-06-19,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06675_D5,06675,D5,archive_price_performance,waiting_until_due,2026-06-21,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06106_T1_allotment,06106,T1_allotment,archive_allotment_results,waiting_until_due,2026-06-23,06106_allotment_results_pending_2026_06_15,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Allotment result facts are not archived yet.
|
||||
06106_D1,06106,D1,archive_price_performance,waiting_until_due,2026-06-24,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet.
|
||||
06106_T2_grey_market,06106,T2_grey_market,archive_grey_market_result,waiting_until_due,2026-06-24,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet.
|
||||
06106_D5,06106,D5,archive_price_performance,waiting_until_due,2026-06-28,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06658_D20,06658,D20,archive_price_performance,waiting_until_due,2026-07-04,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06675_D20,06675,D20,archive_price_performance,waiting_until_due,2026-07-06,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06106_D20,06106,D20,archive_price_performance,waiting_until_due,2026-07-13,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06658_D60,06658,D60,archive_price_performance,waiting_until_due,2026-08-13,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06675_D60,06675,D60,archive_price_performance,waiting_until_due,2026-08-15,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06106_D60,06106,D60,archive_price_performance,waiting_until_due,2026-08-22,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
|
@@ -0,0 +1,22 @@
|
||||
ticker,stage,status,required,due_date,completed_at,last_source_id,data_gap_id,last_sync_run_id,updated_at,notes
|
||||
06106,T0_prospectus,complete,1,2026-06-15,2026-06-15,06106_prospectus_candidate_2026_06_15,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Prospectus source and offering terms are archived.
|
||||
06106,T1_allotment,pending_not_due,1,2026-06-23,,,06106_allotment_results_pending_2026_06_15,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Allotment result facts are not archived yet.
|
||||
06106,T2_grey_market,pending_not_due,1,2026-06-24,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet.
|
||||
06106,D1,pending_not_due,1,2026-06-24,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet.
|
||||
06106,D5,pending_not_due,1,2026-06-28,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06106,D20,pending_not_due,1,2026-07-13,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06106,D60,pending_not_due,1,2026-08-22,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06658,T0_prospectus,complete,1,2026-06-05,2026-06-05,06658_prospectus_2026_06_05,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Prospectus source and offering terms are archived.
|
||||
06658,T1_allotment,complete,1,2026-06-12,2026-06-12,06658_allotment_results_2026_06_12,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Allotment result facts are archived.
|
||||
06658,T2_grey_market,pending_due,1,2026-06-15,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet.
|
||||
06658,D1,pending_due,1,2026-06-15,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet.
|
||||
06658,D5,pending_not_due,1,2026-06-19,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06658,D20,pending_not_due,1,2026-07-04,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06658,D60,pending_not_due,1,2026-08-13,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06675,T0_prospectus,complete,1,2026-06-09,2026-06-09,06675_prospectus_2026_06_09,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Prospectus source and offering terms are archived.
|
||||
06675,T1_allotment,pending_not_due,1,2026-06-16,,,06675_allotment_results_pending_2026_06_15,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Allotment result facts are not archived yet.
|
||||
06675,T2_grey_market,pending_not_due,1,2026-06-17,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet.
|
||||
06675,D1,pending_not_due,1,2026-06-17,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet.
|
||||
06675,D5,pending_not_due,1,2026-06-21,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06675,D20,pending_not_due,1,2026-07-06,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
06675,D60,pending_not_due,1,2026-08-15,,,,sync_state_seed_2026_06_15,2026-06-15T06:30:00Z,Price/performance source is not archived yet. Due date uses calendar days until trading-calendar support is added.
|
||||
|
Reference in New Issue
Block a user