init push
This commit is contained in:
187
docs/data-migration-wiki-ckeditor.html
Normal file
187
docs/data-migration-wiki-ckeditor.html
Normal file
@@ -0,0 +1,187 @@
|
||||
<h1>Data Migration Wiki</h1>
|
||||
<p>
|
||||
This page is the living technical document for migration scope, decisions, limitations, and open tasks.
|
||||
Update it for every migration wave.
|
||||
</p>
|
||||
|
||||
<h2>Scope</h2>
|
||||
<ul>
|
||||
<li>Database schema migrations in <code>migrations/</code>.</li>
|
||||
<li>Data migration SQL (legacy to current domain model).</li>
|
||||
<li>Non-DB migration concerns: binary files (PDF/images), queue backlog, manual user actions.</li>
|
||||
</ul>
|
||||
|
||||
<h2>Source of Truth</h2>
|
||||
<ul>
|
||||
<li>Versioned DB migrations: <code>migrations/*.sql</code>.</li>
|
||||
<li>Legacy helicopter import flow:
|
||||
<ul>
|
||||
<li><code>migrations/goose/20260414_helicopter_import_staging.sql</code></li>
|
||||
<li><code>migrations/goose/20260415_helicopter_import_apply.sql</code></li>
|
||||
<li><code>migrations/scripts/migration_helicopter.sql</code> (deprecated helper note)</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>Queue migration notes: <code>docs/queue-sqs-migration.md</code>.</li>
|
||||
<li>File manager APIs and attachment model:
|
||||
<ul>
|
||||
<li><code>docs/file-manager-upload.md</code></li>
|
||||
<li><code>docs/file-manager-attachment.md</code></li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<h2>Implemented Data Migrations</h2>
|
||||
<ol>
|
||||
<li>
|
||||
<strong>20260326_move_responsible_flight_rescuer_to_air_rescuer_profiles.sql</strong>
|
||||
<ul>
|
||||
<li>Copies <code>pilot_profiles.responsible_flight_rescuer</code> into <code>air_rescuer_profiles.responsible_flight_rescuer</code>.</li>
|
||||
<li>Drops old column from <code>pilot_profiles</code>.</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
<strong>20260403_file_manager_file_lifecycle_phase1.sql</strong>
|
||||
<ul>
|
||||
<li>Normalizes legacy file lifecycle states (<code>pending/uploading</code>) into newer states (<code>validated/processing</code>).</li>
|
||||
<li>Carries <code>upload_error</code> into <code>failure_reason</code>.</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
<strong>20260415_helicopter_import_apply.sql</strong>
|
||||
<ul>
|
||||
<li>Imports rows from <code>helicopter_old_import</code> into <code>helicopters</code>.</li>
|
||||
<li>Maintains old-to-new ID mapping via <code>helicopter_old_id_map</code>.</li>
|
||||
<li>Explicitly non-reversible (<code>Down</code> keeps data as audit trail).</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ol>
|
||||
|
||||
<h2>Known Migration Decisions and Gaps</h2>
|
||||
|
||||
<h3>Confirmed "Will Not Migrate" (Current Decision)</h3>
|
||||
<ol>
|
||||
<li>
|
||||
Legacy helicopter columns intentionally ignored:
|
||||
<ul>
|
||||
<li><code>counter</code></li>
|
||||
<li><code>service</code></li>
|
||||
<li><code>disabled</code></li>
|
||||
<li><code>visible</code></li>
|
||||
</ul>
|
||||
<p>Decision is documented in <code>migrations/scripts/migration_helicopter.sql</code>.</p>
|
||||
</li>
|
||||
</ol>
|
||||
|
||||
<h3>Confirmed "Not Migrated Yet" (Open)</h3>
|
||||
<ol>
|
||||
<li>
|
||||
Legacy SQS queue backlog
|
||||
<ul>
|
||||
<li>Current worker does not consume old SQS backlog.</li>
|
||||
<li>Required action: drain/discard SQS backlog before/at cutover.</li>
|
||||
<li>Source: <code>docs/queue-sqs-migration.md</code>.</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
Binary files (PDF/images) into new file manager storage
|
||||
<ul>
|
||||
<li>Current migrations create file manager tables and attachment relations, but no SQL migration imports binary objects or bulk-inserts legacy metadata into <code>file_files</code>/<code>attachments</code>.</li>
|
||||
<li>This means legacy documents may need manual re-upload unless a dedicated import job is created.</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
Legacy <code>photo_file_id</code> references in contact profile tables
|
||||
<ul>
|
||||
<li>Contact profile schemas still include <code>photo_file_id</code> string columns.</li>
|
||||
<li>No migration currently maps these into <code>attachments</code> + <code>image_attachment_id</code>/<code>foto_attachment_id</code>.</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ol>
|
||||
|
||||
<h2>Manual Actions Required at Cutover</h2>
|
||||
<ol>
|
||||
<li>Validate which legacy PDFs/images must remain accessible on day 1.</li>
|
||||
<li>For files without automated import, perform manual re-upload via file manager flow.</li>
|
||||
<li>Re-link uploaded files to business references (attachments) and to entity-level attachment columns where relevant.</li>
|
||||
<li>Obtain client sign-off for any files intentionally left unmigrated.</li>
|
||||
</ol>
|
||||
|
||||
<h2>Migration TODO Register</h2>
|
||||
<table border="1" cellpadding="6" cellspacing="0">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>ID</th>
|
||||
<th>Item</th>
|
||||
<th>Type</th>
|
||||
<th>Status</th>
|
||||
<th>Owner</th>
|
||||
<th>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>MIG-001</td>
|
||||
<td>Decide final strategy for legacy PDFs/images: automated import vs manual re-upload</td>
|
||||
<td>Decision</td>
|
||||
<td>Open</td>
|
||||
<td>TBD</td>
|
||||
<td>Blocks final cutover checklist</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MIG-002</td>
|
||||
<td>Produce inventory of all legacy file references and classify critical/non-critical</td>
|
||||
<td>Analysis</td>
|
||||
<td>Open</td>
|
||||
<td>TBD</td>
|
||||
<td>Include PDFs + photos</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MIG-003</td>
|
||||
<td>If automation chosen, implement importer to populate <code>file_files</code> and <code>attachments</code> safely</td>
|
||||
<td>Engineering</td>
|
||||
<td>Open</td>
|
||||
<td>TBD</td>
|
||||
<td>Prefer idempotent batch import with audit log</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MIG-004</td>
|
||||
<td>Define mapping from legacy contact <code>photo_file_id</code> to new attachment model</td>
|
||||
<td>Engineering</td>
|
||||
<td>Open</td>
|
||||
<td>TBD</td>
|
||||
<td>Covers pilot/doctor/air rescuer/technician/FA/staff</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MIG-005</td>
|
||||
<td>Document and execute SQS backlog drain/discard plan before production cutover</td>
|
||||
<td>Ops</td>
|
||||
<td>Open</td>
|
||||
<td>TBD</td>
|
||||
<td>See queue migration doc</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MIG-006</td>
|
||||
<td>Create client-facing exception list for data that cannot/will not be migrated</td>
|
||||
<td>Governance</td>
|
||||
<td>Open</td>
|
||||
<td>TBD</td>
|
||||
<td>Must be approved before go-live</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>Cutover Validation Checklist</h2>
|
||||
<ol>
|
||||
<li>Run DB migrations in target environment.</li>
|
||||
<li>Run all required data migration scripts (including helicopter import flow if applicable).</li>
|
||||
<li>Validate row counts and sample records for migrated domains.</li>
|
||||
<li>Validate file accessibility for mandatory records (especially PDFs needed operationally).</li>
|
||||
<li>Validate queue behavior post-cutover (<code>pending</code> to <code>published</code>, no legacy SQS dependency).</li>
|
||||
<li>Record migration result, exceptions, and follow-up tickets on this page.</li>
|
||||
</ol>
|
||||
|
||||
<h2>Change Log</h2>
|
||||
<ul>
|
||||
<li>2026-04-17: Initial wiki created from current migration scripts and docs.</li>
|
||||
<li>2026-04-17: Added CKEditor-ready HTML version.</li>
|
||||
</ul>
|
||||
Reference in New Issue
Block a user