importer
Technical notes on web development, DevOps, and AI integration.
2 articles
- 23:31backend
When the Source Shrinks: Reconcile in the Importer
63 rows in, 65 rows in the database: the importer itself deletes entities the source no longer lists.
TL;DR: The importer treats deletion as part of its contract: any category entity whose attributes.no is missing from the source JSON gets removed in the same run. Idempotency keys use source numbers instead of slugs to prevent resurrection, and ON DELETE CASCADE cleans up media. Dry-run plus result counters verified it: 65 seeded, 63 kept, 2 deleted, then zero changes on rerun.
#data-engineering#importer#database - 22:34backend
Swapped Coordinates and the Honest Import
22 heritage records, one swapped column pair, and six drafts: what an honest data import looks like.
TL;DR: While importing a heritage site spreadsheet, the author found latitude and longitude columns swapped, plus typo'd prefixes, duplicate coordinates, and one missing record. The fixed importer parses with column swap, preserves raw DMS strings, and routes suspect rows to draft with a geo_review flag instead of publishing. Result: 16 records published, 6 drafted, and reruns are fully idempotent.
#data-engineering#importer#geospasial