autowisp.tests.test_database_migration module

Class Inheritance Diagram

Unit tests for project database migration.

These need only a throwaway SQLite database, not a full project, so they subclass unittest.TestCase directly.

class autowisp.tests.test_database_migration.BackendMixin

Bases: object

Supplies clean project databases on whichever backend is under test.

SQLite gets a fresh file per engine; a server has only the one database, so it is emptied before each test instead.

static add_stray_index(engine, name)

Add an index the models do not declare, to produce drift.

Drift is provoked by adding something rather than removing it: the index this branch introduces cannot be dropped on MySQL (see create_legacy_schema()).

static create_legacy_schema(engine)

Build the pre-migration schema: everything but the new index.

Not create_all-then-drop. InnoDB refuses to drop image_observing_session because it is the index backing image’s foreign key on observing_session_id – MySQL indexes a foreign key column whether or not anyone asked. Leaving the index out of the metadata is both portable and a truer picture of a 1.8.1 database, which never had it.

static has_index(engine, table, name)

Whether table carries an index called name.

static list_triggers(engine)

The names of every trigger defined in the project database.

make_engine(name='project.db')

An engine for a clean project database on the current backend.

migrate(engine)

Migrate, confirming the backup where the backend demands one.

reset_backend()

Return the backend to empty part way through a test.

The asymmetry setUp lives with, reached one level in: a test that builds several schemas in turn – one per released version, say – gets a fresh file per engine on SQLite and the same database every time on a server, where the second iteration would otherwise start from whatever the first left behind.

setUp()

Leave a clean project database ready for the test.

class autowisp.tests.test_database_migration.TestAdditiveMigrations(methodName='runTest')

Bases: BackendMixin, TestCase

The pre-Alembic helper still adds missing nullable columns.

It is private now and has exactly one caller – the baseline path of migrate_project() – but it is what every legacy project passes through, so it keeps its own tests.

_apply()

Run the helper the way migrate_project does: on one connection.

_make_old_project()

Simulate an existing project: a pipeline_run table from before code_version and the error table existed, holding a row.

setUp()

Leave a clean project database ready for the test.

test_adds_missing_code_version()

An old pipeline_run (no code_version) gains the column.

test_adds_missing_resolved_to_old_error_table()

An older error table (no resolved) gains the column.

test_creates_missing_error_table()

An existing project gains the new error table.

test_idempotent()

Re-running is a no-op and does not raise.

test_preexisting_rows_and_data_preserved()

Existing rows survive; the new column is NULL for them.

class autowisp.tests.test_database_migration.TestCheckProjectSchema(methodName='runTest')

Bases: BackendMixin, TestCase

The read-only gate every project open – and every worker – runs.

setUp()

Leave a clean project database ready for the test.

test_does_not_mutate_a_stale_database()

The gate must never issue DDL: workers run it concurrently.

test_passes_on_a_current_database()

No exception, and nothing written.

test_raises_on_a_database_from_the_future()

A newer AutoWISP migrated it: advise upgrading, not migrating.

Reachable whenever a centralised database is shared – one user upgrades and migrates, another opens it on older code.

test_raises_on_an_unbaselined_database()

A legacy project is refused, pointing at wisp-migrate.

class autowisp.tests.test_database_migration.TestConcurrentMigration(methodName='runTest')

Bases: BackendMixin, TestCase

Two migrators racing on one database must not corrupt or crash it.

Exercises whichever lock the backend uses: BEGIN IMMEDIATE on SQLite, GET_LOCK on a server. Alembic supplies neither.

setUp()

Leave a clean project database ready for the test.

test_two_migrators_reach_head_without_error()

Both calls return, and the database ends up correctly migrated.

class autowisp.tests.test_database_migration.TestMigrateProject(methodName='runTest')

Bases: BackendMixin, TestCase

The three database states migrate_project() has to handle.

_legacy_engine()

A 1.8.1-era database: full schema, unstamped, missing the index.

test_backup_is_taken_before_migrating()

SQLite is copied aside; a server is the administrator’s job.

test_crash_window_between_ddl_and_version_update()

DDL applied but the stamp lost – the state MySQL can crash into.

MySQL commits DDL implicitly, so a failure between the schema change and the alembic_version update leaves exactly this. Re-running must reach head rather than failing on an index that already exists.

test_fresh_database_is_stamped_without_running_revisions()

create_all builds the current schema, so nothing is applied.

test_legacy_database_is_baselined_then_upgraded()

An unstamped project reaches head in one pass, no user action.

test_migration_is_idempotent()

A second run applies nothing and reports no change.

test_older_database_reaches_the_same_state()

One missing a pre-baseline column still lands at head.

test_project_creation_leaves_no_backup()

There is nothing to preserve when the database is empty.

test_server_refuses_without_a_confirmed_backup()

A shared database is not migrated as a side effect of anything.

The SQLite path has no equivalent: it copies the file aside itself.

class autowisp.tests.test_database_migration.TestRevisionChain(methodName='runTest')

Bases: TestCase

The revision chain is linear, ordered, and consistently named.

A fork is what actually breaks upgrade head: two branches each adding a revision both point at the same parent, and Alembic then refuses to upgrade because the head is ambiguous. Catching that here means catching it in CI rather than at a user’s next project open.

setUp()

Hook method for setting up the test fixture before exercising it.

test_baseline_is_the_root()

BASELINE_REVISION names the one revision with no parent.

test_no_revision_has_two_parents()

No merge revisions: the history stays linear.

test_numbers_increase_along_the_chain()

Numeric prefixes strictly increase from base to head.

Catches a duplicate number and a revision merged out of order.

test_revision_ids_are_numbered_slugs()

Every revision id matches the NNNN_slug convention.

test_revision_ids_fit_the_version_table()

Ids stay within Alembic’s VARCHAR(32) version column.

SQLite ignores a declared length, so an over-long id passes there and only fails on MySQL, mid-upgrade, with “Data too long for column ‘version_num’” – after some revisions have already been applied. Cheaper to catch here.

test_single_head()

Exactly one head: no fork left unresolved by a merge or rebase.

class autowisp.tests.test_database_migration.TestSchemaDrift(methodName='runTest')

Bases: BackendMixin, TestCase

The revision chain and the ORM models describe the same schema.

A revision may not import the models – it has to keep meaning the same schema forever, while the models move – so anything a revision creates is declared twice, once in data_model and once in the revision. Nothing keeps the two in step except this check, which is why the plan calls for it rather than for sharing the definitions.

Worth running per backend: the comparison is over reflected types, and what MySQL reports for a column is not what SQLite does.

test_created_database_matches_the_models()

Control: create_all builds from the models, so it must agree.

test_drift_is_actually_detected()

The check discriminates – an empty result means agreement.

Without this the two tests above would pass just as happily if get_schema_drift() always returned nothing.

test_migrated_database_matches_the_models()

A database the revisions built agrees with the models.

Note this cannot catch a model changed with no revision to match: the “before” state here comes from today’s metadata too, so both sides move together. TestUpgradeFromRelease is the test that catches that, by building the “before” state from released code.

class autowisp.tests.test_database_migration.TestSqliteMigrationLock(methodName='runTest')

Bases: TestCase

The SQLite half of the migration lock really does exclude a second writer.

Alembic does no locking of its own, so without this two migrators can both read the same revision and both try to apply it. On SQLite the protection is BEGIN IMMEDIATE, which pysqlite does not issue by default – it defers BEGIN until the first write, leaving a window in which both have already read. The server equivalent is GET_LOCK, covered by TestConcurrentMigration when pointed at one.

_engine(timeout=30.0)

An engine with a busy timeout, as interface.py builds them.

_write_from_elsewhere()

Write from an independent connection, failing fast if locked.

setUp()

Hook method for setting up the test fixture before exercising it.

test_begin_immediate_excludes_a_second_writer()

With the guard, the write lock is held from the start.

test_guard_is_removed_afterwards()

The listeners are per-call and must not leak onto the engine.

test_transaction_alone_does_not_hold_the_write_lock()

Baseline: the window this guard closes is real.

Without the guard an open transaction that has not yet written lets another connection write, which is what allows two migrators to both read a stale revision.

class autowisp.tests.test_database_migration.TestTimestampTriggers(methodName='runTest')

Bases: BackendMixin, TestCase

Every table carrying timestamp keeps a trigger maintaining it.

Nothing else checks this. get_schema_drift is alembic’s comparison of tables, columns, indexes and constraints, and a trigger is none of those – so the schema checks elsewhere in this file pass unchanged with every trigger in the database dropped.

test_creating_the_schema_installs_all_of_them()

Creation covers the provenance tables, not just data_model’s.

The triggers used to be attached to each class import_table_definitions discovered, and that discovery is a glob which does not descend into data_model/provenance – so twelve tables carried a timestamp column that nothing ever updated. Comparing against the metadata catches a repeat.

test_migrating_reinstates_a_dropped_trigger()

Covers the path taken when the revisions have nothing to do.

A database already at head skips the upgrade entirely, so the repair cannot ride along with a revision; it has to be a check made on the way past.

class autowisp.tests.test_database_migration.TestUpgradeFromRelease(methodName='runTest')

Bases: BackendMixin, TestCase

A database built by a released AutoWISP reaches today’s schema.

This is the test that catches a model changed without a revision to match. Every other check here builds its “before” state from today’s metadata, so a missing revision moves both sides together and goes unnoticed. Here the starting schema is built by the released code itself, checked out from its tag, so the revision chain is the only thing that can close the gap.

That released package is loaded in a subprocess: it defines the same module names as the code under test, so importing both into one interpreter would have whichever came first shadow the other.

_build_release_schema(source, engine)

Create the release’s schema, running that release’s own code.

_export_release(ref)

Extract the autowisp package as of ref into a temp dir.

release_baselines = ('1.8.1', '2.0.0')

Released versions a project database may be upgraded from.

Add each new release tag as it ships; every entry gets its own upgrade-to-current check.

setUp()

Leave a clean project database ready for the test.

test_a_value_too_long_to_keep_stops_the_migration()

Narrowing a column refuses rather than truncating.

Refusing is the point: on a server not running in strict mode the ALTER would truncate the value silently.

Uses condition_expression.expression (1000 -> 768 in 0005) rather than image.raw_fname, which narrows identically in 0004. The guard lives in the shared resize_varchar_column, so either exercises it – but condition_expression has no foreign keys, whereas an image row needs an image_type and an observing session, and that in turn needs an observer, camera, telescope, mount, observatory and target. A server enforces every one of those, so the alternative was either a dozen rows of fixture or switching the checks off, and neither has anything to do with column widths.

test_every_release_keeps_its_timestamp_triggers()

Upgrading does not cost the database its triggers.

SQLite cannot alter a column in place, so batch_alter_table rebuilds the table and the drop takes its triggers with it. The rebuilt table is created by the revision rather than from the models, so nothing puts them back – a 1.8.1 database used to lose seven this way, and the check above could not see it.

test_every_release_upgrades_to_the_current_schema()

Each released schema, once migrated, agrees with today’s models.

autowisp.tests.test_database_migration._git(*args, binary=False)

Run git at the repository root; return output, or None if it fails.

The root, not this file’s directory: git archive refuses a pathspec reaching outside the current directory, so it has to be invoked from the top level.

autowisp.tests.test_database_migration._repo_root()

Return the repository’s top level, or None outside a checkout.

autowisp.tests.test_database_migration.expected_timestamp_triggers()

The triggers the models install when they create the schema.

Derived from the metadata rather than listed, so a table added later is covered without anyone remembering to extend a literal here.

autowisp.tests.test_database_migration.on_server()

Whether this run is pointed at a MySQL/MariaDB server.

The same switch the rest of the suite uses (see autowisp.tests), so one variable turns everything onto a server rather than each part having its own idea of where to look. Setting it runs these scenarios against one too, covering the backend-specific paths – GET_LOCK, implicitly committed DDL, type comparison in the drift check – with the tests that already exist rather than a parallel copy that would drift out of step.