The first version of almost every workspace migration script looks deceptively simple: export an object from the source, import it into the destination, repeat.
That works until the migration becomes real. Then the questions change. What if the destination path is wrong? What if something already exists there? What if the notebook still depends on mounts, DBFS paths, secrets or workspace-specific tables? How do you prove what moved? How do you roll back a bad batch?
The useful mental model: migration is not a copy operation. It is a controlled state transition with evidence.
Start with inventory, not execution
A migration tool should know what it is about to move before it moves anything. That means creating an inventory that becomes the control plane for the migration.
At minimum, each object should carry source identity, intended destination, object type, owner or domain, migration decision, complexity flags and current migration state.
| Field | Why it matters |
|---|---|
| Source path | Defines the exact object being migrated. |
| Target path | Makes destination ownership explicit before execution. |
| Object type | Notebook and job migrations require different handling. |
| Migration flag | Prevents “everything in the workspace” becoming the default scope. |
| Complexity flags | Identifies risky references before the object reaches the target. |
| Status | Supports repeatability, retries and audit history. |
Dry run should be a first-class mode
“Dry run” should not mean printing what the script thinks it might do. It should execute every safe validation step and stop immediately before mutation.
A useful dry run answers questions such as:
- Does the source object still exist?
- Is the destination path valid?
- Would the operation overwrite anything?
- Can required permissions be satisfied?
- Does the object contain known risky patterns?
- Are dependent objects included in the same migration wave?
The output should be a report, not console noise. If a team cannot review and sign off the dry-run result, the feature is mostly cosmetic.
Separate safe automation from manual review
Trying to make every notebook automatically portable is usually the wrong optimisation. Some references are deterministic and safe to transform. Others are context-dependent.
For example, a known two-part table name may be safely expanded to a three-part Unity Catalog reference when the target catalog mapping is unambiguous. A hard-coded mount path or secret scope usually needs more context.
This classification is more useful than pretending an accelerator can remove all migration judgement. Good automation removes predictable work and makes unpredictable work visible.
Overwrite protection is not optional
A bulk migration utility that silently replaces target objects is dangerous. The safer default is to fail when a destination object already exists unless overwrite was explicitly approved.
Even when overwrite is allowed, preserve the previous target state first. A migration tool should assume that rollback will eventually be needed, because at scale it probably will.
Jobs need dependency awareness
Jobs are not just another object type. They contain references to notebooks, compute, parameters, libraries, identities, schedules and sometimes environment-specific configuration.
That means migration order matters. If a job points to a notebook, the notebook should exist in the target before the job becomes active. A simple dependency graph is enough to prevent many avoidable failures.
Evidence is part of the product
When a migration spans hundreds of objects, memory and chat messages are not a control system. Every run should generate durable evidence.
The report should make it easy to answer:
- What was attempted?
- What succeeded?
- What failed and why?
- Which objects need manual review?
- What source and destination paths were used?
- When did the operation happen?
- Which run or batch produced the result?
Logs are for debugging. Reports are for operating the migration. You normally need both.
The migration engine should be idempotent
A failed batch should be safely rerunnable. That requires deterministic inputs, explicit state, and clear rules for existing target objects.
If retrying a run can create duplicate content or produce a different target layout, the migration mechanism is fragile.
A simple architecture
The actual export/import calls are a small part of this architecture. Most of the engineering value sits around them.
What I would measure
Once the tool works, measure more than migration speed. Useful metrics include automation coverage, manual-review rate, first-pass success rate, validation failure rate, retry rate and rollback count.
Those metrics reveal whether the accelerator is genuinely reducing migration risk or simply moving objects faster.
The uncomfortable truth: a fast migration script without controls is just a faster way to create inconsistent target environments.
Where this goes next
The natural next step is to treat the migration accelerator as a small product rather than a notebook someone knows how to run: versioned configuration, reusable policy rules, structured reports, a clear rollback model and eventually a lightweight interface around the workflow.
The interesting part is not the API calls. It is turning a one-off engineering script into a capability another team can use safely without knowing how it was built.
I’m documenting more patterns around data-platform migrations, architecture and platform automation here as I encounter them.
← Back to all notes