Here are our four top laws-to-live-by for designing any large-scale data conversion or content migration:
- Develop Specifications.
- Design for subset runs.
- Know how to roll back.
- Report errors early, don’t hide them.
Includes: functional specifications, technical specifications, tool configuration plan, test case design, and deployment plan design. Refer to the best practices section within requirements – it lists a basic set of analysis patterns. Completing and verifying the list of rules or mappings is a design task.
Design for subset runs
If the data set is large enough, design the import process so that it can be run as an incremental set in a shorter timeframe. Faster develop-test cycles that way. During the detailed design and implementation, try to employ approaches that make the process more easily testable. For example, being able to pause and resume the process, or to run across a subset of the content, or to run only one stage of the process. In one of our past major DMS content import projects, the upload process required at least seven hours per major division: this stretched the implement-test-fix cycle out into iterations that lasted days.
Know how to roll back
Design your fallback strategy for production. If something goes terribly, terribly wrong, how do you put things back the way they were? It could be as simple as “after we tell all users to get out of the systems, then we will take a full back-up of everything before we start. Make the decisions explicit, though.
Report errors early, don’t hide them
Design to detect and report on deltas (things that break the expected patterns, possibly caused by things added after the initial analysis was put together). Plan to detect “deltas” in a “dry run” up front, so that the most critical issues can be addressed before kicking off the real import.
This article is part of a series discussing many aspects of content migration from an unstructured storage system to a document management system. The overview page is here: Content Migration to a DMS – Articles.