A Problem Solving Experience

• Skill Set Involved:  Long term thinking,  considering sustainability, seeing the big picture where future functioning might be complicated by a short term solution.

• Problem:  Migrating content from one system to another system where original timestamps of the incoming files were not preserved upon ingestion.

• Solution:  Create temporary custom scripts to handle the timestamp anomaly, then remove the scripts to avoid confusion in the future.

The Role

I was the project lead for a migration of files from one commercial document/content management system (I will call it "CMS-1") to another commercial system ("CMS-2"). Once the content had been migrated, CMS-1 would be decommissioned.

I had two full-time developer resources and one part-time developer resource to handle the back-end coding work to facilitate the transfer. I gathered all the business requirements and functioned as the liaison between the internal client group and the developers.

The Metadata Mapping Challenge

Pre-migration testing revealed that when the documents coming from CMS-1 were ingested by CMS-2, CMS-2 overwrote the original datestamps/timestamps of the incoming documents with the date/time of ingestion. If we had moved forward with the migration at that point, the timestamps on all 70,000+ documents would have been changed, and made it seem as if all those files were created and last modified on the same date (the date of ingestion). Many of the documents were several years old or older, so the timestamps in CMS-2 would not have been even remotely accurate. 

The Developers’ Proposed Solution

The developers' proposed solution was to create two new custom fields in CMS-2 to house the original timestamp dates of the files coming from CMS-1. They would map the true timestamp data from CMS-1 to those custom fields in CMS-2. Then after the migration, they would hide the timestamp fields generated by CMS-2 that showed the ingestion date.

The Problems with that Solution

I felt that the developers’ solution of creating new custom fields for timestamps and then hiding the default timestamp fields was too much of a “shell game” that would be hard to explain in the future, and would not be sustainable over the long term. 

I thought about how this would play out in the future. When the migration was complete, and CMS-1 had been sunsetted, the users would be creating new files and uploading those to CMS-2. But the timestamp data for those new files would automatically be housed in CMS-2’s default timestamp fields, which had been hidden after the migration. 

Since the custom timestamp fields used for the migration would have remained blank for those new files, in order to see the accurate timestamp data for the new files, the developers would have to unhide the default timestamp fields.

This would have resulted in all files in the repository showing four timestamp fields. The newer files would have nothing in the custom fields, but the older documents would have four timestamp fields, displaying two different creation dates and two different last modified dates. 

This would have created massive confusion down the road as to which dates were accurate. It would also have meant that inaccurate search results would be returned if a user searched for all documents created within a certain date range, but searched only on the CMS-2 generated timestamp fields.

My Solution

I instructed the developers to write two custom scripts. I wrote out the script behavior and process flow that I wanted them to implement. 

The first script would map the original timestamps of the incoming documents to new (temporary) custom fields. Once the documents were ingested into CMS-2, the second script would be run, and pull the timestamp data from those custom fields to overwrite the CMS-2 generated timestamps. 

I told the developers that when the ingestion was completed, and the field mapping scripts had been run - but before the new CMS-2 repository was opened for use - to remove the custom timestamp fields and the custom scripts.  Leave no trace. Leave a clean repository. Then the only custom fields needed for the future would be up to the users to request, if they wanted to include a client account number or some other classification data. 

If any of the older documents were edited in the future, they would automatically show a new last modified date - a timestamp auto-generated by CMS-2 and displayed in the default field for that metadata. No “shell game” mapping or field hides/unhides would need to remain, since CMS-1 was decommissioned after the migration.

Takeaway

Sustainability and clarity for future users, over speed of implementation (within reason), is always a better strategy. Even if a one-step solution is quicker, two steps are better if it means less complication and confusion down the road.

Short term solutions are often required for various reasons - a contract expiring or an emergency of some sort. But in most cases, it's best to think ahead, think strategically, asking questions such as:

• Will the proposed solution create tech debt that we must deal with in another few months or even in another year or two?

• Will this break something downstream?

• When users start using the new system, what possible scenarios of use (something outside the norm) might not be compatible with the solution?

Note that these questions should be asked even if the solution is not intended to be short term. Always be thinking strategically. Preemptively consider all the possible ways the solution could fail. Be harsh when interrogating your own analysis and proposals, because if you do not, others may do it for you.

Go Long!