Un-dead Lotus Notes Documents: How ‘ghosts’ can haunt your databases

Have you ever seen a deleted document mysteriously re-appear in a Lotus Notes database? If you answered “yes” you probably weren’t hallucinating … and you’re not alone.

This eerie issue can occur when someone replicates a database after a long period of dormancy—like when somebody’s laptop gets plugged in after a long vacation. These pesky unwanted docs are commonly called ‘ghosts’ or ‘resurrections.’So what prompts these ghost documents to haunt our Lotus Notes databases?

The short answer: It happens when deletion stubs for deleted documents get ‘purged’ before all replica databases can see them. This means copies of documents that should have been deleted will still remain on a replica database. So during the next replication these documents will be regenerated on the replica databases where they had previously been deleted. But that’s surely not enough to satisfy an inquiring Notes mind, so here’s some more in-depth background information on how ghost documents are generated:

Every time a document is deleted in Lotus Notes, a deletion stub is created. The deletion stub is used during replication to tell replicas that a particular document was deleted elsewhere, and therefore should be deleted here to keep things in sync.

There isn’t much to a deletion stub. A stub contains no fields and only the most basic document information (the NoteID, UNID, the Modification Sequence number and the Deletion/Creation dates) necessary to perform its job. But they still do take up disk space so Lotus Notes and Domino offers a provision for deleting—or purging—them after a ‘purge interval’ at which time they’ve (hopefully) outlived their usefulness.

The way Notes handles deletion stubs normally works splendidly…but when troubles arise—watch out!

The purge interval is equal to the date set in the Replication>Options>Space Savers (see the image below) dialog in your Lotus Notes client. Oddly enough, it doesn’t matter if you turn on the ‘Remove documents not modified in last __ (days)’ feature—the date here is going to be used as your database’s deletion stub purge interval no matter what. The default purge interval is 90 days.

Lotus Notes will attempt to purge documents every time 1/3rd of the purge interval elapses. It’s a tad confusing, but think about it this way: If you’re using the default purge interval setting of 90 days, any ‘expired’ stubs lying around in your databases should (ideally) get the heave-ho within 30 days.

Different types of databases have different ‘triggers’ that tell Notes that it’s time to purge expired stubs. Notes checks local databases for expired stubs each time you double-click their Workspace icons. For databases on servers, stubs are purged during the Updall task. Notes handles purging stubs in address books, logs and stats databases a little differently, but we won’t get into that here.
Note: Remotely accessing a database won’t trigger the purge process. Nor will purging be triggered for databases that are already opened by another person or process.

Under normal circumstances, the way Lotus Notes handles deleted documents and deletion stubs in a distributed environment works splendidly. But when troubles arise—watch out!

Imagine a guy at your organization (let’s call him ‘Bob the HR guy’) takes his laptop with him on a extended vacation/sabbatical.

Upon returning to work, he plugs his computer back into the network and replicates all his local Notes databases. That’s where our troubles begin… There are a few hundred documents on this laptop’s local replicas that had been deleted long, long ago on all the other replica databases (but after Bob went on his sabbatical). The deletion stubs on the other replicas had already expired and had been purged, so as far as Notes is concerned all those old documents on Bob’s laptop are perfectly OK and should be replicated.

Soon people start to notice hundreds of documents that they deleted ages ago are back in the database—like ghosts. (Aside: Though these sorts of documents are commonly called ‘ghosts’ in the Lotus Notes world, I’d argue that calling them ‘zombies’ would be more apt).

Soft deletion: Purgatory for documents

In cases where documents are ‘soft deleted’ (like in the mail database), Notes treats things a little differently. With a soft deletion you get a special ‘limited-time-only’ type of deletion stub that still retains all document data. From this ‘soft deletion’ state the document can be revived and transformed back into a normal document if edited or saved. Once a soft deleted document passes the expiry date (you can set the soft deleted document expiry date in the ‘Advanced’ tab of the Notes application properties dialog), it will be stripped of its field data and transformed into a normal deletion stub. The ‘Empty Trash’ action will also transform a soft deletion into a normal deletion stub.

If you have Ytria scanEZ you can use its Post Replication Auditor feature (Tools menu>Post Replication Auditor) to help track down ghost documents in Lotus Notes databases.

The Post Replication Auditor is a snap to use, you just need to open the database you want to check for ghosts then click the Audit button. The audit results are shown in a grid that supports grouping and sorting.
Under the hood, the Post Replication Auditor works by analyzing and comparing creation date information for documents that have been replicated and squaring this information with deletion stub lifetime setting (purge interval).

The following four diagrams show four different situations where a new document can appear in a database after replication. In the fourth case, there’s a possibility that the new document is a ghost, so scanEZ will flag it.

In instances where the ‘Created initially’ date of a replicated document falls within the database’s deletion stub lifetime, the replicated document will not be a ghost.

In instances where the ‘Created initially’ date of a replicated document predates the deletion stub lifetime cutoff date (eg: if the deletion stub lifetime setting was 90 days and this document was ‘created initially’ 95 days ago) but the replica database’s creation date falls within the deletion stub lifetime, the replicated document will not be a ghost.

In instances where the ‘Created initially’ date of a replicated document predates the deletion stub lifetime cutoff date (eg: if the deletion stub lifetime setting was 90 days and this document was ‘created initially’ 95 days ago) the replicated document could possibly be a ghost.
But remember it’s only possibly a ghost. There are specific instances where one could legitimately expect a document or a design’s ‘Created initially’ date to predate the deletion stub lifetime cutoff date (eg: restored backup documents or a design refresh from a template). We use this icon to indicate a possible ghost/resurrection: This little guy is the icon used to flag Lotus Notes ghost documents/designs in scanEZ

Warning: Be extra cautious about design elements flagged as ‘Possible Resurrections.’ If you replace or refresh a design, you might see legitimate designs flagged as possible ghosts because their ‘created initially’ dates are old enough to place them outside your database’s deletion stub lifetime setting.