Both Entourage and Outlook for Mac utilize a database for quick retrieval of messages when browsing and searching. However, each application has a radically different database design from the other and each uses a Microsoft proprietary format.
Entourage and Outlook store their database files in:
~/Documents/Microsoft User Data/Office 2011 Identities/Main Identity/
The ” ~ ” (tilde) represents the user home folder, such as /Users/johndoe, and the Main Identity folder is named this by default but may be a different name. Both Entourage and Outlook require the database files and associated files to be in this exact location.
Moving the database files or renaming any of the enclosing folders will prevent the applications from reading them. If the database location is changed then the applications will most likely create a new database and ask the user to begin setup of a new mail account. Restoring the database files to their original location and restarting the application often resolves missing data problems.
An Identity is simply one set of database files. That one set contains unique user accounts, messages, calendar events, contacts, etc. Multiple identities can reside side-by-side in the Office Identities folder but only one identity can be open at one time. A new identity is named Main Identity, which corresponds to the name of the Main Identity folder in the database folders.
Microsoft’s concept of an identity was to allow multiple users to be able to share one computer but switch to his or her own mail accounts and information. With the introduction of Mac OS X and user Accounts, the purpose for identities has greatly diminished. They now expect end-users to each have a Mac OS X user account with just one identity in each.
Switching between identities in Entourage is different than Outlook. Entourage users can select the Switch Identities… menu command from the Entourage menu and select a different identity. An identity can be renamed here as well to make it more recognizable. Outlook users must launch Outlook with the Option key down, select the desired identity and use the Set default identity menu button at the bottom of the window to make it the default. Identities can be renamed here as well. The next time Outlook runs, the contents of the new identity will display.
Contents of the database
Like most databases, the Entourage and Outlook databases contain primarily data records and indices. Data records are the actual messages themselves and indices are like the index of a book, helping to quickly find a message.
Indices are intended to be frequently accessed (thousands at a time) while the content is displayed in real-time. The message list seen in the Inbox, for example, is drawn from the indices of the database. When a message is opened and displayed, the content is drawn from the data records.
The primary file for both Entourage and Outlook is the Database file (with a capital “D”). For Entourage the Database stores both the indices as well as the data records, resulting in a single file that can grow from several hundred megabytes to several gigabytes in size. Outlook stores only the indices in the Database file and stores the data records as a series of files nested within folders. Those folder are stored in:
~/Documents/Microsoft User Data/Office 2011 Identities/Main Identity/
Outlook’s Data Records folder is divided into sub-folders with the top level of folders identifying the content (Categories, Contacts, Messages, etc.) and deeper level folders dividing the content into groups of no more than 1,000 items each. The numbered folders follow a pattern of 0T/0B/0M/0K (T=Trillion, B=Billion, M=Million and K=Thousand). The number of each message is unique but assigned sequentially.
Once a folder reaches a thousand items, such as the 20K folder, then the 21K folder is created and so on. The unique file names assigned to a message are never reused. As messages are deleted from Outlook, they are deleted from the Data Records folder.
As messages are received the Database file will grow in size. When messages are deleted, however, the Database will not shrink automatically. This is typical with any database. With regular receiving and deletion activity, the Database size will eventually reach a plateau and increase very little afterward.
Users may notice that the Outlook Database file may increase dramatically when an attachment is received as part of a message but they shouldn’t be alarmed that something is wrong or that attachments are ballooning the Database. As a message with an attachment is received, the attachment information is temporarily stored in the indices before moving to its final place in the data content area.
The Database file increases in size due to the attachment size but the blocks of the transaction are not recovered. Instead, they are left empty and will be reused as needed before increasing the size of the Database any further. Simply put, the Database file for Outlook will be as big as the indices needed to reference messages plus the size of the largest attachment ever received.
Microsoft Database Daemon
A background process called the Microsoft Database Daemon is responsible for maintaing the Database and its associated files. The Database daemon runs while either Entourage or Outlook is open and continues running in the background if the applications are quit unless the user has disabled all the features that require the daemon to be running.
The first time Entourage runs, it will add the daemon to the user’s Login Items in his Mac OS X Account settings. This ensures the daemon runs even while the application is quit. The daemon is responsible for Office Reminders notifications, updating My Day while it is open and creating caches for Spotlight searches. While Entourage is running, the daemon will also monitor the health of the Database and alert the user to any problems.
Outlook also adds the Microsoft Database Daemon to the user’s Login Items and is responsible too for Office Reminders notifications, updating My Day and monitoring the health of the Database. It does not, however, create cache files for Spotlight because the actual message files themselves are indexed instead of cached copies.
Microsoft Database Utility
The Entourage and Outlook databases are prone to corruptions issues like any other database format. The Microsoft Database Utility will repair the Database and compact it at user request. It need only be run if alerted to an issue by the Microsoft Database Daemon or if the application begins behaving sporadically. Examples of Database corruption are frequent application crashes, incorrect numbers displayed for unread items and messages appearing under one view but not another.
Repairing the Database on a planned schedule as part of maintenance is not necessary.
To invoke the Microsoft Database Utility in either Entourage or Outlook, quit the application and then launch it again while holding the Option key. The Microsoft Database Utility window will appear and display all existing identities.
Entourage users should select the option to Verify the Database and Rebuild only if prompted. Outlook users will only be able to click the Rebuild… button.
When the Microsoft Database Utility rebuilds the Database, it will first create a backup copy. Should the rebuild exacerbate the problem with the Database rather than eliminate the problem, the user will always be able to return to the backup copy to attempt migrating messages and other data to a new Database.
Outlook’s Database is Time Machine friendly as well as optimized to work other backup solutions. They only need back up just changed files and not all files. Entourage’s single Database file, however, changes with every message received, deleted, moved or modified. That means one new message prompts a backup of the entire Entourage Database file, whether it’s 500MB or 10GB.
Neither Entourage nor Outlook offer backup tools but instead rely on the end-user to create his own solution. Each application requires a different backup strategy.
Outlook’s Database is purposely excluded from Time Machine backups and should be excluded from all backup solutions. While the Microsoft Database Daemon is running, the Database file is in use and cannot be properly copied to a backup location without inducing potential corruption, making it useless when restored. However, should Outlook recognize that the Database is missing, it will alert the user that he has restored his messages from a Time Machine backup (even if a different backup solution was used) and prompt to create a new Database from the existing message files.
Because Entourage’s Database is always in use by the Microsoft Database Daemon, the Database file should only be backed up while its user is logged out or while both Entourage and the daemon are quit. Following are articles that provide options for creating a good Entourage Database backup:
- Update to shell script used with Super Duper! in Snow Leopard
- Backup Entourage with SuperDuper!
- Automated Backup with Shell and AppleScripts
- Using Automator and Entourage to back up the MUD folder
- Alternative method to use Entourage and Time Machine
- Entourage and Time Machine
No solutions exist to allow users to selectively restore messages or other items. All solutions require the entire Database to be restored from backup.