Outlook, Outlook, how does your Database grow?

Database fileAdministrators on the MacEnterprise.org mailing list recently discussed their observations about the behavior of the Microsoft Outlook for Mac Database file. Unlike Microsoft Entourage, which stores messages within the Database file and creates one multi-megabyte or even multi-gigabyte file, Outlook is suppose to store messages separately from the Database file. The purpose is to make Time Machine backups and similar management much easier.

However, one administrator noticed that with a brand new Outlook Database, one message with a 5MB attachment immediately grew the Database file itself by 5MB. This prompted him to speculate whether or not the Outlook Database file really was storing attachments separately as Microsoft claimed it should do.

By the end of the day Andy Ruff, a developer in the Macintosh Business Unit at Microsoft and Lead Program Manager for Outlook for Mac, stepped into the conversation to explain exactly how the Outlook Database works.

Data records and indices

Like most databases, the Outlook database contains primarily data records and indices. Data records are the actual messages themselves and indices are like the index of a book, helping to quickly find a message.

Indices are intended to be frequently accessed (thousands at a time) while the content is displayed in real-time. The message list seen in the Inbox, for example, is drawn from the indices of the database. When a message is opened and displayed, the content is drawn from the data records.

Outlook stores only the indices in the Database (with a capital “D”) file and stores the data records as a series of files nested within folders. The Database file and its folders are located here:

In Andy’s words:

The “Database” file is actually better thought of as the “Index” … Indices are generated from data records over time —when you get a new message, we update the indices and store the data record. For this reason, there’s no “value” to backing-up the index; if catastrophic failure occurs (e.g. Hard disk crashing), Outlook rebuilds the index file by reading each data record.

Why is the Outlook database growing?

Outlook does store attachments and mail messages within the Data Records folders but it stores them separately. However, as a message with an attachment is received, the attachment information is temporarily stored in the indices before moving to its final place within the Data Records folders. This causes the Database file to grow. And, like most databases, when data is removed the blocks that stored the data are not recovered to shrink the database. Instead, they are left in place to be reused by new data.

Andy continues:

That means your Index file will typically be the size of all the indices needed by Outlook plus the largest attachment you’ve received in Outlook.
For more information about the Outlook Database read our Database article.
Share this:
  • Twitter
  • Facebook
  • del.icio.us
  • Digg
  • StumbleUpon
  • Google Bookmarks
  • PDF
  • Print

8 comments to Outlook, Outlook, how does your Database grow?

  • Hi team,

    While on the topic of Mac Outlook databases, it would be great if you could expand the topic a little to that of mail archiving. Archiving is one of the nice features available in the Win versions of Outlook, but never seems to have been addressed efficiently in the Mac versions (Entourage or Outlook).

    Yes I know one can export to an archive, but archiving to a personal folder is so much cleaner in the Win versions, especially when one has to access old mail periodically.

    Can you provide some thoughts around this please?

    Thanks for a great resource!

    Regards,
    Neil

  • Rich

    FWIW, we’ve seen several instances of corrupt Outlook 2011 dbs that can not be revived even after multiple repair passes. This makes little sense to me based on the report above. In a practical sense, how does one invoke the database rebuild if they literally don’t have the original file?

    Thanks for doing what y’all do!

    - Rich

    • The Database file no longer stores your actual data as it did with Entourage. Instead, it’s now just an index file that helps Outlook quickly locate your actual messages stored in the Data Records folder.

      Quit Outlook and delete this file:

      ~/Documents/Microsoft User Data/Office 2011 Identities/Main Identity/Database

      When you launch Outlook, it will notice this file is missing and alert you that you’ve restored from a Time Machine backup. (Crazy, I know!) It will then regenerate the Database file from the information stored in the Data Records folder. This can take a while.

      I’m not sure how much data could be missing from the Data Records folder before the Database file would fail to rebuild.

  • Surprisingly, TimeMachine can’t backup the Outlook database as the file is held open if OutLook is running when the backup happens. It is also possible that the Database will be corrupted if OutLook hangs and requires a Force Quit. Deleting the Database file and allowing the Database Utility to rebuild said Database file appears to be the only resolution in this instance as well.

    • This is actually by design.

      No database can be backed up wile it’s in use and the Microsoft Database Daemon is constantly using the Database file. That means, no, it cannot be easily backed up with Time Machine or any backup system.

      Unlike Entourage, however, Outlook’s dependence on maintaining the Database file isn’t nearly as important. It can always create a new Database file from the items in the Data Records folder. The Database file is now expendable and replaceable.

      • Allan Marcus

        just about every major database platform, including SQL Server, allows for hotback.

  • danny

    if i intend to delete the whole account, do i remove all the database file and data record file automatically? or i need to delete these 2 files manually.

    Thank u
    Danny

    • If you do not want to try and recover your data, you can delete the entire Main Identity in the Finder.

      ~/Documents/Microsoft User Data/Office 2011 Identities/your identity

      Another option is to open the Database Utility, select the Identity and click on the – button to delete. Hold down Option key when launching Outlook to access the Database Utility.

      See this link for help troubleshooting:

      Troubleshoot Outlook (how to rebuild and repair)