Open Calendaring

The following image represents a very rough and high-level draft of how the MAPI objects interact in the Open Connector message store.  The “Profile” object holds reference to all the MAPI Service Provider instances.  Also note that the “ABLogon” object is central to the other logon objects due to the fact that it holds all the credential information.

A single ABLogon can service multiple MAPI objects through-out the connector.  But we still have to work out the best way to get these objects mapped to each other at runtime.  According to the MAPI documentation, care to should be taken not to a assume that the objects are in the same process space.  Eg.  A Transport Logon object can be initiated on the “Spooler Thread” and have to communicate to the Address-book logon object.  We’ll see how closely Outlook requires us to follow this rule.

A big issue we face is instantiating as few of these objects as possible so the connector runs a lot faster.  In many case, this will require us going against the MAPI documentation, or at least being very smart about it.

Also seemly unrelated objects also need to find each other at runtime.  For instance, the ‘otlkcon_account‘ object which represents accounts in remote systems may need to find the correct ‘IMsgStore‘ object to download events to.  In this case, the mapping can be done by Principal URI.

We’ve updated the development plan for the Version 0.5 Milestone 4.  This milestone will actually consist of several snapshot releases.

• Milestone 3.25 will have the “Shared Calendar” or “Delegate” feature support and also iCalendar Recurrence support.  Those are currently being worked on.
• Milestone 3.5 will have the first support for invite processing via CalDAV Inbox and Outbox folders.
• Milestone 3.75 will introduce further scheduling invite support such as invite declining, exceptions and recurrence.
• Milestone 4 will include object caching support and further performance optimization.

Realistically these changes move the delivery of Milestone 4 to September 30th for the earliest.

Work continues on getting delegate support ( aka “shared calendars” ) working with Outlook 2003.  Things are just starting off, but looks promising.  Shared calendars is the least documented Outlook feature we’ve targeted for support so far, so this might take a while.  Still, I’m confident we’ll have shared this and recurrence done by August 31st.

Apart for these two major features, we’re spending time getting the newer providers ( address-book especially ) working well.  We’ve run into the unexpected problem of MapiLogonEx() calling our IABProvider::Logon() method each time it’s called.  Logon method is very slow, so we’ll have to speed that up.

Also, it looks like some COM Addins are having a very hard time with our connector.  This was recently discovered for at least one popular Outlook add-in so we’re looking into this as well.

The Open Connector source code can be downloaded using Subversion from http://sf.net/projects/otlkcon/.  Please follow the instructions on the SourceForge website to get the source code.

The source has changed quite a bit over the years.  The project began as a plain C library written with NMake makefiles.  We tried to use as few Microsoft APIs as possible in a misguided attempt at being at least partially cross-platform.  Eventually the project was re-written entirely in C++, not because of how painful it was to write COM Classes in plain C, but mainly to make use of the STL Library.  We also switched to using as many OS APIs as possible to avoid having to manage dependencies.

Currently the source code is separated into 3 main projects.  That number grows and shrinks every few months by the way.

The ‘OpenConnector‘ sub-project is a C#/.Net executable server.  This is the program that displays the Open Connector configuration menu and also handles most of Open Connector’s network traffic.  This executable is started when Outlook.exe starts and shuts down when Outlook ends.  Look in here if you’d like to make Open Connector produce more efficient Network traffic, or would like to clean up the GUI.

The ‘opncon‘ sub-project is a C++ dynamic link library.  It builds ‘opncon32.dll‘ is the actual MAPI DLL that Outlook.exe loads.

The ‘tests‘ sub-project holds the unit-tests for the project.  We’re in bad need to have these worked.  The tests use the boost unit-test framework.

Currently the build has no external binary dependencies, outside what comes with Microsoft Visual Studio and the Operating System.  We have 2 external source dependencies.  SQLite is an in-process SQL database that we include with our source.  The other external dependency is the Boost library, which is used extensively in our source code but not included with the source.  Please download the latest boost library from http://boost.org/ and move the source to a folder named ‘boost‘ in Open Connector’s source code root folder.

Microsoft Visual C++ 2005 is the current build environment, but we plan to upgrade to 2008 in the future.  I haven’t tried but the express versions of the suite may work.

I use the ‘Debug_Installed‘ build target the most during development.  This target has a post-build action that copies the built opncon32.dll into the ‘C:\WINDOWS\System32‘ folder.  This saves the step of copying the dlll over manually whenever there’s a rebuild.

The DLL uses a lot of DebugPrint() calls to log its actions.  Make sure that ‘log to debugger’ is set in the configuration GUI, since it is off by default.  These logs can be seen in the ‘Output‘ window of the debugger.

We also use a lot of DebugBreak() calls as assertions when in a debug build.  Usually you can choose to continue on those and things work ok.

Open Connector is a MAPI Message Service.

A basically “MAPI Message Service” is a set of Outlook extensions that provide the capability to communicate with a network or device.  For instance you may be familiar with the native “Microsoft Exchange/PST” Message Service which allows Outlook users to save to “.PST” files and interact with remote Microsoft Exchange servers.  There’s a lot more information on Message Services on MSDN if you are interested.

A message service is divided into providers.  The four most common providers are the “Message Store Provider” which is responsible for storing Outlook’s messages and folders locally, “Transport Provider” which is responsible for connecting Outlook to the network, “Address-book Provider” which is responsible for providing user information to Outlook, and “Free/Busy Provider” which is responsible for providing free/busy information to Outlook.

For instance, the native PST provider that allows you to store your email in a “PST” file is basically a message store provider.  It stores messages And when you sync Outlook with an exchange server, you’re using the Exchange Transport Provider.  Or when you use an LDAP server to get an email recipient information, you are using the LDAP Address-book provider.

Open Connector is a full Message Service.  It provides a message store provider, transport provider, address-book provider, and free/busy provider.

Our message store provider’s “O_IMsgStore” class implements the MAPI “IMsgStore” interface and stores messages and folders locally in a single-file database.

Our transport provider’s “O_IXPProvider” class implements the MAPI “IXPProvider” interface and communicates with CalDAV servers only for now.

Our address-book provider’s “O_IABProvider” class implements the MAPI “IABProvider” interface and retrieves information from LDAP address-books.

And finally, our free/busy provider’s “O_IFreeBusySupport” class implements the MAPI “IFreeBusySupport” interface and again retrieves Free/Busy information from CalDAV servers for now.

When Outlook is started, it finds all registered providers and starts any that are specified in the user’s “Outlook Profile“.   Since multiple profiles can exist, the provider’s class may have multiple objects created in the same process space.  In fact, MAPI gives precious few guarantees to message service writers on how they objects will be used at runtime.  A provider object can be used across multiple threads, they can be created and destroyed in any order.  Providers will often depend on each other for capabilities ( eg. the message store provider may require user credentials from the address-book provider ), but they can’t take for granted that that any other provider will be created at all, much less is in the same process space.

All this means that there’s a great deal of trouble to put together reliable communication and synchronization between MAPI service providers.

At some point Open Connector will have an installation wizard that will simply ask a few questions about your calendar server then go off to build a suitable set of configuration options. But until someone is nice enough to write this wizard for us, we are forced to fill out the tabs in the “Open Connector Configuration” progam manually.

The configuration program is written in C# and is very easy to modify. It simply reads and writes registry entries that the main Open Connector DLL picks up and uses at run-time. It’s source-code can be found in the “OpenConnector” sub-project and it is part of the “OpenConnector.exe” process. This program can be started from the “Programs” sub-menu of the “Start” button, under “Open Connector Groupware”.

The General Tab

1. The “Database File” field holds the full path of the file that will store all your local connector information, like events, contacts, etc.
2. The “Enable Transport Provider” check-box allows you to turn on or off our connector’s access to the network. Not selecting this check-box will disable calendar syncing with a CalDAV server for instance. The Transport Provider is one of the core components of the Open Connector MAPI Library we’ve developed. But you can always leave it off if you’d just like to work with local events and messages.
3. The “Enable Addressbook Provider” check-box allows the use of our address-book features. As of writing, only LDAP address-books are supported, but more back-ends are planned for the future. The Address-book Provider is needed for calendar scheduling, free/busy support. You can leave it off it you’d like to focus on basic calendar synchronization and not scheduling.
4. Ignore “User Master Password” for now. Incomplete feature.

The Debug Tab

1. The “Log to File” feature allows Open Connector to write all debug information to a specified file. The default file is stored under the user’s “Application Data” folder, but can be changed to anything else. Use the “Log File” edit box on the same page to specify an alternate path.
2. Ignore the “Log to Memory” feature. This allows Open Connector to write debug information to a shared-memory segment that a local server can retrieve. Good for local process that’s monitoring the DLL. Will probably be removed.
3. The “Log to Debugger” option allows Open Connector DLL to use the Win32 API DebugPrint() function to log messages to the debugger that’s running Outlook.exe. We use this one a lot, but it’s obviously geared for developers. Testers who aren’t interested in debugging will probably just need the “Log to File” feature turned on. The resulting log file could be sent to the developers.
4. The “Debug Level” feature specifies how much debug information is logged to the various logging targets. The value of “-1” specifies that all possible information that can be logged should be logged. A value of “0” turns of logging. This value is a bit-mask, ie. “-1” is actually 0xFFFFFFFF in hex and turns on all bits in the flag, “0” is 0×00000000 and turns off all bits. Logging slows done the provider considerably, so set to 0 for speed.

The Accounts Tab

1. The “Account” Edit box names the current account.  Internally, a lot of features refer to the account by its name.  Hence, although it is possible to rename an account it is not advisable.  We will have support for proper account renaming in the future.
2. For now the “Type” drop-down list should be set to “CalDAV“.  Again, more protocols are planned.  The “File” option creates an account that reads and writes data to and from an XML file.  This option is incomplete and should not be used.
3. The “Your Name” option is a string used to identify the mailbox owner.  Set that to anything you’d like.
4. The “Username” option is the username of the server that you will be receiving information from, eg. Your CalDAV server.  This must be a server valid username.  Check with your server administrator for this value.
5. The “Password” edit-box holds the password that corresponds to the username you entered above.
6. The “Email Address” edit-box stores the email account address that matches your server account.  It’s very important for some server protocols, eg. CalDAV that that address relates to the “Username” and “Password” values that you entered above.  In other words, you can’t just use any of your email addresses at your disposal.  You’d have to use the email address that was associated with your CalDAV account when it was created.
7. The “Home URI” edit-box is the full URL of the account on the remote server.  This is the generic description that would match any server type.  But more specifically to CalDAV, this should be the URL to the folder/collection that your ‘inbox‘ is probably in.  This needs to be the full path also. Eg. ‘http://cal.example.com/ucaldav/user/kervin‘ for Bedework or ‘http://cal.example.com/calendars/users/kervin/‘  for Apple Calendar Server.  A proper URL in this edit-box is manatory.
8. The ‘Default Calendar‘ edit should have the name of the calendar collection that holds Outlook’s default events and calendar data.  You have to select this folder by first clicking “Save” account, then “Browse…” account on the “Accounts” tab.  In the new window  that pops-up go to the “Tools” menu, then select “Connect…“.  If all goes well you should now see the remote folder hierarchy from your server.  Select the calendars you’d like to have synced with Outlook by clicking on the check box next too their names.  Click ok.  Any folders selected in the “Browse” screen before should now be selectable in the “Default Calendar” drop-down list.
9. The ‘Principal URI‘ edit-box is also manatory.  This should point to the WebDAV Principal URL for the owner of the mailbox.  Eg. ‘http://cal.example.com/principals/users/kervin/‘
10. The “Default Account” check-box selects the current account as main mailbox’s owner.  This shoudl be checked for one account, and only one account always.
11. The other check-boxes in that group should be always checked for now.
12. The “Address-book” options store information for accessing remote address-books.  For now only LDAP address-books are supported.   The “Username” is the DN string for the user that will log onto the LDAP account.  The “Password” is it’s password.  The “Home URI” is the full LDAP URL for the Organizational Unit that will be synced with the local address-book, eg. “ldap://dir.example.com/ou=people,dc=openconnector,dc=com“.

If you are going to test the connector, there are a few basic things you should know about how it works.

1. Currently the connector is installed as a single DLL named ‘opncon32.dll’ in your system folder.  On 32-bit Windows XP that is usually C:\Windows\SYSTEM32\.  But check your configuration to make sure.  Usually deleting this DLL file after Outlook.exe is shutdown is enough to disable the connector and restore your Outlook.  As of writing, the connector installs no other DLLs.  We use to have a separate SQLite binary installed as well, but now that’s compiled statically.  Also, up until recently, the connector was named ‘mstore32.dll’.  Please remove the old file if it’s still in your system folder ( but make sure it’s the same ‘mstore32.dll’ )
2. The connector uses a single data file store in your ‘Application Data’ folder under ‘Open Connector Groupware’.  For example, on my computer this is ‘C:\Documents and Settings\kervin\Application Data\Open Connector Groupware‘  This is only the default path for the data file.  It can be changed by running the ‘Open Connector Configuration’ program.  It’s alright to delete this file.  If the database file is not found, the connector simply creates and populates a new one when Outlook loads up.  Of course you will lose your local copy of any events you’ve downloaded, but those will be downloaded again on the next sync with the calendar server.  This data folder also holds Open Connector’s default log file.  Frequently while testing I delete both those files and restart Outlook to give the connector a fresh start.  Note that as of writing, the connector takes as much as 2 1/2 minutes to completely build it’s database on first start.  Subsequent loads take about a minute probably.
3. The connector installs a few registry keys in ‘HKEY_CURRENT_USER/Software/Open Connector Groupware‘.  Your configuration options are stored there.  The important account information is encrypted but can be decrypted if someone has your account ( obviously ), so as usual please use precaution.  This key and subkeys can be removed at anytime and the configuration program will simply write them on the next time it is run and changes saved.
4. You never have to restart windows for Open Connector.  Well, that’s probably a good idea in production, especially after an uninstall, but not for testing.  We don’t have a production release as yet.  You will have to restart Outlook.exe a few times though.  You should not have to reinstall Windows either
5. You should not have to reinstall Outlook.  The connector only does basic Outlook configuration changes, nothing that would corrupt an installation.  I’ve been developing for Outlook for almost 10 years now, I don’t think I’ve ever re-install due to corruption.  If your Outlook is behaving funny, manually delete the connector DLL and you should be fine.  If it’s still acting up, it’s probably one of your other plugins ( …and you would at that point owe the connector an apology )
6. The connector has a local server named ‘OpenConnector.exe‘.  This program will be in what ever folder you installed Open Connector in, but by default goes into ‘C:\Program Files\Open Connector Project\Open Connector Groupware‘.  You may have change this when you installed.  And also your ‘Program Files’ folder may be different.  This executable is started when Outlook.exe loads and runs until it ends.  It does some of the work for the actual connector, like communicating to a lot of the remote server.  So if you have a firewall on your computer, you should allow it to access the internet.  This is also the Open Connector Configuration program.
7. The ‘Open Connector Configuration‘ program is a stand-alone executable that can be accessed from your Start Menu.  Outlook.exe does not have to be running to using the configuration program.  Just go to “Programs” then “Open Connector Groupware” and it should be there.  This is a convenient place to configure the connector if Outlook.exe is taking too slow to load for instance.
8. Don’t get too fancy.  We are still working on very basic functionality.  The MAPI API is a huge beast, so it will take a while to get everything work correctly.  We will get there, but the current builds are for preview / developer testing mainly.

We’d love to support as many platforms and configurations as possible with our first stable release ( Version 0.5 ) but unfortunately limited developer time forces us to focus on a smaller set of configurations.

Currently we’re committed to supporting…

Operating Systems : Windows XP

Platform : x86-32bit

Outlook Versions : Outlook 2003 and 2007

We’re assuming that the CPU and Memory specifications are average for computer purchased in the last 4 years.

Windows Vista and 64-bit platforms are notably absent from the supported configurations list.  That does not mean we will not support those machines, but we can’t promise that we will either.

As for server software, we will support the latest version of Apple Calendar Server and Bedework Calendar System.  Again, hopefully we will get some of the other CalDAV servers tested and supported as well.

For the last few weeks I’ve been chasing a Milestone 4 blocker bug in our message store provider. From mapping known MAPI symbols to the stack at various stages, it seems that Outlook fails while running the HrValidateIPMSubtree() function call with our provider. Specifically, MAPI seems to think that a c-string we’re returning is actually multi-byte. My guess so far is stack corruption sometime much early in the program flow, but the mystery remains unsolved.

The sad thing is that I know it’s a probably a very small and simple change that’s causing this, but finding that in the 30K+ lines of C++ with no real unit-tests to speak of is proving difficult.

A very exhaustive set of unit-tests is on the road-map. I’ve already started one using the boost::unit_test framework, but it’s only in the beginning stages and I won’t get anytime to work on it for sometime to come. This would be a great place for a volunteer with not much MAPI experience to help out.

Hopefully, I can get this fixed soon so I can continue working on Milestone 4’s delegate and recurrence support ( and the dozens of prerequisites their depend on… )

I’ve received quite a few emails about Open Connector’s status and I apologize for the delay. I guess like most programmer’s I’ve concentrated a lot more on the code than communication. This blog changes all of that

We’ve reached the third milestone build of the Open Connector version 0.5. I will post more about our current roadmap in the future. For now, version 0.5 is basically our first non-developer Beta release.

Milestone 3 is a significant step. It introduces the last two MAPI connectors that we needed. The first is the “Address-book Provider” which allows Open Connector to retrieve calendar and mail user information from any LDAP Directory server, eg. OpenLDAP. These downloaded users are stored in Outlook for use with calendar delegates and free/busy lookups. The second provider is the “Free/Busy Provider” which allows Open Connector to display Free/Busy data from any CalDAV server user in Microsoft Outlook.

Milestone 3 is very buggy right now and so I would actually recommend waiting for Milestone 3.5 or 4 to take a look at things. Unless you are a very experienced MAPI/C++/Win32 developer who’s willing to help with the project.

What’s in the near future? Well Milestone 3.5 should have delegate support ( if we can actually get Outlook to behave ) and also the very delayed recurrence support. Although Milestone 4’s targets have not been fully defined, it should be centered around getting the rest of the CalDAV scheduling protocol features to work.