After much sadness and remorse, I started sifting through the application logs to see what happened. After looking through the tens of thousands of hack attempts (keep trying, guys! ;-), I found the error entry I was looking for, and I wasn't happy with the message: That assembly does not allow partially trusted callers. Curses! We've hit this problem before on the Rackspace Cloud. They operate their sites in a modified medium trust environment, which means (wait for it) that all assemblies that live on your site are automatically only partially trusted! Yay!
So what does this mean? It basically means that any assembly you attempt to reference from an assembly you've deployed to your cloud site must be marked with AllowPartiallyTrustedCallers. Unfortunately, many of the .NET libraries are not marked with this attribute. In this case, the Workflow library was unhappy with us trying to call it in a partially trusted way, so we have to revamp some of our backend processing to not use Workflow. +1 for the coolness factor of using Workflow Foundations, minus several million for finding this defect so close to launch.
