Thou Shall Always Know What is “in” Production
For the background on this series you can read my introduction in Improving Enterprise Development in the Cloud.
Have you ever been in this situation?
You have carefully listened to your customers about what new functionality that they want implemented in a future release. You believe you have a very good understanding of their requirements and you reiterate back to them what you heard. Everybody agrees and you and your team are ready to go and work on the new feature(s).
After your team goes through iterations of designing, implementing, testing and clarifying issues with the customer, you are ready to promote these changes up through QA, UAT and finally to production. Everything looks great and you and your team celebrate and start working on the next set of features.
A couple of days later you get a call from your customer that there is a high-priority bug that needs to be addressed, immediately. As in, drop everything, this is the single most important thing that needs to be addressed because of the impact it is having.
What would be your next course of action? Duplicate the problem in a development environment, right? How would you do that? Pull the source code that has been (hopefully) tagged from your repository. (You are using a source code repository like CVS, Subversion, Git, or Mercurial, right?) You might also be required to replicate data in order to duplicate the issue.
Once you were able to duplicate the issue in your development environment, you would track down the root cause, implement a fix, check in your changes, test and push the revised code to QA, UAT and back in to production.
Unfortunately this same practice is not available with the Force.com platform, because not everything is accessible through the Force.com Metadata API. 
The Metadata Api is used to “retrieve, deploy, create, update or delete customization information, such as custom object definitions and page layouts, for your organization.” If you are interested in knowing the complete list of components that cannot be retrieved or deployed using the MetadataAPI, read the Development Lifecycle Guide (PDF, HTML) under the Which Metadata Components are Accessible section.
In the context of this conversation what this means is that at the time of this writing there is no way of extracting a complete representation (configuration and code) of what constitutes a Salesforce.com organization at any point in time and storing it within a source code repository. Stop and read that again. I’ll wait.
(Another consequence of this limitation, that is outside of the scope of this article, is that migrating changes from a development organization to other organizations will likely require the coordination of manual configuration changes. In other words, I can’t completely automate the migration…)
The scenario I described earlier regarding how to fix a major production issues will not work in Force.com because of this limitation. Instead, if you really want to have an identical copy of your production organization, you are required to create a Sandbox, do your work testing and fixing there and then migrate the changes either automatically or manually into production. Alternatively you could work in production, which might be a risky proposition…
Enterprise Development, as I stated in the introduction, must effectively coordinate the efforts of a development team. Historically, my teams have organized around a source code repository as the source of truth and has worked really well. With Salesforce.com and Force.com, the source of truth is not contained within a source code repository their source of truth is a Salesforce.com organization. This means that we must invent new practices for working more effectively and making a request for Salesforce to expose everything through it’s Metadata API. (I understand that this is not a trivial problem to solve)
In the meantime, what practices have you invented for working with Salesforce/Force.com more effectively in an Enterprise Environment?
Improving Enterprise Development in the Cloud – Part 2
Enterprise Development, as I stated in the introduction, must effectively coordinate the efforts of a development team. By organizing around a source code repository, you have a source of truth than enables you to always know what is in production, helping your team to quickly replicate your production environment.
Another powerful practice is to back up your production environment before deploying any new changes. This seemingly simple practice is necessary to mitigate against the risk of production deployments that don’t go according to plan. I have personally witnessed production deployments that tested fine in development, QA and UAT environments not work in production due to configuration and/or data discrepancies. When this happens, having a plan of action to be able to roll back to a previous configuration is critical to minimize downtimes for people using the systems.
Backing up production before migration becomes even more important when you cannot completely automate the migration process. On the Force.com platform, because there is no way of extracting a complete representation of configuration and code that describes a Salesforce.com organization, there will be certain types of configuration that will need to be done manually. This means that certain configurations may not exactly match how the lower environments were configured and cause problems in production. And if you don’t remember the exact steps, or have incomplete or missing documentation, you would like to be able to roll back to a known good state.
I did find a suggestion at ideas.salesforce.com that speaks about having the ability to restore salesforce.com using a snapshot metaphor. If you think that this would be a valuable addition to salesforce.com, I encourage you to vote on it here.
In the meantime, what practices have you invented for minimizing the risks of failed production deployments within the force.com platform?