Tuesday, September 08, 2009

SaaS contingency plans need more than software escrow

Although I'm a supporter of the software-as-a-service (SaaS) model of application delivery, there's one area that still make me nervous: contingency planning. What happens if my SaaS provider goes out of business? Or, one day decides he doesn't want me as a customer? How do I ensure continuity of my system? How do I get my data?

Furthermore, it doesn't take a business failure of my SaaS provider for there to be issues about my getting full access to my data. What if I want to switch providers? I wrote a post on this last year, entitled: SaaS: plan to get out before you get in. It's now 10 months later and I still have the same concerns.

Last month, Ben Kepes suggested that software escrow is part of the answer. Traditionally, software escrow is a service whereby a trusted third-party has access to the vendor's source code and the customer's software license agreement allows the customer to gain legal access to said source code in the event of business failure by the vendor. Ben writes:
Others have noted escrow services as being valuable in the event that a vendor goes under, providing,as they do, certainty over data ownership. However there is a bigger issues with the ability of escrow services to cover the IP of a third party application and therefore give ongoing certainty around the functionality of a third party integration. As Escrow Associate points out;

Some agents make the process even simpler by ensuring the integrity and currency of the source code with software escrow verification services. In a SaaS environment, these testing services can also recreate the hosting environment and provide access for end user testing. These services protect both providers and subscribers, because they know that the escrow account contains useable code.
He also points to Escrow Associates as a provider of software escrow services is now starting to offer escrow services specifically for SaaS providers.

Now, I will admit, I do not know much about the services provided by Escrow Associates or any other escrow agent specifically targeting the needs of SaaS customers. But here are some concerns that I would have if I were doing contingency planning for a critical SaaS application:
  1. Does the escrow provider periodically test its ability to transfer both the software and all customer data to its backup hosting site? Access to the software is one thing: access to a completely operational system with my data is another thing.
  2. What levels of service can I expect from the escrow provider in terms of access to my recovered system: 24 hours, 48 hours, a week, a month?
  3. What actions do I need to take as a customer to ensure access to the escrowed version of the system. For example, do I need to make DNS changes?
  4. What level of performance can I expect from the escrow provider? Is the escrow provider prepared to offer the same service levels as the SaaS vendor was giving?
  5. Can I test my business continuity plan periodically with the escrow provider?
These are questions I came up with in just about five minutes. You can see the point. It's not just a matter of getting access to the escrowed software, or even to my data, but a matter of having continuity of access to a completely working system.

I'm waiting for the first example of a prominent or even not-so-prominent SaaS provider ceasing operations to see how this sort of thing gets handled in practice. It's probably already happened, but I'm not aware of it. A few cases, and I believe these business continuity needs will become more clear.

Read Ben's post and also be sure to read the many excellent comments, which also express similar concerns to those I mentioned above.

Update, Sep. 9: Be sure to read the comments on this post, which contain much additional discussion.

Update, Sep. 15: See additional discussion in the comments section on Vinnie's post, in response to this post, especially the comment from Laef Olson of RightNow Technologies, a SaaS provider, who points out the need for financial and operational transparency as a way of giving SaaS customers time to make a transition if required.

Update, Sep. 16. Jeff Gordon provides additional clarity to evaluating the continuity needs of various applications.

Related posts
Addressing business continuity issues with SaaS providers
SaaS: plan to get out before you get in
All not sweet with NetSuite

19 comments:

Anonymous said...

Something I've always wondered is: what happens if a SaaS company goes bust and creditors take legal possession of the hardware, buildings and other assets?

Are the creditors entitled to cut the power and to remove equipment (or otherwise put it out of action) until an agreement is reached? Is there anything to stop them liquidating the assets as soon as they like? Are they obliged to keep the data available? What rights do you have in terms of access and timescales? Will they have all the passwords if unhappy employees have left? etc etc

Jeff Gordon said...

I just posted the same comment at the original site... but thought we might as well have the conversation here, too. :)

---- begin comment from cloud ----
Escrow doesn't solve ANY SaaS issues unless the SaaS vendor is Microsoft, Sun, Apple (or running entirely on a linux platform)... because the problem with Escrow (as it existed in the past all the way through today) is that you don't get all of the third party infrastructure needed to bring up the service/application. And even if you do have the infrastructure vendor as the SaaS vendor, chances are that your escrow agreement doesn't provide all of the infrastructure apps necessary to run the service/app - you ONLY get the service/app code (and maybe your data - which you should have regardless and not have to pay extra for).

As a side note: in the decade plus that I've been doing IT-related contract negotiations, and the decade before that when I was solely on the sysadmin side of things, I have never seen a successful escrow release.

Leigh Monette said...

I'm in favor of SaaS solutions for many reasons, but these types of concerns make me hesitate.

I'm inclined to continue down the same old path: maintain my own systems, but implement features that mimic cloud solutions: ubiquitous access from multiple platforms. Let's face it, as a mobile professional, that's really the key for me - to get what I want, when I need it, no matter where I am.

A regularly synced, off-site backup system should handle the major pitfalls. But even that raises the specter of unavailability...

Lincoln Murphy said...

@anonymous - The answer is "it depends" and I'm sure the other folks on here, including Frank, can add more value to your question. But overall, the answer that matters the most is "the clients are probably out of luck, at least for a while".

You do bring up some interesting points, including the fact that no matter how good their internal security measures are while they are in business, once a creditor has taken possession and/or sold the assets to another entity, how well will the security be preserved. One thing to keep in mind is that in a SaaS business, one of the most valuable assets is often (not always) the customers. A vendor can be gross margin profitable but be burning through cash trying to acquire new customers. If they run out of cash and must forfeit the business, presumably the existing client base could sustain the business for a new owner and even make them a little money. Given that, it would be in their best interest to maintain integrity and security within the application in order to keep the existing customer base. Done properly, the business could be put in "zombie" mode and just operated without trying to grow in an effort to at least achieve some return on the original investment or loan by the creditor. To me, someone focused on growing revenue for SaaS vendors, this seems like a bad idea, but it could be done.

I added a comment on the Cloud Ave. post cited above that talks about some of the other issues. This is a concern, and this is why you should do your due diligence on SaaS vendors you intended to work with.

@Jeff - cross-site agreement here (also on Cloud Ave.)

@Leigh - Don't be too concerned. Obviously the rewards for you outweigh the risks. In fact, for the majority of users of SaaS (generalization) that seems to be the case. Do understand that there *ARE* ways to implement Business Continuity in SaaS and many vendors do it quite well. There is no standard yet, and certainly no 3rd party providers seem to have implemented it correctly (yet), but they will.

For the vendors that need a Business Continuity solution now (i.e. the public SaaS companies, the large enterprise-focused vendors, etc.), they've self-funded their (clients') own. The small SaaS vendors, and more importantly their clients, are the ones that are currently not protected. The theory has been that they don't generally deal with mission-critical applications or serious data and the threat of them going under is not enough to offset the expense of dealing with BC issues. In other words, there hasn't been a big enough pain (i.e. failure) to warrant the aspirin of a true BC solution for small to medium sized SaaS vendors. Someone will figure it out and bring a solution to market. Until then, many small SaaS vendors are at least offering data portability options so you can at least have access to you data should something happen. (I don't think this is the right way... the context of data is often in the application, but I guess it will do in a pinch)

I've never been a proponent of the "small SaaS vendors simply don't provide a valuable enough service or manage value-enough data to worry about." If people use those vendors, and invest time (even if not large amounts of money) into the system, then they don't want it to go away. Something might not be "mission critical" to an enterprise, but it could be to a department or small business or individual professional. The value of data is relative, and it really isn't up to someone to say "Vendor A's client's data is more valuable because they are an MRP system and Vendor B is just a CRM system." It does come down to scale and the (in)ability of smaller SaaS companies to fully fund their own Business Continuity like the larger companies.

Very interesting topic indeed...

- Lincoln Murphy
Sixteen Ventures

Frank Scavo said...

Lincoln's expansive comment reflects my own feelings about this subject exactly. Jeff, as usual, hits the mark, and I find his observation sobering that he has "never seen a successful escrow release." He's not just talking about SaaS here: he means, any escrow release, even for on-premise software. Makes total sense.

So software escrow for SaaS providers is a red herring. It doesn't address the real problem, which is how do I as a customer plan for business continuity if a SaaS provider is providing a critical system? The industry doesn't have a consistently good answer yet, and until it does this issue is and should be a barrier to SaaS adoption.

Lincoln's reply to the Anonymous question about creditors liquidating a bankrupt SaaS provider is also correct. Computer hardware has very little liquidation value. The hardware is much more valuable supporting an ongoing business, and as Lincoln points out, as long as their are paying customers there is probably a way to make a SaaS provider cash-flow positive by just cutting out all sales and marketing for example and just focusing on keeping the lights on.

So, generally, there should be some transition time. But the question remains, how do I as a customer make that transition?

Software escrow has little if any value in this scenario.

Jeff Gordon said...

There's one potential solution for business continuity that I really like and I've seen implemented only a handful of times - the "on-premise, vendor-maintained solution".

This is where the vendor essentially brings up the service within the customer's data center environment (ie: the customer has to first HAVE a data center for this solution to be a possibility). Then the vendor comes in physically or remotely to do all of the various maintenance tasks associated with the service. They worry about hardware problems, software issues, updates/upgrades, patches, bug fixes, etc. (Again, the customer has to either provide them with physical access - sometimes an issue; or with virtual access - also sometimes an issue.)

But the benefits are immediately visible: if the vendor goes away tomorrow, no big deal, you've still got the entire infrastructure running at your site... and you've got access to everything you need to migrate the data out of the system and go somewhere else if necessary.

Frank Scavo said...

Jeff, sounds like a traditional data center outsourcing arrangement, in the customer's own facility.

This doesn't seem to address the SaaS business continuity issue where I the customer want a cost-effective multi-tenant SaaS solution. Can't easily transfer that to an inhouse data center. Hence the need for some other third-party to be functioning almost as a hot-site for the SaaS provider, with rights/contracts in place to transition support under a whole range of scenarios (bankruptcy, natural disaster, etc.).

Jeff Gordon said...

Sorta' Frank. I think that's probably a good comparison.

But as a customer, I really don't care about multi-tenancy, rather, as you stated, I care about cost. I want something cheap. Always cheap. And if it's always cheap that you want, it's never going to be what you really want or need OTHER than cheap.

I like the idea of having a hot-site managed by someone else. I imagine the contracts would be a bit tricky, though, as if I were the SaaS provider, licensing my own infrastructure needs from other vendors, I would have a difficult time getting the assignment language worded in such a way as to provide the failover to the hot-site.

Maybe, and more interesting to me, would be the concept of putting the entire system into a legally-created trust. The trustee would be the SaaS vendor until such time as they went bust... and the backup trustee would be a group comprised of the CEO/CIOs of each of the customers using the solution.

Because the problem we're really trying to solve is smooth transition of this entire infrastructure... which doesn't work well when you consider the current state of software licenses and how vendors like to control their software when customers go bankrupt (grab any license you currently have and look under termination - most allow the vendor to terminate during customer's bankruptcy... not transfer to someone else).

So you need to find a solution that would enable the entire infrastructure to change ownership as a result of bankruptcy to some other entity (hence the trust) but still allow the SaaS vendor to retain control so long as they're a viable entity.

Frank Scavo said...

Jeff, now you're talking! Just like individuals have living trusts to provide for smooth transition to their heirs upon death, so a "living trust" for a SaaS provider would allow smooth transition to customers in the event of corporate death (bankruptcy).

This would be the perfect contingency plan, as it addresses the core issue: legal ownership.

Now, I'm not a lawyer and the Spectator does not give legal advice. And I don't know if any SaaS provider has looked at this option as a way of addressing customer concerns about business continuity, but they should.

Actually, this solution would also works for on-premise providers, wouldn't it? And it would be a lot cleaner than software escrow, which as you say, you never have seen successfully executed.

Michael Dunham said...

While all of these approaches are worth considering - we should all understand that the first consideration of a SaaS customer should not be perpetual access. At best, service failure continuity is an opportunity for transition to another solution in a planned way. After the service has died, there will be no updates and no matter who takes over reliability will take a backseat to simple operation.

Frankly, my feeling is much of this is FUD brought on by traditional vendors. What assurance do you have the company who developed your licensed software will continue to exist, provide maintenance updates, and keep on a path that is benificial to your business needs? Because of the business model behind SaaS - the vendor has much to lose if they don't continue to provide the service you are paying for and continue to improve their application. Of course, not every SaaS vendor really understands the importance of the customer relationship in their business model, but that is another subject.

Continuity plans are necessary for any line of business productivty system. Locally hosted applications often have backup failures, server outages and many other issues. The point is to understand your critical business operations and know what you will do to continue them in various scenarios while you move to an alternative arrangement. Getting into unnatural legal and contractual obligations beyond what is needed to accomplish that simple aim is counterproductive at best.

Mike Dunham
Scio Consulting

Ramiro Vergara R. said...

I have always been a disbeliever in the SaaS model.
If you can hardly trust investing in purchasing a license from "world class ERP" providers (just visit http://www.erpgraveyard.com/tombs.html ), much less you can trust providers in the SaaS arena. Sure, you can somewhat protect your investment having the right contract clauses but at the end of the day your most valuable information is in the hands of a supplier that may or may not dissapear and create chaos in your company.
Ramiro

Jeff Gordon said...

Mike:

I'm not worried about perpetual access. I'm worried about continuity - those are different concerns. Perpetuity implies that I want access forever, whereas continuity means that I don't want unexpected unavailability.

So the types of things we're considering here aren't "fixes" - they're band-aids, planned ways of handling a situation where the vendor goes away today and you're not affected until you want to be. The concept of escrow, for example, was to provide the customer with access to the source code for ongoing support... not for actual perpetual use. Eventually, you'll have to select a new vendor and move on.

This isn't just FUD from traditional s/w vendors (though I agree that it IS FUD to some degree). These issues apply to both traditional and SaaS vendors... if they go out of business today, I've got to find a new vendor to handle my issue. The question then is "when".

If I don't have any kind of alternative plan, in the SaaS world, I'm down until I pick a new vendor... whereas with the traditional vendor, I keep plugging along until the software dies. So I'm incented on the SaaS side to figure out a way to keep the status quo during a disaster. Thus, the potential solutions we're discussing.

But in the end, in either case, we're moving to a new vendor (which is why I still advocate always keeping your finger on the pulse of a second or third choice vendor).

Frank Scavo said...

Mike, I agree with Jeff on this. The problem is not perpetual access, it is transitional access.

Furthermore, you are right that the same issues apply to on-premise software licenses, but the issue is much more pronounced with SaaS vendors because I as a customer am not just relying on the vendor for software but for a complete solution, including data and infrastructure.

Finally, while traditional vendors might use this issue as FUD, as you say, be assured that correspondents such as Jeff and me do NOT represent vendors and we certainly have no interest in spreading fear and uncertainty. In this matter, we are only interested in helping buyers think through their options, understand their risks, and take appropriate steps to mitigate them.

Mike Dunham said...

I'm in total agreement - as I said, perpetual access is not a reasonable goal for continuity planning in any situation.

My problem is - that is the level the discussion gets too often when we get tangled in legal arrangements and contracts. I think there is reasonable horizon (time limit) and to a great degree what that horizon is depends on the individual use of an application (I might use it for 60 percent of my work - you might use it only marginally) and the value to my work. We need to focus on those issues first and what it will take to transition off the service or application and provide alternatives until it or a replacement is.

If I know my level of risk, I can decide what cost I can afford to bear for mitigation. If I don't really know my risk - any number of contracts and alternative providers are just noise I can't interpret.

There is another side though - in the best case the service vendor will determine what the likely level of risk is for the largest percent of their customer base and provided alternatives as a part of their service. I have worked with several vendors who are in critical line of business services and approach their market from that point of view and done quite well with it

Nina Baker said...

As a vendor of traditional, on-premises software solution, I do agree that cloud computing will work for many businesses. However, for our market sector – distributors supplying to manufacturers – it is not a good solution. There are just too variables with cloud computing that could go wrong and would seriously hamper their business.

As Google’s fiasco with Gmail earlier this month proves, the service could be over-whelmed and not accessible for several hours. The same applies if local phone or cable service is disrupted. For distributors supplying product to manufacturers that utilize JIT inventory, this would be a huge headache and loss of business. They make their reputations by being able to receive product and turn it around right away.

See Dianna Dilworth’s article for Direct Marketing News on this subject. http://directline.dmnewsblogs.com/2009/09/02/gmail-outage-raises-questions-about-cloud-computing/

Another very important thing to consider is that SaaS vendors must host it somewhere – there is a physical location that operates this service. What happens if disaster strikes this location? One major SaaS vendor lists this very concern on its SEC report:

Our services are delivered primarily out of a single data center. Any disruption of service at this facility could interrupt or delay our ability to deliver our service to our customers. We host our services and serve our customers primarily from a third-party data center facility with SAVVIS located in California. We do not control the operation of this facility. This facility is vulnerable to damage or interruption from earthquakes, hurricanes, floods, fires, terrorist attacks, power losses, telecommunications failures and similar events. Our data facility is located in an area known for seismic activity, increasing our susceptibility to the risk that an earthquake could significantly harm the operations of this facility. It also could be subject to break-ins, computer viruses, sabotage, intentional acts of vandalism and other misconduct.

This same company “offers” their customers an off-site back-up service for an additional charge! I’ve lived in Southern California and I can tell you the threat of earthquakes and fires is very real. If my company goes up in flames, our customers will still be able to operate. If they have a flood, hurricane, or have an angry employee destroy their server (all of which have happened), we have a disaster recovery plan in place to help them get back to business ASAP.

And yes Ramiro, many on-premise software providers have bitten the dust as is evidence by the ERP Graveyard, but those clients did not lose service, they were simply absorbed into a bigger company (and getting lesser service for higher rates, but that’s another post!).

Cloud Computing, like Oracle CEO Larry Ellison has said, is the latest fashion. Whether it is right for somone’s business should be thoroughly investigated before jumping on the bandwagon.

vinnie mirchandani said...

orry, Frank, I think it is more of a red herring - see my post Is SaaS too defensive

http://dealarchitect.typepad.com/deal_architect/2009/09/is-saas-too-defensive.html

Frank Scavo said...

My reply to my friend Vinnie (cross-posted from his blog):

Vinnie, as you know, I have no horse in this race, so can hardly be raising FUD in my post at
http://fscavo.blogspot.com/2009/09/saas-contingency-plans-need-more-than.html

Furthermore, I point out and Jeff Gordon ably confirms in the comments, software escrow is not the answer.

Contingency planning for potential loss of SaaS service is not a high priority for many SaaS customers (e.g. non-mission critical). But when the application is critical, it would be irresponsible to not the question of "what do I do if this provider goes out of business" or "what happens if this provider is negligent in his business continuity planning."

And yes, this is needed for on-premise solutions also. But the issue is more critical for SaaS solutions, because with on-premise software, if the vendor goes belly up, I don't lose my infrastructure. SaaS providers, on the other hand, are providing infrastructure and they are holding customer data. To simply say that these issues are red herrings doesn't them go away.

I don't think these issues are insurmountable, nor necessarily difficult to resolve. Jeff Gordon has some innovative suggestions. But I think these issues require more than a lot of hand-waving by vendors.

Amanda Kilgore said...

I work for Escrow Associates and would love to get the owner in on this conversation or possible set up a conference call. This is a very interesting blog. You guys can contact me anytime at 770.518.2451 ext.320 My Name is Amanda Kilgore and I am sure Chris would be willing to speak on this subject in detail at any time.

Florence said...

We do not have good software escrow agents here in India yet. But with each passing day, young software entrepreneurs are feeling the need of presence of good & reliable escrow agents without having to rely upon agents based abroad.