RIA Fires Google Apps After Google Notifies Advisor Of A 15-Month Failure In Gmail's Archiving System; Gmail Archiver Accidentally Deleted Messages It Was Supposed To Retain

Thursday, December 12, 2013 13:32
edit
RIA Fires Google Apps After Google Notifies Advisor Of A 15-Month Failure In Gmail's Archiving System; Gmail Archiver Accidentally  Deleted Messages It Was Supposed To Retain

Tags: advisor technology | client emails | compliance | email | FINRA | google | RIA compliance

An Investment Advisor rep with a background as a programmer recently replaced Google Apps' email archiving system after being notified that Gmail's archiver accidentally deleted messages that should have been retained for the 15-month period from March 28, 2012 to June 18, 2013. 

 
Google notified the advisor on September 5 of the failure, gave him a refund, and apologized profusely. In a lengthy mea culpa to Gmail archiving users affected by the failure, Google said it has rectified the problem and lists numerous steps it has taken to reduce the risk of another such failure. (The email sent by Google to the financial advisor about the failure of the Google Vault archiving system is pasted below.)

This Website Is For Financial Professionals Only


 

The failure was specifically attributed by Google to its Vault, where email archives are retained. Google says Vault failed to archive messages deleted by users. That’s not supposed to happen. To be clear, when a Gmail archiving user deletes an email from his mailbox, it’s supposed to be retained by Google's archiving system. That’s the system of record for an RIA, where regulators should be able to access all emails—especially the ones you've deleted.  

 
“In the weeks since we discovered the problem in July, we researched methods to identify and recover the deleted messages,” Venkat Panchapakesan, Vice President, Engineering at Google wrote in the September 5 email sent to Google Gmail archiving users who lost emails an unknown amount of emails. “We were able to restore messages that users deleted between June 19, 2013 and July 17, 2013 to the Vault archive. However, as part of normal operations, Gmail systems purge and permanently remove older messages, and we could not retrieve messages that users deleted between March 28, 2012 and June 18, 2013.”
 
Ironically, the RIA, whose owner asked not to be named, is run by an advisor who knows how to code in several computer programming languages and is a sophisticated technology buyer.
 
It's interesting to note that, if you search Google for press coverage failure of Google Vault to archive emails, you won't readily find any results or social banter mentioning Gmail archiving failure.  
 
“What I learned from this instance is that using software designed to be compliant for our industry is the most efficient way to handle these things,” he says. “While Google Apps was a cheap alternative, their history with snooping, changing terms and conditions, and deleting supposedly archived emails has very much soured me on them.”
 
----------------------------------
 

 

From: Google Apps Vault [mailto: This e-mail address is being protected from spambots. You need JavaScript enabled to view it ]

Sent: Thursday, September 5, 2013 4:10 PM

To: NAME REDACTED

Subject: Please Read: Archiving issue with your Google Apps Vault service

 

Dear Google Apps Vault Administrator,

We want to inform you of an issue that occurred with your Google Apps Vault service. Vault is designed to keep an archive of your organization's messages, including messages that your users delete. However, Google recently determined that the Vault service did not retain some deleted messages as it should have. Messages under legal hold were properly archived and not affected by the issue.

In this letter, we’ll explain what happened and how we fixed the problem, the refund you will receive, and how to contact us for any assistance.

Background: Vault retention rules

Admins can set up a default retention rule to control how long Vault retains their organizations’ messages. For example, the default rule can be set to retain all messages for 3 years. When users delete messages from Gmail, the messages are removed from their mailboxes, but should remain available in Vault.

Admins can also set custom retention rules in Vault. For example, they can set a custom rule to retain some users’ messages for 7 years instead of the 3-year default retention rule for all users. In the event of an investigation, admins can place a user on a legal hold so the user’s messages are exempt from deletion by any retention rules.

What happened with the Vault archive

On July 17, 2013, we discovered that the default retention rule had not been working as intended since March 28, 2012, the initial release of the service.

If a user deleted a message from their mailbox, the default retention rule did not archive the message in Vault. Instead, these deleted messages were permanently removed from Google's servers by the normal Gmail deletion process. This means:

             The default retention rule did not retain messages that users deleted between March 28, 2012 and June 18, 2013. As a result, these messages deleted by users are not included in your archive, unless they were subject to a custom retention rule or legal hold.

             All messages that your users did not delete have been archived by your default retention rule as expected.

             All messages subject to legal holds and custom retention rules have been archived as expected. Custom retention rules and legal holds were unaffected by this issue.

We have fixed this issue, and messages deleted by users after June 19, 2013 are now properly archived by the Vault default retention rule.

We sincerely apologize to your organization and users for not archiving your messages according to the default retention rule. We understand that you entrusted Vault with your messages, and we fell short of providing you with the complete service you expected and paid for.

The impact of this issue

In the weeks since we discovered the problem in July, we researched methods to identify and recover the deleted messages. We were able to restore messages that users deleted between June 19, 2013 and July 17, 2013 to the Vault archive. However, as part of normal operations, Gmail systems purge and permanently remove older messages, and we could not retrieve messages that users deleted between March 28, 2012 and June 18, 2013.

After thorough investigation, we determined that we do not have data about the number of messages affected by this issue. We regret that this information is unavailable. The actual impact to your organization depends on which messages were subject to the default retention rule (and not retained by any legal holds or custom rules), and how many of those messages your users deleted. Many Gmail users tend to archive rather than delete, in which case the messages were retained in Vault as expected. If you want to review your retention and archiving rules, please see these step-by-step instructions.

Ensuring the reliability of the service

We want to share our findings about the cause and how we will prevent issues like this from recurring. Our initial implementation of the default retention rule contained a defect in the archiving of deleted messages. As we did not have the correct monitoring in place, the retention defect persisted until we recently discovered the issue.

The Vault team and other engineers at Google investigated our internal processes and performed a complete technical review from development to production. Here are the actions we are taking:

             In engineering: We are expanding engineering reviews for all changes and new features to ensure that the code does what it's meant to do and works well with other Google systems. For the product requirements and development phases, we have developed more detailed use cases that describe combinations of default retention rules, custom rules, and legal holds.

             In testing: We are overhauling our quality assurance processes and rebuilding the test environment. To better detect potential issues, we are increasing our testing rigor and coverage—expanded reviews of quality assurance plans, improved tests, and more test cases, with deeper focus on retention.

             In use: We are improving our system reporting and analytics to monitor the health of Vault and its retention system as customers use the service—for example, to automatically alert the engineering and support teams of any unusual changes in the volume of archived messages. This will help us both detect and respond quickly to any issues that may arise in production.

Your service refund

Because Vault did not perform as intended, we are issuing you a full refund from the day you began payment for the service through July 31, 2013.  Within the next 30 days, we will send you information about receiving your refund.

Our commitment to you

We want you to know that our team is taking this issue extremely seriously. The Vault service has not lived up to the standards that you, as our customer, expect from us. We apologize to you for this issue—we can and will do better for you.

We are committed to providing an archiving service that's reliable, secure, and responsive to your business needs. We use Vault for our own retention of email, and we are confident in the service's capabilities. In the coming months, we will be working hard to earn back your trust.

If you have questions or need assistance, please contact us at our hotline at 1 855-675-1504 toll free in North America or 1 604-675-1504 (between 8AM to midnight EST, Monday through Friday) or by email at This e-mail address is being protected from spambots. You need JavaScript enabled to view it .

Sincerely,

Venkat Panchapakesan

Vice President, Engineering

Google

 

________________________________________

© 2013 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043.

You have received this mandatory email service announcement to update you about important information regarding your Google Apps Vault product or account.

 

 

 

 

 

Comments (3)

...
Chris Winn
While this was a Google issue, it is important to point out that an advisor's CCO and management is responsible for supervising their advisory persons, testing their systems and performing vendor due diligence. I make mention here not to defend Google nor knock the advisor.

The purpose is to remind everyone of the accountability that comes along with your job. An advisor can delegate to vendors, partners, employees, etc. However, while one can delegate certain responsibilities, you cannot delegate your accountability. If you have a vendor that fails, that is your failure.

The best way to mitigate the risk is periodic due diligence. If you don't look you are unlikely to find an issue.

Chris Winn , December 19, 2013
...
agluck
If a COO or CCO chooses a Google Apps to do the job for their firm, with its history of problems, how would regulators treat the RIA?

I was just warming up to buying a Google Chromebook. (The Acer C720 is the best one at the moment because it's a touchscreen at a breakthrough price), when this news about Google's Gmail email archiving failing.

I have been very slow to adopt public cloud systems for the very same reasons cited by the advisor who contacted me to say Gmail archiving is unreliable for professionals.

I've been saying this for years and nobody seems to care. I guess most regulators will let an RIA slide if his vendor loses records that are supposed to be retained. So RIAs have little risk, from that perspective.

However, a fiduciary who is supposed to be prudent, should not be using consumer apps to do a job in a professional practice. It's just a bad idea.

I believe it is irresponsible for a professional to use apps produced for the masses for communication subject to regulatory compliance. Email archiving can be done for nothing by printing out your emails or copying folders in Outlook. But it is a bad idea to do that because the possibility of human error is so high. I believe advisors should use apps created for the financial services industry for communications subject to books and records rules.

I've used Dropbox and SkyDrive and Box. Box is best. But I believe it is wreckless for a professoinal firm to use Dropbox and even SkyDrive. These apps make it too likely that an RIA will share client data that you were not supposed to share or delete information that you are supposed to retain. I may be biased, but that does not make me wrong.


Would love to hear Chris Winn and other compliance pros chime in on this
agluck , December 19, 2013
...
Chris Winn
My 2 cents...

There are several issues at play here. As I mentioned in my prior post, not matter which vendors a firm chooses the remain accountable. Remember LPL just paid $9M for this using "financial services industry" vendors.

In our routine compliance reviews, we encounter advisor security breaches with systems from enterprise-grade vendors to stand-alone machines.

Many applications start as consumer-grade and evolve to enterprise-grade applications. Google's failure was very public for one reason. They're Google. We expect more at nearly $1,100 per share.

I agree with enterprise-grade applications (or add-ons to enhance applications) are necessary. Many cloud applications you mention have enterprise-grade features, such as two-factor authentication, IP address whitelists, browsers-specific authentication, and forced complex passwords. However, many find these features inconvenient and either don't enable them or find a way to shut them off.

If we analyze the LPL issue, they used very reputable vendors (not naming anyone). But if it was not implemented or monitored, the result is failure.

If we think of the localized approach of using Outlook or saving hardcopies, or even on premise servers, these approaches all have potential shortcomings as well. I have encountered numerous instances where someone was reliant on Outlook and had a hard drive failure with no back-up. That is certainly not a solution.

The only answer is to evaluate not only the technology, but how the advisor and its clients will use the technology. Further there needs to be a real plan monitor and back-up the applications, even the enterprise-grade ones.

It is important to note as well that Google Vault is only available via Google Apps for Business, which is an enterprise-grade application suite. There are service-level agreements, SSAE16 audits, and a full array of security features. It is fair to say that they test most features with the Gmail public before rolling out to businesses. They clearly blew it at Google, but it seems the advisor did too, unfortunately.

I do not have an issue with registered investment advisors using these services if they implement the right controls. If they are not equipped or do not desire to manage the security with the same accountability the offer as client fiduciaries, they should only use industry-specific systems. Most industry systems don't give the options to loosen security.
Chris Winn , December 20, 2013

Write comment

You must be logged in to post a comment. Please register if you do not have an account yet.

busy