RIA Fires Google Apps After Google Notifies Advisor Of A 15-Month Failure In Gmail's Archiving System; Gmail Archiver Accidentally Deleted Messages It Was Supposed To Retain Hot
The failure was specifically attributed by Google to its Vault, where email archives are retained. Google says Vault failed to archive messages deleted by users. That’s not supposed to happen. To be clear, when a Gmail archiving user deletes an email from his mailbox, it’s supposed to be retained by Google's archiving system. That’s the system of record for an RIA, where regulators should be able to access all emails—especially the ones you've deleted.
From: Google Apps Vault [mailto:This email address is being protected from spambots. You need JavaScript enabled to view it.]
Sent: Thursday, September 5, 2013 4:10 PM
To: NAME REDACTED
Subject: Please Read: Archiving issue with your Google Apps Vault service
Dear Google Apps Vault Administrator,
We want to inform you of an issue that occurred with your Google Apps Vault service. Vault is designed to keep an archive of your organization's messages, including messages that your users delete. However, Google recently determined that the Vault service did not retain some deleted messages as it should have. Messages under legal hold were properly archived and not affected by the issue.
In this letter, we’ll explain what happened and how we fixed the problem, the refund you will receive, and how to contact us for any assistance.
Background: Vault retention rules
Admins can set up a default retention rule to control how long Vault retains their organizations’ messages. For example, the default rule can be set to retain all messages for 3 years. When users delete messages from Gmail, the messages are removed from their mailboxes, but should remain available in Vault.
Admins can also set custom retention rules in Vault. For example, they can set a custom rule to retain some users’ messages for 7 years instead of the 3-year default retention rule for all users. In the event of an investigation, admins can place a user on a legal hold so the user’s messages are exempt from deletion by any retention rules.
What happened with the Vault archive
On July 17, 2013, we discovered that the default retention rule had not been working as intended since March 28, 2012, the initial release of the service.
If a user deleted a message from their mailbox, the default retention rule did not archive the message in Vault. Instead, these deleted messages were permanently removed from Google's servers by the normal Gmail deletion process. This means:
• The default retention rule did not retain messages that users deleted between March 28, 2012 and June 18, 2013. As a result, these messages deleted by users are not included in your archive, unless they were subject to a custom retention rule or legal hold.
• All messages that your users did not delete have been archived by your default retention rule as expected.
• All messages subject to legal holds and custom retention rules have been archived as expected. Custom retention rules and legal holds were unaffected by this issue.
We have fixed this issue, and messages deleted by users after June 19, 2013 are now properly archived by the Vault default retention rule.
We sincerely apologize to your organization and users for not archiving your messages according to the default retention rule. We understand that you entrusted Vault with your messages, and we fell short of providing you with the complete service you expected and paid for.
The impact of this issue
In the weeks since we discovered the problem in July, we researched methods to identify and recover the deleted messages. We were able to restore messages that users deleted between June 19, 2013 and July 17, 2013 to the Vault archive. However, as part of normal operations, Gmail systems purge and permanently remove older messages, and we could not retrieve messages that users deleted between March 28, 2012 and June 18, 2013.
After thorough investigation, we determined that we do not have data about the number of messages affected by this issue. We regret that this information is unavailable. The actual impact to your organization depends on which messages were subject to the default retention rule (and not retained by any legal holds or custom rules), and how many of those messages your users deleted. Many Gmail users tend to archive rather than delete, in which case the messages were retained in Vault as expected. If you want to review your retention and archiving rules, please see these step-by-step instructions.
Ensuring the reliability of the service
We want to share our findings about the cause and how we will prevent issues like this from recurring. Our initial implementation of the default retention rule contained a defect in the archiving of deleted messages. As we did not have the correct monitoring in place, the retention defect persisted until we recently discovered the issue.
The Vault team and other engineers at Google investigated our internal processes and performed a complete technical review from development to production. Here are the actions we are taking:
• In engineering: We are expanding engineering reviews for all changes and new features to ensure that the code does what it's meant to do and works well with other Google systems. For the product requirements and development phases, we have developed more detailed use cases that describe combinations of default retention rules, custom rules, and legal holds.
• In testing: We are overhauling our quality assurance processes and rebuilding the test environment. To better detect potential issues, we are increasing our testing rigor and coverage—expanded reviews of quality assurance plans, improved tests, and more test cases, with deeper focus on retention.
• In use: We are improving our system reporting and analytics to monitor the health of Vault and its retention system as customers use the service—for example, to automatically alert the engineering and support teams of any unusual changes in the volume of archived messages. This will help us both detect and respond quickly to any issues that may arise in production.
Your service refund
Because Vault did not perform as intended, we are issuing you a full refund from the day you began payment for the service through July 31, 2013. Within the next 30 days, we will send you information about receiving your refund.
Our commitment to you
We want you to know that our team is taking this issue extremely seriously. The Vault service has not lived up to the standards that you, as our customer, expect from us. We apologize to you for this issue—we can and will do better for you.
We are committed to providing an archiving service that's reliable, secure, and responsive to your business needs. We use Vault for our own retention of email, and we are confident in the service's capabilities. In the coming months, we will be working hard to earn back your trust.
If you have questions or need assistance, please contact us at our hotline at 1 855-675-1504 toll free in North America or 1 604-675-1504 (between 8AM to midnight EST, Monday through Friday) or by email at This email address is being protected from spambots. You need JavaScript enabled to view it..
Sincerely,
Venkat Panchapakesan
Vice President, Engineering
________________________________________
© 2013 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043.
You have received this mandatory email service announcement to update you about important information regarding your Google Apps Vault product or account.
This Website Is For Financial Professionals Only
Office 365 For Advisor Compliance
Registered Reps With Their Own Independent RIA Must Keep Their Own Email And Social Media Archives In Addition To Their Broker-Dealer's Records
Ernst & Young Survey Discloses Willingness Of CFOs To Take Bribes
Disputes Over Axys Performance Numbers No Excuse For Fudging Client Reports, SEC Says
