Rackspace hosted Exchange suffered a disastrous blackout starting December 2, 2022 and is still continuous as of 12:37 AM December 4th. At first referred to as connection and login problems, the assistance was eventually upgraded to announce that they were dealing with a security incident.
Rackspace Hosted Exchange Issues
The Rackspace system went down in the early morning hours of December 2, 2022. Initially there was no word from Rackspace about what the issue was, much less an ETA of when it would be dealt with.
Customers on Buy Twitter Verified reported that Rackspace was not reacting to support emails.
This has actually been rather the day with #Rackspace. Every hosted exchange client has actually been down for 14 hours or so. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they succumbed to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace consumer privately messaged me over social networks on Friday to relate their experience:
“All hosted Exchange customers down over the previous 16 hours.
Not sure the number of business that is, but it’s significant.
They’re serving a 554 long delay bounce so individuals emailing in aren’t aware of the bounce for numerous hours.”
The official Rackspace status page used a running upgrade of the blackout however the initial posts had no details aside from there was an interruption and it was being investigated.
The first official upgrade was on December 2nd at 2:49 AM:
“We are examining a concern that is impacting our Hosted Exchange environments. More information will be posted as they become available.”
Thirteen minutes later Rackspace started calling it a “connection concern.”
“We are examining reports of connectivity concerns to our Exchange environments.
Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their email client(s).”
By 6:36 AM the Rackspace updates explained the continuous problem as “connectivity and login problems” then later that afternoon at 1:54 PM Rackspace revealed they were still in the “investigation phase” of the failure, still attempting to figure out what failed.
And they were still calling it “connectivity and login concerns” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
4 hours later on Rackspace referred to the circumstance as a “significant failure”and started offering their clients complimentary Microsoft Exchange Strategy 1 licenses on Microsoft 365 as a workaround until they understood the problem and could bring the system back online.
The main guidance stated:
“We experienced a significant failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any further concerns while we continue work to restore service. As we continue to resolve the source of the problem, we have an alternate option that will re-activate your ability to send and get emails.
At no cost to you, we will be providing you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 until additional notice.”
Rackspace Hosted Exchange Security Occurrence
It was not till almost 24 hr later on at 1:57 AM on December 3rd that Rackspace formally announced that their hosted Exchange service was struggling with a security occurrence.
The announcement even more exposed that the Rackspace specialists had powered down and disconnected the Exchange environment.
“After further analysis, we have figured out that this is a security occurrence.
The known impact is isolated to a portion of our Hosted Exchange platform. We are taking needed actions to evaluate and safeguard our environments.”
Twelve hours later on that afternoon they updated the status page with more info that their security group and outdoors professionals were still dealing with fixing the outage.
Was Rackspace Service Affected by a Vulnerability?
Rackspace has actually not released information of the security occasion.
A security event usually involves a vulnerability and there are 2 severe vulnerabilities currently in the wile that were patched in November 2022.
These are the two most current vulnerabilities:
Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack enables a hacker to check out and change data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an assailant is able to run harmful code on a server.
An advisory published in October 2022 explained the effect of the vulnerabilities:
“An authenticated remote assailant can perform SSRF attacks to escalate benefits and carry out arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mailbox server, the enemy can possibly get to other resources via lateral motion into Exchange and Active Directory environments.”
The Rackspace interruption updates have actually not shown what the specific issue was, only that it was a security occurrence.
The most existing status upgrade since December fourth stated that the service is still down and consumers are motivated to migrate to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make development in dealing with the occurrence. The availability of your service and security of your data is of high importance.
We have committed comprehensive internal resources and engaged world-class external knowledge in our efforts to lessen unfavorable impacts to consumers.”
It’s possible that the above kept in mind vulnerabilities belong to the security occurrence impacting the Rackspace Hosted Exchange service.
There has actually been no statement of whether consumer details has actually been jeopardized. This event is still ongoing.
Included image by Best SMM Panel/Orn Rin