File sharing company says backups unaffected, but non-backed up work could be lost

LucidLink logo

File sharing platform LucidLink suffered an outage, with the company reporting that it went down at 13:55 BST on 29 April and was mostly back online by 07:00 BST on 1 May. 

The company has blamed a hacking attempt, a, “malicious attempt against our core metadata service,” and says that all work that is backed up will be available when the service resumes. Backups are scheduled to run every six hours on LucidLink, and the company says that any work completed after the point of the last backup could be unrecoverable.

As of the morning of 1 May, the company has reported that the “vast majority” of filespaces have been restored, with a “small number” needing further manual intervention. These will be dealt with on a case-by-case basis. LucidLink also revealed that it will be creating a report on what happened, with the aim of making sure it can’t happen again.

It’s most recent update, at 07:00 on 1 May, LucidLink said: “We are happy to announce the restoration of filespace metadata service is now complete, and access to the vast majority of filespaces has been fully restored. As we expected, a small number of metadata instances failed to initialize properly and will require manual intervention. This is a time-consuming process, and our team will continue working until all customers are fully restored.

“If you are one of the customers who still has a filespace in a disconnected state, please create a ticket in our support system. We will proactively create tickets on behalf of affected accounts as we identify those without open tickets. Moving forward, we will switch our communications from general mass updates to individual tracking in our support ticket system.

“We understand that your organizations require additional in-depth accounting of what took place and why, as well as the steps being taken to mitigate the possibility of future occurrences. Now that we have brought the majority of our customers back online and reinforced our systems, we will continue implementing measures and will provide a report for you and your management. I do not yet have a timeline for this, but we are committed to sharing our findings.

“Thank you to everyone who has supported, encouraged, and provided moral support these past days. It is the energy of our customers that fuels the passion of our entire team.”

In the company’s previous update, at 16:30 BST on 30 April, it said that it believes the entire issue will be resolved in four to six hours, with some filespaces repaired much earlier - at 17:00 BST.

It stated: “No file data prior to the last backup before the outage was lost and no personal or corporate information was leaked. However, the metadata service for each filespace was damaged during this attack, which is causing the connection outage. We’re fixing the metadata service, which will restore access to your data. Each filespace’s metadata service is restored individually.

“We’ve identified some filespaces with metadata service that are unaffected but are unable to connect to the discovery service due to the metadata infrastructure attack. We have allocated additional resources to focus on that subset and will manually reconnect these filespaces to the discovery service, bringing them online. This should happen within the next hour, by 16:00 UTC.

“In addition, we are now in step two of the process, restoring the metadata state for all individual filespaces, and are on track to restore access to filespaces in the next 4-6 hours. We will do this on a rolling basis to get customers back online faster. During this process, we will not be able to notify customers individually when their filespace is back online. Customers will be able to monitor their connection status on the LucidLink app.”

In the previous update, at 13:15 BST on 30 April, CEO Peter Thompson reported that the issue will take another six to eight hours to be resolved (pointing to a 19:00-21:00 BST fix time in the UK), and that any work completed after 7:55 BST on 29 April was in danger of being unrecoverable, as that was the earliest an automated backup will have been completed before the outage. Individual users may have different backup times, but that is the “worst case”.

He wrote: “As promised, our team is continuing to work urgently on resolving this issue, and I wanted to provide an update on the current status.

“To restore filespace access from our backup systems, we are in the process of rebuilding the metadata infrastructure. This infrastructure consists of a discrete instance for each filespace. The recovery process is a 2-step process: 1) build the infrastructure and 2) restore each individual instance from a specific filespace backup.

“Our teams are well into step one of that process, rebuilding the infrastructure.

“Upon completing step two of this process, we will connect the new infrastructure to the discovery service, and customers will regain full access to their filespaces. Our estimate for this process is six to eight hours (18:00 to 20:00 UTC). We will do everything we can to accelerate this timeline.

“In the last update, we referred to the backup window. Here is additional information regarding the timing of this incident. The service went down at 12:55 UTC. As I noted above, our systems backup each filespace individually on a rolling basis every six hours. We cannot provide your individual backup time, but the worst case would be six hours prior to the incident or 6:55 UTC.

“Thank you for your patience and continued support. We will provide our next update at 15:00 UTC to inform you of our progress in regaining access to filespaces for all customers.

“We have created a status page to provide you with updates on the outage that occurred. We will be updating this page at least every three hours: https://lnkd.in/dKrXGKdE.”

Writing on Linkedin earlier, Thompson stated: “I know that everyone is eagerly anticipating the next update.

“Our priority is resolving filespace access for all customers. Rest assured, we will provide more detailed information on what caused the outage in the future.

“For additional clarity, what we initially thought was a DDoS service attack against our discovery service seems to have been a more malicious attempt against our core metadata service.

“No file data prior to the last backup before the outage was lost and no personal or corporate information was leaked. The metadata service now needs to be restored from our backups. As a result, there is a slight chance that a few hours of work from the point of the last backup to when the attack shut down the service will be unrecoverable.

“Our teams worked on developing the process for service restoration all through the night. They will next test and begin the restoration process.

“We will provide the next status update by 8am EST on Tuesday, April 30th.”

Social media users have complained at the situation.

LucidLink has been contacted for further comment.