Affecting System - PHL1 Network

• 03/16/2024 16:45
• Last Updated 03/16/2024 21:04

RESOLUTION @ 8:27 PM Eastern

Our data center's upstream connectivity was impacted due to a local hub being down. Service has been successfully restored.

UPDATE @ 7:18 PM Eastern

There is a hub that is down with an upstream ISP that is causing a larger outage not localized just to the PHL1 datacenter. Engineers are working to replace this equipment. No ETA as of yet, but we will continue to post here as more information is gathered.

INCIDENT START

We are currently investigating a datacenter wide outage in our PHL1 region. More updates to follow as more information is found.

Affecting System - ISP1

• 01/10/2024 02:15 - 01/27/2024 21:26
• Last Updated 01/27/2024 21:25

Resolution

Our ISP was undergoing a DDOS attack which caused these disruptions. This is now resolved.

Or customers are currently experiencing a loss of services related to an outage which is currently ongoing affecting our primary ISP. We are work with them to restore service ASAP. We are also working with our secondary ISP provider to determine why automated failover doesn't appear to have functioned properly.

More updates to follow as progress is made.

Affecting Other - phl1-c1-pm007.serviceplatform.network

• 09/13/2023 05:01 - 01/10/2024 02:48
• Last Updated 09/13/2023 17:46

FINAL UPDATE @ 5:12 PM Eastern

Various attempts at rebuilding the array have failed. At this point, we have no other option but to rebuild the node from scratch, as the data is not recoverable.

Please email support@launchvps.com if your VPS is affected, and we will provide solutions for your account.

UPDATE @ 2:05 PM Eastern

Attempts at rebuilding the array have failed. Continuing with additional methods.

UPDATE @ 11:56 AM Eastern

No meaningful updates yet. Continuing to troubleshoot the ZFS array and bring the system online.

UPDATE @ 9:37 AM Eastern

Progress made on RAID card issue. Attempting to rebuild ZFS array and bring node back online.

UPDATE @ 8:15 AM Eastern

Appears to be a RAID card issue. Contacting remote datacenter hands to assist with replacement.

UPDATE @ 6:41 AM Eastern

Continuing the investigation. Testing boot configurations to try to return the machine online. Attempting to rule out RAID card issue.

START @ 5:30 AM Eastern

This node is currently offline with a kernel panic. It appears that a disk has failed. We are investigating and attempting to resolve the issue.

Affecting System - Network

• 02/21/2023 21:16
• Last Updated 02/22/2023 10:41

UPDATE - This is now resolved.

We are currently seeing reports of some access issues with VPS on the 154.16.159.0/24 subnet.

It appears this only affects traffic routing through Equinix. 

We have raised issues with our IP vendor and attempted to contact Equinix for support.

We will continue to follow-up and post all updates to this issue here.

Affecting Server - phl1-c1.serviceplatform.network

• 11/18/2022 10:00 - 11/18/2022 12:32
• Last Updated 11/18/2022 12:33

RESOLUTION

Upgrades are completed.

Please let us know if you encounter any further issues.

UPDATE 

Host upgrade has begun. VPS and website reboots/downtime will start shortly.

START

LaunchVPS will be performing maintenance on KVM, LXC, and Shared Web hosts during the timeframe specified below. The hosts will be evacuated and upgraded to a new version of our virtualization platform software. Your service will lose connectivity for up to 5-10 minutes as hosts are rebooted. Working through these upgrades will allow us to offer more robust plans and additional features in the near future as well as ensure the stability and security of the platform.

Date and Time of Maintenance: November 18th, 2022 @ 14:00 UTC through 19:00 UTC

Description: In order to accommodate future features, stability, and supportability of the virtualization infrastructure, the current hosts will be upgraded and rebooted.

Services Affected: All KVM, LXC and cPanel Shared Web hosting services.

Downtime: Approximately 5-10 minutes of downtime while the service is rebooted.

Affecting Server - phl-cp-web001.webplatform.network

• 05/11/2021 17:44 - 05/12/2021 07:46
• Last Updated 05/12/2021 07:46

UPDATE

This has been resolved and all services are operating normally.

INCIDENT START

We are seeing issues with DNS which is affecting all services hosting on our cPanel Web Hosting product.

This has been corrected and all services will come online shortly as DNS propagates throughout the internet.

More updates to follow.

Affecting Other - Routing

• 11/10/2020 00:00 - 05/08/2021 10:32
• Last Updated 05/08/2021 10:32

UPDATE @ 4:12 PM

A new ROA has been pushed and the subnet now shows as valid as shown here:

https://rpki-validator.ripe.net/announcement-preview?asn=AS397384&prefix=154.16.159.0%2F24

Now, we will wait for the change to propagate. Once propagated, the issues should be resolved.

INCIDENT START

We have been made aware of an issue with the RoA status of our subnet 154.16.159.0/24. 

There appears to have been an error with the RoA for the subnet. We are working to correct this.

Due to this, the BGP routes are not accepted by some providers.

We are actively working with our IPv4 provider to correct this. We will continue to update as more information is available.

We apologize for this issue. Please feel free to contact us via opening a ticket if you have any further questions or concerns.

Thank you.

Affecting System - All VPS hosted on PHL1-C1-PM004

• 07/29/2020 21:30 - 07/30/2020 00:55
• Last Updated 07/30/2020 00:55

RESOLUTION @ 12:53 AM on 7/30/2020 EASTERN

The node is back online. After a ZFS scrub the server booted normally and all VPS have started as expected.

If you continue to have any issues, please do not hesitate to contact our support department via creating a ticket.

Thank you and we apologize for the long-running outage.

UPDATE @ 11:42 PM EASTERN

The resilvering process completed. The node is still not rebooting although the symptoms are different. We will run a scrub on the ZFS pool and try to reboot once again.

UPDATE @ 10:58 PM EASTERN

The ZFS resilvering process is running for the failed drive via a bootable rescue CD. Once this is complete, we will attempt to boot the node once more.

UPDATE @ 09:59 PM EASTERN

Attempting to boot with a rescue CD to rebuild ZFS array.

START @ 09:30 PM EASTERN

PHL1-C1-PM004 is offline. The node was performing a rebuild of a failed disk. During this rebuild, a process caused extremely high load on the node which rendered the node unreachable via network connection.

After a reboot, the node is reporting that it is missing a critical boot device. 

We are investigating.

Affecting System - PHL1-C1-PM001 Virtualization Node

• 08/16/2019 11:02 - 08/16/2019 13:01
• Last Updated 08/16/2019 11:05

PHL1-C1-PM001 is showing high load issues. You may have issues connecting to your VPS while we investigate the issue.

Affecting Other - PHL1

• 04/20/2019 20:00 - 04/20/2019 21:30
• Last Updated 04/20/2019 21:34

LaunchVPS will be performing maintenance on virtualization hosts and cPanel Shared Web hosts during the timeframe specified below. The hosts will have the latest security and performance patches installed. After the patch installation, the systems will be rebooted.

Date and Time of Maintenance: Saturday, April 20th, 2019 @ 08:00 PM Eastern

Description: Security and performance patches will be installed and each system will be rebooted in order for the updates to take effect.

Services Affected: All KVM, LXC, and cPanel Web hosting services.

Downtime: Approximately 5-15 minutes of downtime while the system is rebooted.

Contact: Email support@launchvps.com with any issues or questions.

Affecting Server - phl1-c1.serviceplatform.network

• 08/22/2018 21:00 - 08/22/2018 22:00
• Last Updated 10/30/2018 10:42

START

Affecting Server - phl1-c1.serviceplatform.network

• 10/30/2018 09:11 - 10/30/2018 09:25
• Last Updated 10/30/2018 09:27

UPDATE @ 9:25 AM Eastern

Node is back online and VPS are powering on. If you VPS is still unreachable it should be back online within 10 minutes. If it is not, please create a ticket with us so we can resolve ASAP.

Our apologies for the brief outage. We are working toward resolving this issue so this will not be a problem in the future.


ISSUE START

We are currently rebooting PHL1-C1-PM003 due to issues which are affecting VPS rebuilds and stops/restarts. We have reported this issue as a bug and are working to resolve it with future updates. Unfortunately, we must perform this reboot to bring all services back to a 100% functional state.

Updates will be provided within 15-20 minutes once the node is back online.

Affecting Server - phl1-c1.serviceplatform.network

• 08/11/2018 20:00 - 08/11/2018 21:01
• Last Updated 08/11/2018 21:43

RESOLUTION @ 9:01 PM Eastern

PHL1-C1-PM001 is now back online and VPS are powered on. Maintenance is complete.

UPDATE @ 8:47 PM Eastern

PHL1-C1-PM002 is now back online and VPS are powered on. PHL1-C1-PM001 will now be undergoing a reboot.

UPDATE @ 8:08 PM Eastern

PHL1-C1-PM003 is now back online and VPS are powering on. PHL1-C1-PM002 will now be undergoing a reboot.

UPDATE @ 8:00 PM Eastern

PHL1-C1-PM003 is now being rebooted.


START

Affecting Server - phl1-c1.serviceplatform.network

• 04/18/2018 09:40 - 04/18/2018 11:48
• Last Updated 04/18/2018 11:48

RESOLUTION @ 11:48 AM Eastern

All nodes are back online and VPS are powered on. Please let us know if you encounter any issues.

UPDATE @ 10:34 AM Eastern

PHL1-C1-PM001 has been rebooted and is now back online. All VPS are currently powering on. 

Moving on to reboot PHL1-C1-PM003.

UPDATE @ 9:59 AM Eastern

PHL1-C1-PM002 has bee rebooted and is now back online. All VPS are in a powered-on state. 

Moving on to reboot PHL1-C1-PM001.


INCIDENT START @ 9:40 AM Eastern

We are currently seeing management API issues and disk I/O issues on the PHL1 virtualization cluster. This appears to be a bug which is resolved by patching the running kernel. We have installed the new kernel and are performing reboots to apply the kernel.

Affecting Other - Node - phl1-c1-pm001

• 03/20/2018 17:04 - 03/20/2018 17:22
• Last Updated 03/20/2018 17:16

INCIDENT RESOLVED @ 22:16 UTC

VPS are back online. Investigation into root cause continues.

INCIDENT BEGIN @ 22:04 UTC

Node is exibiting high and load and critical services are failing.

Node is currently rebooting. Service will be online with-in 15 minutes.

• 02/25/2018 20:00 - 02/25/2018 20:30
• Last Updated 02/25/2018 20:16

INCIDENT RESOLUTION @ 8:16 PM

RAM has been updaded and all VPS are back online. Please feel free to reach out to us with any questions.


INCIDENT START @ 8:00 PM Eastern

LAX1-C1-PM002 is currently down for a scheduled RAM upgrade. A notification has been sent via email prior to this maintenance.

An additional update will be added once the maintenance has been completed.

Affecting System - PHL Network

• 01/24/2018 15:01 - 01/24/2018 16:07
• Last Updated 01/24/2018 16:35

INCIDENT RESOLVED @ 3:53 Eastern

Root cause has been identifed. Proper precautions have been taken and the affected areas of the network are back online.

INCIDENT BEGIN @ 2:40 PM Eastern

We are currently a DDos attack which is affecting the 143.16.159.0/24 range.

We are working to remedy this issue as soon as possible. Until the issue is resolved, customers using this range may see limited network connectivity.

More infomration to follow shortly.

Affecting Other - IPv6 Network

• 07/18/2017 12:07 - 01/16/2018 15:43
• Last Updated 01/16/2018 15:44

UPDATE 1/16/2018 @ 3:43 PM Eastern

The core router was upgraded and IPv6 speeds are optimal. Please reach out to us if you have any further questions. 

UPDATE 9/7/2017 @ 11:05 AM Eastern

Our datacenter provider will be upgrading the core network infrastructure in order to resolve this issue. The upgrade will be completed in 1 week.

Updates to follow.

INITIAL POST

We are tracking an issue where IPv6 networking is performing slowly for external internet facing requests.

This can also affect the speeds of IPv4 networking if IPv6 is the preferred route.

We are currently working with our data center support staff to pin down the issue. We have validated the issue does not present itself within switching or virtualization layer.

If you are affected by this issue, please review the following workarounds:

- Disable IPv6. If you do not rely on IPv6, you can disable the protocol until the issue is resolved.
- Prefer IPv4 over IPv6. Edit your gia.conf file in Linux appropriately to accomplish this. In Windows, please view this article for information on how to edit the prefix policies.

As always, please create a ticket if you require any assistance or have any questions. We will keep this issue updated as more information becomes available.

Affecting Other - PHL1 Network

• 01/09/2018 13:13 - 01/09/2018 15:29
• Last Updated 01/09/2018 15:30

RESOLUTION @ 3:29 PM Eastern

All systems are now online. Please create a ticket if you continue to have any issues. We apologize for any inconvience due to this issue.

UPDATE @ 2:44 PM Eastern

VPS systems with an IP from the 154.16.159.0/24 subnet look to be back online. Testing is still on-going, but so far all looks well. We will continue to test and resovle this incident when we determine the issue is in fact resolved.

UPDATE @ 2:21 PM Eastern

The 154.16.159.0/24 subnet is still seeing a BGP peering issue with Level3. We have escalated this issue to their support and are working with them to resolve. We currently do not have an ETA, but we are working to resolve this ASAP.

UPDATE @ 1:51 PM Eastern

We are still seeing issues with the 154.16.159.0/24 subnet. This is being worked now and should be back online shortly.

UPDATE @ 1:34 PM Eastern

Some services are beginning to come back online. Updates to follow.

UPDATE @ 1:19 PM Eastern

There are currently issues around BGP configuration in PHL. The network team is currently working to bring service back online.


INCIDENT START @ 1:13 PM Eastern

We are currently investigating a network issue in PHL. Currently, all PHL services are down.

Updates to follow shortly.

Affecting Other - Node - phl1-c1-pm001

• 01/07/2018 14:50 - 01/07/2018 15:24
• Last Updated 01/07/2018 15:31

INCIDENT RESOLVED @ 20:24 UTC

VPS are back online. Investigation into root cause continues.


INCIDENT BEGIN @ 20:09 UTC

Node is exibiting high and load and critical services are failing.

Node is currently rebooting. Service will be online with-in 15 minutes.

Affecting System - lax1-c1-pm001.serviceplatform.network

• 01/03/2018 07:15 - 01/03/2018 07:44
• Last Updated 01/03/2018 07:47

INCIDENT RESOLVED @ 12:44 PM UTC

System was rebooted and all VPS are powered on. We will continue to invesigate the root cause of the host issue.

Please do noe hestiate to contact us with any questions.


INCIDENT BEGIN @ 12:15 PM UTC

LAX1-C1-PM001.serviceplatform.network is not resonding to ping. Investigation is underway.

Affecting System - PHL1-C1-PM003

• 12/19/2017 12:43 - 12/19/2017 13:21
• Last Updated 12/19/2017 15:51

RESOLUTION

Node is back online and all VPS are powered on sucessfully. Incident is resolved.

INCIDENT START

Node is being rebooted due to high load caused by replication services which are hung.

Updates to following within 30 minutes.

Affecting Other - PHL1 Systems - Shared, LXC, KVM

• 11/11/2017 11:00 - 11/20/2017 21:00
• Last Updated 11/20/2017 11:21

FINAL UPDATE @ November 20th 4:21 PM UTC

Migrations are ccomplete. An update email will follow detailing new features and upgrades.

UPDATE @ November 17th 2:36 PM UTC

Migrations are conitnuing thorughout the rest of the week. 

UPDATE @ November 11th 4:22 PM UTC

PHL1-C1-PM003 is powering back on after hardware upgrade. LXC, KVM services on this node will be back online shortly. Shared Web services will also return online in the next 10 minutes.


SCHEDULED MAINTENANCE INITIAL SUMMARY

LAUNCH VPS will be performing maintenance on KVM, LXC, and Shared Web hosts during the timeframe specified below. The hosts will be evacuated and upgraded to a new version of our virtualization platform software. Your service will lose connectivity for up to 5-10 minutes as it is migrated to a new host. Working through these upgrades will allow us to offer more robost plans and additional features in the near future.

Date and Time of Maintenance: November 11th, 2017 @ 16:00 UTC to November 12th, 2017 @ 02:00 UTC

Description: In order to accommodate future features and supportability of the virtualization infrastructure, the current hosts will be evacuated and upgraded.

Services Affected: All KVM, LXC and Shared Web hosting services.

Downtime: Approximately 5-10 minutes of downtime while the service is migrated.

Contact: Email support@launchvps.com with any issues or questions.


Please do not hesitate to contact us with any questions or concerns. 

Affecting Other - Node - phl1-p1-nb-srv001

• 11/10/2017 10:16 - 11/10/2017 10:40
• Last Updated 11/10/2017 10:41

INCIDENT CLOSED @ 15:40 UTC

All VPS are back online. Root cause is continuing to be investigated.

Please reach out to us if you have any issues with your VPS powering on or functioning.


UPDATE @ 15:26 UTC

Node is back online and VPS are starting. If your VPS is not online yet, please log-in to the control panel and select the "Boot" option. 

We are continuing to monitor the node and will wrtie an update when all VPS are retruned to a powered-oin state.


INCIDENT BEGIN @ 15:16 UTC

LXC containers and KVM VPS were not responding to start/stop requests. Load on the server spiked to very high level. Investigating cause.

Node is currently rebooting.

Affecting Other - Node - phl1-p1-nb-srv001

• 10/05/2017 23:02
• Last Updated 10/05/2017 23:33

INCIDENT CLOSED @ 3:31 AM UTC

Reboot complete. All VPS are back online. Root cause is continuing to be investigated.

INCIDENT BEGIN @ 3:02 AM UTC

LXC containers were not responding to start/stop requests. Load on the server spiked to very high level. Investigating cause.

Node is currently rebooting.

Affecting System - PHL1-P1-NB-SRV001

• 07/05/2017 04:15
• Last Updated 07/05/2017 22:26

RESOLUTION @ 8:59 AM UTC
 
All VPS have returned to an online state. We will continue to monitor and investigate the issue for root cause.

UPDATE @ 8:26 AM UTC
 
Node has been rebooted. VPS are currently coming back online.

INCIDENT BEGIN @ 8:15 AM UTC

Node has been found to be unresponsive at console. We are investigating the issue.

Affecting System - LAUNCH VPS Network

• 05/14/2017 08:32 - 05/14/2017 13:17
• Last Updated 05/22/2017 13:03

We have detected an issue which affects the LAUNCH VPS network as a whole. You may see intermittent packet loss or system slowness.

We believe the issue has been resolved and linked to a misbehaving container. We will keep this event open until further verification is performed.

Please reach out to us if you continue to have issues.

Affecting System - PHL1 Virtualization Cluster

• 03/11/2017 23:00 - 03/12/2017 00:00
• Last Updated 03/12/2017 23:53

LAUNCH VPS will be performing maintenance on KVM and LXC virtualization hosts during the timeframe specified below. The hosts will be moved to a new rack in order to facilitate current and future growth. The hosts will be powered down and moved to an adjacent rack. From there, they will be powered on. All VPS will be verified they enter a powered-on state.

Date and Time of Maintenance: March 11th, 2017 @ 04:00 AM UTC

Description: In order to accommodate future growth of the virtualization infrastructure, the current hosts will be powered down and moved.

Services Affected: All KVM and LXC virtual server services.

Downtime: Approximately 30-60 minutes of downtime while the system is moved and powered-on.

Contact: Email support@launchvps.com with any issues or questions.

Communications: Further updates can be tracked by viewing Twitter and this status page.

Please do not hesitate to contact us with any questions or concerns. 

Affecting System - PHL1 Cluster API Services

• 02/03/2017 10:00 - 02/03/2017 12:00
• Last Updated 02/03/2017 16:35

We had an issue with our virtualization API services which resulted in reboots of our virtualization cluster. After many attempts to resolve the issue using methods which would not impact running VPS, we had no choice but to perform a reboot to resolve the issue.

We believe we have found the underlying issue which caused the instability. 

It appears a new NFS server which was attached to the cluster to host ISOs, LXC templates, and backups has been unstable during backup operations. This appears to be due to high I/O. The system will disconnect intermittently which causes issues with VPS with connected ISO media. This, in turn, will begin to affect the stability of cluster/management services, including API operations.

We will be replacing this NFS system as well as making other configuration changes which should alleviate the issue.

We apologize for the downtime and appreciate your patience as we work through this issue.

Thank you so much for your continued support. Please do not hesitate to email support@launchvps.com if you have any questions.

• 01/25/2017 11:02
• Last Updated 01/25/2017 12:09

INCIDENT RESOLVED:

All VPS are now online. We apologize for this unexpected outage and will begin investigation to ensure this does not occur in the future.

Please create a ticket with us if you have any questions or concerns.

• 01/03/2017 18:00 - 01/03/2017 19:56
• Last Updated 01/03/2017 15:25

LAUNCH VPS will be performing maintenance on the virtualization node where your VPS resides. We will need to shutdown the server in order to upgrade the amount of physical RAM. This will cause your VPS to shutdown and reboot, as well. An audit will occur after the node is powered-on to be sure all VPS are in a powered-on state. It is still recommeneded that customers check their VPS after the maintenance to verify all essential services are functioning properly.

Date and Time of Maintenance: January 3rd, 2017 @ 23:00 UTC

Description: Node will be shutdown in order to upgrade physical RAM.

Services Affected: KVM and LXC VPS services on the node, PHL1-P1-NA-SRV001.

Downtime: Approximately 10-20 minutes of downtime while the system is shutdown, upgraded, and powered back on.

Contact: Email support@launchvps.com with any issues or questions.

Communications: Further updates can be tracked by following this network incident page.


Please do not hesitate to contact us with any questions or concerns. 

• 08/09/2016 11:00 - 08/09/2016 11:30
• Last Updated 08/09/2016 14:21

LAUNCH VPS will be performing maintenance on the virtualization node on which your virtual server resides. We will need to reboot the server in order to enable nested virtualization features for current and future customer requests. The host restart will cause your VPS to reboot. An audit will run after the node reboot to be sure all VPS are in a powered-on state.

Date and Time of Maintenance: August 9th 2016 @ 3:00 PM UTC

Description: In order to enable nested virtualization functionality for our customers a virtualization host reboot will occur.

Services Affected: KVM and LXC VPS services on the node, PHL1-P1-NA-SRV001.

Downtime: Approximately 5-15 minutes of downtime while the system is rebooted.

Contact: Email support@launchvps.com with any issues or questions.


Please do not hesitate to contact us with any questions or concerns.

• 06/11/2016 14:44 - 06/11/2016 14:46
• Last Updated 06/11/2016 17:31

The virtualization node, PHL1-P1-NA-SRV001, experienced an unexpected reboot earlier this afternoon. The reboot appears to be linked to third party software which LAUNCH VPS uses to manage kernel updates. The software has functioned very well over time, but appears to have encountered a fault.

Unfortunately, it appears that directly after this software performed a live update, the virtualization node rebooted which is not the expected behavior. The node returned online without further issue. The 3rd party software is now functioning as it should. All VPS which were hosted on this node also experienced a reboot. LAUNCH VPS verified that all virtual servers returned to a powered-on state and there were no errors within the virtualization management software. A spot check was also performed to ensure our customer's VPS appeared to be functioning as they should.

LAUNCH VPS will investigate this issue further. We will also report this issue to the software vendor and ensure their team joins on the investigation.

We sincerely apologize for this inconvenient service interruption. If you have any questions or concerns please do not hesitate to contact us by creating a support ticket in the LAUNCH VPS control panel.

Thank you for hosting with LAUNCH VPS.

• 01/22/2016 22:39 - 01/23/2016 10:00
• Last Updated 01/23/2016 20:14

UPDATE @ 1:15 AM UTC 1/23/2016 BELOW

Updates are installed and bandwidth counting is functional.

All customer virtual servers are are started.

Maintenance is complete.

INITIAL POST BELOW

LAUNCH VPS will be performing maintenance on this KVM virtualization node. We will need to perform a reboot on the host node in order to apply patches. These patches are required in order to further troubleshoot issues related to bandwidth counting metrics for all virtual servers.