All systems operational

Notice history

Nov 2023

MC.LON3, MC.LON4, MC.GRA2 Node Replacement
  • Completed
    November 27, 2023 at 8:14 PM
    Completed
    November 27, 2023 at 8:14 PM

    This has been completed.

  • In progress
    November 27, 2023 at 4:00 PM
    In progress
    November 27, 2023 at 4:00 PM

    Maintenance is now in progress

  • Planned
    November 27, 2023 at 4:00 PM
    Planned
    November 27, 2023 at 4:00 PM

    We will be performing node replacements for MC.LON3, MC.LON4, and MC.GRA2. This operation is part of our ongoing effort to optimize our services and take advantage of better pricing options from our hosting provider in these specific locations.

    Key Details of the Maintenance:

    1) Temporary Shutdown: All gameservers operating in the MC.LON3, MC.LON4, and MC.GRA2 locations will be temporarily powered off during the maintenance period. You will need to start your server once the maintenance has been completed.

    2) Game Panel Access: Access within the Game Panel for these nodes will be restricted while the maintenance is underway.

    3) IP Address Changes:

    • The majority of IP addresses will remain unchanged.
    • For clients using machine IPs (the direct IP obtained when pinging the hostname of these nodes), please be aware that these IPs will be renumbered.
    • This change should affect only a limited number of clients. We will provide individual notices to those customers once the migration is complete.

    4) Database Migration: Databases associated with these nodes will also be moved during this process.

    5) Data Integrity: We do not anticipate any data loss as a result of this maintenance.

    We understand the importance of uninterrupted service and are committed to keeping the downtime to a minimum. We appreciate your understanding and cooperation as we work to enhance our infrastructure and continue delivering exceptional service.

    Should you have any questions or concerns, please do not hesitate to reach out to our support team.

WinterNode Game Control Panel Unavailable
  • Resolved
    Resolved

    No further tickets or issues have been reported at this time.

    Please reach out to our Support Team through Live Chat or Discord if you need assistance with anything or have something to report.

  • Monitoring
    Monitoring

    Panel Team was able to restore more functionality and the number of reports have been reduced/squashed.

    You may experience some database creation issues due to firewall settings on our end (de02 & de03 being introduced into the PoP mix), if this is case, please reach out to our Support Team through Live Chat or Discord. However, you may need to wait until our normal internal operation team returns in the morning. We apologize for the inconvenience in this regard.

  • Identified
    Identified

    Panel Team was able to temporarily get the Panel back online again after bringing in an additional Point of Presence (PoP) to takeover traffic from the normal set of webservers. They are still actively working to stabilize this issue.

    In the event the Game Panel goes down again, please remember that the unavailability of the Game Panel does not affect online game servers as they are hosted on separate infrastructure.

  • Investigating
    Investigating

    Our Panel Team is currently investigating an incident regarding the inaccessibility of our Game Control Panel.

    All game servers currently turned on are not affected, as they are hosted on separately infrastructure.

WinterNode Game Control Panel Unavailable
  • Resolved
    Resolved

    The Panel Team has decided to rollback to Netherlands as our Game Panel has been affected for quite a bit of time.

  • Identified
    Identified

    Our Panel Team is conducting maintenance to switch from the Netherlands webserver to a Germany webserver.

    The Panel Team is actively working to restore accessibility of our Game Control Panel.

    The unavailability of the Game Panel does not affect online game servers as they are hosted separately.

Chicago nodes - connection issues
  • Resolved
    Resolved

    No further issues have been reported.

    Marking as Resolved.

  • Monitoring
    Monitoring

    An upstream routing issue was identified and remediated to resolve the issue.
    We are currently monitoring the issue. Please open a ticket if you are having troubles accessing your server.

  • Investigating
    Investigating

    We have detected a problem related to WinterNode.com Chicago services and are actively investigating.

    There are server connection issues resulting in not being able to connect to the server.

Oct 2023

WinterNode Game Control Panel Partially Unavailable
  • Resolved
    Resolved

    The panel should be back online and accessible. Thank you for being patient.

  • Investigating
    Investigating

    We are currently investigating an incident regarding the inaccessibility of our Game Control Panel for certian nodes. If you happen to have the issue accessing your server. Please open a support ticket.

Chicago Network Maintenance
  • Completed
    October 14, 2023 at 10:00 AM
    Completed
    October 14, 2023 at 10:00 AM

    Maintenance has completed successfully

  • In progress
    October 14, 2023 at 4:00 AM
    In progress
    October 14, 2023 at 4:00 AM

    Maintenance is now in progress

  • Planned
    October 14, 2023 at 4:00 AM
    Planned
    October 14, 2023 at 4:00 AM

    Our upstream provider has scheduled a maintenance window that is expected to have no impact to service.

    However, out of an abundance of caution due to the maintenance-related connectivity issues experienced last night, we are declaring this as an official maintenance window.

    We apologize for the short notice, but would like to offer you and your clients the greatest possible advance notice that further packet loss or connectivity loss may be experienced as our upstream providers improve capacity and redundancy.

    MAINTENANCE ID:       CHI603.101323
    START/END DATE:       10/13/23-10/14/23
    START TIME:           2300 CDT (-5 UTC)
    ESTIMATED END TIME:   0500 CDT (-5 UTC)

WinterNode Game Control Panel Unavailable
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Our Panel Team has informed us that the Game Panel should be recovered. Our monitoring also confirms that our Game Panel is online again.

    We'll be monitoring for the next few hours.

  • Identified
    Identified

    Our Panel Team has acknowledged and determined the issue regarding the Game Panel being unavailable.

    "We have 3 nodes up, but for some reason they can't vote a primary quorum, we're looking into it, this is likely the root cause of what's been going on over the last few days…starting recovery now"

  • Investigating
    Investigating

    We are currently investigating an incident regarding the inaccessibility of our Game Control Panel.

Undetermined problem - details to follow
  • Resolved
    Resolved

    We experienced another short blip for a few minutes around 12:55 AM Pacific Time. We are now outside of the scheduled network maintenance window, which ended at 5AM Central Time.

  • Monitoring
    Monitoring

    Our data center is currently performing network maintenance that was originally expected to have no impact on service or connectivity, and that is clearly not the case. We are monitoring the situation and are in contact with the facility.

  • Investigating
    Investigating

    We have detected a problem related to WinterNode.com services and are actively investigating.

    More details to follow.

WinterNode Game Panel Degraded Performance
  • Resolved
    Resolved

    No further prolonged outages have occurred since our last update. We'll be marking this as resolved for the time being.

    As a friendly reminder, issues with our Panel do not impact the accessibility of your game servers as they are hosted on separate infrastructure.

    We open similar incidents for all our services as well to keep you informed. You may view them on https://status.winternode.com/history/1 and https://resources.winternode.com for uptime as reported by our monitoring software.

    If you are getting the error "We're having some trouble connecting to your server, please wait…", please open a ticket in our Discord Server, as we may need to restart the daemon which interfaces between the Game Panel and your server. The easiest way to check, is to visit the Web File Manager.

  • Monitoring
    Monitoring

    We've received acknowledgement of the incident from our Panel Provider at 1:44 AM PT. A temporary fixed has been put into place at 2:30AM PT and an investigation into what occurred is underway by the Panel Team. The Game Panel is now online again.

  • Investigating
    Investigating

    We're getting a bit tired of formal messages… We know the Panel is down again 🤷‍♂️.

    Game servers hosted by us are not affected. It's been a rough couple of days for our Panel Provider. 😐 Feel free to take a look at the previous post-mortem provided to learn more about the current infrastructure setup.

  • Resolved
    Resolved

    A post-mortem was provided by our Panel Team - https://gamepanel.notion.site/11-10-2023-0d4aaef1626f4494b6238ef7044de522

    In summary: Currently all traffic is diverted to a small set of webserver Point of Presence (POPs) (mostly nl-ams-web01), but in the upcoming days 3 new POPs will be introduced and ready to serve you our Game Panel with lower latency and resiliency.

  • Monitoring
    Monitoring

    Our Panel Team is actively working on improving our Panel infrastructure to diversity the number of providers used to serve our Game Panel. After everything falls into place, it is expected to be uphill from here and a big reduction in the chances of downtime, as more webservers will be available to takeover any traffic from Points of Presence that go unavailable.

    Game Servers themselves are managed separately and are not affected.

    More information will be provided soon. We do receive automated alerts for whenever the Game Panel is unavailable.

Sep 2023

VPS Hypervisor Upgrade
  • Completed
    September 23, 2023 at 6:00 PM
    Completed
    September 23, 2023 at 6:00 PM

    Maintenance has completed successfully

  • In progress
    September 23, 2023 at 5:00 PM
    In progress
    September 23, 2023 at 5:00 PM

    Maintenance is now in progress

  • Planned
    September 23, 2023 at 5:00 PM
    Planned
    September 23, 2023 at 5:00 PM

    This notice is to make you aware of an upcoming scheduled maintenance that we will be conducting and have applicable impact to some of our VPS Nodes.

    SCHEDULED MAINTENANCE WINDOW
    Scheduled Start: Saturday, September 23, 2023 12 PM CT

    Scheduled Finish: Saturday, September 23, 2023 1 PM CT

    SCOPE OF WORK
    VPS1.CHI
    VPS2.CHI
    VPS3.CHI
    VPS4.CHI

    IMPACT
    We will be upgrading the hypervisor. All customer VMs will be briefly powered down before the node is restarted during the upgrade process. We anticipate customer VMs to be offline for approximately 10 minutes during this window.

CHI Services were temporarily Unavailable
  • Resolved
    Resolved

    We experienced another small blip around 11:14PM PT for a few minutes, but did not result in another major outage of our CHI services.

    A daemon restart was required and fulfilled for all CHI nodes around 9:50PM PT to re-establish connectivity with our Panel.

    Follow-ups have been scheduled for the morning, we'll be marking this incident as resolved for now and release a Reason For Outage if available.

    If you are experiencing any lingering issues, please reach out to our Support Team on Discord.

  • Monitoring
    Monitoring

    At 9:13PM PT we received automated alerts regarding a problem related to WinterNode.com services.

    At 9:22 PM PT affected services returned to normal operational status, our team will be monitoring to escalate further if necessary.

Sep 2023 to Nov 2023