WinterNode - WinterNode Game Panel Degraded Performance – Incident details

All systems operational

WinterNode Game Panel Degraded Performance

Resolved
Major outage
Started about 1 year agoLasted 2 days

Affected

Control Panels

Major outage from 8:06 PM to 3:54 PM

Game Control Panel

Major outage from 8:06 PM to 3:54 PM

Updates
  • Resolved
    Resolved

    No further prolonged outages have occurred since our last update. We'll be marking this as resolved for the time being.

    As a friendly reminder, issues with our Panel do not impact the accessibility of your game servers as they are hosted on separate infrastructure.

    We open similar incidents for all our services as well to keep you informed. You may view them on https://status.winternode.com/history/1 and https://resources.winternode.com for uptime as reported by our monitoring software.

    If you are getting the error "We're having some trouble connecting to your server, please wait…", please open a ticket in our Discord Server, as we may need to restart the daemon which interfaces between the Game Panel and your server. The easiest way to check, is to visit the Web File Manager.

  • Monitoring
    Monitoring

    We've received acknowledgement of the incident from our Panel Provider at 1:44 AM PT. A temporary fixed has been put into place at 2:30AM PT and an investigation into what occurred is underway by the Panel Team. The Game Panel is now online again.

  • Investigating
    Investigating

    We're getting a bit tired of formal messages… We know the Panel is down again 🤷‍♂️.

    Game servers hosted by us are not affected. It's been a rough couple of days for our Panel Provider. 😐 Feel free to take a look at the previous post-mortem provided to learn more about the current infrastructure setup.

  • Resolved
    Resolved

    A post-mortem was provided by our Panel Team - https://gamepanel.notion.site/11-10-2023-0d4aaef1626f4494b6238ef7044de522

    In summary: Currently all traffic is diverted to a small set of webserver Point of Presence (POPs) (mostly nl-ams-web01), but in the upcoming days 3 new POPs will be introduced and ready to serve you our Game Panel with lower latency and resiliency.

  • Monitoring
    Monitoring

    Our Panel Team is actively working on improving our Panel infrastructure to diversity the number of providers used to serve our Game Panel. After everything falls into place, it is expected to be uphill from here and a big reduction in the chances of downtime, as more webservers will be available to takeover any traffic from Points of Presence that go unavailable.

    Game Servers themselves are managed separately and are not affected.

    More information will be provided soon. We do receive automated alerts for whenever the Game Panel is unavailable.