Service Under Maintenance

About This Site

Welcome to the Wilmaa's Tribe's home for incidents reporting. Here you can find the up to date status of our services and explore past incidents.

Uptime over the past 90 days. View historical uptime.
YalloTV App's Under Maintenance
90 days ago
99.97 % uptime
Today
Android TV Operational
90 days ago
100.0 % uptime
Today
Apple TV Operational
90 days ago
100.0 % uptime
Today
Mobile Android Operational
90 days ago
100.0 % uptime
Today
Mobile iOS Operational
90 days ago
100.0 % uptime
Today
Embedded Player Operational
90 days ago
100.0 % uptime
Today
LG TV Operational
90 days ago
100.0 % uptime
Today
Philips TV Operational
90 days ago
100.0 % uptime
Today
Samsung TV Operational
90 days ago
100.0 % uptime
Today
Web Operational
90 days ago
100.0 % uptime
Today
Streamprovider Operational
90 days ago
99.64 % uptime
Today
Other Under Maintenance
90 days ago
100.0 % uptime
Today
yallo.ch authentication provider Operational
90 days ago
100.0 % uptime
Today
MySports App's Operational
90 days ago
100.0 % uptime
Today
Apple TV Operational
90 days ago
100.0 % uptime
Today
Web Operational
90 days ago
100.0 % uptime
Today
Android TV Operational
90 days ago
100.0 % uptime
Today
Mobile iOS Operational
90 days ago
100.0 % uptime
Today
Mobile Android Operational
90 days ago
100.0 % uptime
Today
Streamprovider Operational
90 days ago
100.0 % uptime
Today
Subscriptions/Salesforce Operational
90 days ago
100.0 % uptime
Today
Other Operational
90 days ago
100.0 % uptime
Today
Backend Operational
90 days ago
100.0 % uptime
Today
Sunrise Moments Operational
90 days ago
99.99 % uptime
Today
Frontend (NextJS) Operational
90 days ago
99.99 % uptime
Today
Backend Operational
90 days ago
100.0 % uptime
Today
Tickets & Events Pipeline Operational
90 days ago
100.0 % uptime
Today
SSO/Login Operational
90 days ago
100.0 % uptime
Today
Infrastructure/WAF Operational
90 days ago
100.0 % uptime
Today
Storyblok CMS Operational
90 days ago
100.0 % uptime
Today
Wilmaa Internal Tools and Platforms Operational
90 days ago
100.0 % uptime
Today
VPN Operational
90 days ago
100.0 % uptime
Today
Internal Development Tools and Infrastructure Operational
90 days ago
100.0 % uptime
Today
Internal Office Tools and Infrastructure Operational
90 days ago
100.0 % uptime
Today
Everything else Operational
90 days ago
100.0 % uptime
Today
Wilmaa TV Platform Operational
90 days ago
100.0 % uptime
Today
Wilmaa Middleware Operational
90 days ago
100.0 % uptime
Today
Swiss-Ski Operational
90 days ago
100.0 % uptime
Today
App Operational
90 days ago
100.0 % uptime
Today
Sandbox Operational
90 days ago
100.0 % uptime
Today
Testing Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Apr 11, 2026

No incidents reported today.

Apr 10, 2026

No incidents reported.

Apr 9, 2026
Android Mobile NG v2.0.9 - Sprint 6.2
Completed - The scheduled maintenance has been completed.
Apr 9, 12:00 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 9, 11:00 CEST
Scheduled - Scope of this process:
-Staging / Pre-Production
-Production
-QA Testing
-BE Confirmation
Apr 8, 13:23 CEST
Backend maintenance
Completed - The scheduled maintenance has been completed.
Apr 9, 11:49 CEST
In progress - Scheduled maintenance posttponed to 09th
Apr 9, 08:30 CEST
Scheduled - Backend releases
Mar 31, 19:11 CEST
Apr 8, 2026
Apple tvOS OG v1.68.1 - Sprint 6
Completed - The scheduled maintenance has been completed.
Apr 8, 12:01 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 8, 11:01 CEST
Scheduled - Scope of this process:
-Staging / Pre-Production
-Production
-QA Testing
-BE Confirmation
Apr 1, 09:29 CEST
Apple NG iOS Mobile v2.3.0 - Sprint 6
Completed - The scheduled maintenance has been completed.
Apr 8, 12:00 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 8, 11:00 CEST
Scheduled - Scope of this process:
-Staging / Pre-Production
-Production
-QA Testing
-BE Confirmation
Apr 1, 09:25 CEST
Apr 7, 2026
Rewards Production Release
Completed - The scheduled maintenance has been completed.
Apr 7, 13:30 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 7, 09:30 CEST
Scheduled - We will be undergoing scheduled maintenance during this time.
Apr 1, 10:27 CEST
Long Loading Times
Resolved - This incident has been resolved.
Apr 7, 09:37 CEST
Monitoring - Incident Summary

At 10:15, we observed a significant increase in traffic to the rewards page, primarily triggered by a newsletter distribution. This surge exposed performance limitations in the new rewards platform architecture.

⸻

System Context

The previous rewards platform relied on a single component handling both frontend and backend logic.

With the new platform:
• The original core component remained largely unchanged
• A new secondary component was introduced
• Communication between components is handled via a Google Cloud PSC connection
• The new component is responsible for integrating with external systems, primarily GLOX

⸻

What Happened

Under increased load:
• Requests to GLOX began timing out after 15 seconds (application timeout threshold)
• Initial investigation considered a network issue, but no supporting evidence was found

⸻

Root Cause Analysis (Current Understanding)

The secondary component was designed for high performance using asynchronous processing. For each request to GLOX, it performs several operations:
1. Validation of key material received via PSC (to verify request origin)
2. 3scale credential validation and refresh
3. Telemetry/metrics collection

All of these operations were implemented asynchronously.

Under high load, we believe:
• The accumulation of asynchronous tasks led to contention within the event loop
• This resulted in a degradation of throughput, effectively resembling an I/O loop deadlock scenario

⸻

Mitigation

We implemented the following changes:
• Reduced pressure on the event loop by limiting asynchronous task consumers
• Moved key refresh operations to dedicated threads

These changes have improved system stability.

⸻

Why This Was Not Detected Earlier

During load testing:
• External systems (including GLOX) were mocked
• Testing focused on internal application performance only

As a result, the interaction between asynchronous processing and real external dependencies under load was not fully validated.

⸻

Current Status
• Overall system metrics have returned to normal levels
• However, a small number of requests are still timing out

This indicates:
• There may be additional contributing factors
• The incident likely resulted from a combination of architectural limitations and external dependencies

⸻

Next Steps
• Continue investigating residual timeouts
• Reproduce the issue in test environment with the original codebase
• Perform end-to-end load testing including real external integrations
• Review architecture for backpressure handling and async workload isolation
Mar 26, 19:37 CET
Investigating - A lot of timeouts causing long loading times
Mar 26, 17:55 CET
Apr 6, 2026

No incidents reported.

Apr 5, 2026

No incidents reported.

Apr 4, 2026

No incidents reported.

Apr 3, 2026

No incidents reported.

Apr 2, 2026

No incidents reported.

Apr 1, 2026

No incidents reported.

Mar 31, 2026

No incidents reported.

Mar 30, 2026
TVP Web Philips Production Release
Completed - The scheduled maintenance has been completed.
Mar 30, 15:46 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 30, 09:15 CEST
Update - We will be undergoing scheduled maintenance during this time.
Mar 27, 13:52 CET
Scheduled - Scope:
- Staging
- Pre-prod
- QA testing
- BE approval
- Prod release
Mar 27, 13:31 CET
TVP Web Tizen Production Release
Completed - The scheduled maintenance has been completed.
Mar 30, 15:46 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 30, 09:15 CEST
Update - We will be undergoing scheduled maintenance during this time.
Mar 27, 13:53 CET
Scheduled - Scope:
- Staging
- Pre-prod
- QA testing
- BE approval
- Prod release
Mar 27, 13:32 CET
TVP Web Browser Production Release
Completed - The scheduled maintenance has been completed.
Mar 30, 15:46 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 30, 09:16 CEST
Update - We will be undergoing scheduled maintenance during this time.
Mar 27, 13:53 CET
Update - Scope:
- Staging
- Pre-prod
- QA testing
- BE approval
- Prod release
Mar 27, 13:40 CET
Scheduled - We will be undergoing scheduled maintenance during this time.
Mar 27, 13:37 CET
TVP Web LG Production Release
Completed - The scheduled maintenance has been completed.
Mar 30, 14:15 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 30, 09:16 CEST
Update - We will be undergoing scheduled maintenance during this time.
Mar 27, 13:52 CET
Scheduled - Scope:
- Staging
- Pre-prod
- QA testing
- BE approval
- Prod release
Mar 27, 13:36 CET
TVP Embedded Player Production Release
Completed - The scheduled maintenance has been completed.
Mar 30, 12:40 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 30, 09:40 CEST
Scheduled - We will be undergoing scheduled maintenance during this time.
Mar 30, 09:37 CEST
Apple tvOS OG v1.68.0 - Sprint 6.1
Completed - The scheduled maintenance has been completed.
Mar 30, 12:01 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 30, 11:00 CEST
Scheduled - Scope of this process:
-Staging / Pre-Production
-Production
-QA Testing
-BE Confirmation
2 replies
Mar 27, 09:32 CET
Android TV OG v1.68.0 - Sprint 6
Completed - The scheduled maintenance has been completed.
Mar 30, 12:00 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 30, 11:00 CEST
Scheduled - Scope of this process:
-Staging / Pre-Production
-Production
-QA Testing
-BE Confirmation
Mar 27, 09:06 CET
Mar 29, 2026

No incidents reported.

Mar 28, 2026

No incidents reported.

← Incident History Powered by Atlassian Statuspage