VTB issue

Incident Report for TravelSpirit

Postmortem

Leadup

In the days before January 12th, performance issues are observed with our backup storage provider for the Visual Tour Builder database.
Support case opened with our Cloud Provider. They confirm known issue: daily saturation during midnight until early morning at Neurenberg location.
Workarounds implemented: upload chunk size increased from 5 MB to 256 MB, full backup interval changed from daily to weekly, backup schedule moved outside peak hours.
January 9th at 12:00: last successful full database backup completes before the incident.

Fault

January 12th at 00:11: scheduled security maintenance begins on production database cluster to enhance system security.
At 00:14 the database cluster enters unexpected failure state requiring restoration from backup.
At 00:30 on-call engineer alerted and begins immediate recovery procedures.

Timeline

January 12th (Sunday)
00:30 – 06:00: Multiple recovery attempts fail due to our Cloud Provider Object Storage timeouts during overnight saturation period. Each Point-in-Time Recovery attempt fails partway through due to S3 endpoint instability.
06:46: First recovery cluster deployed using workaround (S3 caching proxy).
08:14: Status page updated - "We are currently facing issues with opening and transferring VTBs. We have identified the issue and working on a solution."
09:15: Initial ETA missed - the Object Storage continues to cause recovery failures.
10:58: Status update - "Our estimate to be back online is now by the end of the day."
18:36: Status update - "We are working around the clock to recover from a database disruption that started last night at 00:30."
21:03: Status update - "We are making good progress and now expect to see initial recovery in the next hours."

January 13th (Monday)
01:05: Engineering team pauses overnight work due to fatigue; recovery strategy refined.
05:09: Recovery strategy pivoted to direct NVMe storage for improved I/O performance. Accepted to restore an outdated version of the database, which increased the chances of a recovery.
06:55: Production database core restored and validated, albeit an outdated version.
07:05: Status update - "We have been able to restore all VTB Data that was created or last edited before 09-01-2025 at 12:00 noon. Any changes to existing VTBs or VTBs created after that timestamp are currently unavailable."
09:13 – 14:30: Extended recovery session to restore individual tenant databases with edit timestamps after 09-01-2026 12:00 noon.
17:58: VTB Standalone and primary services fully restored.

January 14th (Tuesday)
09:00: Final databases restorations completed.
11:43: Status update - "The script for restoring the missing VTBs and data is running."
14:12 – 14:48: Brief issue with VTB creation resolved.

January 15th (Wednesday)
01:51: Final application updates deployed resolving remaining edge cases.
06:47: Status update - "The Visual Tour Builder is fully operational again and all VTB data has been restored."
10:07: Issue marked as resolved.

Root Cause

Primary cause:The production database cluster experienced an unexpected failure during infrastructure maintenance, requiring restoration from backups.

Contributing factor: Cloud Provider Object Storage performance issues during overnight hours (documented by provider) caused repeated recovery failures between 00:30 and 06:00, requiring implementation of retry logic and caching layer to complete restoration.

Scale factor: Multiple tenant databases required sequential restoration, each dependent on downloading data from the affected storage endpoint.

Learnings we took from this

When storage performance issues arose, temporary workarounds were implemented rather than migrating to an alternative provider. This resulted in the most recent full backup being from January 9, 12:00 CET, necessitating a three-day transaction log replay during recovery.

Follow up tasks

✅ Migrate all database backups to a different Cloud Provider S3 enterprise storage (completed January 15-17)
✅ Restore daily full backup schedule (completed January 15)
✅ Implement backup health monitoring and alerting (completed January 17)
✅ Deploy secondary recovery clusters in standby (completed January 14)
🔄 Implement cross-region backup replication (planned February 2026)
🔄 Automated backup verification testing (planned February 2026)
🔄 Disaster recovery runbook documentation (in progress)

Posted Jan 23, 2026 - 15:11 CET

Resolved

The issue has been resolved. All systems are back to normal, and data is available again.
We will publish a detailed post-mortem report next week.

Posted Jan 15, 2026 - 10:07 CET

Update

The Visual Tour Builder is fully operational again and all VTB data has been restored (including VTBs modified after 09-01-2026 12:00).
VTBs which are created between 09-01 12:00 and 12-01 23:00 are restored from backup are marked with a RECOVERED tag, but might have minor issues. No data has been lost and the development team is working on a restoring those.

Posted Jan 15, 2026 - 06:47 CET

Update

The creation of the VTB's is possible again. We continue to work on the restore of the data from the VTB's between 09-01 and 12-01.

Posted Jan 14, 2026 - 14:48 CET

Update

The creation of new VTB's is not working, we're looking into the cause.

Posted Jan 14, 2026 - 14:12 CET

Update

The script for restoring the missing VTB's and data is running for the VTB Stand Alone application. Once that has finalized, we can restore the data for customer with a TravelSpirit BackOffice subscription.

Posted Jan 14, 2026 - 11:43 CET

Update

Recovery plan is ready. We've verified the restore can be completed without service interruption, so we will proceed tomorrow during business hours. Missing data has been located and validated.

Posted Jan 14, 2026 - 01:03 CET

Update

We're continuing to work on restoring VTBs created or modified after 9 January, 12:00. This is taking longer than initially expected as we're prioritizing data quality and integrity. We want to ensure all restored VTBs are complete and error-free.

Additionally, since the application is now fully operational and many of you are actively working in VTB, we're taking extra care to prevent any conflicts between the recovery process and your current work.

All VTBs created or modified after 9 January, 12:00 remain safely stored in our backups. We're working through the recovery systematically and will continue to update this page as we make progress. We expect to be able to give a better time frame by end of business day.

Posted Jan 13, 2026 - 14:03 CET

Monitoring

A fix has been implemented and we are monitoring the results.

We have been able to restore all VTB Data that was created or last Edited before 09-01-2026 at 12:00 noon. Any changes to existing VTB's or VTB's created after that timestamp are currently unavailable. You may see newer VTB's visible in your Travelplan, but you can not open them.

This data is not lost. However, we need more time to restore this last part of our backups. We expect to restore these last VTB's later today. You can freely use the VTB for creating and editing new and existing proposals. We will restore any missing VTB's with timestamp after 09-01-2026 at 12:00 with a clear (RESTORED) flag so you can recognize them once restored.

A Post Mortem will be posted as soon as the incident is fully resolved.

Posted Jan 13, 2026 - 07:05 CET

Update

We have restored the Visual Tour Builder application to an online state and are now working on making all existing VTB data available.

Posted Jan 13, 2026 - 05:24 CET

Update

We are still working on this issue

Posted Jan 12, 2026 - 23:03 CET

Update

We are still working on this issue. We are making good progress and now expect to see initial recovery in the next hours.

Posted Jan 12, 2026 - 21:03 CET

Update

We are working around the clock to recover from a database disruption that started last night at 00:30. We will continue working on restoring the Visual Tour Builder database throughout the evening and night.

Posted Jan 12, 2026 - 18:36 CET

Update

We are continuing to work on a solution for the current issue. Our estimate to be back online is now by the end of the day.

Posted Jan 12, 2026 - 10:58 CET

Identified

We currently facing issues with opening and transferring VTB's. We have identified the issue and working on a solution.

Posted Jan 12, 2026 - 08:14 CET

This incident affected: Visual Tour Builder.