GotSport down - Tournaments are going to have fun this weekend

RandomSoccerFan

RandomSoccerFan

GOLD
Looks like GotSport has been knocked offline for a few days (link). No details available, but it's either user error or ransomware, and the latter should be considered a version of the former at this point. We just got a message from a tournament we are in for Saturday with an alternate spreadsheet, and they were lucky they had fully downloaded everything prior to it going down. The hope is they have usable backups for the years of game data stored there and can get it back online prior to next weekend.

gotsport error.jpg
 
Looks like it will continue to be down at least through Wednesday. The tournament we played in (and won!) this weekend was managed with a shared Google Sheets. TBH, they kept it up to date and accurate, with quick updates and no real issues - but I'm sure that on the back end is was much more work for them getting the spreadsheets set up and managed throughout the weekend. Hopefully all is back to normal for next weekend.

gotsport error5.jpg
 
RandomSoccerFan said:
Looks like it will be down until sometime next week.

View attachment 15769
Click to expand...
Someone is learning a hard lesson in the difference between backups of data (which provide copies of the data only) and system level backups which provide backups of the entire server. (Which generally provide a "snapshot" of the entire server at a specific point in time.)

In a nutshell having the data is nice but if you dont have the server + applications getting things running again means rebuilding the entire server + applications so the data can be restored to a running configuration.

I bet in the future they backup both data and servers configs.
 
In theory, I'm sure they thought they had full backups of everything. It's too bad that their first real test of them is showing how ineffective the backup strategies and implementations were. This is why you test. Maybe not every week, or month, but going more than a year without a full recovery test isn't a prudent bet for any business that relies on the system for its livelihood.

Full snapshots of the servers online should be the bare minimum, and would be the worst case in going back to a few days ago, or a week ago. The complications of recovering from backups for live systems that have continuous data input, is how do you capture the interim data in such a way that it can be loaded correctly to the recovered system - that's often the harder part of full recovery tests. But for something like GotSport, I imagine if someone said click this switch and everything is the same as it was yesterday or the day before, and people have to reenter data for those few days, they'd likely have taken that offer immediately.
 
RandomSoccerFan said:
The complications of recovering from backups for live systems that have continuous data input, is how do you capture the interim data in such a way that it can be loaded correctly to the recovered system - that's often the harder part of full recovery tests.
Click to expand...
Transaction logs that are being continually written, to record all creates/changes on the system and continually written offline so that they are not affected by an outage.

Restore the server data, run through the transaction logs for the period from the last backup to the outage and you have an operational system as up to date as you can get without complete mirroring etc.

Its all in the design as they say ...
 
Of course, but the devil is always in the details. "The System" is often a collection of various apps and data feeds that create a variety of different transaction logs, from other disparate systems, divisions, or even companies. The simpler everything is, the easier it is to envision how backups and restores could be designed and implemented. The more complicated it is, and now you find yourself with 200 people planning for 6 months prep for how to do a full data center disaster recovery test over a weekend, followed by 3 months of post-mortem about what didn't go as well as it could have with which entities - to better prepare for the next one 6 months later.

Virtualization and Cloud technologies have made much of this both conceptionally and operationally more effective/quicker/safer/cheaper, but none of it has gotten us much closer to vaunted big red "Easy" button we all wish we could press!

If there is any good news for GotSport, I can say that every time I've been a witness to an IT-related cockup in this vein, money/resources/support magically appears to at least try and address what led to the problem in the first place to keep it from embarrassing the company again any time soon. Never let a crisis go to waste, and all that....
 
I hope they do a public post mortem on this. It looks like they're hosted at "flexential" (never heard of it myself). Not sure if they're using their cloud services or if they're colocating bare metal. None of that sounds good considering GCP, AWS, or Azure would all be more standard choices. My guess is they don't have backups. They're probably using a nosql data store where the data is spread across several nodes (cassandra/scylladb, mongodb, etc). The data is probably still "there" but they lost the mapping of it all. This is just a guess. If they had backups of a simple SQL database, they would've been back up days ago.
 
The thing that really makes me raise an eyebrow is "Gotsport has assembled the country's best forensic data recovery experts". That makes me think they're working with a company like DriveSavers or Gillware. If that's the case, then they have some disks that have gone bad. I REALLY REALLY hope that's not what's going on here. I've gone through that process and it's not pretty. If this is the case, the future is looking pretty bleak for GotSport. If this goes on much longer I wouldn't be surprised if entrepreneurial minds don't start conjuring up something. I've never seen this kind of outage before.
 
NorCalDad said:
The thing that really makes me raise an eyebrow is "Gotsport has assembled the country's best forensic data recovery experts". That makes me think they're working with a company like DriveSavers or Gillware. If that's the case, then they have some disks that have gone bad. I REALLY REALLY hope that's not what's going on here. I've gone through that process and it's not pretty. If this is the case, the future is looking pretty bleak for GotSport. If this goes on much longer I wouldn't be surprised if entrepreneurial minds don't start conjuring up something. I've never seen this kind of outage before.
Click to expand...
I agree this is not good.

Backups (like insurance) are only important when you need them.
 
What I don't understand is that this is not a complex site with not a whole lot of data. I mean, the backup for the entire site is probably in the GB's and not TB's.
 
They are almost certainly using firms like that. They likely had disks go bad, backups that were unusable, and knowledge on how to recreate some critical pieces left the org years ago. They are completely hosed. When they say "All the data is there", I'm imagining them all standing around in a circle pointing at a physical hard drive saying "but it's right there!" when in reality it's way beyond usable. Unfortunately, those disks are probably sitting at Flexential, nowhere near where anyone from GotSport physically resides, but the mental picture still makes me chuckle.

NorCalDad said:
If this is the case, the future is looking pretty bleak for GotSport. If this goes on much longer I wouldn't be surprised if entrepreneurial minds don't start conjuring up something.
Click to expand...

Eh, there's just not much money in it. I don't think many of the thousand+ clubs that are using GotSport want to swap out for a new solution, or pay any real money for it either. What they provide is just slightly above a free service from a single club's perspective - but at scale it's useful for the sport as a whole. If they come back online with none of the old data, but are able to provide tournament/league scheduling going forward as people load their new data, odds are everything starts working like it did before, rather than any new real investment of alternatives.
 
RandomSoccerFan said:
Eh, there's just not much money in it. I don't think many of the thousand+ clubs that are using GotSport want to swap out for a new solution, or pay any real money for it either. What they provide is just slightly above a free service from a single club's perspective - but at scale it's useful for the sport as a whole. If they come back online with none of the old data, but are able to provide tournament/league scheduling going forward as people load their new data, odds are everything starts working like it did before, rather than any new real investment of alternatives.
Click to expand...

What do clubs pay GotSport anyway? Curious what they're pulling in.
 
You must log in or register to reply here.
Back
Top