Osgrid is back online provisionally. It’s possible that certain assets may become unavailable for a while when the conversion process runs across them. You are able to use the grid and build, new assets will not be affected. Due to the back end team doing their database-voodoo the cluster now has enough “breathing space” to serve both the grid and the conversion processes in the background. Huge thanks to the back end team for their time and efforts to get us operational again.
Impact on regular operations is limited. What is important for now, is that you don’t wipe your existing region caches. Some older assets might become unavailable for a while. Now is also not the best time for making exports of your inventory (IAR’s). This is a bit of a nuisance, but VS keeping the grid offline until this process is finished, it’s a side effect we will have to live with.
Q & A :
Do you expect more downtime ? Not for as far as this process is concerned. That’s why it took this initial time. Everything is being monitored. Can never say never, but as it looks now, all is stable and chugging along, running as expected. We might have some brief region interruptions when plaza’s are being moved around as mentioned in the previous post.
When will all be back to normal ? This process can run anywhere between the next few weeks or next few months. With the grid being operational, syncing, and adding data there is no possible way to put a timeline on it. It has no “progress bar” or the likes you can stare at.
My region won’t work ? Check what you always check. Is the IP address still the same, port forwarding correct ? Did windows update automagically reset your firewall rules. The downtime should have zero impact on your region, your databases, or it’s configuration.
My Avatar won’t load properly ? Same, Possibly relog, try logging in a different region, rebake, change outfit, open edit appearance. If you just updated viewers / did that before the outage, click help / whitelist and follow instructions. This seems to be a thing, according to FS team. Don’t wipe your cache. That rarely is a fix, and will just force the viewer to download your whole region and inventory again. Slow for you, slow for those around you, and unlikely to make things better. Also older assets might be unavailable for some time, and in rare cases you might even need to grab another/different avatar.
Can’t we do this stuff without being down for so long ? Not without throwing money at it. The asset cluster is scalable, but when OSgrid got modernized over the past 4 years we didn’t financially plan for 4 times more data growth. So it’s sort of an unexpected surprise. We’re already happy it’s fixable. Also mind, we don’t have a dedicated team of open source database experts that sit and wait for some nice puzzle to appear, with a huge job attached behind it to fill their ‘spare time’ with. Again this is not an excuse, this is the reality we deal with.
Slow communications ? Not really. Just like normal. The website shows Offline, X has an alert message, Discord had a message. Other social media follows later, as the admin managing those is in the EU timezone, so picks up on that stuff when he sees it. (Mind you we don’t call each other on cellphones at midnight to tell somethings up with the grid. It’s not like the world is ending). An admin sees stuff failing, so he acts. And than maybe has his coffee and goes to his boss to do his 8 hour paid job, before he can even investigate. Sometimes another admin can pick up and fix it. Sometimes not. And before we can explain in a coherent story what’s wrong, what’s planned, and whats the answer to the “when” question. It just takes a little time. Screaming assumptions is generally not useful, and the result wont change from an uninformed message.
Ever considered starting with fresh Databases ? Yes. And No. No as in, will not happen. 2014 was a very costly outage due to a hardware failure. A lot of money and effort was spent back than in saving peoples worlds and things, and saving OSgrid. We inherited that. We can’t just wipe that and “start over”. Many of those people are no longer with us, but their legacy is. This grid sprouted from an idea, it has a soul, a history, with roots.
The OSgrid board would never consider such a “refresh”, unless the majority of the community would demand such. In the end, OSgrid is owned by its users. But we can’t and wont run away when things get hard. That’s not the mentality that made OSgrid exist for 17 years. And sure we’ve made mistakes, we have flaws, we’re not perfect. We do what we can, with the means we have, and are grateful for your trust and support in keeping OSgrid alive & online.
