Log in

No account? Create an account
Catastrophic Data Loss - The Official LJ of FPS (and other dumb things) [entries|archive|friends|userinfo]
The Official LJ of FPS (and other dumb things)

[ website | Funday Pawpet Show Webpage ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Catastrophic Data Loss [Apr. 29th, 2008|08:55 am]
The Official LJ of FPS (and other dumb things)


[mood |distresseddistressed]

There was a Major failure of a RAID in the datacenter that hosts like 1/2 the furry fandom. Extent of data loss is not known at this time. Down time is not known at this time, and depends on if I have to re-write websites from scratch (Which will totally suck the big one). The status of things like running FPS live this sunday are still up in the air, as it may end up getting cancelled, but it too early to tell. The list of domains that I own that are currently affected:


All mail lists and mail to these domains are also down.

Also affected by this: lion.anthrochat.org and all of irc services. So popular irc channels like #fursuit are affected by this.

And before everyone asks.. No there was not a backup, since this is supposed to be a RAID, which is supposed to be data security.

The people at the data center are working on it, trying to put the bombshell back together from the exploded fragments. Hopefully I will get a status report soon :(

*crossposted to a bunch of places to get the word out*

[User Picture]From: rhanlav
2008-04-29 10:12 am (UTC)
Ack! Uh, damn. Thats not good. I hope things get back to normal soon.
(Reply) (Thread)
(Deleted comment)
[User Picture]From: reese_tora
2008-04-29 03:59 pm (UTC)
you beat me to it, RAID is NOT a backup, repeated becasue it's so very true.

incidentally, what is it with data loss this month... I hear this school district in the mid west had thier servers, onsite backups, and off site backups all fail simultaneously, losing three months of grades, and a librarian my mother communicates had a loss of a few months worth of card catalogue data(server and backups) recently as well...
(Reply) (Parent) (Thread)
From: actonrf
2008-04-29 11:09 am (UTC)
I can not agree more. Still the "data center" should keep a back up copy of the files. Secondly there no excuse not keeping a backup and working copy locally. I never rely on the web hosting copy to maintain data security and redundancy.
(Reply) (Thread)
[User Picture]From: dragonheadthing
2008-04-29 12:13 pm (UTC)

"hosts like 1/2 the furry fandom."... That's like, almost half the internet for me.
(Reply) (Thread)
[User Picture]From: aj_hyena
2008-04-29 12:59 pm (UTC)

On the bright side...

Who says YOU have to write websites? :P Give a couple of us one page to "repair" off a site, and we can have everything up in a matter of hours. For mostly-text like Pawpet.tv, at least.

If things are dead, I know of a couple servers we can at least get chat back for, so as not to remain out of touch.

I will be pissed if it's cancelled this week (as I'm not just going down to FL for a -day- if there's no show), but hey, Murphy's Law.

Data "security". Makes me wonder if it wasn't a deliberate attack. *cynic*

Ageis J. Hyena, signing off, over and out.
(Reply) (Thread)
[User Picture]From: dmnkttn
2008-04-29 02:22 pm (UTC)

Re: On the bright side...

... tell your inner cynic it's wrong this time. The failure was not a deliberate attack - just a really unfortunate flaw in the RAID. Which, I think, sucks worse.
(Reply) (Parent) (Thread)
[User Picture]From: aj_hyena
2008-04-29 03:06 pm (UTC)
Unfortunate at the most inopportune time, most definately. Then again that's always how these things happen.

In either case here's hoping -everything- gets recovered quickly, not just the FPS site.
(Reply) (Parent) (Thread)
[User Picture]From: kuddlepup
2008-04-29 03:00 pm (UTC)

Re: On the bright side...

I think the sites Yappy would be most concerned about are the ones for Osceola as those are ones for local government type organizations.

It wasn't a deliberate thing, it's just something that happened. And yes, it's a pain because it's impacting lots of things that many of us use daily. More than the pawpet site, it's impacted many people's e-mail capacity and other site specific information. (historical data, archives, logs, etc.)

Everyone is working under a strained capacity, so please bear with us. We're all trying to get things done with limited resources atm.
(Reply) (Parent) (Thread)
[User Picture]From: aj_hyena
2008-04-29 03:08 pm (UTC)
Eeeek. Yeah.

I wasn't aware Yappy worked for the government in some way... or was the "hub" that got wrecked just storage for both the FPS site and local govt? (I'm clueless about this sort of thing normally, so.)
(Reply) (Parent) (Thread)
[User Picture]From: reese_tora
2008-04-29 04:08 pm (UTC)
I guess this explains why I couldn't access certain sites you didn't mention last night; someone really does have all the internet on their computer, and that went down.

As a network/system administrator, the idea of eschewing backups for RAID makes me cringe.
(Reply) (Thread)
[User Picture]From: atkelar
2008-04-29 04:10 pm (UTC)
Yes, sch&"/§& happens... It's the part that makes life interesting. Anybody can succeed in a perfect world but it takes something to succeed in this one.

I think I experienced the downfall of the system first hand. I was browsing the art-jam archives when all the sudden: no response from server... Well, even non-windows systems go down now and then so I didn't give it much of a thought.

Anyway, hang in there and if I can do anything to help, let me know...

PS: I refrained from the old "RAID vs. Backup" speech because... well let's say there's always a next video submission when (not IF!) FPS comes back - and I'll try to get it ready for Sunday just in case... :-)
(Reply) (Thread)
From: zalno
2008-04-29 05:02 pm (UTC)
Well, damn. That sucks major ass. But then again, it isn't like we have other things to keep us occupied.

*goes off to play The World Ends With You in the meantime*
(Reply) (Thread)
[User Picture]From: film2edit
2008-04-29 06:29 pm (UTC)
Eeep. I hope they can be recovered!

If you need help rebuilding any of the sites, please let me know, and I'll try to help you.
(Reply) (Thread)
[User Picture]From: kuddlepup
2008-04-29 09:28 pm (UTC)
We may..and thanks very much for the offer. The site might need a whole revamp. Will see what the next few days entails.
(Reply) (Parent) (Thread)
[User Picture]From: thesonicgod
2008-04-29 10:20 pm (UTC)
Wow, this was totally unexpected... is the RAID failure software or hardware based? (Do the disks still work?)

web.archive.org didn't seem to hold onto anything useful. :/
(Reply) (Thread)
[User Picture]From: gamer_wolf
2008-04-30 12:58 am (UTC)
Yikes, I'm sorry to hear that this happened. I hope it all gets fixed soon.
(Reply) (Thread)