Server Crash

Gizmo
High Level Donor
Gizmo's picture
Posts: 397
Joined: 2007-03-06
User is offlineOffline
Server Crash

Server crashed again.  Unfortunately because of an ongoing problem that has affected the server for months, the backups stopped running.  This made it to where I had to restore it far further than normal.  All I can really say is I apologize for this.

 I have informed Brian that I need to distance myself from administrating the site.  When I originally took over it was because the previous guy had to leave the country and gave rather short notice on needing to find another solution.  I picked up where he left off, and then moved it to a server that was hosted somewhere where I have had good luck with customer service.  Then about 6 months ago I got a new job and moved out of my parents house to my own place.  This caused not only a lot of stress but also a lack of the time I used to have to dedicate to working on the site.  Also, originally I had a lot more disposable income (was paying about 200 in rent to my parents, now paying 600 + utiltiies which comes out to about another 200-300 depending).  Tho I got a raise in the new job, it doesn't quite offset that difference at all.  Luckily many people here have stepped up and donating some money to help out on the server costs (which I appreciate much).  

That all being said, I feel that I can't dedicate the time and love that the server needs.  Brian, Kelly and Rook (among many others) work very hard and for Brian, Rook and Kelly this is pretty much what they do for work (certainly would be completely if that was possible at this point, and its getting there).  However I don't have that same kind of dedication nor time.

I felt it necessarily to explain some of this so others can hopefully see a little bit where I am coming from.  This all being said, again I apoligize for todays issues.  Also, Brian and I will be looking for people to take over some of the backend tasks (like checking to make sure the backups run, optimizing mysql tables and what not).  If you have experience in mysql and/or linux (especially command line stuff) please let Brian or I know.  As always you can find me on skype as gizmoiscariot or you can email me at .  


dead_again
Special AgentWebsite Admin
dead_again's picture
Posts: 251
Joined: 2007-05-13
User is onlineOnline
sorry to hear that happened.

sorry to hear that happened. I know how daunting these things can be, seeing that I am a network administrator. I added you to my contact list on skype so feel free to IM me or, if your lucky to catch me, call me. I have a lot of Linux CLI (command line interface) experience and some MySQL experience. I also know python for scripting and some shell scripting.

Idiocracy was not a documentary; Can we please keep it that way?


mr804
ModeratorSpecial Agent
mr804's picture
Posts: 80
Joined: 2007-11-04
User is offlineOffline
I have  14 year experiance

I have  14 year experiance with many different flavors of linux/unix if whoamever needs help. Feel free to message me.


Sapient
High Level DonorRRS CO-FOUNDERRRS Core MemberWebsite Admin
Sapient's picture
Posts: 4747
Joined: 2006-04-18
User is onlineOnline
Folks, feel free to use

Folks, feel free to use google cache to try and find works you wrote in the last few days but are now gone.  Kelly lost about 50 hours of editing and site work, I lost about 75-100.   I hope to make a vid on the issue.  

 Don't let it get you down folks.  We remain determined and are consistently looking to figure out the problem and prepare the right solution.  

 

He's not the man he wants you to think he is. Go to The Real John McCain to learn about the double talk express. View these videos about John McCain to learn about the flip-flopping double talker. Help me get rid of a transferable credit with Southwest Airlines and you'll save $60.


wavefreak
Theist
wavefreak's picture
Posts: 1826
Joined: 2007-05-10
User is offlineOffline
Are you hosting or

Are you hosting or co-locating?


cam
Posts: 77
Joined: 2007-11-19
User is offlineOffline
Is their an email list for

Is there an email list for tech people? A collective of sorts? I'm a sysadmin.


I AM GOD AS YOU
SuperfanBronze Member
Posts: 3723
Joined: 2007-09-29
User is onlineOnline
   I AM  is CRYING TEARS

   I AM  is CRYING TEARS  


dave805
dave805's picture
Posts: 78
Joined: 2007-12-27
User is offlineOffline
So is this why all my posts

So is this why all my posts are gone and i had to resign up on the forums?? I thought at first i was banned..


Sapient
High Level DonorRRS CO-FOUNDERRRS Core MemberWebsite Admin
Sapient's picture
Posts: 4747
Joined: 2006-04-18
User is onlineOnline
Yup Dave, sorry bout that.

Yup Dave, sorry bout that.

He's not the man he wants you to think he is. Go to The Real John McCain to learn about the double talk express. View these videos about John McCain to learn about the flip-flopping double talker. Help me get rid of a transferable credit with Southwest Airlines and you'll save $60.


Visual_Paradox
ModeratorSpecial Agent
Visual_Paradox's picture
Posts: 315
Joined: 2007-04-07
User is offlineOffline
Yes, early this morning the

Yes, early this morning the server crashed. Sapient said in the Stickam chatroom last night (read: "this morning&quotEye-wink he was going to get a little bit of sleep but I noticed the server crash thirty minutes later, causing him to be woken up to work on the problem. After a while I think they (Brian and Kelly) decided to wait for Gizmo. But yes, the information that didn't appear in the latest backup was lost. As Brian said, you can use Google's cache to help put some of that information back on the website, and I'm sure everyone would appreciate it if you helped in this regard.

Brian, I just sent you an e-mail regarding a question about how certain information in the database is stored, and I included 9 suggestions for improving the XHTML/CSS code for the website. You can find it easily by looking for an email subject-line that begins with "Website Code" or an e-mail address that starts with "megaman". (That's from a video game folks, not my ego.) I would appreciate a response.


shelleymtjoy
ModeratorRRS local affiliate
shelleymtjoy's picture
Posts: 1330
Joined: 2006-12-26
User is offlineOffline
Gizmo, as a user I offer my

Gizmo, as a user I offer my extreme thanks for all the work you have done.  clapping I have no MySQL experience which is why I haven't voiced an offer in the past. 

 Although I do have Linux experience.  [MS - Telecommunications with concentrations in Systems Engineering and Network Security (from the fornesics side).]  I don't really have time to help with this but since there is such a need I can make time and/or take more adderall.Eye-wink  I'm not really sure if anything I know is needed but you all know where to reach me. 

BTW - When things calm down I do need to talk to someone about atheist volunteers.  There is a secular group in the area here that is in the process of forming a non-profit.

 

Beltway Atheists "We're good people, just not god people."


dave805
dave805's picture
Posts: 78
Joined: 2007-12-27
User is offlineOffline
Sapient wrote:

Sapient wrote:
Yup Dave, sorry bout that.

No reason to be sorry about it.. Mehh Things happen.. Have to live with it. But thanks for the reply sapient.

Now the question is.. What exactily was the cause and how do we prevent it from happening in the future? If i can be any help regarding this issue let me know. I have a bit of experiance in the Network admin field.

 

"because of an ongoing problem that has affected the server for months" << What exactily was/is this ongoing problem ? If you don't mind me asking. 


MattShizzle
Superfan
MattShizzle's picture
Posts: 6840
Joined: 2006-03-31
User is offlineOffline
Man that sucks. I couldn't

Man that sucks. I couldn't get on all yesterday morning. Really sucks losing all the site did. Hopefully more can get back.


wavefreak
Theist
wavefreak's picture
Posts: 1826
Joined: 2007-05-10
User is offlineOffline
dave805 wrote: Sapient

dave805 wrote:

Sapient wrote:
Yup Dave, sorry bout that.

No reason to be sorry about it.. Mehh Things happen.. Have to live with it. But thanks for the reply sapient.

Now the question is.. What exactily was the cause and how do we prevent it from happening in the future? If i can be any help regarding this issue let me know. I have a bit of experiance in the Network admin field.

 

"because of an ongoing problem that has affected the server for months" << What exactily was/is this ongoing problem ? If you don't mind me asking.

 

I have to wonder what type of hosting contract you guys have. Why isn't the host doing nightly backups? I know you're going to get a lot of arm chair quarter backing, but if this happened at the company that hosts our customers we would fire their asses. What type of SLA to you guys have? If it is a matter of cost, you lost over 100 hours of work. This has a cost to you, maybe not in upfront dollars. But, hell, even at 5 bucks an hour, you just burned 500 dollars of your valuable time. That's $40 a month  That should buy you a much more reliable host - and you are certainly worth more than $5/hr.


Gizmo
High Level Donor
Gizmo's picture
Posts: 397
Joined: 2007-03-06
User is offlineOffline
wavefreak wrote: dave805

wavefreak wrote:
dave805 wrote:

Sapient wrote:
Yup Dave, sorry bout that.

No reason to be sorry about it.. Mehh Things happen.. Have to live with it. But thanks for the reply sapient.

Now the question is.. What exactily was the cause and how do we prevent it from happening in the future? If i can be any help regarding this issue let me know. I have a bit of experiance in the Network admin field.

 

"because of an ongoing problem that has affected the server for months" << What exactily was/is this ongoing problem ? If you don't mind me asking.

 

I have to wonder what type of hosting contract you guys have. Why isn't the host doing nightly backups? I know you're going to get a lot of arm chair quarter backing, but if this happened at the company that hosts our customers we would fire their asses. What type of SLA to you guys have? If it is a matter of cost, you lost over 100 hours of work. This has a cost to you, maybe not in upfront dollars. But, hell, even at 5 bucks an hour, you just burned 500 dollars of your valuable time. That's $40 a month That should buy you a much more reliable host - and you are certainly worth more than $5/hr.

The hosting is just a dedicated server that we rent.  All they are responsible for is network connectivity and hardware.  Everything beyond that has been generally my responsibility.  Hency why im asking for help.  Between my normal 40+ hour a week job and other things that life demands, I have not been able to dedicate the time that is now required to handle a site of this size and magnitude, hence my request for help (which several have volunteered and we are talking with those now).  We have now taken steps to make sure that at least the backups are running which now it should be nore more than a day should I need to restore something.  However, not knowing what caused the issue in the first place (the other recent issues have been caused by module changes in drupal that indirectly caused the problem, or at least seemed to correlate to that).  This time however there was no noticible changed that should have caused this problem.   


wavefreak
Theist
wavefreak's picture
Posts: 1826
Joined: 2007-05-10
User is offlineOffline
Gizmo - I hope you didn't

Gizmo - I hope you didn't take it wrong. I understand how much work is involved and you deserve kudos for the time you volunteer. I'm just suggesting a more comprehensive hosting solution might be in order going forward. A lot of energy is put into this site. It's a shame to see hard work go up in smoke because of insufficient resources. My company's  host does hourly transaction log backups of every single database. Some information relative to Sarbanes-Oxley is replicated every ten minutes. Granted, this is paid for under the terms of service, but soultions for data protection are out there. Given the growth of RRS, it may be time to kick it up a notch.


aiia
ModeratorSilver Member
aiia's picture
Posts: 1395
Joined: 2006-09-12
User is onlineOnline
Looks like the server

Looks like the server crashed again today

That fact that you cannot prove god’s non-existence is inconsequential because we cannot prove the non-existence of anything.


a seeker
Theist
a seeker's picture
Posts: 47
Joined: 2007-12-31
User is offlineOffline
Seems ironic. This is a

Seems ironic. This is a forum for scientifically minded folk, many of whom obviously know computers rather well and it seems to have a computer problem.

 Whereas the Christian forum i hang out in contains people of... lets be nice and say a more right brain tendancy and is so far ok!

 

Oh come now! you've got to see the funny side! 

 

BTW i LIKE firefox. I'm a convert. Halleighlujah. 


Watcher
ModeratorSpecial AgentBronze Member
Watcher's picture
Posts: 1682
Joined: 2007-07-10
User is offlineOffline
Jesus keeps crashing our

Jesus keeps crashing our server.

That bastard...

Science. It works, bitches.

"I'd take the awe of understanding over the awe of ignorance any day."
Douglas Adams

MattShizzle wrote:
You literally are my least favorite person on this site.


Hambydammit
High Level DonorHigh Level ModeratorMod GodRRS Core MemberWebsite AdminPlatinum Member
Hambydammit's picture
Posts: 5708
Joined: 2006-10-22
User is offlineOffline
As soon as we can convince

As soon as we can convince atheists to donate 10% of their income to us, I guarantee we won't have any more server problems.

 

If superior creatures from space ever visit earth, the first question they will ask, in order to assess the level of our civilization, is: 'Have they discovered evolution yet?' -- Richard Dawkins


a seeker
Theist
a seeker's picture
Posts: 47
Joined: 2007-12-31
User is offlineOffline
LMAO!!!!

LMAO!!!!

 

Well Jim Caviezel who played the lead in the passion of the Christ got struck by lightning TWICE whilst filming.

 

Subtle huh?

 

Re tithing. Bummer. They got that one all tied up. You'd be accused of plagerising.

 

 

 

 

 


aiia
ModeratorSilver Member
aiia's picture
Posts: 1395
Joined: 2006-09-12
User is onlineOnline
tithing is a fancy word for

tithing is a fancy word for panhandling

That fact that you cannot prove god’s non-existence is inconsequential because we cannot prove the non-existence of anything.


a seeker
Theist
a seeker's picture
Posts: 47
Joined: 2007-12-31
User is offlineOffline
Had to look up panhandling.

Had to look up panhandling. Found begging, sponging and spanging.

 

And i'd have to agree. CS lewis wrote interestingly on politics. He wrote that the behaviour a system likes is not neccessarily the behaviour which is good for a system (and vice versa).

I think the church has misunderstood / misinterpreted the tithing laws for its own advantage.

 

PLEASE can i have my theist badge back. I feel dishonest posting without it.!

 


Hambydammit
High Level DonorHigh Level ModeratorMod GodRRS Core MemberWebsite AdminPlatinum Member
Hambydammit's picture
Posts: 5708
Joined: 2006-10-22
User is offlineOffline
Funny thing, actually. 

Funny thing, actually.  When I was in college, I tried to start my own religion.  I worked at a pool hall, and the deal was simple.  Whenever somebody wanted something, they would come to the pool hall and give me ten dollars.  Then they could ask me for whatever they wanted.  Sometimes, if it was something that might have happened anyway, I would let them have it.  Most of the time, I said, "No."

It never caught on the way I expected.  After all, I had a great model to work from.

 

If superior creatures from space ever visit earth, the first question they will ask, in order to assess the level of our civilization, is: 'Have they discovered evolution yet?' -- Richard Dawkins


Sapient
High Level DonorRRS CO-FOUNDERRRS Core MemberWebsite Admin
Sapient's picture
Posts: 4747
Joined: 2006-04-18
User is onlineOnline
It's crashed a few times

It's crashed a few times over the last few weeks.

 The good news is that the problems are about to go away.  We're bout to go up to a new level of server.  I'll be dipping in to my personal savings again and getting tons of help from the the long volunteer work of what I'd like to start referring to as the "Rational Power Developer Squad Geek Nerd Force."  

 

He's not the man he wants you to think he is. Go to The Real John McCain to learn about the double talk express. View these videos about John McCain to learn about the flip-flopping double talker. Help me get rid of a transferable credit with Southwest Airlines and you'll save $60.


dead_again
Special AgentWebsite Admin
dead_again's picture
Posts: 251
Joined: 2007-05-13
User is onlineOnline
After extensive work on the

After extensive work on the database I have gotten page load times down to the 4-6 second range. With any luck our database problems are over with and things (hopefully) will get better. With the new server page load times will most likely decrease even further. Expect great things in the coming weeks/months!

Idiocracy was not a documentary; Can we please keep it that way?


Sapient
High Level DonorRRS CO-FOUNDERRRS Core MemberWebsite Admin
Sapient's picture
Posts: 4747
Joined: 2006-04-18
User is onlineOnline
Hambydammit wrote: As soon

Hambydammit wrote:

As soon as we can convince atheists to donate 10% of their income to us, I guarantee we won't have any more server problems. 

That too.  I was planning on working around the problem and dipping into my personal savings again.  appx $3,000 for a new server coming out of "fairx" mutual fund in a few days.  

He's not the man he wants you to think he is. Go to The Real John McCain to learn about the double talk express. View these videos about John McCain to learn about the flip-flopping double talker. Help me get rid of a transferable credit with Southwest Airlines and you'll save $60.


Cpt_pineapple
agnostic deistTheist
Cpt_pineapple's picture
Posts: 2328
Joined: 2007-04-12
User is onlineOnline
Aren't you guys making

Aren't you guys making money off the many ads I see around now?

 


Sapient
High Level DonorRRS CO-FOUNDERRRS Core MemberWebsite Admin
Sapient's picture
Posts: 4747
Joined: 2006-04-18
User is onlineOnline
Cpt_pineapple

Cpt_pineapple wrote:

Aren't you guys making money off the many ads I see around now?

 

Yup. Until about a month ago we made about $1 a day.  Recently we restricted 200 top bidding christian sites from advertising.  Since then, users click on ads more frequently.  We think things will settle out to about $3 or $4 per day with the new method.

 The server I'll be buying in the next few days will be about $3,000 and we might need another one in a year, and then another in a year after that if we continue to grow at this pace.  

Essentially the ads work out to be about 2% of what the RRS house needs to get by.  It's not very much.  Amazon is about another 2% of what we need to get by.

This has got me thinking, I should make a chart of where the money comes from.  I think many would be surprised.  I top the "investors" list in terms of money invested/lost, unless of course you put Mr Anonymous above me.   

He's not the man he wants you to think he is. Go to The Real John McCain to learn about the double talk express. View these videos about John McCain to learn about the flip-flopping double talker. Help me get rid of a transferable credit with Southwest Airlines and you'll save $60.