Server Crash
Offline Server crashed again. Unfortunately because of an ongoing problem that has affected the server for months, the backups stopped running. This made it to where I had to restore it far further than normal. All I can really say is I apologize for this.
I have informed Brian that I need to distance myself from administrating the site. When I originally took over it was because the previous guy had to leave the country and gave rather short notice on needing to find another solution. I picked up where he left off, and then moved it to a server that was hosted somewhere where I have had good luck with customer service. Then about 6 months ago I got a new job and moved out of my parents house to my own place. This caused not only a lot of stress but also a lack of the time I used to have to dedicate to working on the site. Also, originally I had a lot more disposable income (was paying about 200 in rent to my parents, now paying 600 + utiltiies which comes out to about another 200-300 depending). Tho I got a raise in the new job, it doesn't quite offset that difference at all. Luckily many people here have stepped up and donating some money to help out on the server costs (which I appreciate much).
That all being said, I feel that I can't dedicate the time and love that the server needs. Brian, Kelly and Rook (among many others) work very hard and for Brian, Rook and Kelly this is pretty much what they do for work (certainly would be completely if that was possible at this point, and its getting there). However I don't have that same kind of dedication nor time.
I felt it necessarily to explain some of this so others can hopefully see a little bit where I am coming from. This all being said, again I apoligize for todays issues. Also, Brian and I will be looking for people to take over some of the backend tasks (like checking to make sure the backups run, optimizing mysql tables and what not). If you have experience in mysql and/or linux (especially command line stuff) please let Brian or I know. As always you can find me on skype as gizmoiscariot or you can email me at .
- Login to post comments
sorry to hear that happened. I know how daunting these things can be, seeing that I am a network administrator. I added you to my contact list on skype so feel free to IM me or, if your lucky to catch me, call me. I have a lot of Linux CLI (command line interface) experience and some MySQL experience. I also know python for scripting and some shell scripting.
Your god's silence speaks loud and clear
I have 14 year experiance with many different flavors of linux/unix if whoamever needs help. Feel free to message me.
Folks, feel free to use google cache to try and find works you wrote in the last few days but are now gone. Kelly lost about 50 hours of editing and site work, I lost about 75-100. I hope to make a vid on the issue.
Don't let it get you down folks. We remain determined and are consistently looking to figure out the problem and prepare the right solution.
Please become a Patron of Brian Sapient
Support our activism efforts by making your Amazon purchases via this link.
Are you hosting or co-locating?
Is there an email list for tech people? A collective of sorts? I'm a sysadmin.
I AM is CRYING TEARS
So is this why all my posts are gone and i had to resign up on the forums?? I thought at first i was banned..
Yup Dave, sorry bout that.
Please become a Patron of Brian Sapient
Support our activism efforts by making your Amazon purchases via this link.
Yes, early this morning the server crashed. Sapient said in the Stickam chatroom last night (read: "this morning"
he was going to get a little bit of sleep but I noticed the server crash thirty minutes later, causing him to be woken up to work on the problem. After a while I think they (Brian and Kelly) decided to wait for Gizmo. But yes, the information that didn't appear in the latest backup was lost. As Brian said, you can use Google's cache to help put some of that information back on the website, and I'm sure everyone would appreciate it if you helped in this regard.
Brian, I just sent you an e-mail regarding a question about how certain information in the database is stored, and I included 9 suggestions for improving the XHTML/CSS code for the website. You can find it easily by looking for an email subject-line that begins with "Website Code" or an e-mail address that starts with "megaman". (That's from a video game folks, not my ego.) I would appreciate a response.
Stultior stulto fuisti, qui tabellis crederes!
Gizmo, as a user I offer my extreme thanks for all the work you have done.
I have no MySQL experience which is why I haven't voiced an offer in the past.
Although I do have Linux experience. [MS - Telecommunications with concentrations in Systems Engineering and Network Security (from the fornesics side).] I don't really have time to help with this but since there is such a need I can make time and/or take more adderall.
I'm not really sure if anything I know is needed but you all know where to reach me.
BTW - When things calm down I do need to talk to someone about atheist volunteers. There is a secular group in the area here that is in the process of forming a non-profit.
No reason to be sorry about it.. Mehh Things happen.. Have to live with it. But thanks for the reply sapient.
Now the question is.. What exactily was the cause and how do we prevent it from happening in the future? If i can be any help regarding this issue let me know. I have a bit of experiance in the Network admin field.
"because of an ongoing problem that has affected the server for months" << What exactily was/is this ongoing problem ? If you don't mind me asking.
Man that sucks. I couldn't get on all yesterday morning. Really sucks losing all the site did. Hopefully more can get back.
Matt Shizzle has been banned from the Rational Response Squad website. This event shall provide an atmosphere more conducive to social growth. - Majority of the mod team
I have to wonder what type of hosting contract you guys have. Why isn't the host doing nightly backups? I know you're going to get a lot of arm chair quarter backing, but if this happened at the company that hosts our customers we would fire their asses. What type of SLA to you guys have? If it is a matter of cost, you lost over 100 hours of work. This has a cost to you, maybe not in upfront dollars. But, hell, even at 5 bucks an hour, you just burned 500 dollars of your valuable time. That's $40 a month That should buy you a much more reliable host - and you are certainly worth more than $5/hr.
My Artwork
The hosting is just a dedicated server that we rent. All they are responsible for is network connectivity and hardware. Everything beyond that has been generally my responsibility. Hency why im asking for help. Between my normal 40+ hour a week job and other things that life demands, I have not been able to dedicate the time that is now required to handle a site of this size and magnitude, hence my request for help (which several have volunteered and we are talking with those now). We have now taken steps to make sure that at least the backups are running which now it should be nore more than a day should I need to restore something. However, not knowing what caused the issue in the first place (the other recent issues have been caused by module changes in drupal that indirectly caused the problem, or at least seemed to correlate to that). This time however there was no noticible changed that should have caused this problem.
Go here if you would like to donate to help pay for server costs
Gizmo - I hope you didn't take it wrong. I understand how much work is involved and you deserve kudos for the time you volunteer. I'm just suggesting a more comprehensive hosting solution might be in order going forward. A lot of energy is put into this site. It's a shame to see hard work go up in smoke because of insufficient resources. My company's host does hourly transaction log backups of every single database. Some information relative to Sarbanes-Oxley is replicated every ten minutes. Granted, this is paid for under the terms of service, but soultions for data protection are out there. Given the growth of RRS, it may be time to kick it up a notch.
My Artwork
Looks like the server crashed again today
Seems ironic. This is a forum for scientifically minded folk, many of whom obviously know computers rather well and it seems to have a computer problem.
Whereas the Christian forum i hang out in contains people of... lets be nice and say a more right brain tendancy and is so far ok!
Oh come now! you've got to see the funny side!
BTW i LIKE firefox. I'm a convert. Halleighlujah.
Jesus keeps crashing our server.
That bastard...
As soon as we can convince atheists to donate 10% of their income to us, I guarantee we won't have any more server problems.
Atheism isn't a lot like religion at all. Unless by "religion" you mean "not religion". --Ciarin
http://hambydammit.wordpress.com/
Books about atheism
LMAO!!!!
Well Jim Caviezel who played the lead in the passion of the Christ got struck by lightning TWICE whilst filming.
Subtle huh?
Re tithing. Bummer. They got that one all tied up. You'd be accused of plagerising.
tithing is a fancy word for panhandling
Had to look up panhandling. Found begging, sponging and spanging.
And i'd have to agree. CS lewis wrote interestingly on politics. He wrote that the behaviour a system likes is not neccessarily the behaviour which is good for a system (and vice versa).
I think the church has misunderstood / misinterpreted the tithing laws for its own advantage.
PLEASE can i have my theist badge back. I feel dishonest posting without it.!
Funny thing, actually. When I was in college, I tried to start my own religion. I worked at a pool hall, and the deal was simple. Whenever somebody wanted something, they would come to the pool hall and give me ten dollars. Then they could ask me for whatever they wanted. Sometimes, if it was something that might have happened anyway, I would let them have it. Most of the time, I said, "No."
It never caught on the way I expected. After all, I had a great model to work from.
Atheism isn't a lot like religion at all. Unless by "religion" you mean "not religion". --Ciarin
http://hambydammit.wordpress.com/
Books about atheism
It's crashed a few times over the last few weeks.
The good news is that the problems are about to go away. We're bout to go up to a new level of server. I'll be dipping in to my personal savings again and getting tons of help from the the long volunteer work of what I'd like to start referring to as the "Rational Power Developer Squad Geek Nerd Force."
Please become a Patron of Brian Sapient
Support our activism efforts by making your Amazon purchases via this link.
After extensive work on the database I have gotten page load times down to the 4-6 second range. With any luck our database problems are over with and things (hopefully) will get better. With the new server page load times will most likely decrease even further. Expect great things in the coming weeks/months!
Your god's silence speaks loud and clear
That too. I was planning on working around the problem and dipping into my personal savings again. appx $3,000 for a new server coming out of "fairx" mutual fund in a few days.
Please become a Patron of Brian Sapient
Support our activism efforts by making your Amazon purchases via this link.
Aren't you guys making money off the many ads I see around now?
Yup. Until about a month ago we made about $1 a day. Recently we restricted 200 top bidding christian sites from advertising. Since then, users click on ads more frequently. We think things will settle out to about $3 or $4 per day with the new method.
The server I'll be buying in the next few days will be about $3,000 and we might need another one in a year, and then another in a year after that if we continue to grow at this pace.
Essentially the ads work out to be about 2% of what the RRS house needs to get by. It's not very much. Amazon is about another 2% of what we need to get by.
This has got me thinking, I should make a chart of where the money comes from. I think many would be surprised. I top the "investors" list in terms of money invested/lost, unless of course you put Mr Anonymous above me.
Please become a Patron of Brian Sapient
Support our activism efforts by making your Amazon purchases via this link.
I guess I don't understand why you buy your own hardware. Why not pay for a commercial web host? Or try something like Amazon'd cloud computing. I pay $10 a month for a basic web site with ecommerece capability and MySQL or MS SQL Server for a back end. What are the requirements of this site that force you into fronting your own hardware?
My Artwork
This was the solution everyone on the dev team came up with after several days of going through different options. Though we had different opinions on how we should implement it, we all agreed that a new server was in order.
If we were to go with a canned service like what you suggested wavefreak we would be limited on a number of fronts such as MySQL utilization limitations (that is what we need a lot of), inability to have SSH (console) access to the server, and the inability to do things like write our own custom backup scripts.
I thought the same thing you did wavefreak until I saw what was happening under the hood. Even though the up front investment is heavy, the long term (monthly) cost will be much lower now.
This server is not just for rationalresponders.com, it is for a multitude of sites. rationalresponders.com happens to be the biggest of them, but sites like margaretdowney.com, freethinkingteens.com, greydonsquare.com, wotmwatchdog.org and asktheatheist.com, to name a few, have to share space with rationalresponders.com.
Also, the dev team thought buying a server to be the best decision because Brian shared some of his future plans with us that would require more capabilities than a canned hosting service could offer.
Luckily for the RRS several volunteers have stepped up (The "Rational Power Developer Squad Geek Nerd Force" as Sapient calls us) to tackle recent problems and to start helping develop a better site with many more features for everyone. Fixing the low level stuff like the database issues is just the beginning of an endless battle. Bear with us as we strive to make the site great.
Your god's silence speaks loud and clear
In a tech collective I am in we use Server Beach. This gives us full control of a real machine, but it's not purchase of the hardware, it's a lease, so we can migrate to a new machine when need be, and continue paying on a monthly basis (though its cheaper to pay a year in advance).
Take a look, it may or may not be a good option for you.
That is what we are currently doing. Moving to a server we own allows for more flexibility than renting one. Imagine it the same way as renting a house. You can leave it at the end of your lease no problems, but while you are there you can't make any major changes to the property without consent and if you do, you must leave those changes with the property. Buying a server is like owning a mobile home. You can move it to whatever "trailer park" you want to at a moments notice.
Your god's silence speaks loud and clear
That may be true josh but Buying vs renting has its own pitfalls.
1. Costs of a server/hardware/setup/maintance/etc (can be much more then you would ever pay for a rented server)
2. Bandwidth.. You realisticaly need a dedicated T1 line or better for a server. (Specialy when you have as much high bandwidth entinsive content as this site. Video's etc.)
Then again Buying also has its advantages..
1. Ability to put whatever content you want on the server (Within the law
)
2. Phisical access to the server at a moments notice incase something happens.. etc
In my opinion (Just my opinion) the costs of buying and hosting your own server are prohibitive atm. Seems to me it would be better to go with a hosting company. Then again thats just my uninformed opinion. I do not know all the complex issues you guys are dealing with.
Hit the nail on the head there. And we're too busy working on it to explain all of it. You're advice was good but not factoring in some specifics we need.
briefly...
1. we'll be co-locating the server (about $100 per month)
2. Our Super Rational Power Squad Nerd Geek Force is highly capable of setting up and maintaining the server on a volunteer basis.
Feb 5th (as late as March 5th if I miss a buying deadline and/or issues arise) look for our site to have a top 25,000 site 3 month average, to have about 3-4 seconds a page load, and to be a top 15,000 site before the end of the year. (secretly some people have a much higher goal)
Look for www.MargaretDowney.com and www.AskTheAtheist.com to see huge growth as well. (both sites should be top 150,000 sites by years end and that's supposedly conservative)
www.asktheatheist.com jumped 6,000,000 in ranking in 3 months!
Please become a Patron of Brian Sapient
Support our activism efforts by making your Amazon purchases via this link.
Thank you Sapient for responding
Could you adress 2 small issues ..
1. It seems their is a Mysql error currently (Is this just me or everyone)
Warning: Cannot modify header information - headers already sent by (output started at /home/rrcom/public_html/includes/database.mysql.inc:120) in /home/rrcom/public_html/includes/bootstrap.inc on line 550
Secondly.. Are my posts being moderated.. (It would seem a couple recent posts have disapered) Or maybe it is just the server having issues.