How’d they do that? Breaking the server.
Posted December 4th, 2008 by andrewSome of you may have noticed the hour that Fluther was down this afternoon. I thought I’d give you a little insight into what we’ve been I’ve been up to while Bacio fiddles the bits on the server.
Not wanting anyone to accuse us of not running with the cool kids, I decided to implement a STOMP messaging queue system* for Fluther in November. Now, whenever a question gets asked we can offload the matching algorithm to a different machine. The result? You get a much snappier Fluther.
I’ve also rewritten our twisted-based notify server to use the message queue as well, so watch out for better emails when there’s activity on your questions.
Here’s how this afternoon went down. Many of these steps are run automatically using our custom capistrano scripts:
- Post an MOTD that the site will go down.
- Remove one of our webservers from behind perlbal, our load balancer. We use this to test our production environment later.
- Take the site down. (Disable the fluther apache site, enable the maintenance apache site)
- Using capistrano, update all of our servers with the latest messaging code (which we call lemur)
- Update all the servers with the latest fluther code
- Migrate the database (we’ve just switched over from using a home-brewed capistrano migration system to south, it’s nifty!)
- Update the new notifier codebase (we call it hydra)
- Restart all of our processes: apache, lighttpd, memcached, lemur, and hydra
- Check the logs to make sure that everything is running.
- Use our web server that we’ve taken out of the load balancer to run a few spot tests.
- Reenable the website
- Add our web server back into the load balancing pool.
In practice, there were a few snags (mostly dependency issues with the new codebase), but overall everything is running very smoothly now.
* The messaging system is a customized version of Brian Rosner’s engine library and resembles starling — it’s pluggable for different queue types, provides a nice interface for deferring functions, and has sample clients that consume messages off the queue. Let me know if this interests you!
Explore posts in the same categories: Fluther, How'd they do that?, News, Technical
December 4th, 2008 at 6:34 pm
I just love it when you speak computer-ese. Thanks for all you’re doing to make the site better!
December 4th, 2008 at 7:02 pm
Okay…what?
Glad you and Bacio are on the job. :^)
December 4th, 2008 at 7:12 pm
Ok, I understood nothing after the first sentence, but I want to thank you guys for doing so much for keeping things running smoothly. Thanks for all you do to give us such a great place to be!!
December 4th, 2008 at 7:22 pm
Apparently Bacio has earned his keep, at last. (As have you, Andrew).
I thought this was going to be about tennis. Pity.
December 5th, 2008 at 7:32 am
This meant nothing to me, but I’m glad you’re improving the site! I love it here.
December 5th, 2008 at 8:19 am
Huh?
I have no idea what you are talking about…..? Did I StumbleUpon a foreign language site?
December 5th, 2008 at 4:11 pm
This will almost have to be fluthered.
December 5th, 2008 at 4:42 pm
You have Bac-Os running the server? Not even *real* bacon?
December 5th, 2008 at 6:24 pm
Bacio keeps kosher.
December 6th, 2008 at 1:18 am
Wow;
I knew you had to be smart to make these things work, but I didn’t realize that you had your very own language. I am thankful for all you brainy folks that keep my Internet safe for democracy! You are the super hero’s of the 21st Century!
December 6th, 2008 at 11:30 am
–eyes rolling to back of head–
That’s why they pay you the big bucks! Keep up the good work Andrew and Ben!
December 6th, 2008 at 7:00 pm
I have no idea what you just said but thank you so much for Fluther!!!
December 8th, 2008 at 12:16 pm
Very cool. One of the big problems we had with using a messaging queue was getting performance out of atomic gets. Is this a 1:1 pass off?
December 10th, 2008 at 1:51 pm
Interestingly, while reading these comments, I began searching for the “Great Answer” button.
Thanks a bunch, guys!
March 3rd, 2009 at 6:55 am
Hello webmaster
I would like to share with you a link to your site
write me here preonrelt@mail.ru