The Fluther Blog

Back to Fluther

What is Fluther?

Fluther is a free Q&A collective that specializes in getting
fast answers from the right people. Check it out!

How’d they do that? Breaking the server.

5:46 pm

Some of you may have noticed the hour that Fluther was down this afternoon. I thought I’d give you a little insight into what we’ve been I’ve been up to while Bacio fiddles the bits on the server.

Not wanting anyone to accuse us of not running with the cool kids, I decided to implement a STOMP messaging queue system* for Fluther in November. Now, whenever a question gets asked we can offload the matching algorithm to a different machine. The result? You get a much snappier Fluther.

I’ve also rewritten our twisted-based notify server to use the message queue as well, so watch out for better emails when there’s activity on your questions.

Here’s how this afternoon went down. Many of these steps are run automatically using our custom capistrano scripts:

  • Post an MOTD that the site will go down.
  • Remove one of our webservers from behind perlbal, our load balancer. We use this to test our production environment later.
  • Take the site down. (Disable the fluther apache site, enable the maintenance apache site)
  • Using capistrano, update all of our servers with the latest messaging code (which we call lemur)
  • Update all the servers with the latest fluther code
  • Migrate the database (we’ve just switched over from using a home-brewed capistrano migration system to south, it’s nifty!)
  • Update the new notifier codebase (we call it hydra)
  • Restart all of our processes: apache, lighttpd, memcached, lemur, and hydra
  • Check the logs to make sure that everything is running.
  • Use our web server that we’ve taken out of the load balancer to run a few spot tests.
  • Reenable the website
  • Add our web server back into the load balancing pool.

In practice, there were a few snags (mostly dependency issues with the new codebase), but overall everything is running very smoothly now.

 

* The messaging system is a customized version of Brian Rosner‘s engine library and resembles starling — it’s pluggable for different queue types, provides a nice interface for deferring functions, and has sample clients that consume messages off the queue. Let me know if this interests you!