Pound makes lighty and Mongrel play nice
Posted by David July 03, 2006 @ 05:02 PM
With the rocket rise of Mongrel, we’ve seen a growing number of folks jump ship from lighttpd to Apache 2.2 because of mod_proxy_balancer. It’s great to see that Mongrel is putting Apache back on the map as a premiere Rails web server, but unless you desire Apache for other reasons, you certainly don’t have to jump ship.
The trouble with lighttpd is the state of its mod_proxy implementation, which leaves a lot to be desired when used as a balancer between multiple Mongrel backends. But because the whole Rails deployment stack is going straight HTTP, it’s surprisingly easy to rectify. All you need is to add a more capable load balancer to the mix and you’re golden.
One such balancer that has seen a lot of play lately is Pound (OS X install notes). It’s light, fast, and proven on mega sites. So here’s what you do if want to stay with lighttpd and still use Mongrel:
- Setup lighttpd on port 80 with mod_proxy to point at one back-end server (see the Mongrel lighty docs, but just only use one backend instead of four).
- Setup Pound on a high port, like 7999, and make it point to any number of Mongrel processes (see the Mongrel Pound docs).
- Start n number of Mongrels, from say port 8000 through 8002, using either mongrel_cluster or the soon-to-be-committed Mongrel-compatible script/process/spawner
And bingo, you should now have a production-ready stack ready to take on the world. This is a pretty good outline of how we at 37signals intend to use Mongrel in production shortly.
You can also take pleasure in the fact that Jan Knesche is busy at work making the Pound crutch unnecessary. Over the Summer, he has promised to elevate mod_proxy to be on par with the competition, and this three-way stack should again become a two-way one.
Thanks for saving me from FCGI!
Wow, the pun with Mongrel and Pound is just too good to coincidental. It looks like Pound has been around for a while though, so it must be. What’s up with that?
Isn’t it time someone created a packaged solution containing a quick install of Mongrel and Lighty with simple balancing set up? It could be called “Dog pound”...
Not sure I understand this entirely. lighty uses mod_proxy to send requests to pound, and then pound load balances to mongrel? I thought the whole problem with lighty is that mod_proxy was broken in the first place. Is it only broken when load balancing?
has anyone set up mongrel behind a hardware load balancer? the network I inherited has some sort of ServerIron WebMasterDeluxe 5000 load balancer parked in front of it, which is currently just bouncing http traffic between two apache installs.
I’d like to dump apache and do straight up ServerIron to Mongrel, but haven’t seen that anyone has yet tried this.
Forgive my ignorance, but if Pound is working, why do you need Lighttpd at all? What is its role in the three-way stack? Serve static files?
balanced diet: Yes, two companies I know just put a LB in front of a pack of mongrels.
CJK: You can also go this route with just pound+mongrel. Isn’t HTTP great.
Pat: The problem is that the lighttpd mod_proxy doesn’t recover from backends that go down.
Pawel: Yes, we get great joy in serendipitous puns like that.
I think I need a picture with boxes and lines. :-)
Pat, yes, it’s only broken for error handling and load balancing. Pound handles those parts, so lighty can do really stupid simple proxying.
Carl-Johan, yes, lighty is there for speed of static files. But definitely consider it an optimization. You don’t need lighty, but for sites doing hundreds of thousands or millions of requests per day, it’ll be a very good idea to use lighty for static files.
Randy perhaps this will help, its no map though.
HTTP_REQUEST -> LIGHTTPD LIGHTTPD -> POUND (in this case I think it’s on the same server) POUND
load balanced with failover> MONGREL_CLUSTER (also running on the same server?)I’ve been doing a lot of research on clustering and this sounds like a very simple single server solution thats easily expandable by just pointing pound to other servers later on.
All thats missing is a little memcached. :)
@randy: check this out: http://blog.innerewut.de/articles/2006/04/21/scaling-rails-with-apache-2-2-mod_proxy_balancer-and-mongrel
it’s got loads of boxes and lines ;)
i like the current appoach a lot! maily because of the UN*X style of ‘loosly coupledness’...
yet i do see some issues: – manjor configuration clutter, you’ll have to keep a port administration of some sort. – shared hosts, have to keep an even larger port administration
i understand that this is due to the fact that the load balancing is not bound to a single system.
isn’t it possible to have some sort of socket (in stead of port) based connection between a load balancer and the mongrel process. or am i right now just reinventing some sort of fastCGI?
or could a solution be found in a mod_rails, that like mod_php or mod_perl just needs one configuration file to manage (starting, stopping, respawing, etc.) a bunch of rails apps. (maybe im proposing something very stupid, if so please let me know why)
note: i understand that capistrano, mongrel_cluster, and the “soon-to-be-committed Mongrel-compatible script/process/spawner” also would simplify the task of administering a bit.
the rubyonrails manuals have been offline for several days now and I really need them! I’m not sure who to email to get it fixed :-(
@Jonathan Hoyt:
what do you mean by the ‘manuals’?
thisone works for me: http://api.rubyonrails.com/ (you should be anble to generate then yr self from the rails code)
and here you find commentable api docs for rails: http://rails.outertrack.com/ (a good backup if you depend on online documentation)
_cies.
What Manuals? http://api.rubyonrails.org/ ?
Works from here.
nice idea, except i cant seem to get the remote_ip – all requests are logged as originating from localhost – accessing request.env gives the same.
i did some digging but no luck; i believe whats happening is that lighttpd mod_proxy AND pound are adding an x-forwarded-for header and that mongrel isnt able to deal with the ‘double’ (i wouldnt really expect it to)
any ideas? for some daft reason i cant post to the mongrel list
An alternative method to avoid the double-proxy is the use Pound’s support for regex-based passing of requests to different backends.
This way, you can redirect requests for /images, /javascripts, etc, to lighty, and the rest to the mongrel backends.
My issue is now when I use :only_path => false in url_for and link_to, pound’s port is used instead of 80. :(
Any tips?
That’s a bummer about the port. That’s going to be an issue for me too. The only other thing I can think of is to pass an environment variable to Rails with the port.
not sure what you have against using apache, but having another link in your processing chain sounds like your are asking for trouble.
pound is just another app to maintain, and make sure it is up and running.
disclaimer: I’m a apache developer.
I don’t know where else to comment but what’s up with all the docs being down. I can’t get to any of the manuals…
Ian: I have nothing against Apache. It’s a great web server. I’m using both Apache and lighttpd at 37signals. For running many applications, I tend to prefer to give each its own lighttpd process.
When talking about Rails and Load Balancing, no one seems to mention sandard LVS based solutions such as http://www.ultramonkey.org/
Is there someone with experience of both Mongrel and LVS that can compare the two approaches?
@Scott: I’m having the same problem with request.remote_ip. When Pound is running with Mongrel, remote_ip gets set to 127.0.0.1. I tried commenting out the line in the Pound source code (http.c) that adds the X-Forwarded-For header, but that didn’t change anything. Unfortunately, this is a blocker for us.
request.request.uri is also different. It no longer contains the query parameters (as was the case under lightpd/dispatch.fcgi).
With regards to my last comment: the request.request_uri difference wasn’t caused by Pound – it was Mongrel. When running lighttpd/dispatch.fcgi, request.request_uri contains the query parameters. When running lighttpd/mongrel, request.request_uri does not contain the query parameters. Not sure which behavior is correct, but when looking at migrating, it is important to know that there is a difference.
On my very small setup I’ve decided to put Pound on port 80 and let it handle balancing and virtual hosting for my Rails apps with Mongrel and then send everything else to Apache(running on a higher port) for my legacy sites. Works very well.
I couldn’t find any way around the request.remote_ip = 127.0.0.1 problem. The issue is that Pound blindly overrides the X-Forwarded-For header already added by Lighttpd.
The only solution I found was to use Pen instead, which lets you not add the X-Forwarded-For header. In addition the source for pen does have a check to see if the X-Forwarded-For header is already present, and if so, not add another one, but this wasn’t working for me.
What did work was to simply not include the -H command line option.
Alternativly, I’ve been running Pound in front of Lighttpd and it’s working great.
@Tyler: This setup solves the request.remote_ip issue.
The write-up also includes a fix for loggin the actual remote IP of requests by the servers behind Pound.
http://blog.tupleshop.com/articles/2006/07/08/deploying-rails-with-pound-in-front-of-mongrel-lighttpd-and-apache
Rob
I also found the remote_ip issue to be a deal-breaker with Pound. Thanks to Lars’ suggestion of Pen as an alternative, all is working well again.
I’ve been testing a few different configurations, Lighttpd and FCGI, Lighttpd proxying to a few mongrel processes, and Pound proxying to lighttpd for static and mongrel for rails stuff. one thing i noticed when benchmarking was that i was getting 75% failed requests when using mongrel when the load was high. with lighttpd and fcgi, i got 9% failed requests. Anyone else notice behaviour like this? Is mongrel really production-ready? Also lighttpd+fcgi was on the order of 10x faster than lighttpd proxying for mongrel. This was being tested on a Fedora Core 5 box, latest versions of everything as of last friday.
Jordan
@Rob: Thanks for the write up! Initial tests show that it works great.
I used pound on a webserver cluster with ca. 200 hits/second. Above that pound started to behave erratically as it hit 100% CPU load on a 1 GHz machine. As such I was very satisfied by pound and would recommend it to anyone with a larger machine or lesser hits. I found that by using the loadbalancer of the Linux HA Project (http://www.linux-ha.org/) you get more functionality (you may balance every tcp protocol) for less CPU seconds. 200 hits/second HTTP and many many MySQL connections resulted in a mere 1% CPU load.