I am now going to try to get you to setup a small, tiny, little version of a good deployment that matches the configuration of the site at http://mongrel2.org, with all the examples running. This configuration will give you all the tools you need to make automated and managed deployments, but it is using small scale tools. The idea is that you learn what is involved in a nice, easy-to-manage setup, using simple things first, then you can extrapolate that out into your own setup or something better.
It may seem obvious, but I’ll go over the things you need in order to continue on in this section:
That will get you going at first and, as we go, we’ll do various other setups to get our application working.
Why should you learn programming? The trend is that if you are a system administrator who can’t code, you are on your way out. Eventually, you’ll be in charge of automating systems; not manually managing them, and if you don’t believe me then what do you think all those managed service companies are doing? Alright, so you need to learn to code, but most of the books suck for really learning if you know nothing.
This is why I started my own book: Learn Python The Hard Way, for people who know nothing about programming but need or want to learn. It teaches Python, but it mostly teaches all the things programmers actually learn before they learn programming. When you’re done with my book, you’ll have your “programming brown belt”. That means you can then move onto one of many other free online books and really learn programming, and have a higher chance of actually learning it.
If you can’t code Python then you can probably muddle through this and you may learn something, but learning Python will be important later.
But don’t read “Dive Into Python”. It is a horrible introduction.
When I started working on this little manual, I wanted to get you into setting up a well-managed and automated deployment system. The m2sh program does much of the automation you need, but Mongrel2 also has to talk to quite a few separate little pieces that run as separate processes. Trying to juggle all these processes without a tool to help is a nightmare. You end up writing init scripts and merging them into your boot process and all sorts of crazy antics just so you can run a stupid hello world demo.
What I needed was a “user space process manager”. These are programs that run other programs, but, more importantly, try to keep those other programs running without much human intervention. When you need to deploy a ton of processes that all have to be running, these USPMs are fantastic. They usually read some startup profile describing what needs to start and what they depend on, and then it kicks everything into gear and watches them. If any of the processes crash, they try to restart them. Very simple.
There’s just one catch: all of them suck. There’s daemontools, which barely builds (if at all) and then assumes that daemons don’t fork. Stupid. There’s minit, which bafflingly required dietlibc to even compile and assumed it was going to be the one true init (not user space at all). There’s cinit, which got through a compile, then barfed on its documentation, and the end result is some huge number of weird shell scripts to make it work, and, again, it wants to be the one true init. Finally , runit is some of the worst C code I’ve seen in years and has the same weird design as daemontools.
After trying every single one, I just gave up. Either they didn’t build, were too complex, expected to be the one true init, poorly documented, not maintained, and definitely not going to work for this manual. My only choice was to shave a yak and write my own.
The end result is procer, which lives in tools/procer and does most of what you need in a USPM. It works a lot like daemontools or minit, but is much simpler, with these differences:
Of course, if you have something else you like then, please, use it. Anything that automates process management will be your friend. In this manual, to keep things simple and easily understood, I’ll be using procer to tell you how to setup everything.
I wrote procer mostly for this book, but I also use it for my Mongrel2 deployments. It works for me but you can try other solutions. By default, Mongrel2 will work with either daemontools/runit style, or init.d style launchers. If Mongrel2 runs as a regular user, it assumes that you want runit style (don’t fork, write to stdout/stderr). If you run as root, it assumes you want init.d style like what procer uses (fork, drop priv, chroot, etc.).
You should check out proclaunch as another alternative that is similar to procer, and inspired by procer, but written in Perl with a few more features.
Either way, Mongrel2 is practical, and does generally the right thing with today’s tools. Want to use daemontools? Fine, just run it mongrel2 config.sqlite server_uuid and it’ll work right. Want to put it in init.d or use procer or similar? Fine, run it as root.
Installing procer is very easy. It’s a single little binary and it lives in tools/procer in the Mongrel2 source. Here’s how you’d install it totally from scratch as if you hadn’t even built Mongrel2 yet:
That’s the entire install process, and now procer is in /usr/local/bin so you can use it. In the rest of this chapter you’ll learn how to use procer by just setting up the Mongrel2 demo completely and messing around with it.
We need to plan this deployment to make sure we get the end result correct:
Once you have this setup working, you can then start to make your own deployments and tweak things as you need for your own applications. Remember that the goal is to get you to automate everything as much as possible, so you can go further than this then do it.
We’ll need a place to put all this stuff and run it so that Mongrel2 can chroot there, procer knows where its profiles are, and its all nice and clean. For these instructions, we’re just going to make some directories in your home directory, but feel free to change this up later if you find a better way.
# go home first cd ˜/ # create the deployment dir mkdir deployment cd deployment/ # fill it with the directories we need mkdir run tmp logs static profiles # Note: On some systems zeromq needs access to /proc from # the chroot - on Linux this command should do it (make # sure you mount it at boot time as well): ## mkdir -p proc && sudo mount --bind /proc proc # create the procer profile dirs for each thing cd profiles/ mkdir chat mp3stream handlertest web mongrel2 cd .. # copy the mongrel2.conf sample from the source to here cp ˜/mongrel2/examples/configs/mongrel2.conf mongrel2.conf # setup the mongrel2 database initially m2sh load # see our end results ls
Hopefully, you’re starting to see how you could easily automate this so that you don’t have to do this all the time. I’m just showing you how to “make the sausage” so that you know where everything goes. Future versions of m2sh will most likely create deployment directories like this automatically.
What we’ve done here is the following:
Now we’re ready to get the configuration working. Here’s the thing, though: you should try to alter the configuration yourself. I’ve already given you the file and you are going to have to make the changes to meet the requirements for this deployment directory. Here’s what you have to change in mongrel2.conf to make everything work right:
Once you have that all edited, you should be able to run m2sh load -db config.sqlite -config mongrel2.conf and it’ll just load it up. Try using m2sh servers and m2sh hosts to take a peek.
To test it out at this stage you can just run the config.sqlite that you did with these commands:
That’s enough to make sure it runs, but you’ve got nothing running, so it mostly won’t work at all. Just start up and then kill it right after.
Now we want to make procer start everything for us and keep it running. How procer works is you put a few special files into a directory in profiles. This directory (say chat) is the profile for that app. When you start procer, you point it at the main profiles directory and it tries to run it. It’s dead simple and very easy to automate, so we’ll do it by hand and then you can do some automation later.
Let’s first setup a basic config that gets our skeleton profiles and make sure procer can run everything:
cd profiles/ ls # should see: chat handlertest mongrel2 mp3stream web # make all the restart settings for i in *; do touch $i/restart; done # make all the empty dependencies for i in *; do touch $i/depends; done # setup the pid_files to some sort of default for i in *; do echo $PWD/$i/$i.pid > $i/pid_file; done cat chat/pid_file # get the run script setup to do nothing for i in *; do echo ’#!/bin/sh’ > $i/run; done for i in *; do chmod u+x $i/run; done # check out what we did ls -lR
With all of that, you can then try to run procer to watch it fail but still try to run everything:
sudo procer $PWD $PWD/../run/procer.pid less error.log
This is assuming that you are still in the profiles directory. You should see the file error.log get created and probably some messages printed to the screen. Just ignore any mention of Mongrel2 since that’s probably just cruft from the libm2.a we haven’t removed.
Take a look in the error.log and you’ll see it’s not necessarily errors but information on how things were run. You should see something like this for each profile:
I’ve cleaned this up a bit and, again, ignore that it’s saying “Mongrel2”; that’s just cruft from the library since it was originally designed for Mongrel2. What you can see here is the following:
Next up, let’s get Mongrel2 running inside procer:
cd ˜/deployment # make mongrel2 run as root sudo chown root.root profiles/mongrel2 # tell procer where mongrel2 puts its pid_file # notice the > not >> on this echo "$PWD/run/mongrel2.pid" > profiles/mongrel2/pid_file # make the run script start mongrel2 (notice the >> on this) echo "cd $PWD" >> profiles/mongrel2/run echo "m2sh start -db config.sqlite -host localhost" >> profiles/mongrel2/run # check out the results cat profiles/mongrel2/run #!/bin/sh cd /home/YOU/deployment m2sh start -db config.sqlite -host localhost
Obviously, you don’t have to use a series of echo commands to make these scripts. You can edit them just fine, we’re just doing it this way so that you can follow along easier.
Now, make sure you don’t have any other Mongrel2 processes running, and then start procer again to see if it starts this configuration correctly.
cd ˜/deployment # clear out the error.log for testing rm profiles/error.log # start procer sudo procer $PWD/profiles $PWD/procer.pid # see if procer is running ps ax | grep procer # should see: # 17934 ? Ss 0:00 procer /home/zedshaw/deployment/profiles /home/zedshaw/deployment/procer.pid # see if mongrel2 is running ps ax | grep mongrel2 # should see: # 17944 ? Ssl 0:00 mongrel2 config.sqlite ba0019c0-9140-4f82-80ca-0f4f2e81def7
To watch procer in action, try doing m2sh stop -db config.sqlite -host localhost -murder and then look at profiles/error.log and watch Mongrel2 come right back.
We’ve got a good setup of procer going and it keeps Mongrel2 running, so let’s setup a similar thing for each of our little Python demos that we’ll need. In order to do this, though, we sort of have to “hack in” making them daemonize and create PID files with a little shell script help. Let’s start with the chat demo and, assuming your mongrel2 source is in ˜/projects/mongrel2, you will change profiles/chat/run to be like this:
This little script uses some funky features you might not be familiar with, but which are nice to learn, so let’s take a look:
When you run this manually, you should see something like this:
./profiles/chat/run nohup: redirecting stderr to stdout # you’ll only see the above if you needed nohup ps ax | grep chat # should see: 19305 pts/1 Sl 0:00 python chat.py kill -TERM 19305
After all that, you can then try out procer again to see if it properly runs the chat demo as well as mongrel2:
# run procer to get stuff started sudo procer $PWD/profiles $PWD/run/procer.pid # see if it’s all running ps ax | grep procer # should see: # 19607 ? Ss 0:00 procer /home/zedshaw/deployment/profiles /home/zedshaw/deployment/run/procer.pid ps ax | grep mongrel2 # should see: # 19621 ? Ssl 0:00 mongrel2 config.sqlite ba0019c0-9140-4f82-80ca-0f4f2e81def7 ps ax | grep chat # should see: # 19609 ? Sl 0:00 python chat.py # try killing chat to see if it comes back kill -TERM ‘cat profiles/chat/chat.pid‘ ps ax | grep chat # should see: # 19669 ? Sl 0:00 python chat.py
If you go look at profiles/error.log, you’ll see that procer is also running each of them as the right user, with chat being run as you, but Mongrel2 being run as root so it can chroot/drop privileges properly.
Rather than give you a walk through each of these setups, here’s the run scripts for the remaining files:
#!/bin/sh set -e DEPLOY=/home/YOU/deployment SOURCE=/home/YOU/projects/mongrel2 cd $SOURCE/examples/http_0mq # WARNING: on some systems the nohup doesn’t work, like OSX # try running it without nohup python -u http.py > http.log 2>&1 & echo $! > $DEPLOY/profiles/handlertest/handlertest.pid
#!/bin/sh set -e DEPLOY=/home/YOU/deployment SOURCE=/home/YOU/projects/mongrel2 cd $SOURCE/examples/mp3stream # WARNING: on some systems the nohup doesn’t work, like OSX # try running it without nohup python -u handler.py > mp3stream.log 2>&1 & echo $! > $DEPLOY/profiles/mp3stream/mp3stream.pid
Once everything is running and procer is maintaining it, you just need to see if things work. Here’s some curl commands to try:
There’s some nice subtle features you get from using procer to run your stuff:
The key thing, though, is that you now have the whole application for the mongrel2.org demo up and running, including automated process management, configuration, and managing everything.
The final thing we have to do is get the static content we need to try out the chat demo:
If you get a good response then you should be able to go to http://localhost:6767/chatdemo/ and the chat should work. Notice also that you just killed mongrel2 with m2sh and it came back because of procer. If you do your curl check too fast, you might miss it, so just wait a bit.
You should have been testing the configuration as you went, but the main things to test are:
If you run into problems, make sure that you can run each little piece and that the files you were supposed to make are correct. The best tool to use is diff.
That ends this chapter, and at this point you should know how to setup nearly everything Mongrel2 has to offer right now. You should have a good idea of how procer will work or not for your real deployments, and how it’s used by me for my own deployments.
A major improvement that we may eventually make is automating setup of procer profiles, and just better overall management of the profiles with m2sh. If you feel like hacking on that, just go ahead and try and let us know.
Other than that, automate, automate, automate.
Mongrel2 enforces the correct behavior when you run as root, which is to drop priv and chroot. This makes the server more secure, and it also simplifies your deployments. Since everything you do always runs in a chroot, you now just need to rsync that chroot directory, or put it into a git or hg repo, and you’re set. You’re literally forced to make your deployments portable to different directories and systems.
As of the 1.0 push for Mongrel2, we haven’t done much work on how you deploy all the different languages. They sort of sprung up during development and our plan is to expand that out in the 2.0 version so that deployment is very well documented for all the different languages we support. That means you’ll probably run into some snags and things we didn’t anticipate.
The following are some general points we’ve come up with while deploying our own apps, with more to come as we work on the 2.0 version:
There’s a few additional tips for people who want to use alternative process supervision like daemontools, runit, or init.d setups. No matter what you use, you should probably follow this advice:
As we work on the next phase of Mongrel2 development, this will improve, so watch for news about deployment and real applications.