Chapter 4
Deploying

I am now going to try to get you to setup a small, tiny, little version of a good deployment that matches the configuration of the site at http://mongrel2.org, with all the examples running. This configuration will give you all the tools you need to make automated and managed deployments, but it is using small scale tools. The idea is that you learn what is involved in a nice, easy-to-manage setup, using simple things first, then you can extrapolate that out into your own setup or something better.

4.1 Mongrel2 Deployment Requirements

It may seem obvious, but I’ll go over the things you need in order to continue on in this section:

Mongrel2

I know, hard to believe, but you actually need to have Mongrel2 installed.

m2sh

Again, not sure why, but some folks think they don’t need this. Unless you’ve written your own, you need m2sh.

Python

Some systems (like Debian) don’t install all of Python. Make sure your Python setup is good.

root

You’ll need root access on your box. Either through sudo or some other means.

Basic Python coding

Right now, you should be able to do some basic Python.

That will get you going at first and, as we go, we’ll do various other setups to get our application working.

4.1.1 Introducing procer

When I started working on this little manual, I wanted to get you into setting up a well-managed and automated deployment system. The m2sh program does much of the automation you need, but Mongrel2 also has to talk to quite a few separate little pieces that run as separate processes. Trying to juggle all these processes without a tool to help is a nightmare. You end up writing init scripts and merging them into your boot process and all sorts of crazy antics just so you can run a stupid hello world demo.

What I needed was a “user space process manager”. These are programs that run other programs, but, more importantly, try to keep those other programs running without much human intervention. When you need to deploy a ton of processes that all have to be running, these USPMs are fantastic. They usually read some startup profile describing what needs to start and what they depend on, and then it kicks everything into gear and watches them. If any of the processes crash, they try to restart them. Very simple.

There’s just one catch: all of them suck. There’s daemontools, which barely builds (if at all) and then assumes that daemons don’t fork. Stupid. There’s minit, which bafflingly required dietlibc to even compile and assumed it was going to be the one true init (not user space at all). There’s cinit, which got through a compile, then barfed on its documentation, and the end result is some huge number of weird shell scripts to make it work, and, again, it wants to be the one true init. Finally , runit is some of the worst C code I’ve seen in years and has the same weird design as daemontools.

After trying every single one, I just gave up. Either they didn’t build, were too complex, expected to be the one true init, poorly documented, not maintained, and definitely not going to work for this manual. My only choice was to shave a yak and write my own.

The end result is procer, which lives in tools/procer and does most of what you need in a USPM. It works a lot like daemontools or minit, but is much simpler, with these differences:

1. It is much simpler, with only a single command to start all your stuff and keep it running.
2. It will build anywhere Mongrel2 builds, because it reuses the libm2.a library from the Mongrel2 project.
3. It doesn’t want to be the one true init, or even expect to be running constantly. You can start it and stop it and it will only run what’s not already running.
4. It assumes that programs will always daemonize and create a PID file. This turns out to be way easier to manage than what daemontools does, so I’m sort of baffled why daemontools is how it is.
5. It has dependency management so that you can have processes start only after others have finished.
6. It still uses simple files to configure itself that are in separate directories.
7. It can be run as root and, like Mongrel2, it will drop privileges to the owner of the profile directory before it runs the command. This is incredibly useful because it lets you setup scripts that run as other users without much configuration or fuss.
8. It is dinky, tiny and well written so you can understand it, even though it’s written in C.
9. Best of all, I can use it in this book and you won’t go insane trying to install it or use it like the others.

Of course, if you have something else you like then, please, use it. Anything that automates process management will be your friend. In this manual, to keep things simple and easily understood, I’ll be using procer to tell you how to setup everything.

4.1.2 Installing procer

Installing procer is very easy. It’s a single little binary and it lives in tools/procer in the Mongrel2 source. Here’s how you’d install it totally from scratch as if you hadn’t even built Mongrel2 yet:

cd projects/mongrel2 
make clean all && sudo make install

That’s the entire install process, and now procer is in /usr/local/bin so you can use it. In the rest of this chapter you’ll learn how to use procer by just setting up the Mongrel2 demo completely and messing around with it.

4.2 The Plan

We need to plan this deployment to make sure we get the end result correct:

1. Create a deployment area where everything will live.
2. Create a config.sqlite that will work with the demos in examples.
3. Setup procer to run Mongrel2 and the three demo Python scripts for chat, handlertest, and mp3stream, and have it run the fake backend web.py project so we have something to proxy to.
4. Get all the static file content working.
5. Test out that procer is keeping things running and play with taking things down and up and using m2sh to work with the deployment.

Once you have this setup working, you can then start to make your own deployments and tweak things as you need for your own applications. Remember that the goal is to get you to automate everything as much as possible, so you can go further than this then do it.

4.3 Step 1: The Deployment Area

We’ll need a place to put all this stuff and run it so that Mongrel2 can chroot there, procer knows where its profiles are, and its all nice and clean. For these instructions, we’re just going to make some directories in your home directory, but feel free to change this up later if you find a better way.

# go home first 
cd ˜/ 
 
# create the deployment dir 
mkdir deployment 
cd deployment/ 
 
# fill it with the directories we need 
mkdir run tmp logs static profiles 
 
# Note: On some systems zeromq needs access to /proc from 
# the chroot - on Linux this command should do it (make 
# sure you mount it at boot time as well): 
## mkdir -p proc && sudo mount --bind /proc proc 
 
# create the procer profile dirs for each thing 
cd profiles/ 
mkdir chat mp3stream handlertest web mongrel2 
cd .. 
 
# copy the mongrel2.conf sample from the source to here 
cp ˜/mongrel2/examples/configs/mongrel2.conf mongrel2.conf 
 
# setup the mongrel2 database initially 
m2sh load 
 
# see our end results 
ls

Hopefully, you’re starting to see how you could easily automate this so that you don’t have to do this all the time. I’m just showing you how to “make the sausage” so that you know where everything goes. Future versions of m2sh will most likely create deployment directories like this automatically.

What we’ve done here is the following:

1. Setup a ˜/deployment directory we’ll put everything in.
2. Created run, tmp, logs, and profiles that Mongrel2 and procer need to run.
3. In profiles we started dirs for chat, mp3stream, handlertest, web and mongrel2, that procer will read files out of to get all our gear up and running.
4. Copied the mongrel2.conf example file over to our deployment so we can modify it.
5. Initialized the config.sqlite file we’ll be filling in with our modififed mongrel2.conf.

4.4 Step 2: The mongrel2.org Configuration

Now we’re ready to get the configuration working. Here’s the thing, though: you should try to alter the configuration yourself. I’ve already given you the file and you are going to have to make the changes to meet the requirements for this deployment directory. Here’s what you have to change in mongrel2.conf to make everything work right:

1. Get rid of the test_directory handler, since we won’t need it, and any routes that mention it.
2. Change the base of chat_demo_dir to ’static/chatdemo/’, which we’ll setup at the end.
3. Modify the server chroot so that it’s /home/YOU/deployment/.
4. Use the m2sh uuid command to make some new UUIDs for all the existing ones. This is optional, but probably a good idea to get in the habit now.
5. Change the port for web_app_proxy so it points to 8080 instead of 80.
6. Finally, change any mention of “mongrel2.org” into “localhost” so that you can run it locally.

Once you have that all edited, you should be able to run m2sh load -db config.sqlite -config mongrel2.conf and it’ll just load it up. Try using m2sh servers and m2sh hosts to take a peek.

To test it out at this stage you can just run the config.sqlite that you did with these commands:

m2sh start -db config.sqlite -host localhost 
# hit ˆC to exit out 
m2sh start -db config.sqlite -host localhost -sudo 
less logs/error.log 
m2sh stop -db config.sqlite -host localhost -murder

That’s enough to make sure it runs, but you’ve got nothing running, so it mostly won’t work at all. Just start up and then kill it right after.

4.5 Step 3: Setup procer

Now we want to make procer start everything for us and keep it running. How procer works is you put a few special files into a directory in profiles. This directory (say chat) is the profile for that app. When you start procer, you point it at the main profiles directory and it tries to run it. It’s dead simple and very easy to automate, so we’ll do it by hand and then you can do some automation later.

Let’s first setup a basic config that gets our skeleton profiles and make sure procer can run everything:

cd profiles/ 
ls 
# should see: chat  handlertest  mongrel2  mp3stream  web 
 
# make all the restart settings 
for i in *; do touch $i/restart; done 
 
# make all the empty dependencies 
for i in *; do touch $i/depends; done 
 
# setup the pid_files to some sort of default 
for i in *; do echo $PWD/$i/$i.pid > $i/pid_file; done 
cat chat/pid_file 
 
# get the run script setup to do nothing 
for i in *; do echo ’#!/bin/sh’ > $i/run; done 
for i in *; do chmod u+x $i/run; done 
 
# check out what we did 
ls -lR

With all of that, you can then try to run procer to watch it fail but still try to run everything:

sudo procer $PWD $PWD/../run/procer.pid 
less error.log

This is assuming that you are still in the profiles directory. You should see the file error.log get created and probably some messages printed to the screen. Just ignore any mention of Mongrel2 since that’s probably just cruft from the libm2.a we haven’t removed.

Take a look in the error.log and you’ll see it’s not necessarily errors but information on how things were run. You should see something like this for each profile:

I’ve cleaned this up a bit and, again, ignore that it’s saying “Mongrel2”; that’s just cruft from the library since it was originally designed for Mongrel2. What you can see here is the following:

1. It starts up and says it found 5 profiles.
2. It starts chat, and says there’s no PID file so it’s good to continue.
3. It reports what ACTION it’s running, so you can see the config.
4. It spawns off your run script, drops privilege and says it’s WAITING for your script to exit.
5. After your script runs, it looks for the PID file you gave in pid_file and, if it’s not there, it exits that action.
6. It does this for all of them and, since none of them run right, procer exits.

Next up, let’s get Mongrel2 running inside procer:

cd ˜/deployment 
# make mongrel2 run as root 
sudo chown root.root profiles/mongrel2 
 
# tell procer where mongrel2 puts its pid_file 
# notice the > not >> on this 
echo "$PWD/run/mongrel2.pid" > profiles/mongrel2/pid_file 
 
# make the run script start mongrel2 (notice the >> on this) 
echo "cd $PWD" >> profiles/mongrel2/run 
echo "m2sh start -db config.sqlite -host localhost" >> profiles/mongrel2/run 
 
# check out the results 
cat profiles/mongrel2/run 
#!/bin/sh 
cd /home/YOU/deployment 
m2sh start -db config.sqlite -host localhost

Obviously, you don’t have to use a series of echo commands to make these scripts. You can edit them just fine, we’re just doing it this way so that you can follow along easier.

Now, make sure you don’t have any other Mongrel2 processes running, and then start procer again to see if it starts this configuration correctly.

cd ˜/deployment 
# clear out the error.log for testing 
rm profiles/error.log 
 
# start procer 
sudo procer $PWD/profiles $PWD/procer.pid 
 
# see if procer is running 
ps ax | grep procer 
 
# should see: 
# 17934 ?        Ss     0:00 procer /home/zedshaw/deployment/profiles /home/zedshaw/deployment/procer.pid 
 
# see if mongrel2 is running 
ps ax | grep mongrel2 
 
# should see: 
# 17944 ?        Ssl    0:00 mongrel2 config.sqlite ba0019c0-9140-4f82-80ca-0f4f2e81def7

To watch procer in action, try doing m2sh stop -db config.sqlite -host localhost -murder and then look at profiles/error.log and watch Mongrel2 come right back.

4.5.1 The Python Examples

We’ve got a good setup of procer going and it keeps Mongrel2 running, so let’s setup a similar thing for each of our little Python demos that we’ll need. In order to do this, though, we sort of have to “hack in” making them daemonize and create PID files with a little shell script help. Let’s start with the chat demo and, assuming your mongrel2 source is in ˜/projects/mongrel2, you will change profiles/chat/run to be like this:

#!/bin/sh 
set -e 
 
DEPLOY=/home/YOU/deployment 
SOURCE=/home/YOU/projects/mongrel2 
 
cd $SOURCE/examples/chat 
# WARNING: on some systems the nohup doesn’t work, like OSX 
# try running it without 
nohup python -u chat.py > chat.log 2>&1 & 
echo $! > $DEPLOY/profiles/chat/chat.pid

This little script uses some funky features you might not be familiar with, but which are nice to learn, so let’s take a look:

1. The first trick is set -e, which tells bash to bail if there’s any errors in your script. This is a huge life saver in system scripts.
2. Next, you point some variables at where the deployment and Mongrel2 source live, remembering to not type YOU but your username.
3. After that, you run the chat.py using a program called nohup. This basically daemonizes your script by redirecting output and preventing the program from exiting, and then you background it with &.
4. The final thing we do is echo the magic variable $! (the PID of the last process started in the background) to the chat.pid file in the profile directory.

When you run this manually, you should see something like this:

./profiles/chat/run 
nohup: redirecting stderr to stdout 
# you’ll only see the above if you needed nohup 
 
ps ax | grep chat 
 
# should see: 19305 pts/1    Sl     0:00 python chat.py 
 
kill -TERM 19305

After all that, you can then try out procer again to see if it properly runs the chat demo as well as mongrel2:

# run procer to get stuff started 
sudo procer $PWD/profiles $PWD/run/procer.pid 
 
# see if it’s all running 
ps ax | grep procer 
 
# should see: 
# 19607 ?        Ss     0:00 procer /home/zedshaw/deployment/profiles /home/zedshaw/deployment/run/procer.pid 
 
ps ax | grep mongrel2 
 
# should see: 
# 19621 ?        Ssl    0:00 mongrel2 config.sqlite ba0019c0-9140-4f82-80ca-0f4f2e81def7 
 
ps ax | grep chat 
 
# should see: 
# 19609 ?        Sl     0:00 python chat.py 
 
# try killing chat to see if it comes back 
kill -TERM ‘cat profiles/chat/chat.pid‘ 
 
ps ax | grep chat 
 
# should see: 
# 19669 ?        Sl     0:00 python chat.py

If you go look at profiles/error.log, you’ll see that procer is also running each of them as the right user, with chat being run as you, but Mongrel2 being run as root so it can chroot/drop privileges properly.

Rather than give you a walk through each of these setups, here’s the run scripts for the remaining files:

#!/bin/sh 
set -e 
 
DEPLOY=/home/YOU/deployment 
SOURCE=/home/YOU/projects/mongrel2 
 
cd $SOURCE/examples/http_0mq 
# WARNING: on some systems the nohup doesn’t work, like OSX 
# try running it without 
nohup python -u http.py > http.log 2>&1 & 
echo $! > $DEPLOY/profiles/handlertest/handlertest.pid

#!/bin/sh 
set -e 
 
DEPLOY=/home/YOU/deployment 
SOURCE=/home/YOU/projects/mongrel2 
 
cd $SOURCE/examples/mp3stream 
# WARNING: on some systems the nohup doesn’t work, like OSX 
# try running it without 
nohup python -u handler.py > mp3stream.log 2>&1 & 
echo $! > $DEPLOY/profiles/mp3stream/mp3stream.pid

#!/bin/sh 
set -e 
 
DEPLOY=/home/YOU/deployment 
SOURCE=/home/YOU/projects/mongrel2 
 
cd $SOURCE/examples/chat 
# WARNING: on some systems the nohup doesn’t work, like OSX 
# try running it without 
nohup python -u www.py > www.log 2>&1 & 
echo $! > $DEPLOY/profiles/web/web.pid

4.5.2 Testing The New Setup

Once everything is running and procer is maintaining it, you just need to see if things work. Here’s some curl commands to try:

curl http://localhost:6767/ 
# Hello, World! 
curl http://localhost:6767/handlertest

4.5.3 Nice Features of Procer

There’s some nice subtle features you get from using procer to run your stuff:

Faster Development

A great thing about procer is once you get all of this setup, it cuts down on a lot of your setup time and development time because it will properly restart things for you. This means you can simply make changes to code or configs, and then just kill the process and procer will kick it back over automatically.

Easy Automation

You should start to see how you could automate creating profiles for new processes since the setup is consistent.

profiles/run.log

All your commands will have their output sent to this file so you can see how they might be blowing up in your scripts.

Restart State Maintained

Since procer is just tracking PID files and processes, if you shut it down, it won’t kill the world. When you start it back up, it just starts new stuff or stuff it needs, then goes back to supervising. This means you can change the configs for procer then just kick it over and it’ll do the right thing.

The key thing, though, is that you now have the whole application for the mongrel2.org demo up and running, including automated process management, configuration, and managing everything.

4.6 Step 4: Static Content

The final thing we have to do is get the static content we need to try out the chat demo:

cp -r ˜/projects/mongrel2/examples/chat/static static/chatdemo 
m2sh stop -db config.sqlite -host localhost -murder 
curl -I http://localhost:6767/chatdemo/

If you get a good response then you should be able to go to http://localhost:6767/chatdemo/ and the chat should work. Notice also that you just killed mongrel2 with m2sh and it came back because of procer. If you do your curl check too fast, you might miss it, so just wait a bit.

4.7 Step 5: Testing And Troubleshooting

You should have been testing the configuration as you went, but the main things to test are:

1. The /chatdemo/ works and you can send messages. Try a few different browsers.
2. You can get a simple message from the /handlertest/ and that’s about it.
3. See if you can get the mp3streamer to stream some mp3s. Put a few in its directory, then kill it so procer brings it back. Then, point mplayer at http://localhost:6767/mp3stream and it should work.
4. Check that you can make the proxy go to the web.py app you start in the chat demo’s directory.
5. See if you can stop things and have procer bring them back.
6. Stop procer and then start it again to see if it properly doesn’t step on things.

If you run into problems, make sure that you can run each little piece and that the files you were supposed to make are correct. The best tool to use is diff.

4.8 Further Improvements

That ends this chapter, and at this point you should know how to setup nearly everything Mongrel2 has to offer right now. You should have a good idea of how procer will work or not for your real deployments, and how it’s used by me for my own deployments.

A major improvement that we may eventually make is automating setup of procer profiles, and just better overall management of the profiles with m2sh. If you feel like hacking on that, just go ahead and try and let us know.

Other than that, automate, automate, automate.

4.9 Deployment Tips

Mongrel2 enforces the correct behavior when you run as root, which is to drop priv and chroot. This makes the server more secure, and it also simplifies your deployments. Since everything you do always runs in a chroot, you now just need to rsync that chroot directory, or put it into a git or hg repo, and you’re set. You’re literally forced to make your deployments portable to different directories and systems.

As of the 1.0 push for Mongrel2, we haven’t done much work on how you deploy all the different languages. They sort of sprung up during development and our plan is to expand that out in the 2.0 version so that deployment is very well documented for all the different languages we support. That means you’ll probably run into some snags and things we didn’t anticipate.

The following are some general points we’ve come up with while deploying our own apps, with more to come as we work on the 2.0 version:

1. Don’t run things as root if you can. It’s bad habit that everyone tries to do their sysadmin completely as root, so Mongrel2 is designed to be run very easily as under a regular user account. The only time you really should be running as root is when you do a quick -sudo to m2sh to start mongrel2 up so it can chroot.
2. Use the chroot to keep your deployment simpler. I literally do all my work locally and then just rsync my changes up to my remote staging server. Everything has to live in the chroot anyway, and the chroot enforces that it is completely self-contained.
3. Use Python’s virtualenv or anything similar to get yourself a totally local environment. Too many systems, such as OSX, have very outdated packages and will change versions on you without telling you. The best way to make sure your software keeps working (and works as one cohesive deployment) is to use a virtualenv inside your chroot. It should even work cross-platform if you don’t have compiled packages in there.
4. Create a user for your application and live in there. I don’t have any root access on my stuff. Everything is run as a user named after the website, and is deployed right in the /home/USER directory. I login as that user, manage as that user, and I don’t give them sudo access. For the times when I need to sudo to restart or run Mongrel2, I then use a separate login that I have open (with screen) and do it there. This reduces your risk of hacks, but also just simplifies things. It’s no problem for me to move my configuration over to new machines with this setup, or deploy clusters. I know that as long as there’s the right user on the target, I’m set.
5. Use GNU screen or die.
6. Keep your config.sqlite and the .conf file in your chroot, and keep your content and everything else under that. This makes sure that the config isn’t accessible outside your content directories. Mongrel2 helps you get this right by not allowing certain Dir configurations that would expose your chroot to the world.

There’s a few additional tips for people who want to use alternative process supervision like daemontools, runit, or init.d setups. No matter what you use, you should probably follow this advice:

1. Whatever you use for process management, make sure it can run stuff as not root and can do chroot for you. If you’re running your Mongrel2 as root, you’re doing it wrong. Actually, if you’re running any services as root that don’t absolutely need to be, you’re doing it wrong.
2. Mongrel2 is happy to run as a regular user, and assumes that if you do not run as root, then you probably want to run under daemontools or similar. It won’t chroot or drop priv and logs to stdout/stderr.
3. If you need to bind to port 80 but run under daemontools as a regular user, then use privbind to do it. This tool will run any command, like mongrel2 but it does it in a way that lets the executable grab ports below 1024. This restriction on ports is actually really stupid so don’t worry about doing this.
4. Make sure your process monitor is not a single point of failure. Some of them out there will take your whole world down if they crash. Try doing a harsh kill on your process manager and see how it behaves. As much as they like to tell you not to worry about this because they “run forever”, everything has bugs and stupid people tend to kill things they don’t get. If taking one process down nukes your whole server, then that’s a bad design.

As we work on the next phase of Mongrel2 development, this will improve, so watch for news about deployment and real applications.