Deployment for Dummies? (Clojure + Luminus + Mongodb + Digitalocean)

This accurately reflects my methods here.

I just threw up a very basic database-backed web application on Digital Ocean (no, you can’t see it, it’s purely an internal thing for a data collection project I have going with a research team).

In getting it fired up, I basically patched together instructions from a bunch of tutorials, horribly screwing it up the first time (the application went down before the first research assistant ever managed to get any data in it).

This is my attempt to record all of the steps I followed on the, thus-far more successful (one of my RAs actually got some data in there! It’s still running 36 hours later!), second iteration of the deployment. It represents the steps I followed on September 6, 2016, and may be obsolete if you’re reading this much later.

This is more of a brain-dump for a very dirty deploy than a sensible tutorial. I openly admit to have barely-skimmed important documentation. I don’t know jack-all about devops. There’s one point where I appeal to magic. and it gets really aggressively punchy toward the end. Needless to say, I don’t guarantee that this is correct, just that it seems to be working for me right now. This is my first time, remember? Interspersed through this article, I’ve cited my sources, so go look there for more information. This is also a kind of rough draft (as of 9–7–16); I may correct it or tweak things as I learn more.

On the other hand, it might still be useful for others who don’t really know anything about deploying stuff. I pieced this information together from chasing down a bunch of random sources and a ton of Google, so it’s about time someone put it all in one place.

Assumptions:

1. You have a working Clojure web application, compiled as a uberjar, that you can successfully run on localhost and which does what you intend it to do, including talking to a local database install.
2. That application is based on the (excellent) Luminus template .
3. You want to use Mongodb as your database backend (and you are using the scaffolding that Luminus gives you for that).
4. You want to deploy on Digital Ocean.
5. You’re willing to pay more than the bare minimum. (I’m running my little app off the $10/month 1gb ram instance, because I have very light uses. Might end up upgrading to the $20/month one. Wouldn’t dare do any JVM thing on the $5/month 512mb ram one.)
6. You want to start from absolutely zero.
7. You don’t want to do anything fancy with webservers — the server you have bundled in your JAR is fine for you (I’m using Httpkit) — you don’t, for example, want to make Ngnix do the real work, or somehow get Apache involved, or some other kind of devops dark magic.
8. You’re not playing around with sensitive data and hence not overly concerned with security, so we won’t be worrying about complexities like setting up keys rather than passwords, using https, firewalls, etc.
9. Corollary of 7 and 8: you don’t need anything fancy. You’re not doing industrial scale stuff, you just need to deliver code over a webserver so that people can interact with it in fairly small quantities.
10. You know the basics of operating a Linux commandline, ssh, etc., and also have a real dev environment at home (like a *nix/Mac with standard terminal utilities, not some Microsoft thing).
11. You aren’t going to need to deal with any DNS stuff. (Me, I’m just running a backend fronted by a custom chrome extension for my research assistants, so I’m just pointing them to the raw IP address www.xxx.yyy.z:3000. If you need a domain name, check here for instructions on Digitalocean.)

With those considerations in mind, here’s how I got from uberjar to app.

1. Digitalocean. Get an account if you don’t have one, and then start a new droplet (Digitaloceanese for “virtual server”), which is just clicking a button from their web interface. Like I said up above, I seem to be going fine with the $10/month 1gb droplet.
2. Set up your droplet (still in the web interface here) with a 64-bit Ubuntu 16.04. Important: use the 64-bit version, not the 32-bit version. There’s documentation floating around telling you that 64-bit won’t do you any good on servers with less than 2gb of ram, but what they don’t tell you (and what I found out to my horror the first time around) is that the actual production version of Mongodb wants a 64-bit OS.
3. The kindly folks at Digitalocean will e-mail you a root password and IP address for your new server. SSH into that (ssh root@your.ip.address) and promptly change the password.
4. Create non-root user, and give that user sudo privs. Let’s suppose you want to call your non-root user kittycat. Then, while still logged in as root:

adduser kittycat
usermod -aG sudo kittycat

Most of the steps thus far came from Digitalocean’s basic setup guide; go look at it for additional security steps you might want to take, and that I’m skipping, because, as noted above, no sensitive data here folks. (and now medium’s editing interface is going to break automatic numbered list formatting, sigh.)

5. SSH in as your newly created user, make sure everything works.

6. Create a directory in your new user’s home directory for the app, and, just for convenience, create another for the environment for that app. Supposing that your app is named meowmeow, for example, we might create meowmeow and meowmeow/env. Absolute paths will then be /home/kitty/meowmeow and /home/kitty/meowmeow/env. We’ll need these later.

7. While we’re here, go ahead and upload the app. SCP is easiest for this: from a commandline at your own computer:

scp meowmeow.jar kittycat@host:/home/kittycat/meowmeow/meowmeow.jar

8. We should probably install the necessary stuff to run the app. Because we’re not mucking around with fancy stuff, this just means installing Java and Mongodb. First, Mongo. This is actually going to be a bit complicated because Mongo. I followed these instructions from Digitalocean, so see them for more details.

9. Mongo wants its own freaking package repository, so first you need the keys for that:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927

(also, medium interface, why the hell are you changing double-dashes to en-dashes in code blocks? STOP IT, YOU’RE NOT MICROSOFT.)

10. Then you need the actual repository:

echo “deb http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.2 multiverse” | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list

11. Then you need APT to want to know the repository exists:

sudo apt-get update

12. Then finally you can actually get Mongo:

sudo apt-get install -y mongodb-org

13. Ok, now this is going to make Java nice and easy. Go get the latest version:

sudo apt-get install default-jre

Bam. Done. There are more instructions from Digitalocean here, like setting all kinds of environment variables, but I seem to be able to get it to work fine without all that.

14. So you’ve got your app, and you’ve gotten the two real dependencies for your app. Maybe you should actually run them? Let’s start with a brittle test:

14a. You’ll probably need to set an environment variable so your jar can find your database:

export DATABASE_URL=”mongodb://127.0.0.1/meowmeow_prod”

(or whatever you want to call it) should do the trick there.

At this point, you should be able to fire up the app by running mongod and then running your jar. Check that now, and make sure nothing goes crashey-crashey. Make sure you can visit the website, or curl in and get something back, or otherwise do whatever it is that your app is supposed to do.

But we’re not done yet, because this is brittle: if the server restarts or something, then your app is going to stop, and then (if you’re me) your research assistants are going to send you e-mails that say stuff like “Professor Gowder, I tried to log into the system, but nothing happened,” and then you’ll miss lunch blowing up and redeploying the entire server and barely make it to the class you have to teach and possibly get an ulcer.

15. So we’re going to Control-C our way out of all that jar jazz, plus kill the mongo server.*

• incidentally, this kindly SO user sussed out a neat way to just drop mongo with a one-liner from bash rather than mucking around with the special mongo shell (repl? whatever it’s supposed to be):

mongo --eval “db.getSiblingDB(‘admin’).shutdownServer()"

Now we’ll use a neat tool called systemd to tell our application to restart when the server restarts.

N.B.: right now, I’m not sure how to tell systemd to start one application before another. You’re gonna want the database to start before the jar. For whatever reason, this seems to happen for me—possibly because I configured mongo before the jar, as described below, possibly because systemd starts what it’s told to start in parallel and the JVM startup time is so horrible that by the time it goes looking for the database it’s already there, possibly because magic. Hell, for all I know it happens in the right order because the name of my jar starts with an r, and systemd just goes in alphabetical order. If that’s the case, please go back and instead of naming the app we’re imagining you to have meowmeow, name it zowzow. Thanks.

If anyone knows why systemd is magically working in the right ordrr, please do comment to let me know.

Also, I imagine that the “after” line in the config files could guarantee this by declaring mongo to be a dependency of the jar, but figuring out what to put in the config file to make this happen would involve reading way more documentation than I’m in a mood for right now, so, sorry. I’ll try to edit at some point down the road when things stop working and I have to stop relying on magic.

16. Let’s tell systemd to start mongo on server restart. Fire up your favorite text editor with sudo, and tell it to edit /etc/systemd/system/mongodb.service, e.g.

sudo vim /etc/systemd/system/mongodb.service

(Yes, that’s right, we’re using vim. I guess you can use nano or something too. Or, if you really feel like being stubborn, you can just copy-paste the below through echo and a pipe.)

Into that file, enter the following:

[Unit]
Description=ITS A DATABASE YO foo bar loren ipsum
After=network.target

[Service]
User=mongodb
ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf

[Install]
WantedBy=multi-user.target

If you want an explanation of what all these lines do, go back to the Digitalocean page on Mongo. Also, there are some more details in yet another Digitalocean tutorial, which I confess to have only skimmed bits of (this might also have an answer to the magic why stuff starts when).

Start it, and then make sure it’s running:

sudo systemctl start mongodb
sudo systemctl status mongodb

If the output of that second command doesn’t have any sad things like “failed” or “inactive” or “kernel panic” or “just throw away your computer now, LUSER,” then you’re probably ok. We’ll enable this in a minute, but for now, let’s turn to the JAR .

17. New file to edit with your favorite editor: /lib/systemd/system/meowmeow.service

Note first how this file is in lib and the other was in etc. It turns out (see random wiki reference from Archland) that systemd can read files from both locations, but that etc takes precedence. I literally just noticed this right now as I type this, when I did the actual deploy I put them in different directories only because I switched to following the tutorial on the Luminus website for the jar. But that might be the answer to why mongodb magically starts before the jar for me (if we interpret the ambiguous term “takes precedence” to mean not just “decides in case of conflicts” but also “goes first” — which it might! Maybe.). So just stick this one in lib and hope?

Anyway. Put this stuff in there:

[Unit]
Description=Foo bar app loren ipsum kitty meow duh
After=network.target 
[Service] 
WorkingDirectory=/home/kittycat/meowmeow
EnvironmentFile=-/home/kittycat/meowmeow/env
Environment=”DATABASE_URL=mongodb://127.0.0.1/meowmeow_prod"
ExecStart=/usr/bin/java -jar /home/kittycat/meowmeow.jar 
User=kittycat
[Install] 
WantedBy=multi-user.target

[Very special update: so here’s an article that delves into the nitty-gritty of systemd dependency and load order handling. Turns out that the way to actually guarantee that the jar loads after the database would be to stick mongodb.service in the “after” line in this file. But also it’s kinda parallel and so maybe JVM startup time is saving the existing code in my actual case. Look how non-lazy I am in the service of you, the reader.]

Note that all of these directories and commands to execute and stuff have to be absolute, not relative, pathnames.

Do the start and status commands from above again, though, obviously, with meowmeow instead of mongodb. Then go see if your website is running again. If it all works, and Russian hackers haven’t already pwned your webapp, then you’re probably ready to:

18. Tell it to start automatically on restart.

sudo systemctl daemon-reload 
sudo systemctl enable mongodb.service
sudo systemctl enable meowmeow.service

I’m not completely sure that the first line there is necessary, but it has 15 upvotes, so, hey. I’m also not sure that the .service extension is necessary on the enable commands (the Luminus tutorial has ’em, the Digitalocean tutorial doesn’t, when I was just typing in whatever the internet gods told me to do to get this running I accordingly put it in for the jar but not for the database… and it seems to have worked).

19. Now the moment of truth. Reboot the server:

sudo shutdown -r now

Wait a few moments. Then see if your app works. If it does, you’re done.*

• Well, done, if you’re not doing anything serious, and don’t need, e.g., security of really any kind at all. If stop here on anything in actual industrial production that has sensitive data or where it matters if evil hackers make you part of a botnet or basically in any circumstance where you don’t feel free to trust everyone to whom you give the ip address to this server, then you’ll get fired and possibly imprisoned if you stop here, but why the hell are you relying on a tutorial from someone who started by pointing out that he doesn’t know what he’s talking about anyway?