Home Server Nginx How do I configure HHVM for multiple virtual hosts?

Get social!
HHVMI chanced upon HHVM the other day, so I thought I would give it a try on my server. For those of you who haven't heard about it, HHVM is an open-source virtual machine (originally developed by the folks at Facebook) that runs PHP (and Hack) programs from the command line or via Apache, Nginx, etc. I recently read that benchmarks showed that HHVM can outperform the native PHP engine "hands down", so I figured I would give it a try. Installing HHVM is pretty straightforward. Since I use NGINX, I followed the instructions here: https://www.digitalocean.com/community/tutorials/how-to-install-hhvm-with-nginx-on-ubuntu-14-04 Then, I wrote a small bash script (nginx-benchmark.sh) to benchmark a single virtual host:

# Usage: ./nginx-benchmark.sh <domain-name> <[http|https]> <[1|'']>
if [ ! -n "$SKIP_WGET" ]
    # Make sure access.log has all links from site
    wget -r -l 10 -O out.html $HTTP://www.$DOMAIN 
    rm out.html
awk '{ print "www."ENVIRON["DOMAIN"]$7 }' /var/log/nginx/$DOMAIN/access.log | sort | uniq > urls.txt
siege -c 15 -f urls.txt -i -b -t 60S 
Here's an example of what I saw:

Transactions:              16722 hits
Availability:              98.98 %
Elapsed time:             119.94 secs
Data transferred:         193.51 MB
Response time:              0.11 secs
Transaction rate:         139.42 trans/sec
Throughput:             1.61 MB/sec
Concurrency:               14.89
Successful transactions:       16731
Failed transactions:             172
Longest transaction:           10.06
Shortest transaction:           0.00


Transactions:              10360 hits
Availability:              98.93 %
Elapsed time:             119.61 secs
Data transferred:         121.30 MB
Response time:              0.17 secs
Transaction rate:          86.61 trans/sec
Throughput:             1.01 MB/sec
Concurrency:               14.92
Successful transactions:       10353
Failed transactions:             112
Longest transaction:           13.01
Shortest transaction:           0.00
As you can see, the results were pretty impressive. Being a shoot-first-ask-questions-later type of guy, I decided to convert all my virtual hosts over to HHVM. The problem I ran into was that there was no clear-cut way to tell HHVM to run as a specific user for each virtual host, which I like to do for security reasons. After googling around for an hour or so, I finally figured out that I would need to run multiple instances of HHVM with different config files for each. Not wanting to write 50 separate config files, I called on Python for an assist. First, I created a directory under /etc/hhvm called conf.d, then created within it a file called template.ini. It looks like this:
; %%DOMAIN%%

; hhvm specific 

hhvm.server.type = fastcgi
hhvm.server.user = %%USER%%
hhvm.server.default_document = index.php
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/nginx/%%DOMAIN%%/hhvm-error.log
hhvm.repo.central.path = /var/run/hhvm/hhvm-%%USER%%.hhbc

; php options
session.save_handler = files
session.save_path = /home/php/sessions/%%USER%%
session.gc_maxlifetime = 1440

; hhvm specific 
pid = /var/run/hhvm/%%USER%%.pid
hhvm.log.level = Warning
hhvm.log.always_log_unhandled_exceptions = true
hhvm.log.runtime_error_reporting_level = 8191
hhvm.mysql.typed_results = false
hhvm.enable_zend_sorting = 1
You may have to make some changes to the file above to suit your specific setup. Specifically, check the hhvm.log.file and session.save_path entries. Next. I wrote a quick Python script (man, I love Python) and called it make-ini.py. I dropped it in the same directory (conf.d) as template.ini:
#!/usr/bin/env python

import sys
import time

if len(sys.argv) != 2:
    print 'Usage: ' + sys.argv[0] + ' <domain-name>'

domain = sys.argv[1]
user = domain.replace('.', '_')

print 'Creating hhvm setup for domain', domain, 'and user', user

template_ini_str = open('template.ini', 'r').read()
template_ini_str = template_ini_str.replace('%%USER%%', user).replace('%%DOMAIN%%', domain)

open(user + '.ini', 'w').write(template_ini_str)
The script basically looks at all the folders in /var/www (which are domain names) and create the appropriate .ini file for each domain. You'll definitely need to make some adjustments to it to suit your environment. After running the above script, I now had all the config files set up. Next, I needed to tell my server to load them at start up. To do this I first disabled the default hhvm script in services:
update-rc hhvm disable
Then I created another template and Python script to create configs for supervisord. If you've never used it, it's a simple-to-use alternative to the normal sys-init scripts (or now systemd [ugh!]). Just run:
sudo apt-get supervisor
on Ubuntu (or any Debian-based system) to install. I navigated to the /etc/supervisor/conf.d directory and created template.conf_ (note the underscore at the end -- you don't want supervisor to mistake this file as a config file!).
# %%USER%%
command=hhvm -m server -c /etc/hhvm/conf.d/%%USER%%.ini
stdout_logfile = /var/log/hhvm/%%USER%%.log  
stderr_logfile = /var/log/hhvm/%%USER%%.err
Then I created the make-conf.py script to create all the config files for supervisor:
#!/usr/bin/env python

import sys
import time

if len(sys.argv) != 2:
    print 'Usage: ' + sys.argv[0] + ' <domain-name>'

domain = sys.argv[1]
user = domain.replace('.', '_')

print 'Creating hhvm supervisor setup for domain', domain, 'and user', user

template_ini_str = open('template.conf_', 'r').read()
template_ini_str = template_ini_str.replace('%%USER%%', user).replace('%%DOMAIN%%', domain)

open(user + '.conf', 'w').write(template_ini_str)
Now I had all the necessary config files to run an HHVM process for each virtual user. I restarted the server and sure enough there were no HHVM processes running. What the what? Turns out I needed to create the /var/run/hhvm directory beforehand (and recreate it at each boot). The HHVM config files use that directory to store socket files and .hhbc files (I'm not sure what they are for). I stuffed the following lines in the supervisor init file (/etc/init.d/supervisor):
# For hhvm --DR
mkdir -p /var/run/hhvm 2>/dev/null
chmod 3777 /var/run/hhvm 2>/dev/null
That way the hhvm directory gets created and set with the proper file permissions. Now, before you start yelling at me and threatening to take away my sys admin badge, I realize that the supervisor init script was a poor choice to set up the HHVM run folder, but I couldn't think of anywhere better. Perhaps I should have created a custom init script for this purpose. I also looked into pre-commands for HHVM and supervisor, but couldn't find anything good. Anyway, keep in mind that if you update supervisor, you will be prompted to overwrite the init script. If you do, HHVM will break on your first reboot. Moving forward, I rebooted and HHVM was up and running. All that was left was to set up NGINX. I decided to perform this task manually, so as not to break 50 virtual web servers simultaneously. Basically what I did was for all my config files in /etc/nginx/sites-enabled I removed the existing php location block -- the ones that  start like this:
location ~ \.php$ {
and replaced it with the following:
    location ~ \.(hh|php)$ {
        fastcgi_keep_conn on; 
        fastcgi_pass  unix:///var/run/hhvm/xenabeast_com.sock; 
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include        fastcgi_params;
That did it! A restart on NGINX after each config file change and some testing of each virtual host worked (almost) flawlessly. I had problems with phpMyAdmin (MySQL database issues) and roundcube (couldn't find html class). I finally gave up on them (for now). Other than that all my virtual hosts were running HHVM (and no customer's have screamed so far). Since converting to HHVM, I've been monitoring the server closely and I am not seeing any performance hits. Actually the reverse is true: running HHVM as actually decreased the load on the server.

Bad Behavior has blocked 38 access attempts in the last 7 days.