Nginx: Reverse Proxy for Jenkins & Apache SSL Hosts

I have a server hosting several staging and testing instances of various websites that are accessible via secured and unsecured connections.

I intend to set up a Jenkins CI server and make it accessible only via a secured connection. Of course, I would like the rest of the hosted websites to keep running like before.

The solution: use Nginx as a reverse proxy that passes requests either to the Jenkins server or to Apache.

Jenkins is configured to run at port 8080. I would like Nginx to listen to connections over port 443 on the Jenkins URL and pass them to the localhost:8080. Also, all incoming connection to the Jenkins URL will be redirected to port 443 and from there via the proxy to localhost:8080

The secure Jenkins connection will require a certificate. Since the CI server is not public, I can save some money and use a self-signed certificate (Or better yet, get a Let’s Encrypt certificate).

To keep the Apache hosts operational, I will have to configure Apache to listen to ports other than 80 and 443, which now will be taken by Nginx.

The following instructions are largely based on tutorials published at DigitalOcean How To Configure Nginx with SSL as a Reverse Proxy for Jenkins and How To Configure Nginx as a Web Server and Reverse Proxy for Apache on One Ubuntu 16.04 Server

Installing Nginx

The Nginx installation has been straightforward and unproblematic:

sudo apt-get update
sudo apt-get install nginx

Nginx SSL Certificate

As said above, I am going to create and use a self-signed certificate. The command is as follows:

sudo mkdir -p /etc/nginx/ssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/cert.key -out /etc/nginx/ssl/cert.crt

The command prompts me to enter a few details for the certificate and then creates the key and certificate files at the specified paths.

I have to change the permissions on the key file so:

sudo chmod 400 /etc/nginx/ssl/cert.key

Configuring the Proxy for Jenkins

At /etc/nginx/sites-available I create a host configuration file jenkins with the following content:

server {
   listen 80;
   listen [::]:80;
   server_name jkns.myserver.com;
   return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl;
    server_name jkns.myserver.com;

    ssl_certificate           /etc/nginx/ssl/nginx.crt;
    ssl_certificate_key       /etc/nginx/ssl/nginx.key;

    ssl on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;

    access_log            /var/log/nginx/jenkins.access.log;

    location / {
      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;

      # Fix the "It appears that your reverse proxy set up is broken" error.
      proxy_pass          http://localhost:8080;
      proxy_read_timeout  90;

      proxy_redirect      http://localhost:8080 https://jkns.myserver.com;
    }
}

Configuration file explained

This explanation is based on the Nginx documentation.

The first configuration block listens to incoming requests on port 80 for the Jenkins server URL and redirects them permanently to the secure URL. The line with [::]:80is for IPv6 addresses that are specified in square brackets

server {
   listen 80;
   listen [::]:80;
   server_name jkns.myserver.com;
   return 301 https://$server_name$request_uri;
}

The next block is the secure host configuration for the Jenkins server. It listens to port 443 with the sslparameter. I also have to specify the locations of the certificate and key files:

server {
    listen 443 ssl;
    server_name jkns.myserver.com;
 
    ssl_certificate           /etc/nginx/ssl/nginx.crt;
    ssl_certificate_key       /etc/nginx/ssl/nginx.key;

...
}

Next follows the configuration for the ngx_http_ssl_module.

The ssl on directive is made obsolete in version 1.15.0 in favor of the listen directive and can be omitted.

The setting ssl_session_cache sets the types and sizes of caches that store the session parameters. The parameter builtin specifies the use of the cache that is built-in OpenSSL and can be used by one worker process only. The number 1000 indicates the cache size in the session. The parameter sharedalso allows the use of a cache shared between all worker processes. Its size is set to 10 megabytes:

ssl_session_cache  builtin:1000  shared:SSL:10m;

The directive ssl_protocols enables the specified protocols, and the directive ssl_ciphers enables the specified ciphers. The line with ciphers is presented in the format understood by OpenSSL. The ssl_preser_server_ciphers set to on indicates that the server ciphers should be preferred over the client ciphers when using SSLv3 and TLS protocols:

ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;

Next, a line to specify the path to the host access log:

access_log            /var/log/nginx/jenkins.access.log;

Finally, the location block configures the proxy settings for the Jenkins server. The settings are for the module ngx_http_proxy_module.

First, several entries for the directiveproxy_set_header. This directive allows redefining or appending fields to the request header passed to the proxied server. The field name is the token that comes after the directive, e.g., Host, and the new value follows it.

A note to the $proxy_add_a_forwarded_for variable. Its value is the X-Forwarded-For client request header field with the $remote_addr variable appended to it, separated by a comma. If the X-Forwarded-For field is not present in the client request header, the $proxy_add_x_forwarded_for variable is equal to the $remote_addr variable.

The $scheme value in the X-Forwarded-Proto field is expected to be https

proxy_set_header        Host $host;
proxy_set_header        X-Real-IP $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header        X-Forwarded-Proto $scheme;

If Jenkins shows the It appears that your reverse proxy set up is broken error, we must make sure that the parameter value in the directiveproxy_pass matches the Jenkins URL. In my case, the localhost URL works fine.

Finally, this line:

proxy_redirect      http://localhost:8080 https://jkns.myserver.com;

Here, the first parameter is the redirect URL and the second is the replacement.

I make sure that the configuration file is symlinked to the sites-enabled directory:

sudo ln -s /etc/nginx/sites-available/jenkins /etc/nginx/sites-enabled/jenkins

After the restart, the new site configuration is enabled:

sudo service nginx restart

Configuring the Reverse Proxy for Apache Hosts

As mentioned in the beginning, I have several staging and testing sites running on the server. They must be accessible via https. Additionally, a few placeholder sites should be accessed unencrypted.

To start, I change the Apache ports.conf file where I change lines listening to ports 80 and 443 to 8081 and 8443 respectively:

Listen 8081

        Listen 8443

Then, I go to the files containing the virtual host configuration and change the ports there as well:

...

...

I disable the files containing secure virtual hosts’ configurations because they will not be needed in the new setup.

Then, I create a new configuration file at /etc/nginx/sites-available/apachethat will contain the reverse proxy configurations for the Apache hosts.

First, I create entries for unsecure websites:

server {
    listen 80;
    server_name my-unsecure-site.com www.my-unsecure-site.com;

        location / {
        proxy_pass http://my_server_ip_address:8081;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
server {
...
}

Now, I need to configure server entries for staging and testing sites that will be accessible both via secure and unsecure connections:

server {
    listen 80;
    listen 443 ssl;
    server_name stage.my-site.com testing.my-site.com;

    ssl on;
    ssl_certificate /etc/ssl/certs/my-site.com.crt;
    ssl_certificate_key /etc/ssl/private/my-site.com.key;

    location / {
        proxy_pass http://my_server_ip_address:8081;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
server {
    listen 80;
    listen 443 ssl;
    server_name stage.my-other-site.com testing.my-other-site.com;

    ssl on;
    ssl_certificate /etc/ssl/certs/my-other-site.com.crt;
    ssl_certificate_key /etc/ssl/private/my-other-site.com.key;

    location / {
        proxy_pass http://my_server_ip_address:8081;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Now, in an incoming connection should be secure, the configuration can access the certificate and key files and encrypt it. The request is then forwarded to the port 8081 that Apache listens to.

This, however, is a problematic configuration. With it, the browser will receive the default server certificate, i.e., my-site.com.crt regardless of the requested server name. This is caused by SSL protocol behavior The SSL connection is established before the browser sends an HTTP request and Nginx does not know the name of the requested server. Therefore, it may only offer the default server’s certificate.

There are several solutions discussed in the documentation, and for a production setup, I will have to implement one of them. These, however, are testing instances that already use mismatched certificates (I did not order a wildcard or name-specific certificates for them), so this will do.

Configuring mod_rpaf

Source: Digital Ocean

For this configuration to work, we need an Apache module mod_rpaf. This module rewrites the values of REMOTE_ADDR, HTTPS, and HTTP_PORT based on the values provided by the reverse proxy. We will install the latest version by compiling it from the source.

First, we need to install packages required by this module:

sudo apt-get install unzip build-essential apache2-dev

Then, I download the latest stable release of the module from GitHub:

wget https://github.com/gnif/mod_rpaf/archive/stable.zip
unzip stable.zip

I change to the unzipped module directory and compile and install the module:

cd mod_rpaf-stable
sudo make
sudo make install

Next, I create the configuration loader file for the module:

sudo vi /etc/apache2/mods-available/rpaf.load

I add this line to the configuration loader file :

LoadModule rpaf_module /usr/lib/apache2/modules/mod_rpaf.so

Next, I create the configuration file in the same directory:

sudo vi /etc/apache2/mods-available/rpaf.conf

The content of the configuration file is this:

    RPAF_Enable             On
    RPAF_Header             X-Real-Ip
    RPAF_ProxyIPs           my_server_ip_address 
    RPAF_SetHostName        On
    RPAF_SetHTTPS           On
    RPAF_SetPort            On

Here’s a brief description of each directive. See the mod_rpaf README file for more information.

• RPAF_Header – The header to use for the client’s real IP address.
• RPAF_ProxyIPs – The proxy IP to adjust HTTP requests for.
• RPAF_SetHostName – Updates the vhost name so ServerName and ServerAlias work.
• RPAF_SetHTTPS – Sets the HTTPS environment variable based on the value contained in X-Forwarded-Proto.
• RPAF_SetPort – Sets the SERVER_PORT environment variable. Useful for when Apache is behind a SSL proxy.

Finally, I enable the module and restart Apache:

sudo a2enmod rpaf
sudo service apache2 restart

Possible problems

In the logfile at /var/log/nginx/error.log you might see entries like:

bind() to 0.0.0.0:443 failed (98: Address already in use)
bind() to 0.0.0.0:80 failed (98: Address already in use)

This indicates that these ports are currently in use. This command frees them, then restarts Nginx:

sudo fuser -k 443/tcp
sudo service nginx restart