Nginx-plus as a reverse proxy for Atlassian JIRA Data Center edition

Update: Moved the code to Github instead of inline in this blog post.

At work we run the Atlassian JIRA Data Center edition, which is basically a multi-node setup of the normal JIRA server setup. Since there was very little information online about an actual working nginx (plus) load balancing config for this setup, I decided to publish the one we ended up with and works well for us.

This one is built for nginx-plus, but if you leave out the health checks you can probably run in on the open source nginx version too.

Check the Github repository: https://github.com/karelbemelmans/jira-datacenter-nginx-load-balancer

That’s all.

Another year is almost gone.. this was my 2018

I haven’t posted a single blog post yet this year, so it’s time for a catch-up post. It’s not that 2018 was a boring year, I’ve just been too caught up into work and real life to take time to write some stuff down.

nginx

If you would ask me to summarize the year into one word, I would tell you “nginx”. Nginx has become our main tool at work to build our global load balancing service, where nginx takes on the role of CDN, edge load balancer (via an anycast setup), API gateway, reverse proxy and sidecar proxy for applications.

We used a combination of open source nginx and the licensed nginx-plus version (which has a ton more features than the open source one) to replace old, monolithic hardware load balancers with (mostly F5 BigIp’s) and it worked out great. With nginx we can now see load balancers as software solutions that run on everyday harware and perfectly scale out horizontally, and not as exotic hardware powerhouses that need specialised hardware (and support contracts) to keep running.

Expect a big blog post coming in 2019 where I go a bit more into depth about our setup, for now I’ll leave you hungry for the details.

Public Cloud

I’m still a big fan of AWS (I renewed my AWS Solutions Architect - Associate certification), but I’ve not worked with it quite as much as I’d like. Google and Azure are still somewhere at the bottom of my todolist, I just haven’t had the time or a project to dig deeper into them.

The public cloud is an awesome tool for a lot of problems nowadays, but there are still a lot of cases when a public cloud is not the best solution. When it comes to my own experience:

  • If you already have datacenters all over the world, you better use them all the way and not go hybrid.
  • The cloud only works if you can go all-in and not just see it as “yet another virtualisation platform” and keep that cloud-agnostic approach. You really need to embrace the polycloud or stick to on-prem.
  • But maybe you can go all-in for just a small set of your infrastructure? I’ll give you a big tip: Big Data, data processing, machine learning and A.I really shine on the public cloud.

Company values

Oh boy, have I learned a lot about what a big multinational company thinks about their values and how every employe has to make sure they live up to them.

My job is no longer about just coming up with the right technical solution, I also have to learn how to deal with all kinds of people on a daily basis, and get them to accept that my solution is also in their best interest.

It’s quite a challenge but I’d like to think I’ve gotten a lot better at it the past year.

Data

Big Data, Machine Learning and A.I. have been important topics the past years, but in 2019 they will really explode in adaption by even non-tech companies. I will def. post some things about that the next year.

Podcasts

I still haven’t started with them I’m afraid, reading things at my own pace is still my preferred method of getting to know new things. I’ve also read more books (non-fiction) than any other year before.

Personal life

I stopped sharing personal things online, with the exception of maybe a few home network updates. Posting vague pictures with cheesy comments on Instagram for a hand full of close friends is as far as I’m willing to go nowadays.

Golf

Let me step aside from that previous paragraph already: In 2018 I started to play golf and I absolutely love it. It will be a long journey before I’m any good at it, but I’m convinced it’s going to be fun and worth the frustrations I’m going through right now.

Docker nginx container with GeoIp database

This is a Docker nginx container that includes the MaxMind GeoIP Country database. It injects the X-Origin-Country-Code and X-Origin-Country-Name headers into http requests to the backend with the country code of the requester.

Note: This container needs some more configuration before it actually runs, like the backend “backend” needs to exist. But you are most likely to copy/paste parts of this code into your own project anyway :)

Docker PHP container with New Relic

I created a small example Docker container that runs PHP with the New Relic agent installed. The New Relic agent on the Docker host cannot monitor things inside the containers, so we need to install the agent inside each Docker container.

The Dockerfile looks like this:

# My PHP base container: https://github.com/karelbemelmans/docker-php-base
FROM karelbemelmans/php-base:7.1
MAINTAINER mail@karelbemelmans.com

# You should override this when you run the container!
# It will get appended to the New Relic appname in the entrypoint scripts e.g. my-php-container-local
ENV environment local

# Install New Relic
RUN set -x && DEBIAN_FRONTEND=noninteractive \
  && wget -O - https://download.newrelic.com/548C16BF.gpg | apt-key add - \
  && echo "deb http://apt.newrelic.com/debian/ newrelic non-free" > /etc/apt/sources.list.d/newrelic.list \
  && apt-get update \
  && apt-get install -y newrelic-php5 \
  && newrelic-install install \
  && rm -rf /var/lib/apt/lists/*

# We need to copy the New Relic config AFTER we installed the PHP extension
# or we get warnings everywhere about the missing PHP extension.
COPY config/newrelic.ini /usr/local/etc/php/conf.d/newrelic.ini

# Generate an example PHP file in the webroot
RUN echo '<?php phpinfo();' > /var/www/html/index.php

# Our entrypoint script that also modifies the New Relic config file
COPY entrypoint.sh /
CMD ["/entrypoint.sh"]

The entrypoint.sh scripts:

#!/bin/bash -e

# Update the New Relic config for this environment
echo "newrelic.appname=my-php-container-${environment}" >> /usr/local/etc/php/conf.d/newrelic.ini

# Proceed with normal container startup
exec apache2-foreground

Besides that you need to create the config/newrelic.ini file too with your license id (and probably more options):

# First of all enable the extension
extension=newrelic.so

# Our license key is required
newrelic.license="REPLACEME"

# Enable this if you configured your account for High Security
#newrelic.high_security=true

Source code on Github: https://github.com/karelbemelmans/docker-php-newrelic

Note: This container is built on top of my PHP 7 Base Docker container, which you might also find useful.

Common misconceptions about the public cloud

I’m currently a big fan of public clouds, mostly because of the Infrastructure as a Service tools they offer: by uploading a simple JSON or YAML template I can create infrastructure services and network services that scale accross multiple physical and even geographical locations, and automatically install all the applications that run on top of them.

But when I talk to people in other companies I still hear a lot of misconceptions about how all of this works and what it costs. I will try to get rid of the biggest ones in this blog post.

Disclaimer: While most of the things I’m going to address are not unique to a specific public cloud, I will be using AWS as a reference since that’s the public cloud I’m currently most familiar with.

Sidenote: Have a look at my CloudFormation templates on Github for some examples to use on Amazon Web Services.

Misconception: You have no idea about costs on a public cloud

False.

1. Set limits

While the public cloud offers you a virtually endless amounts of resources to use, you can, and MUST, set limits on everything. E.g. when you are creating an Auto Scaling Group (a service that creates and destroys instances depending on the resources needed) you always set an upper and a lower limit for the number of instances it can create when it executes a scaling action.

2. Set warnings

Pretty trivial to point out, but you can track your costs on a daily basis, with warnings if a certain threshold has been reached. But it’s your job to monitor those costs and act upon them if they are not what you expected them to be.

A big aide in this is using tags for your resources. Tags allow you to group resources together easily in the billing overview. Tags could include Environment (e.g. prod, staging, test, …), CostCenter (a common used tag for grouping resources per department), Service (e.g. Network, VPC, Webservers) and whatever tag you want to use. The key really is “more is better” when it comes to tags since that allows you to refine to a very low level.

3. Simulate your costs well

Before moving to the public cloud it’s perfectly doable to create a cost similation of what the setup will cost. AWS offers you the Simple Monthly Calculator and TCO Calculator. It is however YOUR job to do this as detailed as possible, with taking storage usage and bandwidth usage into account to make this a good estimate.

4. Don’t keep things running 24 / 7 if they don’t need to

On AWS you pay per hour you use a resource, e.g. a virtual server. On Google Compute Engine you even pay per minute, so destroying resources when you don’t need them is a must to keep costs down.

Using Infrastructure as a Code you can create templates that will build your infrastructure, networking and application setups, as I’ve stated above already. But this also allows you to create identical stacks for a staging, QA or development environment whenever you need it, which you can destroy again when you are done using them.

A simple example would be a QA environment, identical to the production environment, that only runs during office hours, since nobody will be using it outside of those.

If you provide enough input parameters for your IaaC templates you can even optimize costs more: production runs in 2 physical locations, QA could run in only one, since it does not require the same level of high availability.

Misconception: But AWS crashes all the time :(

One of AWS’s slogans is actually Design for faillure.

Hardware crashes, there is no hosting company that will give you a 100% uptime guarantee on any single piece of hardware. Even with the most redundant setup there are still weak links in the chain that can break. What you need to do on the public cloud is to make sure that your application can handle faillure

Examples:

  • run Auto Scaling instances behind a load balancer instead of a single host (yes you will need to redesign your application for this)
  • run in multiple datacenters and even in multiple geographical regions
  • get a good understanding of which service from the public cloud is offered as high-available and what you still have to do yourself

Designing your application for the public cloud will be a challenge, but there are enough example cases already that shows you how to do this. And with container technology becoming more mature every month this suddenly got a whole lot easier to achieve.

Misconception: We don’t want vendor lock-in to public cloud provider X

In all the applications I have created or moved to the cloud there was very little vendor lock-in besides the IaaC tool we used (and you can even use something like Terraform to remove that lock-in). Things that all applications use but are not specific to any public cloud:

  • Linux virtual servers
  • Docker containers
  • MySQL/PostgreSQL databases
  • memcache and/or Redis
  • NFS storage
  • DNS

The main thing of the application is still the Docker container, where your actual code runs. On AWS this will run on ECS with ALB’s, but you can just as well run the containers on Google Compute Engine or Microsoft Azure with the equivalents of those systems, it will not require change in your application code at all.

But… if you want to make most use of public cloud provider X you will need to develop for it. On AWS you would e.g. make your application run on ECS and use S3, SNS and SQS to glue things together. But once you do this you will realise how powerfull and virtually without limits the public cloud is.

I hope you found this blog post usefull, feel free to leave a comment below.