You don’t have enough static analysis


Pretty much every programming language out there has tools that statically analyze your source code and detect different problems. These problems can range from simple things like ensuring that you have consistent casing for variable names in Java to ruthlessly enforcing method limits in Ruby. If you’ve ever used one of these tools, they may seem overbearing and not worth the hassle, but they will soon prove their value once your application becomes larger, has multiple developers, or is business critical and can’t afford outages caused by trivial mistakes. Static analysis tools are a super-low cost solution for improving the quality of a code-base.

Continue reading “You don’t have enough static analysis”

Structured and auditable changes to infrastructure

Note: I’m going to use AWS services as most of my examples for this post, but that’s just because I’m most familiar with them, the patterns found below are not limited to just AWS and can be applied to any cloud provider or self-hosted where similar patterns exist.


Every service has some amount of supporting infrastructure required to support it. This includes any virtual servers (EC2 or other), storage (ex. S3, DynamoDB), load balancing, etc. basically any resources that your service uses that is not your direct business logic could be considered infrastructure. If you use continuous integration and change control on your business logic, then why would you not apply the same rules to your infrastructure?

Allowing and requiring developers to make changes using the UI introduces risk that one might make a mistake and bring down your production service. Continuing from my last post about infrastructure names, you could also make a mistake in any of regional clones.
Continue reading “Structured and auditable changes to infrastructure”

Vending Software Good Practices – Docker Security

Docker containers are the latest craze taking the world by storm. They enable software vendors to have more control over how their software is executed reducing the amount of work that software hosters need to be responsible for. By shifting the burden of figuring out environment requirements on to the software vendor, certain critical decisions that help improve security can be made once and only once and distributed to end-users. This reduces the cost barrier of having more stable/secure software as users no-longer have to think about intricacies of security and management, which we can see that users rarely take the time to invest in.

Docker containers have a number of different security mechanisms. I won’t go into details on that, if you’re interested in learning more, make sure to read the Docker security documentation page.


In Linux kernels, each process has a set of capability flags that the kernel checks when the process makes certain privileged syscalls. Processes running as root automatically get certain capabilities assigned to it.

Some example capabilities:

  • CAP_NET_BIND_SERVICE – Enables processes to bind to ports < 1024. By default, non-root processes can’t find to these reserved ports. Dropping this capability prevents even root processes from binding to these ports
  • Even more on the man page

According to the principal of least privilege, running with fewer capabilities will reduce the attack surface of a given piece of software.

Docker compose.yml

Docker compose files are a popular way to vendor an entire service stack to users. With it you can describe one or more Docker containers in a YAML-based format. More information is available in the official docs. A little used feature enables you to specify which capabilities your service requires.

For example, this is the configuration that I use for running NGINX on my server:

  image: nginx:1.9.10
    - ALL
    - CHOWN
    - SETGID
    - SETUID

In this example, I enable a whitelist for capabilities instead of using the default list that Docker provides and enable only the minimal capabilities that are required. This list enables NGINX to modify file permissions (for access logs,) bind to port 80 and 443, and change the process user account. The default whitelist is available in the Docker source code here. Based on this, we’re reducing the attack surface that a malicious actor can leverage.

Docker compose is fully self-contained and doesn’t require the user to make any changes to their environment to start using. Docker compose and capabilities are a low-cost way to start reducing the attack surface of an application. Every service owner should attempt to run their application with –cap-drop ALL, then selectively enable capabilities until their application works, then vend that list as a best practice.

AppArmor/Security Profiles

Capabilities are a cheap way to begin to improve security, but they can only restrict a limited subset of kernel sys calls, making fine grained security control impossible. This is where mandatory access control and AppArmor strives. For distributions that support it (such as Ubuntu,) AppArmor is an opt-in security model that enables you to whitelist and/or blacklist specific sys calls, along with the parameters of those sys calls. For example, you could configure a Docker container application to only be able to open TCP connections to specific IP ranges and ports. Docker supports the ability to run containers with specific AppArmor profiles. While this requires more work on the user’s side to use, security conscious service vendors could vend an AppArmor profile along with their service that users could install. I plan to go into more detail on this in the future.


Anybody who builds a Docker container should leverage the security model that Docker provides by running with least privileges and capabilities, then include that configuration in vendor configuration, like Docker compose files. By doing this, your end users all will be able to take advantage of slightly reduced attack surface area, with only minimal effort on your side. Capabilities are in no way fool-proof, and one should never believe that they will significantly reduce the attack surface, but it’s better than nothing.

Dynamic AWS resource discovery for one-click region spin-ups

Disclaimer: At the time of this article’s writing, I work at Amazon, but not in AWS. This article is based on my own research and ideas and is not the official position of Amazon. This article is not intended as marketing material for AWS, only as some architectural patterns for you to use if you do leverage AWS.

AWS provides a number of different resources that you can use to build services using, including S3 buckets, SQS queues, etc. When you create a new instance of that resource, you must pick a name that usually must be unique in a given namespace. Depending on your naming scheme, you may also have to start embedding resource names in code or configuration files. This makes spinning up new regions difficult as now you have to update configuration with names for every stage/region that you might use. This may not seem like that big of a deal, but consider that you may have tens of different SQS queues, S3 buckets, etc. for each region/stage. This can begin to combinatorically explode as you now have # regions * # stages * # resources of different configuration definitions. This results in a lot of boilerplate.

But what if there was a better way?

Continue reading “Dynamic AWS resource discovery for one-click region spin-ups”

Fast development environments

Setting up new hosts entries for every different web site that you develop is hard. This workflow allows you to completely automate it. First thing you’ll want to do is setup a wildcard DNS record that points to your host. This allows you to dynamically setup new development websites without having create new DNS records for each one of them. I created a fake internal-only TLD on my local network’s DNS server that automatically returns the IP address of my development VM for any query to *.devvm. If you don’t have access to that, you could re-use an actual domain and automatically forward something like * to the VM. For example, I have the ASUS RT-AC68U router for my personal network. So I SSH’d to the router, typed vi /etc/dnsmasq.conf, then appended:

Continue reading “Fast development environments”

Creating Simple Omnibox Shortcuts in Chrome

Lately I’ve been dealing a lot with development stuff that requires I specify a particular port in the URL when visiting the application or service I’m trying to access. I quickly grew tired of having to type in the url (or letting it autofill) and then adding the correct port at the end. Since I’m running many web applications on my desktop at home that all run on different ports, there were a lot of results popping up for I wanted a quicker way to access particular applications on my local network, and thus I set out for ways to create omnibox shortcuts.

It turns out that creating omnibox shortcuts in chrome is actually really easy, although maybe I’m just exploiting a different tool to do what I want with it. Basically we’re going to add our own custom “search engines” to the omnibox. But we’re not going to search for anything. We’re just going to use the awesome ability to add custom search engines to also add shortcuts.


In here you’ll see that a lot of websites you visit appear to already be adding search engines to your browser for you under the “Other search engines” section. How nice of them. Let’s do the same thing, except we’re going to add our own like this:

LocalPlex    plex

That “search engine” will actually enable us to just type “plex” into the omnibox and hit enter, chrome will automatically take you to in that tab.  Nifty. While we’re at it, I set up my home router to use HTTPS only for the administration pages, but since the AiCloud software built into the router listens on port 443, we have to access the admin panel via a different port. We’ll use 8443.

HomeRouter    homerouter

And now when we type homerouter and hit enter, it’ll take us to the correct port over https so we can login. Neat!

You can also use this to add actual searching mechanisms for your favorite sites as long as you know the URL query structure, but to be honest I don’t really care enough about that feature. I just wanted shortcuts in my omnibox. Now you can have shortcuts in your omnibox too.

99 Problems, OCSP Ain’t One (Anymore)

So it took me a while to figure out why OCSP Stapling wasn’t working on the server I’m building with Adam. I figured I’d write down what I found here to sort of cover my problems. This is by no means a comprehensive list or solution, just what I found worked for me.

It seems that for OCSP to work properly you need to include it in the default server block.

In /etc/nginx/sites-enabled/default:

server {
    listen 443;
    server_name _;
    ssl on;
    ssl_stapling on;
    ssl_certificate /etc/nginx/certs/your_cert.pem;
    ssl_certificate_key /etc/nginx/certs/your_cert.key;

For whatever reason this wasn’t working alone for me, so I also added ssl_stapling on; to the http block in /etc/nginx/nginx.conf

Now OCSP seems to be working correctly on my other subdomains, this appears to be due to a limitation with openssl tests not allowing SNI.

You can test your own OCSP Stapling status using the following command:

openssl s_client -connect -tls1 -tlsextdebug -status

It appears that on the first load it’s not necessarily cached, so try running the command twice back to back to confirm whether you see:

OCSP response: no response sent


OCSP Response Data:    OCSP Response Status: successful (0x0)    Response Type: Basic OCSP Response    Version: 1 (0x0)    Responder Id: 90AF6A3A945A0BD890EA125673DF43B43A28DAE7    Produced At: Oct 18 00:36:24 2014 GMT    Responses:    Certificate ID:      Hash Algorithm: sha1      Issuer Name Hash: 7AE13EE8A0C42A2CB428CBE7A605461940E2A1E9      Issuer Key Hash: 90AF6A3A945A0BD890EA125673DF43B43A28DAE7      Serial Number: FD9BEFA92F8BEBCE721B67BED87783E3    Cert Status: good    This Update: Oct 18 00:36:24 2014 GMT    Next Update: Oct 22 00:36:24 2014 GMT

Best of luck to you in your journey for a better SSL server.

Quick Alerts using Python & Pushover

It seems like all the bad things always happen whenever we’re far away from our computers. In my case, I’m trying to purchase a server from a company that seems to sell them as quick as I get them, so knowing right away when they come in is important. Ideally I’d like my whatever device I’m using to let me know right away if something has been made available.

To solve my dilemma, I’ve written a short python script using the chump library to periodically reload the webpage to determine whether or not the server I want is available for purchase. Chump is a pretty awesome python wrapper for the Pushover API that lets you quickly and easily send push notifications to any of your devices, or all of them at once.

Unfortunately, Pushover has a license cost of $4.99, but given what it comes with it seemed like a no-brainer for me to pick this up. Today it’s a simple script to alert me when a server is available for purchase, tomorrow it’ll be server monitoring alerts via push notifications.

Once you’ve created a Pushover account, it’ll give you a user key that is used to uniquely identify you and send push notifications to you. To get started, you’ll want to create an application so you can receive an application key. In this example, we’re going to use USERKEY and APPKEY to denote the string values of your user key and application key.

Once you’ve gotten these two pieces of information and installed chump (pip install chump), you can begin your own notification script like so:

from chump import Application
myApp = Application(‘APPKEY’)
#check to see if we’re authenticated successfully
if myApp.is_authenticated:
     print “Application Successfully Authenticated!”
    print “Application Failed to Authenticate!”
    raise SystemExit
myUser = myApp.get_user(‘USERKEY’)
#check to see if the get_user returned successfully
if myUser.is_authenticated:
    print “User Successfully Authenticated!”
    print “User Failed to Authenticate!”
    raise SystemExit
#We’re not going to check user devices because we want to send to all
myAlert = myUser.sendMessage(“Testing Chump & Pushover!”)
if myAlert.is_sent:
    print str(myAlert.sent_at)
    print “Failed to send message! Aborting!”
    raise SystemExit

And now we’ve sent a basic message via Pushover using chump. For more information on chump, check it out on readthedocs. I’m not going to get too far into the details of my automated system, but this should at least get you started sending push notifications to your devices.