Best Practices for Elasticsearch mappings

At first, Elasticsearch may appear to be schemaless since you can add new fields any time you want, but every field in a document must match the mapping.

Dynamic Templates reduce boilerplate

How many times have you opened up a mapping file to something like this where the same type definition is repeated over and over again?

{
  "properties": {
    "foo": {
      "type": "keyword"
    },
    "foo": {
      "type": "keyword"
    },
    "foo": {
      "type": "keyword"
    },
    "baz": {
      "type": "keyword"
    },
    "other": {
      "type": "text"
    },
    ...
  }
}

It’s super easy to refactor this into an alternative where by default all string values are mapped as keyword, except for the specific field listed as “text”.

{
  "properties": {
    "dynamic_templates": [
      {
        "example_name": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ],
    "other": {
      "type": "text"
    }
  }
}

Disable type detection

For new fields, Elasticsearch can automatically identify what type to use, but it can be wrong or do unexpected things. For example, I’ve seen Elasticsearch accidentally identify a decimal value as a long because the first value to go into the index did not have any decimal points. Then all other documents failed to be indexed because they did not match. This is especially important if you have fields that have a wide range of values (for example, user controlled) because you can’t predict if the first value is going to look like a number or a date, when it should always be considered to be a string.

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html

{
  "mappings": {
    "date_detection": false,
    "numeric_detection": false
  }
}

Fast development environments

Setting up new hosts entries for every different web site that you develop is hard. This workflow allows you to completely automate it. First thing you’ll want to do is setup a wildcard DNS record that points to your host. This allows you to dynamically setup new development websites without having create new DNS records for each one of them. I created a fake internal-only TLD on my local network’s DNS server that automatically returns the IP address of my development VM for any query to *.devvm. If you don’t have access to that, you could re-use an actual domain and automatically forward something like *.dev.technowizardry.net to the VM. For example, I have the ASUS RT-AC68U router for my personal network. So I SSH’d to the router, typed vi /etc/dnsmasq.conf, then appended:

Continue reading “Fast development environments”

Creating Simple Omnibox Shortcuts in Chrome

Lately I’ve been dealing a lot with development stuff that requires I specify a particular port in the URL when visiting the application or service I’m trying to access. I quickly grew tired of having to type in the url (or letting it autofill) and then adding the correct port at the end. Since I’m running many web applications on my desktop at home that all run on different ports, there were a lot of results popping up for 127.0.0.1. I wanted a quicker way to access particular applications on my local network, and thus I set out for ways to create omnibox shortcuts.

It turns out that creating omnibox shortcuts in chrome is actually really easy, although maybe I’m just exploiting a different tool to do what I want with it. Basically we’re going to add our own custom “search engines” to the omnibox. But we’re not going to search for anything. We’re just going to use the awesome ability to add custom search engines to also add shortcuts.

chrome://settings/searchEngines

In here you’ll see that a lot of websites you visit appear to already be adding search engines to your browser for you under the “Other search engines” section. How nice of them. Let’s do the same thing, except we’re going to add our own like this:

LocalPlex    plex    127.0.0.1:32400

That “search engine” will actually enable us to just type “plex” into the omnibox and hit enter, chrome will automatically take you to 127.0.0.01:32400 in that tab.  Nifty. While we’re at it, I set up my home router to use HTTPS only for the administration pages, but since the AiCloud software built into the router listens on port 443, we have to access the admin panel via a different port. We’ll use 8443.

HomeRouter    homerouter    https://192.168.1.1:8443

And now when we type homerouter and hit enter, it’ll take us to the correct port over https so we can login. Neat!

You can also use this to add actual searching mechanisms for your favorite sites as long as you know the URL query structure, but to be honest I don’t really care enough about that feature. I just wanted shortcuts in my omnibox. Now you can have shortcuts in your omnibox too.

99 Problems, OCSP Ain’t One (Anymore)

So it took me a while to figure out why OCSP Stapling wasn’t working on the server I’m building with Adam. I figured I’d write down what I found here to sort of cover my problems. This is by no means a comprehensive list or solution, just what I found worked for me.

It seems that for OCSP to work properly you need to include it in the default server block.

In /etc/nginx/sites-enabled/default:

server {
    listen 443;
    server_name _;
    ssl on;
    ssl_stapling on;
    ssl_certificate /etc/nginx/certs/your_cert.pem;
    ssl_certificate_key /etc/nginx/certs/your_cert.key;
}

For whatever reason this wasn’t working alone for me, so I also added ssl_stapling on; to the http block in /etc/nginx/nginx.conf

Now OCSP seems to be working correctly on my other subdomains, this appears to be due to a limitation with openssl tests not allowing SNI.

You can test your own OCSP Stapling status using the following command:

openssl s_client -connect your.site:443 -tls1 -tlsextdebug -status


It appears that on the first load it’s not necessarily cached, so try running the command twice back to back to confirm whether you see:

OCSP response: no response sent


or:

OCSP Response Data:    OCSP Response Status: successful (0x0)    Response Type: Basic OCSP Response    Version: 1 (0x0)    Responder Id: 90AF6A3A945A0BD890EA125673DF43B43A28DAE7    Produced At: Oct 18 00:36:24 2014 GMT    Responses:    Certificate ID:      Hash Algorithm: sha1      Issuer Name Hash: 7AE13EE8A0C42A2CB428CBE7A605461940E2A1E9      Issuer Key Hash: 90AF6A3A945A0BD890EA125673DF43B43A28DAE7      Serial Number: FD9BEFA92F8BEBCE721B67BED87783E3    Cert Status: good    This Update: Oct 18 00:36:24 2014 GMT    Next Update: Oct 22 00:36:24 2014 GMT

Best of luck to you in your journey for a better SSL server.