Recommended Practices for OpenShift Origin Master Hosts

In addition to pod traffic, the most-used data-path in an OpenShift Origin infrastructure is between the OpenShift Origin master hosts and etcd. The OpenShift Origin API server (part of the master binary) consults etcd for node status, network configuration, secrets, and more.

Optimize this traffic path by:

  • Co-locating master hosts and etcd servers.

  • Ensuring an uncongested, low latency LAN communication link between master hosts.

  • Ensuring the first etcd server listed in /etc/origin/master/master-config.yaml is the local etcd instance.

Recommended Practices for OpenShift Origin Node Hosts

The OpenShift Origin node configuration file at /etc/origin/node/node-config.yaml contains important options, such as the iptables synchronization period, the Maximum Transmission Unit (MTU) of the SDN network, and the proxy-mode.

The node configuration file allows you to pass arguments to the kubelet (node) process. You can view a list of possible options by running kubelet --help.

Not all kubelet options are supported by OpenShift Origin, and are used in the upstream Kubernetes. This means certain options are in limited support.

One of the most commonly discussed node options is max-pods, which limits the number of pods that can run on a node at any given time. Exceeding the max-pods values can result in:

  • Increased CPU utilization on both OpenShift Origin and Docker.

  • Slow pod scheduling.

  • Potential out-of-memory scenarios (depends on the amount of memory in the node).

  • Exhausting the pool of IP addresses.

  • Resource overcommitting, leading to poor user application performance.

In Kubernetes, a pod that is holding a single container actually uses two containers. The second container is used to set up networking prior to the actual container starting. Therefore, a system running 10 pods will actually have 20 containers running.

See the Sizing Considerations section in the installation documentation for the recommended limits for an OpenShift Origin cluster. The recommended sizing accounts for OpenShift Origin and Docker coordination for container status updates. This coordination puts CPU pressure on the master and docker processes, which can include writing a large amount of log data.

Recommended Practices for OpenShift etcd Hosts

etcd is a distributed key-value store that OpenShift Origin uses for configuration.

After profiling etcd under OpenShift Origin, etcd frequently performs small amounts of storage input and output. Using etcd with storage that handles small read/write operations quickly, such as SSD, is highly recommended.

Optimize communication between etcd and master hosts either by co-locating them on the same host, or providing a dedicated network.

etcd hosts are typically memory intensive.

Scaling Hosts Using the Tuned Profile

Tuned is a tuning profile delivery mechanism enabled by default in Red Hat Enterprise Linux and other Red Hat products. Tuned customizes Linux settings, such as sysctls, power management, and kernel command line options, to optimize the operating system for different workload performance and scalability requirements.

OpenShift Origin leverages the tuned daemon and includes Tuned profiles called atomic-openshift-host and atomic-openshift-guest. These profiles safely increase some of the commonly encountered vertical scaling limits present in the kernel, and are automatically applied to your system during installation.

The Tuned profiles support inheritance between profiles. On an OpenShift Origin system, the findings delivered by Tuned will be the union of throughput-performance (the default for RHEL) and atomic-openshift-guest. Tuned will determine if you are running OpenShift Origin on a virtual machine, and, if so, automatically apply virtual-guest tuning as well.

To see which Tuned profile is enabled on your system, run:

# tuned-adm active
Current active profile: atomic-openshift-node-guest

See the Red Hat Enterprise Linux Performance Tuning Guide for more information about Tuned.