$ oc get quota -n demoproject NAME AGE besteffort 11m compute-resources 2m core-object-counts 29m
Using quotas and limit ranges, cluster administrators can set constraints to limit the number of objects or amount of compute resources that are used in your project. This helps cluster administrators better manage and allocate resources across all projects, and ensure that no projects are using more than is appropriate for the cluster size.
As a developer, you can also set requests and limits on compute resources at the pod and container level.
The following sections help you understand how to check on your quota and limit range settings, what sorts of things they can constrain, and how you can request or limit compute resources in your own pods and containers.
A resource quota, defined by a ResourceQuota object, provides constraints
that limit aggregate resource consumption per project. It can limit the quantity
of objects that can be created in a project by type, as well as the total amount
of compute resources and storage that may be consumed by resources in that project.
|
Quotas are set by cluster administrators and are scoped to a given project. |
You can view usage statistics related to any hard limits defined in a project’s quota by navigating in the web console to the project’s Settings tab.
You can also use the CLI to view quota details:
First, get the list of quotas defined in the project. For example, for a project called demoproject:
$ oc get quota -n demoproject NAME AGE besteffort 11m compute-resources 2m core-object-counts 29m
Then, describe the quota you are interested in, for example the core-object-counts quota:
$ oc describe quota core-object-counts -n demoproject Name: core-object-counts Namespace: demoproject Resource Used Hard -------- ---- ---- configmaps 3 10 persistentvolumeclaims 0 4 replicationcontrollers 3 20 secrets 9 10 services 2 10
Full quota definitions can be viewed by running oc export on the object. The
following show some sample quota definitions:
apiVersion: v1
kind: ResourceQuota
metadata:
name: core-object-counts
spec:
hard:
configmaps: "10" (1)
persistentvolumeclaims: "4" (2)
replicationcontrollers: "20" (3)
secrets: "10" (4)
services: "10" (5)| 1 | The total number of ConfigMap objects that can exist in the project. |
| 2 | The total number of persistent volume claims (PVCs) that can exist in the project. |
| 3 | The total number of replication controllers that can exist in the project. |
| 4 | The total number of secrets that can exist in the project. |
| 5 | The total number of services that can exist in the project. |
apiVersion: v1
kind: ResourceQuota
metadata:
name: openshift-object-counts
spec:
hard:
openshift.io/imagestreams: "10" (1)| 1 | The total number of image streams that can exist in the project. |
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-resources
spec:
hard:
pods: "4" (1)
requests.cpu: "1" (2)
requests.memory: 1Gi (3)
limits.cpu: "2" (4)
limits.memory: 2Gi (5)| 1 | The total number of pods in a non-terminal state that can exist in the project. |
| 2 | Across all pods in a non-terminal state, the sum of CPU requests cannot exceed 1 core. |
| 3 | Across all pods in a non-terminal state, the sum of memory requests cannot exceed 1Gi. |
| 4 | Across all pods in a non-terminal state, the sum of CPU limits cannot exceed 2 cores. |
| 5 | Across all pods in a non-terminal state, the sum of memory limits cannot exceed 2Gi. |
apiVersion: v1
kind: ResourceQuota
metadata:
name: besteffort
spec:
hard:
pods: "1" (1)
scopes:
- BestEffort (2)| 1 | The total number of pods in a non-terminal state with BestEffort quality of service that can exist in the project. |
| 2 | Restricts the quota to only matching pods that have BestEffort quality of service for either memory or CPU. |
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-resources-long-running
spec:
hard:
pods: "4" (1)
limits.cpu: "4" (2)
limits.memory: "2Gi" (3)
scopes:
- NotTerminating (4)
| 1 | The total number of pods in a non-terminal state. |
| 2 | Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value. |
| 3 | Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value. |
| 4 | Restricts the quota to only matching pods where spec.activeDeadlineSeconds is nil. For example,
this quota would not charge for build or deployer pods. |
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-resources-time-bound
spec:
hard:
pods: "2" (1)
limits.cpu: "1" (2)
limits.memory: "1Gi" (3)
scopes:
- Terminating (4)
| 1 | The total number of pods in a non-terminal state. |
| 2 | Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value. |
| 3 | Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value. |
| 4 | Restricts the quota to only matching pods where spec.activeDeadlineSeconds >=0. For example,
this quota would charge for build or deployer pods, but not long running pods like a web server or database. |
apiVersion: v1
kind: ResourceQuota
metadata:
name: storage-consumption
spec:
hard:
persistentvolumeclaims: "10" (1)
requests.storage: "50Gi" (2)
| 1 | The total number of persistent volume claims in a project |
| 2 | Across all persistent volume claims in a project, the sum of storage requested cannot exceed this value. |
The following describes the set of compute resources and object types that may be managed by a quota.
|
A pod is in a terminal state if |
| Resource Name | Description |
|---|---|
|
The sum of CPU requests across all pods in a non-terminal state cannot exceed
this value. |
|
The sum of memory requests across all pods in a non-terminal state cannot
exceed this value. |
|
The sum of CPU requests across all pods in a non-terminal state cannot exceed
this value. |
|
The sum of memory requests across all pods in a non-terminal state cannot
exceed this value. |
|
The sum of storage requests across all persistent volume claims cannot
exceed this value. |
|
The sum of CPU limits across all pods in a non-terminal state cannot exceed this value. |
|
The sum of memory limits across all pods in a non-terminal state cannot exceed this value. |
|
The sum of storage limits across all persistent volume claims cannot exceed this value. |
| Resource Name | Description |
|---|---|
|
The total number of pods in a non-terminal state that can exist in the project. |
|
The total number of replication controllers that can exist in the project. |
|
The total number of resource quotas that can exist in the project. |
|
The total number of services that can exist in the project. |
|
The total number of secrets that can exist in the project. |
|
The total number of |
|
The total number of persistent volume claims that can exist in the project. |
|
The total number of image streams that can exist in the project. |
Each quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches the intersection of enumerated scopes.
Adding a scope to a quota restricts the set of resources to which that quota can apply. Specifying a resource outside of the allowed set results in a validation error.
| Scope | Description |
|---|---|
Terminating |
Match pods where |
NotTerminating |
Match pods where |
BestEffort |
Match pods that have best effort quality of service for either |
NotBestEffort |
Match pods that do not have best effort quality of service for |
A BestEffort scope restricts a quota to limiting the following resources:
pods
A Terminating, NotTerminating, or NotBestEffort scope restricts a quota to tracking the following resources:
pods
memory
requests.memory
limits.memory
cpu
requests.cpu
limits.cpu
After a resource quota for a project is first created, the project restricts the ability to create any new resources that may violate a quota constraint until it has calculated updated usage statistics.
After a quota is created and usage statistics are updated, the project accepts the creation of new content. When you create or modify resources, your quota usage is incremented immediately upon the request to create or modify the resource.
When you delete a resource, your quota use is decremented during the next full recalculation of quota statistics for the project. If project modifications exceed a quota usage limit, the server denies the action. An appropriate error message is returned explaining the quota constraint violated, and what your currently observed usage stats are in the system.
When allocating compute resources, each container may specify a request and a limit value each for CPU and memory. Quotas can restrict any of these values.
If the quota has a value specified for requests.cpu or requests.memory,
then it requires that every incoming container make an explicit request for
those resources. If the quota has a value specified for limits.cpu or
limits.memory, then it requires that every incoming container specify an
explicit limit for those resources.
See Compute Resources for more on setting requests and limits in pods and containers.
A limit range, defined by a LimitRange object, enumerates
compute resource
constraints in a project at the pod,
container, image, image stream, and persistent volume claim level, and specifies the amount of resources
that a pod, container, image, image stream, or persistent volume claim can consume.
All resource create and modification requests are evaluated against each
LimitRange object in the project. If the resource violates any of the
enumerated constraints, then the resource is rejected. If the resource does not
set an explicit value, and if the constraint supports a default value, then the
default value is applied to the resource.
|
Limit ranges are set by cluster administrators and are scoped to a given project. |
You can view any limit ranges defined in a project by navigating in the web console to the project’s Settings tab.
You can also use the CLI to view limit range details:
First, get the list of limit ranges defined in the project. For example, for a project called demoproject:
$ oc get limits -n demoproject NAME AGE resource-limits 6d
Then, describe the limit range you are interested in, for example the resource-limits limit range:
$ oc describe limits resource-limits -n demoproject Name: resource-limits Namespace: demoproject Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Pod cpu 200m 2 - - - Pod memory 6Mi 1Gi - - - Container cpu 100m 2 200m 300m 10 Container memory 4Mi 1Gi 100Mi 200Mi - openshift.io/Image storage - 1Gi - - - openshift.io/ImageStream openshift.io/image - 12 - - - openshift.io/ImageStream openshift.io/image-tags - 10 - - -
Full limit range definitions can be viewed by running oc export on the object.
The following shows an example limit range definition:
apiVersion: "v1"
kind: "LimitRange"
metadata:
name: "core-resource-limits" (1)
spec:
limits:
- type: "Pod"
max:
cpu: "2" (2)
memory: "1Gi" (3)
min:
cpu: "200m" (4)
memory: "6Mi" (5)
- type: "Container"
max:
cpu: "2" (6)
memory: "1Gi" (7)
min:
cpu: "100m" (8)
memory: "4Mi" (9)
default:
cpu: "300m" (10)
memory: "200Mi" (11)
defaultRequest:
cpu: "200m" (12)
memory: "100Mi" (13)
maxLimitRequestRatio:
cpu: "10" (14)| 1 | The name of the limit range object. |
| 2 | The maximum amount of CPU that a pod can request on a node across all containers. |
| 3 | The maximum amount of memory that a pod can request on a node across all containers. |
| 4 | The minimum amount of CPU that a pod can request on a node across all containers. |
| 5 | The minimum amount of memory that a pod can request on a node across all containers. |
| 6 | The maximum amount of CPU that a single container in a pod can request. |
| 7 | The maximum amount of memory that a single container in a pod can request. |
| 8 | The minimum amount of CPU that a single container in a pod can request. |
| 9 | The minimum amount of memory that a single container in a pod can request. |
| 10 | The default amount of CPU that a container will be limited to use if not specified. |
| 11 | The default amount of memory that a container will be limited to use if not specified. |
| 12 | The default amount of CPU that a container will request to use if not specified. |
| 13 | The default amount of memory that a container will request to use if not specified. |
| 14 | The maximum amount of CPU burst that a container can make as a ratio of its limit over request. |
Supported Resources:
CPU
Memory
Supported Constraints:
Per container, the following must hold true if specified:
| Constraint | Behavior |
|---|---|
|
If the configuration defines a |
|
If the configuration defines a |
|
If a configuration defines a For example, if a container has |
Supported Defaults:
Default[resource]Defaults container.resources.limit[resource] to specified value if none.
Default Requests[resource]Defaults container.resources.requests[resource] to specified value if none.
Supported Resources:
CPU
Memory
Supported Constraints:
Across all containers in a pod, the following must hold true:
| Constraint | Enforced Behavior |
|---|---|
|
|
|
|
|
|
Each container running on a node consumes compute resources, which are measurable quantities that can be requested, allocated, and consumed.
When authoring a pod configuration file, you can optionally specify how much CPU and memory (RAM) each container needs in order to better schedule pods in the cluster and ensure satisfactory performance.
CPU is measured in units called millicores. Each node in a cluster inspects the operating system to determine the amount of CPU cores on the node, then multiplies that value by 1000 to express its total capacity. For example, if a node has 2 cores, the node’s CPU capacity would be represented as 2000m. If you wanted to use 1/10 of a single core, it would be represented as 100m.
Memory is measured in bytes. In addition, it may be used with SI suffices (E, P, T, G, M, K) or their power-of-two-equivalents (Ei, Pi, Ti, Gi, Mi, Ki).
apiVersion: v1
kind: Pod
spec:
containers:
- image: nginx
name: nginx
resources:
requests:
cpu: 100m (1)
memory: 200Mi (2)
limits:
cpu: 200m (3)
memory: 400Mi (4)| 1 | The container requests 100m cpu. |
| 2 | The container requests 200Mi memory. |
| 3 | The container limits 200m cpu. |
| 4 | The container limits 400Mi memory. |
Each container in a pod can specify the amount of CPU it requests on a node. The scheduler uses CPU requests to find a node with an appropriate fit for a container.
The CPU request represents a minimum amount of CPU that your container may consume, but if there is no contention for CPU, it can use all available CPU on the node. If there is CPU contention on the node, CPU requests provide a relative weight across all containers on the system for how much CPU time the container may use.
On the node, CPU requests map to Kernel CFS shares to enforce this behavior.
To view compute resources for a pod:
$ oc describe pod nginx-tfjxt
Name: nginx-tfjxt
Namespace: default
Image(s): nginx
Node: /
Labels: run=nginx
Status: Pending
Reason:
Message:
IP:
Replication Controllers: nginx (1/1 replicas created)
Containers:
nginx:
Container ID:
Image: nginx
Image ID:
QoS Tier:
cpu: Burstable
memory: Burstable
Limits:
cpu: 200m
memory: 400Mi
Requests:
cpu: 100m
memory: 200Mi
State: Waiting
Ready: False
Restart Count: 0
Environment Variables:
Each container in a pod can specify the amount of CPU it is limited to use on a node. CPU limits control the maximum amount of CPU that your container may use independent of contention on the node. If a container attempts to exceed the specified limit, the system will throttle the container. This allows the container to have a consistent level of service independent of the number of pods scheduled to the node.
By default, a container is able to consume as much memory on the node as possible. In order to improve placement of pods in the cluster, specify the amount of memory required for a container to run. The scheduler will then take available node memory capacity into account prior to binding your pod to a node. A container is still able to consume as much memory on the node as possible even when specifying a request.
If you specify a memory limit, you can constrain the amount of memory the container can use. For example, if you specify a limit of 200Mi, a container will be limited to using that amount of memory on the node. If the container exceeds the specified memory limit, it will be terminated and potentially restarted dependent upon the container restart policy.
A compute resource is classified with a quality of service (QoS) based on the specified request and limit value.
| Quality of Service | Description |
|---|---|
BestEffort |
Provided when a request and limit are not specified. |
Burstable |
Provided when a request is specified that is less than an optionally specified limit. |
Guaranteed |
Provided when a limit is specified that is equal to an optionally specified request. |
A container may have a different quality of service for each compute resource. For example, a container can have Burstable CPU and Guaranteed memory qualities of service.
The quality of service has different impacts on different resources, depending on whether the resource is compressible or not. CPU is a compressible resource, whereas memory is an incompressible resource.
A BestEffort CPU container is able to consume as much CPU as is available on a node but runs with the lowest priority.
A Burstable CPU container is guaranteed to get the minimum amount of CPU requested, but it may or may not get additional CPU time. Excess CPU resources are distributed based on the amount requested across all containers on the node.
A Guaranteed CPU container is guaranteed to get the amount requested and no more, even if there are additional CPU cycles available. This provides a consistent level of performance independent of other activity on the node.
A BestEffort memory container is able to consume as much memory as is available on the node, but there are no guarantees that the scheduler will place that container on a node with enough memory to meet its needs. In addition, a BestEffort container has the greatest chance of being killed if there is an out of memory event on the node.
A Burstable memory container is scheduled on the node to get the amount of memory requested, but it may consume more. If there is an out of memory event on the node, Burstable containers are killed after BestEffort containers when attempting to recover memory.
A Guaranteed memory container gets the amount of memory requested, but no more. In the event of an out of memory event, it will only be killed if there are no more BestEffort or Burstable containers on the system.
To specify compute resources via the CLI:
$ oc run nginx --image=nginx --limits=cpu=200m,memory=400Mi --requests=cpu=100m,memory=200Mi
Opaque integer resources allow cluster operators to advertise new node-level resources that would be otherwise unknown to the system. Users can consume these resources in pod specifications, just like CPU and memory. The scheduler takes care of the resource accounting so that no more than the available amount is simultaneously allocated to pods.
|
Opaque integer resources are Alpha currently, and only resource accounting is implemented. There is no resource quota or limit range support for these resources, and they have no impact on QoS. |
Opaque integer resources are resources that begin with the prefix
pod.alpha.kubernetes.io/opaque-int-resource-. The API server
restricts quantities of these resources to whole numbers. Examples of
valid quantities are 3, 3000m and 3Ki. Examples of invalid
quantities are 0.5 and 1500m.
There are two steps required to use opaque integer resources. First, the cluster operator must advertise a per-node opaque resource on one or more nodes. Second, users must request the opaque resource in pods.
To advertise a new opaque integer resource, the cluster operator should
submit a PATCH HTTP request to the API server to specify the available
quantity in the status.capacity for a node in the cluster. After this
operation, the node’s status.capacity will include a new resource. The
status.allocatable field is updated automatically with the new resource
asynchronously by the kubelet.
|
Since the scheduler uses the node |
The HTTP request below advertises five foo resources on node
openshift-node-1. Note that ~1 is the encoding for the character / in the
patch path. The operation path value in the JSON-Patch is interpreted as a
JSON-Pointer. For more details, refer to
IETF RFC 6901, section 3.
PATCH /api/v1/nodes/openshift-node-1/status HTTP/1.1
Accept: application/json
Content-Type: application/json-patch+json
Host: openshift-master:8080
[
{
"op": "add",
"path": "/status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-foo",
"value": "5"
}
]To consume opaque resources in pods, include the name of the opaque resource as
a key in the the spec.containers[].resources.requests map.
The pod will be scheduled only if all of the resource requests are satisfied
(including CPU, memory, and any opaque resources). The pod will remain in the
PENDING state while the resource request cannot be met by any node.
Example
The pod below requests two CPUs and one foo (an opaque resource).
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
containers:
- name: my-container
image: myimage
resources:
requests:
cpu: 2
pod.alpha.kubernetes.io/opaque-int-resource-foo: 1Resource limits can be set per-project by cluster administrators. Developers do not have the ability to create, edit, or delete these limits, but can view them for projects they have access to.