AUDIT: id="5c3b8227-4af9-4322-8a71-542231c3887b" ip="127.0.0.1" method="GET" user="admin" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/pods" AUDIT: id="5c3b8227-4af9-4322-8a71-542231c3887b" response="200"
The openshift start command is used to launch OpenShift Origin servers. The command
and its subcommands (master to launch a
master
server and node to launch a
node
server) all take a limited set of arguments that are sufficient for launching
servers in a development or experimental environment.
However, these arguments are insufficient to describe and control the full set of configuration and security options that are necessary in a production environment. To provide those options, it is necessary to use the dedicated master and node configuration files.
Master configuration files and
node configuration files are fully specified
with no default values. Therefore, any empty value indicates that you want to
start up with an empty value for that parameter. This makes it easy to reason
about exactly what your configuration is, but it also makes it difficult to
remember all of the options to specify. To make this easier, the configuration
files can be created with the --write-config option and then used with the
--config option.
This section reviews parameters mentioned in the master-config.yaml file.
You can create a new master configuration file to see the valid options for your installed version of OpenShift Origin.
| Parameter Name | Description |
|---|---|
|
Contains admission control plug-in configuration. |
|
Key-value pairs that will be passed directly to the Kube API server that match
the API servers' command line arguments. These are not migrated, but if you
reference a value that does not exist the server will not start. These values
may override other settings in |
|
Key-value pairs that will be passed directly to the Kube controller manager
that match the controller manager’s command line arguments. These are not
migrated, but if you reference a value that does not exist the server will not
start. These values may override other settings in |
|
Used to enable or disable various admission plug-ins. When this type is present
as the configuration object under |
|
Allows specifying a configuration file per admission control plug-in. |
|
A list of admission control plug-in names that will be installed on the master. Order is significant. If empty, a default list of plug-ins is used. |
|
Key-value pairs that will be passed directly to the Kube scheduler that match
the scheduler’s command line arguments. These are not migrated, but if you
reference a value that does not exist the server will not start. These values
may override other settings in |
| Parameter Name | Description |
|---|---|
|
Holds the necessary configuration options for serving assets. |
|
A list of features that should not be started. You will likely want to set this as null. It is very unlikely that anyone will want to manually disable features and that is not encouraged. |
|
Files to serve from the asset server file system under a subcontext. |
|
When set to true, tells the asset server to reload extension scripts and stylesheets for every request rather than only at startup. It lets you develop extensions without having to restart the server for every change. |
|
Key- (string) and value- (string) pairs that will be injected into the console under
the global variable |
|
File paths on the asset server files to load as scripts when the web console loads. |
|
File paths on the asset server files to load as style sheets when the web console loads. |
|
The public endpoint for logging (optional). |
|
An optional, absolute URL to redirect web browsers to after logging out of the web console. If not specified, the built-in logout page is shown. |
|
How the web console can access the OpenShift Origin server. |
|
The public endpoint for metrics (optional). |
|
URL of the the asset server. |
| Parameter Name | Description |
|---|---|
|
Holds authentication and authorization configuration options. |
|
Indicates how many authentication results should be cached. If 0, the default cache size is used. |
|
Indicates how long an authorization result should be cached. It takes a valid time duration string (e.g. "5m"). If empty, you get the default timeout. If zero (e.g. "0m"), caching is disabled. |
| Parameter Name | Description |
|---|---|
|
List of the controllers that should be started. If set to none, no
controllers will start automatically. The default value is * which will start
all controllers. When using *, you may exclude controllers by prepending a |
|
Enables controller election, instructing the master to attempt to acquire a
lease before controllers start and renewing it within a number of seconds
defined by this value. Setting this value non-negative forces
|
|
Instructs the master to not automatically start controllers, but instead to wait until a notification to the server is received before launching them. |
| Parameter Name | Description |
|---|---|
|
The advertised host:port for client connections to etcd. |
|
Contains information about how to connect to etcd. |
|
Holds the necessary configuration options for connecting with an etcd database. |
|
Contains information about how API resources are stored in etcd. These values are only relevant when etcd is the backing store for the cluster. |
|
The path within etcd that the Kubernetes resources will be rooted under. This value, if changed, will mean existing objects in etcd will no longer be located. The default value is kubernetes.io. |
|
The API version that Kubernetes resources in etcd should be serialized to. This value should not be advanced until all clients in the cluster that read from etcd have code that allows them to read the new version. |
|
The path within etcd that the OpenShift Origin resources will be rooted under. This value, if changed, will mean existing objects in etcd will no longer be located. The default value is openshift.io. |
|
API version that OS resources in etcd should be serialized to. This value should not be advanced until all clients in the cluster that read from etcd have code that allows them to read the new version. |
|
The advertised host:port for peer connections to etcd. |
|
Describes how to start serving the etcd peer. |
|
Describes how to start serving the etcd master. |
|
The path to the etcd storage directory. |
| Parameter Name | Description |
|---|---|
|
Describes how to handle grants. |
|
Auto-approves client authorization grant requests. |
|
Auto-denies client authorization grant requests. |
|
Prompts the user to approve new client authorization grant requests. |
|
Determines the default strategy to use when an OAuth client requests a grant.This method will be used only if the specific OAuth client does not provide a strategy of their own. Valid grant handling methods are:
|
| Parameter Name | Description |
|---|---|
|
Allows scheduled background import of images to be disabled. |
|
The format of the name to be built for the system component. |
|
Holds options that describe how to build image names for system components. |
|
Controls limits and behavior for importing images. |
|
Determines if the latest tag will be pulled from the registry. |
|
Controls the number of images that are imported when a user does a bulk import of a Docker repository. This number defaults to 5 to prevent users from importing large numbers of images accidentally. Set -1 for no limit. |
|
The maximum number of scheduled image streams that will be imported in the background per minute. The default value is 60. |
|
The minimum number of seconds that can elapse between when image streams scheduled for background import are checked against the upstream repository. The default value is 15 minutes. |
| Parameter Name | Description |
|---|---|
|
A list of API levels that should be enabled on startup, v1 as examples. |
|
A map of groups to the versions (or |
|
Contains information about how to connect to kubelets. |
|
Holds the necessary configuration options for the Kubernetes master. |
|
The number of expected masters that should be running. This value defaults to 1 and may be set to a positive integer, or if set to -1, indicates this is part of a cluster. |
|
The public IP address of Kubernetes resources. If empty, the first result from
|
|
File name for the .kubeconfig file that describes how to connect this node to the master. |
|
The range to use for assigning service public ports on a host. |
|
The subnet to use for assigning service IPs. |
|
The list of nodes that are statically known. |
| Parameter Name | Description |
|---|---|
|
The CIDR string to specify the global overlay network’s L3 space. |
|
Controls what values are acceptable for the service external IP field. If
empty, no |
|
The number of bits to allocate to each host’s subnet. For example, 8 would mean a /24 network on the host. |
|
Controls the range to assign ingress IPs from for services of type LoadBalancer on bare metal. If empty, ingress IPs will not be assigned. It may contain a single CIDR that will be allocated from. For security reasons, you should ensure that this range does not overlap with the CIDRs reserved for external IPs, nodes, pods, or services. |
|
Provides network options for the node. |
|
The name of the network plug-in to use. |
|
The CIDR string to specify the service networks. |
| Parameter Name | Description |
|---|---|
|
Forces the provider selection page to render even when there is only a single provider. |
|
Used for building valid client redirect URLs for external access. |
|
A path to a file containing a go template used to render error pages during the authentication or grant flow If unspecified, the default error page is used. |
|
Ordered list of ways for a user to identify themselves. |
|
A path to a file containing a go template used to render the login page. If unspecified, the default login page is used. |
|
CA for verifying the TLS connection back to the |
|
Used for building valid client redirect URLs for external access. |
|
Used for making server-to-server calls to exchange authorization codes for access tokens. |
|
Holds the necessary configuration options for OAuth authentication. |
|
Allows for customization of pages like the login page. |
|
A path to a file containing a go template used to render the provider selection page. If unspecified, the default provider selection page is used. |
|
Holds information about configuring sessions. |
|
Allows you to customize pages like the login page. |
|
Contains options for authorization and access tokens. |
| Parameter Name | Description |
|---|---|
|
Holds default project node label selector. |
|
Holds information about project creation and defaults. |
|
The string presented to a user if they are unable to request a project via the project request API endpoint. |
|
The template to use for creating projects in response to projectrequest. It is in the format namespace/template and it is optional. If it is not specified, a default template is used. |
| Parameter Name | Description |
|---|---|
|
Points to a file that describes how to set up the scheduler. If empty, you get the default scheduling rules |
| Parameter Name | Description |
|---|---|
|
Defines the range of MCS categories that will be assigned to namespaces. The
format is |
|
Controls the automatic allocation of UIDs and MCS labels to a project. If nil, allocation is disabled. |
|
Defines the total set of Unix user IDs (UIDs) that will be allocated to projects automatically, and the size of the block each namespace gets. For example, 1000-1999/10 will allocate ten UIDs per namespace, and will be able to allocate up to 100 blocks before running out of space. The default is to allocate from 1 billion to 2 billion in 10k blocks (which is the expected size of the ranges container images will use once user namespaces are started). |
| Parameter Name | Description |
|---|---|
|
Controls whether or not to allow a service account to reference any secret in a namespace without explicitly referencing them. |
|
A list of service account names that will be auto-created in every namespace.
If no names are specified, the |
|
The CA for verifying the TLS connection back to the master. The service account controller will automatically inject the contents of this file into pods so they can verify connections to the master. |
|
A file containing a PEM-encoded private RSA key, used to sign service account
tokens. If no private key is specified, the service account |
|
A list of files, each containing a PEM-encoded public RSA key. If any file contains a private key, the public portion of the key is used. The list of public keys is used to verify presented service account tokens. Each key is tried in order until the list is exhausted or verification succeeds. If no keys are specified, no service account authentication will be available. |
|
Holds the necessary configuration options for a service account. |
| Parameter Name | Description |
|---|---|
|
Allows the DNS server on the master to answer queries recursively. Note that open resolvers can be used for DNS amplification attacks and the master DNS should not be made accessible to public networks. |
|
The ip:port to serve on. |
|
Controls limits and behavior for importing images. |
|
A file containing a PEM-encoded certificate. |
|
TLS cert information for serving secure traffic. |
|
The certificate bundle for all the signers that you recognize for incoming client certificates. |
|
Holds the necessary configuration options for DNS. |
|
Holds the domain suffix. |
|
Holds the IP. |
|
A file containing a PEM-encoded private key for the certificate specified by
|
|
Provides overrides to the client connection used to connect to the master. |
|
The number of concurrent requests allowed to the server. If zero, no limit. |
|
A list of certificates to use to secure requests to specific host names. |
|
The number of seconds before requests are timed out. The default is 60 minutes. If -1, there is no limit on requests. |
|
The HTTP serving information for the assets. |
| Parameter Name | Description |
|---|---|
|
A boolean to enable or disable dynamic provisioning. Default is true. |
FSGroup |
Can be specified to enable a quota on local storage use per unique FSGroup ID.
At present this is only implemented for emptyDir volumes, and if the underlying
|
|
Contains options for controlling local volume quota on the node. |
|
Contains options for configuring volume plug-ins in the master node. |
|
Contains options for configuring volumes on the node. |
|
Contains options for configuring volumes on the node. |
|
The directory that volumes are stored under. |
Audit provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators, or other components of the system.
Audit works at the API server level, logging all requests coming to the server. Each audit log contains two entries:
The request line containing:
A Unique ID allowing to match the response line (see #2)
The source IP of the request
The HTTP method being invoked
The original user invoking the operation
The impersonated user for the operation (self meaning himself)
The impersonated group for the operation (lookup meaning user’s group)
The namespace of the request or <none>
The URI as requested
The response line containing:
The the unique ID from #1
The response code
Example output for user admin asking for a list of pods:
AUDIT: id="5c3b8227-4af9-4322-8a71-542231c3887b" ip="127.0.0.1" method="GET" user="admin" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/pods" AUDIT: id="5c3b8227-4af9-4322-8a71-542231c3887b" response="200"
| Parameter Name | Description |
|---|---|
|
A boolean to enable or disable audit logs. Default is |
|
File path where the requests should be logged to. If not set, logs are printed to master logs. |
|
Specifies maximum number of days to retain old audit log files based on the time stamp encoded in their filename. |
|
Specifies the maximum number of old audit log files to retain. |
|
Specifies maximum size in megabytes of the log file before it gets rotated. Defaults to 100MB. |
The following node-config.yaml file is a sample node configuration file that was generated with the default values as of writing. You can create a new node configuration file to see the valid options for your installed version of OpenShift Origin.
allowDisabledDocker: false
apiVersion: v1
authConfig:
authenticationCacheSize: 1000
authenticationCacheTTL: 5m
authorizationCacheSize: 1000
authorizationCacheTTL: 5m
dnsDomain: cluster.local
dnsIP: 10.0.2.15 (1)
dockerConfig:
execHandlerName: native
imageConfig:
format: openshift/origin-${component}:${version}
latest: false
iptablesSyncPeriod: 5s
kind: NodeConfig
masterKubeConfig: node.kubeconfig
networkConfig:
mtu: 1450
networkPluginName: ""
nodeIP: ""
nodeName: node1.example.com
podManifestConfig: (2)
path: "/path/to/pod-manifest-file" (3)
fileCheckIntervalSeconds: 30 (4)
proxyArguments:
proxy-mode:
- iptables (5)
volumeConfig:
localQuota:
perFSGroup: null(6)
servingInfo:
bindAddress: 0.0.0.0:10250
bindNetwork: tcp4
certFile: server.crt
clientCA: node-client-ca.crt
keyFile: server.key
namedCertificates: null
volumeDirectory: /root/openshift.local.volumes| 1 | Configures an IP address to be prepended to a pod’s /etc/resolv.conf by adding the address here. |
| 2 | Allows pods to be placed directly on certain set of nodes, or on all nodes without going through the scheduler. You can then use pods to perform the same administrative tasks and support the same services on each node. |
| 3 | Specifies the path for the pod manifest file or directory. If it is a directory, then it is expected to contain one or more manifest files. This is used by the Kubelet to create pods on the node. |
| 4 | This is the interval (in seconds) for checking the manifest file for new data. The interval must be a positive value. |
| 5 | The service proxy implementation to use. |
| 6 | Preliminary support for local emptyDir volume quotas, set this value to a resource quantity representing the desired quota per FSGroup, per node. (i.e. 1Gi, 512Mi, etc) Currently requires that the volumeDirectory be on an XFS filesystem mounted with the 'gquota' option, and the matching security context contraint’s fsGroup type set to 'MustRunAs'. |
| Parameter Name | Description |
|---|---|
|
The fully specified configuration starting an OpenShift Origin node. |
|
Node may have multiple IPs, so this specifies the IP to use for pod traffic routing. If not specified, network parse/lookup on the nodeName is performed and the first non-loopback address is used. |
|
The value used to identify this particular node in the cluster. If possible, this should be your fully qualified hostname. If you are describing a set of static nodes to the master, this value must match one of the values in the list. |
|
Controls grace period for deleting pods on failed nodes. It takes valid time duration string. If empty, you get the default pod eviction timeout. |
|
Specifies the client cert/key to use when proxying to pods. |
| Parameter Name | Description |
|---|---|
|
If true, the kubelet will ignore errors from Docker. This means that a node can start on a machine that does not have docker started. |
|
Holds Docker related configuration options |
|
The handler to use for executing commands in Docker containers. |
If you are using Docker 1.9+, you may want to consider enabling parallel image pulling, as the default is to pull images one at a time.
|
There is a potential issue with data corruption prior to Docker 1.9. However, starting with 1.9, the corruption issue is resolved and it is safe to switch to parallel pulls. |
kubeletArguments:
serialize-image-pulls:
- "false" (1)| 1 | Change to true to disable parallel pulls. (This is the default config) |
For some authentication configurations,
an LDAP bindPassword or OAuth clientSecret value is required.
Instead of specifying these values directly in the master configuration file,
these values may be provided as environment variables, external files,
or in encrypted files.
...
bindPassword:
env: BIND_PASSWORD_ENV_VAR_NAME ...
bindPassword:
file: bindPassword.txt ...
bindPassword:
file: bindPassword.encrypted
keyFile: bindPassword.keyTo create the encrypted file and key file for the above example:
$ oadm ca encrypt --genkey=bindPassword.key --out=bindPassword.encrypted > Data to encrypt: B1ndPass0rd!
|
Encrypted data is only as secure as the decrypting key. Care should be taken to limit filesystem permissions and access to the key file. |
When defining an OpenShift Origin configuration from scratch, start by creating new configuration files.
For master host configuration files, use the openshift start command with the
--write-config option to write the configuration files. For node hosts, use
the oadm create-node-config command to write the configuration files.
The following commands write the relevant launch configuration file(s),
certificate files, and any other necessary files to the specified
--write-config or --node-dir directory.
Generated certificate files are valid for two years, while the certification
authority (CA) certificate is valid for five years. This can be altered with the
--expire-days and --signer-expire-days options, but for security reasons, it
is recommended to not make them greater than these values.
To create configuration files for an all-in-one server (a master and a node on the same host) in the specified directory:
$ openshift start --write-config=/openshift.local.config
To create a master configuration file and other required files in the specified directory:
$ openshift start master --write-config=/openshift.local.config/master
To create a node configuration file and other related files in the specified directory:
$ oadm create-node-config \
--node-dir=/openshift.local.config/node-<node_hostname> \
--node=<node_hostname> \
--hostnames=<node_hostname>,<ip_address> \
--certificate-authority="/path/to/ca.crt" \
--signer-cert="/path/to/ca.crt" \
--signer-key="/path/to/ca.key"
--signer-serial="/path/to/ca.serial.txt"
--node-client-certificate-authority="/path/to/ca.crt"
When creating node configuration files, the --hostnames option accepts a
comma-delimited list of every host name or IP address you want server
certificates to be valid for.
Once you have modified the master and/or node configuration files to your specifications, you can use them when launching servers by specifying them as an argument. Keep in mind that if you specify a configuration file, none of the other command line options you pass are respected.
To launch an all-in-one server using a master configuration and a node configuration file:
$ openshift start --master-config=/openshift.local.config/master/master-config.yaml --node-config=/openshift.local.config/node-<node_hostname>/node-config.yaml
To launch a master server using a master configuration file:
$ openshift start master --config=/openshift.local.config/master/master-config.yaml
To launch a node server using a node configuration file:
$ openshift start node --config=/openshift.local.config/node-<node_hostname>/node-config.yaml
OpenShift Origin uses the systemd-journald.service to collect log messages for debugging, using five log message severities. The logging levels are based on Kubernetes logging conventions, as follows:
| Option | Description |
|---|---|
0 |
Errors and warnings only |
2 |
Normal information |
4 |
Debugging-level information |
6 |
API-level debugging information (request / response) |
8 |
Body-level API debugging information |
You can control which INFO messages are logged by setting the loglevel option in the in /etc/sysconfig/atomic-openshift-node or /etc/sysconfig/atomic-openshift-master file. Configuring the logs to collect all messages can lead to large logs that are difficult to interpret and can take up excessive space. Collecting all messages should only be used in debug situations.
|
Messages with FATAL, ERROR, WARNING and some INFO severities appear in the logs regardless of the log configuration. |
You can view logs for the master or the node system using the following command:
# journalctl -r -u <journal_name>
Use the -r option to show the newest entries first.
For example:
# journalctl -r -u atomic-openshift-master.service # journalctl -r -u atomic-openshift-node.service
To change the logging level:
Edit the /etc/sysconfig/atomic-openshift-master file for the master or /etc/sysconfig/atomic-openshift-node file for the nodes.
Enter a value from the Log Level Options table above in the OPTIONS=--loglevel= field.
For example:
OPTIONS=--loglevel=4
Restart the master or node host as appropriate:
# systemctl restart atomic-openshift-master # systemctl restart atomic-openshift-node
After the restart, all new log messages will conform to the new setting. Older messages do not change.
|
The default log level can be set using the Advanced Install. For more information, see Cluster Variables. |
The following examples are excerpts from a master journald log at various log levels. Timestamps and system information have been removed from these examples.
fit failure on node: PodFitsHostPorts
E0222 factory.go:361] Error scheduling default router-1
I0222 event.go:211] Event(api.ObjectReference
I0222 start_master.go:644] Started Origin Controllers
I0222 start_master.go:623] Started Kubernetes Controllers
I0222 endpoints_controller.go:283] Waiting for pods controller
I0222 event.go:211] Event(api.ObjectReference{Kind:"Node"
W0222 nodecontroller.go:671] Missing timestamp for Node
W0222 nodecontroller.go:671] Missing timestamp for Node
E0222 factory.go:361] Error scheduling default router-1
Always TerminationGracePeriodSeconds:0xc20d819b78 ActiveDeadlineSeconds:<nil>
I0222 scheduler.go:117] Failed to schedule: &{TypeMeta:
I0222 event.go:211] Event(api.ObjectReference{Kind:
I0222 replication_controller.go:434] Too few "default"/"router"
I0222 start_master.go:644] Started Origin Controllers
I0222 subnets.go:27] Found existing HostSubnet
I0222 subnets.go:27] Found existing HostSubnet
I0222 common.go:79] Initializing single-tenant plugin
I0222 common.go:54] Starting with configured hostname
I0222 start_master.go:623] Started Kubernetes Controllers
I0222 plugins.go:291] Loaded volume plugin "kubernetes.io/nf"
I0222 plugins.go:291] Loaded volume plugin "kubernetes.io/ho"
I0222 endpoints_controller.go:283] Waiting for pods controller"
I0222 endpoints_controller.go:283] Waiting for pods controller"
W0222 nodecontroller.go:671] Missing timestamp for Node
I0222 nodecontroller.go:604] Recording Registered Node
I0222 nodecontroller.go:416] NodeController observed
controller_utils.go:592] Ignoring inactive pod default/router-1-0ww1g in state Failed, deletion time <nil>
I0222 replication_controller.go:497] Finished syncing controller "default/docker-registry-2" (206.507µs)
I0222 controller_utils.go:592] Ignoring inactive pod default/router-1-zyi9y in state Failed, deletion time <nil>
I0222 replication_controller.go:497] Finished syncing controller "default/docker-registry-1" (176.748µs)
I0222 controller_utils.go:160] Controller default/router-1 either never recorded expectations, or the ttl expired.
I0222 controller_utils.go:160] Controller default/docker-registry-2 either never recorded expectations, or the ttl expired.
I0222 nodecontroller.go:709] Node dell-r430-20.gsslab.pnq.redhat.com ReadyCondition updated. Updating timestamp.
I0222 factory.go:474] Backing off 1m0s for pod &{backoff:60000000000 lastUpdate:{sec:636233 loc:0x59a6560} reqInFlight:1}
; retrying
fit failure on node (ibm-x3650m4-01-vm-02.lab.eng.bos.redhat.com): PodFitsHostPorts
E0222 factory.go:361] Error scheduling default router-1-d7svd: pod (router-1-d7svd) failed to fit in any node
b089d8 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[region:infra] ServiceAccountName:router NodeName: SecurityContext:0xc20ecd00c0 ImagePullSec
]} Spec:{Volumes:[{Name:router-token-h6q3n VolumeSource:{HostPath:<nil> EmptyDir:<nil> GCEPersistentDisk:<nil> AWSElasticBlockStore:<nil> GitRepo:<nil> Secret:0xc20eb
I0222 factory.go:474] Backing off 16s for pod &{backoff:32000000000 lastUpdate:{sec:63623353879 nsec:506358920 loc:0x59a6560} reqInFlight:1}
b118b8 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[region:infra] ServiceAccountName:router NodeName:
I0222 round_trippers.go:271] Request Headers:
I0222 round_trippers.go:264] GET https://dell-r430-20.gsslab
I0222 round_trippers.go:274] User-Agent: openshift/v3.2.
I0222 round_trippers.go:274] Accept: application/json, *
I0222 round_trippers.go:271] Request Headers:
I0222 round_trippers.go:264] GET https://dell-r430-20.gsslab
I0222 request.go:870] Response Body: {"kind":"ClusterRole","
I0222 round_trippers.go:295] Content-Type: application/j
I0222 round_trippers.go:295] Cache-Control: no-store
I0222 round_trippers.go:295] Content-Length: 898
I0222 round_trippers.go:295] Date: Wed, 22 Feb 2017 09:3
I0222 round_trippers.go:292] Response Headers:
I0222 round_trippers.go:289] Response Status: 200 OK in 7 mi
I0222 configgetter.go:127] using watch cache storage (capacity=1000)
I0222 controller_utils.go:592] Ignoring inactive pod