Primeros pasos con Kubernetes (V): Health Check
En Kubernetes hay dos conceptos sobre Health Check:
- Readiness Probe
- Liveness Probe
Las Readiness Probe son pruebas que realizamos sobre un Pod para certificar que se encuentra listo para ser usado y su ciclo de vida finaliza una veza se encuentra listo.
Las Liveness Probe son pruebas que realizamos periódicamente sobre un Pod para certificar que todavía sigue funcionando correctamente y su ciclo de vida dura hasta que se elimina el Pod.
Estas pruebas de Health Check las configuramos en el YAML de Deployment.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: hello
spec:
replicas: 1
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello
image: gcr.io/k8slabs-142707/hello:v1
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /
port: 8081
periodSeconds: 2
timeoutSeconds: 1
successThreshold: 2
initialDelaySeconds: 1
livenessProbe:
httpGet:
path: /
port: 8081
timeoutSeconds: 1
initialDelaySeconds: 10
Como podemos ver marcado en rojo, hemos configurado las dos pruebas.
La prueba de Readiness hemos indicado que se realiza una validación contra la URL / por el puerto 8081 y que daremos el Pod por Ready tras validar 2 veces correctamente (successThreshold) en un periodo de 2 segundos por validación (periodSeconds) que se iniciará tras 5 segundos de haberse arrancado (initialDelaySeconds) con un timeout de validación de 1 segundo antes de considerar error (timeoutSeconds).
La prueba de Liveness se ejecuta también sobre la raíz / sobre el puerto 8081 y se lanza validaciones cada 2 segundos a partir del segundo 20 de haberse arrancado.
Una vez preparado el fichero con pruebas que fallaran (recordemos que exponemos el 8080 y no el 8081), procedemos a realizar el despliegue.
$ kubectl create -f ./apps/hello/deployment.yml
deployment "hello" created
$ kubectl describe pods hello
Name: hello-3633437581-12q25
Namespace: default
Node: gke-bs-default-pool-bbee06a5-ntpd/10.132.0.6
Start Time: Mon, 19 Sep 2016 13:40:39 +0000
Labels: app=hello
pod-template-hash=3633437581
Status: Running
IP: 10.64.0.3
Controllers: ReplicaSet/hello-3633437581
Containers:
hello:
Container ID: docker://f0913a1c2e5f032f4ee790786db2cf68d1e4e6740b60457e17a7ee35f5a04036
Image: gcr.io/k8slabs-142707/hello:v1
Image ID: docker://sha256:2b405bb918b39b5e63b3a76e1abf429fb44edd65b601bf0ead6eccd4d85577dc
Port: 8080/TCP
Requests:
cpu: 100m
State: Running
Started: Mon, 19 Sep 2016 13:40:39 +0000
Ready: False
Restart Count: 0
Liveness: http-get http://:8081/ delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:8081/ delay=1s timeout=1s period=2s #success=2 #failure=3
Environment Variables:
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-64ex2:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-64ex2
QoS Tier: Burstable
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
14s 14s 1 {default-scheduler } Normal Scheduled Successfully assigned hello-3633437581-12q25 to gke-bs-default-pool-bbee06a5-ntpd
14s 14s 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Pulled Container image "gcr.io/k8slabs-142707/hello:v1" already present on machine
14s 14s 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Created Created container with docker id f0913a1c2e5f
14s 14s 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Started Started container with docker id f0913a1c2e5f
4s 4s 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Warning Unhealthy Liveness probe failed: Get http://10.64.0.3:8081/: dial tcp 10.64.0.3:8081: getsockopt: connection refused
12s 2s 7 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Warning Unhealthy Readiness probe failed: Get http://10.64.0.3:8081/: dial tcp 10.64.0.3:8081: getsockopt: connection refused
$ kubectl get pods –w
NAME READY STATUS RESTARTS AGE
hello-3633437581-12q25 0/1 Running 1 1m
NAME READY STATUS RESTARTS AGE
hello-3633437581-12q25 0/1 Running 2 1m
--> HACEMOS UN CONTROL+C PARA CANCELAR LA VIGILANCIA ACTIVA
$ kubectl describe pods hello
Name: hello-3633437581-12q25
...
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
2m 2m 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Created Created container with docker id f0913a1c2e5f
2m 2m 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Started Started container with docker id f0913a1c2e5f
2m 2m 1 {default-scheduler } Normal Scheduled Successfully assigned hello-3633437581-12q25 to gke-bs-default-pool-bbee06a5-ntpd
1m 1m 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Killing Killing container with docker id f0913a1c2e5f: pod "hello-3633437581-12q25_default(a6db53cc-7e6e-11e6-8f94-42010a840076)" container "hello" is unhealthy, it will be killed and re-created.
1m 1m 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Started Started container with docker id 9c23b9167f35
1m 1m 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Created Created container with docker id 9c23b9167f35
1m 1m 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Killing Killing container with docker id 9c23b9167f35: pod "hello-3633437581-12q25_default(a6db53cc-7e6e-11e6-8f94-42010a840076)" container "hello" is unhealthy, it will be killed and re-created.
1m 1m 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Created Created container with docker id 485713ad223d
1m 1m 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Started Started container with docker id 485713ad223d
37s 37s 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Killing Killing container with docker id 485713ad223d: pod "hello-3633437581-12q25_default(a6db53cc-7e6e-11e6-8f94-42010a840076)" container "hello" is unhealthy, it will be killed and re-created.
37s 37s 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Created Created container with docker id 16c82c0e5ec3
2m 37s 4 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Pulled Container image "gcr.io/k8slabs-142707/hello:v1" already present on machine
37s 37s 1 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Normal Started Started container with docker id 16c82c0e5ec3
2m 27s 6 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Warning Unhealthy Liveness probe failed: Get http://10.64.0.3:8081/: dial tcp 10.64.0.3:8081: getsockopt: connection refused
2m 1s 89 {kubelet gke-bs-default-pool-bbee06a5-ntpd} spec.containers{hello} Warning Unhealthy Readiness probe failed: Get http://10.64.0.3:8081/: dial tcp 10.64.0.3:8081: getsockopt: connection refused
Con el comando “kubectl describe pods” te muestra información detallada de los Pods, en nuestro caso del hello. En este detalle, podemos ver un listado de eventos (marcados en rojo). Podemos observar en los eventos como se han realizado varios tests fallidos de Readyness y Liveness.
Con el comando “kubectl get pods –w” se queda a la escucha activa de cualquier cambio en el sistema y tras un tiempo podemos observar como se vuelve a crear un Pod nuevo tras haber eliminado el Pod anterior por no superar las pruebas de Health Check.
Tras detectar la creación de un nuevo Pod, volvemos a llamar al “kubectl describe” y vemos como en los eventos se observan varios eventos de destrucción de Containers y creación del mismo.
Ahora que hemos visto Kubernetes reaccionando ante Containers que no cumplen el Health Check, vamos a configurarlo correctamente y vemos como trabaja Kubernetes ante Health Checks que funcionan.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: hello
spec:
replicas: 1
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello
image: gcr.io/k8slabs-142707/hello:v1
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /
port: 8080
periodSeconds: 2
timeoutSeconds: 1
successThreshold: 2
initialDelaySeconds: 1
livenessProbe:
httpGet:
path: /
port: 8080
timeoutSeconds: 1
initialDelaySeconds: 10
Ahora volvemos a desplegar.
$ kubectl delete -f ./apps/hello/deployment.yml
deployment "hello" deleted
$ kubectl create -f ./apps/hello/deployment.yml
deployment "hello" created
$ kubectl describe pods hello
Name: hello-3462388619-gsjso
Namespace: default
Node: gke-bs-default-pool-bbee06a5-myey/10.132.0.5
Start Time: Mon, 19 Sep 2016 14:14:02 +0000
Labels: app=hello
pod-template-hash=3462388619
Status: Running
IP: 10.64.2.3
Controllers: ReplicaSet/hello-3462388619
Containers:
hello:
Container ID: docker://8569cbd0b41e66438294bccbc5429c3baf1bfa8ac93a23314b3b7e616d617a54
Image: gcr.io/k8slabs-142707/hello:v1
Image ID: docker://sha256:2b405bb918b39b5e63b3a76e1abf429fb44edd65b601bf0ead6eccd4d85577dc
Port: 8080/TCP
Requests:
cpu: 100m
State: Running
Started: Mon, 19 Sep 2016 14:14:03 +0000
Ready: True
Restart Count: 0
Liveness: http-get http://:8080/ delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:8080/ delay=1s timeout=1s period=2s #success=2 #failure=3
Environment Variables:
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
default-token-64ex2:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-64ex2
QoS Tier: Burstable
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
24s 24s 1 {default-scheduler } Normal Scheduled Successfully assigned hello-3462388619-gsjso to gke-bs-default-pool-bbee06a5-myey
23s 23s 1 {kubelet gke-bs-default-pool-bbee06a5-myey} spec.containers{hello} Normal Pulled Container image "gcr.io/k8slabs-142707/hello:v1" already present on machine
23s 23s 1 {kubelet gke-bs-default-pool-bbee06a5-myey} spec.containers{hello} Normal Created Created container with docker id 8569cbd0b41e
23s 23s 1 {kubelet gke-bs-default-pool-bbee06a5-myey} spec.containers{hello} Normal Started Started container with docker id 8569cbd0b41e
21s 21s 1 {kubelet gke-bs-default-pool-bbee06a5-myey} spec.containers{hello} Warning Unhealthy Readiness probe failed: Get http://10.64.2.3:8080/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-3462388619-gsjso 1/1 Running 0 36s
Tras volver a desplegar el Pod, podemos observar como ha fallado una prueba de Readiness, pero tras 20 segundos no se ha vuelto a generar ningún evento más y ya aparece en estado Ready y Running. La prueba de Readiness que ha fallado es debido a que el delay inicial para testear si se encuentra funcionando es de 1 segundo, por lo que todavía no ha arrancado el Pod, pero a posteriori una vez arrancado ya han salido todas las pruebas correctamente.
No hay comentarios
Publicar un comentario