Stream logs of multiple pods in Kubernetes

What if there are multiple replicas of pod are running and you want to check the logs of all of them simultaneously, of course you can not do kubectl logs to get logs of all the pods That’s what, In this article we will see how we can stream logs of multiple pods in CLI –

Let’s first create a pod – 

$ kubectl create deployment pinging --image=alpine -- ping 127.0.0.1
deployment.apps/pinging created


$ kubectl get pod
NAME                   	   READY   STATUS	RESTARTS   AGE
pinging-66dff4b844-pkfxl   1/1 	   Running      0          3m31s

Now if I want to stream the logs of this pod, I can run

$ kubectl logs -f pinging-66dff4b844-pkfxl
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.117 ms
64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.147 ms
64 bytes from 127.0.0.1: seq=2 ttl=64 time=0.174 ms
64 bytes from 127.0.0.1: seq=3 ttl=64 time=0.098 ms
64 bytes from 127.0.0.1: seq=4 ttl=64 time=0.095 ms
64 bytes from 127.0.0.1: seq=5 ttl=64 time=0.116 ms
…………
……

Ok, That’s easy.
Now, Let’s increase the number of replicas of pod

$ kubectl scale deployment pinging --replicas=3
deployment.apps/pinging scaled


$ kubectl get po
NAME                   	   READY      STATUS	RESTARTS    AGE
pinging-66dff4b844-pkfxl   1/1        Running   0           8m45s
pinging-66dff4b844-spddv   1/1        Running   0           23s
pinging-66dff4b844-x29qx   1/1        Running   0           23s

Now if you want to stream the logs of all the pods, How will you do that?

If you are thinking that, I will just get the logs of deployment but turns out that It only shows the logs of first pod of the Deployment, As you can see here – 

$ kubectl logs deploy/pinging
Found 3 pods, using pod/pinging-66dff4b844-pkfxl
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.117 ms
64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.147 ms
64 bytes from 127.0.0.1: seq=2 ttl=64 time=0.174 ms
64 bytes from 127.0.0.1: seq=3 ttl=64 time=0.098 ms
64 bytes from 127.0.0.1: seq=4 ttl=64 time=0.095 ms
64 bytes from 127.0.0.1: seq=5 ttl=64 time=0.116 ms
…………
………

Then, How will you check the logs of all the pods? For this you can use labels.
For that, first find out the labels of your pods

$ kubectl get po --show-labels
NAME                   	  READY  STATUS   RESTARTS  AGE	  LABELS
pinging-66dff4b844-pkfxl  1/1 	 Running  0         14m	  app=pinging,pod-template-hash=66dff4b844
pinging-66dff4b844-spddv  1/1 	 Running  0         6m2s  app=pinging,pod-template-hash=66dff4b844
pinging-66dff4b844-x29qx  1/1 	 Running  0         6m2s  app=pinging,pod-template-hash=66dff4b844

Here you can see there are two app=pinging, pod-template-hash=66dff4b844 labels, these get added automatically when you create a deployment. We will use app=pinging for getting the logs

And, Now stream the logs of all the pods based on this labels

$ kubectl logs -f -l app=pinging
64 bytes from 127.0.0.1: seq=512 ttl=64 time=0.092 ms
64 bytes from 127.0.0.1: seq=514 ttl=64 time=0.137 ms
64 bytes from 127.0.0.1: seq=513 ttl=64 time=0.087 ms
64 bytes from 127.0.0.1: seq=514 ttl=64 time=0.149 ms
64 bytes from 127.0.0.1: seq=515 ttl=64 time=0.090 ms
64 bytes from 127.0.0.1: seq=1013 ttl=64 time=0.091 ms
64 bytes from 127.0.0.1: seq=1014 ttl=64 time=0.160 ms
64 bytes from 127.0.0.1: seq=1015 ttl=64 time=0.110 ms
64 bytes from 127.0.0.1: seq=1016 ttl=64 time=0.109 ms
64 bytes from 127.0.0.1: seq=516 ttl=64 time=0.113 ms
64 bytes from 127.0.0.1: seq=517 ttl=64 time=0.116 ms
……………
………
………

Here you can see the logs of all the pods, though there is a problem as you can not differentiate which line of log belongs to which pod.

We can use Stern for that, Stern allows us to tail multiple pods and multiple containers within pods. Each result is color coded for quicker debugging.

Bonus thing – Let’s increase more replicas of this pod to see one more thing –  

$ kubectl scale deployment pinging --replicas=10
deployment.apps/pinging scaled


$ kubectl logs -f -l app=pinging
error: you are attempting to follow 10 log streams, but maximum allowed concurrency is 5, use --max-log-requests to increase the limit

As you can see, You can not stream the logs of more than 5 pods concurrently.

That’s all for now.
Thank you for reading!!

Stay tuned for more articles on Cloud and DevOps. Don’t forget to follow me for regular updates and insights.

Let’s Connect: LinkedIn