Description

Refhttps://matrix.to/#/!wJiiHsLipVywuWOyNi:ubuntu.com/$4SJQ5ruV0QkcEEG0BJCfiDJmElBPuZH-i4WPpqc97Io?via=ubuntu.com&via=matrix.org

The default installation of Canonicalk8s snap comes with the default limit of apparently 100 (or 110?) pods per node.

Attempting to scale a simple app (solo model, solo app, scale ~100) beyond that limit results some pods not appearing at all, and Juju cli not being very helpful.

Juju status header:

App               Version  Status     Scale  Charm             Channel  Rev  Address         Exposed  Messagejgol-worker                active    98/100  jgol-worker                  0  ...

Juju status unit list omits the units that didn't come up.
juju debug-log doesn't mention those units at all.
When there are many units, it's hard to visually notice which ones are missing.

The work-around is to add--max-pods=NNN flag to Kubelet, in the case of the snap, that's the/var/snap/k8s/common/args/kubelet file.

kubectl describe pods -n xxx shows the affected units with this annotation:

Warning FailedScheduling 12m default-scheduler 0/1 nodes are available: 1 Too many pods. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

I wish that Juju exposed the individual units that were requested but not yet allocated.
If Juju could expose the underlying K8s error, that would be even better.

Juju version

3.6.11

Cloud

No response

Expected behaviour

Something visible injuju status

Reproduce / Test

• concierge prepare --profile k8s
• juju deploy ./some.charm -n 100
• wait a while
• juju status

Notes & References

No response