System information (version)

• OpenCV => 4.5.1
• Operating System / Platform => Ubuntu 64 Bit
• Compiler => gcc

Detailed description

The standard deviation calculation of calibrateCamera is returning NaN for all standard deviations in all cases when provided with comparably low numbers of points.
This is due tothis line, where the divisor and thussigma2 can become negative. The variabletotal usually isnimages*maxPoints or less,nparams = nimages*6+18 (for standard calibration method), andnparams_nz < nparams.
For the casetotal < nparams_nz < nparams the resultingsigma2 is negative.
This can (and nearly always does) happen whenmaxPoints = 6 sototal/nparams < 1.
This is likely also why it has been unnoticed so far.

Changing thetotal tonparams yields seemingly correct results.EDIT: Only for my case wheremaxPoints = 6! This might not be a general or even correct fix!

I've also opened aforum question previously, although I'm positive it is a bug.

Steps to reproduce

1. Generate a sample set of 22 6-point markers
2. CallcalibrateCameraRO, requesting the standard deviations to be calculated

• One sample run yielded the following variable values:total=132 (22*6), nparams=150(22*6+18), nparams_nz=139

3. Observe the returned standard deviations to all be NaN

An example implementation can be found in this larger project (calibrateCameraRO call here).

Issue submission checklist

• [ X] I report the issue, it's not a question
• [X ] I checked the problem with documentation, FAQ, open issues,
  forum.opencv.org, Stack Overflow, etc and have not found solution
• [ X] I updated to latest OpenCV version and the issue is still there
• [ X] There is reproducer code and related data files: videos, images, onnx, etc