Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Added int32, int64 support and type inference to dnn#24411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
asmorkalov merged 28 commits intoopencv:5.xfromalexlyulkov:al/dnn-type-inference
Mar 1, 2024

Conversation

@alexlyulkov
Copy link
Contributor

@alexlyulkovalexlyulkov commentedOct 16, 2023•
edited by asmorkalov
Loading

Related PRs:

Added a type inference to dnn similar to the shape inference, added int32 and int64 support.

  • Added getTypes method for layers that calculates layer outputs types and internals types from inputs types (Similar to getMemoryShapes). By default outputs and internals types = input[0] type
  • Added type inference pipeline similar to shape inference pipeline. LayersShapes struct (that is used in shape inference pipeline) now contains both shapes and types
  • All layers output blobs are now allocated using the calculated types from the type inference.
  • Inputs and constants with int32 and int64 types are not automatically converted into float32 now.
  • Added int32 and int64 support for all the layers with indexing and for all the layers required in tests.

Added int32 and int64 support for CUDA:

  • Added host<->device data moving for int32 and int64
  • Added int32 and int64 support for several layers (just slightly modified CUDA C++ templates)

Passed all the accuracy tests on CPU, OCL, OCL_FP16, CUDA, CUDA_FP16. (except RAFT model)

CURRENT PROBLEMS:

  • ONNX parser always converts int64 constants and layers attributes to int32, so some models with int64 constants doesn't work (e.g. RAFT). The solution is to disable int64->int32 conversion and fix attributes reading in a lot of ONNX layers parsers (New 5.x ONNX parser doesn't support int64 constants #25102)
  • I didn't add type inference and int support to VULCAN, so it doesn't work at all now.
  • Some layers don't support int yet, so some unknown models may not work.

CURRENT WORKAROUNDS:

  • CPU arg_layer indides are implemented in int32 followed by a int32->int64 conversion (the master branch has the same workaround with int32->float conversion)
  • CPU and OCL pooling_layer indices are implemented in float followed by a float->int64 conversion
  • CPU gather_layer indices are implemented in int32, so int64 indices are converted to int32 (the master branch has the same workaround with float->int32 conversion)

DISABLED TESTS:

  • RAFT model

REMOVED TESTS:

  • Greater_input_dtype_int64 (because it doesn't fit ONNX rules, the whole test is just comparing float tensor with int constant)

TODO IN NEXT PULL REQUESTS:

  • Add int64 support for ONNX parser
  • Add int support for more layers
  • Add int support for OCL (currently int layers just run on CPU)
  • Add int tests
  • Add int support for other backends
force_builders=Linux OpenCL,Win64 OpenCL

inps[i] = *ld.inputBlobs[i];
}
layerPtr->finalize(inps, ld.outputBlobs);
layerPtr->preferableTarget = preferableTarget;
Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Moved it to line 590, because this line runs after type inferense and line 590 runs before type inference

* order is the same as in layersIds
*/
voidgetLayerShapes(const MatShape& netInputShape,
const MatType& netInputType,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why type is requires to get only shapes?

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Since the shape inference and the type inference has the same pipeline, I added the type inference into the shape inference pipeline. So this function probably will be renamed into getLayersShapesAndTypes

@alexlyulkovalexlyulkov changed the titleAdded type inference to dnnAdded int32, int64 support and type inference to dnnOct 24, 2023
@asmorkalov
Copy link
Contributor

See build errors:

/build/precommit_linux64/5.x/opencv_contrib/modules/text/src/ocr_holistic.cpp: In member function 'size_t cv::text::OCRHolisticWordRecognizerImpl::getClassCount()':/build/precommit_linux64/5.x/opencv_contrib/modules/text/src/ocr_holistic.cpp:78:63: error: no matching function for call to 'cv::dnn::dnn5_v20231225::Net::getLayerShapes(cv::dnn::dnn5_v20231225::MatShape&, int&, std::vector<std::vector<int> >&, std::vector<std::vector<int> >&)'         net.getLayerShapes(inputShape, id, inShapes, outShapes);                                                               ^In file included from /build/precommit_linux64/5.x/opencv/modules/dnn/include/opencv2/dnn.hpp:76:0,                 from /build/precommit_linux64/5.x/opencv_contrib/modules/text/src/ocr_holistic.cpp:8:/build/precommit_linux64/5.x/opencv/modules/dnn/include/opencv2/dnn/dnn.hpp:749:14: note: candidate: void cv::dnn::dnn5_v20231225::Net::getLayerShapes(const MatShape&, const MatType&, int, std::vector<std::vector<int> >&, std::vector<std::vector<int> >&) const         void getLayerShapes(const MatShape& netInputShape,              ^~~~~~~~~~~~~~/build/precommit_linux64/5.x/opencv/modules/dnn/include/opencv2/dnn/dnn.hpp:749:14: note:   candidate expects 5 arguments, 4 provided/build/precommit_linux64/5.x/opencv/modules/dnn/include/opencv2/dnn/dnn.hpp:756:14: note: candidate: void cv::dnn::dnn5_v20231225::Net::getLayerShapes(const std::vector<std::vector<int> >&, const std::vector<int>&, int, std::vector<std::vector<int> >&, std::vector<std::vector<int> >&) const         void getLayerShapes(const std::vector<MatShape>& netInputShapes,              ^~~~~~~~~~~~~~/build/precommit_linux64/5.x/opencv/modules/dnn/include/opencv2/dnn/dnn.hpp:756:14: note:   candidate expects 5 arguments, 4 provided

@asmorkalov
Copy link
Contributor

@alexlyulkov Please rebase and fix conflicts.

constint requiredOutputs,
constint requiredInternals,
std::vector<MatType>&outputs,
std::vector<MatType>&internals)const;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Do we really need such kind of functionality in public API?
What Users scenarios need all types of all layers at once?

It makes sense to support this scenarios instead:

  • user set network inputs
  • user specify list of requested outputs
  • network compiles (internally calls layer's.finalize())
  • user asks for layer by its name
  • ask types and shapes of layer outputs without passing of any parameters like "inputs/outputs/internals" - pass only index of layer's output.

Note:

  • support multiple outputs per layer
  • user should not know anything about "internals"
  • getMemoryShapes /getLayersShapes() should be deprecated as primary API too

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think it is out of scope of this pull request. I added the new type inference pipeline similar to existing shape inference pipeline because this way produces less architecture changes. I think it will be better to make another pull request with public API modifications.

@fengyuentau
Copy link
Member

Something is broken with RAFT

Test_ONNX_nets.RAFT/0, where GetParam() = OCV/CPU[ERROR:0@65.123] global net_impl.cpp:1174 getLayerShapesRecursively OPENCV/DNN: [GatherElements]:(onnx_node!GatherElements_366): getMemoryShapes() throws exception. inputs=2 outputs=1/1 blobs=0[ERROR:0@65.123] global net_impl.cpp:1180 getLayerShapesRecursively     input[0] = [ 2700 1 2914 ][ERROR:0@65.123] global net_impl.cpp:1180 getLayerShapesRecursively     input[1] = [ 2700 1 81 ][ERROR:0@65.123] global net_impl.cpp:1184 getLayerShapesRecursively     output[0] = [ 2700 1 81 ][ERROR:0@65.123] global net_impl.cpp:1190 getLayerShapesRecursively Exception message: OpenCV(5.0.0-pre) /home/ci/opencv/modules/dnn/src/layers/gather_elements_layer.cpp:68: error: (-215:Assertion failed) inputs[1] == CV_64S || inputs[1] == CV_32S in function 'getTypes'unknown file: FailureC++ exception with description "OpenCV(5.0.0-pre) /home/ci/opencv/modules/dnn/src/layers/gather_elements_layer.cpp:68: error: (-215:Assertion failed) inputs[1] == CV_64S || inputs[1] == CV_32S in function 'getTypes'

}

TEST_P(Test_ONNX_nets,RAFT)
TEST_P(Test_ONNX_nets,DISABLED_RAFT)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Please create ticket about the issue and add reference to it in comments.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Reminder.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Comment on lines -76 to -80
if (inputs_arr.depth() == CV_16F)
{
forward_fallback(inputs_arr, outputs_arr, internals_arr);
return;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why?

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

forward_fallback converts data from float16 to float32 and runs the layer. Now max_umpooling_layer supports CV_16F, so it doesn't need forward_fallback

Comment on lines -189 to -193
if (hasDynamicShapes)
{
updateLayersShapes();
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why?

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

As I understand, this function runs shape inference for the second time for the models with dynamic shapes. I've found out that this is redundant because this line runs before the main shape inference.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@dkurt could you take a look?

}

TEST_P(Test_ONNX_nets,RAFT)
TEST_P(Test_ONNX_nets,DISABLED_RAFT)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Reminder.

Alexander Lyulkovand others added20 commitsMarch 1, 2024 13:52
Copy link
Contributor

@asmorkalovasmorkalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

👍 Looks good to me! Thanks a lot for the contribution!

@asmorkalov
Copy link
Contributor

/build/precommit_docs/5.x/opencv/modules/dnn/include/opencv2/dnn/dnn.hpp:738: warning: The following parameters of cv::dnn::Net::getFLOPS(const std::vector< MatShape > &netInputShapes, const std::vector< int > &netInputTypes) const are not documented:/build/precommit_docs/5.x/opencv/modules/dnn/include/opencv2/dnn/dnn.hpp:721: warning: The following parameters of cv::dnn::Net::getLayerShapes(const MatShape &netInputShape, const int &netInputType, const int layerId, std::vector< MatShape > &inLayerShapes, std::vector< MatShape > &outLayerShapes) const are not documented:/build/precommit_docs/5.x/opencv/modules/dnn/include/opencv2/dnn/dnn.hpp:699: warning: The following parameters of cv::dnn::Net::getLayersShapes(const std::vector< MatShape > &netInputShapes, const std::vector< int > &netInputTypes, std::vector< int > &layersIds, std::vector< std::vector< MatShape > > &inLayersShapes, std::vector< std::vector< MatShape > > &outLayersShapes) const are not documented:/build/precommit_docs/5.x/opencv/modules/dnn/include/opencv2/dnn/dnn.hpp:769: warning: The following parameters of cv::dnn::Net::getMemoryConsumption(const std::vector< MatShape > &netInputShapes, const std::vector< int > &netInputTypes, size_t &weights, size_t &blobs) const are not documented:/build/precommit_docs/5.x/opencv/modules/dnn/include/opencv2/dnn/dnn.hpp:794: warning: The following parameters of cv::dnn::Net::getMemoryConsumption(const std::vector< MatShape > &netInputShapes, const std::vector< int > &netInputTypes, std::vector< int > &layerIds, std::vector< size_t > &weights, std::vector< size_t > &blobs) const are not documented:

@asmorkalovasmorkalov merged commit1d1faaa intoopencv:5.xMar 1, 2024
@opencv-alalek
Copy link
Contributor

opencv-alalek commentedMar 4, 2024•
edited
Loading

Where is the first line of PR description with the link on related opencv_contrib PR?

CI is totally RED.

@asmorkalov
Copy link
Contributor

@asmorkalov
Copy link
Contributor

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@dkurtdkurtdkurt left review comments

@opencv-alalekopencv-alalekopencv-alalek left review comments

@vpisarevvpisarevvpisarev approved these changes

@asmorkalovasmorkalovasmorkalov approved these changes

Assignees

No one assigned

Projects

None yet

Milestone

5.0-alpha

Development

Successfully merging this pull request may close these issues.

6 participants

@alexlyulkov@asmorkalov@fengyuentau@opencv-alalek@vpisarev@dkurt
[8]ページ先頭
©2009-2025 Movatter.jp