NotificationsYou must be signed in to change notification settings
Fork56.4k
Star85.3k

Modified Caffe parser to support the new dnn engine#26208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Jump to bottom

Merged

asmorkalov merged 10 commits intoopencv:5.xfromalexlyulkov:al/new-engine-caffe-parser

Oct 28, 2024

Merged

Modified Caffe parser to support the new dnn engine#26208

asmorkalov merged 10 commits intoopencv:5.xfromalexlyulkov:al/new-engine-caffe-parser

Oct 28, 2024

Conversation

Copy link

Contributor

alexlyulkov commentedSep 27, 2024

Now the Caffe parser supports both the old and the new engine. It can be selected using newEngine argument in PopulateNet.

All cpu Caffe tests work fine except:

Test_Caffe_nets.Colorization
Test_Caffe_layers.FasterRCNN_Proposal

Both these tests doesn't work because of the bug in the new net.forward function. The function takes the name of the desired target last layer, but uses this name as the name of the desired output tensor.
Also Colorization test contains a strange model with a Silence layer in the end, so it doesn't have outputs. The old parser just ignored it. I think, the proper solution is to run this model until the (number_of_layers - 2) layer using proper net.forward arguments in the test.

Pull Request Readiness Checklist

See details athttps://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

alexlyulkov requested review fromasmorkalov andvpisarev

September 27, 2024 09:42

vpisarev mentioned this pull request

Oct 4, 2024

New dnn engine#26056

Merged

6 tasks

asmorkalov added category: dnn feature labels

Oct 8, 2024

asmorkalov added this to the5.0-alpha milestone

Oct 8, 2024

asmorkalov self-assigned this

Oct 8, 2024

alexlyulkov force-pushed theal/new-engine-caffe-parser branch 2 times, most recently from811b6ea tobea0503Compare

October 11, 2024 11:17

asmorkalov pushed a commit that referenced this pull request

Oct 16, 2024

Merge pull request#26056from vpisarev:new_dnn_engine

3cd57ea

New dnn engine#26056This is the 1st PR with the new engine; CI is green and PR is ready to be merged, I think.Merge together withopencv/opencv_contrib#3794---**Known limitations:*** [solved] OpenVINO is temporarily disabled, but is probably easy to restore (it's not a deal breaker to merge this PR, I guess)* The new engine does not support any backends nor any targets except for the default CPU implementation. But it's possible to choose the old engine when loading a model, then all the functionality is available.* [Caffe patch is here:#26208] The new engine only supports ONNX. When a model is constructed manually or is loaded from a file of different format (.tf, .tflite, .caffe, .darknet), the old engine is used.* Even in the case of ONNX some layers are not supported by the new engine, such as all quantized layers (including DequantizeLinear, QuantizeLinear, QLinearConv etc.), LSTM, GRU, .... It's planned, of course, to have full support for ONNX by OpenCV 5.0 gold release. When a loaded model contains unsupported layers, we switch to the old engine automatically  (at ONNX parsing time, not at `forward()` time).* Some layers , e.g. Expat, are only partially supported by the new engine. In the case of unsupported flavours it switches to the old engine automatically (at ONNX parsing time, not at `forward()` time).* 'Concat' graph optimization is disabled. The optimization eliminates Concat layer and instead makes the layers that generate tensors to be concatenated to write the outputs to the final destination. Of course, it's only possible when `axis=0` or `axis=N=1`. The optimization is not compatible with dynamic shapes since we need to know in advance where to store the tensors. Because some of the layer implementations have been modified to become more compatible with the new engine, the feature appears to be broken even when the old engine is used.* Some `dnn::Net` API is not available with the new engine. Also, shape inference may return false if some of the output or intermediate tensors' shapes cannot be inferred without running the model. Probably this can be fixed by a dummy run of the model with zero inputs.* Some overloads of `dnn::Net::getFLOPs()` and `dnn::Net::getMemoryConsumption()` are not exposed any longer in wrapper generators; but the most useful overloads are exposed (and checked by Java tests).* [in progress] A few Einsum tests related to empty shapes have been disabled due to crashes in the tests and in Einsum implementations. The code and the tests need to be repaired.* OpenCL implementation of Deconvolution is disabled. It's very bad and very slow anyway; need to be completely revised.* Deconvolution3D test is now skipped, because it was only supported by CUDA and OpenVINO backends, both of which are not supported by the new engine.* Some tests, such as FastNeuralStyle, checked that the in the case of CUDA backend there is no fallback to CPU. Currently all layers in the new engine are processed on CPU, so there are many fallbacks. The checks, therefore, have been temporarily disabled.---- [x] I agree to contribute to the project under Apache 2 License.- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV- [x] The PR is proposed to the proper branch- [ ] There is a reference to the original bug report and related work- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable      Patch to opencv_extra has the same branch name.- [ ] The feature is well documented and sample code can be built with the project CMake

asmorkalov force-pushed theal/new-engine-caffe-parser branch frombea0503 to4a318f7Compare

October 19, 2024 17:03

Copy link

Contributor

asmorkalov commentedOct 19, 2024

@alexlyulkov I rebased your branch on top of 5.x and fixed conflicts. The original version is available in my fork:https://github.com/asmorkalov/opencv/tree/al/new-engine-caffe-parser

Copy link

Contributor

asmorkalov commentedOct 21, 2024

@dkurt @vpisarev could you take a look?

asmorkalov requested a review fromdkurt

October 21, 2024 11:15

Copy link

Contributor

vpisarev commentedOct 21, 2024

Looks good to me, and it's cool that you managed to combine two Caffe importers, the old one and the new one, in one source file. But it looks like some of the tests fail.

Copy link

Contributor

asmorkalov commentedOct 21, 2024

With the new engine:

2024-10-21T15:02:44.7508887Z [ RUN      ] Net.forwardAndRetrieve2024-10-21T15:02:44.7509953Z unknown file: Failure2024-10-21T15:02:44.7511731Z C++ exception with description "OpenCV(5.0.0-pre) /home/ci/opencv/modules/dnn/src/net_impl.cpp:613: error: (-215:Assertion failed) !layers[0].outputBlobs.empty() in function 'allocateLayers'2024-10-21T15:02:44.7512031Z " thrown in the test body.2024-10-21T15:02:44.7512328Z [  FAILED  ] Net.forwardAndRetrieve (0 ms)

Copy link

Contributor

asmorkalov commentedOct 21, 2024

Also there are a lot of test failures like this:

2024-10-21T15:02:48.5956366Z [ RUN      ] DNNTestNetwork.SSD_VGG16/0, where GetParam() = OCV/CPU2024-10-21T15:02:48.8928234Z unknown file: Failure2024-10-21T15:02:48.8932640Z C++ exception with description "OpenCV(5.0.0-pre) /home/ci/opencv/modules/dnn/src/net_impl.cpp:995: error: (-213:The function/feature is not implemented) The new dnn engine doesn't support inference until a specified layer. If you want to run the whole model, please don't set the outputName argument in the forward() call. If you want to run the model until a specified layer, please use the old dnn engine in function 'forward'2024-10-21T15:02:48.8932911Z " thrown in the test body.2024-10-21T15:02:48.8933491Z [  FAILED  ] DNNTestNetwork.SSD_VGG16/0, where GetParam() = OCV/CPU (297 ms)

I propose to revise test code and replace forward calls where it makes sense.

Alexander Lyulkovand others added4 commits

October 22, 2024 09:11

Modified Caffe parser to support the new dnn engine

2de3f21

Fixed Caffe parser for the new engine, modified few tests for the new…

9d5c6aa

… ingine

Added engine selection to readNetFromCaffe

d16d7e1

Fixed Obj-C generator bindings for Mac and iOS.

51f6d14

asmorkalov force-pushed theal/new-engine-caffe-parser branch froma8b4f23 to51f6d14Compare

October 22, 2024 06:11

asmorkalov reviewed

Oct 22, 2024

View reviewed changes

modules/dnn/src/caffe/caffe_importer.cpp OutdatedShow resolvedHide resolved

modules/dnn/src/caffe/caffe_importer.cpp

		CaffeImportercaffeImporter(prototxt.c_str(), caffeModel.c_str());
		Net net;
		caffeImporter.populateNet(net);
		caffeImporter.populateNet(net, engine == ENGINE_NEW \|\| engine == ENGINE_AUTO);

Copy link

Contributor

asmorkalovOct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

What about fallback, if the new engine cannot handle the model?

Copy link

ContributorAuthor

alexlyulkovOct 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Look like it is unnecessary. The parser doesn't fall in tests

modules/dnn/src/caffe/caffe_importer.cpp

		CaffeImportercaffeImporter(bufferProto, lenProto, bufferModel, lenModel);
		Net net;
		caffeImporter.populateNet(net);
		caffeImporter.populateNet(net, engine == ENGINE_NEW \|\| engine == ENGINE_AUTO);

Copy link

Contributor

asmorkalovOct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

the same here.

asmorkalov added3 commits

October 22, 2024 16:30

Added check to DNN test with named outputs and layers.

81a4500

Forced old DNN engine for the test cases where output names are set.

b4e47fe

More python test suppression.

67b61b7

Copy link

Contributor

asmorkalov commentedOct 23, 2024

Remaining test failures:

2024-10-22T16:02:12.2500917Z [ RUN      ] Net.forwardAndRetrieve2024-10-22T16:02:12.2502780Z unknown file: Failure2024-10-22T16:02:12.2505417Z C++ exception with description "OpenCV(5.0.0-pre) /home/ci/opencv/modules/dnn/src/net_impl.cpp:613: error: (-215:Assertion failed) !layers[0].outputBlobs.empty() in function 'allocateLayers'2024-10-22T16:02:12.2508347Z " thrown in the test body.2024-10-22T16:02:12.2509215Z [  FAILED  ] Net.forwardAndRetrieve (1 ms)

2024-10-22T16:02:53.8966745Z [ RUN      ] Test_Model.DetectionOutput/0, where GetParam() = OCV/CPU2024-10-22T16:02:54.0457031Z [ WARN:0@51.492] global net_impl_backend.cpp:229 setPreferableTarget Targtes are not supported by the new graph egine for now2024-10-22T16:02:54.0547580Z unknown file: Failure2024-10-22T16:02:54.0551203Z C++ exception with description "OpenCV(5.0.0-pre) /home/ci/opencv/modules/dnn/src/net_impl2.cpp:522: error: (-215:Assertion failed) ninputs == 1 && "empty name can only be used to set input if there is just one input" in function 'setMainGraphInput'2024-10-22T16:02:54.0554479Z " thrown in the test body.2024-10-22T16:02:54.0555691Z [  FAILED  ] Test_Model.DetectionOutput/0, where GetParam() = OCV/CPU (158 ms)

Fixed tests with Caffe models and the new dnn engine

f3c819b

Copy link

Contributor

asmorkalov commentedOct 26, 2024

@alexlyulkov Please take a look on Python test failures:

======================================================================ERROR: test_async (test_dnn.dnn_test)Testing OpenCV 5.0.0-preLocal repo path: /home/ci/opencv----------------------------------------------------------------------Traceback (most recent call last):  File "/home/ci/opencv/modules/dnn/misc/python/test/test_dnn.py", line 341, in test_async    netSync.setPreferableBackend(backend)cv2.error: OpenCV(5.0.0-pre) /home/ci/opencv/modules/dnn/src/plugin_wrapper.impl.hpp:313: error: (-213:The function/feature is not implemented) Backend (plugin) is not available: 'openvino' in function 'createPluginDNNNetworkBackend'======================================================================ERROR: test_input_3d (test_dnn.dnn_test)Discovering python tests from: /home/ci/build    found 0 testsDiscovering python tests from: /home/ci/opencv/modules/python/test    found 132 testsDiscovering python tests from: /home/ci/opencv/modules/flann/misc/python/test----------------------------------------------------------------------Traceback (most recent call last):  File "/home/ci/opencv/modules/dnn/misc/python/test/test_dnn.py", line 468, in test_input_3d    net.setPreferableBackend(backend)cv2.error: OpenCV(5.0.0-pre) /home/ci/opencv/modules/dnn/src/plugin_wrapper.impl.hpp:313: error: (-213:The function/feature is not implemented) Backend (plugin) is not available: 'openvino' in function 'createPluginDNNNetworkBackend'======================================================================ERROR: test_set_param_3d (test_dnn.dnn_test)----------------------------------------------------------------------Traceback (most recent call last):  File "/home/ci/opencv/modules/dnn/misc/python/test/test_dnn.py", line 521, in test_set_param_3d    net.setPreferableBackend(backend)cv2.error: OpenCV(5.0.0-pre) /home/ci/opencv/modules/dnn/src/plugin_wrapper.impl.hpp:313: error: (-213:The function/feature is not implemented) Backend (plugin) is not available: 'openvino' in function 'createPluginDNNNetworkBackend'----------------------------------------------------------------------