Intelligent food recognition and meal analysis system based on multi-modal interactionTechnical Field
The invention relates to the field of health management, in particular to an intelligent food recognition and diet analysis system based on multi-modal interaction.
Background
Diet investigation is a key ring in weight management, a common method is 24h diet review method, but the method has a certain limitation that a researched person may have problems such as recall bias, food quantification and the like, and investigation accuracy is greatly limited. The traditional meal investigation tool in recent years comprises (1) a Chinese patent CN118430746A builds a virtual food model through VR wearing equipment, further calculates the weight of food through the volume of virtual food, simplifies the investigation process but cannot solve the subjective influence of user recall bias and food quantification, (2) a Chinese patent CN117936030A adopts a watershed method to divide images, determines image boundaries based on edge detection, further calculates image contours to determine the types and the quantity of food, and then combines the sizes of reference objects to estimate the actual intake of the food, although shortening the investigation time, cannot accurately divide the quantitative problems of mixed dishes and various foods in the mixed dishes, and (3) a Chinese patent CN117976144A distinguishes the weight of food before meal and the weight of food after meal through a food identification technology, calculates the content of food ingredients, and still does not solve the quantitative problems of various foods in the mixed dishes, so that certain errors exist in calculated values and application scenes are limited, and (4) a part of the investigation method is mostly limited to carrying out food identification by using the traditional image identification technology, ignoring the key problems corresponding to the foods and realizing accurate meal management and health management errors in component measurement one by one.
Disclosure of Invention
Aiming at the problems that the existing meal recognition technology cannot accurately quantify mixed food materials and relies on manual recording to have low efficiency, the invention provides an intelligent food recognition and meal analysis system based on multi-modal interaction, which remarkably improves the food recognition accuracy through an image and voice dual-modal interaction recognition mechanism, combines intelligent weighing and data synchronization functions, simplifies the operation flow of users and realizes accurate meal management and health guidance.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows:
An intelligent food recognition and diet analysis system based on multi-modal interaction is characterized by comprising an intelligent food scale hardware module, a multi-modal recognition module, a food database module, a diet analysis module and a mobile phone end synchronization module,
The intelligent food balance hardware module comprises a weighing sensor, a camera, a voice interaction unit and a communication unit;
The multi-mode recognition module is used for collecting food images and extracting image features of various foods before cooking through the camera, matching the food images with the food database module and obtaining food information, and activating the voice interaction unit to receive voice input and secondarily matching the voice interaction unit with the food database module through voice recognition conversion if the matching confidence is lower than a preset threshold value of 75%, so as to obtain the food information;
The food database module is used for storing food names, image features, voice tag libraries, nutrition component data of unit weight and associated codes, supporting dynamic expansion, uploading images, voice descriptions of self-defined food, package food bar codes and corresponding nutrition tag information by a user through a mobile phone APP, and updating the cloud database after being checked by a system administrator.
The meal analysis module is used for receiving the matching result (including the association code) from the multi-mode identification module and gram weight data weighed by the weighing sensor, and calculating the contents of carbohydrate, fat, protein and trace elements recorded in the sixth edition of Chinese food composition table of the actually ingested food by associating the nutrition component data of unit weight in the food database module;
The mobile phone terminal synchronization module is used for transmitting the data generated by the meal analysis module to the mobile phone terminal APP in real time through the communication unit to generate a visual meal analysis report and a health suggestion.
Further, the multi-modal recognition module comprises an image recognition sub-module and a voice interaction sub-module,
The image recognition submodule adopts ResNet-50 improved models of a convolutional neural network, and by adding an SE attention module before the last layer of full connection, the recognition accuracy of food is improved, the voice interaction submodule supports the recognition of Shandong, guangdong and Yun Guichuan dialects and spoken language expression, semantic analysis is carried out based on BERT models subjected to corpus fine adjustment in the food field, and the matching result is optimized.
Further, the food database module also comprises a special meal warehouse sub-module and a packaged food warehouse sub-module,
The special meal warehouse submodule stores infant formula food, special medical food, meal replacement food, nutrient supplements, GI food and health food data;
the packaging food warehouse submodule stores nutrition tag information related to the scanned GS1 standard commodity bar code;
the special meal library sub-module (31) and the packaged food library sub-module (32) also support dynamic expansion, and a user can upload images, voice descriptions and packaged food bar codes of self-defined foods and corresponding nutrition tag information through the mobile phone APP, and update a cloud database after being checked by a system administrator.
Further, the mobile phone synchronization module also comprises a nutrition balance display sub-module and a food suggestion sub-module,
The nutrition balance display submodule supports the health targets of autonomous input of age, sex, weight, fat reduction, muscle increase, sugar control or salt control of a user, generates daily recommended intake based on the standard of Chinese resident dietary nutrient reference intake (2023 edition), compares the daily recommended intake with actual intake data obtained by calculation of a dietary analysis module (4), supports backtracking and time analysis of historical data, and visualizes the nutrition balance in a chart form;
the food suggestion submodule makes reasonable diet suggestions for nutrient components which are insufficient or exceed the recommended intake according to the result of the nutrition balance display submodule.
Furthermore, the communication unit supports Bluetooth and Wi-Fi, ensures real-time synchronization of data and is compatible with mobile terminals of iOS 12 and above and Android 8.0 and above systems.
The beneficial effects of the invention are as follows:
According to the daily requirements of users, the method and the system for detecting the weight of the food meet the daily requirements of the users, and the cleaned food materials are sequentially subjected to gram weight data weighed by the weighing sensor in the intelligent food weighing hardware module before cooking, and are matched with food information in the food database through image or voice recognition, so that diet analysis is performed. The influence of recall bias and subjective quantification of foods of a user is eliminated, each food is prevented from being missed and accurate gram weight data are acquired, and the tedious operation that the user frequently uses a mobile phone to record or photograph is reduced.
Under the action of a nutrition balance degree display submodule in a mobile phone end synchronization module, the system can be matched with a recommended intake according to gender, age and weight, and health targets of fat reduction, muscle increase, sugar control or salt control, a user can also self-define a historical data backtracking period, a comparison result between the historical data backtracking period and the actual intake is visualized through a chart form, and the diet balance consciousness of the user is enhanced.
Under the action of a food suggestion submodule in the mobile phone end synchronization module, according to a self-defined historical data backtracking period analysis result, when the intake of a certain nutrient substance is insufficient or exceeds the recommended intake, the food suggestion submodule sorts the nutrient substances according to the content of the nutrient substance in different foods, simultaneously gives food types and intake recommendations, gradually guides a user to reasonably plan diet by pushing a corresponding food cooking method, and keeps healthy weight.
Drawings
FIG. 1 is a structural framework diagram of the intelligent food identification and meal analysis system based on multimodal interactions of the present invention.
FIG. 2 is an overall flow chart of the intelligent food identification and meal analysis system based on multimodal interactions of the present invention.
In the figure, a 1-intelligent food balance hardware module, a 11-weighing sensor, a 12-camera, a 13-voice interaction unit, a 14-communication unit, a 2-multi-mode identification module, a 21-image identification sub-module, a 22-voice interaction sub-module, a 3-food database module, a 31-special meal library sub-module, a 32-packaged food library sub-module, a 4-meal analysis module, a 5-mobile phone end synchronization module, a 51-nutrition balance degree display sub-module and a 52-food suggestion sub-module.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1 and 2, the intelligent food recognition and diet analysis system based on multi-modal interaction according to a preferred embodiment of the present invention includes an intelligent food balance hardware module 1, a multi-modal recognition module 2, a food database module 3, a diet analysis module 4, and a mobile phone synchronization module 5.
The intelligent food balance hardware module 1 comprises a weighing sensor 11, a camera 12, a voice interaction unit 13 and a communication unit 14.
The multi-mode recognition module 2 is used for collecting food images and extracting image features of various foods before cooking through the camera 12, matching the food images with the food database module 3 to obtain food information, and activating the voice interaction unit 13 to receive voice input and secondarily matching the food database module 3 through voice recognition conversion if the matching confidence is lower than a preset threshold value 75% to obtain the food information. Before cooking, various foods comprise two forms before and after the cutter changing treatment, when the matching of the foods fails, the system prompts that the picture identification fails, please complain about the names of the foods, and the user can match with the food database 3 again through voice interaction, so that the accuracy of the food identification is improved.
The food database module 3 is used for storing food names, image features, voice tag libraries, nutrition component data of unit weight and associated codes, supporting dynamic expansion, and a user can upload images, voice descriptions, package food bar codes and corresponding nutrition tag information of self-defined food through the mobile phone APP, and update the cloud database after being checked by a system administrator. The food name and the nutrient composition data of unit weight are from the sixth edition of the Chinese food composition Table.
The meal analysis module 4 is configured to receive the matching result (including the association code) from the multimodal recognition module 2 and the gram weight data weighed by the weighing sensor 11, and calculate the carbohydrate, fat, protein and trace element content recorded in the sixth edition of the chinese food composition table of the actually ingested food by associating the nutrition component data per unit weight in the food database module 3.
The mobile phone terminal synchronization module 5 transmits the data generated by the meal analysis module 4 to the mobile phone terminal APP in real time through the communication unit 14 to generate a visual meal analysis report and a health suggestion.
The multi-modality recognition module 2 includes an image recognition sub-module 21 and a voice interaction sub-module 22.
The image recognition sub-module 21 adopts ResNet-50 improved model of convolutional neural network, and improves the recognition accuracy of food by adding SE attention module before the last layer of full connection.
The voice interaction sub-module 22 supports Shandong, guangdong and Yun Guichuan dialects and spoken language expression recognition, performs semantic analysis based on BERT models subjected to corpus fine adjustment in the food field, and optimizes matching results.
The food database module 3 also includes a special meal library sub-module 31 and a packaged food library sub-module 32.
The special meal pool sub-module 31 stores infant formula, special medical food, meal replacement food, nutrient supplements, GI food and health food data, and the package food pool sub-module 32 stores nutritional tag information associated with the scanned GS1 standard commodity bar code. Meanwhile, the food database module supports dynamic expansion, and a user can upload images, voice descriptions and package food bar codes of self-defined foods and corresponding nutrition tag information through the mobile phone APP, and update the cloud database after checking by a system administrator.
The mobile phone synchronization module 5 further includes a nutrition balance display sub-module 51 and a food suggestion sub-module 52.
The nutrition balance display sub-module 51 supports the user to autonomously input the health targets of age, sex, weight, fat reduction, muscle increase, sugar control or salt control, generates daily recommended intake based on the standard of Chinese resident dietary nutrient reference intake (2023 edition), compares the daily recommended intake with the actual intake data calculated and acquired by the dietary analysis module 4, supports the backtracking and time analysis of historical data, and visualizes the nutrition balance in a chart form.
The food advice sub-module 52 makes a reasonable meal advice for nutritional ingredients that are insufficient or exceed the recommended intake based on the results of the nutritional balance presentation sub-module 51.
The nutrition balance display sub-module 51 of the present embodiment can calculate the body quality index according to the height and weight of the user, display all the information of the food related to the sixth edition of the Chinese food composition table by default through the mobile phone APP, and display the food related information by user definition according to the user's requirement, for example, the user only pays attention to heat, carbohydrate, protein and fat, and then select the corresponding options in the nutrition balance display sub-module 51. In addition, the food suggestion sub-module 52 also provides corresponding meal suggestions according to the results of the customized presentation by the nutrition balance presentation sub-module 51.