CN113838210A

Movatterモバイル変換

Info

Publication number: CN113838210A
Application number: CN202111059999.3A
Authority: CN
Inventors: 骞爱荣; 赖林彬; 张文娟; 侯怡; 张红; 唐再煜; 兰玉乾; 李郁
Original assignee: Northwestern Polytechnical University
Current assignee: Northwestern Polytechnical University
Priority date: 2021-09-10
Filing date: 2021-09-10
Publication date: 2021-12-24

Abstract

Translated fromChinese

本发明公开了一种将超声图像转换为3D模型的方法及装置，涉及影像处理技术领域，更加直观，有助于医患沟通；进而大大提高了检测结果的准确率；同时也有助于医学解剖、学习等。方案要点为：获取多张待转换超声图像；提取每一张待转换超声图像的特征图，得到特征图图谱；根据特征图图谱上各点坐标，获取特征图图谱上各点的类别及目标检测框；提取特征图图谱上前景类内各点与目标检测框组成待检测图像；采用阈值分割法，获取自动分割图像；根据自动分割图像获取3D转换模型。本发明主要用于医学临床诊断。

The invention discloses a method and a device for converting an ultrasonic image into a 3D model, and relates to the technical field of image processing. , learning, etc. The main points of the scheme are: acquiring multiple ultrasound images to be converted; extracting the feature map of each ultrasound image to be converted to obtain a feature map map; obtaining the category and target detection of each point on the feature map map according to the coordinates of each point on the feature map map frame; extract the points in the foreground class on the feature map and the target detection frame to form the image to be detected; adopt the threshold segmentation method to obtain the automatic segmentation image; obtain the 3D transformation model according to the automatic segmentation image. The present invention is mainly used for medical clinical diagnosis.

Description

Method and device for converting ultrasonic image into 3D model

Technical Field

The invention relates to the technical field of image processing, in particular to a method and a device for converting an ultrasonic image into a 3D model.

Background

The ultrasonic image is a tomographic image of human body formed by ultrasonic diagnostic apparatus using echo principle, i.e. a beam of ultrasonic wave is transmitted by probe of the apparatus to enter into body, and is scanned in linear, sector or other form, when it meets interface of two tissues with different acoustic impedances, the ultrasonic wave is reflected back, and after being received by probe, it is displayed on screen through signal amplification and information processing. Is commonly used in medical clinical diagnosis, especially in the examination of fetal heart malformations during pregnancy.

In the prior art, doctors generally look at ultrasonic images to determine whether the fetal heart is malformed during pregnancy. By the method, whether most fetal hearts have malformations or not can be detected. However, due to the individual difference of the fetus and the complexity of the image such as attenuation, speckle, shadow, signal loss, etc., if the ultrasound image is manually marked by a doctor for observation and detection, not only much time and effort are required, but also the accuracy of the detection result is not high.

Disclosure of Invention

The invention provides a method and a device for converting an ultrasonic image into a 3D model, wherein a characteristic graph map is obtained by extracting a characteristic graph of each ultrasonic image to be converted; acquiring the category and the target detection frame of each point according to the coordinates of each point on the feature map; extracting each point in the foreground class and a target detection frame to form an image to be detected; obtaining an automatic segmentation image by adopting a threshold segmentation method; further acquiring a 3D conversion model; compared with the prior art, the method has the advantages that the 2D ultrasonic image is converted into the 3D model, so that the method is more intuitive and is beneficial to doctor-patient communication; thereby greatly improving the accuracy of the detection result; and also facilitates medical anatomy, learning, etc.

In order to achieve the purpose, the invention adopts the following technical scheme:

the invention provides a method for converting an ultrasonic image into a 3D model, which comprises the following steps:

and acquiring a plurality of ultrasonic images to be converted.

And extracting the characteristic diagram of each ultrasonic image to be converted to obtain a characteristic diagram spectrum.

Acquiring the category and the target detection frame of each point on the feature map according to the coordinates of each point on the feature map; the categories are classified into a foreground category and a background category.

And extracting each point in the foreground class on the characteristic map and the target detection frame to form an image to be detected.

Obtaining an automatic segmentation image by adopting a threshold segmentation method; namely, each point in the image to be detected is segmented according to a preset threshold value, and an automatic segmentation image is obtained.

And acquiring a 3D conversion model according to the automatic segmentation image.

Further, obtaining the category and the target detection frame of each point on the feature map according to the coordinates of each point on the feature map, includes:

uniformly dividing k × H × W anchor regions on the feature map; wherein, the anchor regions have different scales, k is 9, H is the height of the feature map, and W is the width of the feature map.

Respectively corresponding each point on the characteristic diagram map to each anchor area to obtain corresponding coordinates; .

And respectively sending the corresponding coordinates of each point on the feature map into a softmax classifier, and acquiring the category and the target detection frame of each point on the feature map.

Further, after the corresponding coordinates of each point on the feature map are sent to a softmax classifier, and the category and the target detection frame of each point on the feature map are obtained, the method further includes:

calculating offset, and optimizing the type and the target detection frame of each point on the feature map.

Further, the threshold segmentation method includes:

a global threshold segmentation method and a self-adaptive local threshold segmentation method; wherein the global threshold segmentation method comprises BINARY, TRUNC, TOZERO; the adaptive local threshold segmentation method comprises MEAN _ C and GAUSSIAN _ C.

Further, after acquiring the 3D conversion model according to the automatic segmentation image, the method further includes:

and optimizing the 3D conversion model by adopting a redrawing grid method.

Further, the ultrasound image to be converted includes:

the ultrasonic image to be converted is obtained from an ultrasonic instrument and is an ultrasonic image of a fetal heart in 23 to 27 weeks during pregnancy.

Further, the method for converting an ultrasound image into a 3D model further includes:

carrying out format conversion on the ultrasonic image to be converted; namely, the original format of the ultrasonic image to be converted is converted into the JPEG format.

Meanwhile, the invention also provides an ultrasound image conversion device, which comprises:

the first acquisition unit is used for acquiring a plurality of ultrasonic images to be converted.

And the extraction unit is used for extracting the characteristic diagram of each ultrasonic image to be converted to obtain a characteristic diagram map.

The second acquisition unit is used for acquiring the category and the target detection frame of each point on the feature map according to the coordinates of each point on the feature map; the categories are classified into a foreground category and a background category.

And the composition unit is used for extracting each point in the foreground class on the characteristic map and the target detection frame to form an image to be detected.

A third obtaining unit, which obtains an automatic segmentation image by adopting a threshold segmentation method; namely, each point in the image to be detected is segmented according to a preset threshold value, and an automatic segmentation image is obtained.

And the conversion unit is used for acquiring a 3D conversion model according to the automatic segmentation image.

Further, the second obtaining unit includes:

a divider: the characteristic map is used for uniformly dividing k H W anchor regions; wherein, the anchor regions have different scales, k is 9, H is the height of the feature map, and W is the width of the feature map.

A corresponding device: the characteristic diagram is used for respectively corresponding each point on the characteristic diagram map to each anchor area to obtain corresponding coordinates; .

A classifier: and the coordinate detection device is used for classifying the corresponding coordinates of each point on the feature map respectively to obtain the category and the target detection frame of each point on the feature map.

Further, the ultrasound image conversion apparatus further includes: a first optimizer and a second optimizer;

the first optimizer: and the method is used for calculating the offset and optimizing the type and the target detection frame of each point on the feature map.

The second optimizer: and optimizing the 3D conversion model by adopting a redrawing grid method.

Further, the ultrasound image conversion apparatus further includes:

a format converter: and the format converter is used for converting the original format of the ultrasonic image to be converted into a JPEG format.

Further, the ultrasound image conversion apparatus further includes:

a display unit: for displaying the 3D conversion model.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below.

FIG. 1 is a flow chart illustrating a method for converting an ultrasound image into a 3D model according to an embodiment of the present invention;

FIG. 2 is a flow chart illustrating another method for converting an ultrasound image into a 3D model according to an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of an apparatus for converting an ultrasound image into a 3D model according to an embodiment of the present invention;

fig. 4 is a schematic structural diagram of another apparatus for converting an ultrasound image into a 3D model according to an embodiment of the present invention.

Detailed Description

The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example 1

An embodiment of the present invention provides a method for converting an ultrasound image into a 3D model, as shown in fig. 1, including:

101. and acquiring a plurality of ultrasonic images to be converted.

The ultrasonic image to be converted is an ultrasonic diagnostic apparatus which utilizes the echo principle, namely, an instrument probe emits a beam of ultrasonic into the body and carries out linear, sector or other forms of scanning, when the ultrasonic image meets the interface of two tissues with different acoustic impedances, the ultrasonic image is reflected back, and after the ultrasonic image is received by the probe, the ultrasonic image is amplified and processed by information and displayed on a screen to form a human body tomogram. In order to ensure the accuracy of the subsequent transformation, at least 600 ultrasound images to be transformed are generally selected, and 800 ultrasound images to be transformed are selected in the embodiment.

102. And extracting the characteristic diagram of each ultrasonic image to be converted to obtain a characteristic diagram spectrum.

Specifically, each ultrasonic image to be converted is sent to the convolution layer and passes through a group of conv + relu + posing layers to be used as a feature extractor to extract an image feature map.

103. Acquiring the category and the target detection frame of each point on the feature map according to the coordinates of each point on the feature map; the categories are classified into a foreground category and a background category.

The method comprises the steps of respectively introducing each point on a feature map into preset coordinate systems of a plurality of different scales, namely, the coordinates of each point on the feature map in different coordinate systems are different, and judging whether each point belongs to a foreground class or a background class and a target detection frame according to the coordinates of each point on the feature map.

104. And extracting each point in the foreground class on the characteristic map and the target detection frame to form an image to be detected.

105. Obtaining an automatic segmentation image by adopting a threshold segmentation method; namely, each point in the image to be detected is segmented according to a preset threshold value, and an automatic segmentation image is obtained.

The threshold segmentation method comprises a global threshold segmentation method and a self-adaptive local threshold segmentation method; the global threshold segmentation method comprises BINARY, TRUNC and TOZERO; the adaptive local threshold segmentation method comprises MEAN _ C and GAUSSIAN _ C. The above-mentioned division methods will be described in detail below, and the user can select them as desired when the present embodiment is applied.

BINARY is the simplest threshold segmentation method, and changes the pixel points larger than the threshold value into the maximum value, and changes the other pixel points into 0, and the specific formula is as follows:

in the formula: dst (x, y) represents the value of the image after threshold segmentation at the (x, y) pixel, and src (x, y) represents the gray value of the original image at the (x, y) pixel.

The TRUNC reserves the pixel information of which the gray scale is smaller than the threshold value on the basis of the BINARY, and the pixel points of which the gray scale is larger than the threshold value are converted into the preset threshold value instead of the maximum value. The concrete formula is as follows:

TOZERO selects the image information with the reserved gray value larger than the threshold value, and sets the gray value to be 0 for other pixel points, and the specific formula is as follows:

the MEAN _ C and the GAUSSIAN _ C are adaptive threshold segmentation, and different from the three methods, the threshold values of the two methods are automatically determined according to local neighborhood, and then the BINARY method is used for segmentation after the threshold values are determined. The concrete formula is as follows:

wherein, if the method is the MEAN _ C method, T (x, y) represents the difference value of the average value of the local neighborhood blocks and a constant C; for the GAUSSIAN _ C method T (x, y) we denote the difference of the GAUSSIAN weighted sum of the local neighborhood blocks and the constant C.

106. And acquiring a 3D conversion model according to the automatic segmentation image.

And converting each automatic segmentation image into a 3D conversion model in a superposition mode. That is, since the interval d between adjacent layers of the image acquired by the ultrasonic instrument is 0.2mm, the image from the i-th layer to the i + 1-th layer can be considered to be the same when d is sufficiently small due to the continuity of the object. The specific mathematical formula is as follows:

model(x，y，z)＝dst_i(x，y)d_i＜z＜d_i+1 (5)

in the formula: model (x, y, z) represents the value of the 3D model at (x, y, z) (a value of 0 indicates that the location is not part of the object, otherwise part of the object). dst_i(x, y) represents the image segmentation result of the ith layer.

The invention provides a method for converting an ultrasonic image into a 3D model, which comprises the steps of extracting a characteristic diagram of each ultrasonic image to be converted to obtain a characteristic diagram map; acquiring the category and the target detection frame of each point according to the coordinates of each point on the feature map; extracting each point in the foreground class and a target detection frame to form an image to be detected; obtaining an automatic segmentation image by adopting a threshold segmentation method; further acquiring a 3D conversion model; compared with the prior art, the method has the advantages that the 2D ultrasonic image is converted into the 3D model, so that the method is more intuitive and is beneficial to doctor-patient communication; thereby greatly improving the accuracy of the detection result; and also facilitates medical anatomy, learning, etc.

Example 2

An embodiment of the present invention provides a method for converting an ultrasound image into a 3D model, as shown in fig. 2, including:

201. and acquiring a plurality of ultrasonic images to be converted.

In this embodiment, the ultrasound image to be converted is obtained from an ultrasound instrument, and is an ultrasound image of a fetal heart in 23 to 27 weeks during pregnancy. The number of the ultrasonic images to be converted is 1000.

202. And converting the format of the ultrasonic image to be converted, namely converting the original format of the ultrasonic image to be converted into a JPEG format.

Typically, the images acquired by the ultrasound apparatus are in voc format, comprising a series of cross-sectional images of the fetal heart from below to above and longitudinal sectional images from front to back and from left to right. In this embodiment, a bottom-up fetal heart cross-section image is selected and converted into an image withsize 243 × 200 and format JPEG by mics.

In addition, the format conversion of the ultrasound image to be converted in this embodiment further includes: after the pictures which do not contain the heart part are removed, the rest JPEG images are subjected to rotation, translation, scaling, mirror image, cutting and Gaussian noise method to amplify the ultrasonic image set samples to be converted. And labeling the fetal heart boundary manually by using LabelImg on the amplified ultrasound image set, and classifying labels into four categories of 'ht' (cardiac apical, top of heart), 'fcv' (four-chamber heart view, four chamber view), 'vot' (Vascular outflow tract), and 'tvv' (Three-vessel interface, Three vessels view) according to different heart parts contained in the fetal heart image. And generating an xml file according to the labeling information, making the xml file and the picture into a PASCAL VOC2007 data set format, and finally generating txt index files named as test, train and val so as to be used in the subsequent process.

203. And extracting the characteristic diagram of each ultrasonic image to be converted to obtain a characteristic diagram spectrum.

Specifically, each ultrasonic image to be converted with the size of 243 × 200 and the format of JPEG is sent to a convolution layer and passes through a group of conv + relu + posing layers to be used as a feature extractor to extract an image feature map.

204. Uniformly dividing k × H × W anchor regions on the feature map; wherein, the anchor regions have different scales, k is 9, H is the height of the feature map, and W is the width of the feature map.

205. And respectively corresponding each point on the characteristic diagram map with each anchor area to obtain corresponding coordinates.

Based on the above: and uniformly dividing the feature map into 9H W anchor regions, wherein each point on the feature map has 9 corresponding anchors, namely anchor coordinates with three length-width scales of 1: 1, 1: 2 and 2: 1, and the anchor coordinates can cover the coordinates of various sizes on the ultrasonic image to be converted.

206. And respectively sending the corresponding coordinates of each point on the feature map into a softmax classifier, and acquiring the category and the target detection frame of each point on the feature map.

207. Calculating offset, and optimizing the type and the target detection frame of each point on the feature map.

The formula of the loss function used in the embodiment of the present invention is shown below, and includes formulas (6), (7), (8), (9) and (10):

wherein i denotes the subscript of each sample, p_iAnd t_iVectors representing the ith probability and the four parameterized coordinates, respectively. While

And

vectors representing the ith probability and the four parameterized coordinates, respectively. N is a radical of_clsAnd N_regThe normalized sizes of the classification term and the regression term are respectively expressed, the parameters play a role in balancing two loss functions, and the specific parameters refer to the formula 10.

208. And extracting each point in the foreground class on the characteristic map and the target detection frame to form an image to be detected.

Specifically, the target detection frame mask is generated using equation (11).

In the formula: mask (x)_i，y_j) The values of the pixel points with coordinates (i, j) in the mask are represented, and x, y, w, h are the four parameterized coordinates of the prediction box.

The ultrasound image information of the fetal heart region can be extracted using equation 12:

masked(x_i，y_j)＝mask(x_i，y_j)*img(x_i，y_j) (12)

in the formula: masked (x)_i，y_j) A value indicating a pixel point of coordinates (i, j) in the image after extracting the fetal heart region from the mask, and img (x)_i，y_j) And (3) representing the value of a pixel point with coordinates (i, j) in the original image.

209. Obtaining an automatic segmentation image by adopting a threshold segmentation method; namely, each point in the image to be detected is segmented according to a preset threshold value, and an automatic segmentation image is obtained.

The threshold segmentation method comprises a global threshold segmentation method and a self-adaptive local threshold segmentation method; wherein the global threshold segmentation method comprises BINARY, TRUNC and TOZERO; the adaptive local threshold segmentation method includes MEAN _ C and GAUSSIAN _ C, and each segmentation method is described in more detail in embodiment 1 of the present invention, which will not be described herein again, and the TRUNC global segmentation method is selected in this embodiment.

And detecting the ultrasonic image to be converted after the format conversion to obtain the position, the area size, the score and the sequence of the fetal heart part of the image, and then transmitting the related information into a TRUNC global segmentation method based on the score sequence to obtain an automatic segmentation image of the fetal heart area.

210. And acquiring a 3D conversion model according to the automatic segmentation image.

The embodiment converts the automatic segmentation image into a 3D conversion model in a superposition mode. That is, since the interval d between two adjacent ultrasound images to be converted acquired by the ultrasound apparatus is 0.2mm, and since continuity exists between the ultrasound images to be converted, when d is sufficiently small, the images from the ith to the (i + 1) th can be considered to be the same. Specifically, as shown in formula 13:

model(x，y，z)＝dst_i(x，y)d_i＜z＜d_i+1 (13)

in the formula: model (x, y, z) represents the value of the 3D conversion model at (x, y, z) (a value of 0 indicates that the location is not part of the object, otherwise part of the object). dst_i(x, y) represents the i-th image segmentation result.

211. And optimizing the 3D conversion model by adopting a redrawing grid method.

After the 3D conversion model is completed, the connection of the 3D conversion model has a step-like interval difference, and in order to make the surface of the 3D conversion model smoother, the present embodiment adopts a method of redrawing a mesh for optimization.

Furthermore, the embodiment of the invention converts the format of the ultrasonic image to be detected, so that the application range is wider; the characteristic diagram atlas and the 3D conversion model are optimized, and the accuracy of the detection result is further improved.

Example 3

An embodiment of the present invention provides an apparatus for converting an ultrasound image into a 3D model, as shown in fig. 3, including:

the first acquiringunit 11 is configured to acquire a plurality of ultrasound images to be converted.

And the extractingunit 12 is configured to extract a feature map of each ultrasound image to be converted to obtain a feature map.

The second obtainingunit 13 is configured to obtain the category and the target detection frame of each point on the feature map according to the coordinates of each point on the feature map; the categories are classified into a foreground category and a background category.

And thecomposition unit 14 is used for extracting each point in the foreground class on the characteristic map and forming an image to be detected with the target detection frame.

A third obtainingunit 15, which obtains an automatic segmentation image by using a threshold segmentation method; namely, each point in the image to be detected is segmented according to a preset threshold value, and an automatic segmentation image is obtained.

Aconversion unit 16, configured to obtain a 3D conversion model according to the automatically segmented image.

The specific implementation manner, detailed technical information, and the like of each component in the embodiment of the present invention have been described in both embodiment 1 and embodiment 2, and are not described herein again, and can be automatically and correspondingly searched during implementation or understanding.

The invention provides an ultrasonic image conversion device, comprising: the first acquisition unit acquires a plurality of ultrasonic images to be converted; the extraction unit extracts a characteristic diagram of each ultrasonic image to be converted to obtain a characteristic diagram map; the second acquisition unit acquires the type and the target detection frame of each point on the characteristic diagram spectrum; the composition unit extracts each point in the foreground class on the characteristic diagram atlas and a target detection frame to form an image to be detected; the third acquisition unit acquires an automatic segmentation image by adopting a threshold segmentation method; the conversion unit acquires a 3D conversion model according to the automatic segmentation image and converts the 2D ultrasonic image into the 3D model, so that compared with the prior art, the method is more intuitive and is beneficial to doctor-patient communication; thereby greatly improving the accuracy of the detection result; and also facilitates medical anatomy, learning, etc.

Example 4

An embodiment of the present invention provides an apparatus for converting an ultrasound image into a 3D model, as shown in fig. 4, including:

the first obtainingunit 21 is configured to obtain a plurality of ultrasound images to be converted.

Aformat converter 22 for converting the original format of the ultrasound image into a JPEG format.

The extractingunit 23 is configured to extract a feature map of each ultrasound image to be converted to obtain a feature map.

The second obtainingunit 24 is configured to obtain categories and target detection frames of each point on the feature map according to coordinates of each point on the feature map; the categories are classified into a foreground category and a background category.

The divider 241: the characteristic map is used for uniformly dividing k H W anchor regions; wherein, the anchor regions have different scales, k is 9, H is the height of the feature map, and W is the width of the feature map.

The corresponding device 242: and the characteristic map is used for respectively corresponding each point on the characteristic map to each anchor area to obtain corresponding coordinates.

The classifier 243: and the coordinate detection device is used for classifying the corresponding coordinates of each point on the feature map respectively to obtain the category and the target detection frame of each point on the feature map.

The first optimizer 25: and the method is used for calculating the offset and optimizing the type and the target detection frame of each point on the feature map.

And thecomposition unit 26 is used for extracting each point in the foreground class on the characteristic map and forming an image to be detected with the target detection frame.

A third obtainingunit 27 that obtains an automatic segmentation image by using a threshold segmentation method; namely, each point in the image to be detected is segmented according to a preset threshold value, and an automatic segmentation image is obtained.

Aconversion unit 28 for obtaining a 3D conversion model from the automatically segmented image.

And thesecond optimizer 29 is used for optimizing the 3D conversion model by adopting a redrawing grid method.

Adisplay unit 210 for displaying the 3D conversion model.

Furthermore, the format converter is added to convert the original format of the ultrasonic image into the JPEG format, so that the application range is wider; calculating the offset by a first optimizer, and optimizing the category and the target detection frame of each point on the feature map; the second optimizer optimizes the 3D conversion model by adopting a redrawing grid method, and the display unit displays the 3D conversion model, so that the accuracy and the intuition of the detection result are further improved.

The above description is only for the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.