Disclosure of Invention
In order to solve the above problems, embodiments of the present invention provide a method and a system for extracting a remote sensing image road, which overcome the above problems or at least partially solve the above problems.
In a first aspect, an embodiment of the present invention provides a method for extracting a remote sensing image road, including:
predicting road nodes and connectivity among the nodes based on a trained deep convolutional neural network, wherein the deep convolutional neural network is obtained by carrying out supervised training on a high-resolution remote sensing image marked with a real road area;
and recovering the road topological structure based on the road nodes and the connectivity among the nodes so as to rebuild the road network.
Further, the predicting the road nodes and the connectivity among the nodes based on the trained deep convolutional neural network comprises:
predicting a road node significance graph and a node connectivity graph with the same size as the original image;
extracting road nodes from the road node significance map based on a non-maximum value inhibition method;
and predicting the connectivity among the road nodes according to the extracted road nodes and the node connectivity graph.
Further, the predicting the road node significance map and the node connectivity map with the same size as the original image comprises the following steps:
extracting road characteristics from the input remote sensing image based on a pre-trained coding structure;
the road node saliency map and the node connectivity map are predicted based on a decoding structure for road node prediction and a decoding structure for connectivity prediction.
Further, the loss function of the deep convolutional neural network is:
Loss=Lossnode+Lossconnect;
therein, LossnodeLoss function, Loss, which is a road node saliency mapconnectIs a supervised loss function of the node connectivity graph.
Further, the supervised loss function of the road node saliency map is:
wherein S is a predicted node significance map,
for the significance map of the real node, p is each pixel position on the significance map, and N is the number of all pixels.
Further, the supervised loss function of the node connectivity graph is:
wherein C is a predicted node connectivity graph,
for the significance map of the real nodes, p is each pixel position on the connectivity map, and N is the number of all pixels.
Further, the restoring the road topology structure based on the road nodes and the connectivity between the nodes to reconstruct the road network includes:
deducing a set of edges required for constructing a road network based on a preset node connection algorithm;
reconstructing a road topology based on the set of edges to reconstruct the road network.
In a second aspect, an embodiment of the present invention further provides a device for extracting a remote sensing image road, including:
the prediction module is used for predicting road nodes and connectivity among the nodes based on the trained deep convolutional neural network, and the deep convolutional neural network is obtained by carrying out supervised training on the high-resolution remote sensing image marked with the real road area;
and the road network reconstruction module is used for recovering the road topological structure based on the road nodes and the connectivity among the nodes so as to reconstruct the road network.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
a processor, a memory, a communication interface, and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the memory stores program instructions which can be executed by the processor, and the processor calls the program instructions to execute the remote sensing image road extraction method.
In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, where the computer instructions cause the computer to execute the method for extracting a road from a remote sensing image.
The method and the device for extracting the road based on the remote sensing image provided by the embodiment of the invention have the advantages that the nodes with strong connectivity are connected, the continuity of the extracted road is obviously improved, the road node significance map and the connectivity map generated by the small road and the wider road have no difference, the road node significance map and the connectivity map are treated equally during training, the effect on the extraction of the small road is better, and the accuracy of the finally obtained road extraction result is obviously improved.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some embodiments, but not all embodiments, of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic flow chart of a remote sensing image road extraction method provided by an embodiment of the present invention, as shown in fig. 1, including:
101. predicting road nodes and connectivity among the nodes based on a trained deep convolutional neural network, wherein the deep convolutional neural network is obtained by carrying out supervised training on a high-resolution remote sensing image marked with a real road area;
102. and recovering the road topological structure based on the road nodes and the connectivity among the nodes so as to rebuild the road network.
It should be noted that, in order to overcome the defects of the road extraction method in the prior art, the embodiment of the invention preferably designs a remote sensing image road extraction method based on road node prediction and connectivity estimation. Compared with the traditional method for extracting roads based on semantic segmentation, the method uses an undirected graph G (V, E) to represent the road network, wherein V represents a set of road nodes, and E represents a set of edges of connecting nodes required by constructing the road network. According to the embodiment of the invention, characteristics easy to classify can be directly learned from training data through end-to-end training of the convolutional neural network, the significance graph of the road node V and the connectivity graph between the nodes are predicted, and the connectivity between the nodes is deduced through the connectivity graph
The prediction of the value of V is made,
we have then devised a road node connection algorithm, which is based on V,
road topological structure is rebuilt to form a road network.
Specifically, instep 101, the embodiment of the present invention trains a convolutional neural network, learns features that are easy to classify from training data, and predicts a saliency map of road nodes and a connectivity map between nodes. After the node significance map is predicted by the network, non-maximum value suppression is adopted to obtain road nodes, and the connectivity among the nodes is predicted according to the node connectivity map.
Further, instep 102, according to the inferred road node sets and node connectivity, the embodiment of the present invention designs a node connection algorithm to infer the sets of edges required for constructing the road network, and reconstructs a road topology structure to form the road network.
The method for extracting the road based on the remote sensing image provided by the embodiment of the invention has the advantages that the nodes with strong connectivity are connected, the continuity of the extracted road is obviously improved, the road node significance map and the connectivity map generated by the small road and the wide road have no difference, the road node significance map and the connectivity map are treated equally during training, the effect on the extraction of the small road is better, and the accuracy of the finally obtained road extraction result is obviously improved.
On the basis of the above embodiment, the predicting the road nodes and the connectivity between the nodes based on the trained deep convolutional neural network includes:
predicting a road node significance graph and a node connectivity graph with the same size as the original image;
extracting road nodes from the road node significance map based on a non-maximum value inhibition method;
and predicting the connectivity among the road nodes according to the extracted road nodes and the node connectivity graph.
The embodiment of the invention firstly utilizes a pre-trained coding structure to extract road characteristics from an input remote sensing image. And predicting the road node significance graph and the connectivity graph between the nodes according to the extracted features by using a decoding structure for road node prediction and a decoding structure for connectivity prediction.
And then extracting road nodes according to the predicted road node significance map by utilizing non-maximum value inhibition. Specifically, in the node saliency map, if the value of the node p is a maximum value within the 10 × 10 neighborhood, p is a road node.
Then according to the extracted road node set V ═ V
1,v
2,...v
n}(v
iRepresenting the ith node in the node set), and estimating the connectivity between the nodes by combining the node connectivity graph obtained by network prediction, namely whether edges are connected between the nodes,
represents v
i,v
jThe connection is carried out by the edges,
representing a borderless connection.
Denotes v
i,v
jThe connectability between is calculated as follows:
d (u) denotes a position from viDirection vjPosition on the straight line segment of di,djDenotes vi,vjThe position on the image, C, represents the node connectivity graph predicted by the network.
The calculation is as follows:
on the basis of the above embodiment, the predicting the road node saliency map and the node connectivity map with the same size as the original image includes:
extracting road characteristics from the input remote sensing image based on a pre-trained coding structure;
the road node saliency map and the node connectivity map are predicted based on a decoding structure for road node prediction and a decoding structure for connectivity prediction.
It can be known from the content of the above embodiment that the embodiment of the present invention firstly needs to extract the road features of the remote sensing image. Specifically, a remote sensing image is used as input, and a pre-training residual error network (ResNet) basic network with a full connection layer removed is used as a main network. In the remote sensing image, the extraction of road information is extremely dependent on context information, and in order to acquire larger receptive field information, a plurality of layers of cavity convolution operations are added behind a basic network, so that each neuron of a convolution neural network can have a receptive field with the size of a global image. Through a series of convolution and down-sampling operations, the resolution is finally obtained
Wherein W and H represent the width and height of the original input remote sensing image, respectively. Then the obtained image road characteristics (the resolution is
) The feature map of (1) is inputted with a decoding structure for road node prediction and a decoding structure for node connectivity map prediction, and a road node saliency map and a node connectivity map of resolution wxh are outputted.
On the basis of the above embodiment, the loss function of the deep convolutional neural network is as follows:
Loss=Lossnode+Lossconnect;
Lossnodeloss function, Loss, which is a road node saliency mapconnectIs a supervised loss function of the node connectivity graph.
This is a loss function that is finally defined in view of predicting the node significance map and the connectivity map at the same time.
On the basis of the above embodiment, the supervised loss function of the road node saliency map is:
wherein S is a predicted node significance map,
for the significance map of the real node, p is each pixel position on the significance map, and N is the number of all pixels.
On the basis of the above embodiment, the supervision loss function of the node connectivity graph is:
wherein C is a predicted node connectivity graph,
for the significance map of the real nodes, p is each pixel position on the connectivity map, and N is the number of all pixels.
On the basis of the foregoing embodiment, the restoring a road topology structure based on the road nodes and connectivity between the nodes to reconstruct a road network includes:
deducing a set of edges required for constructing a road network based on a preset node connection algorithm;
reconstructing a road topology based on the set of edges to reconstruct the road network.
From the above description of the embodiments, it can be seen that the embodiments of the present invention need to be based on the inferred road node set V and the node connectivity
Node connection algorithm is designed to infer the set of edges needed to build a road networkAnd E, reconstructing a road topological structure to form a road network.
Specifically, E { } is initialized first, and then v is set for each road node
iLooking for distances from it less than d
TRoad node set V
DWill V
DThe nodes in the interior are arranged in descending order of distance, for V
DEach node v in
jEstimating its connectivity to the current node
If it is
And edge { v
i,v
jIf there is no crossing relation with any edge in E, add { v }
i,v
jTo E. Finally, repeating the above sequencing and adding process until V
DAll the points in (1) are processed.
Fig. 2 is a test result of the remote sensing image road extraction method based on a depglobe (global satellite image) data set according to the embodiment of the present invention; fig. 3 is a test result of the remote sensing image road extraction method based on the spaenet data set according to the embodiment of the invention. Among them, SpaceNet is a collection of remote sensing images provided by DigitalGlobe commercial satellite Inc. Referring to fig. 2 and 3, the invention performs road extraction based on remote sensing images, and the obtained road network result is shown in fig. 2 and 3, for example, it can be seen that the invention can stably and accurately perform road extraction on remote sensing images in different scenes.
By combining the above processes, the method provided by the embodiment of the invention has the following three advantages:
1. end-to-end training without post-processing
The road network extraction is carried out by predicting the road nodes and the node connectivity, the road topological structure extraction and the road network construction can be directly carried out, any post-processing such as skeletonization and road element connection is not needed, and the quality of the extracted road network is high. In addition, the network designed by the embodiment of the invention has simple structure, low complexity and easy training.
2. The road network has high integrity and good road topology quality
According to the embodiment of the invention, the predicted road nodes are subjected to connectivity estimation, and the nodes with strong connectivity are connected, so that the continuity of the extracted road is obviously improved, and the extracted result has fewer broken results. In addition, the graph representation of the road is also helpful for extracting the road topology.
3. The fine road extraction effect is better
For the road extraction method based on semantic segmentation, the extraction result of the fine road is poor because the proportion of the fine road on the image is low. The embodiment of the invention is different from a semantic segmentation method, the road node significance map and the connectivity map generated by the small road and the wider road have no difference, and are treated equally during training, the effect on extracting the small road is better, and the precision of the finally obtained road extraction result is obviously improved.
Fig. 4 is a schematic structural diagram of a remote sensing image road extraction device according to an embodiment of the present invention, as shown in fig. 4, including: aprediction module 201 and a roadnetwork reconstruction module 202, wherein:
theprediction module 201 is configured to predict road nodes and connectivity among the nodes based on a trained deep convolutional neural network, where the deep convolutional neural network is obtained by performing supervised training on a high-resolution remote sensing image labeled with a real road region;
the roadnetwork reconstruction module 202 is configured to restore a road topology structure based on road nodes and connectivity between the nodes, so as to reconstruct a road network.
Specifically, theprediction module 201 and the roadnetwork reconstruction module 202 may be used to execute the technical solution of the embodiment of the method for extracting a remote sensing image road shown in fig. 1, and the implementation principle and the technical effect are similar, which are not described herein again.
The remote sensing image-based road extraction device provided by the embodiment of the invention has the advantages that the nodes with strong connectivity are connected, the continuity of the extracted road is obviously improved, the road node significance map and the connectivity map generated by the small road and the wide road have no difference, the road node significance map and the connectivity map are treated equally during training, the effect on the extraction of the small road is better, and the accuracy of the finally obtained road extraction result is obviously improved.
On the basis of the above embodiment, the predicting the road nodes and the connectivity between the nodes based on the trained deep convolutional neural network includes:
predicting a road node significance graph and a node connectivity graph with the same size as the original image;
extracting road nodes from the road node significance map based on a non-maximum value inhibition method;
and predicting the connectivity among the road nodes according to the extracted road nodes and the node connectivity graph.
On the basis of the above embodiment, the predicting the road node saliency map and the node connectivity map with the same size as the original image includes:
extracting road characteristics from the input remote sensing image based on a pre-trained coding structure;
the road node saliency map and the node connectivity map are predicted based on a decoding structure for road node prediction and a decoding structure for connectivity prediction.
On the basis of the above embodiment, the loss function of the deep convolutional neural network is as follows:
Loss=Lossnode+Lossconnect;
Lossnodeloss function, Loss, which is a road node saliency mapconnectIs a supervised loss function of the node connectivity graph.
On the basis of the above embodiment, the supervised loss function of the road node saliency map is:
wherein S is a predicted node significance map,
for the significance map of the real node, p is each pixel position on the significance map, and N is the number of all pixels.
On the basis of the above embodiment, the supervision loss function of the node connectivity graph is:
wherein C is a predicted node connectivity graph,
for the significance map of the real nodes, p is each pixel position on the connectivity map, and N is the number of all pixels.
On the basis of the foregoing embodiment, the restoring a road topology structure based on the road nodes and connectivity between the nodes to reconstruct a road network includes:
deducing a set of edges required for constructing a road network based on a preset node connection algorithm;
reconstructing a road topology based on the set of edges to reconstruct the road network.
Fig. 5 is a block diagram of an electronic device according to an embodiment of the present invention, and referring to fig. 5, the electronic device includes: a processor (processor)301, a communication Interface (communication Interface)302, a memory (memory)303 and abus 304, wherein theprocessor 301, thecommunication Interface 302 and thememory 303 complete communication with each other through thebus 304.Processor 301 may call logic instructions inmemory 303 to perform the following method: predicting road nodes and connectivity among the nodes based on a trained deep convolutional neural network, wherein the deep convolutional neural network is obtained by carrying out supervised training on a high-resolution remote sensing image marked with a real road area; and recovering the road topological structure based on the road nodes and the connectivity among the nodes so as to rebuild the road network.
An embodiment of the present invention discloses a computer program product, which includes a computer program stored on a non-transitory computer readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer can execute the methods provided by the above method embodiments, for example, the method includes: predicting road nodes and connectivity among the nodes based on a trained deep convolutional neural network, wherein the deep convolutional neural network is obtained by carrying out supervised training on a high-resolution remote sensing image marked with a real road area; and recovering the road topological structure based on the road nodes and the connectivity among the nodes so as to rebuild the road network.
Embodiments of the present invention provide a non-transitory computer-readable storage medium, which stores computer instructions, where the computer instructions cause the computer to perform the methods provided by the above method embodiments, for example, the methods include: predicting road nodes and connectivity among the nodes based on a trained deep convolutional neural network, wherein the deep convolutional neural network is obtained by carrying out supervised training on a high-resolution remote sensing image marked with a real road area; and recovering the road topological structure based on the road nodes and the connectivity among the nodes so as to rebuild the road network.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to each embodiment or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.