GEOGRAPHIC DATA PROCESSING METHODS AND SYSTEMS FOR DETECTING ENCROACHMENT BY OBJECTS INTO A GEOGRAPHIC CORRIDOR
TECHNICAL FIELD
The present disclosure relates to geographic data processing methods and systems. In particular, the present disclosure relates to methods and systems for detecting encroachment by objects into a geographic corridor. BACKGROUND
Third party encroachment is a major threat to geographic corridors such as pipelines. Managing the risk from such encroachment is a real challenge to pipeline operators. If this threat is not properly managed, the consequences can lead to major catastrophe which impacts public safety, asset damage, financial loss and tarnish a company’s reputation.
On top of ground patrolling, helicopter and autonomous drone surveillance have been widely used for pipeline aerial inspection and monitoring of third party encroachment threats. The data produced from these inspections are in the forms of aerial photos and videos. However, this can result in massive datasets which must then be manually analyzed to identify the threats captured along the pipeline length. The processes are time consuming and costly due to manual interpretation involving massive datasets, many resources and prone to human error during data analysis.
SUMMARY
According to a first aspect of the present disclosure a method of detecting encroachment by objects into a geographic corridor from geographic data is provided. The method comprises: receiving current geographic data comprising current image data of a geographic area comprising the geographic corridor and location tag data; generating a plurality of sets of machine learning object detection results from the current image data, each set of machine learning object detection results being generated using a different machine learning object detection methodology; combining the plurality of sets of machine learning object detection results to generate a combined set of machine learning object detection results; filtering the combined set of machine learning object detection results using location tag data associated with machine learning object detection results of the combined set of machine learning object detection results to generate a set of machine learning encroachment detections; and generating a set of encroachment detections comprising the set of machine learning encroachment detections.
In an embodiment, the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated from the current image data at a first resolution and a set of a set of machine learning object detection results generated from the current image data at a second resolution.
In an embodiment, the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated with a first object detection algorithm and a set of object detection results generated with a second object detection algorithm.
In an embodiment, the set of encroachment detections comprises a set of confidence indications determined from the number of sets of machine learning object detection results from plurality of sets of machine learning object defection results in which each encroachment detection of the set of encroachment detections occurs.
In an embodiment, the method further comprises: receiving historicai orthophoto data comprising historicai geographic data of the geographic area comprising the geographic corridor, the historic geographic data comprising historical image and location tag data; detecting changes between the historical image data and the current image data to generate a set of change detection results; and filtering the set change detection results using location tag data associated with the set of change detection results to generate a set of change detection encroachment detections, wherein the set of encroachment detections further comprises the set of change detection encroachment detections. in an embodiment, wherein detecting changes between the historical image data and the current image data to generate a set of change detection results comprises appiying a piurality of image comparison methodologies to the historical image data and the current image data.
In an embodiment, the method further comprises accessing a database storing information on mapped objects and locations corresponding to the mapped objects, and filtering the mapped objects to generate a set of mapped object detections comprising mapped objects having a location overlapping with the geographic corridor, wherein the set of encroachment detections further comprises the set of mapped object detections.
In an embodiment, the method further comprises: receiving land lot information indicating an expected position of a target object in the geographic corridor; identifying an actual position of the target object in the geographic corridor; and determining a deviation between the actual position of the target object and the actual position of the target object in the geographic corridor.
In an embodiment, the method further comprises generating a risk score for the geographic corridor using the set of encroachment detections.
According to a third aspect of the present disclosure a computer readable medium storing processor executable instructions which when executed on a processor cause the processor to carry out a method as set out above is provided.
According to a third aspect of the present disclosure, a geographic data processing system for detecting encroachment by objects into a geographic corridor is provided. The geographic data processing system comprises: a processor and a data storage device storing computer program instructions operable to cause the processor to: receive current geographic data comprising current image data of a geographic area comprising the geographic corridor and location tag data; generate a plurality of sets of machine learning object detection results from the current image data, each set of machine learning object detection results being generated using a different machine learning object detection methodology; combine the plurality of sets of machine learning object detection results to generate a combined set of machine learning object detection results; filter the combined set of machine learning object detection results using location tag data associated with machine learning object detection results of the combined set of machine learning object detection results to generate a set of machine learning encroachment detections; and generate a set of encroachment detections comprising the set of machine learning encroachment detections. in an embodiment, the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated from the current image data at a first resolution and a set of a set of machine learning object detection results generated from the current image data at a second resolution.
In an embodiment, the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated with a first object detection algorithm and a set of object detection results generated with a second object detection algorithm.
In an embodiment, the set of encroachment detections comprises a set of confidence indications determined from the number of sets of machine learning object detection results from plurality of sets of machine learning object detection results in which each encroachment detection of the set of encroachment defections occurs. in an embodiment, the data storage device further stores computer program instructions operable to cause the processor to: receive historical orthophoto data comprising historical geographic data of the geographic area comprising the geographic corridor, the historic geographic data comprising historical image and location tag data; detect changes between the historical image data and the current image data to generate a set of change detection results; and filter the set change detection results using location tag data associated with the set of change detection results to generate a set of change detection encroachment detections, wherein the set of encroachment detections further comprises the set of change detection encroachment detections. in an embodiment, the data storage device further stores computer program instructions operable to cause the processor to: detect changes between the historical image data and the current image data to generate a set of change detection results by applying a plurality of image comparison methodologies to the historical image data and the current image data.
In an embodiment, the data storage device further stores computer program instructions operable to cause the processor to: access a database storing information on mapped objects and locations corresponding to the mapped objects, and filter the mapped objects to generate a set of mapped object detections comprising mapped objects having a location overlapping with the geographic corridor, wherein the set of encroachment detections further comprises the set of mapped object detections.
In an embodiment, the data storage device further stores computer program instructions operable to cause the processor to: receive land lot information indicating an expected position of a target object in the geographic corridor; identify an actual position of the target object in the geographic corridor; and determine a deviation between the actual position of the target object and the actual position of the target object in the geographic corridor.
In an embodiment, the data storage device further stores computer program instructions operable to cause the processor to: generate a risk score for the geographic corridor using the set of encroachment detections. Further embodiments of the present invention are set out in the following clauses:
1 , A method of detecting encroachment by objects into a geographic corridor from geographic data, the method comprising: receiving current geographic data comprising current image data of a geographic area comprising the geographic corridor and location tag data; generating a plurality of sets of machine learning object detection results from the current image data, each set of machine learning object detection results being generated using a different machine learning object detection methodology; combining the plurality of sets of machine learning object detection results to generate a combined set of machine learning object detection results; filtering the combined set of machine learning object defection results using location tag data associated with machine learning object detection results of the combined set of machine learning object detection results to generate a set of machine learning encroachment detections; and generating a set of encroachment detections comprising the set of machine learning encroachment detections. 2. A method according to clause 1, wherein the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated from the current image data at a first resolution and a set of a set of machine learning object detection results generated from the current image data at a second resolution.
3. A method according to clause 1 or clause 2, wherein the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated with a first object detection algorithm and a set of object detection results generated with a second object detection algorithm.
4. A method according to any one of clauses 1 to 3, wherein the set of encroachment detections comprises a set of confidence indications determined from the number of sets of machine learning object detection results from plurality of sets of machine learning object detection results in which each encroachment detection of the set of encroachment detections occurs.
5. A method according to clause 1 , further comprising: receiving historical orthophoto data comprising historical geographic data of the geographic area comprising the geographic corridor, the historic geographic data comprising historical image and location tag data; detecting changes between the historical image data and the current image data to generate a set of change detection results; and filtering the set change detection results using location tag data associated with the set of change detection results to generate a set of change detection encroachment detections, wherein the set of encroachment detections further comprises the set of change detection encroachment detections.
6. A method according to clause 5, wherein detecting changes between the historical image data and the current image data to generate a set of change detection results comprises applying a plurality of image comparison methodologies to the historical image data and the current image data.
7. A method according to any preceding clause, further comprising accessing a database storing information on mapped objects and locations corresponding to the mapped objects, and filtering the mapped objects to generate a set of mapped object detections comprising mapped objects having a location overlapping with the geographic corridor, wherein the set of encroachment detections further comprises the set of mapped object detections,
8. A method according to any preceding clause, further comprising: receiving land lot information indicating an expected position of a target object in the geographic corridor; identifying an actual position of the target object in the geographic corridor; and determining a deviation between the actual position of the target object and the actual position of the target object in the geographic corridor.
9. A method according to any preceding clause, further comprising generating a risk score for the geographic corridor using the set of encroachment detections.
10. A computer readable medium storing processor executable instructions which when executed on a processor cause the processor to carry out a method according to any one of clauses 1 to 9.
11. A geographic data processing system for detecting encroachment by objects into a geographic corridor, the geographic data processing system comprising: a processor and a data storage device storing computer program instructions operable to cause the processor to: receive current geographic data comprising current image data of a geographic area comprising the geographic corridor and location tag data; generate a plurality of sets of machine learning object detection results from the current image data, each set of machine learning object detection results being generated using a different machine learning object detection methodology; combine the plurality of sets of machine learning object detection results to generate a combined set of machine learning object detection results; filter the combined set of machine learning object detection results using location tag data associated with machine learning object detection results of the combined set of machine learning object detection results to generate a set of machine learning encroachment detections; and generate a set of encroachment detections comprising the set of machine learning encroachment detections.
12. A geographic data processing system according to clause 11, wherein the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated from the current image data at a first resolution and a set of a set of machine learning object detection results generated from the current image data at a second resolution.
13. A geographic data processing system according to clause 11 or clause 12, wherein the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated with a first object detection algorithm and a set of object detection results generated with a second object detection algorithm,
14. A geographic data processing system according to any one of clauses 11 to 13, wherein the set of encroachment detections comprises a set of confidence indications determined from the number of sets of machine learning object detection results from plurality of sets of machine learning object detection results in which each encroachment detection of the set of encroachment detections occurs. 15. A geographic data processing system according to clause 11 , wherein the data storage device further stores computer program instructions operable to cause the processor to: receive historical orthophoto data comprising historical geographic data of the geographic area comprising the geographic corridor, the historic geographic data comprising historical image and location tag data; detect changes between the historical image data and the current image data to generate a set of change detection results; and filter the set change detection results using location tag data associated with the set of change detection results to generate a set of change detection encroachment detections, wherein the set of encroachment detections further comprises the set of change detection encroachment detections. 18. A geographic data processing system according to clause 15, wherein the data storage device further stores computer program instructions operable to cause the processor to: detect changes between the historical image data and the current image data to generate a set of change detection results by applying a plurality of image comparison methodologies to the historical image data and the current image data.
17. A geographic data processing system according to any one of clauses 11 to 16 wherein the data storage device further stores computer program instructions operable to cause the processor to: access a database storing information on mapped objects and locations corresponding to the mapped objects, and filter the mapped objects to generate a set of mapped object detections comprising mapped objects having a location overlapping with the geographic corridor, wherein the set of encroachment detections further comprises the set of mapped object detections.
18. A geographic data processing system according to any one of clauses 11 to 17 wherein the data storage device further stores computer program instructions operable to cause the processor to: receive land lot information indicating an expected position of a target object in the geographic corridor; identify an actual position of the target object in the geographic corridor; and determine a deviation between the actual position of the target object and the actual position of the target object in the geographic corridor,
19. A geographic data processing system according to any one of clauses 11 to 18 wherein the data storage device further stores computer program instructions operable to cause the processor to: generate a risk score for the geographic corridor using the set of encroachment detections,
BRIEF DESCRIPTION OF THE DRAWINGS
In the following, embodiments of the present invention will be described as non-limiting examples with reference to the accompanying drawings in which:
FIG.1 is a block diagram showing a geographic data processing system according to an embodiment of the present invention;
FIG.2 is a flow chart showing method of processing geographic data to detect encroachment by objects into a geographic corridor according to an embodiment of the present invention;
FIG.3 is a flow chart showing method of processing geographic data to detect encroachment by objects into a geographic corridor according to an embodiment of the present invention;
FIG.4 is a flowchart showing an overview of processing in a method according to an embodiment of the present invention;
FIG.5 is a flowchart showing a machine learning workflow in a method according to an embodiment of the present invention;
FIG.6 is a flowchart showing processing of mapped object data in a method according to an embodiment of the present invention; FIG.7 is a flowchart showing an image processing workflow in a method according to an embodiment of the present invention; and
FIG.8 is a flowchart showing an image processing workflow in a method according to an embodiment of the present invention.
DETAILED DESCRIPTION
The present disclosure relates to the autonomous detection of third-party threats such as intrusion, encroachment, and crossing of a geographic corridor such a pipeline from aerial visual imagery collected by aerial vehicle survey or satellite imagery. The system may be capable of detecting the following: vehicles and machinery within and beside the geographic corridor, buildings or structures encroaching or beside the geographic corridor, any suspicious changes within the geographic corridor such as foreign objects, and utilities such as transmission towers within or encroaching the geographic corridor. The system may also be configured for the following: identification of location class through counting the number of buildings within the geographic corridor, deviations of objects such as pipelines from their land lots, and detection of roads or waterways crossing the geographic corridor.
FIG.1 shows a geographic data processing system according to an embodiment of the present invention. The geographic data processing system 100 is a computer system with memory that stores computer program modules which implement geographic data processing methods according to embodiments of the present invention.
The geographic data processing system 100 comprises a processor 110, a working memory 112, an input interface 114, a user interface 116, an output interface 118, program storage 120 and data storage 140. The processor 110 may be implemented as one or more central processing unit (CPU) chips. The program storage 120 is a non-volatile storage device such as a hard disk drive which stores computer program modules. The computer program modules are loaded into the working memory 112 for execution by the processor 110. The input interface 114 is an interface which allows data, such as image data to be received by the geographic data processing system 100. The input interface 114 may be a wireless network interface such as a Wi-Fi or Bluetooth interface, alternatively it may be a wired interface. The user interface 116 allows a user of the geographic data processing system 100 to input selections and commands and may be implemented as a graphical user interface. The output interface 118 outputs data and may be implemented as a display or a data interface.
The program storage 120 stores a machine learning object detection module 122, an image processing change detection module 124, a mapped object detection module 126, a land lot deviation detection module 128, an encroachment detection module 130, and a risk score calculation module 132. The computer program modules cause the processor 110 to execute various geographic data processing which is described in more detail below. The program storage 120 may be referred to in some contexts as computer readable storage media and/or non-transitory computer readable media. As depicted in FIG.1 , the computer program modules are distinct modules which perform respective functions implemented by the geographic data processing system 100. It will be appreciated that the boundaries between these modules are exemplary only, and that alternative embodiments may merge modules or impose an alternative decomposition of functionality of modules. For example, the modules discussed herein may be decomposed into sub-modules to be executed as multiple computer processes, and, optionally, on multiple computers. Moreover, alternative embodiments may combine multiple instances of a particular module or sub-module. It will also be appreciated that, while a software implementation of the computer program modules is described herein, these may alternatively be implemented as one or more hardware modules (such as field-programmable gate array(s) or application-specific integrated circuit(s)) comprising circuitry which implements equivalent functionality to that implemented in software.
The data storage 140 stores machine learning model 150. The machine learning models 150 comprise multiple models which are trained for image recognition. As shown in FIG.1 , the machine learning models 150 comprise a first model for recognizing a first object 152, a second model for recognizing the first object 154, a first model for recognizing a second object 156 and a second model for recognizing the second object 158. The geographic data processing system 100 is coupled to a first object map database 160 and a second object map database 162. The first object map database 160 and the second object map database 162 are databases which store labelled maps of objects such as roads, utility towers and transmission lines.
FIG.2 is a flow chart showing method of processing geographic data to detect encroachment by objects into a geographic corridor according to an embodiment of the present invention. The method 200 shown in FIG.2 is carried out by the geographic data processing system 100 shown in FIG.1.
In step 202, the geographic data processing system 100 receives current geographic data. The current geographic data is received by the input interface 114 of the geographic data processing system 100. The geographic data comprises current image data of a geographic corridor and location tag data which identifies the locations of the current image data. The current geographic data may comprise orthophoto data, or any image captured from above data such as satellite imagery, or drone images and videos.
In step 204, the machine learning object detection module 122 is executed by the processor 110 of the geographic data processing system 100 to generate a plurality of sets of machine learning object detection results from the current image data. Each of the sets of machine learning object detection results is generated using a different machine learning methodology. For example, different machine learning object detection results may be generated by using different machine learning models stored in the data storage 140. Thus, a first set of machine learning object detection results may be generated using the first machine learning model for the first object 152 and a second set of machine learning object detection results may be generated using the second machine learning model for the first object 156. Further, different machine learning object detection results may be generated by converting the current image data to different resolutions and applying the machine learning models to the different resolutions of image data. In step 204, multiple sets of machine learning object detection results are generated, these sets of machine learning object detections may comprise duplicate detections corresponding to the same object being detected by different methodologies. In step 206, the machine learning object detection module 122 is executed by the processor 110 of the geographic data processing system 100 to combine the plurality of sets of machine learning object detection results to obtain a combined set of machine learning object detection results. In step 206, the combined results may be given a confidence indication depending on the number object detection sets from the plurality of object detection sets in which a detection occurs. Thus, if an object is detected by multiple machine learning object detection methodologies then that detection is given a high confidence indication.
In step 208, the machine learning object detection module 122 is executed by the processor 110 of the geographic data processing system 100 to filter the machine learning detections based on location. As described above, the location tag data indicates locations associated with the current image data. A set of machine learning encroachment detections is generated by filtering the combined set of machine learning object detection results based on their location. Object detection results having a location corresponding to the geographic corridor are identified as being in the set of machine learning encroachment detections. In step 210, the encroachment detection module 130 is executed by the processor 110 of the geographic data processing system 100 to generate a set of encroachment detections. The set of encroachment detections may correspond to the set of machine learning encroachment detections. In some embodiments as described more detail below, the set of encroachment detections may include detections generated by other detection methods.
In step 212, the risk score calculation module 132 is executed by the processor 110 of the geographic data processing system to calculate a risk score for the geographic corridor using the set of encroachment detections. The final list of detections are passed to a rule-based engine that consumes the detections and their meta data (such as the type of detection such as vehicle, building, road crossing, and machinery) to produce a final risk score for each area of interest segment (for example a pipe segment). The geographic corridor may be divided into segments (i.e. each 1km is one segment), the goal is to come up with a final third-party threat score for each segment based on all the detections from drone surveys and manual patrolling. The score may be used to provision resources to each segment for further routine inspections patrolling. For example, the score may be calculated according to the following rules: if a segment is found to have machinery (e.g. excavator) within 20m from a pipeline, then the risk score is increased by 10, if there is a highway crossing the segment then the risk score is increased by 2; if there is a 1-lane road crossing the segment, then the risk score is increased by 5; if there are objects (such as buildings, garbage, etc.) detected within pipeline right of way, then the risk score is increased by 10; if there are industrial facilities within 50m from pipeline then the risk score is increased by 5; if the segment is within a predefined location class A (more than 10 buildings within 200m) then the risk score is increased by 10. The final score represents an estimate of the chance of third-party encroachments/intrusion or damage to pipeline.
FIG.3 is a flow chart showing method of processing geographic data to detect encroachment by objects into a geographic corridor according to an embodiment of the present invention. The method 300 shown in FIG.3 includes the steps of the method 200 described above with reference to FIG.2 and also includes additional steps which allow additional processing to influence the encroachment detections and the risk score. The method 300 shown in FIG.3 is carried out by the geographic data processing system 100 shown in FIG.1.
In step 302, the geographic data processing system 100 receives current geographic data. The current geographic data is received by the input interface 114 of the geographic data processing system 100. The geographic data comprises current image data of a geographic corridor and location tag data which identifies the locations of the current image data. The current geographic data may comprise orthophoto data, or any image captured from above such as satellite imagery, or drone images and videos.
In step 304, the machine learning object detection module 122 is executed by the processor 110 of the geographic data processing system 100 to perform machine learning object detection. Step 304 corresponds to steps 204, 206 and 208 described above with reference to FIG.2, and the output is a set of machine learning encroachment detections.
In step 306, the geographic data processing system 100 receives historic geographic data. The historic geographic data comprises historic image data of the geographic corridor and location tag data which identifies the locations of the historic image data. The historic geographic data may comprise orthophoto data, or any image captured from above such as satellite imagery, or drone images and videos. In step 308, the image processing change detection module is executed by the processor 100 of the geographic data processing system 100 to perform change detection on the geographic data to generate a set of change detection results. As will be described in more detail below, the change detection results are generated by comparing the current image data with the historic image data.
In step 310, the mapped object detection module 126 is executed by the processor 100 of the geographic data processing system 100 to access mapped object data from the first object map database 160 and the second object map database 162. This returns indications of mapped objects within the geographic corridor.
In step 312, the mapped object detection module 126 is executed by the processor 100 of the geographic data processing system 100 to generate mapped object detections. The mapped object detections are a set of indications of mapped objects which are located in the geographic corridor.
In step 314, the geographic data processing system 100 receives land lot information. The land lot information comprises indications expected positions of a target object in the geographic corridor. For example, the land lot information may comprise indications of land purchased as a corridor for a pipeline, in which case, the pipeline would be expected to be positioned within the corridor for the pipeline.
In step 316, the land lot deviation module 128 is executed by the processor 110 of the geographic data processing system to determine a deviation between an actual position of the target object in the geographic corridor and an expected position from the land lot information received in step 314.
In step 318, the encroachment detection module 130 is executed by the processor 110 of the geographic data processing system 100 to generate a set of encroachment detections. The set of encroachment detections is generated from the set of machine learning encroachment detections, the set of change detection results, and the set of mapped object detections. In step 320, the risk score calculation module 132 is executed by the processor 110 of the geographic data processing system 100 to generate a risk score based on the encroachment detections. The risk score may be output from the output interface 118 of the geographic data processing system. FIG.4 is a flowchart showing an overview of processing in a method according to an embodiment of the present invention. The processing 400 shown in FIG.4 is carried out by the geographic data processing system 100 shown in FIG.1 and corresponds to parts of the methods 200 and 300 shown in FIG.2 and FIG.3. The inputs to the processing 400 are image data 410, pipeline coordinates 412 and land lot data 414. In this example, the processing 400 relates to a pipeline. The image data 410 comprises images captured of the geographic corridor including the pipeline and may comprise orthophoto data including tags which identify the geographic locations of the image data. The pipeline coordinates 412 comprises a set of coordinates of the pipeline. The land lot data 414 comprises a set of indications of the planned locations of the pipeline and the land lots purchased for the pipeline.
The image data 416 is subjected to pre-processing 416 which includes image normalization, image scaling, band arrangement, format transformation. Then the pre- processed image data is subjected to three parallel processing streams: a machine learning workflow 420, data mining 430 and computer vision 440. In the machine learning workflow 420 machine learning object detection results are generated. The machine learning workflow 420 is followed by a morphology 422 step in which the detections from the machine learning workflow are cleaned. This is followed by a vectorization 424 step in which the detections are converted into polygons. In the data mining 430, mapped object detections are generated by querying the first object map database 160 and the second object map database 162. In the computer vision 440 processing, change detection algorithms are used to generate change detection results by comparing current image data with historical image data. Detections from the machine learning workflow 420, the data mining 430 and the computer vision 440 processing are combined by a combiner 450. The combiner 450 Combines all detections from machine learning, computer vision and data mining.
The pipeline coordinates 412 are subjected to buffering 418 which adds a certain distance around the vector line of the pipeline and transforms the vector line into polygons which indicate the geographic corridor 460 which corresponds to the pipeline right of way onto which encroachments are detected. The geographic corridor 460 is used to filter the detections. The detections are filtered to those within the area of interest only.
The land lot data 414 is are used to detect deviations between pipeline coordinates and land lot coordinates so that the actual route of the pipeline can be compared with the planned route.
In a geo-filtering 470 processing step, the combined detections are filtered using the geographic corridor 460 in order to identify objects within the right of way of the pipeline. The outputs from the processing 400 are detections 472, statistics 474 and integration 476. The final detections may be fed into a rule-based engine to produce threat score of each pipeline segment.
Each detection may comprise: the location and shape (polygon or square around the object). Represented in vector format (GEOJSON geometry), meta data: this can include: Detection id: id of this detection (unique); class type: what is the detection (car, machinery, road, building, transmission tower, waterway, unknown object, etc); road name: for roads only; road type: highway, primary, secondary, pathway, etc; color: color to represent this object in the map; and confidence indicator: how confident the system is of this detection. FIG.5 is a flowchart showing a machine learning workflow in a method according to an embodiment of the present invention. The machine learning workflow 500 shown in FIG.5 corresponds to the machine learning workflow 420 shown in FIG.4 and is an implementation of step 304 of the method 300 shown in FIG.3 and steps 204 to 206 shown in FIG.2. The machine learning workflow 500 shown in FIG.5 is carried out by the machine learning object detection module 122 shown in FIG.1.
The input into the machine learning workflow 500 is image data 502 which corresponds to images of the geographic corridor. The image data 502 is subjected to pre-processing 504. The pre-processing 504 comprises transforming the image data into different resolutions. This transformation is done to fit different machine learning models as different models may only accept certain resolutions of image.
As shown in FIG.5, the image data 502 is transformed into 5cm / pixel resolution 506, 15 cm / pixel resolution 508 and other resolutions 510. The pre-processed image data is then fed into a plurality of machine learning models. The machine learning models may be for example artificial neural networks trained on image data sets to recognize certain objects. As shown in FIG.5, the machine learning models comprise a first car model 520, a second car model 522, a first road model 524 and a second car model 526. It will be appreciated that these are intended as illustrative examples and a plurality of different models for each of a plurality of different object types may be used. In addition, the different resolution images may be fed into each of the machine learning models to generate multiple sets of results. As described above, several models are used to predict the same thing at different resolutions. The results from the different models are combined by a combiner algorithm. As shown in FIG.5, the results from the first car model 520 and the second car model 522 are combined by an ensemble algorithm 530 and the results from the first road model 524 and the second road model 526 are combined by an ensemble algorithm 532. The ensemble algorithms may be majority voting ensembles. The results from the different models are combined in an aggregate algorithm 540. The outputs from the machine learning models are images which are huge, and these may be transformed into vector formats based on GIS vector format standards which are much more lightweight. As described above results from the same models with different resolution input images and results from different models which detect the same objects are combined. Thus, each of the models may yield different results. The different models which detect the same object may be trained using different datasets. The combined results may include a confidence indicator. If more models produce the same prediction, the confidence for that prediction will be higher. Results from different resolutions may be compared and if a detection in the same location is found in both resolutions, take one of them and increase confidence. The use of different resolutions improves the performance and reduces the rate of false negatives. Machine learning models are trained on specific resolution and images come in different resolutions, having detections from multiple resolutions will ensure the maximum number of detections are found.
FIG.6 is a flowchart showing processing of mapped object data in a method according to an embodiment of the present invention. The method 600 shown in FIG.6 corresponds to steps 310 and 312 of the method 300 shown in FIG.3 and the data mining 430 processing shown in FIG.4. The method 600 shown in FIG.6 is carried out by the mapped object detection module 126 of the geographic data processing system 100 shown in FIG.1. The inputs to the method 600 are image data 602 which as discussed above is images such as orthophotos of the geographic corridor and tags 604 which are a manual collection of keywords to be used in data mining based on application program interface (API) docs.
Boundary extraction 606 is performed on the image data 606. Then the tags 604 are used to obtain open street map (OSM) overpass data from an application program interface (API) 608. Alternative mapping data such as Google Maps or Bing Maps may also be used. The collected data is in OSM vector format. The collected data is then subjected to transformation 610 to convert from OSM format and labeling to GEOJSON format. Then filtering 612 is performed to remove unnecessary or irrelevant data. The outputs from the method are detections 614 and metadata 616. The metadata 616 comprises additional information relating to the detections 614 such as the name of the street or road and the type of the road.
This is used to detect crossings by roads, waterways and utilities such as transmission lines/towers. This info is important to quantify the potential of third-party threats. An area with more secondary roads and utilities is more prone to threats. Using online resources such as Open Street Map to collect information about road and utility networks is much easier than training specific machine learning models t to detect this information from the images. This can also be costly if no labelled data is present. These results are then filtered to only those within the pipeline right of way. It is finally combined with other results from machine learning and computer vision.
FIG.7 is a flowchart showing an image processing workflow in a method according to an embodiment of the present invention. The method 700 shown in FIG.7 corresponds to steps 306 and 308 of the method 300 shown in FIG.3 and the computer vision 440 processing shown in FIG.4. The method 700 shown in FIG.7 is carried out by the image processing change detection module 124 of the geographic data processing system 100 shown in FIG.1.
The inputs to the method 700 are current image data 702 and historic image data 704. The current image data 702 and the historic image data 704 are sets of image data captured from above, for example orthophoto data and may include location data indicating the locations corresponding to the image data. The historic image data 704 may be image data captured during a previous survey of the geographic corridor which is used as a reference to detect changes.
The current image data 702 is subjected to pre-processing 706 and the historic image data 704 is subjected to pre-processing 708. Each of the pre-processing steps 706 and 708 comprise normalization and scaling. Then the pre-processed current image data and the pre-processed historic image data are input into a change detection ensemble 710. The change detection ensemble 710 is an ensemble of algorithms used to detect changes between the current image data 702 and the historic image data 704. Following the change detection ensemble 710, detections are subjected to morphology 712 which comprises filtering noise. Then the filtered detections are subjected to masking 716 and vectorization 716. The masking 716 extracts detections as a binary mask and the vectorization converts the binary mask into polygons. The output is detections 718.
FIG.8 is a flowchart showing an image processing workflow in a method according to an embodiment of the present invention. The method 800 shown in FIG.8 corresponds to the change detection ensemble 710 described above with reference to FIG.7.
As described above with reference to FIG.7, the inputs to the method 800 are current image data 802 and historic image data 804. The current image data 802 and the historic image data 804 are sets of image data captured from above, for example orthophoto data and may include location data indicating the locations corresponding to the image data. The historic image data 804 may be image data captured during a previous survey of the geographic corridor which is used as a reference to detect changes.
The current image data 802 is subjected to pre-processing 806 and the historic image data 804 is subjected to pre-processing 808. Each of the pre-processing steps 806 and 808 comprise normalization and scaling.
The pre-processed current image data and the pre-processed historic image data are input into three different image change detection algorithms: a Gaussian mixture background subtractor 820, a pixel subtraction 830 workflow, and a structural similarity index measure 840 workflow. The Gaussian mixture background subtractor 820 is a segmentation algorithm for background segmentation. It is followed by a morphology 822 operation. The morphological operations may comprise the use of opening followed by closing operations. The pixel subtraction 830 workflow comprises a Gaussian filter 832 followed by a morphology 836 operation and an adaptive Gaussian filter 834 followed by a morphology operation 838. The structural similarity index measure 840 workflow comprises determination of a structural similarity index followed by binary thresholding 842, a find contours operation 844 followed by a morphology operation 846. The detections from the different workflows are combined by a combiner 850 to give the final set of change detection encroachment detections. The combiner 850 may be implemented using an OR binary operation. Alternatively, intensity summation may be used and areas with higher intensity means higher confidence of change detection.
Using different algorithms will produce different change detection results, therefore applying multiple algorithms and combining the results can give an increased number of detections. The goal is to detect large enough changes within the area of interest which could constitute things like garbage, waste containers, etc.
There are two reasons for which combining is advantageous in this application. Firstly, data may not be updated in real-time; drone surveys are done once every few months. Hence computations resource requirements are manageable. Secondly, it is preferred to have more false positives than false negatives. For example, it would be preferable to detect there is an object on the pipeline when there are none rather than missing an object. The final set of detections go through a review process before going into threat and risk quantification.
Whilst the foregoing description has described exemplary embodiments, it will be understood by those skilled in the art that many variations of the embodiments can be made within the scope and spirit of the present invention.