技术领域technical field
本发明属于隧道内事故监测及无线通信领域，涉及一种基于声音时频域分析的隧道事故监测报警系统及方法。The invention belongs to the field of accident monitoring and wireless communication in tunnels, and relates to a tunnel accident monitoring and alarm system and method based on sound time-frequency domain analysis.
背景技术Background technique
近年来大量特长公路隧道陆续建成并投入运营，公路隧道已逐渐由建设高峰期转向运营高峰期，隧道属于国家重要基础设施，维护其安全是十分重要与必要的，为此，隧道应严格实行事故监控监测管理。但受人工检测难度和技术操作等多方面影响，使其成为运营期间面临的首要问题，给隧道运营管理带来难题。In recent years, a large number of extra-long highway tunnels have been built and put into operation one after another. Highway tunnels have gradually shifted from the peak period of construction to the peak period of operation. Tunnels are important national infrastructures, and it is very important and necessary to maintain their safety. For this reason, tunnels should be strictly implemented. Monitoring monitoring management. However, due to the difficulty of manual inspection and technical operations, it has become the primary problem faced during operation, which has brought difficulties to tunnel operation and management.
目前，隧道内事故监测报警存在以下问题：a.监控系统网络智能化较低，大多采用定点或人工巡检，要得到隧道内某点实时状态信息并不现实；b.针对长大隧道，其发生事故后的救援安排与效率更备受考验，隧道过长，事故地点与外界沟通困难，较难开展应急救援活动；c.现有隧道事故监测手段主要是烟雾与视频监测，但视频监控距离与范围覆盖有限，受光线和内部交通因素影响较大，运营成本较高，而烟雾监测距离与范围覆盖更低，及时性较差，容易错过事故救援的黄金时间。At present, there are the following problems in the monitoring and alarming of accidents in the tunnel: a. The network intelligence of the monitoring system is low, and most of them use fixed-point or manual inspections. It is not realistic to obtain real-time status information at a certain point in the tunnel; b. The rescue arrangements and efficiency after the accident are more tested. The tunnel is too long, and the accident site is difficult to communicate with the outside world, making it difficult to carry out emergency rescue activities; c. The existing monitoring methods for tunnel accidents are mainly smoke and video monitoring, but the video monitoring distance The range coverage is limited, it is greatly affected by light and internal traffic factors, and the operating cost is high, while the smoke monitoring distance and range coverage is lower, the timeliness is poor, and it is easy to miss the golden time for accident rescue.
如发明CN 104880245 A提出一种基于车辆撞击噪声定位报警系统，得出并进行特征值计算，该算法精准度较差，对撞击信号的识别度较低且误报率较高；又如发明CN 106887105 A与CN 103077609 A，分别提出一种基于受灾人特征和多传感器感知的隧道监控系统，其可行性、造价以及施工技术要求较高，对于传感器和硬件设施依赖性较强，将实际施工环境与人机协调理想化，脱离于现实。For example, the invention CN 104880245 A proposes a positioning alarm system based on vehicle impact noise, and obtains And carry out the eigenvalue calculation, the accuracy of the algorithm is poor, the recognition of the impact signal is low and the false alarm rate is high; another example is the invention of CN 106887105 A and CN 103077609 A, respectively propose a method based on the characteristics of the victims and multi-sensor The perceptual tunnel monitoring system has high requirements for feasibility, cost and construction technology, and is highly dependent on sensors and hardware facilities. It idealizes the actual construction environment and human-machine coordination, and is divorced from reality.
发明内容Contents of the invention
为克服现有技术中的问题，本发明的目的在于提供一种基于声音时频域分析的隧道事故监测报警系统及方法。In order to overcome the problems in the prior art, the object of the present invention is to provide a tunnel accident monitoring and alarm system and method based on sound time-frequency domain analysis.
为解决现有技术存在的问题，本发明的技术方案是：一种基于声音时频域分析的隧道事故监测报警系统，包括声音采集处理模块、DSP储存分析模块和控制模块，其中：In order to solve the problems existing in the prior art, the technical solution of the present invention is: a tunnel accident monitoring and alarm system based on sound time-frequency domain analysis, including a sound collection and processing module, a DSP storage analysis module and a control module, wherein:
声音采集处理模块包括数值转换器和声音传感器；The sound collection and processing module includes a numerical converter and a sound sensor;
DSP储存分析模块包括ROM闪存模块、SRAM数据存储模块和DSP核心处理模块；DSP storage analysis module includes ROM flash memory module, SRAM data storage module and DSP core processing module;
控制模块包括报警模块、通讯模块和定位模块，其中，通讯模块包括通讯控制装置与通讯传输装置，定位模块用于确定事故车辆的位置信息，报警模块包括报警信号装置、报警控制系统和报警通讯装置；The control module includes an alarm module, a communication module and a positioning module, wherein the communication module includes a communication control device and a communication transmission device, the positioning module is used to determine the location information of the accident vehicle, and the alarm module includes an alarm signal device, an alarm control system and an alarm communication device ;
数值转换器与声音传感器和DSP储存分析模块连接；DSP储存分析模块与数值转换器和控制模块连接；控制模块与DSP核心处理模块和隧道监控中心连接，通讯模块与隧道监控中心的通讯系统连接，报警模块通过继电器接口与急救火灾报警系统连接，定位模块与声音传感器连接，且安置于声音传感器内部；The numerical converter is connected with the sound sensor and the DSP storage analysis module; the DSP storage analysis module is connected with the numerical converter and the control module; the control module is connected with the DSP core processing module and the tunnel monitoring center, and the communication module is connected with the communication system of the tunnel monitoring center. The alarm module is connected to the emergency fire alarm system through the relay interface, and the positioning module is connected to the sound sensor and placed inside the sound sensor;
声音传感器用于收集隧道内的声音，采集到的声音经过数值转换器转换为数字信号后传输给SRAM数据存储模块；DSP核心处理模块用于加载ROM闪存中的代码执行并读取SRAM数据存储模块中的数据，并将运算后得到的指令发送给通讯传输装置。The sound sensor is used to collect the sound in the tunnel, and the collected sound is converted into a digital signal by a digital converter and then transmitted to the SRAM data storage module; the DSP core processing module is used to load the code in the ROM flash memory for execution and read the SRAM data storage module The data in it, and the command obtained after the operation is sent to the communication transmission device.
DSP储存分析模块和控制模块均安置于隧道监控中心，声音采集处理模块中，数值转换器安置于隧道监控中心，声音传感器布置于隧道内壁。Both the DSP storage analysis module and the control module are placed in the tunnel monitoring center, the sound collection and processing module, the digital converter is placed in the tunnel monitoring center, and the sound sensor is arranged on the inner wall of the tunnel.
声音传感器布置于隧道两侧的边墙高度为2～2.5m处，并沿其延伸方向间隔预定距离设定多组。The sound sensors are arranged at the side walls on both sides of the tunnel at a height of 2-2.5m, and multiple groups are set at predetermined distances along the extension direction.
声音传感器采用ARM9声音感知器，数值转换器采用ads5422转换芯片，DSP核心处理模块采用TMS320C54DSP板，ROM闪存模块采用SST39LF/VF160，为1M16bit的CMOS多功能FlashMPF器件。The sound sensor adopts ARM9 sound sensor, the digital converter adopts ads5422 conversion chip, the DSP core processing module adopts TMS320C54DSP board, and the ROM flash memory module adopts SST39LF/VF160, which is a 1M16bit CMOS multifunctional FlashMPF device.
本发明还提供了一种基于声音时频域分析的隧道事故监测报警方法，包括以下步骤：The present invention also provides a tunnel accident monitoring and alarm method based on sound time-frequency domain analysis, comprising the following steps:
步骤1)：收集隧道内实时声音信号，筛选出有效声音信号作为有效帧；Step 1): Collect real-time sound signals in the tunnel, and filter out effective sound signals as effective frames;
步骤2)：有效帧与模板库数字信号进行傅立叶变换并筛选；Step 2): Perform Fourier transform and screening of effective frames and template library digital signals;
步骤3)：有效帧与模板库数字信号进行功率谱转换并筛选；Step 3): Perform power spectrum conversion and screening of effective frames and template library digital signals;
步骤4)：有效帧进行小波分解并筛选作为特征信号；Step 4): Effective frames are subjected to wavelet decomposition and screened as feature signals;
步骤5)：将步骤4中的特征信号带入到神经网络中进行最终判断。Step 5): Bring the characteristic signal in step 4 into the neural network for final judgment.
具体步骤如下：Specific steps are as follows:
步骤1)：筛选声音有效帧：Step 1): Filter valid sound frames:
将采用的声音信号频率设置为8000HZ，采取每帧过50点其阈值为600的数据帧为有效帧，对非有效帧进行舍弃；Set the frequency of the sound signal used to 8000HZ, take the data frame with a threshold of 600 over 50 points per frame as the valid frame, and discard the non-valid frame;
步骤2)：将步骤1)中采取的有效帧与模板库中的特征信号做傅立叶变换并筛选：Step 2): Fourier transform and filter the effective frames taken in step 1) and the feature signals in the template library:
将产生隧道撞击事故时对应发出的声音特征信号与汽车鸣笛声音、汽车发动机声音特征信号转换为数字特征信号并保存至模板库中，对有效帧和模板库中的数字特征信号做傅立叶变换，将其时间域上的特征信号转换为频率域上的特征信号，对上述两个傅立叶积分变换后的函数计算其相关系数，将相关系数大于阈值的数据进行保留，否则进行舍弃，相关系数的求解根据下式计算：Convert the corresponding sound feature signal, car whistle sound, and car engine sound feature signal when a tunnel collision accident occurs into a digital feature signal and save it in the template library, and perform Fourier transform on the effective frame and the digital feature signal in the template library. Convert the characteristic signal in the time domain to the characteristic signal in the frequency domain, calculate the correlation coefficient of the above two functions after Fourier integral transformation, keep the data with the correlation coefficient greater than the threshold, otherwise discard it, and solve the correlation coefficient Calculate according to the following formula:
式(1)中：Cov(X，Y)表示协方差公式，D(x)D(y)分别表示X与Y的方差；In formula (1): Cov(X, Y) represents the covariance formula, and D(x)D(y) represents the variance of X and Y respectively;
步骤3)：将步骤1)中采取的有效帧与模板库中的特征信号做功率谱转换并筛选：Step 3): Perform power spectrum conversion and screening of the effective frames taken in step 1) and the characteristic signals in the template library:
处理长度为1024字节的离散式傅立叶积分变换，频率采用8000Hz，将有效帧与模板库中的数字特征信号转换为功率谱，对上述两个功率谱转换后的函数计算其相关系数，对相关系数超过阈值的有效帧信号进行进一步的计算，功率谱根据下式计算：Discrete Fourier integral transform with a length of 1024 bytes is processed, and the frequency is 8000 Hz. The effective frame and the digital feature signal in the template library are converted into a power spectrum, and the correlation coefficient is calculated for the above two converted functions of the power spectrum. The effective frame signal whose coefficient exceeds the threshold is further calculated, and the power spectrum is calculated according to the following formula:
式(2)中：S(ω)表示有效帧的功率谱，X(T)表示时域信号，P表示为功率谱密度；In formula (2): S(ω) represents the power spectrum of the effective frame, X(T) represents the time domain signal, and P represents the power spectral density;
步骤4)：有效帧信号小波分解，保留特征信号：Step 4): Wavelet decomposition of the effective frame signal, retaining the characteristic signal:
对步骤3)中超过阈值的有效帧全部对应转换至一个固定的合理区间内，即归一化，将超过阈值的有效帧信号进行小波分解，分解出不同时间域与频域的数值信号，剔除高频率信号，保留分解后的低频率信号，通过Mallat算法，将保留的低频率信号逐步分解并将其作为特征信号，小波分解公式如下式表示：In step 3), all valid frames that exceed the threshold are correspondingly converted into a fixed reasonable interval, that is, normalized, and the effective frame signals that exceed the threshold are subjected to wavelet decomposition to decompose numerical signals in different time domains and frequency domains, and eliminate For high-frequency signals, the decomposed low-frequency signals are retained. Through the Mallat algorithm, the retained low-frequency signals are gradually decomposed and used as characteristic signals. The wavelet decomposition formula is expressed as follows:
式(4)中：h表示滤波器系数，Cjn表示长度空间的尺度系数；In formula (4): h represents the filter coefficient, andCjn represents the scale coefficient of the length space;
步骤5)：用基于小波分解进行特征提取的数据训练三层神经网络，将步骤4)保留的特征信号投入到神经网络中进行最终判断；具体训练神经网络过程：将汽车鸣笛和汽车发动机声音与隧道撞击声一起训练，隧道撞击声、汽车鸣笛和汽车发动机声音通过小波分解后作为训练样本备用；用交叉验证的方式提取百分之七十五的样本作为训练数据，其余为测试数据；采用三层神经网络，其中，输入层为样本特征值，隐藏层的个数大于输入层的个数，输出层为识别结果；将神经网络的权值与阈值θ(W，b)初始化，将训练样本投入神经网络中进行迭代计算，调整权值W与阈值b，在训练完成的神经网络中将测试数据带入到神经网络中进行分类，将表现最好的神经网络的权值与阈值作为最终的神经网络分类器；Step 5): Use the data for feature extraction based on wavelet decomposition to train a three-layer neural network, put the characteristic signal retained in step 4) into the neural network for final judgment; the specific training process of the neural network: the sound of the car whistle and the sound of the car engine Train with tunnel impact sound, tunnel impact sound, car whistle and car engine sound are used as training samples after wavelet decomposition; 75% of the samples are extracted by cross-validation as training data, and the rest are test data; A three-layer neural network is used, where the input layer is the sample feature value, the number of hidden layers is greater than the number of input layers, and the output layer is the recognition result; the weight and threshold θ(W, b) of the neural network are initialized, and Put the training samples into the neural network for iterative calculation, adjust the weight W and the threshold b, bring the test data into the neural network for classification in the trained neural network, and use the weight and threshold of the neural network with the best performance as The final neural network classifier;
步骤6)：输入实时特征信号进行神经网络判断输出状态结果：Step 6): Input the real-time characteristic signal to judge the output state result of the neural network:
将隧道内声音传感器2收集到的实时声音信号按照步骤1至4顺序运行，并将运行后得到的特征信号代入步骤5中构建的神经网络模型进行计算，判断实时隧道状态分类。The real-time sound signal collected by the sound sensor 2 in the tunnel is run sequentially according to steps 1 to 4, and the characteristic signal obtained after running is substituted into the neural network model constructed in step 5 for calculation to judge the real-time tunnel state classification.
步骤5中，对权值W与阈值b进行微调参照如下公式进行：In step 5, fine-tune the weight W and threshold b with reference to the following formula:
式(10)中：Wij表示对应权值，α表示收敛速率，表示误差函数对权值求偏导数；In formula (10): Wij represents the corresponding weight, α represents the convergence rate, Indicates that the error function takes a partial derivative of the weight;
式(11)中：表示对应阈值，表示误差函数对阈值求偏导数；In formula (11): Indicates the corresponding threshold, Represents the partial derivative of the error function to the threshold;
步骤5中，对权值W与阈值b进行微调参照如下公式进行：In step 5, fine-tune the weight W and threshold b with reference to the following formula:
式(13)中：xi表示神经网络该层的神经元个数个数。In formula (13): xi represents the number of neurons in this layer of the neural network.
所述三层神经网络中，输入层为11个神经元，隐藏层为15个神经元，输出层为3个神经元。In the three-layer neural network, the input layer has 11 neurons, the hidden layer has 15 neurons, and the output layer has 3 neurons.
与现有技术相比，本发明的优点如下：本发明通过声音信号感知隧道内事故状态，更好地适应隧道运营模式并提高运作效率，也为隧道事故监测报警方向提供了一种新的模式；本发明通过小波分析与改进后神经网络对事故产生的声音信号进行时频域分析，极大提高了对于事故信号的识别精准度、覆盖范围、抗干扰度与信噪比；本发明更全面和直接的获取隧道事故信息，达到对隧道的整体监控，及时预警，减少人员丧亡与财产损失，能够满足快速救援并减少事故影响范围；用基于小波分解进行特征提取的数据训练三层神经网络，将特征信号投入到神经网络中进行最终判断，将汽车鸣笛和汽车发动机声音与隧道撞击声一起训练，隧道撞击声、汽车鸣笛和汽车发动机声音通过小波分解后作为训练样本备用，小波分析是时间(空间)频率的局部化分析，时域分析是指控制系统根据输出量的时域表达式，直接在时间域中对系统进行分析的方法，频域分析是将时间历程波形经过傅立叶变换分解为若干单一的谐波分量，以获得信号的频率结构以及各谐波和相位信息，提高算法的准确率与识别速度；Compared with the prior art, the present invention has the following advantages: the present invention perceives the accident state in the tunnel through the sound signal, better adapts to the tunnel operation mode and improves the operation efficiency, and also provides a new mode for tunnel accident monitoring and alarm direction ; The present invention carries out the time-frequency domain analysis to the sound signal that the accident produces through wavelet analysis and the improved neural network, has greatly improved the recognition accuracy, coverage, anti-interference degree and signal-to-noise ratio to the accident signal; The present invention is more comprehensive and direct access to tunnel accident information, to achieve overall monitoring of the tunnel, timely early warning, reduce casualties and property losses, meet the needs of rapid rescue and reduce the scope of accident impact; use data for feature extraction based on wavelet decomposition to train a three-layer neural network, The feature signal is put into the neural network for final judgment, and the car whistle and car engine sound are trained together with the tunnel impact sound. The tunnel impact sound, car whistle and car engine sound are decomposed by wavelet and used as training samples for backup. Wavelet analysis is Localized analysis of time (space) frequency, time domain analysis refers to the method that the control system directly analyzes the system in the time domain according to the time domain expression of the output quantity, frequency domain analysis is to decompose the time history waveform through Fourier transform For several single harmonic components, to obtain the frequency structure of the signal and the information of each harmonic and phase, and improve the accuracy and recognition speed of the algorithm;
另外，本发明通过合理的公式设计，将公式(10)与(11)右侧增加一个风险系数Rc，从而降低神经网络过拟合的概率，声音传感器的合理位置布置，在有效收集信号的同时避免容易遭到破坏的风险。In addition, the present invention adds a risk coefficient Rc to the right side of formulas (10) and (11) through reasonable formula design, thereby reducing the probability of over-fitting of the neural network. Avoid the risk of being easily compromised.
附图说明Description of drawings
图1为本发明的系统结构示意图；Fig. 1 is a schematic diagram of the system structure of the present invention;
图2为本发明实施例的事故监测报警硬件连接示意图；Fig. 2 is a schematic diagram of hardware connection of accident monitoring and alarming in an embodiment of the present invention;
图3为本发明实施例的改进算法流程图。Fig. 3 is a flow chart of the improved algorithm of the embodiment of the present invention.
图中，1-隧道，2-声音传感器，3-数值转换器，4-ROM闪存模块，5-SRAM数据存储模块，6-报警信号装置，7-报警控制系统，8-报警通讯装置，9-通讯控制装置，10-通讯传输装置，11-DSP核心处理模块。In the figure, 1-tunnel, 2-sound sensor, 3-value converter, 4-ROM flash memory module, 5-SRAM data storage module, 6-alarm signal device, 7-alarm control system, 8-alarm communication device, 9 -communication control device, 10-communication transmission device, 11-DSP core processing module.
具体实施方式Detailed ways
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进行进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本发明，并不用于限定本发明。In order to make the object, technical solution and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.
参见图1与图2，本发明的一种基于声音时频域分析的隧道事故报警系统，包括声音采集处理模块、DSP储存分析模块，控制模块；所述声音采集处理模块包括声音传感器2与数值转换器3，声音传感器2用于收集隧道1内声音信号，数值转换器3通过数模转换将电信号放大并转换为数字信号；所述DSP储存分析模块包括ROM闪存模块4、SRAM数据存储模块5、DSP核心处理模块11，SRAM数据存储模块5将所有数字信号存储并等待DSP核心处理模块11处理，ROM闪存模块4存放识别库与可执行文件，DSP核心处理模块11通过SRAM数据存储模块5与ROM闪存模块4中存放的文件进行识别对比与分析；所述控制模块包括报警模块、通讯模块、定位模块，通讯模块包括通讯控制装置9与通讯传输装置10，用于管控中心接收事故信息与监控报警之间的协调，定位模块用于确定事故车辆的位置信息，报警模块包括报警信号装置6、报警控制系统7和报警通讯装置8，用于发出事故警示信息通知监控中心与防灾救援部门联动。Referring to Fig. 1 and Fig. 2, a kind of tunnel accident warning system based on sound time-frequency domain analysis of the present invention comprises sound acquisition processing module, DSP storage analysis module, control module; Described sound acquisition processing module comprises sound sensor 2 and numerical value Converter 3, sound sensor 2 is used for collecting sound signal in tunnel 1, numerical converter 3 amplifies and converts electrical signal into digital signal through digital-to-analog conversion; Described DSP storage analysis module comprises ROM flash memory module 4, SRAM data storage module 5. DSP core processing module 11, SRAM data storage module 5 stores all digital signals and waits for DSP core processing module 11 to process, ROM flash memory module 4 stores identification library and executable file, DSP core processing module 11 through SRAM data storage module 5 Identify, compare and analyze the files stored in the ROM flash memory module 4; the control module includes an alarm module, a communication module, and a positioning module, and the communication module includes a communication control device 9 and a communication transmission device 10 for the control center to receive accident information and The coordination between monitoring and alarming, the positioning module is used to determine the location information of the accident vehicle, and the alarming module includes an alarming signal device 6, an alarming control system 7 and an alarming communication device 8, which is used to send out accident warning information to notify the monitoring center and disaster prevention and rescue department linkage.
进一步地，所述声音传感器2分布于隧道1整体路段或重要路段，布置于隧道1两侧的边墙上且与汽车高度大致相等位置处，并沿其延伸方向间隔预定距离设定多组，参考折叠式覆盖使相邻声音传感器2之间的监听范围重叠，从而实现其无障碍、无盲区收集，也可根据声音传感器2的感测性质进行实地调整，声音传感器2采用ARM9声音感知器。Further, the sound sensors 2 are distributed in the entire road section or important road sections of the tunnel 1, arranged on the side walls on both sides of the tunnel 1 at positions approximately equal to the height of the vehicle, and set multiple groups at predetermined distances along the extending direction, Referring to the folding coverage, the monitoring ranges between adjacent sound sensors 2 overlap, so as to realize its barrier-free and no blind area collection. It can also be adjusted on the spot according to the sensing properties of the sound sensor 2. The sound sensor 2 uses an ARM9 sound sensor.
进一步地，所述数值转换器3安置于隧道监控中心，与声音传感器2和DSP储存分析模块连接，将从声音传感器2接受到的电信号转换至数据信号并传输给DSP储存分析模块，数值转换器3采用ads5422转换芯片。Further, the numerical converter 3 is placed in the tunnel monitoring center, connected with the sound sensor 2 and the DSP storage analysis module, converts the electrical signal received from the sound sensor 2 into a data signal and transmits it to the DSP storage analysis module, and converts the numerical value Device 3 adopts ads5422 conversion chip.
进一步地，所述DSP储存分析模块与数值转换器3和控制模块连接，安置于隧道监控中心，接收数值转换器3传递的数字信号并做对比分析，将处理结果发送至相对应的控制模块；所述DSP核心处理模块11采用TMS320C54DSP板，ROM闪存模块4采用SST39LF/VF160，为1M16bit的CMOS多功能FlashMPF器件。Further, the DSP storage analysis module is connected with the numerical converter 3 and the control module, placed in the tunnel monitoring center, receives the digital signal transmitted by the numerical converter 3 and performs comparative analysis, and sends the processing result to the corresponding control module; The DSP core processing module 11 adopts a TMS320C54DSP board, and the ROM flash memory module 4 adopts SST39LF/VF160, which is a 1M16bit CMOS multifunctional FlashMPF device.
进一步地，所述控制模块安置于隧道监控中心，与DSP核心处理模块11和相关交通主管与救援部门连接，通过传达的事故状态信息开展应急措施。通讯模块与隧道监控中心的通讯系统连接，报警模块通过继电器接口与相关急救火灾报警系统连接，定位模块与声音传感器2连接，安置于声音传感器2内部，通过音频数字信息识别事故地点对应的声音传感器2，进而通过定位模块确定事故地点信息，定位模块采用Mtk或Mstar GPS芯片。Further, the control module is placed in the tunnel monitoring center, connected with the DSP core processing module 11 and relevant traffic supervisors and rescue departments, and carries out emergency measures through the conveyed accident status information. The communication module is connected to the communication system of the tunnel monitoring center, the alarm module is connected to the relevant emergency fire alarm system through the relay interface, the positioning module is connected to the sound sensor 2, and is placed inside the sound sensor 2, and the sound sensor corresponding to the accident site is identified through audio digital information 2. Then determine the accident location information through the positioning module, which uses Mtk or Mstar GPS chip.
进一步地，当DSP核心处理模块11接收到声音采集模块传递的声音信号时，首先通过时域特征初步分析，通过算法对声音信号进行消噪处理，使其达到60％以上的识别率；然后通过小波分析与改进后神经网络在时域和频域上特征提取，使其在能被时域识别的基础上继续提高对撞击信号的识别度，使其达到90％以上的识别率；当噪音信号高于阈值，DSP核心处理模块11随即与控制模块联动，进行报警救援。Further, when the DSP core processing module 11 receives the sound signal delivered by the sound acquisition module, first through the preliminary analysis of the time domain characteristics, the sound signal is denoised by an algorithm, so that it reaches a recognition rate of more than 60%; then through Wavelet analysis and improved neural network feature extraction in the time domain and frequency domain, so that it can continue to improve the recognition of impact signals on the basis of being recognized in the time domain, so that it can reach a recognition rate of more than 90%; when the noise signal If the value is higher than the threshold, the DSP core processing module 11 is linked with the control module to carry out alarm and rescue.
DSP核心处理模块11是整个隧道事故监测报警系统的核心，它完成音频信号的采集、控制、存储、处理以及与外界通讯等功能，本发明主要在于DSP核心处理模块11中针对声音信号处理算法上的改进：The DSP core processing module 11 is the core of the entire tunnel accident monitoring and alarm system, and it completes functions such as collection, control, storage, processing and communication with the outside world of the audio signal. The present invention mainly lies in the DSP core processing module 11 for the sound signal processing algorithm improvement of:
步骤1)：将本改进算法中采用的声音信号频率设置为8000HZ，由于在隧道内收集到的声音信号数量巨大，为减少其计算量并提高精准度，本改进算法对声音信号进行时间域上的筛选，采取每帧过50点其阈值为600的数据帧为有效帧，对非有效帧进行舍弃。Step 1): Set the frequency of the sound signal used in this improved algorithm to 8000HZ. Due to the huge number of sound signals collected in the tunnel, in order to reduce the amount of calculation and improve the accuracy, this improved algorithm performs time domain analysis on the sound signal. For the screening, the data frames with a threshold value of 600 over 50 points per frame are taken as valid frames, and non-valid frames are discarded.
步骤2)：将产生隧道撞击事故时对应发出的声音特征信号与汽车鸣笛声音特征信号、汽车发动机声音特征信号转换为数字特征信号并保存至模板库中，对有效帧和模板库中的数字特征信号做傅立叶变换，将其时间域上的特征信号转换为频率域上的特征信号，对上述两个傅立叶积分变换后的函数计算其相关系数，将相关系数大于阈值的数据进行保留，否则进行舍弃。相关系数的求解根据下式计算：Step 2): Convert the corresponding sound characteristic signal, car whistle sound characteristic signal, and automobile engine sound characteristic signal when a tunnel collision accident occurs into a digital characteristic signal and save it in the template library. Perform Fourier transform of the characteristic signal, convert the characteristic signal in the time domain to the characteristic signal in the frequency domain, calculate the correlation coefficient of the above two functions after Fourier integral transformation, and keep the data with the correlation coefficient greater than the threshold value, otherwise perform give up. The solution to the correlation coefficient is calculated according to the following formula:
式(1)中：Cov(X，Y)表示协方差公式，D(x)D(y)分别表示X与Y的方差。In formula (1): Cov(X, Y) represents the covariance formula, and D(x)D(y) represents the variance of X and Y respectively.
步骤3)：处理长度为1024字节的离散式傅立叶积分变换，频率采用8000Hz，将有效帧与模板库中的数字特征信号转换为功率谱，对上述两个功率谱转换后的函数计算其相关系数，对互相关系数超过阈值的有效帧信号进行进一步的计算，功率谱根据下式计算：Step 3): Process the discrete Fourier integral transform with a length of 1024 bytes, the frequency is 8000 Hz, convert the effective frame and the digital characteristic signal in the template library into a power spectrum, and calculate the correlation of the above two converted functions of the power spectrum Coefficient, to further calculate the effective frame signal whose cross-correlation coefficient exceeds the threshold, and the power spectrum is calculated according to the following formula:
式(2)中：S(ω)表示有效帧的功率谱，X(T)表示时域信号，P表示为功率谱密度。In formula (2): S(ω) represents the power spectrum of the effective frame, X(T) represents the time domain signal, and P represents the power spectral density.
步骤4)：对超过阈值的有效帧全部对应转换至一个固定的合理区间内，即归一化，将超过阈值的有效帧信号进行小波分解，分解出不同时间域与频域的数值信号，剔除高频率信号，保留分解后的低频率信号，通过Mallat算法，将保留的低频率信号逐步分解并将其作为特征信号，小波分解公式如下式表示：Step 4): Convert all effective frames exceeding the threshold to a fixed reasonable interval, that is, normalize, perform wavelet decomposition on effective frame signals exceeding the threshold, decompose numerical signals in different time domains and frequency domains, and eliminate For high-frequency signals, the decomposed low-frequency signals are retained. Through the Mallat algorithm, the retained low-frequency signals are gradually decomposed and used as characteristic signals. The wavelet decomposition formula is expressed as follows:
式(4)中：h表示滤波器系数，Cjn表示长度空间的尺度系数。In formula (4): h represents the filter coefficient, and Cjn represents the scale coefficient of the length space.
步骤5)：将模板库中的隧道撞击、汽车鸣笛、汽车发动机声音特征信号按照步骤4进行小波分解，将分解到不同频率空间的数字特征信号做为样本，将样本的75％作为训练样本构建神经网络，其余25％做为测试样本检测神经网络的误差。本改进算法采用三层神经网络，输入层为样本特征值，隐藏层的个数大于输入层的个数，输出层为W与b的识别结果。将神经网络的权值与阈值θ(W，b)初始化，将训练样本投入神经网络中进行迭代计算，调整权值W与阈值b，如下式表示：Step 5): Decompose the characteristic signals of tunnel impact, car whistle and car engine sound in the template library by wavelet according to step 4, and use the digital characteristic signals decomposed into different frequency spaces as samples, and use 75% of the samples as training samples Construct the neural network, and the remaining 25% are used as test samples to detect the error of the neural network. This improved algorithm uses a three-layer neural network, the input layer is the sample feature value, the number of hidden layers is greater than the number of the input layer, and the output layer is the recognition result of W and b. Initialize the weight and threshold θ(W, b) of the neural network, put the training samples into the neural network for iterative calculation, and adjust the weight W and threshold b, as shown in the following formula:
z(2)＝W(1)x+b(1) (5)z(2)=W(1)x+b(1) (5)
a(2)＝f(z(2)) (6)a(2)=f(z(2)) (6)
z(3)＝W(2)a(2)+b(2) (7)z(3)=W(2)a(2)+b(2) (7)
D＝f(z(3)) (8)D=f(z(3)) (8)
式(5)与(7)中：W(1)与W(2)表示权值，b(1)与b(2)表示阈值。In formulas (5) and (7): W(1) and W(2) represent weights, b(1) and b(2) represent thresholds.
式(6)与(8)中：a(2)表示训练样本通过神经网络阈值计算得到的数值，D表示一次神经网络迭代的阶段值。In formulas (6) and (8): a(2) represents the value calculated by the training sample through the neural network threshold, and D represents the stage value of one neural network iteration.
进一步地，本改进算法基于传统反向传播误差的神经网络分析，对权值W与阈值b进行微调，公式如下：Furthermore, this improved algorithm is based on the neural network analysis of the traditional backpropagation error, and fine-tunes the weight W and the threshold b. The formula is as follows:
式(10)中：Wij表示对应权值，α表示收敛速率，表示误差函数对权值求偏导。In formula (10): Wij represents the corresponding weight, α represents the convergence rate, Represents the partial derivative of the error function on the weight.
式(11)中：表示对应阈值，表示误差函数对阈值求偏导。In formula (11): Indicates the corresponding threshold, Represents the partial derivative of the error function with respect to the threshold.
采用Widrow-Hoff学习规则，误差函数公式如下表示：Using the Widrow-Hoff learning rule, the error function formula is expressed as follows:
式(12)中：dj表示一次迭代输出值，yi表示初始真实值，本改进算法通过误差函数来分析权值W与阈值b的调整范围。In formula (12): dj represents the output value of an iteration, and yi represents the initial real value. This improved algorithm analyzes the adjustment range of weight W and threshold b through the error function.
进一步地，本改进算法将公式(10)与(11)右侧增加一个风险系数Rc，从而降低神经网络过拟合的概率，Rc如下式表示：Further, this improved algorithm adds a risk coefficient Rc to the right side of formulas (10) and (11), thereby reducing the probability of neural network over-fitting, Rc is expressed as follows:
式(13)中：xi表示特征个数。In formula (13): xi represents the number of features.
通过将模板库中的隧道撞击、汽车鸣笛、汽车发动机声音特征信号进行小波分解与权值W和阈值b的优化，构建出能够判断声音信号并进行状态分类的神经网络。Through the wavelet decomposition and optimization of the weight W and threshold b of the tunnel impact, car whistle, and car engine sound feature signals in the template library, a neural network capable of judging the sound signal and classifying the state is constructed.
步骤6)：将隧道内声音传感器2收集到的实时声音信号按照步骤1—4运行，代入步骤5中构建的神经网络模型进行计算，判断实时隧道状态分类。Step 6): Run the real-time sound signal collected by the sound sensor 2 in the tunnel according to steps 1-4, and substitute it into the neural network model constructed in step 5 for calculation, and judge the real-time tunnel state classification.
参见图3，所述算法流程图描述了隧道事故监测报警系统通过声音信号识别隧道是否发生碰撞事故的方法，其步骤如下：Referring to Fig. 3, described algorithm flow chart has described the method for the tunnel accident monitoring and warning system to identify whether a collision accident occurs in the tunnel through the sound signal, and its steps are as follows:
1.收集隧道内实时声音信号；1. Collect real-time sound signals in the tunnel;
2.筛选出有效声音信号作为有效帧；2. Screen out valid sound signals as valid frames;
3.有效帧与模板库数字信号进行傅立叶变换并筛选；3. Perform Fourier transform and screening of effective frames and template library digital signals;
4.有效帧与模板库数字信号进行功率谱转换并筛选；4. Power spectrum conversion and screening of effective frames and template library digital signals;
5.有效帧进行小波分解并筛选作为特征信号；5. Effective frames are decomposed by wavelet and screened as characteristic signals;
6.特征信号进行神经网络判断输出状态结果。6. The characteristic signal is used to judge the output status result of the neural network.
在整个DSP核心处理模块处理过程中，本改进算法通过对数字信号多个域，多特征的分析与识别，通过小波分析与改进后神经网络进行的时频域分析，通过Matlab的仿真提取，可极大提升对事故声音信号的识别效率，提高对于事故声音信号的识别精准度、抗干扰度与信噪比。In the process of the entire DSP core processing module, the improved algorithm analyzes and recognizes multiple domains and features of the digital signal, analyzes the time-frequency domain through wavelet analysis and the improved neural network, and extracts through the simulation of Matlab. Greatly improve the recognition efficiency of accident sound signals, improve the recognition accuracy, anti-interference degree and signal-to-noise ratio of accident sound signals.
以上内容是结合具体实施例对本发明方法所作的进一步详细说明，不能认定本发明方法的具体实施只限于此。对于本发明所属技术领域的普通技术人员来说，在不脱离本发明构思的前提下做出若干等同替代或明显变型，且性能或用途相同，都应当视为属于本发明由所提交的权利要求书确定的专利保护范围。The above content is a further detailed description of the method of the present invention in conjunction with specific embodiments, and it cannot be assumed that the specific implementation of the method of the present invention is limited thereto. For those of ordinary skill in the technical field to which the present invention belongs, several equivalent substitutions or obvious modifications are made without departing from the concept of the present invention, and the performance or use is the same, all should be regarded as belonging to the present invention by the submitted claims The scope of patent protection determined by the book.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810267017.1ACN108519149B (en) | 2018-03-28 | 2018-03-28 | A tunnel accident monitoring and alarm system and method based on sound time-frequency domain analysis |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810267017.1ACN108519149B (en) | 2018-03-28 | 2018-03-28 | A tunnel accident monitoring and alarm system and method based on sound time-frequency domain analysis |
| Publication Number | Publication Date |
|---|---|
| CN108519149Atrue CN108519149A (en) | 2018-09-11 |
| CN108519149B CN108519149B (en) | 2020-09-01 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810267017.1AExpired - Fee RelatedCN108519149B (en) | 2018-03-28 | 2018-03-28 | A tunnel accident monitoring and alarm system and method based on sound time-frequency domain analysis |
| Country | Link |
|---|---|
| CN (1) | CN108519149B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109597342A (en)* | 2019-01-16 | 2019-04-09 | 郑州轻工业学院 | A kind of the sand dredger monitoring device and method of dynamic group net INTELLIGENT IDENTIFICATION |
| CN109616140A (en)* | 2018-12-12 | 2019-04-12 | 浩云科技股份有限公司 | A kind of abnormal sound analysis system |
| CN109695475A (en)* | 2018-11-30 | 2019-04-30 | 太原理工大学 | A kind of coal mine roadway exception monitoring device and method based on sound |
| CN109754581A (en)* | 2019-01-15 | 2019-05-14 | 深圳市宇航智造技术有限公司 | Calculator room equipment method for early warning, device, equipment and storage medium |
| CN111862519A (en)* | 2020-08-17 | 2020-10-30 | 浙江中辰安全设备有限公司 | Fire early warning fog calculation service gateway |
| CN112881744A (en)* | 2021-02-08 | 2021-06-01 | 中科(湖南)先进轨道交通研究院有限公司 | Train speed and direction measuring device based on train sound and working method thereof |
| CN112951271A (en)* | 2020-11-27 | 2021-06-11 | 长沙理工大学 | Tunnel traffic safety warning method and system based on acoustic assistance |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020011939A1 (en)* | 2000-06-22 | 2002-01-31 | Koichiro Mizushima | Vehicle detection apparatus and vehicle detection method |
| CN101710450A (en)* | 2009-12-21 | 2010-05-19 | 金会庆 | Traffic accident automatic sound detection and alarm terminal |
| CN101877172A (en)* | 2009-12-21 | 2010-11-03 | 金会庆 | An Automatic Detection Method of Traffic Accidents Based on Collision Sound |
| CN102122133A (en)* | 2011-01-21 | 2011-07-13 | 北京工业大学 | Self-adaption wavelet neural network abnormity detection and fault diagnosis classification system and method |
| CN102163427A (en)* | 2010-12-20 | 2011-08-24 | 北京邮电大学 | Method for detecting audio exceptional event based on environmental model |
| CN102568153A (en)* | 2010-12-20 | 2012-07-11 | 淄博高新区联创科技服务中心 | Vehicle collision detection device |
| CN202362932U (en)* | 2011-11-28 | 2012-08-01 | 长安大学 | Information monitoring device of traffic accident scene |
| CN102741896A (en)* | 2009-12-21 | 2012-10-17 | 电话有限公司 | Portable apparatus and method for detecting and notifying of vehicle accidents |
| CN103077609A (en)* | 2012-12-26 | 2013-05-01 | 上海市城市建设设计研究总院 | Tunnel traffic accident monitoring method and system based on sensing of multiple sensors |
| CN103622707A (en)* | 2012-08-20 | 2014-03-12 | 王震 | Traffic safety detecting system and method |
| CN104102923A (en)* | 2014-07-16 | 2014-10-15 | 西安建筑科技大学 | Nipponia nippon individual recognition method based on MFCC algorithm |
| CN105913663A (en)* | 2016-06-30 | 2016-08-31 | 南京嘉瑞斯电力科技有限公司 | Automobile accident noise detection and positioning system based on sound sensor |
| CN105989748A (en)* | 2015-03-20 | 2016-10-05 | 现代自动车株式会社 | Accident information management appratus, vehicle including the same, and accident information management method |
| CN106841403A (en)* | 2017-01-23 | 2017-06-13 | 天津大学 | A kind of acoustics glass defect detection method based on neutral net |
| CN107086036A (en)* | 2017-04-19 | 2017-08-22 | 杭州派尼澳电子科技有限公司 | A kind of freeway tunnel method for safety monitoring |
| CN107545890A (en)* | 2017-08-31 | 2018-01-05 | 桂林电子科技大学 | A kind of sound event recognition method |
| CN207008862U (en)* | 2017-07-11 | 2018-02-13 | 杭州天铂红外光电技术有限公司 | Detect the device of tunnel accident |
| CN107705633A (en)* | 2017-11-06 | 2018-02-16 | 张定宇 | A kind of freeway tunnel automobile second accident prevention system and method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020011939A1 (en)* | 2000-06-22 | 2002-01-31 | Koichiro Mizushima | Vehicle detection apparatus and vehicle detection method |
| CN101710450A (en)* | 2009-12-21 | 2010-05-19 | 金会庆 | Traffic accident automatic sound detection and alarm terminal |
| CN101877172A (en)* | 2009-12-21 | 2010-11-03 | 金会庆 | An Automatic Detection Method of Traffic Accidents Based on Collision Sound |
| CN102741896A (en)* | 2009-12-21 | 2012-10-17 | 电话有限公司 | Portable apparatus and method for detecting and notifying of vehicle accidents |
| CN102163427A (en)* | 2010-12-20 | 2011-08-24 | 北京邮电大学 | Method for detecting audio exceptional event based on environmental model |
| CN102568153A (en)* | 2010-12-20 | 2012-07-11 | 淄博高新区联创科技服务中心 | Vehicle collision detection device |
| CN102122133A (en)* | 2011-01-21 | 2011-07-13 | 北京工业大学 | Self-adaption wavelet neural network abnormity detection and fault diagnosis classification system and method |
| CN202362932U (en)* | 2011-11-28 | 2012-08-01 | 长安大学 | Information monitoring device of traffic accident scene |
| CN103622707A (en)* | 2012-08-20 | 2014-03-12 | 王震 | Traffic safety detecting system and method |
| CN103077609A (en)* | 2012-12-26 | 2013-05-01 | 上海市城市建设设计研究总院 | Tunnel traffic accident monitoring method and system based on sensing of multiple sensors |
| CN104102923A (en)* | 2014-07-16 | 2014-10-15 | 西安建筑科技大学 | Nipponia nippon individual recognition method based on MFCC algorithm |
| CN105989748A (en)* | 2015-03-20 | 2016-10-05 | 现代自动车株式会社 | Accident information management appratus, vehicle including the same, and accident information management method |
| CN105913663A (en)* | 2016-06-30 | 2016-08-31 | 南京嘉瑞斯电力科技有限公司 | Automobile accident noise detection and positioning system based on sound sensor |
| CN106841403A (en)* | 2017-01-23 | 2017-06-13 | 天津大学 | A kind of acoustics glass defect detection method based on neutral net |
| CN107086036A (en)* | 2017-04-19 | 2017-08-22 | 杭州派尼澳电子科技有限公司 | A kind of freeway tunnel method for safety monitoring |
| CN207008862U (en)* | 2017-07-11 | 2018-02-13 | 杭州天铂红外光电技术有限公司 | Detect the device of tunnel accident |
| CN107545890A (en)* | 2017-08-31 | 2018-01-05 | 桂林电子科技大学 | A kind of sound event recognition method |
| CN107705633A (en)* | 2017-11-06 | 2018-02-16 | 张定宇 | A kind of freeway tunnel automobile second accident prevention system and method |
| Title |
|---|
| 凤鹏飞等: "基于音频检测的交通事故自动报警系统研究", 《唐山学院学报》* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109695475A (en)* | 2018-11-30 | 2019-04-30 | 太原理工大学 | A kind of coal mine roadway exception monitoring device and method based on sound |
| CN109616140A (en)* | 2018-12-12 | 2019-04-12 | 浩云科技股份有限公司 | A kind of abnormal sound analysis system |
| CN109616140B (en)* | 2018-12-12 | 2022-08-30 | 浩云科技股份有限公司 | Abnormal sound analysis system |
| CN109754581A (en)* | 2019-01-15 | 2019-05-14 | 深圳市宇航智造技术有限公司 | Calculator room equipment method for early warning, device, equipment and storage medium |
| CN109597342A (en)* | 2019-01-16 | 2019-04-09 | 郑州轻工业学院 | A kind of the sand dredger monitoring device and method of dynamic group net INTELLIGENT IDENTIFICATION |
| CN111862519A (en)* | 2020-08-17 | 2020-10-30 | 浙江中辰安全设备有限公司 | Fire early warning fog calculation service gateway |
| CN112951271A (en)* | 2020-11-27 | 2021-06-11 | 长沙理工大学 | Tunnel traffic safety warning method and system based on acoustic assistance |
| CN112881744A (en)* | 2021-02-08 | 2021-06-01 | 中科(湖南)先进轨道交通研究院有限公司 | Train speed and direction measuring device based on train sound and working method thereof |
| CN112881744B (en)* | 2021-02-08 | 2023-04-07 | 中科(湖南)先进轨道交通研究院有限公司 | Train speed and direction measuring device based on train sound and working method thereof |
| Publication number | Publication date |
|---|---|
| CN108519149B (en) | 2020-09-01 |
| Publication | Publication Date | Title |
|---|---|---|
| CN108519149A (en) | A tunnel accident monitoring and alarm system and method based on sound time-frequency domain analysis | |
| CN109616140B (en) | Abnormal sound analysis system | |
| CN108445868B (en) | Intelligent automobile fault diagnosis system and method based on modern signal processing technology | |
| CN110879254B (en) | Steel rail crack acoustic emission signal detection method based on improved least square generation type countermeasure network | |
| CN107545890A (en) | A kind of sound event recognition method | |
| CN107086036A (en) | A kind of freeway tunnel method for safety monitoring | |
| CN106898346A (en) | A kind of freeway tunnel safety monitoring system | |
| CN105877766A (en) | Mental state detection system and method based on multiple physiological signal fusion | |
| CN206931362U (en) | A kind of freeway tunnel safety monitoring system | |
| CN109816987A (en) | A kind of automobile whistle electronic police enforces the law capturing system and its grasp shoot method | |
| CN112257533B (en) | A perimeter intrusion detection and identification method | |
| CN108848571B (en) | Rail transit safety monitoring system and monitoring method based on MEMS sensor | |
| Smailov et al. | A novel deep CNN-RNN approach for real-time impulsive sound detection to detect dangerous events | |
| CN117609885A (en) | High-speed rail intrusion monitoring method based on distributed optical fiber sensing and fused neural network | |
| CN115618205A (en) | A portable voiceprint fault detection system and method | |
| CN111951505B (en) | Fence vibration intrusion positioning and mode identification method based on distributed optical fiber system | |
| CN102346948B (en) | Circumference invasion detection method and system | |
| CN118629217A (en) | A highway sound recognition accident detection system | |
| CN110555457A (en) | Engine lubricating oil wear signal characteristic processing system and method | |
| CN110472563B (en) | Vertical ladder vibration over-large fault diagnosis method based on wavelet packet decomposition and neural network | |
| KR102421158B1 (en) | Smart Volume Control System of Voice Information System According to Noise Levels by Real-Time Acoustic Analysis | |
| CN114743562A (en) | Method and system for recognizing airplane voiceprint, electronic equipment and storage medium | |
| CN206648005U (en) | A kind of intelligent road-lamp of built-in impact sound detection function | |
| CN116910690A (en) | A target classification system based on data fusion | |
| CN109389994A (en) | Identification of sound source method and device for intelligent transportation system |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee | Granted publication date:20200901 |