CN113011321A - Spectral signal denoising method, system, terminal and readable storage medium based on joint dictionary - Google Patents
Spectral signal denoising method, system, terminal and readable storage medium based on joint dictionaryDownload PDFInfo
- Publication number
- CN113011321A CN113011321ACN202110285933.XACN202110285933ACN113011321ACN 113011321 ACN113011321 ACN 113011321ACN 202110285933 ACN202110285933 ACN 202110285933ACN 113011321 ACN113011321 ACN 113011321A
- Authority
- CN
- China
- Prior art keywords
- dictionary
- spectrum
- noise
- sparse representation
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/02—Preprocessing
- G06F2218/04—Denoising
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2136—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on sparsity criteria, e.g. with an overcomplete basis
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/28—Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Image Processing (AREA)
Abstract
Translated fromChinese
Description
Technical Field
The invention belongs to the technical field of spectral signal processing, and particularly relates to a spectral signal denoising method, a system, a terminal and a readable storage medium based on a joint dictionary.
Background
When the ultraviolet-visible spectroscopy is used for detecting the concentration of a substance component, the absorption spectrum of the ultraviolet-visible spectroscopy not only contains complex and various substance information, but also is influenced by the environment where the instrument is located, so that the spectrum signal often contains a large amount of noise information. When the concentration of the substance to be detected is low, the amplitude difference between the spectrum signal and the noise signal is not large, and the noise interference is easy to cause, so that the subsequent quantitative analysis is influenced. Therefore, when the ultraviolet-visible spectroscopy is used for concentration detection, the spectral signals need to be denoised, so that the spectral analysis precision and capability are improved.
The currently commonly used spectral signal denoising algorithm mainly comprises Savitzky-Golay (SG) convolution denoising, Kalman filtering denoising, wavelet denoising and the like. The core of the SG convolution algorithm lies in the selection of the size of a sliding window, and generally needs to be preset in advance; kalman filtering needs to be given by people, and when the model is not accurate, the filtering effect is reduced; the effect of wavelet de-noising depends greatly on the determination of wavelet basis functions and the number of layers, and meanwhile, the selection of a threshold value is required. In summary, each time these common denoising methods are used, some parameters need to be changed manually, and the spectra of the same category cannot be processed adaptively.
Disclosure of Invention
The invention aims to provide a spectral signal denoising method, a system, a terminal and a readable storage medium based on a combined dictionary.
On one hand, the invention provides a spectral signal denoising method based on a joint dictionary, which comprises the following steps:
step S1: constructing a joint dictionary, which comprises:
acquiring pure spectrum data and noise spectrum data;
constructing a pure spectrum dictionary by using the pure spectrum data and constructing a noise spectrum dictionary by using the noise spectrum data;
combining the pure spectrum dictionary and the noise spectrum dictionary to form a joint dictionary;
processing a target spectrum curve by adopting a minimum angle regression algorithm based on the combined dictionary to obtain an optimal sparse representation coefficient matrix;
obtaining a noise reduction spectrum of the target spectrum curve by using the optimal sparse representation coefficient matrix and the joint dictionary, evaluating, and if the noise reduction spectrum does not meet the evaluation requirement, adjusting the joint dictionary until the evaluation requirement is met;
step S2: and denoising the spectral curve to be denoised by using the joint dictionary in the step S1, wherein the spectral curve to be denoised is processed by using a minimum angle regression algorithm to obtain an optimal sparse representation coefficient matrix, and then a denoising spectrum of the spectral curve to be denoised is obtained by using the optimal sparse representation coefficient matrix and the joint dictionary.
Optionally, a process of obtaining a noise reduction spectrum by using the optimal sparse representation coefficient matrix and the joint dictionary, where the noise reduction spectrum is represented by:
Y1＝D×X1
in the formula, Y1For noise-reducing spectra, D is a joint dictionary, X1Representing a coefficient matrix for the optimal sparsity;
the joint dictionary D is represented as:
D＝[D1,D2]wherein D is1Representing a pure spectral dictionary, D2Representing a noise spectrum dictionary.
Optionally, the pure spectrum dictionary and the noise spectrum dictionary are both obtained by using a dictionary learning algorithm of K-SVD, and the process is as follows:
s201: input sample data S e RN*MAnd the maximum iteration number P, wherein if the pure spectrum dictionary is obtained by training, the corresponding sample data S is the pure spectrum data; if the noise spectrum dictionary is obtained through training, corresponding sample data S is noise spectrum data, N is the number of wavelength points of each sample spectrum, M is a sample dimension, and R is a real number;
s202: randomly selecting K samples and constructing an initial dictionary D0∈RN*KThat is, the spectrum data of K samples are formed, and then an initial sparse representation coefficient matrix X is obtained by calculation0；
S203: based on initial dictionary and initial expression coefficient matrix X0Updating and iterating each dictionary atom column by column until the maximum iteration number P is met, and if the maximum iteration number is not met, entering next updating iteration by using the updated dictionary D and the sparse representation coefficient matrix X;
wherein, the kth column dictionary atom d in the dictionarykThe updates of (2) are as follows:
calculating an error matrix based on the following formula
In which Y is a sample signal, djFor the jth column of dictionary atoms in the current dictionary,
for the jth row of data, Ω, in the current sparse representation coefficient XkIs Nx | omegakI matrix, definition set
For the error momentMatrix of
Carrying out SVD to obtain a matrix U and a matrix V;
finally, the first column in the matrix U is selected as the updated dictionary atom dkAnd using the result of the multiplication of the first column in matrix V with Δ (1,1) as the kth row of data in the matrix X of sparse representation coefficients
Optionally, the process of processing the spectral curve by the least-angle regression method to obtain the optimal sparse representation coefficient matrix is as follows:
s21: traversing dictionary atoms based on the joint dictionary and the target spectrum curve to obtain a combination of sparse representation coefficients;
firstly, selecting dictionary atoms closest to the target spectral curve from the combined dictionary, and calculating residual error Y based on the selected atoms′The following are:
wherein Y is a target spectrum,
to make use of selected atoms dkAnd coefficient XkForming a target approximation value;
then, selecting the residual Y from the joint dictionary′Calculating the next residual error according to the formula until the residual error is smaller than a preset threshold value or the selected dictionary atom meets a preset requirement;
if a plurality of dictionary atoms are closest to the target spectrum curve or the residual error, selecting all the dictionary atoms meeting the requirement, and further obtaining a combination of a plurality of sparse representation coefficients;
s22: rejecting combinations without noise dictionary atoms from the combinations of the plurality of sparse representation coefficients in step S21;
s23: the combination of the optimal sparse representation coefficients is selected from the remaining combinations of sparse representation coefficients in step S22 as an optimal sparse representation coefficient matrix.
When the combination of the optimal sparse representation coefficients is selected in step S23, the fitting degree is the minimum, and the fitting degree formula is as follows:
in the formula, R2Is degree of fitting, I1Is noisy spectral data requiring noise reduction, I0Is the spectral data after the noise reduction is completed,
is the mean of the noise spectrum.
The method comprises the steps of denoising a target spectrum curve by using a sparse representation coefficient combination and a joint dictionary to obtain denoised spectrum data, and selecting a group of sparse representation coefficient combinations with minimum fitting degrees by using the fitting degrees as standards. Other noise spectrum data may be selected in other ways.
The invention improves on the coefficient selection condition of the conventional minimum angle regression algorithm. The influence of spectrum atoms and noise atoms on an actual target spectrum is fully considered, the selection condition of sparse representation atom combination is improved, the fitting degree of a reconstruction curve and a mixing curve is used as a judgment standard, and more pure spectrum dictionary atoms are used for obtaining the optimal sparse representation coefficient combination. Compared with the conventional minimum angle regression algorithm, the method has the advantages that the sensitivity of the algorithm to noise is reduced, the spectral characteristics of the component to be detected are better reflected, and the denoising effect is better.
Optionally, the spectral curve to be processed and the target spectral curve are the same type of spectral curve, and the target spectral curve is the same as the spectral curve to be processed in terms of chemical substances. That is, the two are spectral data in the same environment or similar environments, such as spectral data collected in the same target environment or spectral data collected in a simulated target environment or spectral data collected in an environment similar to the target environment.
Optionally, the test environment of the pure spectral data is a darkroom; and adopting Gaussian random noise to simulate environmental noise data in the process of acquiring the noise spectrum data.
In another aspect, the present invention further provides a denoising system based on the above method, including: the system comprises a data acquisition unit, a pure spectrum dictionary construction unit, a noise spectrum dictionary construction unit, a joint dictionary construction unit, an optimal sparse representation coefficient acquisition unit and a noise reduction unit;
the data acquisition unit is used for acquiring pure spectrum data and noise spectrum data;
the pure spectrum dictionary construction unit is used for constructing a pure spectrum dictionary by utilizing the pure spectrum data;
the noise spectrum dictionary construction unit is used for constructing a noise spectrum dictionary by using the noise spectrum data;
the combined dictionary construction unit is used for combining the pure spectrum dictionary and the noise spectrum dictionary to form a combined dictionary;
the optimal sparse representation coefficient acquisition unit is used for processing a target spectral curve by adopting a minimum angle regression algorithm based on the joint dictionary to obtain an optimal sparse representation coefficient matrix;
the noise reduction unit is used for obtaining a noise reduction spectrum of the target spectrum curve by utilizing the optimal sparse representation coefficient matrix and the joint dictionary;
the evaluation unit is used for evaluating whether the noise reduction spectrum meets the evaluation requirement; if not, adjusting the joint dictionary until the evaluation requirement is met;
the optimal sparse representation coefficient matrix obtaining unit is further used for processing the spectral curve to be denoised by adopting a minimum angle regression algorithm to obtain an optimal sparse representation coefficient matrix;
the denoising unit is further configured to obtain a denoising spectrum of the spectral curve to be denoised by using the optimal sparse representation coefficient matrix and the joint dictionary.
In another aspect, the present invention provides a terminal, including a memory and a processor, where the memory stores a computer program, and the processor calls the computer program to execute: the spectral signal denoising method based on the joint dictionary comprises the following steps.
In a fourth aspect, the present invention also provides a readable storage medium storing a computer program, the computer program being invoked by a processor to perform: the spectral signal denoising method based on the joint dictionary comprises the following steps.
Advantageous effects
1. The invention provides a spectral signal denoising method based on a joint dictionary, which is characterized in that a pure spectral dictionary is constructed by using pure spectral data and a noise spectral dictionary is constructed by using the noise spectral data, so that the joint dictionary is obtained, pure spectral atoms with material absorption spectral characteristics are reserved in the composition of the joint dictionary, and when a target spectrum is sparsely represented, the absorption spectral characteristics of a material to be detected in the spectrum can be better obtained, so that the filtering effect is improved.
2. In a further preferable scheme of the invention, the coefficient selection condition of the minimum angle regression algorithm is improved, the influence of spectrum atoms and noise atoms on an actual target spectrum is fully considered, the selection condition of sparse representation atom combination is improved, the fitting degree of a reconstruction curve and a mixing curve is used as a judgment standard, and more pure spectrum dictionary atoms are used for obtaining the optimal sparse representation coefficient combination. Compared with the conventional minimum angle regression algorithm, the method has the advantages that the sensitivity of the algorithm to noise is reduced, the spectral characteristics of the component to be detected are better reflected, and the denoising effect is better.
Drawings
FIG. 1 is a flow chart of a spectral signal denoising method based on a joint dictionary provided by the invention;
FIG. 2 is a block diagram of the process of training and using a joint dictionary in the present invention;
FIG. 3 is a set of training sample spectra obtained;
FIG. 4 is a graph of a spectrum for which noise reduction is required in the present invention;
FIG. 5 is a graph of the effect of noise reduction on noisy spectral data using three different noise reduction methods and the present method;
FIG. 6 is a graph showing the effect of the spectral signal denoising method based on the joint dictionary after denoising noise spectral data.
Detailed Description
The invention provides a spectral signal denoising method, a system, a terminal and a readable storage medium based on a joint dictionary. The present invention will be described in further detail with reference to specific examples, but the embodiments of the present invention are not limited thereto.
As shown in fig. 1, the spectral signal denoising method based on the joint dictionary provided in this embodiment includes the following steps:
step S101: several sets of spectral data are acquired. Wherein the acquired spectral data comprises pure spectral data and noisy spectral data. In this embodiment, each set of samples is repeatedly collected under the same environmental condition, and finally, the mean value of the spectral curve of each set of samples is taken as the spectral curve of the set of samples.
Wherein, aiming at the pure spectral data, the testing environment is set as a darkroom, the used solutions are all prepared by using analytical pure chemical reagents, the used water is deionized water, and the spectral curve obtained under the conditions is taken as a pure spectral curve. The spectrometer used in this example was Shimadzu UV-2600, the sampling interval of the spectrometer was 1nm, and the wavelength range was 185-400 nm.
For example, the nitrate nitrogen and nitrite concentrations are set separately at 0.1, 0.2, 0.4, 0.5, 0.6, 0.8, 1.0, 1.2, 1.4, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0, 6.5, 7.0, 7.5, 8.0, 8.5, 9.0, 9.5, 10 in mg/L, and the mixed solution of nitrate and nitrite is set in a cross form (9 solutions are set in a cross form, the nitrate nitrogen concentration is 0.4, 0.6, 0.8, and the nitrite nitrogen concentration is 0.6, 1.0, 1.4, in mg/L). Sampling samples which are individually prepared with nitrate nitrogen and nitrite in a darkroom environment to obtain pure spectral data; and aiming at the mixed solution of nitrate and nitrite, sampling in a target environment to obtain spectral data as a target spectral curve or sampling in a darkroom environment to obtain pure spectral data, and adding noise data to obtain spectral data as the target spectral curve. In some examples, the solution corresponding to the pure spectral data may also be a mixed solution, and similarly, the solution corresponding to the target spectral curve may also be a solution with a single component, and both solutions may contain the same component.
Wherein the noise spectral data is targeted. The environmental noise has the characteristics of small amplitude change and random occurrence, and is similar to the occurrence characteristics of Gaussian random noise. In the embodiment, Gaussian noise data is used for simulating environmental noise data, and Matlab is used for generating 0.1-10dB Gaussian noise with an interval of 0.1dB, wherein the data point of the Gaussian noise data is the same as the length of each sample spectrum.
Step 102: and constructing a pure spectrum dictionary by using the pure spectrum data and constructing a noise spectrum dictionary by using the noise spectrum data. Wherein, the processing is carried out by utilizing a K-SVD algorithm.
Aiming at the pure spectrum data, in order to retain the absorption spectrum characteristics of the sample, the pure spectrum data is used as a training sample, and a K-SVD algorithm is applied to process the pure spectrum data to obtain a trained pure spectrum dictionary D1。
And (3) processing the noise spectrum data serving as a sample by using a K-SVD algorithm to obtain a trained noise spectrum dictionary D2。
The implementation process of the K-SVD algorithm is explained by taking pure spectral data as an example:
s201: inputting pure spectrum data sample S epsilon RN*MAnd a maximum number of iterations P;
s202: randomly selecting K samples and constructing an initial dictionary D0∈RN*KThen, an initial sparse representation coefficient matrix X is obtained through calculation0. The method can be calculated by the existing algorithm, such as an OMP algorithm;
s203: based on initial dictionary and initial expression coefficient matrix X0Updating and iterating each dictionary atom column by column until the maximum iteration number P is met, and if the maximum iteration number is not met, entering next updating iteration by using the updated dictionary D and the sparse representation coefficient X;
wherein, the kth column dictionary atom d in the dictionarykThe updates of (2) are as follows:
calculating an error matrix based on the following formula
The formula is as follows:
the reasoning process is as follows:
definition set
Wherein omegakIs Nx | omegakMatrix, | in (ω)k(i) I) is 1, and the other points are 0;
then, toThe error matrix
Carrying out SVD to obtain a matrix U and a matrix V;
finally, the first column in the matrix U is selected as the updated dictionary atom dkAnd using the result of the multiplication of the first column in the matrix V with Δ (1,1) as the kth row of data in the sparse representation coefficient X
And updating each row of atoms row by row according to the method, judging whether the maximum iteration times are met, and if not, continuing the iteration.
For noisy spectral data, the manner is similar to that described above, and therefore, the description is omitted.
Step 103: and combining the pure spectrum dictionary and the noise spectrum dictionary to form a joint dictionary. As shown in fig. 2, this is expressed as follows:
D＝[D1,D2]wherein D is1Representing a pure spectral dictionary, D2Representing a noise spectrum dictionary;
Step 104: and processing the target spectrum curve by adopting a minimum angle regression algorithm based on the combined dictionary to obtain an optimal sparse representation coefficient.
The method comprises the steps of sampling a mixed solution of nitrate and nitrite in a target environment to obtain spectral data serving as a target spectral curve or sampling the mixed solution in a darkroom environment to obtain pure spectral data, and adding mixed noise data to obtain the spectral data serving as the target spectral curve.
The process of processing the spectral curve by adopting the minimum angle regression algorithm to obtain the optimal sparse representation coefficient is as follows:
s21: traversing dictionary atoms based on the joint dictionary and the target spectrum curve to obtain a combination of sparse representation coefficients;
firstly, selecting dictionary atoms closest to the target spectral curve from the combined dictionary, and calculating residual error Y based on the selected atoms′The following are:
wherein Y is a target spectrum,
to make use of selected atoms dkAnd coefficient XkForming a target approximation value;
then, selecting the residual Y from the joint dictionary′Calculating the next residual error according to the formula until the residual error is smaller than a preset threshold value or the selected dictionary atom meets a preset requirement;
if a plurality of dictionary atoms are closest to the target spectrum curve or the residual error, selecting all the dictionary atoms meeting the requirement, and further obtaining a combination of a plurality of sparse representation coefficients;
s22: rejecting combinations without noise dictionary atoms from the combinations of the plurality of sparse representation coefficients in step S21;
s23: the combination of the optimal sparse representation coefficients is selected from the remaining combinations of sparse representation coefficients in step S22 as an optimal sparse representation coefficient matrix. The optimum is preferably selected based on the minimum fitting degree, and in other possible embodiments, the optimum may be selected based on other error criteria.
The fit is formulated as follows:
in the formula, R2Is degree of fitting, I1Is noisy spectral data requiring noise reduction, I0Is the spectral data after the noise reduction is completed,
is the mean of the noise spectrum.
Step 105: and obtaining a noise reduction spectrum of the target spectrum curve by using the optimal sparse representation coefficient matrix and the joint dictionary, evaluating, and if the evaluation requirement is not met, adjusting the joint dictionary until the evaluation requirement is met.
Wherein the denoised spectrum Y1＝D×X1Wherein D represents a joint dictionary, X1A sparse representation coefficient matrix is represented.
Generally speaking, the adjustment method is to modify the iteration times of the pure spectrum dictionary and the noise spectrum dictionary learning, and re-learn the dictionary to obtain a new joint dictionary.
Step S106: and (4) denoising the same type of spectral curve by using the joint dictionary meeting the evaluation requirement to obtain a denoised spectrum.
Therefore, the above processes S101-S105 can be understood as constructing a joint dictionary that meets the requirements; and step S106, denoising the spectral curve to be denoised in practical application by using the constructed combined spectrum.
In practical denoising application, the denoising method can be summarized as follows:
step S1: constructing a joint dictionary based on the mode of the steps S101-S105;
step S2: step S2: and denoising the spectral curve to be denoised by using the joint dictionary in the step S1, wherein the spectral curve to be denoised is processed by using a minimum angle regression algorithm to obtain an optimal sparse representation coefficient matrix, and then a denoising spectrum of the spectral curve to be denoised is obtained by using the optimal sparse representation coefficient matrix and the joint dictionary.
In some implementations, the present invention further provides a denoising system based on the above spectral signal denoising method, including: the system comprises a data acquisition unit, a pure spectrum dictionary construction unit, a noise spectrum dictionary construction unit, a joint dictionary construction unit, an optimal sparse representation coefficient acquisition unit and a noise reduction unit;
the data acquisition unit is used for acquiring pure spectrum data and noise spectrum data;
the pure spectrum dictionary construction unit is used for constructing a pure spectrum dictionary by utilizing the pure spectrum data;
the noise spectrum dictionary construction unit is used for constructing a noise spectrum dictionary by using the noise spectrum data;
the combined dictionary construction unit is used for combining the pure spectrum dictionary and the noise spectrum dictionary to form a combined dictionary;
the optimal sparse representation coefficient acquisition unit is used for processing a target spectral curve by adopting a minimum angle regression algorithm based on the joint dictionary to obtain an optimal sparse representation coefficient matrix;
the noise reduction unit is used for obtaining a noise reduction spectrum of the target spectrum curve by utilizing the optimal sparse representation coefficient matrix and the joint dictionary;
the evaluation unit is used for evaluating whether the noise reduction spectrum meets the evaluation requirement; if not, adjusting the joint dictionary until the evaluation requirement is met;
the optimal sparse representation coefficient matrix obtaining unit is further used for processing the spectral curve to be denoised by adopting a minimum angle regression algorithm to obtain an optimal sparse representation coefficient;
and the denoising unit is further used for obtaining a denoising spectrum of the spectrum curve to be denoised by using the optimal sparse representation coefficient and the joint dictionary.
For the specific implementation process of each unit module, refer to the corresponding process of the foregoing method. It should be understood that, the specific implementation process of the above unit module refers to the method content, and the present invention is not described herein in detail, and the division of the above functional module unit is only a division of a logic function, and there may be another division manner in the actual implementation, for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or may not be executed. Meanwhile, the integrated unit can be realized in a hardware form, and can also be realized in a software functional unit form.
In some implementations, the present invention also provides a terminal comprising a memory and a processor, the memory storing a computer program, the processor invoking the computer program to perform: the spectral signal denoising method based on the joint dictionary comprises the following steps:
step S101: several sets of spectral data are acquired.
Step 102: and constructing a pure spectrum dictionary by using the pure spectrum data and constructing a noise spectrum dictionary by using the noise spectrum data.
Step 103: and combining the pure spectrum dictionary and the noise spectrum dictionary to form a joint dictionary.
Step 104: and processing the target spectrum curve by adopting a minimum angle regression algorithm based on the combined dictionary to obtain an optimal sparse representation coefficient matrix.
Step 105: and obtaining a noise reduction spectrum of the target spectrum curve by using the optimal sparse representation coefficient matrix and the joint dictionary, evaluating, and if the evaluation requirement is not met, adjusting the joint dictionary until the evaluation requirement is met.
Or performing:
step S1: constructing a joint dictionary based on the mode of the steps S101-S105;
step S2: step S2: and denoising the spectral curve to be denoised by using the joint dictionary in the step S1, wherein the spectral curve to be denoised is processed by using a minimum angle regression algorithm to obtain an optimal sparse representation coefficient matrix, and then a denoising spectrum of the spectral curve to be denoised is obtained by using the optimal sparse representation coefficient matrix and the joint dictionary.
For the implementation process of each step, please refer to the specific implementation process of the foregoing method, which is not described herein again.
In a fourth aspect, the present invention also provides a readable storage medium storing a computer program, the computer program being invoked by a processor to perform: the spectral signal denoising method based on the joint dictionary comprises the following steps:
step S101: several sets of spectral data are acquired.
Step 102: and constructing a pure spectrum dictionary by using the pure spectrum data and constructing a noise spectrum dictionary by using the noise spectrum data.
Step 103: and combining the pure spectrum dictionary and the noise spectrum dictionary to form a joint dictionary.
Step 104: and processing the target spectrum curve by adopting a minimum angle regression algorithm based on the combined dictionary to obtain an optimal sparse representation coefficient matrix.
Step 105: and obtaining a noise reduction spectrum of the target spectrum curve by using the optimal sparse representation coefficient matrix and the joint dictionary, evaluating, and if the evaluation requirement is not met, adjusting the joint dictionary until the evaluation requirement is met.
Or performing:
step S1: constructing a joint dictionary based on the mode of the steps S101-S105;
step S2: step S2: and denoising the spectral curve to be denoised by using the joint dictionary in the step S1, wherein the spectral curve to be denoised is processed by using a minimum angle regression algorithm to obtain an optimal sparse representation coefficient matrix, and then a denoising spectrum of the spectral curve to be denoised is obtained by using the optimal sparse representation coefficient matrix and the joint dictionary.
For the implementation process of each step, please refer to the specific implementation process of the foregoing method, which is not described herein again.
It should be understood that in the embodiments of the present invention, the Processor may be a Central Processing Unit (CPU), and the Processor may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The memory may include both read-only memory and random access memory, and provides instructions and data to the processor. The portion of memory may also include non-volatile random access memory. For example, the memory may also store device type information.
The readable storage medium is a computer readable storage medium, which may be an internal storage unit of the controller according to any of the foregoing embodiments, for example, a hard disk or a memory of the controller. The readable storage medium may also be an external storage device of the controller, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the controller. Further, the readable storage medium may also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used for storing the computer program and other programs and data required by the controller. The readable storage medium may also be used to temporarily store data that has been output or is to be output.
The performance of the method is compared by using three methods, namely a conventional wavelet hard threshold denoising algorithm, an orthogonal matching tracking algorithm in sparse representation and a minimum angle regression algorithm in sparse representation.
In order to better verify the effectiveness of the method, the noise reduction performance of the method and other methods is quantitatively evaluated by respectively adopting the signal-to-noise ratio and the prediction mean square error between a reference signal and a filtered signal:
1) signal-to-noise ratio (SNR)
2) Prediction mean square error (RMSE)
Where N is the signal curve length, X (N) is the reference curve,
is the noise reduction curve.
The curves of fig. 4 are subjected to noise reduction processing by using three conventional methods and the method of the present invention, and the noise reduction effect is shown in fig. 5. The three evaluation indexes described above were calculated respectively, and the experimental results of the three prior methods and the method of the present invention were compared, as shown in the following table:
quantization parameter comparison table:
the denoising result quantization parameter comparison table shows that:
the signal-to-noise ratio between the curve denoised by the method and the reference curve is obviously higher than that of the other three methods, and the prediction mean square error is obviously lower than that of the other three methods. Therefore, it can be shown that the method of the present invention has the best denoising effect on the spectrum curve of this type, as shown in fig. 6.
It should be understood that the joint dictionary constructed through learning has the best filtering and denoising effect on the spectral signals of the same samples under the same environmental condition. According to the spectral signal denoising method based on the joint dictionary, provided by the invention, the characteristics of the spectrum and the noise are learned, a pure spectrum dictionary and a noise spectrum dictionary are respectively constructed, the joint dictionary is formed by cascading, and then the optimal sparse representation coefficient is obtained by utilizing an improved minimum angle regression method, so that the spectrum denoising is completed. Compared with the existing common method, the method has the best denoising effect, obtains good effects on curve smoothness and various performance indexes, and can be used for denoising the spectrum curve because the dictionary atoms contain the spectrum characteristics of the substance.
It should be emphasized that the examples described herein are illustrative and not restrictive, and thus the invention is not to be limited to the examples described herein, but rather to other embodiments that may be devised by those skilled in the art based on the teachings herein, and that various modifications, alterations, and substitutions are possible without departing from the spirit and scope of the present invention.
Claims (10)
1. A spectral signal denoising method based on a joint dictionary is characterized in that: the method comprises the following steps:
step S1: constructing a joint dictionary, which comprises:
acquiring pure spectrum data and noise spectrum data;
constructing a pure spectrum dictionary by using the pure spectrum data and constructing a noise spectrum dictionary by using the noise spectrum data;
combining the pure spectrum dictionary and the noise spectrum dictionary to form a joint dictionary;
processing a target spectrum curve by adopting a minimum angle regression algorithm based on the combined dictionary to obtain an optimal sparse representation coefficient matrix;
obtaining a noise reduction spectrum of the target spectrum curve by using the optimal sparse representation coefficient matrix and the joint dictionary, evaluating, and if the noise reduction spectrum does not meet the evaluation requirement, adjusting the joint dictionary until the evaluation requirement is met; step S2: and denoising the spectral curve to be denoised by using the joint dictionary in the step S1, wherein the spectral curve to be denoised is processed by using a minimum angle regression algorithm to obtain an optimal sparse representation coefficient matrix, and then a denoising spectrum of the spectral curve to be denoised is obtained by using the optimal sparse representation coefficient matrix and the joint dictionary.
2. The method of claim 1, wherein: a process of obtaining a noise reduction spectrum by using the optimal sparse representation coefficient matrix and the joint dictionary, wherein the noise reduction spectrum is represented as:
Y1＝D×X1
in the formula, Y1For noise-reducing spectra, D is a joint dictionary, X1Representing a coefficient matrix for the optimal sparsity;
the joint dictionary D is represented as:
D＝[D1，D2]wherein D is1Representing a pure spectral dictionary, D2Representing a noise spectrum dictionary.
3. The method of claim 1, wherein: the pure spectrum dictionary and the noise spectrum dictionary are obtained by adopting a dictionary learning algorithm of K-SVD, and the process is as follows:
s201: input sample data SN*MAnd the maximum iteration number P, wherein if the pure spectrum dictionary is obtained by training, the corresponding sample data S is the pure spectrum data; if the noise spectrum dictionary is obtained through training, corresponding sample data S is noise spectrum data, N is the number of wavelength points of each sample spectrum, and M is a sample dimension.
S202: randomly selecting K samples and constructing an initial dictionary D0∈SN*KThen, an initial sparse representation coefficient matrix X is obtained through calculation0；
S203: based on initial dictionary and initial expression coefficient matrix X0Updating and iterating each dictionary atom column by column until the maximum iteration number P is met, and if the maximum iteration number is not met, entering next updating iteration by using the updated dictionary D and the sparse representation coefficient matrix X;
wherein, the kth column dictionary atom d in the dictionarykThe updates of (2) are as follows:
calculating an error matrix based on the following formula
In which Y is a sample signal, djFor the jth column of dictionary atoms in the current dictionary,
for the jth row of data, Ω, in the current sparse representation coefficient XkIs Nx | omegakI matrix, definition set
Is composed of
Data corresponding to the ith wavelength point;
for the error matrix
Carrying out SVD to obtain a matrix U and a matrix V;
finally, the first column in matrix U is selected as the updated dictionary atom dk, and the product of the first column in matrix V and Δ (1,1) is used as the kth row of data in the sparse representation coefficient matrix X
4. The method of claim 1, wherein: the process of processing the spectral curve by the minimum angle regression method to obtain the optimal sparse representation coefficient matrix is as follows:
s21: traversing dictionary atoms based on the joint dictionary and the target spectrum curve to obtain a combination of sparse representation coefficients;
firstly, selecting dictionary atoms closest to the target spectral curve from the combined dictionary, and calculating residual errors Y' based on the selected atoms as follows:
wherein Y is a target spectrum,
to make use of selected atoms dkAnd coefficient XkForming a target approximation value;
then, selecting a dictionary atom which is closest to the residual error Y' from the combined dictionary, and calculating the next residual error according to the formula until the residual error is smaller than a preset threshold value or the selected dictionary atom meets preset requirements;
if a plurality of dictionary atoms are closest to the target spectrum curve or the residual error, selecting all the dictionary atoms meeting the requirement, and further obtaining a combination of a plurality of sparse representation coefficients;
s22: rejecting combinations without noise dictionary atoms from the combinations of the plurality of sparse representation coefficients in step S21;
s23: the combination of the optimal sparse representation coefficients is selected from the remaining combinations of sparse representation coefficients in step S22 as an optimal sparse representation coefficient matrix.
5. The method of claim 1, wherein: when the combination of the optimal sparse representation coefficients is selected in step S23, the fitting degree is the minimum, and the fitting degree formula is as follows:
in the formula, R2Is degree of fitting, I1Is noisy spectral data requiring noise reduction, I0Is the spectral data after the noise reduction is completed,
is the mean of the noise spectrum.
6. The method of claim 1, wherein: the spectral curve to be processed and the target spectral curve are the same type of spectral curve, the target spectral curve is opposite to the spectral curve to be processed, and the chemical substances of the target spectral curve and the target spectral curve are the same.
7. The method of claim 1, wherein: the testing environment of the pure spectrum data is a darkroom; and adopting Gaussian random noise to simulate environmental noise data in the process of acquiring the noise spectrum data.
8. A denoising system based on the method of any one of claims 1-7, wherein: the method comprises the following steps: the system comprises a data acquisition unit, a pure spectrum dictionary construction unit, a noise spectrum dictionary construction unit, a joint dictionary construction unit, an optimal sparse representation coefficient acquisition unit and a noise reduction unit;
the data acquisition unit is used for acquiring pure spectrum data and noise spectrum data;
the pure spectrum dictionary construction unit is used for constructing a pure spectrum dictionary by utilizing the pure spectrum data;
the noise spectrum dictionary construction unit is used for constructing a noise spectrum dictionary by using the noise spectrum data;
the combined dictionary construction unit is used for combining the pure spectrum dictionary and the noise spectrum dictionary to form a combined dictionary;
the optimal sparse representation coefficient acquisition unit is used for processing a target spectral curve by adopting a minimum angle regression algorithm based on the joint dictionary to obtain an optimal sparse representation coefficient matrix;
the noise reduction unit is used for obtaining a noise reduction spectrum of the target spectrum curve by utilizing the optimal sparse representation coefficient matrix and the joint dictionary;
the evaluation unit is used for evaluating whether the noise reduction spectrum meets the evaluation requirement; if not, adjusting the joint dictionary until the evaluation requirement is met;
the optimal sparse representation coefficient matrix obtaining unit is further used for processing the spectral curve to be denoised by adopting a minimum angle regression algorithm to obtain an optimal sparse representation coefficient matrix;
the denoising unit is further configured to obtain a denoising spectrum of the spectral curve to be denoised by using the optimal sparse representation coefficient matrix and the joint dictionary.
9. A terminal, characterized by: comprising a memory storing a computer program and a processor that invokes the computer program to perform: the method for denoising spectroscopic signals based on a joint dictionary as set forth in any one of claims 1 to 7.
10. A readable storage medium, characterized by: a computer program is stored, which is invoked by a processor to perform: the method for denoising spectroscopic signals based on a joint dictionary as set forth in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110285933.XACN113011321B (en) | 2021-03-17 | 2021-03-17 | Spectral signal denoising method, system, terminal and readable storage medium based on joint dictionary |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110285933.XACN113011321B (en) | 2021-03-17 | 2021-03-17 | Spectral signal denoising method, system, terminal and readable storage medium based on joint dictionary |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113011321Atrue CN113011321A (en) | 2021-06-22 |
| CN113011321B CN113011321B (en) | 2022-05-06 |
Family
ID=76409229
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110285933.XAActiveCN113011321B (en) | 2021-03-17 | 2021-03-17 | Spectral signal denoising method, system, terminal and readable storage medium based on joint dictionary |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113011321B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114861737A (en)* | 2022-06-07 | 2022-08-05 | 常熟理工学院 | Remote disturbance feature extraction method and system of distributed optical fiber sensing system |
| CN116152578A (en)* | 2023-04-25 | 2023-05-23 | 深圳湾实验室 | Training method and device for noise reduction generation model, noise reduction method and medium |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120250748A1 (en)* | 2011-04-04 | 2012-10-04 | U.S. Government As Represented By The Secretary Of The Army | Apparatus and method for sampling and reconstruction of wide bandwidth signals below nyquist rate |
| US20140136585A1 (en)* | 2009-11-03 | 2014-05-15 | Vivaquant Llc | Physiological signal denoising |
| CN105989595A (en)* | 2015-02-12 | 2016-10-05 | 中国科学院西安光学精密机械研究所 | Multi-temporal remote sensing image change detection method based on joint dictionary learning |
| CN107274360A (en)* | 2017-05-24 | 2017-10-20 | 南京师范大学 | A kind of high spectrum image denoising method based on Fisher dictionary learnings, low-rank representation |
| CN107833208A (en)* | 2017-10-27 | 2018-03-23 | 哈尔滨工业大学 | A kind of hyperspectral abnormity detection method based on changeable weight depth own coding |
| CN109632310A (en)* | 2019-01-18 | 2019-04-16 | 北京化工大学 | A Bearing Fault Diagnosis Method Based on Feature Enhancement |
| CN109946253A (en)* | 2019-04-08 | 2019-06-28 | 中南大学 | A spectral denoising method |
| CN110400276A (en)* | 2019-08-28 | 2019-11-01 | 上海海洋大学 | Hyperspectral image denoising method and device |
| CN110458777A (en)* | 2019-08-05 | 2019-11-15 | 湖南大学 | A hyperspectral image denoising method, system and medium based on adaptive rank correction |
| CN110503060A (en)* | 2019-08-28 | 2019-11-26 | 中南大学 | A spectral signal denoising method and system thereof |
| CN111538759A (en)* | 2020-04-20 | 2020-08-14 | 中南大学 | Industrial process intelligent monitoring method and system based on distributed dictionary learning |
- 2021
- 2021-03-17CNCN202110285933.XApatent/CN113011321B/enactiveActive
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140136585A1 (en)* | 2009-11-03 | 2014-05-15 | Vivaquant Llc | Physiological signal denoising |
| US20120250748A1 (en)* | 2011-04-04 | 2012-10-04 | U.S. Government As Represented By The Secretary Of The Army | Apparatus and method for sampling and reconstruction of wide bandwidth signals below nyquist rate |
| CN105989595A (en)* | 2015-02-12 | 2016-10-05 | 中国科学院西安光学精密机械研究所 | Multi-temporal remote sensing image change detection method based on joint dictionary learning |
| CN107274360A (en)* | 2017-05-24 | 2017-10-20 | 南京师范大学 | A kind of high spectrum image denoising method based on Fisher dictionary learnings, low-rank representation |
| CN107833208A (en)* | 2017-10-27 | 2018-03-23 | 哈尔滨工业大学 | A kind of hyperspectral abnormity detection method based on changeable weight depth own coding |
| CN109632310A (en)* | 2019-01-18 | 2019-04-16 | 北京化工大学 | A Bearing Fault Diagnosis Method Based on Feature Enhancement |
| CN109946253A (en)* | 2019-04-08 | 2019-06-28 | 中南大学 | A spectral denoising method |
| CN110458777A (en)* | 2019-08-05 | 2019-11-15 | 湖南大学 | A hyperspectral image denoising method, system and medium based on adaptive rank correction |
| CN110400276A (en)* | 2019-08-28 | 2019-11-01 | 上海海洋大学 | Hyperspectral image denoising method and device |
| CN110503060A (en)* | 2019-08-28 | 2019-11-26 | 中南大学 | A spectral signal denoising method and system thereof |
| CN111538759A (en)* | 2020-04-20 | 2020-08-14 | 中南大学 | Industrial process intelligent monitoring method and system based on distributed dictionary learning |
Non-Patent Citations (4)
| Title |
|---|
| WANG X ET AL: "Faulty feeder detection based on mixed atom dictionary and energy spectrum energy for distribution network", 《IET GENERATION TRANSMISSION & DISTRIBUTION》* |
| Y. ZHANG ET AL: "novel image compression-encryption hybrid algorithm based on the analysis sparse representation", 《OPT. COMMUN》* |
| 张静妙等: "基于低秩字典学习的高光谱遥感图像去噪", 《控制工程》* |
| 郑国梁等: "改进LMS的光谱信号去噪算法", 《光谱学与光谱分析》* |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114861737A (en)* | 2022-06-07 | 2022-08-05 | 常熟理工学院 | Remote disturbance feature extraction method and system of distributed optical fiber sensing system |
| CN116152578A (en)* | 2023-04-25 | 2023-05-23 | 深圳湾实验室 | Training method and device for noise reduction generation model, noise reduction method and medium |
| CN116152578B (en)* | 2023-04-25 | 2023-07-18 | 深圳湾实验室 | Training method and device for noise reduction generation model, noise reduction method and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113011321B (en) | 2022-05-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Hancock et al. | The reliability paradox in assessing structural relations within covariance structure models | |
| CN105352895B (en) | High-spectrum remote sensing data vegetation information extraction method | |
| CN109946253B (en) | Spectrum denoising method | |
| Chen et al. | A background and noise elimination method for quantitative calibration of near infrared spectra | |
| CN113011321A (en) | Spectral signal denoising method, system, terminal and readable storage medium based on joint dictionary | |
| CN109883547A (en) | A Broadband Spectral Signal Denoising Method Based on Wavelet Threshold Difference | |
| CN110503060B (en) | A spectral signal denoising method and system thereof | |
| CN111259269A (en) | Abnormal point detection method, apparatus and storage medium based on time series data | |
| CN117074333B (en) | COD soft measurement model construction method based on ultraviolet-visible light absorption spectrum | |
| CN111766210A (en) | A multispectral measurement method of nitrate nitrogen in complex seawater near shore | |
| CN115795272B (en) | Spectral line denoising method based on fractional order iterative discrete wavelet transform | |
| CN115015120A (en) | Fourier infrared spectrometer and temperature drift online correction method thereof | |
| CN110579746A (en) | Echo signal processing method, device, equipment and storage medium | |
| CN111198167B (en) | Method for detecting concentration of metal ions in solution, terminal device and storage medium | |
| Badawi et al. | Real-time low-cost drift compensation for chemical sensors using a deep neural network with hadamard transform and additive layers | |
| CN115931727B (en) | Chlorophyll a remote sensing inversion method, device and system and storage medium | |
| CN113971641A (en) | Wavelet threshold denoising method, device, equipment and medium | |
| CN118501079A (en) | Gas absorption spectrum signal denoising method and system | |
| CN110118958B (en) | Broadband radar complex echo denoising method based on variational coding-decoding network | |
| CN109839362B (en) | Infrared spectrum quantitative analysis method based on progressive denoising technology | |
| CN119619047A (en) | A red soil organic matter spectrum inversion method and device for removing the influence of free iron | |
| CN118965185A (en) | Soil texture classification method and device based on hyperspectral imagery | |
| CN109670531A (en) | A kind of denoising method of the near infrared light spectrum signal based on Hodrick-Prescott filter | |
| Malik et al. | Support vector regression with digital band pass filtering for the quantitative analysis of near‐infrared spectra | |
| CN110632024B (en) | Quantitative analysis method, device and equipment based on infrared spectrum and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |