Disclosure of Invention
The embodiment of the application provides a sleep state detection method and wearable equipment, which are used for improving the accuracy of sleep state detection of a user and further improving the wearing experience of the user.
In order to achieve the above purpose, the embodiment of the present application adopts the following technical scheme:
In a first aspect, a method for detecting a sleep state is provided and applied to a wearable device, where the wearable device acquires PPG data and ACC data. Then, the wearable device may process PPG data to obtain an R-R interval RRI sequence, and process ACC data to obtain a motion sequence. The wearable device may then predict a sleep state of a wearer of the wearable device based on the RRI sequence and the motion sequence, wherein the sleep state of the wearer includes a fast eye movement state, a deep sleep state, a shallow sleep state, or an awake state.
According to the RRI sequence and the movement sequence, the wearable device can obtain four classification results including a rapid eye movement state, a deep sleep state, a shallow sleep state and a waking state. Therefore, the sleeping state of the wearer can be comprehensively detected, the probability of false detection of the sleeping state is reduced, the detection precision of the sleeping state is improved, a foundation is provided for the subsequent sleeping intervention of the wearer in time according to the sleeping state, and the use experience of the wearer is improved.
In one possible implementation manner of the first aspect, the process of acquiring PPG data and ACC data by the wearable device may specifically include that, in a case that a wearing operation of the wearable device by the wearer is detected, the wearable device acquires PPG data by the PPG sensor in real time, and acquires ACC data by the ACC sensor.
In the application, if the wearable device is worn on a designated part of the body of a wearer, the wearable device can acquire PPG data through a PPG sensor in real time and acquire ACC data through an ACC sensor. Therefore, the sleeping condition of the wearer can be detected in real time, the sleeping intervention of the wearer can be performed on the wearer according to the sleeping condition, so that the sleeping quality of the wearer can be improved, and the wearer can fall asleep better. In addition, because PPG data can also be used for detecting the physiological condition (such as blood oxygen, blood pressure, etc.) of the wearer, therefore, if the PPG sensor can gather PPG data in real time, the wearable equipment can detect the physiological condition of the wearer according to the PPG data in real time, so that the user can conveniently check the physiological condition of the user in real time, and the user can be reminded of taking a rest in time under the condition that the physiological condition of the wearer is in an abnormal state, and the wearing experience of the user is further improved. Moreover, because ACC data can also be used for detecting the motion condition (such as step number) of a wearer, if an ACC sensor can acquire ACC data in real time, the wearable device can detect the motion condition of the wearer according to the ACC data in real time, so that the user can know the motion condition of the user conveniently, and further a reasonable regulation and control motion plan is realized, so that the wearer can exercise healthily.
In a possible implementation manner of the first aspect, the process of acquiring PPG data and ACC data by the wearable device may specifically include that, in a case where a wearing operation of the wearable device by the wearer is detected, the wearable device may acquire PPG data within a first preset period of time, and acquire ACC data within a second preset period of time.
In the application, because the PPG data and the ACC data are used for detecting the sleep state of the wearer, and the wearer generally sleeps in a fixed period, the wearable device can only collect the PPG data and the ACC data in the fixed period, and thus, the utilization rate of collection resources can be improved.
In a possible implementation manner of the first aspect, the process of acquiring PPG data and ACC data by the wearable device may specifically include that the wearable device acquires the ACC data. Then, the wearable device can perform feature extraction on the ACC data to obtain a motion sequence, wherein the motion sequence is used for representing whether a wearer to which the wearable device belongs is in a motion state or not. Thereafter, the wearable device acquires PPG data if the motion sequence characterizes that the wearer is not in motion.
In the application, the wearable equipment only needs to acquire the PPG data when the motion sequence indicates that the wearer is not in a motion state, namely, the wearer is likely to fall asleep, so that the accuracy of sleep state detection is ensured, and the utilization rate of acquisition resources is improved, thereby reducing the waste of storage resources.
In a possible implementation manner of the first aspect, the process of obtaining PPG data and ACC data by the wearable device may specifically include receiving, by the wearable device, a data acquisition indication of the electronic device if the first condition is met. Thereafter, in response to the data acquisition indication, the wearable device acquires PPG data as well as ACC data.
The first condition comprises at least one of the following conditions that the current time is in a preset sleep period, and the electronic equipment is in a screen-off state, and the screen-off duration reaches a preset duration.
According to the application, the electronic equipment is automatically turned off in consideration of the fact that the electronic equipment does not receive touch operation of a wearer for a long time, namely, the electronic equipment is converted from a bright screen state to a off screen state, or the electronic equipment can be converted from the bright screen state to the off screen state under the condition that the electronic equipment receives pressing operation of the wearer for a shutdown key, so that the wearable equipment can acquire PPG data and ACC data under the condition that the wearable equipment receives data acquisition instructions sent by the electronic equipment. Therefore, the waste of subsequent computing resources caused by invalid acquisition of the sensor can be reduced, and the utilization rate of the computing resources is improved.
In addition, considering that the wearer may not use the mobile phone for a long time due to daytime operation, if the electronic device is in a screen-off state, and the screen-off time reaches the preset time, the possibility that the wearer falls asleep is indicated, so that the wearable device can acquire the PPG data and the ACC data under the condition of receiving the data acquisition instruction sent by the electronic device. Therefore, accurate detection of the intelligent watch is realized, waste of subsequent computing resources caused by invalid acquisition of the sensor is reduced, and the utilization rate of the computing resources is improved.
In one possible implementation manner of the first aspect, the process of obtaining PPG data and ACC data by the wearable device may specifically include that the wearable device obtains a pressure value between the wearable device and a wrist of a wearer, where the pressure value is used to characterize a fit degree between a dial of the wearable device and the wrist of the wearer. Then, in case the pressure value is within a preset pressure interval, the wearable device may acquire PPG data as well as ACC data.
According to the application, when the pressure value is in the preset pressure interval, the wearing degree of the wearable device is proper, so that the wearable device can directly acquire PPG data and ACC data. Therefore, accurate collection of data can be realized, the occurrence of inaccurate data collection caused by too loose or too tight wearing of the intelligent watch is reduced, and the detection accuracy of the sleep state is further improved.
In a possible implementation manner of the first aspect, the method further includes that the wearable device may send out a prompt message when the current time is within a preset sleep period and the pressure value is not within a preset pressure interval. The prompting information is used for prompting a wearer to adjust the wearing tightness degree of the wearable equipment.
According to the application, when the pressure value is not in the preset pressure interval, the situation that the wearable equipment is too loose or too tight is indicated, so that the wearable equipment can output prompt information when the current time is in the preset sleep period and the pressure value is not in the preset pressure interval, and thus, a wearer can be timely informed to adjust the tightness of the wearable equipment, a foundation is provided for accurately acquiring data subsequently, and the detection accuracy of the sleep state is further improved.
In a possible implementation manner of the first aspect, the method further includes that the wearable device interpolates the RRI sequence to obtain a target RRI sequence with the same sampling rate as the motion sequence. Thereafter, the wearable device may predict a sleep state of the wearer based on the target RRI sequence and the motion sequence.
In the application, the RRI sequence after processing (namely the target RRI sequence) and the motion sequence can have the same sampling rate by carrying out interpolation processing on the RRI sequence. Therefore, the time alignment among different sequences can be realized, namely, different sequences under the same acquisition time are aligned, the occurrence of the situation that the detection result of the sleep state is influenced due to the fact that time points among the sequences are different (for example, the target RRI sequence of the first second and the motion sequence of the second are used as the input of the same moment) is reduced, and a basis is provided for the follow-up accurate detection of the sleep state.
In a possible implementation manner of the first aspect, the process of predicting the sleep state of the wearer by the wearable device may specifically include that the wearable device may input the target RRI sequence and the motion sequence into a sleep state detection model to obtain the sleep state of the wearer. Wherein the sleep state detection model has the ability to predict sleep states based on the target RRI sequence and the motion sequence.
In the application, the wearable device can output the four-classification result only through one state detection model, so that the occurrence of the condition of computing resource waste caused by multi-model detection can be reduced, the utilization rate of computing resources is improved, and the detection efficiency of sleep states is further improved.
In a possible implementation manner of the first aspect, the sleep state detection model includes a CNN layer, where the CNN layer is used to perform feature extraction on a target RRI sequence and an operation sequence, and the CNN layer includes a convolution kernel, and the process of predicting a sleep state of a wearer by the wearable device specifically may include that the wearable device may continuously use the RRI sequence and the motion sequence in a first period of time as input, and operate the sleep state detection model, where the sleep state detection model slides the convolution kernel with a preset step size, and outputs a continuous sleep state of the wearer in the first period of time.
According to the application, the sleep state of the wearer in the period of the convolution kernel is detected by sliding the convolution kernel on the input data (namely the RRI sequence and the motion sequence), so that the real-time detection of the sleep state can be realized, the intelligent watch can intervene in time on the wearer according to the real-time detection result, the sleep condition of the wearer is improved, and the sleep quality of the wearer is further improved.
In a possible implementation manner of the first aspect, the width of the convolution kernel is a preset convolution time, the preset convolution time is a time length of input data included in one convolution kernel, and the height of the convolution kernel is the number of parameters of the input data, where the input data includes an RRI sequence and a motion sequence.
In the application, the height of the convolution kernel is the number of parameters of the input data, that is, the convolution kernel comprises all the input parameters, so that the sleep state can be more comprehensively detected, the intelligent watch can more accurately detect the sleep state of the wearer, further, a foundation is provided for the subsequent timely sleep intervention of the wearer according to the sleep state, and the use experience of the wearer is improved.
In a possible implementation manner of the first aspect, the method further includes the wearable device acquiring skin resistance data. Then, the wearable device can perform feature extraction on the skin resistance data to obtain a skin resistance sequence. Wherein the sampling rate of the skin resistance sequence is the same as the sampling rate of the motion sequence described above. Thereafter, the wearable device may predict a sleep state of a wearer of the wearable device from the skin resistance sequence, RRI sequence, and motion sequence.
In the application, the wearable device can also predict the sleep state of the wearer according to the skin resistance sequence. So, can follow the sleep state of a plurality of angle detection wearers for the intelligent wrist-watch can be more accurate detect the sleep state of wearers, and then in time intervene to the wearers according to the sleep state for follow-up in time to provide the basis, promoted the user experience of wearers.
In a possible implementation manner of the first aspect, the method further includes performing sleep intervention on the wearer by the wearable device according to the sleep state.
In the application, after the wearable device obtains the sleep state of the wearer, the wearable device can perform sleep intervention on the wearer according to the sleep state. Thus, the sleeping condition of the wearer can be improved, and the sleeping improvement service matched with the current sleeping state is provided, so that the sleeping quality of the wearer is improved.
In a second aspect, the application provides a wearable device comprising a PPG sensor, an ACC sensor, a display screen, a memory and one or more processors, the PPG sensor, ACC sensor, display screen, memory and processors being coupled, the PPG sensor being for acquiring PPG data, the ACC sensor being for acquiring ACC data, the display screen being for displaying an image generated by the processors, the memory being for storing computer program code comprising computer instructions that when executed by the processors cause the wearable device to perform a method as described above.
In a possible implementation manner of the second aspect, the wearable device further includes a GSR sensor, the GSR sensor is coupled to the PPG sensor, the ACC sensor, the display screen, the memory, and the processor, and the GSR sensor is configured to collect skin resistance data.
In a third aspect, the application provides a computer readable storage medium comprising computer instructions which, when run on an electronic device, cause the electronic device to perform the method as described above.
In a fourth aspect, the application provides a computer program product for, when run on an electronic device, causing the electronic device to perform the method as described above.
In a fifth aspect, a chip is provided, which includes an input interface, an output interface, a processor, and a memory, where the input interface, the output interface, the processor, and the memory are connected by an internal connection path, and the processor is configured to execute a code in the memory, and when the code is executed, the processor is configured to execute a method as described above.
The advantages achieved by the electronic device according to the second aspect, the computer readable storage medium according to the third aspect, the computer program product according to the fourth aspect, and the chip according to the fifth aspect may refer to the advantages of the first aspect and any one of the possible design manners thereof, and are not described herein.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. In the description of the present application, unless otherwise indicated, "and/or" in the present application is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B, and that three cases where a exists alone, a exists together with B, and B exists alone, where a and B may be singular or plural. Also, in the description of the present application, unless otherwise indicated, "a plurality" means two or more than two. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (a, b, or c) of a, b, c, a-b, a-c, b-c, or a-b-c may be represented, wherein a, b, c may be single or plural. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same item or similar items having substantially the same function and effect. It will be appreciated by those of skill in the art that the words "first," "second," and the like do not limit the amount and order of execution, and that the words "first," "second," and the like do not necessarily differ. Meanwhile, in the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as examples, illustrations or explanations. Any embodiment or design described herein as "exemplary" or "e.g." in an embodiment should not be taken as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion that may be readily understood.
In some application scenarios, a wearable device (such as the smart watch 200 shown in fig. 1) may be secured on the wrist of a wearer (or referred to as a user) by a wristband. The wearable device may detect a physiological parameter of the wearer and display the physiological parameter (e.g., heart rate) to make the wearer aware of the current physiological condition.
In some embodiments, as shown in fig. 2, the wearable device may obtain the R-R interval (RR INTERVAL, RRI) sequence from a photoplethysmography (photoplethysmographic, PPG) signal (or PPG data) acquired by a PPG sensor. Wherein, the R-R interval refers to the time limit between two adjacent R waves on an electrocardiogram. In the QRS complex, the break point at which the Q wave is located is the start of the R wave, and the time elapsed from this start to the next start is called the RR interval, i.e., the interval of each heart beat. The QRS complex is used to reflect changes in left and right ventricular depolarization potentials and time. Under normal conditions, the time limit of the R-R interval should be between 0.6 and 1.0 seconds, if the R-R interval is less than 0.6 seconds, the condition of tachycardia is indicated to the wearer, and if the R-R interval is less than 1.0 seconds, the condition of bradycardia is indicated to the wearer. In addition, when the R-R intervals are unequal, the wearer is shown to have arrhythmia, and obvious RR intervals are unequal on an electrocardiogram of atrial fibrillation.
Specifically, the wearable device may acquire the PPG signal acquired by the PPG sensor. Then, the wearable device may perform peak detection on the PPG signal to obtain an RRI sequence.
In case the RRI sequence is obtained as described above, the wearable device may extract Heart Rate Variability (HRV) information from the RRI sequence. Wherein the HRV information may include at least one of mean, standard deviation, and root mean square of adjacent RR interval differences of RRI sequences per minute. Then, the wearable device can detect the first sleep state of the wearer according to the HRV information to obtain the sleep condition of the wearer. Wherein the sleep condition is used to characterize the sleep quality of the wearer. It will be appreciated that a longer duration of time that the wearer is in a deep sleep state will be indicative of a higher quality of sleep for the wearer and a longer duration of time that the wearer is in a rapid eye movement state will be indicative of a lower quality of sleep for the wearer.
In some embodiments, the wearable device may input the HRV information into the first state detection model, resulting in a first sleep state of the wearer. Wherein the first sleep state may include at least one of a fast eye movement (rapid eye movement, REM) state, a deep sleep state, and a shallow sleep state. The fast eye movement state refers to a sleep state in which brain waves appearing during sleep become faster in frequency and lower in amplitude, and also show such things as an increase in heart rate, an increase in blood pressure, a relaxation of muscles, and a continuous left and right swing of eyeballs. The deep sleep state is also called a deep sleep state, and refers to a sleep state in which the cortical cells are in a fully resting state. The light sleep state is also called a light sleep state, which refers to a sleep state in which the activity frequency of the body is high and the sleep sensitivity is high, that is, a user in the light sleep state is easy to turn over, dream, and is more easy to wake up than a user in the deep sleep state.
The first state detection model is a model capable of detecting a first sleep state of a user. The first state detection model may be a decision tree model, a support vector machine (support vector machine, SVM) model, a time convolution network (temporal convolutional network, TCN) or the like, which is not particularly limited as long as a classification model can be used as the first state detection model.
It can be seen that the first sleep state of the wearer detected by the wearable device through the PPG signal only includes a fast eye movement state, a deep sleep state and a shallow sleep state, and does not include an awake state. That is, the sleep state detection method cannot comprehensively detect the sleep state of the wearer, that is, the sleep state detection mode is limited, and thus a situation that a sleep state is erroneously detected (for example, a sleep state which is not in the first sleep state is erroneously detected as the first sleep state, that is, an awake state is erroneously detected as the fast eye movement state) may occur due to incomplete sleep state detection, so that the accuracy of the sleep state detection is lower, and the wearing experience of the user is finally affected.
In other embodiments, as shown in fig. 3, the wearable device may derive a motion sequence from an accelerometer (accelerometer, ACC) signal (or ACC data) acquired by an ACC sensor. Wherein the movement sequence is used to characterize whether the wearer is in motion. Specifically, the wearable device may acquire the ACC signal acquired by the ACC sensor. Then, the wearable device can perform feature extraction on the ACC signal to obtain the motion sequence.
Then, the wearable device can detect the second sleep state of the wearer according to the motion sequence, so as to obtain the sleeping condition of the wearer. Wherein the second sleep state may comprise a sleep state and/or an awake state. The sleep state refers to a state in which a person is sleeping. The awake state refers to the conscious state where the cerebral cortex and the whole body are awake. The falling asleep condition is used to characterize whether the wearer is in a falling asleep state.
In one implementation, the wearable device may input the above-described motion sequence into a state machine, resulting in a second sleep state of the wearer. Wherein the state machine is capable of performing state transition according to a preset sleep state (or called a second sleep state) according to a motion sequence, that is, the wearable device can determine whether the wearer is in the sleep state according to the motion sequence through the state machine.
In another implementation, the wearable device may input the above-described motion sequence into a second state detection model, resulting in a second sleep state of the wearer. The second state detection model is a model capable of detecting a second sleep state of the user. The second state detection model may be the same as the first state detection model, or may be different from the first state detection model, and is not particularly limited.
The training process of the second state detection model may specifically include that the wearable device acquires a motion sequence set, where the motion sequence set includes a plurality of motion sequences, and each motion sequence carries a tag of a second sleep state. Then, the wearable device can train the pre-built second state detection model according to the motion sequence set to obtain a trained second state detection model, so that the classification of the second sleep state of the wearer is realized.
It can be seen that the second sleep state of the wearer detected by the wearable device through the ACC signal only includes a sleep state and an awake state, and cannot be distinguished from the sleep state. That is, the sleep state detection method cannot detect the sleep condition of the wearer, and cannot further analyze the sleep quality of the wearer, so that the sleep intervention is performed on the wearer, and the wearing experience of the user is lower.
Therefore, in order to accurately detect the sleep state of the wearer, the embodiment of the application provides a sleep state detection method, in which the wearable device acquires the RRI sequence and the movement sequence. Wherein the RRI sequence is derived based on the PPG signal and the motion sequence is derived based on the ACC signal. Then, the wearable device may input the RRI sequence and the motion sequence into a state detection model to obtain a sleep state of the user. Wherein the sleep state includes a fast eye movement state, a deep sleep state, a shallow sleep state, or an awake state.
In the embodiment of the application, the wearable device inputs the RRI sequence and the motion sequence into the state detection model at the same time, so that four classification results including a rapid eye movement state, a deep sleep state, a shallow sleep state and a waking state can be obtained. Therefore, the sleeping state of the wearer can be comprehensively detected, the probability of false detection of the sleeping state is reduced, the detection precision of the sleeping state is improved, and the wearing experience of the user is further improved. In addition, the wearable device can output four classification results only through one state detection model, so that the occurrence of the condition of computing resource waste caused by multi-model detection can be reduced, the utilization rate of computing resources is improved, and the detection efficiency of sleep states is further improved.
The wearable device is, for example, a device capable of acquiring physiological parameters of a wearer, such as a smart watch, a smart bracelet, or the like, in contact with the wearer. The hardware structure of the wearable device will be described below with reference to fig. 4, taking the wearable device as a smart watch as an example.
As shown in fig. 4, the smart watch 200 includes a watch body and a wristband (or watchband) connected to each other, wherein the watch body may include a front case (not shown in fig. 4), a touch screen 210 (also called a touch panel), a display screen 220, a bottom case (not shown in fig. 4), and a processor 230, a memory 250, a Microphone (MIC) 260, a communication module 270, a PPG sensor 281, an ACC sensor 282, and an ambient light sensor 283. Although not shown, the smart watch may also include a power source, a power management system, an antenna, speakers, a gyroscopic sensor, etc. Those skilled in the art will appreciate that the smart watch structure shown in fig. 4 is not limiting and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
The following describes each functional component of the smart watch 200:
The touch panel 210, also referred to as a touch pad, may collect touch operations thereon by a user of the wristwatch (e.g., operations of the user on or near the touch panel using any suitable object or accessory such as a finger, stylus, etc.), and actuate the responsive connection device according to a predetermined program.
The display 220 may be used to display information entered by the user or provided to the user as well as various menus of the wristwatch. Alternatively, the display 220 may be configured in the form of an LCD, OLED, or the like. Further, the touch panel 210 may cover the display 220, and when the touch panel 210 detects a touch operation thereon or thereabout, the touch panel is transferred to the processor 230 to determine the type of touch event, and then the processor 230 provides a corresponding visual output on the display 220 according to the type of touch event. Although in fig. 4, the touch panel 210 and the display 220 are two separate components to implement the input and output functions of the wristwatch, in some embodiments, the touch panel 210 and the display 220 may be integrated to implement the input and output functions of the wristwatch.
The processor 230 is used for system scheduling, control of a display screen, a touch screen, processing of data sent by sensors (such as PPG sensor 281, ACC sensor 282, ambient light sensor 283 described above), etc. The processor 230 may also be referred to as a main control unit, which may include a computing unit, which may process data, etc.
The memory 250 is used to store software programs and data, and the processor 230 performs various functional applications and data processing of the wristwatch by running the software programs and data stored in the memory. The memory 250 mainly includes a storage program area that can store an operating system, an application program required for at least one function (such as a sound playing function, an image playing function, etc.), and a storage data area that can store data created from using a hand table (such as audio data, a phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include nonvolatile memory, such as magnetic disk storage devices, flash memory devices, or other volatile solid-state storage devices.
The communication module 270, the smart watch may interact information with other electronic devices (e.g., a mobile phone, a tablet computer, etc.) through the communication module 270. By way of example, the communication module 270 may include a wireless communication module and a mobile communication module. Alternatively, the wireless communication module may include a Bluetooth (BT) module, a global navigation satellite system (global navigation SATELLITE SYSTEM, GNSS), a wireless local area network (wireless local area networks, WLAN) (e.g., (WIRELESS FIDELITY, wi-Fi) network). The mobile communication module can provide a solution comprising 2G/3G/4G/5G wireless communication applied to the smart watch.
The PPG sensor 281 may be used to measure heart rate of the user, and/or blood oxygen saturation, etc. physiological data of the human body. Taking the example of detecting the heart rate of the user, in a specific implementation, the PPG sensor 281 may emit a light signal of a specified wavelength that may impinge on arterial blood vessels under skin tissue and be reflected back to the PPG sensor 281. When the heart beats, the contraction and expansion of the blood vessel changes the volume of blood in the arterial vessel, and thus affects the absorption or attenuation of the optical signal by the arterial vessel, and thus affects the reflection of the optical signal. The PPG sensor 281 may detect the heart rate of the user based on the change in the reflected light signal. It should be noted that the PPG sensor 281 may also detect the heart rate of the user by other methods, which is not limited by the present application.
The ACC sensor 282 may be used to detect the magnitude of acceleration of the smart watch 200 in various directions (typically three axes, i.e., x-axis, y-axis, and z-axis). In some embodiments, the ACC sensor 282 may detect the number of steps, stride frequency, and stride in different exercise modes, such as walking, running, or cycling, to provide real-time exercise data to the user. In other embodiments, the ACC sensor 282 may also detect a user's falling asleep, i.e., whether the user is asleep or awake.
The ambient light sensor 283 is used for detecting the illumination condition of the environment in which the smart watch is located.
It should be understood that the illustrated smart watch 200 is merely one example of a wearable device, and that the smart watch 200 may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration of components. The various components shown in fig. 4 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.
Based on the above-described electronic device, the embodiment of the application provides a method for detecting a sleep state. The method can be applied to the health detection (such as sleep detection) scene of the wearable equipment, namely, the wearable equipment in the method is the electronic equipment with the health detection function. The method according to the embodiment of the present application will be described below by taking a wearable device as an example of a smart watch. Specifically, as shown in fig. 5, the method for detecting a sleep state may include S601 to S604.
S601, the intelligent watch collects PPG data through a PPG sensor and collects ACC data through an ACC sensor.
Specifically, in the case where the wearing operation of the wearer with respect to the smart watch is detected, the smart watch may collect PPG data through the PPG sensor and may collect ACC data through the ACC sensor. It can be appreciated that the smart watch can detect the sleep state of the wearer only when the smart watch is worn on a designated part (such as the wrist) of the wearer, so as to determine whether the current sleep quality of the wearer is good, that is, whether sleep intervention is required for the wearer. If the smart watch is not worn at a designated location on the wearer's body, the smart watch does not need to perform data collection to avoid unnecessary power loss.
Generally, the smart watch may emit an optical signal into skin tissue of a wearer (or may be understood as blood, blood vessels, etc. within the skin tissue) through a PPG sensor, such as a light-emitting diode (LED) in the PPG sensor. A PPG sensor, such as a Photodiode (PD) in a PPG sensor, may receive a light signal reflected back through skin tissue. The optical signal includes pulsation information of the blood vessel of the wearer, which is used as PPG data. It should be appreciated that after receiving the reflected optical signal, the PD may convert the optical signal into an electrical signal, and convert the electrical signal into a digital signal that may be utilized by the smart watch through analog-to-digital conversion. The PPG data described above may actually refer to the digital signal.
In some embodiments, if the smart watch is worn at a designated location on the wearer's body, the smart watch may collect PPG data via the PPG sensor in real time, and ACC data via the ACC sensor. Therefore, the sleeping condition of the wearer can be detected in real time, the intelligent watch can intervene in sleeping of the wearer according to the sleeping condition, so that the sleeping quality of the wearer is improved, and the wearer can fall asleep better. In addition, because PPG data can also be used for detecting the physiological condition (such as blood oxygen, blood pressure, etc.) of the wearer, therefore, if the PPG sensor can gather PPG data in real time, the intelligent watch can detect the physiological condition of the wearer in real time according to the PPG data, so that the user can conveniently check the physiological condition of the user in real time, and the user can be reminded of taking a rest in time under the condition that the physiological condition of the wearer is in an abnormal state, and the wearing experience of the user is further improved. Moreover, because ACC data can also be used for detecting the motion condition (such as step number) of a wearer, if an ACC sensor can acquire ACC data in real time, the intelligent watch can detect the motion condition of the wearer according to the ACC data in real time, so that the user can know the motion condition of the user conveniently, a reasonable regulation and control motion plan is realized, and the wearer can be ensured to perform healthy motion.
In other embodiments, to reduce unnecessary waste of resources, the smart watch may collect PPG data by the PPG sensor during a first preset period and collect ACC data by the ACC sensor during a second preset period. The first preset period and the second preset period may be the same or different, and are not particularly limited. It can be appreciated that, because PPG data and ACC data in the embodiment of the present application are used to detect the sleep state of the wearer, and the wearer generally sleeps in a fixed period, the smart watch may collect PPG data and ACC data only in the fixed period, so that the utilization rate of the collection resources may be improved.
Generally, the ACC data is used to detect whether the wearer is asleep, that is, the smart watch only needs to determine whether the wearer has been asleep according to the ACC data, and the PPG data is used to detect which stage of the sleep stages (that is, the first sleep state) the wearer is asleep, that is, after the wearer has been asleep, the smart watch detects the sleep stages of the wearer through the PPG data. Wherein the first sleep state includes at least one of a fast eye movement state, a deep sleep state, and a shallow sleep state. Thus, the smart watch may collect PPG data by the PPG sensor in case ACC data is collected by the ACC sensor and the motion sequence derived from the ACC data characterizes that the wearer is not in motion, i.e. the wearer may be asleep. That is, the smart watch may not collect PPG data first, that is, collect only ACC data. Moreover, in the case that the wearer is in a sleeping state, the smart watch may not collect ACC data, but only PPG data. Therefore, the accuracy of sleep state detection can be ensured, and the utilization rate of acquisition resources is improved, so that the waste of storage resources is reduced.
In one implementation manner, the first preset period and the second preset period may be preset according to actual situations. In an example, taking the first preset period and the second preset period as the same, if most of the wearers sleep between 8 pm and 8 am, the smart watch may set the period between 8 pm and 8 am as the first preset period and the second preset period. In another example, taking the case that the first preset period and the second preset period are different, if most of the wearers fall asleep from 8 pm to 11 pm, the ACC data corresponding to the second preset period is only used to detect whether the wearers are in a sleep state, so the smart watch may set the period between 8 pm and 11 pm as the second preset period. If most of the wearers sleep from 10 pm to 8 am on the next day, and the PPG data corresponding to the first preset period is only used to detect which sleep stage of the sleep stages the wearers are in, the smart watch may set the period of time between 10 pm and 8 am on the next day as the first preset period.
In another implementation manner, the first preset period and the second preset period may be set according to a historical sleep condition of a wearer to whom the smart watch belongs. In an example, taking the case that the first preset time period and the second preset time period are the same, if the wearer to which the smart watch belongs usually sleeps between 11 pm and 7 am, the smart watch may set the time period between 11 pm and 7 am as the first preset time period and the second preset time period. In another example, taking the case that the first preset time and the second preset time period are different, if the wearer to which the smart watch belongs normally falls asleep from 11 pm to 1 am, and the ACC data corresponding to the second preset time period is only used to detect whether the wearer is asleep, the smart watch may set the time period from 11 pm to 1 am, as the second preset time period. If the wearer to which the smart watch belongs is sleeping at 12 pm to 8 am, and the PPG data corresponding to the first preset period is only used to detect which sleep stage of the sleep stages the wearer is in, the smart watch can set the period of time between 12 pm and 8 am as the first preset period.
It can be understood that when the wearer is in a sleep state or is about to fall asleep, the electronic device (such as a mobile phone) is not triggered to perform a corresponding operation, that is, the electronic device does not receive any touch operation (such as clicking operation) of the wearer, and the electronic device belonging to the wearer establishes communication connection with the smart watch. If the electronic device does not receive the touch operation of the wearer for a long time, the screen is automatically turned off, that is, the electronic device is converted from the bright screen state to the off screen state, or the electronic device is converted from the bright screen state to the off screen state under the condition that the touch operation of the wearer for the shutdown key is received, so that the electronic device can send a data acquisition instruction to the intelligent watch under the condition that the electronic device is in the off screen state and the off screen duration reaches the preset duration. Wherein the data acquisition indication is used for representing that the wearer is resting, to indicate that the smart watch triggers the PPG sensor to acquire PPG data, and triggers the ACC sensor to acquire ACC data. Then, in the case where the smart watch receives the data collection instruction, the smart watch may collect PPG data through the PPG sensor and ACC data through the ACC sensor in response to the data collection instruction. Therefore, the waste of subsequent computing resources caused by invalid acquisition of the sensor can be reduced, and the utilization rate of the computing resources is improved.
Further, considering that the wearer may not use the mobile phone for a long time due to daytime operation, the electronic device may determine whether the current time (or referred to as the first time) is within a preset sleep period (e.g. 22:00-9:00 am) or not, and if the electronic device is in the off-screen state and the off-screen duration reaches the preset duration only if the current time is within the preset sleep period, the electronic device may send the data acquisition instruction to the smart watch. Therefore, the intelligent watch can be accurately detected, the waste of subsequent computing resources caused by invalid acquisition of the sensor is reduced, and the utilization rate of the computing resources is improved.
The preset sleep period may be preset in actual situations (such as general sleep situations of most wearers), or may be set by historical sleep situations of the wearer to which the smart watch belongs, which is not specifically limited.
In some embodiments, considering that the tightness of wearing the smart watch by the wearer may affect the accuracy of the data collected by the sensor, that is, if the smart watch is worn too loosely or the smart watch is worn too tightly, the PPG data and the ACC data may be collected inaccurately, which may further affect the detection accuracy of the subsequent sleep state. Therefore, the smart watch may collect a pressure value between the smart watch and the wrist of the wearer through the pressure sensor, wherein the pressure value is used to characterize a degree of fit between the dial of the smart watch and the wrist of the wearer. Then, the acquisition of ACC data and PPG data is started only when the pressure value is within a preset pressure interval. Therefore, accurate collection of data can be realized, the occurrence of inaccurate data collection caused by too loose or too tight wearing of the intelligent watch is reduced, and the detection accuracy of the sleep state is further improved.
It can be appreciated that when the pressure value is within the preset pressure interval, the wearing degree of the intelligent watch is proper, so that the intelligent watch can directly collect the PPG data through the PPG sensor and collect the ACC data through the ACC sensor, that is, the prompt information about the wearing degree of the prompt is not required to be output. When the pressure value is not in the preset pressure interval, the condition that the intelligent watch is too loose or too tight is indicated. Therefore, in order to improve the accuracy of data acquisition and further improve the accuracy of sleep state detection, the smart watch can output prompt information when the current time is within a preset sleep period and the pressure value is not within a preset pressure interval. The prompting information is used for prompting a wearer to adjust the wearing tightness degree of the intelligent watch.
The preset pressure interval is preset according to actual conditions. Specifically, when the pressure value is greater than the first preset pressure and less than the second preset pressure, the smart watch may determine that the pressure value is located within the preset pressure interval. Wherein the first preset pressure is less than the second preset pressure. That is, the preset pressure interval may be determined based on the first preset pressure and the second preset pressure.
In an example, under the condition that the pressure value is smaller than or equal to the first preset pressure, the intelligent watch is worn too loosely, so that in order to improve accuracy of data acquisition, the intelligent watch can output first prompt information to prompt a user that the current wearing degree is loose. For example, the smart watch may output the first prompt information by voice (e.g., the smart watch outputs the prompt information that the wearing degree is loose and the wearing is tight) and/or the smart watch may display the first prompt information (e.g., the smart watch displays the prompt information that the wearing degree is loose and the wearing is tight). So, can reduce to wear the condition emergence that the pine influences data acquisition because of intelligent wrist-watch, promote data acquisition's degree of accuracy, and then promote sleep state's detection precision.
In an example, when the pressure value is greater than or equal to the second preset pressure, it is indicated that the smart watch is worn too tightly, so, in order to improve accuracy of data acquisition, the smart watch may output second prompt information to prompt the user that the current wearing degree is tight. For example, the smart watch outputting the second prompt information may be outputting the second prompt information by voice (e.g., the smart watch outputs the prompt information that is worn more tightly and more loosely), and/or the smart watch displaying the second prompt information (e.g., the smart watch displays the prompt information that is worn more tightly and more loosely). So, can reduce the condition emergence that influences data acquisition because of the intelligent wrist-watch wears too tightly, promote data acquisition's degree of accuracy, and then promote sleep state's detection precision.
It should be noted that, the personal information used in the technical solution of the present application is limited to information that is only agreed by the individual, including but not limited to notifying and reminding the user to read the related user protocol (notification) and signing the protocol (authorization) including the information of the authorized related user before the user uses the function.
Specifically, the PPG data and the ACC sensor can be used to detect the current sleep state of the wearer, so that the smart watch intervenes in sleep of the wearer according to the sleep state, and improves the sleep quality of the wearer. Therefore, in order to protect the privacy of the use of the user, it is necessary to determine whether the wearer has turned on the sleep detection function before the PPG sensor collects PPG data and the ACC sensor collects ACC data. In the case where the sleep detection function is turned on, that is, in the case where the smart watch receives an on operation of the wearer for the sleep detection function, the smart watch may collect PPG data through the PPG sensor and collect ACC data through the ACC sensor, thereby determining a sleep state of the wearer.
It should be noted that, the collection sequence of the PPG data and the ACC data collected by the smart watch is not limited. For example, the smart watch may collect PPG data by the PPG sensor and ACC data by the ACC sensor at the same time. For another example, the smart watch may collect PPG data through the PPG sensor first, and then collect ACC data through the ACC sensor. For another example, the smart watch may collect ACC data through an ACC sensor and then collect PPG data through a PPG sensor.
S602, the intelligent watch respectively processes the PPG data and the ACC data to obtain a target detection sequence, wherein the target detection sequence comprises an RRI sequence and a motion sequence.
Specifically, after the PPG data and the ACC data are collected, the smart watch may process the PPG data and the ACC data respectively to obtain a target detection sequence. The target detection sequence is used for detecting the current sleeping state of the wearer. The target detection sequence may include an RRI sequence and a motion sequence. The RRI sequence is used to characterize the interval of each heart beat, i.e., the interval of each beat. The movement sequence is used to characterize whether the wearer is in motion.
In one implementation, for the PPG data described above, the smart watch may perform peak detection on the PPG data to obtain an RRI sequence. In an example, the smart watch may perform peak detection on the PPG data by looking for local maxima (or peaks) and minima (or troughs) in the PPG data. In another example, the smart watch may also peak detect PPG data by having a first derivative (e.g., li derivative) of an adaptive threshold.
In some embodiments, the smart watch may perform peak detection on the PPG data to obtain a peak point of at least one waveform in the PPG data. Thereafter, the smart watch may calculate a distance between peak points of adjacent waveforms and determine the distance as one RR interval. Then, the smart watch may determine the RRI sequence according to RR intervals corresponding to peak points of all waveforms in the PPG data. That is, the RRI sequence is obtained based on RR intervals corresponding to peak points of all waveforms in the PPG data.
It should be noted that the waveform of the PPG signal may reflect the physiological characteristics of the wearer. Illustratively, as shown in fig. 6, the PPG signal includes 4 waveforms, each representing a complete pulse waveform period. Wherein, the point A1 in the waveform 1, the point A2 in the waveform 2, the point A3 in the waveform 3 and the point A4 in the waveform 4 are peak points, namely R waves. In some embodiments, the wearable device may determine the RR interval, i.e., RR1, by calculating the distance between point A1 and point A2, and RR2 by calculating the distance between point A2 and point A3, and RR3 by calculating the distance between point A3 and point A4.
In another implementation manner, for the ACC data, the smart watch may perform feature extraction on the ACC data to obtain a motion sequence.
In some embodiments, the smart watch may calculate the ACC data according to a preset calculation interval to obtain a motion value. It will be appreciated that the above described motion sequence may comprise at least one motion value, that is, the motion sequence is composed based on at least one motion value. The smart watch may then determine whether the wearer is in motion based on the motion value. In one example, the motion value may be calculated by way of energy. In another example, the motion value may also be calculated by means of a variance.
It should be noted that the preset calculation interval may be set according to actual requirements. In this embodiment, in order to facilitate the subsequent sleep state detection, the preset calculation interval may be set to 1s, that is, the smart watch calculates a motion value every 1 s. In other embodiments, the preset calculation interval may be set to 5s, 10s, or the like, which is not particularly limited.
In one implementation, the smart watch may determine whether the motion value is greater than a preset value. The smart watch may determine that the wearer is in a motion state when the motion value is greater than a preset value, or may determine that the wearer is in a stationary state, i.e., the wearer is not in a motion state, when the motion value is less than or equal to the preset value.
Illustratively, taking the above-mentioned motion value as an example, the motion value is calculated based on energy, the motion value may be calculated by the following formula:
Wherein E () is energy, that is, a motion value, i is the preset interval (for example, 1 s), t is the sampling rate (for example, 100 Hz) of ACC data, x is the acceleration value of the smart watch on the x axis, y is the acceleration value of the smart watch on the y axis, and z is the acceleration value of the smart watch on the z axis.
Specifically, after the smart watch obtains the motion value through the first formula, the smart watch can determine whether the motion value is greater than a preset value (i.e., a preset energy value), and if the motion value is greater than the preset energy value, the smart watch can determine that the wearer is in a motion state. If the movement value is less than or equal to the preset energy value, the smart watch can determine that the wearer is not in a movement state.
For example, in the case that the motion value is obtained by a variance-based calculation, the smart watch may determine whether the motion value is greater than a preset value (i.e., a preset variance value), and if the motion value is greater than the preset variance value, the smart watch may determine that the wearer is in a motion state. If the movement value is less than or equal to the preset variance value, the smart watch may determine that the wearer is not in a movement state.
S603, the intelligent watch carries out interpolation and normalization processing on the RRI sequence to obtain a target RRI sequence.
Specifically, after the RRI sequence is obtained, the smart watch may perform interpolation processing on the RRI sequence to obtain an interpolated RRI sequence.
It should be noted that, since the RRI sequence is used to characterize the interval time of each beat, the interval time of the beats of the same wearer at different sampling times is also different. The motion sequence is calculated according to the ACC data according to a preset calculation interval, that is, the interval time corresponding to the motion sequence is the same, that is, the sampling rate of the motion sequence is fixed. Therefore, in order to ensure the accuracy of the subsequent sleep state detection, the smart watch may interpolate the RRI sequence to obtain an interpolated RRI sequence. Therefore, the sampling rate of the target RRI sequence is consistent with the sampling rate of the motion sequence, and a basis is provided for the subsequent accurate detection of the sleep state.
The manner in which the intelligent watch performs interpolation processing on the RRI sequence may be polynomial interpolation, spline interpolation, or the like, which is not limited in detail. Wherein the polynomial interpolation (polynomial interopolation) is to make a proper specific function (or called polynomial) using the function value of the known points of the function f (x) in a certain section, and to use the value of the specific function as an approximation of the function f (x) at other points of the section. The spline interpolation is a method of modeling raw data in the form of a spline function. The spline function (spline function) is a type of segment (slice) smoothing function that also has some smoothness at the intersections of segments.
In one implementation, after obtaining the interpolated RRI sequence, the smart watch may normalize the interpolated RRI sequence to obtain a target RRI sequence. Wherein the sampling rate of the target RRI sequence is the same as the sampling rate of the motion sequence described above. It can be understood that by performing normalization processing on the interpolated RRI sequence, the interpolation result, that is, the interpolated RRI sequence, can be improved, so that the target RRI sequence and the motion sequence can be spliced into a time sequence with the same length (e.g., 1 s), that is, the sampling time corresponding to the target RRI sequence and the motion sequence is the same, and all the sampling time is the preset calculation interval. Therefore, a foundation can be provided for subsequently improving the accuracy of sleep state detection.
For example, as shown in fig. 7, after the RRI sequence is subjected to interpolation and normalization, the sampling time of the obtained target RRI sequence corresponding to the motion sequence is the same. Taking a sample time of 1 second as an example, the first column of cells in the schematic diagram includes the data corresponding to the target RRI sequence and the motion sequence in the first second. The second column of cells includes data corresponding to the target RRI sequence and the motion sequence within a second. The third column of cells includes data corresponding to the target RRI sequence and the motion sequence within a third second. The fourth column of cells includes data corresponding to the target RRI sequence and the motion sequence within a fourth second. The fifth column of cells includes data corresponding to the target RRI sequence and the motion sequence within the fifth second. The sixth column of cells includes data corresponding to the target RRI sequence and the motion sequence within a sixth second. The seventh column of cells includes data corresponding to the target RRI sequence and the motion sequence within the seventh second.
In some embodiments, the smart watch may not perform the step of S603, that is, the smart watch may directly detect the sleep state of the wearer according to the RRI sequence and the motion sequence, without performing interpolation and normalization processing on the RRI sequence. Thus, the detection efficiency of the sleep state can be improved.
In other embodiments, the smart watch may perform interpolation processing on the RRI sequence only, and not perform normalization processing on the interpolated RRI sequence. Then, the intelligent watch can detect the sleep state of the wearer directly according to the interpolated RRI sequence and the motion sequence. Thus, the detection efficiency of the sleep state can be improved.
S604, the intelligent watch inputs the target RRI sequence and the motion sequence into a sleep state detection model to obtain the sleep state of the wearer.
Specifically, after the target RRI sequence is obtained, the smart watch may predict the sleep state of the wearer according to the target RRI sequence and the motion sequence. The sleep state is used for representing the current sleeping condition of the wearer. The sleep state may include a fast eye movement state, a deep sleep state, a shallow sleep state, or an awake state.
In one implementation, the smart watch may input the target RRI sequence and the motion sequence into a sleep state detection model to obtain a sleep state of the wearer. Wherein the sleep state detection model has the ability to predict sleep states based on the target RRI sequence and the motion sequence.
In some embodiments, after obtaining the target RRI sequence, the smart watch may first vector-combine the target RRI sequence and the motion sequence to obtain a combined sequence table. Then, the intelligent watch can input the combined sequence table into a sleep state detection model to obtain the sleep state of the wearer. It can be understood that, by performing interpolation and normalization processing on the RRI sequence, the sampling rates of the target RRI sequence and the motion sequence are the same, so that the target RRI sequence and the motion sequence are vector-combined, so that time alignment between different sequences can be realized, that is, different sequences under the same acquisition time are aligned, and the occurrence of a detection result that the sleep state is affected due to the fact that time points between the sequences have differences (for example, the target RRI sequence of the first second and the motion sequence of the second are used as inputs at the same time) is reduced, thereby providing a basis for the subsequent accurate detection of the sleep state.
Wherein, the sleep state detection model is a classification model. In this embodiment, the classification model may be a convolutional neural network (convolutional neural networks, CNN). Specifically, the classification process of the convolutional neural network includes the steps of obtaining the score of each class of the current sample through a full-connection layer after passing through a convolutional layer and a pooling layer, and classifying through a normalized exponential function (softmax) according to the scores of different classes to obtain the class of the current sample. Among them, softmax is an activation function for multi-classification problems. In other embodiments, the classification model may be a decision tree, a support vector machine (support vector machine, SVM), or the like, and is not particularly limited as long as the classification model can be used as a sleep state detection model.
In one implementation, the sleep state detection model may include at least one of an N-layer convolutional neural network, a long short-term memory (LSTM), a full-connectivity layer, and a softmax layer. In this embodiment, N is 3. In other embodiments, N may also be other values, for example, N may be 2,5, etc., and is not specifically limited.
The CNN layer (or referred to as a convolution layer) is configured to perform a feature extraction operation on the target RRI sequence and the motion sequence. The LSTM layer is used for processing data having time-series properties, i.e. for processing the target RRI sequence as well as the motion sequence. The full connection layer is used for connecting the features extracted by the convolution layer and outputting a final classification result, that is, the full connection layer is used for combining and integrating the features. Softmax is used to normalize the output sum to become a probability distribution of the predicted class.
Specifically, the smart watch may input the target RRI sequence and the motion sequence into the three-layer convolution layer to obtain a first output value (or referred to as a first output feature). The smart watch may then input the first output value into the LSTM layer, resulting in a second output value (or referred to as a second output characteristic). Then, the smart watch can input the second output value into the full connection layer and the softmax to obtain a four-classification result, namely, the sleep state of the wearer.
It will be appreciated that each convolution layer includes a plurality of convolution kernels that slide over the input data (i.e., the target RRI sequence and the motion sequence) in a preset sliding step size (otherwise known as a preset step size). The preset sliding step length refers to a length (or referred to as time) corresponding to each sliding of the convolution kernel in the convolution operation, that is, a length corresponding to one sliding of the convolution kernel. The preset sliding step length is preset according to actual requirements, for example. For example, the preset sliding step may be 30s, 15s, etc., which is not particularly limited.
In some embodiments, the smart watch may operate the sleep state detection model with the target RRI sequence and the motion sequence in the first period as inputs, and the sleep state detection model slides the convolution kernel in a preset step length, and outputs a continuous sleep state of the wearer in the first period. The first time period is a time period corresponding to the time period when the intelligent watch needs to perform sleep state detection, namely a time period when the wearer falls asleep or is about to fall asleep. In an example, the first period of time may be a period of time preset according to sleep habits of most wearers, for example, the first period of time may be 20:00 to 10:00 am the next day. In another example, the first period of time may also be a period of time preset according to sleep habits of a wearer to whom the smart watch belongs, for example, the first period of time may be 23:00 to 8:00 am the next day.
In this embodiment, by means of sliding the convolution kernel on the input data, the sleep state of the wearer in the period where the convolution kernel is located (that is, the preset convolution time described below) is detected, so that real-time detection of the sleep state can be achieved, the smart watch can intervene in time on the wearer according to the real-time detection result, the sleep condition of the wearer is improved, and the sleep quality of the wearer is further improved.
The size of the convolution kernel is determined based on a preset convolution time and the number of parameters corresponding to the input data (or called input value), that is, the width of the convolution kernel is the preset convolution time, and the height of the convolution kernel is the number of parameters of the input data. The predetermined convolution time is the length of time that the input data can be contained in one convolution kernel. It can be understood that, since the parameters corresponding to the input data in this embodiment are the target RRI sequence and the motion sequence, the number of parameters is 2. If the input data further includes data of other parameters, the number of the parameters may also be the sum of the numbers corresponding to the other parameters, which is not limited in particular.
For example, as shown in fig. 8, the input data of 15s for each cell, that is, 15s for one cell is taken as an example. Specifically, the convolution kernel a slides on the input data according to a preset sliding step length. It can be seen that the parameter corresponding to the first row of cells is the target RRI sequence and the parameter corresponding to the second row of cells is the motion sequence, so it can be explained that the sliding direction of the convolution kernel is sliding laterally and the convolution kernel is sliding from left to right. The width of the convolution kernel a is a preset convolution time (60 s), that is, each convolution kernel includes a target RRI sequence obtained by the smart watch of 60s and a motion sequence. The height of the convolution kernel is the number of parameters (2) of the input data. It will be appreciated that the input data includes data corresponding to the target RRI sequence and data corresponding to the motion sequence, and thus the number of parameters may be 2. And, the preset sliding step length of the convolution kernel is 30s, that is, the smart watch can output the sleep state of the wearer once every 30 s.
In one implementation, the three CNNs included in the sleep state detection model may be named CNN layer one, CNN layer two, and CNN layer three, respectively. The CNN layer one may include at least one of a first preset number of convolution kernels, a maximum pooling, and an activation function. The first preset number is a preset number according to actual conditions. In this embodiment, the first preset number is 64. In other embodiments, the first preset number may also be other values, for example, may be 32, etc., which is not limited specifically. The maximum pooling (max pooling) is one of the common pooling operations in the convolutional neural network, and the spatial dimension of the feature vector can be reduced through the maximum pooling operation, so that the parameter number and the calculation complexity of the model are reduced, and the robustness of the model is enhanced. Illustratively, the activation function may be a linear rectification function (RECTIFIED LINEAR unit, reLU), a sigmoid function, a hyperbolic tangent function (hyperbolic tangent function, tanh), or the like, which is not particularly limited.
Wherein the CNN layer two includes at least one of a first preset number of convolution kernels, a maximum pooling, and an activation function. The CNN layer three includes at least one of a second predetermined number of convolution kernels, a maximum pooling, and an activation function. The second preset number is twice the first preset number, that is, the number of convolution kernels in the CNN layer three may be 128, 64, etc., which is not limited specifically. It can be understood that, since the third CNN layer includes the second preset number of convolution kernels, the feature vector corresponding to the first output value input to the LSTM layer is the second preset number 1. In this embodiment, the feature vector is 128×1.
For example, as shown in fig. 9, after obtaining the target RRI sequence and the motion sequence, the smart watch may sequentially input the target RRI sequence and the motion sequence as input data into the CNN layer one, the CNN layer two, and the CNN layer three, and input an output result (or referred to as the first output feature) into the LSTM layer. Wherein the CNN layer one includes 64 convolution kernels, max pooling, and activation functions. CNN layer two includes 64 convolution kernels. CNN layer three includes 128 convolution kernels.
The feature vector size of the output features of the LSTM layer is set to a third preset number, and the time length corresponding to the first output feature passing through the LSTM layer should not be less than the time length of inputting data all night, that is, the sequence length passing through the LSTM layer is the maximum length of the CNN layer after the convolution operation.
In one implementation, the length of time for inputting data all night may be preset according to the actual situation. For example, if the wearer would not normally sleep for more than 15 hours, the smart watch may set the length of time for which data is entered overnight to 15 hours. Further, taking the above input data corresponding to 15s for each cell as an example, it is understood that 15 hours is equivalent to 54000s, and thus the maximum length is 3600 cells.
In another implementation, the length of time for inputting data all night may also be preset according to the historical sleep condition of the wearer to whom the smart watch belongs. For example, if the sleep time of the wearer to which the smart watch belongs does not normally exceed 10 hours, the smart watch may set the time period for inputting data all night to 10 hours. Further, taking the above-described input data corresponding to 15s for each cell as an example, it can be understood that 10 hours is equivalent to 36000s, and thus the maximum length is 2400 cells.
In some embodiments, the training process of the sleep state detection model may specifically include that the smart watch acquires an input data set. The input data set comprises a plurality of pieces of input data and each piece of input data carries a real state label, and the real state label is used for representing a sleep state corresponding to the input data. For example, if the real state tag is in an awake state, the input data corresponding to the real state tag is data obtained by the smart watch when the wearer is in the awake state. The input data includes an RRI sequence and a motion sequence, and the RRI sequence is the same as the motion sequence in sampling rate. Then, for each piece of input data, the smart watch can input the input data into a sleep state detection model to be trained to obtain a prediction state label. And then, the intelligent watch can carry out parameter adjustment on the sleep state detection model to be trained according to the prediction state label and the real state label carried by the input data to obtain a trained sleep state detection model.
For example, as shown in fig. 10, the sleep state detection process may specifically include that the smart watch may obtain PPG data through a PPG sensor, and perform peak detection on the PPG data to obtain an RRI sequence. And the intelligent watch can acquire ACC data through the ACC sensor, and perform feature extraction on the ACC data to obtain a motion sequence. Then, the intelligent watch can conduct interpolation and normalization processing on the RRI sequence to obtain a target RRI sequence. Then, the smart watch can perform vector combination on the target RRI sequence and the motion sequence, and sequentially input the combined sequence table into three convolution layers, an LSTM layer and a softmax to obtain the sleeping state of the wearer. The sleep state may be a fast eye movement state, a deep sleep state, a shallow sleep state, and an awake state.
In one implementation, the input data may also include skin resistance data (or referred to as skin resistance signals). Wherein the skin resistance data is used to reflect sweat gland secretion of the wearer. The skin resistance signal may be acquired by a galvanic skin (GALVANIC SKIN response, GSR) sensor. The GSR sensor is used to measure changes in skin conductance to provide information on the physiological and emotional states of the wearer. It will be appreciated that if the wearer's mood swings more (e.g., sudden tension) the more sweat glands are secreted by the wearer, the more conductive the wearer's body will be, which in turn will result in a reduced skin resistance.
Specifically, the smart watch may collect skin resistance data through the GSR sensor described above. And then, the intelligent watch can perform feature extraction on the skin resistance data to obtain a skin resistance sequence. Wherein the sampling rate of the skin resistance sequence is the same as the sampling rate of the motion sequence described above. Then, the smart watch may input the skin resistance sequence, the target RRI sequence, and the motion sequence into the sleep state detection model to obtain a sleep state of the wearer. So, can follow the sleep state of a plurality of angle detection wearers for the intelligent wrist-watch can be more accurate detect the sleep state of wearers, and then in time intervene to the wearers according to the sleep state for follow-up in time to provide the basis, promoted the user experience of wearers.
Illustratively, as shown in fig. 11, the sampling times corresponding to the skin resistance sequence, the target RRI sequence, and the motion sequence are the same. Taking a sample time of 1 second as an example, the first column of cells in the schematic diagram includes data corresponding to the target RRI sequence, the motion sequence, and the skin resistance sequence in the first second. The second column of cells includes data corresponding to the target RRI sequence, the motion sequence, and the skin resistance sequence for a second. The third column of cells includes data corresponding to the target RRI sequence, the motion sequence, and the skin resistance sequence for a third second. The fourth column of cells includes data corresponding to the target RRI sequence, the motion sequence, and the skin resistance sequence for a fourth second. The fifth column of cells includes data corresponding to the target RRI sequence, the motion sequence, and the skin resistance sequence for a fifth second. The sixth column of cells includes data corresponding to the target RRI sequence, the motion sequence, and the skin resistance sequence for a sixth second. The seventh column of cells includes data corresponding to the target RRI sequence, the motion sequence, and the skin resistance sequence for a seventh second.
In some embodiments, after obtaining the sleep state of the wearer, the smart watch may perform sleep intervention on the wearer according to the sleep state. Thus, the sleeping condition of the wearer can be improved, and the sleeping improvement service matched with the current sleeping state is provided, so that the sleeping quality of the wearer is improved.
In an example, the smart watch may intercept all notification information in the case where the sleep state of the wearer is in the deep sleep state. Therefore, the occurrence of the condition that the sleeping of the wearer is influenced due to the reminding of the notification message can be reduced, a good sleeping environment is provided for the wearer, and the use experience of the wearer is improved.
In another example, in the case that the sleep state of the wearer is in the fast eye movement state, the smart watch may play sleep aiding contents such as sleep aiding music to help the wearer enter the deep sleep state fast, so as to improve the use experience of the wearer.
In other embodiments, after obtaining the sleep state of the wearer, the smart watch may send the sleep state to the target electronic device (e.g., a mobile phone) in real time. The target electronic device is a device which establishes communication connection with the intelligent watch. The communication connection comprises a wired communication connection or a wireless communication connection. The process by which the target electronic device 100 communicates with the smart watch 200 will be described below in connection with the communication system shown in fig. 12.
As shown in fig. 12, the communication system may include a target electronic device 100 and a smart watch 200. By way of example, the target electronic device may be a cell phone, tablet, notebook, ultra-mobile personal computer, UMPC, netbook, personal digital assistant (personaldigital assistant, PDA), or the like. The application is not limited in any way to the particular type of electronic device 100.
In one possible implementation, the target electronic device 100 may establish a wireless communication connection with the smart watch 200, and the target electronic device 100 and the smart watch 200 may communicate data information with each other over the wireless communication connection. For example, the smart watch 200 may transmit data information such as sleep data of a wearer to the target electronic device 100 through a wireless communication connection, and the target electronic device 100 may transmit data information such as a display screen state and/or a device state of the target electronic device 100 to the smart watch 200 through a wireless communication connection. The sleep data may refer to the time taken by each sleep state such as a light sleep state, a deep sleep state, and a fast eye movement state, and the duty ratio of each sleep state in the sleep period, etc., throughout the sleep period of the user. The data information that the target electronic device 100 and the smart watch 200 send to each other will be described in detail in the following embodiments, and will not be described in detail here. In particular, the wireless communication connection may be one or more of bluetooth, wireless fidelity direct connection (WIRELESS FIDELITY DIRECT, WIFI DIRECT), or wireless fidelity software access point (WIRELESS FIDELITY software access point, WIFI softAP), etc.
In another possible implementation, the target electronic device 100 may also establish a wired communication connection with the smart watch 200 for data interaction. For example, the target electronic device 100 and the smart watch 200 may establish a wired connection through a universal serial bus (universal serial bus, USB) and transmit data information to each other based on the wired communication connection.
Then, in the case that the target electronic device receives the sleep state of the wearer, the target electronic device may perform sleep intervention on the wearer according to the sleep state. That is, the process of performing sleep intervention on the wearer may be performed by the smart watch, or may be performed by the target electronic device that establishes a communication connection with the smart watch, which is not limited in particular.
In still other embodiments, the smart watch may send the time taken by each sleep state throughout the sleep period to the target electronic device upon detecting a shift in the sleep state of the wearer from the first sleep state to the awake state. Therefore, the target equipment can analyze the sleep quality of the wearer according to the occupied time of each sleep state, provide a sleep analysis report for the wearer, facilitate the wearer to clearly know the sleep condition of the wearer, and promote the use experience of the wearer.
In some embodiments, the application provides a computer readable storage medium comprising computer instructions which, when run on an electronic device, cause the electronic device to perform the method as described above.
In some embodiments, the application provides a computer program product which, when run on an electronic device, causes the electronic device to perform the method as described above.
It will be apparent to those skilled in the art from this description that, for convenience and brevity of description, only the above-described division of the functional modules is illustrated, and in practical application, the above-described functional allocation may be performed by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional modules to perform all or part of the functions described above.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical functional division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another apparatus, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and the parts displayed as units may be one physical unit or a plurality of physical units, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a readable storage medium. Based on such understanding, the technical solution of the embodiments of the present application may be essentially or a part contributing to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including several instructions for causing a device (may be a single-chip microcomputer, a chip or the like) or a processor (processor) to perform all or part of the steps of the method described in the embodiments of the present application. The storage medium includes a U disk, a removable hard disk, a Read Only Memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.
The foregoing is merely illustrative of specific embodiments of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.