A method improves a picked-up signal in a hearing system. The hearing system has at least one hearing device, particularly a hearing aid. The hearing aid device has an associated first directional microphone that has an adjustable first directional characteristic with a preferential direction. The first directional microphone converts sound into a first signal that is adopted in the picked-up signal. A speech activity of a user of the hearing system is monitored, and recognition of a speech activity of the user prompts the preferential direction of the first directional characteristic to be adjusted in comparison with a frontal direction of the user such that the sound sensitivity of the first directional microphone undergoes attenuation in the frontal direction.
|
1. A method for improving a picked-up signal in a hearing system having at least one hearing device, the at least one hearing aid device containing an associated first directional microphone having an adjustable first directional characteristic with a preferential direction, which comprises the steps of:
converting, via the first directional microphone, sound into a first signal that is adopted in the picked-up signal;
monitoring speech activity of a user of the hearing system; and
prompting the preferential direction of the first directional characteristic to be adjusted in comparison with a frontal direction of the user such that sound sensitivity of the first directional microphone undergoes attenuation in the frontal direction in dependence on recognition of the speech activity of the user.
2. The method according to
3. The method according to
4. The method according to
5. The method according to
converting, via the second directional microphone, the sound into a second signal that is adopted in the picked-up signal; and
prompting a preferential direction of a second directional characteristic to be adjusted in comparison with the frontal direction of the user such that the second directional characteristic undergoes attenuation in the frontal direction upon recognition of the speech activity of the user.
6. The method according to
7. The method according to
8. The method according to
9. The method according to
10. The method according to
11. The method according to
12. A binaural hearing system, comprising:
two hearing devices each having an associated directional microphone being configured to perform a method according to
13. The binaural hearing system according to
|
|||||||||||||||||||||||||||||
This application is a continuation of patent application U.S. Ser. No. 15/179,297, filed Jun. 10, 2016; this application also claims the priority, under 35 U.S.C. § 119, of German patent application No. DE 10 2015 210 652.7, filed Jun. 10, 2015; the prior applications are herewith incorporated by reference in their entireties.
The invention relates to a method for improving a picked-up signal in a hearing system having at least one hearing device, particularly a hearing aid. The at least one hearing device has an associated first directional microphone that has an adjustable first directional characteristic having a preferential direction, and the first directional microphone converts sound into a first signal that is adopted in the picked-up signal.
For a hearing aid, reception of one's own voice by the user plays a particularly important part, since subjective awareness of one's own voice, as perceived when wearing the hearing device, significantly influences acceptance of a hearing aid by the user. In this case, perception of one's own voice is firstly influenced by the signal path of the hearing aid, that is to say via pickup of the voice by the microphone system, appropriate signal processing of the microphone signal in a hearing aid and reproduction by the loudspeaker or generally the electro acoustic transducer of the hearing aid, with the sound produced being supplied directly to the ear of the user.
Secondly, a sound conductor provided for this supply—be it an ear mold of a “behind the ear” instrument (BTE) or be it the output of an “in the ear” or “completely in the canal” instrument (ITE or CIC)—usually causes extensive closure of the auditory canal, which can result in an occlusion effect for a speech activity of the user. This occurs primarily as a result of sound waves from one's own voice, which are routed by body-borne sound transmission primarily via the jaw bone to the auditory canal, and, owing to the closure of the auditory canal, cannot escape to the outside, which means that one's own voice when speaking is perceived as dull-sounding.
Whereas problems in the perception of one's own voice that are based on an occlusion effect are often countered by additional mechanical ventilation channels in the hearing aid or, more recently, also by special algorithms in the signal processing, the problems with the sound of the voice that are induced in the single path usually involve attempts to improve perception of the user's own voice by adaptive signal processing. By way of example, many hearing aids use automatic gain control (AGC), the gain of which can be lowered in the relevant frequency ranges for this purpose when recognizing the user's own speech activity. Similarly, a compression ratio, the threshold or the response times of compression (“compression ratio”, “threshold”, “attack/release”) can be adapted for recognizing speech activity. Such recognition of the user's own speech activity is possible quickly and efficiently in this case.
However, such measures in the signal processing influence the entire signal component, at least in the relevant frequency ranges, as appropriate, and hence alter the signal components in frequency ranges essential to realistic voice reproduction by lowering the gain or altering the compression parameters. The acoustic perception of the surroundings is also affected thereby, which can be perceived as unpleasant particularly in lively conversation situations with multiple interlocutors. This can thus impair auditory perception significantly.
Published, non-prosecuted German patent application DE 10 2005 032 274 A1, corresponding to U.S. Pat. No. 7,853,931, discloses a hearing apparatus and a corresponding method for detection of the user's own voice. In this regard, a first microphone for picking up ambient sound and a second microphone for picking up auditory canal sound are provided. From a comparison of the signals from a first microphone and the second microphone, it is inferred that the user of the hearing device is speaking and its own voice is present. When the presence of his own voice is established, a directional microphone, in particular, is deactivated. In this way, the directional microphone is operated in an interference-free manner, since the directional microphone would undesirably always orient itself to the frontal direction if the user's own voice were present.
Published, non-prosecuted German patent application DE 10 2011 087 984 A1 (corresponding to U.S. Pat. No. 8,873,779) proposes, for the purpose of recognizing the user's own voice, a hearing apparatus that contains at least two independent analysis devices, each of which is configured to take and receive audio signal as a basis for obtaining speech activity data that are dependent on the speaker activity of the wearer of the hearing apparatus. A fusion device is configured to receive the speech activity data from the analysis devices and to take the speech activity data as a basis for recognizing whether or not the wearer is currently speaking.
The invention is based on the object of specifying a method that, in a hearing system having at least one hearing device, is intended to improve the sound of a picked-up signal for the further processing, particularly in respect of perception by a user, as simply and efficiently as possible when there is voice activity.
The invention achieves the cited object by a method for improving a picked-up signal in a hearing system having at least one hearing device, particularly a hearing aid. The at least one hearing aid device has an associated first directional microphone that has an adjustable first directional characteristic having a preferential direction and the first directional microphone converts sound into a first signal that is adopted in the picked-up signal. A speech activity of a user of the hearing system is monitored, and recognition of a speech activity of the user prompts the preferential direction of the first directional characteristic to be adjusted in comparison with a frontal direction of the user such that the sound sensitivity of the first directional microphone undergoes attenuation in the frontal direction.
Embodiments that are advantageous and in some cases inventive in themselve are the subject matter of the sub claims and the description that follows.
In particular, the first directional microphone is formed by a plurality of, in particular, spaced single microphones, the first directional characteristic being formed by a superimposition of the directional characteristics of the single microphones. In this case, the single microphones may each have an essentially omnidirectional characteristic. In particular, the directional characteristic results from appropriate signal processing of the signals from the single microphones. In this case, the invention contains, in a binaural hearing system, particularly the preferred variant wherein one of the single microphones forming the first directional microphone is arranged in one, in that case the first, hearing device and wherein another of these single microphones is arranged in a further, second hearing device. In the case of a single hearing device, the first directional microphone is formed by arranging preferably two or more essentially omnidirectional single microphones together in the then one hearing device.
In this case, the picked-up signal is intended to be understood to mean that signal in which the first signal from the first directional microphone, into which a sound from the surroundings is converted with the spatial sensitivity prescribed by the selected directional characteristic, is adopted for further signal processing. In this case, the preferential direction of the first directional characteristic is intended to be understood to mean that spatial direction in which the first directional microphone has the highest sensitivity in this case, that is to say that in the case of a sound at constant level that hits the first directional microphone from different directions, arrival from the preferential direction prompts the highest signal level to be produced in the first signal.
Preferably, the first directional characteristic has axial or mirror symmetry, the preferential direction being situated on the access of symmetry or in the plane of symmetry. In this case, the frontal direction of the user is defined by the line of vision of its head, which runs parallel to the plane of symmetry of the head. Preferably, when the hearing device is worn properly by the user, the first directional characteristic can be set in relation to the frontal direction of the user. The first directional characteristic can have preferably a monotonous decrease in sound sensitivity in an angle range around the preferential direction, so that the attenuation in the sound sensitivity is achieved in the frontal direction for an adjustment by an increase in the angle between the preferential direction and the frontal direction.
In this context, the invention is first of all based on the consideration that any regulation or correction of the user's own voice in a picked-up signal, which is accomplished only by the AGC or by adaptation of the parameters for the compression, also has effects on the perception of the sound from the surroundings. Specifically in a hearing situation in which the user is conducting a lively conversation with one or else more interlocutors, and the interaction in the conversation means that the user's own voice activity can occasionally also coincide with a useful signal formed by the speech of one of the interlocutors, this would result in rapid changes in the volume and/or dynamics of the useful signal formed by the speech of the interlocutor, and also for the sound from the surroundings.
The invention makes use, then, of the empirically determinable circumstance that the diffraction of the sound that leaves the mouth during speech activity means that the sound of the user's own voice that the user perceives at one ear is essentially perceived as coming from the frontal direction. This means that, owing to the fact that the line of vision and hence the frontal direction of the user is oriented toward an interlocutor in most hearing situations formed by a conversation, the direction from which the useful signal provided by the speech activity of the interlocutor is perceived essentially coincides with the direction from which the user's own voice is perceived while speaking, provided that no further adaptation of the directions of perception is performed. The effect that can therefore be achieved by adjusting the preferential direction of the first directional characteristic in comparison with the frontal direction of the user is that a smaller proportion of the user's own voice is adopted in the picked-up signal. For agreeable auditory perception of the user's own voice, attenuation in relation to the signal processing of other useful signals of between 6 dB and 10 dB is usually sufficient in the signal path. If the first directional characteristic has a monotonous decrease in the sound sensitivity away from the preferential direction, then a moderate difference between the preferential direction of the first directional characteristic and the frontal direction is sufficient for attenuation of the picked-up speech signal from the user by between 6 dB and 10 dB.
This furthermore results in background noise, which cannot be explicitly associated with a clear source in special terms, not undergoing substantial attenuation owing to the cited measure, which means that the measures do not lead to undesirable fluctuations in the level of the background noise in the picked-up signal that is to be processed further. To a special degree, this also makes use of the circumstance that specifically background noise is perceived as less directional, in spatial terms, in low frequency bands, which make a considerable contribution to the basic perception of a hearing situation, owing to the smaller numbers of waves, and hence are even less affected by the adjustment of the preferential direction of the directional characteristic and hence the directional characteristic as such. In particular, this applies to the case in which adjusting the preferential direction of the directional characteristic involves the directional characteristic itself not being altered further, apart from a purely spatial rotation.
Favorably, the preferential direction of the first directional characteristic is adjusted at an angle of between 5° and 20°, preferably between 5° and 10°, in comparison with the preferential direction of the user. To alter the orientation of the directional characteristic, adjustment of the preferential direction in the cited angle range is usually sufficient in order to attain adequate attenuation of the signal component of the user's own speech in the picked-up signal. Owing to the only slight alteration of the preferential direction, however, the perception of background noise in a hearing situation, which background noise has a certain directional dependency, such as e.g. music from a locatable loudspeaker at a distance of several meters, is also not really influenced, which means that the adaptation becomes even less noticeable for a user.
Preferably, the picked-up signal is monitored for a hearing situation having a directional main sound source, wherein without recognition of a speech activity of the user, recognition of a hearing situation having a directional main sound source prompts the preferential direction of the first directional characteristic to be oriented to the frontal direction of the user. A directional main sound source is intended to be understood to mean a sound source that has considerable directional dependency in relation to the sound level of other signal components, and whose sound level lies distinctly above the level of other signal components, regardless of their spatial direction, in the direction of the maximum (for example a loudspeaker box or the like). The frontal direction of the user, which usually essentially corresponds to his line of vision, can be ascertained by further analysis of the picked-up signal in this case, or can be determined by a preset in the hearing device, subject to the orientation of the first directional microphone when the hearing device is worn properly.
If a directional main sound source is now recognized in the hearing situation without the user's own speech activity being recognized at the same time, then it is assumed that the sound from the main sound source is the useful signal for the user, which useful signal is therefore intended to find its way into the picked-up signal at particularly high resolution. In this case, the acoustic resolution—that is to say the signal-to-noise ratio—is achieved by appropriate adaptation of the directional characteristic through adjustment of the preferential direction toward the main sound source.
It is found to be advantageous in this case if the picked-up signal is monitored for a hearing situation corresponding to a conversation, and without recognition of a speech activity of the user, recognition of a hearing situation corresponding to a conversation prompts the preferential direction of the first directional characteristic to be oriented to the frontal direction of the user. A hearing situation corresponding to a conversation, possibly with background noise having little directionality, is a particularly occurrent case for a hearing situation with a main sound source, which in this case is formed by an interlocutor. When such a hearing situation is then first of all recognized, it is assumed that the line of vision of the user is usually in the direction of the interlocutor who is currently active in the user is not himself speaking and hence no speech activity of his own is recognized. Hence, by setting the preferential direction of the directional characteristic in a frontal direction, an efficient improvement in the signal-to-noise ratio can be attained for the speech signal from the interlocutor. This applies particularly when the frontal direction is ascertained not by further analysis of the picked-up signal but rather on the basis of the orientation of the first directional microphone when the hearing device is worn properly.
In a further advantageous refinement, the hearing system contains a further hearing device, particularly a hearing aid. The further hearing device has an associated second directional microphone and the second directional microphone converts sound into a second signal that is adopted in the picked-up signal. The recognition of a speech activity of the user prompts the preferential direction of the second directional characteristic to be adjusted in comparison with a frontal direction of the user such that the second directional characteristic undergoes attenuation in the frontal direction.
In particular, the hearing system is in the form of a binaural hearing system in which the picked-up signal, which adopts the first signal from the at least one hearing device and the second signal from the further hearing device, is used to form a stereo reproduction signal such that production of a first reproduction channel and a second reproduction channel involves the use of signal components from both the first signal and the second signal. In this case, a first reproduction channel can contain firstly only signal components of the first signal, whose directional characteristic is also influenced on the basis of the signal components of the second signal, or can directly include signal components of the first signal and the second signal. In a binaural hearing system, the user's own voice is perceived as coming from the frontal direction on account of the propagation time difference in the signal components of the voice, which difference fails to materialize between the first signal and the second signal, and is amplified as appropriate. For a binaural hearing system, the proposed method can reduce the signal components of the user's own voice in the picked-up signal in a particularly simple manner and without a high level of complexity in the signal processing, no noticeable alterations occurring in the signal level or in background noise picked up from the surroundings.
Preferably, in the binaural hearing system, the first directional microphone and the second directional microphone are respectively formed by a first single microphone in one, in that case the first, hearing device and by a second single microphone in the further, in that case the second, hearing device, wherein in respect of the first directional characteristic and in respect of the second directional characteristic, the signals from the first single microphone and from the second single microphone are processed for one hearing device and the other hearing device separately. In other words, the single microphones of the two hearing devices, namely a left and a right hearing device, are connected up to form a directional microphone. The first and second directional characteristics result from signal processing on the signals from the first and second single microphones that are separate from the two hearing devices. Preferably, the two single microphones have an essentially omnidirectional characteristic. The separate signal processing means that each of the two hearing devices has an associated specific, namely a first, e.g. left, directional characteristic and a second, e.g. right, directional characteristic.
Expediently, the preferential direction of the second directional characteristic is adjusted at an angle of between 5° and 20°, preferably between 5° and 10°, preferably in comparison with the frontal direction of the user. Expediently, the preferential directions of the first directional characteristic and the second directional characteristic are each adjusted in the cited angle range in this case. To alter the orientation of the directional characteristic, adjustment of the preferential direction in the cited angle range is usually sufficient in order to attain adequate attenuation of the signal component of the user's own speech in the picked-up signal. Owing to the only slight alteration of the preferential direction, however, the perception of background noise in the hearing situation, which background noise has a certain directional dependency, such as e.g. music from a locatable loudspeaker at a distance of several meters, is also not really influenced, which means that the adaptation becomes even less noticeable for a user.
It is found to be additionally advantageous if the picked-up signal is monitored for a hearing situation corresponding to a conversation, wherein recognition of a speech activity of an interlocutor prompts the preferential direction of the second directional characteristic to be oriented to the frontal direction of the user. Expediently, the preferential directions of the first directional characteristic and the second directional characteristic are each oriented to the frontal direction of the user in this case. Specifically in the case of a binaural hearing system having a rich spatial sound, this allows particularly good intelligibility of the speech signal from the interlocutor to be attained in the cited hearing situation with little complexity in the signal processing.
Preferably, recognition of a speech activity of the user prompts the preferential directions of the first directional characteristic and the second directional characteristic to be adjusted in comparison with the frontal direction of the user such that the preferential directions of the first directional characteristic and the second directional characteristic are averted from one another. In particular, the adjustment can be made symmetrically in this case, i.e. the preferential direction of the first directional characteristic and the preferential direction of the second directional characteristic are each adjusted by the same angular amount in an outward direction away from the frontal direction. As a result, depending on the directionality in the directional characteristics, a kind of channel is formed between the directional characteristics of the two hearing devices by the averted setting, which channel has reduced sound sensitivity. The sound of the user's own voice propagates in this channel in this case.
Alternatively, recognition of a speech activity of the user prompts the preferential directions of the first directional characteristic and the second directional characteristic to be adjusted in comparison with the frontal direction of the user such that the preferential directions of the first directional characteristic and the second directional characteristic, setting out as vectors from the respective hearing device, intersect one another. In particular, the adjustment is made symmetrically in this case, i.e. the preferential direction of the first directional characteristic and the preferential direction of the second directional characteristic are each adjusted in an inward direction or toward one another by the same angular amount with respect to the frontal direction. Adjustment of the preferential directions of the directional characteristics toward one another reduces the signal component of the user's own voice that is adopted in the picked-up signal by the shadowing effect that the head exerts on the sound of the voice when the directional characteristic is appropriately oriented. In addition, when the directional characteristics cross over, sudden interjections by an interlocutor standing head-on can continue to be captured almost uninfluenced.
It is found to be additionally advantageous if the picked-up signal is broken down into a plurality of frequency bands, and recognition of a speech activity of the user prompts a preferential direction of the first directional characteristic and/or a preferential direction of the second directional characteristic to be adjusted in comparison with the frontal direction of the user in at least one frequency band. If recognition of a speech activity of the user prompts the directional characteristics to be adjusted in comparison with the frontal direction only in those frequency bands either in which an excessive signal component in the picked-up signal is perceived as particularly disagreeable by the user or which have a particularly high level of directionality in the speech signal and hence are particularly simple to be influenced by adjusting the directional characteristic, then it is possible to save resources in the signal processing of the other frequency bands.
The invention further cites a binaural hearing system having two hearing devices, particularly hearing aids, that each have at least one directional microphone that is set up to perform the method described previously. In this case, the advantages specified for the method and the developments thereof could be transferred mutatis mutandis to the binaural hearing system.
Other features which are considered as characteristic for the invention are set forth in the appended claims.
Although the invention is illustrated and described herein as embodied in a method for improving a picked-up signal in a hearing system, it is nevertheless not intended to be limited to the details shown, since various modifications and structural changes may be made therein without departing from the spirit of the invention and within the scope and range of equivalents of the claims.
The construction and method of operation of the invention, however, together with additional objects and advantages thereof will be best understood from the following description of specific embodiments when read in connection with the accompanying drawings.
Each of the corresponding parts and magnitudes are each provided with the same reference symbols throughout the figures.
Referring now to the figures of the drawings in detail and first, particularly to
The instantaneous conversation situation is thus such that the user 1 orients his frontal direction 28 (which is provided by the line of vision of the head) toward the interlocutor 22 to allow better listening. If, in the present conversation situation, the interlocutor 22 is now the main protagonist, and if the other interlocutors 23 to 26 make only occasional interjections in this case, then the preferential direction 12 of the first directional characteristic 10 and the preferential direction 18 of the second directional characteristic 16 are oriented in the frontal direction 28 of the user 1, which means that the latter can perceive the speech signal from the interlocutor 22 particularly well via the binaural hearing system 2.
When there is speech activity of the user 1, the sound of his own speech is diffracted around the head such that the sound of his own speech hits each ear and hence each of the two hearing devices 6, 7 essentially from the frontal direction. The frontal perception is furthermore conditional upon the sound of his own speech arriving at both ears without a propagation time difference, so that the symmetrical perception means that there is resultant awareness of a sound hitting head-on. This means that when a first directional characteristic of the first directional microphone 8 and the second directional characteristic 16 of the second directional microphone 14 are each oriented in the frontal direction 28, the sound of the user's 1 own voice is adopted to a special degree in the picked-up signal of the binaural hearing system 2. The gain in the signal processing of the binaural hearing system 2 means that this can lead to the user perceiving his own speech at a disagreeably loud level.
In order to prevent such perception, the directional characteristics 10, 16 of the two directional microphones 8, 14 of the binaural hearing system 2 are adjusted contrary to the frontal direction 28 when the user 1 himself is speaking. This is shown in
A further option for the adjustment of the directional characteristics 10, 16 in comparison with the frontal direction 28 of the user 1 for his own speech activity is shown schematically in a plan view in
When there is a speech activity of the user 1, the attenuation of the signal component that is adopted in a picked-up signal of the binaural hearing system 2 occurs primarily as a result of shadowing effects by the head of the user 1. In this case, the directional characteristics 10, 16 are set such that the regions with particularly high sound sensitivity are already partially covered (“shadowed”) by the head of the user, so that the sound of the user's 1 own speech that hits the directional microphones 8, 14 head-on also undergoes attenuation thereby in the picked-up signal. The variant presented in this case additionally has the advantage that the interlocutor 22 is still captured well by the directional characteristics 16, 10 of the directional microphones 8, 14, i.e. despite the adjustment of the preferential directions 12, 18, the sound sensitivity of the directional microphones 8, 14 in immediate surroundings of the interlocutor 22 is still good enough, given a standard conversational distance of 0.5 to 1.5 m, to pick up any spontaneous replies and/or interjections by the interlocutor 22 in the picked-up signal of the binaural hearing system 2 as well at a sufficiently high signal level.
If no speech activity 50 is detected, then a prerequisite made for conversation is that the user directs his eyes toward an interlocutor 22, who can be regarded as the main sound source 52 for the given hearing situation 20. In this case, the directional characteristics 10, 16 are oriented in a frontal direction in the manner explained above. This can be effected particularly also on the basis of a speech activity 54 that is recognized for the interlocutor, i.e. adjustment of the directional characteristics 10, 16 in the relevant frequency bands toward the frontal direction is effected only when the signal coming from the direction of the interlocutor 22 has a correspondingly significant level, which means that the speech activity 54 is detected as a result. The picked-up signal 42 formed by the frequency bands 48 can then continue to be handled in accordance with the applications provided for the binaural hearing system 2.
In a preferred variant, the two hearing devices 6, 7 each comprise an essentially omnidirectional single microphone 55 or 56 (see
Although the invention has been illustrated and described in more detail by means of the preferred exemplary embodiment, the invention is not restricted by this exemplary embodiment. Other variations can be derived therefrom by a person skilled in the art without departing from the scope of protection of the invention.
The following is a summary list of reference numerals and the corresponding structure used in the above description of the invention:
Rosenkranz, Tobias Daniel, Lugger, Marko, Kamkar-Parsi, Homayoun
| Patent | Priority | Assignee | Title |
| 11070923, | Apr 18 2019 | SIVANTOS PTE LTD | Method for directional signal processing for a hearing aid and hearing system |
| Patent | Priority | Assignee | Title |
| 6449593, | Jan 13 2000 | RPX Corporation | Method and system for tracking human speakers |
| 7853031, | Jul 11 2005 | Sivantos GmbH | Hearing apparatus and a method for own-voice detection |
| 8462969, | Apr 22 2010 | SIVANTOS PTE LTD | Systems and methods for own voice recognition with adaptations for noise robustness |
| 8873779, | Dec 08 2011 | SIVANTOS PTE LTD | Hearing apparatus with own speaker activity detection and method for operating a hearing apparatus |
| 9124986, | Jun 27 2011 | Oticon A/S | Feedback control in a listening device |
| 20070098192, | |||
| 20120189147, | |||
| 20160234609, | |||
| DE102005032274, | |||
| DE102011087984, | |||
| DE60022304, | |||
| EP2381702, | |||
| EP2541973, |
| Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
| Mar 19 2018 | Sivantos Pte. Ltd. | (assignment on the face of the patent) | / |
| Date | Maintenance Fee Events |
| Mar 19 2018 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
| Aug 29 2022 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
| Date | Maintenance Schedule |
| Mar 12 2022 | 4 years fee payment window open |
| Sep 12 2022 | 6 months grace period start (w surcharge) |
| Mar 12 2023 | patent expiry (for year 4) |
| Mar 12 2025 | 2 years to revive unintentionally abandoned end. (for year 4) |
| Mar 12 2026 | 8 years fee payment window open |
| Sep 12 2026 | 6 months grace period start (w surcharge) |
| Mar 12 2027 | patent expiry (for year 8) |
| Mar 12 2029 | 2 years to revive unintentionally abandoned end. (for year 8) |
| Mar 12 2030 | 12 years fee payment window open |
| Sep 12 2030 | 6 months grace period start (w surcharge) |
| Mar 12 2031 | patent expiry (for year 12) |
| Mar 12 2033 | 2 years to revive unintentionally abandoned end. (for year 12) |