During mixing of two musical tracks, the variations in combined output volume are reduced by analyzing either the intrinsic amplitude at which each track was mastered or the output amplitude (i.e. subsequent to amplification of the audio signal), and modifying either the intrinsic amplitude or amplification during the mixing phase. Musical clashes during mixing are avoided by analyzing intrinsic amplitudes of the two tracks at similar frequencies to detect the likelihood of a clash, and in the event a clash is detected, reducing the output amplitude of one of the tracks at the relevant frequency.
|
23. An apparatus for equalising output amplitude of first and second sections of musical content intended for simultaneous output, the apparatus comprising:
first and second audio data sources on which the first and second sections of the musical content are stored; first and second music players, adapted to generate first and second audio signals from the first and second audio data sources respectively; an amplifier for amplifying the first and second audio signals; an analyser, adapted to analyse at least the first and second audio signals, and to determine an extent of variation between a constant value and a combined intrinsic amplitude of the first and second audio signals resulting from simultaneous playing of the first and second audio signals.
1. A method for automated mixing of at least two pieces of musical content comprising the steps of:
selecting first and second sections of first and second tracks respectively, over which transition between playing the first and second tracks will be made; analysing intrinsic recorded amplitude of the first and second tracks over the first and second sections respectively; simultaneously playing the first and second sections of the first and second tracks; effecting the transition from playing the first track to playing the second track by reducing output amplitude of the first track over duration of the first section and increasing output amplitude of the second track over duration of the second section; and using analysis of the intrinsic recorded amplitude of at least one of the first and second tracks to equalise variations in net output volume from the first and second tracks over the duration of the transition.
2. The method according to
3. The method according to
4. The method according to
5. The method according to
6. The method according to
7. The method according to
8. The method according to
9. The method according to
generating a copy of at least one of the sections of one of the first and second tracks over which the transition will be made; modifying at least one intrinsic amplitude value from the copy to generate a modified copy; and, amplifying the modified copy during the transition.
10. The method according to
pairing at least selected intrinsic amplitude values of the copy with contemporaneous amplitude values from a template profile of amplitude with time; and modifying amplitude values from the copy so that the amplitude values are equal to the contemporaneous values from the template profile.
11. The method according to
summing a plurality of contemporaneous intrinsic amplitude values of the first and second tracks over the duration of the transition, to generate sets of contemporaneous summed amplitudes; and normalising each of the sets of contemporaneous summed amplitudes so that the sum is equal to a predetermined constant amplitude, by modifying the intrinsic amplitude values of the copy of at least one of the first and second tracks.
12. The method according to
13. The method according to
14. The method according to
15. The method according to
generating an indication of variation in output amplitude for a plurality of temporal juxtapositions for simultaneous playing of the first and second tracks; selecting a temporal juxtaposition of the plurality of temporal juxtapositions having the lowest indicated variation in output amplitude; and, effecting transition from the first track to the second track by playing the first and second tracks in the temporal juxtaposition selected in the selecting step.
16. The method according to
17. The method according to
18. The method according to
19. The method according to
sampling the output amplitude of the first and second tracks, resulting in sampled output amplitude; comparing the sampled output amplitude to a template profile of output amplitude with time; and adjusting amplification of at least one of the first and second tracks in accord with a result of the comparing step.
20. The method according to
21. The method according to
22. The method according to
24. The apparatus according to
25. The apparatus according to
26. The apparatus according to
27. The apparatus according to
28. The apparatus according to
29. The apparatus according to
|
The present invention relates to the automated compilation of pieces of musical content, usually referred to as "tracks", and more particularly, to compilation in which one track is phased in over the top of another, preferably in a manner providing an apparently seamless transition between tracks. This is known in current vernacular as "mixing".
Our co-pending UK application (HP docket 30001926) discloses, inter alia, a system and method for the automated compilation of tracks which are typically stored as digital audio, such as on compact disc. In this system, the outputs of two digital audio players are fed to an output, such as a set of speakers. The speed at which tracks from the two CD players are played is adjusted, so that the beat of an incoming track is matched to the speed of a track currently playing (known as "time stretching"), and once this has been achieved an automated cross-fading device reduces the output volume of the current track while increasing the output volume of the incoming track, thereby to provide a seamless transition between them.
A first aspect of the present invention addresses the issue of amplification of each of the tracks during the transition phase from one track to another, or "cross-fade". In an automated system, in order to try to provide a seamless transition between tracks, amplification of the outgoing track will typically be reduced at the same rate as the amplification of the incoming track is increased, with the reduction and increase in amplification starting at the same time. Frequently tracks are mixed so that the incoming track is faded in over the end of the outgoing track, as a result of which the volume on the outgoing track may well be reducing, since many dance tracks end simply by fading out the volume to zero, or start by fading in the volume from zero (i.e. the intrinsic amplitude or "mastered volume" of the recording is reduced to zero, or increased from zero, as the case may be). In such a situation, unless the fade-out rate of the intrinsic amplitude (and thus for a constant level of amplification, the volume) at the end of the outgoing track matches the fade-in rate of the intrinsic amplitude at the beginning of the incoming track, and both are in turn matched with the rate of cross-fading the amplification from one track to another, the transition between the tracks will be subject to a variation in volume which is undesirable, since it disturbs the seamless transition between incoming and outgoing tracks.
Accordingly, a first aspect of the present invention provides a method for the automated mixing of at least two pieces of musical content comprising the steps of:
selecting first and second sections of first and second tracks respectively, over which transition between playing the first and second tracks will be made;
sampling intrinsic recorded amplitude of the first and second tracks over the first and second sections respectively;
simultaneously playing the first and second sections of the first and second tracks;
effecting transition from playing the first track to playing the second track by reducing output volume of the first track over duration of the first section and increasing output volume of the second track over duration of the second section; and
using sampling of the intrinsic amplitude of at least one of the first and second tracks to equalise variations in net output volume from the first and second tracks over the duration of the transition.
Equalisation of variations in recorded amplitude may result merely in a reduction in variations of net output volume in comparison to what would otherwise be the case, or may result in a substantially constant net output volume, depending upon the extent of equalisation. Equalisation may be achieved typically either by altering the amplification of one or both tracks over the course of the transition, altering the intrinsic recorded amplitude of one or both tracks, or a combination of both techniques.
In one embodiment of equalisation by regulation of amplification for one or both of the tracks, a series of synchronous intrinsic amplitude values are sampled from each of the tracks, and contemporaneous values are then summed to determine the extent, if any, to which the combined intrinsic amplitude varies over the transition phase. The resultant variation in intrinsic amplitude is then used to generate an amplification profile which is then applied proportionally to one or both the tracks during the transition to equalise the net output volume. Equalisation by modification of intrinsic amplitude may use the contemporaneous summed amplitude values to generate discrete error values by which summed amplitude should be altered in order to maintain a constant value over the transition phase.
In an alternative embodiment amplification or intrinsic amplitude modification is used to configure predetermined sections of tracks to predetermined introduction and playout template profiles of amplitude against time, so that any two tracks conforming to the profile (either by variation in amplification or intrinsic amplitude) may be mixed together.
In yet a further embodiment an indication of variation in combined amplitude is generated for a plurality of temporal juxtapositions of two tracks, and the temporal juxtaposition having the lowest indicated variation is selected.
Typically, the equalisation will be performed on the basis of the sampling of the intrinsic amplitude in a particular frequency range determined as dominant, and this will in turn typically be determined on the basis of the frequency of the beat used for time stretching the incoming track and outgoing tracks.
A second and independent aspect of the present invention is concerned with the musical elements present in the outgoing and incoming tracks, such as vocal lines, melodic instrument parts, or percussion signatures (from, e.g. snare drums, symbols or handclaps etc.). It is not unusual for such elements in the outgoing and incoming tracks to clash, even though the fundamental beats of the two tracks have been matched, and the volume of the two tracks has been equalised over the cross fade. The result of such a clash is that when these elements are heard together the result is an unappealing mix.
Accordingly, a second aspect of the present invention provides a method for automated mixing of first and second music tracks comprising the steps of:
selecting first and second sections of the first and second tracks respectively, over which a transition between the first and second tracks will occur;
for at least selected intrinsic peak amplitudes of the first track, determining, in accordance with at least one predetermined criterion, whether a musical clash exists with an intrinsic peak amplitude from the second track; and
in the event of a clash, reducing output amplitude of at least one of the tracks at least at a frequency of one of the clashing intrinsic peak amplitudes, and over a time interval at least equal to duration of the aforesaid one of the intrinsic peak amplitudes.
The reduction in output amplitude (which will typically also be a reduction in output volume) of a given frequency band may again, as with the first aspect of the present invention, be implemented either via adjustment of amplification over at least the frequency of one of the clashing peak amplitudes (although this is only possible where the system provides for differing amplification levels for different frequency bands), or by copying at least the section of the track in question into addressable memory, and altering the intrinsic recorded amplitude levels for that frequency band.
Yet a further independent aspect of the present invention provides a method of mixing first and second tracks including the steps of:
analysing variations in amplitude with time and frequency for both tracks;
on the bas is of the analysis, defining at least one frequency band common to both tracks; and
equalising output amplitude of the tracks in the frequency band during mixing from one track to another.
Thus the frequency band to be used in order to provide equalisation is defined on the basis of the musical characteristics of the tracks to be mixed, rather than using predetermined frequency bands which may not be appropriate having regard to the frequencies of the two tracks to be mixed.
Embodiments of the invention will now be described, by way of example, and with reference to the accompanying drawings, in which:
Referring now to
The illustrated system is operable to decrease or "fade out" the output volume (i.e. the amplitude of the output audio signal, which in this example is made manifest by the speakers 60) of one track from one of the audio sources, e.g. audio source 1, while simultaneously increasing or "fading in" the output volume from another track of audio source 2; ideally this is done in a manner providing a seamless mix between the outgoing and incoming tracks. The provision of such a seamless mix first of all requires that the beats of the outgoing and incoming tracks are matched. This is done by automatically regulating the speed at which one or both of the respective tracks are played, and synchronising the beats of the tracks. The automation of such a process is described in our co-pending European application (HP docket 30001926). Additionally, the output volume of each of the tracks must be regulated to ensure that there are no dramatic increases or decreases in net output volume (i.e. the combined output volume of the tracks playing on audio players 10 and 20) during the course of the transition from the outgoing track to the incoming track.
Referring now to
Referring now to
Referring now to
It is not essential to provide sampled outputs from the individual channels based on peak amplitude. For example, in an alternative configuration an integrating circuit may be used in conjunction with the master clock to provide a series of average amplitude values over the course of each clock cycle.
The sampled outputs from channels Ch1, Ch2, Ch3 are stored in a designated memory MC1, MC2, MC3 respectively (typically provided by designated areas of RAM 80), in a series of what may be thought of as temporal intrinsic peak amplitude coordinates, i.e. comprising a digital intrinsic peak amplitude value, e.g. AC1 (typically 16-24 bits long per audio channel) in conformity with current CD and DVD player standards) and a corresponding time value indicating the time elapsed since the start of the transition phase at which that peak intrinsic amplitude occurred. These three sets of coordinates may be represented in visual terms by three histograms, from which a rapid appreciation of the relative intrinsic amplitude and timing of the peaks can be obtained, and in
Having generated intrinsic amplitude coordinates by sampling the transition section of each track, the coordinates from the dominant channel are then used to provide equalisation of the net output volume. Sampled outputs of the two tracks Z1 and Z2 from the dominant frequency channel which are to occur contemporaneously during the mix are summed together (remembering that the outputs in the low frequency range are synchronised as a result of time stretching and automatic synchronisation in accordance with our co-pending European application 00303960.0) to provide a series of summed contemporaneous values of peak intrinsic amplitude against time, i.e. summed contemporaneous peak amplitude coordinates (ΣACnNBCnN, NT+tCnN). These summed peak amplitude coordinates are illustrated schematically in the histogram of
To adjust the amplification gain over the transition period, a profile of amplification level or gain with time is generated from the summed peak amplitude coordinates, and is then applied to the two tracks. The amplification profile is generated by taking the amplitude value from each summed peak amplitude coordinate, and comparing it to the relatively constant intrinsic amplitude prior to entering the transition phase (NB any differences in intrinsic "constant" amplitude of the two tracks is normalised prior to mixing, either by an adjustment in amplification gain which is phased-in linearly during the transition phase, or by a modification of the intrinsic amplitude of the incoming track, in this instance Z2). In the current example, the intrinsic amplitude of the channel Ch1 frequency band (or in a different example whichever other frequency band is determined as being dominant) prior to entering the transition phase is equal to a substantially constant value α, and the amplification gain q is at a constant value Q. However, at a time NT+t after the start of the transition phase the summed peak amplitude ΣACnNBCnN has dropped below a by an amount δα, given by the expression (ΣACnNBCnN-α) to the value (α+δα).
against time is generated, which in turn may be used to approximate a continuous profile of amplification gain against time during the course of the transition phase (e.g. by fitting a curve to the discrete values) and this profile is shown in FIG. 6C.
The amplification profile is then applied to the outputs of the two audio players 10, 20 without discrimination as to frequency range (since the output of the players is not naturally split into frequency bands) over the duration of the transition phase. The gain levels specified by the amplification profile may be split between the amplifiers 30, 40 of the audio players 10, 20 in any ratio desired, provided that at any instant the net amplification gain applied to the two tracks Z1, Z2 (i.e. the linear sum of the gain applied to tracks individually) is equal to the amplification gain specified by the profile at that instant. In one embodiment the gain values will be split 50--50 between the two players, so that the fade-out and fade-in of the two tracks as a result of their intrinsic amplitude is replicated in relative terms in the transition phase. Alternatively, the relative intrinsic peak amplitudes of the two tracks during the transition phase may be taken into account, in which case the gain is apportioned between the amplifiers 30, 40 so the fade-out and fade-in is substantially linear. Alternatively the amplification profile is applied to only one track.
Although reference has frequently been made to the use of digital audio players in conjunction with the method and apparatus of the present invention, it is not necessary to use such players for implementation of the invention. For example, amplification could be applied to digital audio of the final mix (or near final mix), and used to produce a final mix audio file that is stored in memory.
Equalisation of the net output volume by modification of intrinsic amplitudes may also be performed using the summed contemporaneous peak amplitude coordinates show n in FIG. 6A. Once again each summed peak amplitude ΣACnNBCnN is compared with the pre-transition phase "constant" level α, to generate a value δαN equal to the difference between them. As previously, each value δαN has a positive sign if the summed peak amplitude ΣACnNBCnN is larger than α, and a negative sign if smaller. In the present example each summed peak amplitude ΣACnNBCnN is smaller than α, and so each summed peak amplitude must be increased by (ΣACnNBCnN-δαN) in order to make it equal to α. The total increase required in the summed peak amplitude ΣACnNBCnN for equalisation is then apportioned between the individual intrinsic peak amplitudes in proportion to their size, so the Nth intrinsic peak amplitude value ACnN will be increased by a value:
and the Nth intrinsic peak amplitude value BCnN will be increased by a value
From these absolute values ΔAN and ΔBN of peak amplitude incrementation, a set of proportional reduction values ΔAN/ACnN, and ΔBN/BCnN are easily calculable. These discrete proportional reduction values may then be used to approximate a continuous profile of proportional amplitude modification against time (for example by fitting a curve to the points as in the case of the curve of FIG. 6C), which may then in turn be used to modify each intrinsic amplitude value (as opposed simply to the peak intrinsic amplitude values) of the respective track Z1 or Z2 by an amount proportional to its amplitude. Once the intrinsic amplitudes of the tracks Z1 or Z2 have been modified, the tracks may then be mixed simply by maintaining a constant amplification gain on each track throughout the duration of the mix, since equalisation of the net volume has been performed by the creation of the modified amplitude values.
Physical modification of the intrinsic amplitudes involves copying the transition section of each track Z1, Z2 to a RAM, and then modifying the copied version of the transition section which is stored in the RAM. This is feasible, since the maximum frequency of a CD-quality digital audio signal is approximately 22 KHz, and so is sampled at 44.1 KHz in order to capture all the variations in amplitude (i.e. two "values" of amplitude per cycle). If the transition between the tracks lasts for ten seconds, then 0.88 Mb of memory will be required for each track (digital audio usually operating on 16 bits rather than 8), meaning a tot al required RAM capacity of less then 2 Mb.
In a further embodiment of the present invention, equalisation is performed by considering each of the tracks separately. Referring now to
In a further modification, a combination of amplification adjustment and modification to intrinsic amplitude may be employed, either to tailor two tracks together individually as described above, or to configure tracks to a template profile.
In an alternative embodiment variations in net output volume are minimised by matching sampled fade-out and fade-in sections of two track s in a variety of temporal juxtapositions, i.e. different instances of starting to play the fade-in part of one track simultaneously with the fade-out part of another, and the temporal juxtaposition yielding the smallest variation in net output volume over the duration of the transition is adopted. While this embodiment may not necessarily provide full, or substantially full equalisation, it nevertheless reduces net output volume variations in comparison to what they would otherwise be, and has the virtue of being simple and therefore quicker than the other embodiments. Referring now to
Thus |δαN| is the absolute value of the difference between the sum of contemporaneous peak amplitude values, and the value α is established as the substantially constant amplitude prior to the transition phase. In the example illustrated in
The two sets of peak amplitudes are then re-juxtaposed, with the first and last peak amplitudes of tracks Z2 and Z2 summed together as illustrated in
A further independent aspect of the present invention relates to a qualitative aspect of providing an appealing mix between two tracks. Referring again to
Referring once again to
Preferably, in the event that this frequency blending technique is to be employed in a system also employing techniques to equalise net output volume, the volume equalisation processing is performed first, so that any effect this may have on the output volume of elements from a given non-dominant frequency band may be taken into account, both in determining whether a clash is likely to occur, and in modifying output volumes for musical elements in a particular frequency band.
As mentioned previously in connection with
Clashes may however be prevented without defining further frequency bands. For example, to provide the maps of
Referring now to
It is possible that the reduction in peak amplitude could take an amplitude from one box and into another, thus causing a further reduction in the peak amplitude, which could in theory result in an iterative reduction of some frequencies to negligible (i.e. non audible) levels, it is necessary either to restrict the number of iterations of the process described above, or to stop the process once the non-dominant amplitudes have dropped below a predetermined level.
Analysis of the response of the human ear to different frequencies has shown that, over the range of audible frequencies, the ear is more responsive to some frequencies than others. Thus an audio signal having a constant output volume, whose frequency increases steadily to sweep through the spectrum of audible frequencies, will seem to a listener to be louder at some frequencies in the audible range than others (see for example "The Computer Music Tutorial, Curtis Roads, MIT Press 1998, pp. 1049-1069). In a modification of the technique described above therefore, the sizes of the boxes in amplitude-frequency-time space are weighted in accordance with the established response of the ear. That is to say that at frequencies which the ear is less responsive the boxes are smaller (i.e. a clash between two signals is considered likely only if they are extremely similar), and vice versa.
The range of amplitudes, frequencies and the time interval which define a clash between two peak amplitudes from different tracks have been defined above using Cartesian coordinates, and so boxes within frequency-amplitude-time space have naturally resulted. This is merely for convenience, and any boundary conditions for clashes deemed most appropriate may be defined. Thus for example it is perfectly feasible to define a range of frequencies within which a clash may occur, which range varies with variations in amplitude and time, resulting in e.g., a sphere in frequency-amplitude-time space which defines a clash.
The methods described thus far have all related to analysis and processing of the audio data which occurs prior to playing. It is however possible to perform a degree of equalisation in real time. For example, using a simplified version of the apparatus of
Patent | Priority | Assignee | Title |
10356147, | Aug 24 2007 | BANK OF AMERICA, N A , AS SUCCESSOR COLLATERAL AGENT | Customized perishable media content based on user-specified preference for static or variable location |
10587660, | Aug 24 2007 | BANK OF AMERICA, N A , AS SUCCESSOR COLLATERAL AGENT | Selection of preferred content |
10909577, | Aug 24 2007 | BANK OF AMERICA, N A , AS SUCCESSOR COLLATERAL AGENT | Mapping user notifications to specific media streams |
11265355, | Aug 24 2007 | BANK OF AMERICA, N A , AS SUCCESSOR COLLATERAL AGENT | Customized perishable media content based on user-specified preference for static or variable location |
11563787, | Aug 24 2007 | iHeartMedia Management Services, Inc. | Mobile device selection of preferred content |
11587124, | Aug 24 2007 | iHeartMedia Management Services, Inc. | Passing control of inserting third-party content into a media stream |
6849795, | May 15 1998 | NRI R&D PATENT LICENSING, LLC | Controllable frequency-reducing cross-product chain |
7151215, | Apr 28 2003 | MEDIATEK INC. | Waveform adjusting system for music file |
7189913, | Apr 04 2003 | Apple Inc | Method and apparatus for time compression and expansion of audio data with dynamic tempo change during playback |
7233832, | Apr 04 2003 | Apple Inc | Method and apparatus for expanding audio data |
7272235, | Jun 26 2003 | Microsoft Technology Licensing, LLC | Method and apparatus for audio normalization |
7425674, | Apr 04 2003 | Apple, Inc. | Method and apparatus for time compression and expansion of audio data with dynamic tempo change during playback |
7596234, | Jun 26 2003 | Microsoft Technology Licensing, LLC | Method and apparatus for playback of audio files |
7663051, | Mar 22 2007 | Qualcomm Incorporated | Audio processing hardware elements |
8054994, | Jun 26 2003 | Microsoft Technology Licensing, LLC | Method and apparatus for audio normalization |
8175730, | Jun 30 2009 | Sony Corporation | Device and method for analyzing an information signal |
8553504, | Dec 08 2008 | Apple Inc. | Crossfading of audio signals |
8671214, | Aug 24 2007 | BANK OF AMERICA, N A , AS SUCCESSOR COLLATERAL AGENT | Customizing perishable content of a media channel |
8719349, | Aug 24 2007 | BANK OF AMERICA, N A , AS SUCCESSOR COLLATERAL AGENT | System and method for providing a radio-like experience |
9082135, | Aug 24 2007 | BANK OF AMERICA, N A , AS SUCCESSOR COLLATERAL AGENT | Third party delivery of preferred perishable content |
9300618, | Aug 24 2007 | BANK OF AMERICA, N A , AS SUCCESSOR COLLATERAL AGENT | Customizing perishable content of a media channel |
9699232, | Aug 24 2007 | BANK OF AMERICA, N A , AS SUCCESSOR COLLATERAL AGENT | Adding perishable content to media stream based on user location preference |
9860288, | Aug 24 2007 | BANK OF AMERICA, N A , AS SUCCESSOR COLLATERAL AGENT | Synchronization of preferred perishable content |
9990655, | Aug 24 2007 | BANK OF AMERICA, N A , AS SUCCESSOR COLLATERAL AGENT | Live media stream including personalized notifications |
Patent | Priority | Assignee | Title |
5418321, | Dec 15 1992 | Gateway, Inc | Audio channel system for providing an analog signal corresponding to a sound waveform in a computer system |
5548655, | Oct 01 1992 | Hudson Soft Co., Ltd. | Sound processing apparatus |
5802187, | Jan 26 1996 | United Microelectronics Corp. | Two-channel programmable sound generator with volume control |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Apr 24 2002 | Hewlett-Packard Limited | Hewlett-Packard Company | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 012848 | 0804 | |
Apr 26 2002 | Hewlett-Packard Company | (assignment on the face of the patent) | ||||
Jul 28 2003 | Hewlett-Packard Company | HEWLETT-PACKARD DEVELOPMENT COMPANY, L P | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 013862 | 0623 |
Date | Maintenance Fee Events |
Sep 18 2006 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Sep 20 2010 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Oct 24 2014 | REM: Maintenance Fee Reminder Mailed. |
Mar 18 2015 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
Mar 18 2006 | 4 years fee payment window open |
Sep 18 2006 | 6 months grace period start (w surcharge) |
Mar 18 2007 | patent expiry (for year 4) |
Mar 18 2009 | 2 years to revive unintentionally abandoned end. (for year 4) |
Mar 18 2010 | 8 years fee payment window open |
Sep 18 2010 | 6 months grace period start (w surcharge) |
Mar 18 2011 | patent expiry (for year 8) |
Mar 18 2013 | 2 years to revive unintentionally abandoned end. (for year 8) |
Mar 18 2014 | 12 years fee payment window open |
Sep 18 2014 | 6 months grace period start (w surcharge) |
Mar 18 2015 | patent expiry (for year 12) |
Mar 18 2017 | 2 years to revive unintentionally abandoned end. (for year 12) |