This invention introduces a class of multi-band linear phase lapped biorthogonal transforms with fast, VLSI-friendly implementations via lifting steps called the LiftLT. The transform is based on a lattice structure which robustly enforces both linear phase and perfect reconstruction properties. The lattice coefficients are parameterized as a series of lifting steps, providing fast, efficient in-place computation of the transform coefficients as well as the ability to map integers to integers. Our main motivation of the new transform is its application in image and video coding. Comparing to the popular 8×8 DCT, the 8×16 LiftLT only requires 1 more multiplication, 22 more additions, and 6 more shifting operations. However, image coding examples show that the LiftLT is far superior to the DCT in both objective and subjective coding performance. Thanks to properly designed overlapping basis functions, the LiftLT can completely eliminate annoying blocking artifacts. In fact, the novel LiftLT's coding performance consistently surpasses that of the much more complex 9/7-tap biorthogonal wavelet with floating-point coefficients. More importantly, our transform's block-based nature facilitates one-pass sequential block coding, region-of-interest coding/decoding as well as parallel processing.
|
0. 12. A method of coding, storing or transmitting, and decoding a block of M×M intensities from a digital image selected by an M×M window moving recursively over the image, comprising:
a. processing the intensities in an M×M block coder comprising the steps of:
i. processing the intensities through an initial stage;
ii. scaling the result of the preceding step in each channel;
b. processing the result of the preceding step through a transform coder using a method of processing blocks of samples of digital signals of integer length m comprising processing the digital samples of length m with an invertible linear transform of dimension m, said transform being representable as a cascade, using the steps, in arbitrary order, of:
i) at least one +/−1 butterfly step,
ii) at least one lifting step with rational complex coefficients, and
iii) at least one scaling factor;
c. transmitting or storing the output coefficients of said M×M block coder;
d. receiving the output coefficients in a decoder; and
e. processing the output coefficients in the decoder into a reconstructed image using the inverse of the coder of steps a. and b.
0. 26. A method of coding, storing or transmitting, and decoding sequences of intensities of integer length m recursively selected from a time ordered string of intensities arising from electrical signals, the method comprising the steps of
a) recursively processing the sequences of intensities of integer length m with an invertible forward linear transform of dimension m, said transform being representable as a cascade using the steps, in a preselected arbitrary order, of:
ii) at least one ±1 butterfly step,
iii) at least one lifting step with rational complex coefficients, and
iv) applying at least one scaling factor;
b) compressing the resulting transform coefficients;
c) storing or transmitting the compressed transform coefficients;
d) receiving or recovering from storage the transmitted or stored compressed transform coefficients;
e) decompressing the received or recovered compressed transform coefficients; and
f) recursively processing the decompressed transform coefficients with the inverse of the forward linear transform of dimension m, said inverse transform being representable as a cascade using the steps, in the exact reverse order of the preselected arbitrary order, of:
ii) at least one inverse butterfly corresponding to each of the at least one ±1 butterfly step;
iii) at least one inverse lifting step corresponding to each of the at least one lifting step with rational complex coefficients; and,
iv) applying at least on inverse scaling factor corresponding to the at least one scaling factor.
8. An apparatus for coding, compressing, storing or transmitting, and decoding a block of M×M intensities from a digital image selected by an M×M window moving recursively over the image, comprising:
a. an M×M block transform comprising:
i. an initial stage
ii. a normalizing factor in each channel
b. a cascade comprising a plurality of dyadic rational lifting transforms, each of said plurality of dyadic rational lifting transforms comprising
i. a first bank of pairs of butterfly lifting steps with unitary coefficients between adjacent lines of said transform;
ii. a bank of delay lines in a first group of m/2 alternating lines;
iii. a second bank of butterfly lifting steps with unitary coefficients, and
iv. a bank of pairs of butterfly lifting steps with coefficients of 1/2 between m/2−1 pairs of said m/2 alternating lines;
c. means for transmission or storage of the output coefficients of said M×M block transform; and
d. an inverse transform comprising
i. a cascade comprising a plurality of dyadic rational lifting transforms, each of said plurality of dyadic rational lifting transforms comprising
a) a bank of pairs of butterfly lifting steps with coefficients of 1/2 between said m/2−1 pairs of said m/2 alternating lines;
b) a first bank of pairs of butterfly lifting steps with unitary coefficients between adjacent lines of said transform;
c) a bank of delay lines in a second group of m/2 alternating lines, ; and
d) a second bank of pairs of butterfly lifting steps with unitary coefficients between adjacent lines of said transform;
ii. a de-scaling bank, ; and
iii. an inverse initial stage.
1. An apparatus for coding, storing or transmitting, and decoding M×M sized blocks of digitally represented images, where m is an even number, comprising
a. a forward transform comprising
i. a base transform having m channels numbered 0 through M−1, half of said channel numbers being odd and half being even;
ii. an equal normalization factor in each of the m channels selected to be dyadic-rational;
iii. a full-scale butterfly implemented as a series of lifting steps with a first set of dyadic rational coefficients;
iv. m/2 delay lines in the odd numbered channels;
v. a full-scale butterfly implemented as a series of lifting steps with said first set of dyadic rational coefficients; and
vi. a series of lifting steps in the odd numbered channels with a second specifically selected set of dyadic-rational coefficients;
b. means for transmission or storage of the transform output coefficients; and
c. an inverse transform comprising
i. m channels numbered 0 through M−1, half of said channel numbers being odd and half being even;
ii. a series of inverse lifting steps in the odd numbered channels with said second set of specifically selected dyadic-rational coefficients;
iii. a full-scale butterfly implemented as a series of lifting steps with said first set of specifically selected dyadic-rational coefficients;
iv. m/2 delay lines in the even numbered channels;
v. a full-scale butterfly implemented as a series of lifting steps with said first set of specifically selected dyadic-rational coefficients;
vi. an equal denormalization factor in each of the m channels specifically selected to be dyadic-rational; and
vii. a base inverse transform having m channels numbered 0 through M−1.
10. A method of coding, compressing, storing or transmitting, and decoding a block of M×M intensities from a digital image selected by an M×M window moving recursively over the image, comprising the steps of:
a. Processing the intensities in an M×M block coder comprising the steps of:
i. processing the intensities through an initial stage;
ii. scaling the result of the preceding step in each channel;
b. processing the result of the preceding step through a cascade comprising a plurality of dyadic rational lifting transforms, each of said plurality of dyadic rational lifting transforms comprising
i. a first bank of pairs of butterfly lifting steps with unitary coefficients between adjacent lines of said transform;
ii. a bank of delay lines in a first group of m/2 alternating lines;
iii. a second bank of butterfly lifting steps with unitary coefficients, and
iv. a bank of pairs of butterfly lifting steps with coefficients of 1/2 between m/2−1 pairs of said m/2 alternating lines;
c. transmitting or storing the output coefficients of said M×M block coder;
d. receiving the output coefficients in a decoder; and
e. processing the output coefficients in the decoder, comprising the steps of
i. processing the output coefficients through a cascade comprising a plurality of dyadic rational lifting transforms, each of said plurality of dyadic rational lifting transforms comprising
a) a bank of pairs of butterfly lifting steps with coefficients of 1/2 between said m/2−1 pairs of said m/2 alternating lines;
b) a first bank of pairs of butterfly lifting steps with unitary coefficients between adjacent lines of said transform;
c) a bank of delay lines in a second group of m/2 alternating lines;
d) a second bank of pairs of butterfly lifting steps with unitary coefficients between adjacent lines of said transform;
e) a de-scaling bank; and
f. processing the results of the preceding step in an inverse initial stage.
9. A method of coding, storing or transmitting, and decoding M×M sized blocks of digitally represented images, where m is an even number a power of 2, comprising
a. transmitting the original picture signals to a coder, which effects the steps of
i. converting the signals with a base transform having m channels numbered 0 through M−1, half of said channel numbers being odd and half being even, ;
ii. normalizing the output of the preceding step with a dyadic rational normalization factor in each of said m channels;
iii. processing the output of the preceding step through two lifting steps with a first set of identical dyadic rational coefficients connecting each pair of adjacent numbered channels in a butterfly configuration;
iv. transmitting the resulting coefficients through m/2 delay lines in the odd numbered channels;
v. processing the output of the preceding step through two inverse lifting steps with the first set of dyadic rational coefficients connecting each pair of adjacent numbered channels in a butterfly configuration; and
vi. applying two lifting steps with a second set of identical dyadic rational coefficients connecting each pair of adjacent odd numbered channels to the output of the preceding step;
b. transmitting or storing the transform output coefficients;
c. receiving the transform output coefficients in a decoder; and
d. processing the output coefficients in a decoder, comprising the steps of
i. receiving the coefficients in m channels numbered 0 through M−1, half of said channel numbers being odd and half being even;
ii. applying two inverse lifting steps with dyadic rational coefficients connecting each pair of adjacent odd numbered channels;
iii. applying two lifting steps with dyadic rational coefficients connecting each pair of adjacent numbered channels in a butterfly configuration;
iv. transmitting the result of the preceding step through m/2 delay lines in the even numbered channels;
v. applying two inverse lifting steps with dyadic rational coefficients connecting each pair of adjacent numbered channels in a butterfly configuration;
vi. denormalizing the result of the preceding step with a dyadic rational inverse normalization factor in each of said m channels; and
vii. processing the result of the preceding step through a base inverse transform having m channels numbered 0 through M−1.
2. The apparatus of
3. The apparatus of
4. The apparatus of
5. The apparatus of
6. The apparatus of
7. The apparatus of
11. The apparatus of
0. 13. The method of
0. 14. The method of
0. 15. The method of
a) said invertible transform is an approximation of a biorthogonal transform;
b) said biorthogonal transformation comprises a representation as a cascade of at least one butterfly step, at least one orthogonal transform, and at least one scaling factor;
c) said at least one orthogonal transform comprises a cascade of
i) at least one ±1 butterfly step,
ii) at least one planar rotation, and
iii) at least one scaling factor;
b) said at least one planar rotation being represented by equivalent lifting steps and scale factors; and,
c) said approximation is obtained by replacing floating point coefficients in the lifting steps with rational coefficients.
0. 16. The method of
0. 17. The method of
0. 18. The method of
a) said invertible transform is an approximation of a unitary transform;
b) said approximation of the unitary transform comprises a representation of the unitary transform as a cascade of at least one butterfly step, at least one orthogonal transform, and at least one scale factor;
c) said at least one orthogonal transform being represented as a cascade of
(1) at least one ±1 butterfly steps,
(2) at least one planar rotation, and
(3) at least one scaling factor;
d) said at least one planar rotation being represented by equivalent lifting steps and scale factors; and,
e) said approximation being derived by using approximate rational values for the coefficients in the lifting steps.
0. 19. The method of
0. 20. The method of
0. 21. The method of
where a, b are selected from the group:
±{8, 5, 4, 2, 1, 1/2, 1/4, 3/4, 5/4, 1/8, 3/8, 2/5, 5/8, 7/8, 1/16, 3/16, 5/16, 7/16, 9/16, 11/16, 13/16, 15/16, 25/16}.
0. 22. The method of
0. 23. The method of
0. 24. The method of
0. 25. The method of
0. 27. The method of
0. 28. The method of
0. 29. The method of
a) said invertible transform is an approximation of a biorthogonal transform;
b) said biorthogonal transformation comprises a representation as a cascade of at least one butterfly step, at least one orthogonal transform, and at least one scaling factor;
c) said at least one orthogonal transform comprising a cascade of
i) at least one ±1 butterfly step,
ii) at least one planar rotation, and
iii) at least one scaling factor;
b) said at least one planar rotation being represented by equivalent lifting steps and scale factors; and,
c) said approximation being obtained by replacing floating point coefficients in the lifting steps with rational coefficients.
0. 30. The method of
0. 31. The method of
0. 32. The method of
a) said invertible transform is an approximation of a unitary transform;
b) said approximation of the unitary transform comprises a representation of the unitary transform as a cascade of at least one butterfly step, at least one orthogonal transform, and at least one scale factor;
c) said at least one orthogonal transform being represented as a cascade of
(1) at least one ±1 butterfly steps,
(2) at least one planar rotation, and
(3) at last one scaling factor;
d) said at least one planar rotation being represented by equivalent lifting steps and scale factors; and,
e) said approximation being derived by using approximate rational values for the coefficients in the lifting steps.
0. 33. The method of
0. 34. The method of
0. 35. The method of
where a, b are selected from the group:
±{8, 5, 4, 2, 1, 1/2, 1/4, 3/4, 5/4, 1/8, 3/8, 2/5, 5/8, 7/8, 1/16, 3/16, 5/16, 7/16, 9/16, 11/16, 13/16, 15/16, 25/16}.
0. 36. The method of
0. 37. The method of
0. 38. The method of
0. 39. The method of
|
where Ui0, Ui1, Vi0, Vi1 are diagonalizing orthogonal matrices and Γi, Δi are diagonal matrices with positive elements.
It is well known that any M/2×M/2 orthogonal matrix can be factorized into M(M−2)/8 plane rotations θi and that the diagonal matrices represent simply scaling factors αi. Accordingly, the most general LT lattice consists of KM(M−2)/2 two dimensional rotations and 2M diagonal scaling factors αi. Any invertible matrix can be expressed as a sequence of pairwise plane rotations θ and scaling factors αi as shown in FIG. 3.
It is also well known that a plane rotation can be performed by 3 “shears”:
This can be easily verified by computation
Each of the factors above is capable of a “lifting” step in signal processing terminology. The product of two which effects a linear transform of pairs of coefficients:
The signal processing flow diagram of this operation is shown in FIG. 4. The crossing arrangement of these flow paths is also referred to as a butterfly configuration. Each of the above “shears” can be written as a lifting step.
Combining the foregoing, the shears referred to can be expressed as computationally equivalent “lifting steps” in signal processing. In other words, we can replace each “rotation” by 3 closely-related lifting steps with butterfly structure. It is possible therefore to implement the complete LT lattice shown in
In the simplest but currently preferred embodiment, to minimize the complexity of the transform we choose a small overlapping factor K=2 and set the initial stage E0 to be the DCT itself Many other coding transforms can serve for the base stage instead of the DCT, and it should be recognized that many other embodiments are possible and can be implemented by one skilled in the art of signal processing.
Following the observation in H. S. Malvar, “Lapped biorthogonal transforms for transform coding with reduced blocking and ringing artifacts,” ICASSP97, Munich, April 1997, we apply a scaling factor to the first DCT's antisymmetric basis to generate synthesis LT basis functions whose end values decay smoothly to exact zero—a crucial advantage in blocking artifacts elimination. However, instead of scaling the analysis by √{square root over (2)} and the synthesis by 1/√{square root over (2)}, we opt for 25/16 and its inverse 16/25 since they allow the implementation of both analysis and synthesis banks in integer arithmetic. Another value that works almost as well as 25/16 is 5/4. To summarize, the following choices are made in the first stage: the combination of U00 and V00 with the previous butterfly form the DCT;
and Γo=U00=V00=IM/2. See FIG. 2.
After 2 series of ±1 butterflies W and the delay chain Λ(z), the LT symmetric basis functions already have good attenuation, especially at DC (ω=0). Hence, we can comfortably set U1=IM/2.
As noted, V1 is factorizable into a series of lifting steps and diagonal scalings. However, there are several problems: (i) the large number of lifting steps is costly in both speed and physical real-estate in VLSI implementation; (ii) the lifting steps are related; (iii) and it is not immediately obvious what choices of rotation angles will result in dyadic rational lifting multipliers. In the current invention, we approximate V1 by (M/2)−1 combinations of block-diagonal predict-and-update lifting steps, i.e.,
Here, the free parameters ui and pi can be chosen arbitrarily and independently without affecting perfect reconstruction. The inverses are trivially obtained by switching the order and the sign of the lifting steps. Unlike popular lifting implementations of various wavelets, all of our lifting steps are of zero-order, namely operating in the same time epoch. In other words, we simply use a series of 2×2 upper or lower diagonal matrices to parameterize the invertible matrix V1.
Most importantly, fast-computable VLSI-friendly transforms are readily available when ui and pi are restricted to dyadic rational values, that is, rational fractions having (preferably small) powers of 2 denominators. With such coefficients, transform operations can for the most part be reduced to a small number of shifts and adds. In particular, setting all of the approximating lifting step coefficients to −1/2 yields a very fast and elegant lapped transform. With this choice, each lifting step can be implemented using only one simple bit shift and one addition.
The resulting LiftLT lattice structures are presented in
The inverse or synthesis lattice is shown in FIG. 6. This system comprises a set of four fast dyadic lifting steps 11, a butterfly-structured set of lifting steps 12, a delay line 13 on four of the eight channels, 16/25 inverse normalization 14, and an inverse DCT block 15.
The LiftLT is sufficiently fast for many applications, especially in hardware, since most of the incrementally added computation comes from the 2 butterflies and the 6 shift-and-add lifting steps. It is faster than the type-I fast LOT described in H. S. Malvar, Signal Processing with Lapped Transforms, Artech House, 1992. Besides its low complexity, the LiftLT possesses many characteristics of a high-performance transform in image compression: (i) it has high energy compaction due to a high coding gain and a low attenuation near DC where most of the image energy is concentrated; (ii) its synthesis basis functions also decay smoothly to zero, resulting in blocking-free reconstructed images.
Comparisons of complexity and performance between the LiftLT and other popular transforms are tabulated in Table 1 and Table 2. The LiftLT's performance is already very close to that of the optimal generalized lapped biorthogonal transform, while its complexity is the lowest amongst the transforms except for the DCT.
To assess the new method in image coding, we compared images coded and decoded with four different transforms:
Table 1 contains a comparison of the complexity of these four coding systems, comparing numbers of operations needed per 8 transform coefficients:
No.
Transform
Multiplications
No. Additions
No. Shifts
8 × 8 DCT
13
29
0
8 × 16 Type-I Fast LOT
22
54
0
917 Wavelet, 1-level
36
56
0
8 × 6 Fast LiftLT
14
51
6
In such a comparison, the number of multiplication operations dominates the “cost” of the transform in terms of computing resources and time, and number of additions and number of shifts have negligible effect. In this table, it is clear that the fast LiftLT is almost as low as the DCT in complexity and more than twice as efficient as the wavelet transform.
Table 2 sets forth a number of different performance measures for each of the four methods:
DC
Stopband
Coding
Atten.
Atten.
Mir. Freq.
Transform
Gain (dB)
(−dB)
(−dB)
Atten. (−dB)
8 × 8 DCT
8.83
310.62
9.96
322.1
8 × 16 Type-I Fast LOT
9.2
309.04
17.32
314.7
8 × 16 Optional LT
9.62
327.4
13.5
55.54
8 × 16 Fast LiftLT
9.54
312.56
13.21
304.85
The fast LiftLT is comparable to the optional 8×16 LT transform in coding gain and stopband attenuation an significantly better than the DCT.
Reconstructed images for a standard 512×512 “Barbara” test image at 1:32 compression ratio are shown in
Lena
Goldhill
Barbara
Comp.
9/7 WL
8 × 8
8 × 16
8 × 16
9/7 WL
8 × 8
8 × 16
8 × 16
9/7 WL
8 × 8
8 × 16
8 × 16
Ratio
SPIHT
DCT
LOT
LiftLT
SPIHT
DCT
LOT
LiftLT
SPIHT
DCT
LOT
LiftLT
8
40.41
39.91
40.02
40.21
36.55
36.25
36.56
36.56
36.41
36.31
37.22
37.57
16
37.21
36.38
36.69
37.11
33.13
32.76
33.12
33.22
31.4
31.11
32.52
32.82
32
34.11
32.9
33.49
34
30.56
30.07
30.52
30.63
27.58
27.28
28.71
28.93
64
31.1
29.67
30.43
30.9
28.48
27.93
28.34
28.54
24.86
24.58
25.66
25.93
100
29.35
27.8
28.59
29.03
27.38
26.65
27.08
27.28
23.76
23.42
24.32
24.5
128
28.38
26.91
27.6
28.12
26.73
26.01
26.46
26.7
23.35
22.68
23.36
23.47
PSNR is an acronym for power signal to noise r-atio and represents the logarithm of the ratio of maximum amplitude squared to the mean square error of the reconstructed signal expressed in decibels (dB).
The LiftLT outperforms its block transform relatives for all test images at all bit rates. Comparing to the wavelet transform, the LiftLT is quite competitive on smooth images—about 0.2 dB below on Lena. However, for more complex images such as Goldhill or Barbara, the LiftLT consistently surpasses the 9/7-tap wavelet. The PSNR improvement can reach as high as 1.5 dB.
Tran, Trac D., Topiwala, Pankaj
Patent | Priority | Assignee | Title |
10834400, | Aug 19 2016 | FastVDO LLC | Enhancements of the AV1 video codec |
10880557, | Jun 05 2015 | FastVDO LLC | High dynamic range image/video coding |
11265559, | Jun 05 2015 | FastVDO LLC | High dynamic range image/video coding |
7689052, | Oct 07 2005 | Microsoft Technology Licensing, LLC | Multimedia signal processing using fixed-point approximations of linear transforms |
7773671, | Sep 18 2001 | Microsoft Technology Licensing, LLC | Block transform and quantization for image and video coding |
7839928, | Sep 18 2001 | Microsoft Technology Licensing, LLC | Block transform and quantization for image and video coding |
7881371, | Sep 18 2001 | Microsoft Technology Licensing, LLC | Block transform and quantization for image and video coding |
8942289, | Feb 21 2007 | Microsoft Technology Licensing, LLC | Computational complexity and precision control in transform-based digital media codec |
8971405, | Sep 18 2001 | Microsoft Technology Licensing, LLC | Block transform and quantization for image and video coding |
9609336, | Apr 16 2013 | FastVDO LLC | Adaptive coding, transmission and efficient display of multimedia (acted) |
Patent | Priority | Assignee | Title |
5081645, | Aug 06 1990 | AWARE, INC | Novel spread spectrum codec apparatus and method |
5339265, | Aug 31 1992 | University of Maryland at College Park | Optimal unified architectures for the real-time computation of time-recursive discrete sinusoidal transforms |
5592569, | May 10 1993 | COMPETITIVE TECHNOLOGIES, INC | Method for encoding and decoding images |
5604824, | Sep 22 1994 | FOTO-WEAR, INC | Method and apparatus for compression and decompression of documents and the like using splines and spline-wavelets |
5764698, | Dec 30 1993 | MEDIATEK INC | Method and apparatus for efficient compression of high quality digital audio |
5805739, | Apr 02 1996 | Polycom, Inc | Lapped orthogonal vector quantization |
5812219, | Nov 09 1994 | FUNAI ELECTRIC CO , LTD | System and method for transmitting picture or sound signals |
5857036, | Mar 04 1996 | HEWLETT-PACKARD DEVELOPMENT COMPANY, L P | System and method for the fractal encoding of datastreams |
5859788, | Aug 15 1997 | The Aerospace Corporation | Modulated lapped transform method |
5883981, | Oct 18 1996 | S AQUA SEMICONDUCTOR, LLC | Lattice vector transform coding method for image and video compression |
5898798, | Oct 18 1995 | U S PHILIPS CORPORATION | Region-based texture coding and decoding method and corresponding systems |
5901251, | Mar 18 1997 | Hewlett-Packard Company | Arithmetic coding compressor using a context model that is adaptive to variable length patterns in bi-level image data |
5903669, | Jan 31 1995 | Canon Kabushiki Kaisha | Image processing apparatus and method |
5946038, | Feb 27 1996 | FUNAI ELECTRIC CO , LTD | Method and arrangement for coding and decoding signals |
5960123, | Jul 27 1995 | FUJIFILM Corporation | Method and apparatus for enhancing contrast in images by emphasis processing of a multiresolution frequency band |
5973755, | Apr 04 1997 | Microsoft Technology Licensing, LLC | Video encoder and decoder using bilinear motion compensation and lapped orthogonal transforms |
5995668, | Oct 25 1995 | IPG Electronics 503 Limited | Segmented picture coding method and system, and corresponding decoding method and system |
5999656, | Jan 17 1997 | RICOH CO LTD , HAVING A PLACE OF BUSINESS IN TOKYO, JAPAN; Ricoh Corporation | Overlapped reversible transforms for unified lossless/lossy compression |
6018753, | Jul 29 1997 | WSOU Investments, LLC | Interpolating filter banks in arbitrary dimensions |
6094631, | Jul 09 1998 | Winbond Electronics Corp. | Method of signal compression |
6104982, | Dec 01 1995 | Schlumberger Technology Corporation | Compression method and apparatus for seismic data |
6144771, | Jun 28 1996 | HANGER SOLUTIONS, LLC | Method and apparatus for encoding and decoding images |
6144773, | Feb 27 1996 | Vulcan Patents LLC | Wavelet-based data compression |
6198412, | Jan 20 1999 | WSOU Investments, LLC | Method and apparatus for reduced complexity entropy coding |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jul 29 2003 | Fast VDO LLC | (assignment on the face of the patent) | / | |||
Aug 16 2007 | PT-IP, LLC | FastVDO, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 019747 | /0027 | |
Sep 12 2014 | FastVDO LLC | RPX Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 038249 | /0458 |
Date | Maintenance Fee Events |
Nov 17 2009 | M2552: Payment of Maintenance Fee, 8th Yr, Small Entity. |
Feb 21 2014 | REM: Maintenance Fee Reminder Mailed. |
Jul 16 2014 | EXPX: Patent Reinstated After Maintenance Fee Payment Confirmed. |
Oct 22 2014 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Oct 22 2014 | M1558: Surcharge, Petition to Accept Pymt After Exp, Unintentional. |
Oct 22 2014 | PMFP: Petition Related to Maintenance Fees Filed. |
May 11 2016 | STOL: Pat Hldr no Longer Claims Small Ent Stat |
May 31 2016 | PMFG: Petition Related to Maintenance Fees Granted. |
Date | Maintenance Schedule |
Feb 19 2011 | 4 years fee payment window open |
Aug 19 2011 | 6 months grace period start (w surcharge) |
Feb 19 2012 | patent expiry (for year 4) |
Feb 19 2014 | 2 years to revive unintentionally abandoned end. (for year 4) |
Feb 19 2015 | 8 years fee payment window open |
Aug 19 2015 | 6 months grace period start (w surcharge) |
Feb 19 2016 | patent expiry (for year 8) |
Feb 19 2018 | 2 years to revive unintentionally abandoned end. (for year 8) |
Feb 19 2019 | 12 years fee payment window open |
Aug 19 2019 | 6 months grace period start (w surcharge) |
Feb 19 2020 | patent expiry (for year 12) |
Feb 19 2022 | 2 years to revive unintentionally abandoned end. (for year 12) |