Techniques and tools for high dynamic range (“HDR”) image generation and rendering are described herein. In several described embodiments, images having distinct exposure levels are aligned. In particular embodiments, the alignment of a reference image to a non-reference image is based at least in part on motion vectors that are determined using covariance computations. Furthermore, in certain embodiments, saturated areas, underexposed areas, and/or moving objects are ignored or substantially ignored during the image alignment process. Moreover, in certain embodiments, a hierarchical pyramid block-based scheme is used to perform local motion estimation between the reference image and the non-reference image.
|
17. A digital image processing method for generating HDR images, the method comprising:
by computing hardware:
inputting a reference image, the reference image comprising a digital image at a first exposure level and at a first time;
inputting a non-reference image, the reference image comprising a digital image at a second exposure level and at a second time, the second exposure level being different than the first exposure level, and the second time being different than the first time; and
computing a local motion vector for a block of pixel sample values in the non-reference image indicating relative motion of the block to a corresponding block of pixel sample values in the reference image, the local motion vector being determined at least in part using a metric measuring how the pixel sample values in the block in the non-reference image change together with the pixel sample values in the block in the reference image.
8. A digital image processing device for generating HDR images, the digital image processing device comprising:
a memory storing an image alignment program;
at least one digital signal processing component;
a processing unit operable to execute the image alignment program, wherein said execution of the image alignment program causes the processing unit to:
input a reference image, the reference image comprising a digital image at a first exposure level and at a first time;
input a non-reference image, the reference image comprising a digital image at a second exposure level and at a second time, the second exposure level being different than the first exposure level, and the second time being different than the first time; and
compute a local motion vector for a block of pixel sample values in the non-reference image indicating relative motion of the block to a corresponding block of pixel sample values in the reference image, the local motion vector being determined at least in part using a metric measuring how the pixel sample values in the block in the non-reference image change together with the pixel sample values in the block in the reference image.
10. A method, the method comprising:
by computing hardware:
receiving a reference image, the reference image comprising a digital image at a first exposure level and at a first time;
receiving a non-reference image, the reference image comprising a digital image at a second exposure level and at a second time, the second exposure level being different than the first exposure level, and the second time being different than the first time;
estimating two or more local motion vectors for the non-reference image relative to the reference image, the estimating of the two or more local motion vectors comprising:
downsampling the non-reference image and the reference image one or more times to form a plurality of versions of the non-reference image and the reference image at a plurality of resolutions, wherein a lowest resolution version of the non-reference image has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of the non-reference image, and wherein a lowest resolution version of the reference image has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of the reference image;
for the lowest resolution version of the non-reference image, estimating motion vectors of the blocks of the non-reference image by performing a search for motion vectors in the associated lowest resolution version of the reference image, wherein the search comprises searching for matching blocks at nearby block locations then expanding the search; and
for each successively higher resolution version of the non-reference image, estimating motion vectors of the blocks of the successively higher resolution version of the non-reference image using a refinement search starting from the motion vector estimated for the associated block of the preceding lower resolution version of the non-reference image according to the pyramid structure.
1. One or more non-transitory computer-readable media storing computer-executable instructions which when executed by a computer cause the computer to perform a method, the method comprising:
receiving a reference image, the reference image comprising a digital image at a first exposure level and at a first time;
receiving a non-reference image, the reference image comprising a digital image at a second exposure level and at a second time, the second exposure level being different than the first exposure level, and the second time being different than the first time;
estimating two or more local motion vectors for the non-reference image relative to the reference image, the estimating of the two or more local motion vectors comprising:
downsampling the non-reference image and the reference image one or more times to form a plurality of versions of the non-reference image and the reference image at a plurality of resolutions, wherein a lowest resolution version of the non-reference image has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of the non-reference image, and wherein a lowest resolution version of the reference image has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of the reference image;
for the lowest resolution version of the non-reference image, estimating motion vectors of the blocks of the non-reference image by performing a search for motion vectors in the associated lowest resolution version of the reference image, wherein the search comprises searching for matching blocks at nearby block locations then expanding the search; and
for each successively higher resolution version of the non-reference image, estimating motion vectors of the blocks of the successively higher resolution version of the non-reference image using a refinement search starting from the motion vector estimated for the associated block of the preceding lower resolution version of the non-reference image according to the pyramid structure.
2. The one or more computer-readable media of
3. The one or more computer-readable media of
4. The one or more computer-readable media of
5. The one or more computer-readable media of
6. The one or more computer-readable media of
7. The one or more computer-readable media of
9. The digital image processing device of
11. The method of
12. The method of
13. The method of
14. The method of
15. The method of
16. The method of
18. The digital image processing method of
|
The technology relates to generating and rendering digital images, and more particularly to generating and rendering high dynamic range (“HDR”) digital images.
In digital imaging systems, color generally is represented by coordinates in a multi-dimensional “color space” having three or more channels. Common examples include the well-known RGB and YUV color spaces. In the RGB color space, pixels are specified using coordinates (also referred to as pixel sample values or sample values) that represent intensities of red, green and blue light, respectively. In the YUV color space, pixels are specified using coordinates (pixel sample values or sample values) that represent a luminance value and two chrominance values.
Currently, many image capture, processing, and display devices can only handle pixel sample values having a small dynamic range of 256 (28) discrete values per channel, represented by 8 bits. Such images can be described as having a “bit depth” of 8 bits. In a typical RGB digital image having 8 bits per channel (8 bpc) in red, green and blue color channels, only 256 different values are possible for each of the red, green and blue pixel sample values. Other values (e.g., alpha or opacity values, luminance values, and other such values) also may be constrained by low dynamic range limitations. Some devices can handle up to a 10- or 12-bit dynamic range per channel. However, the human vision system can detect a wide luminance range of 14 orders of magnitude, which translates to around 46 bits. Luminance values in nature can be as high as 108 candela/m2 in bright sunlight, and as low as 10−6 candela/m2 on the underside of a rock on a moonless night.
High dynamic range (“HDR”) imaging presents a more versatile and natural image representation in line with the human vision system. HDR images can present a dynamic range higher than the traditional 8-bit, 10-bit and 12-bit representations to achieve a far higher image quality. HDR images can be used in the same kinds of devices and software tools that process conventional images if the HDR image format is compatible with the device or tool. Several HDR image formats have been developed, and cameras, computer graphics, and display devices have begun to produce, process and display images with increasing dynamic ranges.
Techniques and tools for high dynamic range (“HDR”) image generation and rendering are described herein. In certain embodiments, HDR images are constructed from a collection of lower dynamic range images. For example, HDR images can be generated by merging images taken at different exposures from the same camera for the same scene. However, due to hand shaking or environmental conditions, some camera motion (typically rotational and/or translational) should be expected between each picture. When HDR images are constructed from lower dynamic range images, aligning the images is one aspect of the HDR image generation process. This alignment process can be challenging, however, due to the differences between the images caused by the camera motion and caused by the different exposures at which the various images are taken. Thus, among the several described embodiments are techniques and tools for aligning images having distinct exposures. In particular embodiments, the alignment of the images is based at least in part on motion vectors that are determined using covariance computations between the images. Furthermore, in certain embodiments, saturated areas, underexposed areas, and/or moving objects with local motions are ignored or substantially ignored during the image alignment process. For example, in certain implementations, a Random Sample Consensus (“RANSAC”) method is used to detect and eliminate motion vectors that are outliers due to a moving object, saturation, and/or under exposure.
In some embodiments of the disclosed technology, knowledge of an image capture device's response curve is not needed in advance of the image generation. Furthermore, embodiments of the disclosed technology do not normalize the images prior to alignment, and therefore preserve more information to align images more accurately. Embodiments of the disclosed technology can be used to correct both translational, rotational, or both translation and rotational motion. Further, embodiments of the disclosed technology are more computationally efficient than previous HDR generation techniques.
In one exemplary embodiment disclosed herein, motion analysis is performed between a reference image and a set of one or more non-reference images. The motion analysis can comprise computing one or more covariance values between a set of pixel sample values in the reference image and a set of pixel sample values in one of the non-reference images from the set. Further, based at least in part on the motion analysis, at least the one of the non-reference images from the set can be merged with the reference image to form a higher dynamic range digital image.
In another embodiment disclosed herein, a reference image is received, the reference image comprising a digital image at a first exposure level and at a first time. A non-reference image is also received, the reference image comprising a digital image at a second exposure level and at a second time, the second exposure level being different than the first exposure level, and the second time being different than the first time. Two or more local motion vectors are estimated for the non-reference image relative to the reference image. In particular implementations, the estimating of the two or more local motion vectors comprises downsampling the non-reference image and the reference image one or more times to form a plurality of versions of the non-reference image and the reference image at a plurality of resolutions, wherein a lowest resolution version of the non-reference image has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of the non-reference image, and wherein a lowest resolution version of the reference image has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of the reference image. Further, for the lowest resolution version of the non-reference image, motion vectors of the blocks of the non-reference image can be estimated by performing a full search for motion vectors in the associated lowest resolution version of the reference image; and for each successively higher resolution version of the non-reference image, motion vectors of the blocks of the successively higher resolution version of the non-reference image can be estimated using a refinement search starting from the motion vector estimated for the associated block of the preceding lower resolution version of the non-reference image according to the pyramid structure.
Another embodiment disclosed herein comprises a memory storing an image alignment program, at least one digital signal processing component, and a processing unit operable to execute the image alignment program. In certain embodiments, the image alignment program causes the processing unit to input a reference image, the reference image comprising a digital image at a first exposure level and at a first time; input a non-reference image, the reference image comprising a digital image at a second exposure level and at a second time; and compute a local motion vector for a block of pixel sample values in the non-reference image indicating relative motion of the block to a corresponding block of pixel sample values in the reference image. In particular implementations, the local motion vector is determined at least in part using a metric measuring how the pixel sample values in the block in the non-reference image change together with the pixel sample values in the block in the reference image.
This summary is provided to introduce a selection of concepts in a simplified form that is further described below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. Additional features and advantages of the disclosed technology will be made apparent from the following detailed description of embodiments that proceeds with reference to the accompanying drawings.
I. General Considerations
Disclosed below are representative embodiments of methods, apparatus, and systems for performing HDR image processing, including the aligning of digital images having different exposures. The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Instead, the present disclosure is directed toward all novel and nonobvious features and aspects of the various disclosed embodiments, alone and in various combinations and subcombinations with one another. Furthermore, any features or aspects of the disclosed embodiments can be used alone or in various combinations and subcombinations with one another. The disclosed methods, apparatus, and systems are not limited to any specific aspect or feature or combination thereof, nor do the disclosed embodiments require that any one or more specific advantages be present or problems be solved.
Although the operations of some of the disclosed methods are described in a particular, sequential order for convenient presentation, it should be understood that this manner of description encompasses rearrangement, unless a particular ordering is required by specific language set forth below. For example, operations described sequentially may in some cases be rearranged or performed concurrently. Moreover, for the sake of simplicity, the attached figures may not show the various ways in which the disclosed methods, apparatus, and systems can be used in conjunction with other methods, apparatus, and systems. Furthermore, as used herein, the term “and/or” means any one item or combination of items in the phrase.
The disclosed methods can be implemented using computer-executable instructions stored on one or more computer-readable media (e.g., non-transitory computer-readable media, such as one or more optical media discs, volatile memory components (e.g., DRAM or SRAM), or nonvolatile memory or storage components (e.g., hard drives)) and executed on a computer (e.g., any commercially available computer or a computer or image processor embedded in a device, such as a laptop computer, entertainment console, net book, web book, tablet computing device, smart phone, or other mobile computing device). Any of the intermediate or final data created and used during implementation of the disclosed methods or systems can also be stored on one or more computer-readable media (e.g., non-transitory computer-readable media) and are considered to be within the scope of the disclosed technology.
For clarity, only certain selected aspects of the software-based embodiments are described. Other details that are well known in the art are omitted. For example, it should be understood that the software-based embodiments are not limited to any specific computer language or program Likewise, embodiments of the disclosed technology are not limited to any particular computer or type of hardware. Exemplary computing environments suitable for performing any of the disclosed software-based methods are introduced below.
The disclosed methods can also be implemented using specialized computing hardware that is configured to perform any of the disclosed methods. For example, the disclosed methods can be implemented by an integrated circuit (e.g., an application specific integrated circuit (“ASIC”) (such an ASIC digital signal process unit (“DSP”), a graphics processing unit (“GPU”), or programmable logic device (“PLD”), such as a field programmable gate array (“FPGA”)) specially designed or configured to implement any of the disclosed methods (e.g., dedicated hardware configured to perform any of the disclosed image processing techniques).
The disclosed HDR image generation, HDRimage rendering, or digital image alignment techniques can also be used in a variety of usage and computation scenarios, including image processing on a dedicated image capture device (e.g., a digital camera or a mobile device with a digital camer), image processing on a stand-alone computer, image processing on a network client computer, or image processing on a server computer. Further, various parts of the disclosed HDR image generation, HDR image rendering, or digital image alignment techniques technique can be performed in parallel or cooperatively on multiple computing devices, such as in a client/server, network “cloud” service or peer computing arrangement, among others. Accordingly, it should be recognized that the techniques can be realized on a variety of different electronic and computing devices, including both end user consumer operated devices as well as server computers that may provide the techniques as part of a service offering to customers.
II. Example Computing Environments
With reference to
In addition to the central processing unit 110, the computing environment can include other processing resources, such as digital signal processing DSP or multimedia components 115. The DSP components 115 can be any of the resources that can be utilized advantageously for the digital image alignment techniques disclosed herein. For example, the DSP components can include multimedia DSP ASIC units, GPU shader units, a multicore CPU, advanced multimedia instruction sets for the CPU, or the like.
The computing hardware environment can have additional features. For example, the computing hardware environment 100 includes a storage device 140, one or more input devices 150, one or more output devices 160, and one or more communication connections 170. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing hardware environment 100. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing hardware environment 100, and coordinates activities of the components of the computing hardware environment 100.
The storage device 140 is a type of non-volatile memory and can be removable or non-removable. The storage device 140 includes, for instance, non-transitory computer readable media such as magnetic disks (e.g., hard drives), magnetic tapes or cassettes, optical storage media (e.g., CD-ROMs or DVDs), or any other tangible non-transitory storage medium that can be used to store information and which can be accessed within or by the computing hardware environment 100. The storage device 140 can also store the software 180 for implementing any of the described techniques.
The input device(s) 150 can be a touch input device such as a keyboard, mouse, touch screen, pen, trackball, a voice input device, a scanning device, or another device that provides input to the computing environment 100 (e.g., an image sensing device). The output device(s) 160 can be a display device, touch screen, printer, speaker, or another device that provides output from the computing environment 100.
The communication connection(s) 170 enable communication over a communication medium to another computing entity. The communication medium conveys information such as computer-executable instructions, any of the intermediate or final messages or data used in implementing embodiments of the disclosed technology. By way of example, and not limitation, communication media include wired or wireless techniques implemented with an electrical, optical, RF, infrared, acoustic, or other carrier.
The various methods disclosed herein (e.g., any of the disclosed HDR image processing or image alignment techniques) can be described in the general context of computer-executable instructions stored on one or more computer-readable storage media (e.g., tangible non-transitory computer-readable storage media such as memory 120 and storage 140). As should be readily understood, the terms computer-readable storage media or non-transitory computer-readable media includes the media for storage of data and program instructions such as memory 120 and storage 140, and not modulated data signals alone.
The various methods disclosed herein can also be described in the general context of computer-executable instructions, such as those included in program modules, being executed by a processor in a computing environment. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, and other such software elements that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. Any of the disclosed methods can also be performed using a distributed computing environment (e.g., a client-server network, cloud computing environment, wide area network, or local area network).
In example cloud computing network environment 200, various types of computing services for HDR image sharing, generation, storage or distribution (e.g., image sharing, generation, or social networking sites) are provided by a collection of network-accessible computing and storage resources, referred to as the cloud 210. For example, the cloud 210 can comprise a collection of server computing devices, which may be located centrally or at distributed locations, that provide cloud-based services to various types of users and devices connected via a network, such as the Internet.
In example environment 200, the cloud 210 provides services 220 (such as HDR image storage, HDR image generation, HDR image rendering, image alignment, HDR image sharing, or social networking services, among other examples) for user computing devices. Services can be provided in the cloud 210 through cloud computing service providers, or through other providers of online services. For example, the cloud-based services 220 can include an HDR image generation service, an HDR image rendering service, an image alignment service, an image storage service, an image sharing site, a social networking site, or other services via which user-sourced images are distributed for viewing by others on connected devices 220A-N.
The user may use various image capture devices to capture one or more images (e.g., images at different exposures or HDR images), such as digital cameras, mobile phones, and handheld computing devices. The user can upload the one or more digital images to the service 220 on the cloud 210 either directly (e.g., using a data transmission service of a telecommunications network) or by first transferring the one or more images to a local computer 230, such as a laptop, personal computer or other network connected computing device.
As shown in
Connected device 220A represents a device with a mid-size display screen, such as may be available on a personal computer, a laptop, a tablet or other like network connected devices. Connected device 220B represents a device with display screen with form factors designed to be highly portable (e.g., a small size screen). For example, connected device 220B could be a mobile phone, smart phone, personal digital assistant, and the like. Connected device 220N represents a connected device with a large viewing screen. For example, connected device 220N could be a television screen (e.g., a smart television) or another device that provides image output to a television or an image projector (e.g., a set-top box or gaming console), or other devices with like image display output.
In the illustrated cloud-computing network environment 200, any of the HDR processing and/or image alignment techniques disclosed herein can be implemented and performed at various stages of the image sharing, storage and distribution, and by various ones of the depicted devices depending on the desired use scenario. In one example scenario, any of the HDR processing and/or image alignment techniques is implemented in software on the local computer 230, and applied when images are either initially transferred to the local computer or when uploaded to the cloud-based service 220. In another scenario, any of the HDR processing and/or image alignment techniques is implemented in the cloud, and applied to images as they are uploaded to and stored in the cloud. In another scenario, any of the HDR processing and/or image alignment techniques is implemented by cloud computing services and applied when the images are accessed, transferred, or distributed to another connected device or service. In still other scenarios, any of the HDR processing and/or image alignment techniques is performed on the connected device after images are downloaded or otherwise accessed from the cloud 210.
III. High Dynamic Range Imaging-Overview
Although the human visual system is capable of detecting details in a very wide dynamic range, especially in terms of brightness, humans do not normally pick up details across the full breadth of this dynamic range all at once. For example, it is difficult for humans to pick up details in very bright areas and very dark areas simultaneously. However, by focusing on various local areas in a scene, the human visual system can adapt very quickly to brightness in local areas in order to see details in bright areas (e.g. sky) as well as darker areas (e.g., shadows).
Many existing digital cameras (including most DSLR (digital single lens reflex) cameras) lack direct HDR image capture capabilities and are only capable of delivering each shot from one single perspective at a standard dynamic range. So, there is a large gap between the level of detail that humans can perceive and the level of detail that can be represented in SDR images. A viable solution to the problem is High-Dynamic-Range (HDR) imaging. An HDR image can capture rich information in a scene, and when rendered properly can deliver a very realistic visual experience. Due to the non-linearity and local-adaptive nature of the human visual system, most of the HDR imaging experience can be achieved with normal printouts or existing displays, making the market for HDR images potentially very large. HDR imaging technologies have therefore drawn more attention recently from users, including professional photographers and hobbyists.
A. Exemplary HDR Image Formats
Digital images described herein can be color, grey-scale, or other types of images, and can be represented in a variety of file formats (e.g., GIF, PNG, BMP, TIFF, TIFF Float32, JP2, HDR, OpenEXR, JPEG XR, Radiance RGBE and/or other formats). This section provides details about some image formats that can be used with described techniques and tools for generating HDR images, rendering HDR images, and/or aligning digital images. For example, described techniques and tools can handle HDR images in a JPEG XR format.
JPEG XR is a flexible and powerful format that is useful for encoding HDR images, supporting up to 32-bit floating point numbers. JPEG XR supports HDR image coding while requiring only integer operations (with no divides) for both compression and decompression. JPEG XR is compatible with a file container to store image data and metadata, which can include HD Photo metadata, XMP metadata, and Exif metadata through IFD tags. Compared to existing HDR formats, such as TIFF Float, OpenEXR, and Radiance RGBE, the JPEG XR format provides much higher compression capability while preserving reasonable image quality. The JPEG XR file format is extensible, so additional metadata (either proprietary or standard) can be inserted in JPEG XR files without changing image bit streams.
Radiance RGBE is another format for carrying HDR image data. In an RGBE image, each pixel is represented by 32 bits: one group of bits (e.g., one byte) for a red mantissa (R), one group of bits (e.g., one byte) for a green mantissa (G), one group of bits (e.g., one byte) for a blue mantissa (B), and the remaining bits (e.g., one byte) for a common exponent (E) that is applied to the values represented by the mantissa for each of the R, G and B channels. In RGBE, effective pixel values (fR, fG, fB) are floating point numbers, where fR=R*2(E−128); fG=G*2(E−128); and fB=B*2(E-128). An RGBE rule restricts the largest 8-bit mantissa within the range [128, 255], while the other two 8-bit mantissas are unrestricted (the range is [0, 255]). So, the mapping from RGB to RGBE is unique. The dynamic range that RGBE can present is [2−127, 2+127], which is approximately 76 orders of magnitude. The RGB value reconstructed from an RGBE pixel is unsigned—all nonzero values are positive. However, since the three color components share the same exponent, the precision of the two smaller components is sacrificed. E=0 is a special case, indicating that the corresponding pixel value is zero.
One variant of Radiance RGBE (9:9:9:5) assigns 9 bits each to the red, green and blue mantissa channels, and 5 bits to the exponent, thus representing each pixel in 32 bits as in the 8:8:8:8 format described above. Other representations of RGBE also are possible, with different numbers of bits assigned to the mantissas and the exponent.
32-bit floating point (“32-bit float”) is commonly used in representing floating point image data. Container formats defining 32-bit floating point images include Portable Float Map (“PFM”) and Tagged Image File Format (“TIFF”). The IEEE 754 32-bit single-precision float number includes 1 bit for the sign (s), 8 bits for the exponent (e) and 23 bits for the mantissa (m). 16-bit floats (also referred to as “half”) have one sign bit, five exponent bits and ten mantissa bits. Since the 16-bit and 32-bit floating point representations are structurally identical except for specific differences in lengths of fields, they can be referred to generically as “floating point.”
Some image formats are specified by international standards. For example, the JPEG and JPEG2000 standards set forth requirements for decoders to decode images encoded in JPEG and JPEG2000 format, respectively. A JPEG2000-compliant encoder and decoder (“codec”) provides high-quality images with good compression efficiency. The JPEG XR standard was developed from Windows Media Photo and HD Photo, proprietary image compression formats developed by Microsoft Corporation as part of the Windows Media family of technologies. JPEG XR is described in international standard ISO/IEC 29199-2:2009.
The disclosed technology can be adapted and used to generate HDR images in any of the disclosed formats.
IV. High Dynamic Range Image Generation and Rendering
A. Generalized HDR Imaging System
In examples described herein, SDR images are images with 8 bits per color channel (bpc), and HDR images are 16 bpc images, 32 bpc, or more. More generally, “standard dynamic range” or SDR refers to a display, image, or format with a narrower or lower dynamic range than an HDR image. (SDR images also can be referred to as lower dynamic range, or LDR images.) Similarly, “high dynamic range” or HDR refers to a display, image, or format with a wider or higher dynamic range than an SDR image. For example, described embodiments can be used to generate 32 bpc HDR images from 8 bpc SDR images, or for rendering 32 bpc HDR images for display on a 16 bpc or 8 bpc monitor.
In general, digital images described herein can be captured by or stored on any device capable of capturing digital images (e.g., digital still image cameras, digital video cameras, scanners, or multipurpose devices with image capture capability such as camera phones) or any medium capable of storing digital images (e.g., volatile memory or nonvolatile memory).
In the example shown in
The relationships shown between modules within the system 410 indicate general flows of information in the system; other relationships are not shown for the sake of simplicity. Particular embodiments typically use a variation or supplemented version of the generalized system 410. Depending on the implementation and the type of processing desired, modules of the system can be added, omitted, split into multiple modules, combined with other modules, and/or replaced with like modules. In alternative embodiments, systems with additional or different modules and/or other configurations of modules perform one or more of the described techniques. For example, the system 410 can include a pre-processor that smoothes input digital images using a lowpass filter or other filter so as to selectively remove high-frequency components. Or, the pre-processor performs other pre-processing tasks.
The system 410 can also include one or more encoders that compress image data, outputting a bit stream of compressed digital image information. The exact operations performed by the encoder can vary depending on the compression format. For example, an encoder can compress the HDR image information according to an HDR image format, such as JPEG XR, or an encoder can compress SDR image information according to a format, such as JPEG. The system 410 also can include one or more decoders that decompress compressed image data, outputting a bit stream of decompressed digital image information. The exact operations performed by the decoder can vary depending on the compression format. For example, a JPEG XR-compliant decoder can decompress HDR image information in a JPEG XR file, or a JPEG-compliant decoder can decompress SDR image information in a JPEG file.
B. Generalized HDR Generation and Rendering
In HDR image rendering 520, HDR image information is converted into another form. For example, an HDR image can be rendered as an SDR image (e.g., a JPEG image) and displayed on an SDR display. In certain embodiments, HDR image rendering 520 is performed using the method disclosed below with respect to
Although the exemplary flow chart 500 shows HDR image generation 510 followed by HDR image rendering 520, an HDR image processing system can generate HDR images without rendering them, such as when HDR images are generated for storage or for further processing, but are not displayed. An HDR image processing system also can render HDR images rendering without generating them, such as when the HDR image processing system obtains HDR image information (e.g., via a communication medium) from some other source without first generating the HDR image.
C. Exemplary Methods for High Dynamic Range Image Generation
At 610, LDR image information is input. In the illustrated embodiment, the set of LDR images comprises low exposure image 612, mid exposure image 614, high exposure image 616, which are images of a scene with one or more different exposure settings that will produce a higher dynamic range image 690 when the images are processed and integrated. For example, LDR images 612, 614, 616, are images of a scene taken at different time instances with different exposure settings.
At 620, image alignment is performed. For example, when SDR images 612, 614, 616 represent the visual state of a scene at different time instances, motion that occurs in the sequence of images is analyzed at 620 and the images are aligned so as to allow the images to be merged in order to form an HDR image. When images of an object or scene are taken at different time instances, there is very likely to be image motion (typically, rotational, translational, or zooming) between the different shots of the object or scene. The image alignment performed at 620 is performed in order to quantify the movement so that such movement can be considered when performing further processing.
An exemplary method for performing image alignment is discussed in detail below with reference to
In particular embodiments, global motion analysis is performed as part of the image alignment process. For instance, the global motion analysis method discussed below with respect to
Global motion analysis generally refers to analysis that aims to model motion such as panning, shudder, rotation, zoom, or other movements that affect the entire image. Global motion analysis can involve looking for matching pixel sample values in pixel locations or groups of pixel locations (e.g., blocks) that are consistent with particular kinds of global motion. For example, looking for panning motion in a non-reference image can involve searching for matching pixel sample values (or groups of pixel sample values, such as blocks) in a predominantly horizontal direction in a reference image. Local motion analysis refers to analysis that aims to model movement in individual parts of the image, such as movement of objects or human subjects.
After detecting motion from a particular pixel location (or group of pixel locations, such as a block) to another, the image alignment process involves quantifying the motion. In particular implementations, motion vectors are computed for pixel samples or groups of pixels (e.g., blocks) in an image, relative to a reference image. For example, a motion vector can indicate a displacement in the horizontal dimension and a displacement in the vertical dimension, relative to a reference image. A motion vector of zero can be used to indicate that no motion has been detected for a particular pixel or group of pixels (e.g., a block).
In example implementations, unusually low pixel sample values or unusually high pixel sample values are not considered in the motion analysis because they are often not reliable; for example, the high or low value of the pixel sample value may be due to image noise, or the value may not be an accurate measurement due to dynamic range limitations of an image sensor in the image capture device. The range of acceptable pixel sample values for use in motion analysis, and the way in which the range is determined, can vary. For example, a range of acceptable pixel sample values for use in motion analysis can be predetermined for all images, or determined adaptively for each image or input image set (e.g., based on image information and/or user preferences).
At 630, the images are converted into a linear space. The images 612, 614, 616 are typically in a non-linear space (e.g., RGB), and there is no reliable way to merge them. In some embodiments, the images are converted into a linear space (e.g., into a linear radiance space) using the ICC profile of each image. For example, a gamma-to-linear converter can be used to convert input image signals that are not in a linear domain into a linear domain. Data in still image files (e.g., JPEG files) typically include gamma-encoded values. Thus, in one implementation, RGB samples are converted to the linear domain according to a color profile (e.g., an ICC profile) carried in each image.
At 640, the images are normalized. For example, in certain embodiments, the images are normalized (or scaled) based on their relative exposure levels. In general, normalization involves increasing the pixel sample values of the low exposure image and decreasing the pixel sample values of the high exposure image in order to prevent the high exposure image from dominating the final computations involved in HDR image generation. In certain embodiments, the pixel sample values of the non-reference images (e.g., the pixel sample values of the low exposure image 612 and the high exposure image 616 are multiplied or divided by a scaling factor that corresponds to the relative exposure level for the respective image). In such embodiments, the scaling factor transforms the pixel sample value to a level normalized with the reference image (e.g., the mid exposure image 614). In some embodiments, the scaling operations are applied to the linear data output from method act 630, with the same scaling factor applied across R, G and B pixel sample values of an image. Alternatively, scaling is performed before image alignment, and the motion analysis associated with the image alignment is performed on the scaled pictures. In certain implementations, two normalization ratios (or scaling factors) are computed, a first normalization ratio for normalizing the low exposure image 612 to the mid exposure image 614 and a second normalization ratio for normalizing the high exposure image 616 to the mid exposure image 614. In one particular implementation, the first ratio (ratioLow) is computed by dividing the average mid exposure image pixel sample values (e.g., the average of all R, G, and B values for the mid exposure image) by the average low exposure image pixel sample values (e.g., the average of all R, C, and B values for the low exposure image)). Thus, ratioLow=average(mid exposure)/average(low exposure). Similarly, the second ratio (ratioHigh) is computed by dividing the average mid exposure image pixel sample values (e.g., the average of all R, G, and B values for the mid exposure image) by the average high exposure image pixel sample values (e.g., the average of all R, G, and B values for the high exposure image). Thus, ratioHigh=average(mid exposure)/average(high exposure)). To complete the image normalization process, the RGB pixel sample values of the linear space low exposure image are multiplied by the ratioLow value, and the RGB pixel sample values of the linear space high exposure image are multiplied by the ratioHigh value (which is typically less than 1, and thus can be computationally performed as a division). Although the exemplary embodiment averages R, G, and B values together, alternative embodiments compute different averages for each of the R, G, and B values, resulting in multiple scaling factors.
At 650, the images are merged. In particular, when the images 612, 614, 616 are input, there are multiple sample values for each pixel position in the image. Thus, at 650, single pixel sample values (e.g., a single R pixel sample value, a single G pixel sample value, and a single G pixel sample value) are computed for pixel positions in the HDR image from the multiple pixel sample values at corresponding positions in the images 612, 614, 616. The pixel sample values used during the merging process at 650 can be the pixel sample values of the images 612, 614, 616 after alignment (e.g., after applying the global motion transform computed at 620) and after the pixel values are converted into linear space and normalized (e.g., at 630 and 640).
In one implementation, a weighted average of the pixel sample values is computed for each pixel position (e.g. a weighted arithmetic mean). The weighting factor applied to the pixel sample values in the weighted average (e.g., the weighting factor applied to the pixel sample value of the low exposure image at the pixel position under consideration, the weighting factor applied to the pixel sample value of the mid exposure image at the pixel position under consideration, and the weighting factor applied to the pixel sample value of the high exposure image at the pixel position under consideration) is based on whether the original pixel value before linear conversation and normalization is reliable, In general, the weighting factor is proportional to the confidence level that the pixel value is reliable: the higher the confidence level, the higher the weighting factor, For example, pixel sample values with relatively high values (e.g., over 240) in the high exposure image are typically unreliable, as are pixel sample values with relatively low values (e.g., under 15) in the low exposure image. Such values correspond to areas of the image that are dark or saturated.
The weight applied to a particular pixel sample value can be determined using a variety of techniques but in certain embodiments is computed using a hat function. An example of a suitable hat function is shown in Table 1, The example hat function in Table 1 produces a weighting factor (wt_lut8) that is dependent on the input pixel value (i). The hat function shown in Table 1 produces weighting factors that help ensure that pixels with very low and very high values are considered as unreliable, and thus contribute less to the final image pixel sample value. It should be understood that the variables and behavior of the exemplary hat function should not be construed as limiting but can vary from implementation to implementation.
TABLE 1
An Example Function for Determining Weight of Pixel Sample Values
A hat function: float wt_lut8[256];
for(int i=0; i<256; i++){
wt_lut8[i]=1−powf(((2*i/256.0f)−1.0f), 12.0f);
}
Before the weighted average at a HDR pixel position is computed, the converted and normalized pixel sample values of the images 612, 614, 616 can also be corrected for possible gamma correction quantization errors. An example of a suitable gamma quantization error function is shown in Table 2. The example gamma quantization error function computes a gamma error (gm_error) that is dependent on the input pixel value (i). It should be understood that the variables of the exemplary gamma quantization error function should not be construed as limiting and can vary from implementation to implementation.
TABLE 2
An Example Function for Gamma Error Correction
for(int i=1; i<256; i++){
gm_error[i] = powf((float)(i+.5f)/255, 2.2f) − powf((float)(i−.5f)/255,
2.2f)
}
At 660, the merged image is converted to an appropriate HDR format. For example, the merged pixel data from process block 650 can be encoded into an RGBE or JPEG XR image file. The merged pixel data can be encoded into other formats as well, such as a “half float” file or another file suitable for representing HDR image data. In certain embodiments, conversion into an HDR format is not performed, such as when raw HDR pixel data is the desired output, and no special formatting or encoding is necessary.
The method of
D. Exemplary Methods for High Dynamic Range Image Rendering
At 710, pixel sample values in an HDR digital image are tone mapped to respective lower dynamic range values. Tone mapping can be performed using a global tone map or a global tone map and one or more local tone maps. In general, a tone map refers to a function, g, which maps an image pixel sample value s to a pixel sample value g(s). Global tone mapping is the process of applying a global tone map to each of the pixel sample values in a given image I. For example, a global tone map can be applied to an image to modify the brightness and/or contrast qualities of the image as desired by a user, resulting in a tone-mapped image G(I). Local tone mapping refers to a function L which transforms an image I to a tone-mapped image L(I). As with global tone mapping, a local tone map can be applied to an image with the goal of adjusting brightness and contrast in the image. However, a local tone map can allow for local adjustments which may not be uniformly applied across the image. In certain embodiments, respective LDR values can be identified in a global or local tone mapping lookup table and stored in an LDR image buffer.
At 720, a local contrast value is computed for each of the tone-mapped pixel sample values. For example, a local contrast value is computed based on image data in the linear domain.
At 730, a residual signal is derived for each of the tone-mapped pixel sample values, based at least in part on the respective local contrast value. For example, the system multiplies a pixel sample value in the LDR image buffer by the corresponding local contrast value.
At 740, pixel sample values are computed for an LDR digital image based at least in part on the tone-mapped sample values and the respective residual signals. For example, the system adds together a sample value in the LDR buffer, the corresponding residual signal and a high-frequency dithering signal.
V. Image Alignment Techniques for Use in High Dynamic Range Image Generation
A. Exemplary Generalized Alignment Methods
At 810, a reference image and a non-reference image (the image to be aligned) are input (e.g., loaded, buffered into memory, or otherwise accessed or prepared for further processing). In certain embodiments, the reference image is a mid exposure image and the image to be aligned is either a low exposure image or a high exposure image. For instance, in one particular embodiment, the method 800 is performed once to align a low exposure image to the reference mid exposure image, and once to align a high exposure image to the reference mid exposure image. The order in which the method 800 is performed for the low and high exposure images can vary.
At 820, local motion estimation is performed. For example, and as more fully explained below, a pyramid block-based based motion vector estimation procedure using covariance.
At 830, outlier motion vectors are removed. For example, and as more fully explained below, a Random Sample Consensus (“RANSAC”) method can be used to detect and eliminate motion vectors that are outliers due to moving objects with local motions, saturation, and/or under exposure.
At 840, a global motion transform is estimated. For example, a global motion vector can be estimated by processing the motion vectors computed at process block 820 and can be used to compensate for translation, rotation, and/or zooming of the image to be aligned (e.g., as a result of jitter from hand shakiness).
At 850, image warping is performed to the non-reference image (the image to be aligned). For instance, when the low exposure image is the image to be aligned, the low exposure image can be warped, and when the high exposure image is the image to be aligned, the high exposure image can be warped.
The local motion estimation 820 and image warping 850 processes are typically computationally intensive, and are desirably performed using available multimedia DSP components on the target computing device (e.g., using a specialized ASIC unit (such as the Tesla GPU from Nvidia, or the Larabee unit from Intel), a graphics processing unit (“GPU”) with shader capability, or a multi-core processor or central processing unit having advanced multimedia instructions (e.g., SSEx instructions)). For example, a form of the local motion estimation 820 that is particularly suited for processing utilizing GPU shaders can be implemented and used to accomplish the local motion estimation portion of the digital image alignment method 800 when the target computing device has a GPU shader capability among its available multimedia DPS components. The global alignment program can also include functionality to perform image warping using a D3D API call with bilinear interpolation when the evaluated target computing device is determined to possess that capability among its available multimedia DSP components. The global transform estimation performed at 840 can be performed by specialized hardware but is typically performed by a CPU.
B. Pyramid Block-Based Motion Vector Estimation Using Covariance
In one example implementation, the local motion estimation (e.g., the local motion estimation at process block 830) is performed using a pyramid block-based motion estimation technique with covariance analysis, as illustrated in
The pyramid block-based motion estimation performs motion estimation between the reference image and the non-reference image to be aligned (e.g., between a mid exposure image (the reference image) and a low exposure image (the image to be aligned) or between a mid exposure image (the reference image) and a high exposure image (the image to be aligned)). For the motion estimation, the two images are divided into a regular grid of blocks, such as 16×16 blocks, and the motion estimation calculates a motion vector or displacement of each block of the image to be aligned relative to closely matching image content in the reference image.
The pyramid block-based motion estimation begins by repeatedly down-sampling the resolution of both the non-reference image and the reference image by half (e.g., using a downsampling shader on the GPU of the computing device). In the example shown in
For each resolution of the digital image, the pyramid block-based motion estimation divides the digital images into a grid of blocks. The same size of block is used at each resolution. Preferably a relatively large block size, such as 16×16 or 8×8 pixels is used. In this way, each block of the lowest resolution image will split into 4 blocks of the same block size at the corresponding location in the next higher resolution image, which effectively provides a pyramid structure, or hierarchy, relationship of blocks in the lowest resolution image to those at the corresponding location in the higher resolution images. For purposes of illustration, the blocks in the higher resolution versions of the digital image (e.g., 1/1, 1/2 and 1/4 resolution) that are in a pyramid structure relationship to the upper-left block of the 1/8 resolution image are shown within the thick line boundary in
Next, the pyramid block-based motion estimation performs a full motion vector search based on a covariance analysis on each block in the lowest resolution image over the full reference image to find a position of best matching (or nearest matching) image content in the reference image. The search can be performed, for example, by searching for matching blocks at nearby block locations (e.g. using a covariance analysis), then gradually expanding the search area (e.g., using an expanding spiral pattern) to match a block in the image to be aligned with a block in the reference image. The motion vector typically indicates a displacement in the horizontal dimension and a displacement in the vertical direction of the block in the image to be aligned relative to the reference image. A motion vector of zero can be used to indicate that no motion has been detected for a particular block. This process produces a local motion vector for each block in the lowest resolution version of the image to be aligned corresponding relative to the matching (or nearest matching) image content in the reference image. In particular embodiments, the full search is performed only between the lowest resolution version of the image and the lowest resolution version of the reference image, as shown in
As noted, in certain embodiments of the disclosed technology, a covariance analysis is used to determine the motion vector in the non-reference image to be aligned that best indicates the relative displacement from the matching image content in the reference image (or vice versa). This covariance computation is in contrast to a sum of differences computation. Looking only at the differences between pixel sample values of two images having different exposures is not an effective mechanism for matching image content between images having different exposure levels because the exposure-level differences inherently cause the images to have different pixel values (even if the images are perfectly aligned). A covariance analysis is desirable because covariance measures how much two sets of values change together, regardless of the relative size of the values in the sets. Thus, covariance is effective at detecting and correlating textures and edges between two blocks of image content. Covariance is therefore a useful tool in matching blocks between images having different exposure levels. For instance, in one embodiment, covariances between pixel sample values in a block of the reference image and candidate blocks in the non-reference image to be aligned are computed (or vice versa such that covariances between pixel sample values in a block of the non-reference and candidate blocks in the reference image are computed). The block in the non-reference image having the highest covariance value is then selected as the matching (or nearest matching) block and the corresponding motion vector is computed and used.
In one example, the covariance between two blocks (e.g., a block in the reference image and a candidate block in the non-reference image to be aligned, or vice versa) is computed as follows:
cov(X,Y)=(X−μX)(Y−μY)
(1)
where X is a variate corresponding to pixel sample values from a first block (e.g., the block in the reference image), and Y is a variate corresponding to pixel sample values from a second block (e.g., the candidate block in the image to be aligned). Further, μX=X
and μY=
Y
and are the respective means for x and y. Equation (1) can be more explicitly written as:
where i is an index value, N is the sample size and corresponds to the number of pixel sample values in the block used as part of the covariance computation,
As illustrated in
The pyramid structure relationship of the blocks in the lower resolution digital image to the four corresponding blocks in the next higher resolution digital image emphasizes consistency in the motion estimation calculation among neighboring blocks. As the refinement search is repeated for each successive higher resolution version of the digital image, this emphasized consistency among neighbors provides a more consistent set of local motion vectors for the blocks in the original resolution image upon which to base the global motion transform estimation discussed below.
Once local motion estimation is finished at the original image resolution, the motion vector (optionally along with the covariance result) for each block of the original resolution image is copied, buffered, or otherwise stored for use in global motion estimation.
C. Global Transform Estimation and Removal of Outlying Motion Vectors
where x and y are the original position, x′ and y′ are the transformed position, and s, β, tx, ty are zoom, rotation and translation parameters, respectively.
In general, most users will experience shaky motion effects on translation, rotation and zoom only. Accordingly, the similarity motion transform model of equation (3) is well suited to account for hand shakiness or time evolution between images of different exposure levels which are then used to create a single HDR image. However, alternative implementations of the global motion transform estimation could use other models that model additional or fewer aspects of image jitter from hand shakiness or time evolution.
At 1510, motion vector information between the non-reference image and the reference image is received (e.g., loaded, buffered into memory, or otherwise accessed or prepared for further processing). In certain implementations, covariance information (e.g., covariance information for each block in the image produced during local motion estimation at process block 820 of
At 1520, a set of the motion vectors are selected upon which to base the global motion transform estimation. The motion vectors selected can be motion vectors determined to be more reliable than other motion vectors. In one example implementation, the method 1500 selects motion vectors based on one or more rules. One rule, for example, can be that the motion vectors on picture boundaries are not reliable, and therefore are not selected. For example, camera motion that occurs from image to image can cause the boundary blocks to be partially missing. A further rule can be that blocks that have an original variance value that is below a certain threshold are unreliable, and therefore are not selected. The original variance value measures the relative difference between the pixel sample values in the block. A low value indicates that the pixel sample values in the corresponding block are largely similar or identical to one another. Such low variance values typically occur in areas of a scene that have a large space of a similar color (e.g., a blue sky). On account of the lack of texture or edges, such blocks are not good candidates for producing reliable motion vectors. The threshold value can vary from implementation to implementation but is selected to help eliminate motion vectors that are associated with large smooth spaces in the image. In certain embodiments, all three rules can be used, though fewer or additional criteria of reliability can also be used.
At 1530, an estimate of the global motion transform is computed for the images (e.g., for the relative movement of the non-reference image in relation to the reference image, or vice versa). In certain embodiments, the global motion transform is estimated using equation (3) for the similarity global motion model discussed above. Alternatively, the global motion transform can be estimated based on another motion model.
For estimating the global motion transform, embodiments of the method 1500 also use a Random Sample Consensus (“RANSAC”) method to detect and eliminate motion vectors that are outliers. In the RANSAC method, two motion vectors are chosen at random out of the set of motion vectors selected at 1520. The motion vector processing then solves equation (3) using a Least Mean Square Error (“LMSE”) fit to determine values of s, β, tx, ty parameters for the two randomly selected motion vectors. The RANSAC method then determines a number of the other motion vectors that also are consistent with this solution of the global motion transform equation parameters. The RANSAC method repeats the random selection of other pairs of motion vectors until a largest group of motion vectors consistent with the solution is found. This eliminates motion vectors that are considered outliers, or inconsistent with the consensus LMSE solution for the parameters of the global motion transform.
D. Image Warping with Global Motion Transform
With reference again to
VI. Concluding Remarks
Any of the methods described herein can be performed by computer-executable instructions stored on one or more non-transitory computer-readable media (e.g., storage or other tangible media). Operations can be fully automatic, semi-automatic, or involve manual intervention.
Having described and illustrated the principles of our invention with reference to the described embodiments, it will be recognized that the described embodiments can be modified in arrangement and detail without departing from such principles. It should be understood, for example, that the programs, processes, or methods described herein are not related or limited to any particular type of computing environment, unless indicated otherwise. Various types of general purpose or specialized computing environments may be used with or perform operations in accordance with the teachings described herein. Elements of the described embodiments shown in software may be implemented in hardware and vice versa.
In view of the many possible embodiments to which the principles of our invention may be applied, we claim as our invention all such embodiments as may come within the scope and spirit of the following claims and equivalents thereto.
Sun, Shijun, Wu, Yongjun, Jandhyala, Radhika
Patent | Priority | Assignee | Title |
10218987, | Jun 30 2014 | The University of North Carolina at Chapel Hill | Methods, systems, and computer readable media for performing image compression |
10257421, | Feb 11 2010 | Microsoft Technology Licensing, LLC | Generic platform video image stabilization |
10319079, | Jun 30 2017 | Microsoft Technology Licensing, LLC | Noise estimation using bracketed image capture |
10380463, | Aug 30 2017 | Omron Corporation | Image processing device, setting support method, and non-transitory computer-readable media |
10841494, | Feb 11 2010 | Microsoft Technology Licensing, LLC | Motion vector estimation for video image stabilization |
11004232, | Jul 12 2019 | MEDIATEK INC. | Method and apparatus for performing object detection according to bounding box distribution that is determined by considering lens configuration |
11025830, | May 23 2013 | PROMANTHAN BRAINS LLC, SERIES GAZE | Deghosting camera |
11184559, | Aug 02 2017 | Sony Corporation | Image processing device, image processing method, and imaging device |
11373281, | Feb 23 2021 | Qualcomm Incorporated | Techniques for anchor frame switching |
11558558, | May 23 2013 | PROMANTHAN BRAINS LLC, SERIES GAZE | Frame-selective camera |
8896715, | Feb 11 2010 | Microsoft Technology Licensing, LLC | Generic platform video image stabilization |
9241109, | Apr 27 2012 | Canon Kabushiki Kaisha | Image capturing apparatus, control method, and recording medium for moving image generation |
9349162, | Oct 24 2014 | Matrox Electronic Systems Ltd. | System and method for image color correction |
9525804, | Aug 30 2014 | Apple Inc. | Multi-band YCbCr noise modeling and noise reduction based on scene metadata |
9578240, | Feb 11 2010 | Microsoft Technology Licensing, LLC | Generic platform video image stabilization |
9626745, | Sep 04 2015 | Apple Inc | Temporal multi-band noise reduction |
9641820, | Sep 04 2015 | Apple Inc | Advanced multi-band noise reduction |
9667842, | Aug 30 2014 | Apple Inc. | Multi-band YCbCr locally-adaptive noise modeling and noise reduction based on scene metadata |
9824426, | Aug 01 2011 | Microsoft Technology Licensing, LLC | Reduced latency video stabilization |
9955084, | May 23 2013 | Promanthan Brains LLC, Series Gaze only | HDR video camera |
Patent | Priority | Assignee | Title |
5259040, | Oct 04 1991 | Sarnoff Corporation | Method for determining sensor motion and scene structure and image processing system therefor |
5629988, | Jun 04 1993 | Sarnoff Corporation | System and method for electronic image stabilization |
5973733, | May 31 1995 | Texas Instruments Incorporated | Video stabilization system and method |
6097854, | Aug 01 1997 | Microsoft Technology Licensing, LLC | Image mosaic construction system and apparatus with patch-based alignment, global block adjustment and pair-wise motion-based local warping |
7010174, | Apr 29 2003 | Microsoft Technology Licensing, LLC | System and process for generating high dynamic range video |
7061524, | Nov 13 2001 | BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY, THE | Motion/saturation detection system and method for synthesizing high dynamic range motion blur free images from multiple captures |
7119837, | Jun 28 2002 | Microsoft Technology Licensing, LLC | Video processing system and method for automatic enhancement of digital video |
7227896, | Mar 08 2002 | Sharp Kabushiki Kaisha | Method and apparatus for global motion estimation |
7346109, | Dec 23 2003 | Genesis Microchip Inc. | Motion vector computation for video sequences |
7454136, | Jul 28 2005 | Mitsubishi Electric Research Laboratories, Inc | Method and apparatus for acquiring HDR flash images |
7489341, | Jan 18 2005 | Primax Electronics Ltd. | Method to stabilize digital video motion |
7557832, | Aug 12 2005 | Method and apparatus for electronically stabilizing digital images | |
7649549, | Sep 27 2004 | Texas Instruments Incorporated | Motion stabilization in video frames using motion vectors and reliability blocks |
20050099504, | |||
20060066728, | |||
20060274156, | |||
20060280334, | |||
20070014470, | |||
20070201560, | |||
20070242900, | |||
20070292047, | |||
20070297513, | |||
20090051777, | |||
20090180555, | |||
20090256918, | |||
20090263009, | |||
20100183071, | |||
20100208087, | |||
20100271512, | |||
20110019014, | |||
20120105654, | |||
EP986252, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Feb 25 2011 | Microsoft Corporation | (assignment on the face of the patent) | / | |||
Mar 23 2011 | JANDHYALA, RADHIKA | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 026304 | /0369 | |
Mar 23 2011 | WU, YONGJUN | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 026304 | /0369 | |
Mar 23 2011 | SUN, SHIJUN | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 026304 | /0369 | |
Oct 14 2014 | Microsoft Corporation | Microsoft Technology Licensing, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 034544 | /0001 |
Date | Maintenance Fee Events |
Mar 19 2014 | ASPN: Payor Number Assigned. |
Oct 12 2017 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Oct 13 2021 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Date | Maintenance Schedule |
Apr 29 2017 | 4 years fee payment window open |
Oct 29 2017 | 6 months grace period start (w surcharge) |
Apr 29 2018 | patent expiry (for year 4) |
Apr 29 2020 | 2 years to revive unintentionally abandoned end. (for year 4) |
Apr 29 2021 | 8 years fee payment window open |
Oct 29 2021 | 6 months grace period start (w surcharge) |
Apr 29 2022 | patent expiry (for year 8) |
Apr 29 2024 | 2 years to revive unintentionally abandoned end. (for year 8) |
Apr 29 2025 | 12 years fee payment window open |
Oct 29 2025 | 6 months grace period start (w surcharge) |
Apr 29 2026 | patent expiry (for year 12) |
Apr 29 2028 | 2 years to revive unintentionally abandoned end. (for year 12) |