A method that reconstructs an image function (r), which represents an object function (ƒ) of an object imaged by periodically structured illumination with an illumination grating. It includes: provision of a group of optical sectional images, which are formed after folding operations of the object function (ƒ) with an illumination point spread function (hI), from recorded intermediate images (gi) of the object, wherein the group of optical sectional images contains sub-images (gck, gsk) whose contributions in the image function (r) are formed by kth Fourier components (k>1) of the illumination grating, and demodulation of the group of optical sectional images in order to generate the image function (r). Also described are an imaging method and devices in which the reconstruction method is used.
|
14. A reconstruction apparatus adapted to reconstruct an image function (r), which represents an object function (ƒ) of an object, from a group of optical sectional images which contains sub-images (gck, gsk) which in the image function (r) are formed by kth Fourier components (k>1) of an illumination grating, comprising a demodulation circuit adapted to generate the image function (r) from the sub-images (gck, gsk).
9. An imaging device for obtaining an image function (r) of an object, comprising:
an image recording apparatus that periodically structurally illuminates the object with an illumination grating and detects a plurality of intermediate images (g′),
a sectional image calculator that determines a group of optical sectional images from the intermediate images (gi), and
a reconstruction apparatus that reconstructs an image function (r), which represents an object function (ƒ) of the object, from the group of optical sectional images which contains sub-images (gck, gsk) whose contributions in the image function (r) are formed by kth Fourier components (k>1) of the illumination grating.
1. A method for reconstructing an image function (r) which represents an object function (ƒ) of an object imaged by periodically structured illumination with an illumination grating, comprising:
providing a group of optical sectional images formed after folding operations of the object function (ƒ) with an illumination point spread function (hI), from recorded intermediate images (gi) of the object, wherein the group of optical sectional images contains sub-images (gck, gsk) whose contributions in the image function (r) are formed by kth Fourier components (k>1) of the illumination grating, and
demodulating the group of optical sectional images to generate the image function (r).
2. The method according to
3. The method according to
wherein wk comprises predetermined weights of the kth Fourier components of the illumination grating.
4. The method according to
5. The method according to
determining a grating period and phase position of the illumination grating from sub-images (gck, gsk) formed in the image function (r) by basic Fourier components (k=1) of the illumination grating, wherein multiples of the grating period and phase position of the illumination grating are used to calculate the image function (r).
6. The method according to
7. The method according to
8. An imaging method for obtaining an image function (r) of an object, comprising:
periodically structurally illuminating the object,
detecting a plurality of intermediate images (g′),
determining a group of optical sectional images from the intermediate images (gi), and
reconstructing the image function (r) according to the method of
10. The imaging device according to
11. The imaging device according to
12. The imaging device according to
16. A computer program product on a computer-readable storage medium in a non-transitory manner comprising a program code for implementing a reconstruction method according to
17. A device which comprises a computer-readable storage medium which contains program instructions for implementing a reconstruction method according to
|
This application claims priority of German Patent Application No. 10 2006 044 229.6, filed Sep. 20, 2006, herein incorporated by reference.
This disclosure relates to methods for image processing, in particular, to a method for reconstructing images which have been recorded by a microscope with structured illumination, and to an imaging method for microscopic imaging of an object. The disclosure also relates to devices for implementing the aforementioned methods, in particular, to an imaging device such as a microscope, for example, and a reconstruction device for image reconstruction, and to applications of the methods.
In optical microscopy, various methods have been developed for the depth-resolved imaging of objects, e.g., to obtain one or more sectional images through a sample. By way of example, in confocal microscopy with confocal, point-by-point illumination and imaging, image information is gathered only in the sectional plane of interest. Disadvantages of confocal microscopy exist in particular due to the complex design of the microscope and the time-consuming way in which the image information is gathered.
Another method is structured illumination microscopy (SIM technique). In this method of wide-field microscopy, a periodic grating structure is focussed into the relevant sectional plane of the sample. By imaging the grating structure of an illumination grating into the sectional plane, a modulation of the illumination in real space is achieved, thereby overcoming the so-called “missing cone” problem of wide-field microscopy. From a plurality of intermediate images recorded in each case with a different phase position of the grating structure, an image function can be calculated which represents an image of the sectional plane in the real object (see, for example, WO 2004/038483 A1). However, conventional structured illumination microscopy has the general disadvantage that the image reconstructed from the intermediate images is an approximation, the quality of which may be limited in particular by artifacts and instabilities.
For the efficient use of illumination gratings for structured illumination, use is usually made of transmission gratings which are formed by sharply defined stripes (so-called “Ronchi Ruling”). Conventional structured illumination microscopy has the general disadvantage that a transmission grating with a relatively large period is often selected for light yield reasons. As a result, the axial resolution can be improved only to a limited extent, although the potential that artifacts will occur is relatively high, depending on the observation point spread function. Another disadvantage is that Fourier components which lie outside the fundamental frequency of the transmission grating generate disadvantageous artifacts in the conventionally calculated optical sections (see L. H. Schaefer et al. in “Journal of Microscopy”, Vol. 216, 2004, pages 165-174). It has therefore been found in practice that the quality of image reconstruction may exhibit undesirable limitations, particularly when analysing biological samples. These limitations may be expressed for example in that details of the real sample are imaged only with an insufficient resolution.
It could therefore be advantageous to provide an improved method for image reconstruction from intermediate images of an object imaged with structured illumination, by means of which the disadvantages of conventional techniques are overcome. It could also be advantageous to provide an improved imaging method for obtaining images of objects illuminated with structured illumination and a correspondingly improved imaging device.
We provide a method for reconstructing an image function (r) which represents an object function (ƒ) of an object imaged by periodically structured illumination with an illumination grating, including providing a group of optical sectional images formed after folding operations of the object function (ƒ) with an illumination point spread function (hI), from recorded intermediate images (gi) of the object, wherein the group of optical sectional images contains sub-images (gck, gsk) whose contributions in the image function (r) are formed by kth Fourier components (k>1) of a illumination grating, and demodulating the group of optical sectional images to generate the image function (r).
We also provide an imaging method for obtaining an image function (r) of an object, including periodically structurally illuminating the object, detecting a plurality of intermediate images (g′), determining a group of optical sectional images from the intermediate images (gi), and reconstructing the image function (r) according to the method for reconstructing an image function.
We further provide an imaging device for obtaining an image function (r) of an object, including an image recording apparatus that periodically structurally illuminates the object with an illumination grating and detects a plurality of intermediate images (g′), and a reconstruction apparatus that reconstruct an image function (r), which represents an object function (ƒ) of the object, from the group of optical sectional images which contains sub-images (gck, gsk) whose contributions in the image function (r) are formed by kth Fourier components (k>1) of the illumination grating.
We still further provide a microscope including an imaging device.
We further yet provide a reconstruction apparatus adapted to reconstruct an image function (r), which represents an object function (ƒ) of an object, from a group of optical sectional images which contain sub-images (gck, gsk) which in the image function (r) are formed by kth Fourier components (k>1) of an illumination grating, including a demodulation circuit adapted to generate the image function (r) from the sub-images (gck, gsk).
We also further provide a microscope including a reconstruction apparatus.
We further still provide a computer program product which is located on a computer-readable storage medium, including a program code for implementing the reconstruction method.
We yet further provide a device which includes a computer-readable storage medium which contains program instructions for implementing the reconstruction method.
Further details and advantages of our methods and devices will be described below with reference to the appended drawings, in which:
In method terms, a first aspect is based on the general technical teaching of reconstructing an image function from optical sectional images which have been determined from an imaging with structured illumination and the contributions of which in the image function are formed by kth Fourier components (k>1) of the illumination grating. Advantageously, the set of optical sectional images due to the higher Fourier components of the illumination grating contains sub-images comprising information about the sample which would otherwise be available only by illuminating with a grating of smaller period, which otherwise possibly could not be used purely for functional reasons (light yield, modulation depth achievable by the illumination point spread function).
We found that the higher Fourier components, which in the conventional calculation of an optical section occurred only as oscillatory interference and were discarded (see the aforementioned publication by L. H. Schaefer et al.), can be taken into account in the reconstruction (demodulation of the sectional images) and give rise to the significant advantages of improved artefact suppression and improved resolution due to a larger amount of information being obtained. The image function calculated represents a much improved approximation of the real object function. Another important advantage uses a coarser grating structure which allows more light through and optionally results in an improved signal-to-noise ratio can be used for the structured illumination if necessary, depending on the application, without impairing the quality of the image reconstruction.
The “object function” (ƒ) denotes a real amplitude, dependent on spatial coordinates x and y, of the light (e.g. transmission, reflection or fluorescence) present in the relevant sectional plane of the analysed object (sample). The “image function” (r) is the approximation (image) of the object function determined by the reconstruction, and thus the approximation of the object imaged by periodically structured illumination with an illumination grating. The optical sectional images are calculated from “intermediate images” (gi) which are detected with the periodically structured illumination of the object, known per se, with different phase positions of the grating structure. A plurality of intermediate images recorded with different phases of the illumination grating deliver a plurality of optical sectional images. Within this group of optical sectional images, the sectional images formed by the higher Fourier components of the illumination grating are referred to as “sub-images” (gck, gsk). The “point spread function” (PSF; Fourier-transformed PSF: Optical Transfer Function, OTF) describes the transmission of the light and optionally of the image information in the optical system which is used for imaging. The “illumination point spread function” describes the transmission of the light from the light source via the grating structure to the sample. The “detection point spread function” describes the transmission of the light from the sample to the camera.
If, the grating period and phase position of the illumination grating are determined from further sub-images (gck, gsk) which in the image function (r) are formed by basic Fourier components (k=1) of the illumination grating, advantages are obtained in respect of a relatively low calculation complexity during the reconstruction. Advantageously, multiples of the grating period and phase position of the illumination grating can accordingly be used in order to calculate the image function (r) from all the sub-images taken into account.
The reconstruction method can be applied with optical sectional images immediately after they have been calculated by microscopic imaging with structured illumination. In this case, the sought image function is advantageously formed directly from the optical sectional images. The reconstruction method can also be applied with optical sectional images which are calculated from stored intermediate images (raw data) from conventional microscopic imagings.
Further aspects are mentioned below in connection with a mathematical description of the image reconstruction.
We provide an imaging method for obtaining an image function of an object, in which a plurality of intermediate images are recorded with a periodically structured illumination of the object in each case with different phases of the grating structure used for illumination, and optical sectional images are calculated from these, wherein the optical sectional images are subjected to the method for reconstructing the image function according to the abovementioned first aspect. By means of the imaging method, advantageously depth-resolved images with an improved image quality can be provided.
We also provide an imaging device which has an image recording apparatus which is adapted for a periodically structured illumination of the object and for detecting a plurality of intermediate images, and a reconstruction apparatus which is adapted to reconstruct an image function which represents an object function of the object. The reconstruction apparatus (reconstruction circuit) contains a demodulation circuit which is adapted to reconstruct the image function, which represents the object function of the object, from the group of optical sectional images which contains sub-images (gck, gsk) whose contributions in the image function are formed by kth Fourier components (k>1) of the illumination grating. Preferably, the reconstruction apparatus is designed to implement the image reconstruction method.
The imaging device may be an optical microscope with structured illumination, which is equipped with the reconstruction apparatus.
We also provide devices which thus represent an independent subject matter. Advantageously, a conventional microscope with structured illumination can be retrofitted by simple means with our reconstruction apparatus.
Further independent subject matter includes, but is not limited to, a computer program product on an electronic, magnetic or optical storage medium comprising a program code for implementing the reconstruction method, and a storage medium which contains program instructions for implementing the reconstruction method.
Selected aspects of our methods and devices will be described below with reference to implementing the reconstruction of the image function. Details regarding image recording using a microscope with structured illumination are known per se and therefore will not be explained here.
In the imaging method S0.1 to S2, an image recording using the method of structured illumination is first carried out as in the case of conventional techniques. During the image recording, a plurality of intermediate images gi are first determined in a first sub-step S0.1.
Using a grating structure si which is characterized by a spatial frequency ω and a phase φ and can be described by
si(x,z)=δ(z)[1+m cos(ωx+φi)] (1)
the representation of the intermediate images gi is obtained with the coordinates {circumflex over (x)}k=(xk, yk, zk) and the differentials d{circumflex over (x)}k=dxkdykdzk, as described by L. H. Schaefer et al. in “Journal of Microscopy” (see above), according to:
gi(x)=∫∫∫ƒ(x1)|hD(x−x1)|2×∫∫∫si(x−x0, z−z0−z1)|hI(x0)|2dx0dx1 (2)
(hI and hD: illumination and detection point spread functions) or in a simplified form
gi=hD*(ƒ·[hI*si] (3).
Unlike the conventional approximation
gi=hD*(ƒ·[hI*si])=gw+gc cos φi+gs sin φi (4)
proposed by L. H. Schaefer et al. in the abovementioned publication, according to the invention the intermediate images are represented by
In equation (5), gw represents the conventional microscopic wide-field image and the optical sectional images gck, gsk represent in particular the sub-images which are determined by the basic Fourier components of the illumination grating (k=1) and by the integer (kth) multiples of the Fourier components of the structure of the illumination grating (k>1).
As described in WO 2004/038483 A1 and by L. H. Schaefer et al. in the abovementioned publication, the intermediate images recorded with different phases φi of the illumination grating result in an equation system which, when solved, gives the sought optical sectional images, in particular with the sub-images corresponding to the higher Fourier components of the illumination grating (step S1). Since the vector of unknown variables (gw, gC1 . . . gCN, gS1 . . . gSN)T in equation (5) is expanded in comparison to the conventional approximation (4), more raw data (intermediate images gi) are accordingly required in order to solve the equation system.
If, for example, at least five intermediate images gi are recorded with different phases φ, then equation (5) results in an equation system from which the optical sectional images gck, gsk can be calculated separately in step S1. The equation system is solved for example by the method described by L. H. Schaefer et al. in the abovementioned publication. The optical sectional images gck, gsk are provided as input variables for the subsequent demodulation (step S2).
A heterodyne demodulation takes place in step S2 to reconstruct the resulting image (of the sought image function), in which the image function (r) is calculated according to
(wk; predetermined weights of the kth Fourier components of the illumination grating). The demodulation according to equation (6) is a linear calculation and is therefore used with preference.
The weights wk result from the Fourier synthesis of the grating form used. In the case of an illumination grating whose transmission curve is represented by a square-wave function with a 1:1 duty cycle, w1=1, w3=3, w5=5, . . . etc. The 1:1 duty cycle has the advantages of being easy to implement and of simplifying the calculation complexity.
For the demodulation according to equation (6), precise knowledge of ejωkx, i.e. of the grating period and phase position of each Fourier component of the illumination grating, is required. Preferably, these variables are determined with the required accuracy from the recorded raw data in a first processing step with the grating period of the fundamental frequency (k=1). Since the determination of the grating period and phase position of the fundamental frequency can be formulated as an extremal problem, suitable frequency-sensitive and phase-sensitive cost functions which have the grating period or phase position as a parameter are minimized. With the values thus found for gC1 and gS1, in a further sub-step the image function (r) is calculated, which also contains integer multiples of the grating period.
As shown in
The image recording apparatus 10 comprises a light source 11, a movable grating structure 12, a semi-transparent mirror 13, optical components 14 and a camera 15. These components are known per se from optical imaging systems with structured illumination, and are embodied for example by a microscope with the grating insert and a control system of the ApoTome type (Carl Zeiss AG). The sectional image calculator 20 is for example a calculator circuit by means of which the aforementioned equation system (5) for determining the optical sub-images gck, gsk is solved. The reconstruction apparatus 30 comprises a demodulation circuit 31 for calculating the image function r from the corrected optical sectional images according to equation (6).
The representative features disclosed in the above description, the drawings and the claims may be important both individually and in combination for implementing our methods and devices.
Schäfer, Lutz, Schuster, Dietwald
Patent | Priority | Assignee | Title |
10831012, | Jun 29 2018 | ILLUMINA, INC | Predicting structured illumination parameters |
10928322, | Jan 24 2018 | Illumina, Inc. | Structured illumination microscopy with line scanning |
10996453, | Jan 16 2018 | ILLUMINA, INC | Pattern angle spatial selection structured illumination imaging |
11112591, | Jan 24 2018 | ILLUMINA, INC | Reduced dimensionality structured illumination microscopy with patterned arrays of nanowells |
11119302, | Jun 29 2018 | Illumina, Inc. | Predicting structured illumination parameters |
11150455, | Jan 24 2018 | ILLUMINA, INC | Reduced dimensionality structured illumination microscopy with patterned arrays of nanowells |
11378544, | Jan 08 2018 | Illumina, Inc. | High-throughput sequencing with semiconductor-based detection |
11525993, | Jun 29 2018 | Illumina, Inc. | Predicting structured illumination parameters |
11561196, | Jan 08 2018 | Illumina, Inc. | Systems and devices for high-throughput sequencing with semiconductor-based detection |
11585757, | Jun 14 2018 | ILLUMINA, INC | Device for luminescent imaging |
11650156, | Jan 24 2018 | Illumina, Inc.; Illumina Cambridge Limited | Reduced dimensionality structured illumination microscopy with patterned arrays of nanowells |
11675175, | Jan 16 2018 | ILLUMINA, INC | Multi-arm structured illumination imaging |
11933728, | Jan 24 2018 | Illumina, Inc.; Illumina Cambridge Limited | Reduced dimensionality structured illumination microscopy with patterned arrays of nanowells |
11953464, | Jan 08 2018 | Illumina, Inc. | Semiconductor-based biosensors for base calling |
Patent | Priority | Assignee | Title |
5426297, | Sep 27 1993 | United Technologies Corporation | Multiplexed Bragg grating sensors |
5867604, | Aug 03 1995 | GIM SYSTEMS LTD | Imaging measurement system |
5898517, | Aug 24 1995 | Optical fiber modulation and demodulation system | |
6829397, | Apr 10 2002 | National Tsing Hua University | Dual fiber bragg grating strain sensor system |
6909105, | Mar 02 1999 | MAX-PLANCK-GESELLSCHAFT ZUR FORDERUNG DER WISSENSCHAFTEN E V | Method and device for representing an object |
7009718, | Jun 06 2001 | CITIZEN WATCH CO , LTD | Grating pattern projection apparatus using liquid crystal grating |
7053362, | May 02 2003 | Mitutoyo Corporation | Absolute position miniature grating encoder readhead using fiber optic receiver channels |
7376068, | Aug 19 2000 | Nano-scale resolution holographic lens and pickup device | |
7623235, | Mar 20 2004 | HO, SENG-TIONG | Curved grating spectrometer with very high wavelength resolution |
WO2004038483, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Sep 11 2007 | SCHUSTER, DIETWALD | Carl Zeiss Imaging Solutions GmbH | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 019905 | /0990 | |
Sep 12 2007 | Carl Zeiss Imaging Solutions GmbH | (assignment on the face of the patent) | / | |||
Sep 12 2007 | SCHAFER, LUTZ | Carl Zeiss Imaging Solutions GmbH | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 019905 | /0990 | |
Jul 01 2008 | Carl Zeiss Imaging Solutions GmbH | Carl Zeiss Imaging Solutions GmbH | CHANGE OF ADDRESS FOR ASSIGNEE | 021198 | /0353 | |
Dec 05 2013 | Carl Zeiss Imaging Solutions GmbH | Carl Zeiss Microscopy GmbH | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 031719 | /0751 | |
Mar 07 2024 | Progress Software Corporation | JPMORGAN CHASE BANK, N A , AS ADMINISTRATIVE AGENT | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 066762 | /0833 |
Date | Maintenance Fee Events |
Jan 07 2013 | ASPN: Payor Number Assigned. |
Oct 01 2015 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Oct 09 2019 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Oct 09 2023 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Date | Maintenance Schedule |
Apr 17 2015 | 4 years fee payment window open |
Oct 17 2015 | 6 months grace period start (w surcharge) |
Apr 17 2016 | patent expiry (for year 4) |
Apr 17 2018 | 2 years to revive unintentionally abandoned end. (for year 4) |
Apr 17 2019 | 8 years fee payment window open |
Oct 17 2019 | 6 months grace period start (w surcharge) |
Apr 17 2020 | patent expiry (for year 8) |
Apr 17 2022 | 2 years to revive unintentionally abandoned end. (for year 8) |
Apr 17 2023 | 12 years fee payment window open |
Oct 17 2023 | 6 months grace period start (w surcharge) |
Apr 17 2024 | patent expiry (for year 12) |
Apr 17 2026 | 2 years to revive unintentionally abandoned end. (for year 12) |