Disclosed is a concept for computer-assisted procedures of surgery and diagnosis that target rigid, non-deformable anatomical parts such as bone, tissue, or teeth. The disclosure describes attaching small visual markers to instruments and anatomy of interest (e.g. bone surface), with each marker having a printed known pattern for detection and unique identification in images acquired by a free-moving camera, and a geometry that enables estimating its rotation and translation with respect to the camera using solely image processing techniques.
|
0. 93. A system for computer-aided surgery and diagnosis in an anatomical region of a patient comprising a rigid, non-deformable anatomical part, the system comprising:
a free-moving camera to acquire intra-operative images, video, or images and video of an operating field,
where the free-moving camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f−1(u; k, ξ) with f−1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters;
a world marker (WM) comprising an object configured to be rigidly coupled to the anatomical part such that the WM and the anatomical part do not move one with respect each other; and
an image processing system configured to:
receive the images, video, or images and video acquired by the free-moving camera, wherein the images, video, or images and video include the WM in a field of view of the camera;
receive a 3d reconstruction of points of the anatomical part in a first coordinate system defined using the WM; and
output information derived from the 3d reconstruction for display to assist execution of a clinical procedure using a surgical instrument while keeping the WM in the field of view of the camera, wherein the output comprises guidance including a location of a surgical instrument in the first coordinate system.
0. 73. A system for computer-aided surgery and diagnosis in an anatomical region of a patient comprising a rigid, non-deformable anatomical part, the system comprising:
a camera to acquire intra-operative images, video, or images and video of an operating field,
where the camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f−1(u; k, ξ) with f−1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters;
a world marker (WM) comprising an object configured to be rigidly coupled to the anatomical part such that the WM and the anatomical part do not move one with respect each other, wherein the object comprises a first known pattern that defines a first system of coordinates;
an instrument comprising a surgical instrument or a point location device, wherein the camera is rigidly coupled to the instrument such that camera has a known positional relationship with a portion of the instrument;
an image processing system configured to:
receive the images, video, or images and video acquired by the camera, wherein the images, video, or images and video include the WM in a field of view of the camera;
receive a 3d reconstruction of points of the anatomical part in the first coordinate system; and
output information derived from the 3d reconstruction to a display to assist execution of a clinical procedure using the instrument while keeping the WM in the field of view of the camera, wherein the output comprises a location of the instrument in the first coordinate system.
0. 34. A system for computer-aided surgery and diagnosis in an anatomical region of a patient comprising a rigid, non-deformable anatomical part, the system comprising:
a free-moving camera to acquire intra-operative images, video, or images and video of an operating field,
where the free-moving camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f−1(u; k, ξ) with f−1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters;
a world marker (WM) comprising an object configured to be rigidly coupled to the anatomical part such that the WM and the anatomical part do not move one with respect to each other, wherein the object comprises a first known pattern that defines a first system of coordinates;
one or more surgical instruments, wherein each surgical instrument has at least one tool marker (TM), each TM comprising a respective second known pattern that defines a second system of coordinates in which a position of relevant points and parts of the surgical instrument are known; and
an image processing system configured to:
receive the images, video, or images and video acquired by the camera, wherein the images, video, or images and video include the WM, one or more of the TMs, or the WM and one or more of the TMs in a field of view of the camera;
receive a 3d reconstruction of points of the anatomical part in the first coordinate system; and
output information derived from the 3d reconstruction to a display to assist execution of a clinical procedure using the one or more surgical instruments while keeping the WM, one or more of the TMs, or the WM and one or more of the TMs in the field of view of the camera, wherein the output comprises real-time guidance including a respective location of each surgical instrument in the first coordinate system.
0. 54. A method for computer-aided execution of surgery and diagnosis in an anatomical region comprising a rigid anatomical part, the method comprising:
attaching a world marker (WM) to a surface of the rigid anatomical part, the WM comprising a known pattern defining a first system of coordinates;
acquiring, by a free-moving camera, first intra-operative images, video, or images and video, while keeping the WM in a field of view of the camera,
wherein the free-moving camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f−1(u; k, ξ) with f−being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters;
receiving a 3d reconstruction of points of the anatomical part in the first coordinate system;
during a surgical procedure performed with one or more surgical instruments, wherein each surgical instrument has a respective tool marker (TM), each TM comprising a respective known pattern that defines a respective second system of coordinates in which positions of one or more points on the instrument are known, acquiring, by the camera, second intra-operative images, video, or images and video, while keeping the WM and the surgical instrument TMs in a field of view of the camera;
processing, by the image processing system, the second images, video, or images and video to determine locations of the surgical instruments in the first coordinate system; and
outputting information derived from the 3d reconstruction to a display to assist execution of a clinical procedure using the one or more surgical instruments while keeping the WM, one or more of the TMs, or the WM and one or more of the TMs in the field of view of the camera, wherein the output comprises real-time guidance including a respective location of each surgical instrument in the first coordinate system.
31. A method for computer-aided execution of surgery and diagnosis in anatomical regions comprising rigid, non-deformable parts or tissues, the method comprising:
(i) attaching a visual marker, referred to as the World Marker or WM, in an arbitrary location in the surface of the rigid anatomical part of interest, the visual marker comprising at least one planar facet with a known pattern whose system of coordinates defines a global or world system of coordinates;
(ii) acquiring, by a free-moving camera, intra-operative images and video of an operating field, articular joint, or anatomical cavity;
(iii) using a touch-probe, which is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PT is known, for pin-pointing points and/or outlining curves in the surface of the rigid anatomical part, while keeping both WM and TM of the probe in the field-of-View or FOV of the camera to be visible in the acquired images and video;
(iv) receiving, by an image processing system that may correspond to, or may be part of, a computer and/or any other computing device, the images and video acquired in (iii), which are processed for determining the location of the touch-probe and reconstructing in 3d the pin-pointed points and/or outlined curves that are stored in memory in global or world coordinates;
(v) using the 3d reconstruction results to perform measurements, make inferences, or overlay a surgical plan with the patient's anatomy, in which case a 3d registration method may be used for determining the transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to anatomy of interest;
(vi) performing the surgical procedure with the required instruments, where each instrument has at least one visual marker, also referred to as Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which the position of relevant points and parts in the instrument are known, while keeping both WM and TMs of the instruments in the FOV of the camera to be visible in the acquired images and video;
(vii) receiving, by the image processing system, the images and video acquired in (vi), that are processed such that instruments can be located in world coordinates and real-time guidance can be provided by relating these locations with 3d reconstructions results and/or overlaid surgical plan; and
(viii) outputting, by the image processing system, processing results and guidance information to a display, by either overlaying information in the input video using augmented reality (AR) techniques, or by animating models in a virtual reality (VR) environment;
where the camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x =f−1(u; k, ξ) with f =1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters.
32. A system for video based computer-aided surgery and diagnosis in anatomical regions of a patient comprising rigid, non-deformable anatomical parts or tissues comprising:
(i). a free-moving camera that is meant to acquire intra-operative images and video of an operating field, articular joint or anatomical cavity;
(ii). a visual marker, referred to as a World Marker or WM, which is an object that can be rigidly attached to a surface, and that comprises at least one planar facet with a known pattern having a system of coordinates that defines a global or world system of coordinates;
(iii). a touch-probe that is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PT is known;
(iv). one or more surgical instruments for surgical execution, where each surgical instrument has at least one visual marker, also referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which a position of relevant points and parts in the instrument are known; and
(v). an image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera (i), as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs processing results and guidance information to a display;
e####
wherein the system is configured to execute steps comprising:
(vi). attachment of the WM of (ii) in an arbitrary location in a surface of a rigid anatomical part of interest, such that the WM of (ii) and the rigid anatomical part of interest do not move one with respect to the other;
(vii). 3d reconstruction of points in the surface of the rigid anatomical part of interest by using the touch-probe of (iii) to pin-point the points in the surface while keeping both the WM and the TM of the probe in a field-of-View or FOV of the camera of (i) for determining the location of the touch-probe in the world system of coordinates to create 3d reconstruction results;
(viii). use the 3d reconstruction results to perform measurements, make inferences, or overlay a surgical plan with an anatomical region of the patient, in which case a 3d registration method may be used for determining a transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to the anatomical part of interest; and
(ix). assisted execution of a clinical procedure by using the surgical instruments of (iv) while keeping both WM and TMs of the surgical instruments in the FOV of the camera of (i) such that the surgical instruments can be located in world coordinates and real-time guidance can be provided by relating surgical instrument locations in world coordinates;
wherein an arbitrary point P is reconstructed in global or world coordinates by acquiring an image as described in (vii) that undergoes the following processing steps:
detect, identify, and estimate a 3d pose C of the WM in camera coordinates;
detect, identify, and estimate a 3d pose {circumflex over (T)} of the TM in camera coordinates;
reconstruct a point P that is in contact with the tip of the touch-probe by making
and
store 3d coordinates of point P in a memory.
1. A system for video based computer-aided surgery and diagnosis in anatomical regions of a patient comprising rigid, non-deformable anatomical parts or tissues comprising:
(i). a free-moving camera that is meant to acquire intra-operative images and video of an operating field, articular joint or anatomical cavity;
(ii). a visual marker, referred to as a World Marker or WM, which is an object that can be rigidly attached to a surface, and that comprises at least one planar facet with a known pattern having a system of coordinates that defines a global or world system of coordinates;
(iii). a touch-probe that is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PTT is known;
(iv). one or more surgical instruments for surgical execution, where each surgical instrument has at least one visual marker, also referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which a position of relevant points and parts in the instrument are known; and
(v). an image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera (i), as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs processing results and guidance information to a display;
wherein the system is configured to execute steps comprising:
(vi). attachment of the WM of (ii) in an arbitrary location in a surface of a rigid anatomical part of interest, such that the WM of (ii) and the rigid anatomical part of interest do not move one with respect to the other;
(vii). 3d reconstruction of points in the surface of the rigid anatomical part of interest by using the touch-probe of (iii) to pin-point the points in the surface while keeping both the WM and the TM of the probe in a field-of-View or FOV of the camera of (i) for determining the location of the touch-probe in the world system of coordinates to create 3d reconstruction results;
(viii). use the 3d reconstruction results to perform measurements, make inferences, or overlay a surgical plan with an anatomical region of the patient, in which case a 3d registration method may be used for determining a transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to the anatomical part of interest; and
(ix). assisted execution of a clinical procedure by using the surgical instruments of (iv) while keeping both WM and TMs of the surgical instruments in the FOV of the camera of (i) such that the surgical instruments can be located in world coordinates and real-time guidance can be provided by relating surgical instrument locations in world coordinates with the 3d reconstruction results and/or overlaid surgical plan;
where the camera in (i) is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f−1(u; k,ξ) with f−1 being an inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters.
33. A system for video based computer-aided surgery and diagnosis in anatomical regions of a patient comprising rigid, non-deformable anatomical parts or tissues comprising:
(i). a free-moving camera that is meant to acquire intra-operative images and video of an operating field, articular joint or anatomical cavity;
(ii). a visual marker, referred to as a World Marker or WM, which is an object that can be rigidly attached to a surface, and that comprises at least one planar facet with a known pattern having a system of coordinates that defines a global or world system of coordinates;
(iii). a touch-probe that is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PT is known;
(iv). one or more surgical instruments for surgical execution, where each surgical instrument has at least one visual marker, also referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which a position of relevant points and parts in the instrument are known; and
(v). an image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera (i), as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs processing results and guidance information to a display;
wherein the system is configured to execute steps comprising:
(vi). attachment of the WM of (ii) in an arbitrary location in a surface of a rigid anatomical part of interest, such that the WM of (ii) and the rigid anatomical part of interest do not move one with respect to the other;
(vii). 3d reconstruction of points in the surface of the rigid anatomical part of interest by using the touch-probe of (iii) to pin-point the points in the surface while keeping both the WM and the TM of the probe in a field-of-View or FOV of the camera of (i) for determining the location of the touch-probe in the world system of coordinates to create 3d reconstruction results;
(viii). use the 3d reconstruction results to perform measurements, make inferences, or overlay a surgical plan with an anatomical region of the patient, in which case a 3d registration method may be used for determining a transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to the anatomical part of interest; and
(ix).assisted execution of a clinical procedure by using the surgical instruments of (iv) while keeping both WM and TMs of the surgical instruments in the FOV of the camera of (i) such that the surgical instruments can be located in world coordinates and real-time guidance can be provided by relating surgical instrument locations in world coordinates;
wherein the assisted execution of the clinical procedure is accomplished by acquiring continuous video according to (ix), the video comprising a plurality of images, with each consecutive image undergoing the following processing steps in real-time:
a. detect, identify, and estimate a 3d pose C of the WM in camera coordinates;
b. detect, identify, and estimate a 3d pose {circumflex over (T)} of the TM of each of the one or more surgical instruments in camera coordinates;
c. compute the 3d pose T of TM in WM coordinates using the following equation:
T =C−1 {circumflex over (T)} d. map the information of surgical instrument calibration, that can be points, axes, or CAD models, into world coordinates using the rigid transformation T;
e. relate the surgical instrument calibration information with 3d data stored in memory to make measurements and inferences for the purpose of real-time guidance; and
f. display aiding features either by using Augmented reality (AR), in which case guidance information is overlaid in image or by animating a virtual reality (VR) 3d model.
2. The system of
3. The system of
4. The system of
5. The system of
6. The system of
7. The system of
where T(u) is the pattern template, I(u) is a current frame, Ni is a pattern region, and w is an image warping function given by the function composition
w(u; r, t)=f(x; k, ξ)o h(x; r, t)o f−1(u; k′, ξ′) with f being the projection function and h denoting a homography map that depends on a relative 3d pose r, t.
8. The system of
9. The system of
with {circumflex over (T)}−1 denoting an inverse of a rigid transformation {circumflex over (T)} that encodes a 3d pose r, t of the TM in camera coordinates that is determined from image information.
10. The system of
detect, identify, and estimate a 3d pose C of the WM in camera coordinates;
detect, identify, and estimate a 3d pose {circumflex over (T)} of the TM in camera coordinates;
reconstruct a point P that is in contact with the tip of the touch-probe by making
and
store 3d coordinates of point P in memory.
11. The system of
reconstructing a contour or a sparse mesh of a surface region, in which case the touch-probe is respectively used to outline the contour or randomly grasp the surface region, while the camera acquires a continuous sequence of images as described in (vii), and the processing steps are executed for each frame of the sequence.
12. The system of
13. The system of
detect, identify, and estimate the 3d pose C of the WM in camera coordinates;
map the 3d data from world coordinates into camera coordinates using C; and
project the 3d data into the image using the camera calibration and projection function f (augmented reality).
14. The system of
15. The system of
a. detect, identify, and estimate a 3d pose C of the WM in camera coordinates;
b. detect, identify, and estimate a 3d pose {circumflex over (T)} of the TM of each of the one or more surgical instruments in camera coordinates;
c. compute the 3d pose T of TM in WM coordinates using the following equation:
T=C−1{circumflex over (T)} d. map the information of surgical instrument calibration, that can be points, axes, or CAD models, into world coordinates using the rigid transformation T;
e. relate the surgical instrument calibration information with 3d data stored in memory to make measurements and inferences for the purpose of real-time guidance; and
f. display aiding features either by using Augmented reality (AR), in which case guidance information is overlaid in image, or by animating a virtual reality (VR) 3d model.
16. The system of
17. The system of
18. The system of
19. The system of
20. The system of
W′=C−1C′, with C and C′ being respectively the 3d pose of WM and WM' that are computed from image information.
21. The system of
22. The system of
with Pc being the position of the probe tip in camera coordinates and C−1 a 3d pose of the camera in WM coordinates that is estimated from image information.
23. The system of
with Ĝ denoting a rigid transformation that encodes a 3d pose r, t of a pattern of the calibration grid in camera coordinates t.
24. The system of
PT=LTΛ{circumflex over (T)}−1(Bx) with Λ denoting the operation of line intersection in 3d, Bx being the back-projection line of the point of incidence that is detected in the image, and {circumflex over (T)}−1 (Bx) denoting the line Bx expressed in the local system of coordinates of the visual marker (TM).
25. The system of
PT=ST+λdT. 26. The system of
27. The system of
28. The system of
29. The system of
30. The system of
0. 35. The system of claim 34, wherein the WM comprises one or more of a screw-like object with a flat head or facet, a nail-like object with a flat head or facet to be fixed by pressure, a needle-like object with a flat lateral facet for trans-dermic insertion, or a flat button-like object that is pulled by a thread or guide.
0. 36. The system of claim 34, wherein each pattern comprises recognizable fiducial points for estimating a plane-to-image homography that, given a camera calibration, can be factorized in a rotation and translation that map points in the coordinate system of the pattern into points in the coordinate system of the camera.
0. 37. The system of claim 34 wherein at least one of the known patterns is disposed on a planar surface.
0. 38. The system of claim 34, wherein at least one of the known patterns is disposed on a non-planar surface.
0. 39. The system of claim 34, further comprising a further surgical instrument, wherein the camera is rigidly coupled to the further surgical instrument.
0. 40. The system of claim 39, further comprising:
determining a location of a portion of the further surgical instrument in the first coordinate system according to an image that includes the WM and a known positional relationship between the portion of the further surgical instrument and the camera;
wherein the output further comprises the location of the further surgical instrument.
0. 41. The system of claim 34, wherein:
one of the surgical instruments is an end effector of a robotic tool; and
the image processing system is further configured to send a command to the robotic tool to control the end effector.
0. 42. The system of claim 41, wherein the camera is rigidly coupled to the end effector.
0. 43. The system of claim 34, further comprising a tablet computer, the tablet computer comprising the image processing system and the display.
0. 44. The system of claim 34, further comprising:
a point location device for locating points in the rigid anatomical part, wherein a TM is rigidly coupled to the point location device.
0. 45. The system of claim 44, wherein the point location device comprises one or more of:
a touch probe;
a laser pointer; or
a time-of-flight device.
0. 46. The system of claim 34, further comprising a point location device for locating points in the rigid anatomical part, wherein a TM is rigidly coupled to the point location device, wherein the image processing system is configured to reconstruct the known points in the first coordinate system by, for at least one image:
determining a 3d pose of the WM in camera coordinates;
determining a 3d pose of the point location device TM in camera coordinates; and
reconstructing a point captured by the point location device according to the 3d pose of the WM and the 3d pose of the point location device TM.
0. 47. The system of claim 34, wherein the image processing system is configured to determine the locations of the surgical instruments in the first coordinate system by:
determining a 3d pose of the WM in camera coordinates;
determining a 3d pose of the surgical instrument TMs in camera coordinates; and
determining a 3d pose of the surgical instrument TMs in the first coordinate system according to the 3d pose of the WM in camera coordinates and the 3d pose of the surgical instrument TMs in camera coordinates.
0. 48. The system of claim 34, wherein the camera is calibrated such that image points expressed in pixel coordinates are mapped into image points expressed in distance units according to the projection function, the camera intrinsic parameters, and the camera distortion parameters.
0. 49. The system of claim 48, wherein the image processing system is configured to:
calibrate the camera by acquiring, with the camera, a calibration image of a known calibration grid and determining, based on the calibration image, the camera intrinsic parameters and the camera distortion parameters.
0. 50. The system of claim 49, wherein the image processing system is configured to:
update the camera intrinsic parameters and the camera distortion parameters during the surgical procedure to compensate for one or more of differences in medium, rotation of lens scope with respect to camera-head, or variations in zoom.
0. 51. The system of claim 49, further comprising:
a point location device for locating points in the rigid anatomical part, wherein a TM is rigidly coupled to the point location device;
wherein the calibration image comprises the TM of the point location device and a portion of the point location device, wherein the image processing system is configured to:
calibrate the point location device by determining, based on the calibration image, a location of the portion of the point location device according to an inverse of a rigid transformation that encodes a 3d pose of the TM in camera coordinates.
0. 52. The system of claim 34, wherein receiving the 3d reconstruction comprises performing the 3d reconstruction according to one or more known points of the anatomical part and according to a location of the WM in the images, video, or images and video.
0. 53. The system of claim 34, wherein the WM or at least one of the TMs has multiple planar facets with multiple known patterns, where the locations of the multiple known patterns are known in a common coordinate system of the WM or the at least one of the TMs.
0. 55. The method of claim 54, wherein the image processing system is configured to determine the locations of the surgical instruments in the first coordinate system by:
determining a 3d pose of the WM in camera coordinates;
determining a 3d pose of the surgical instrument TMs in camera coordinates; and
determining a 3d pose of the surgical instrument TMs in the first coordinate system according to the 3d pose of the WM in camera coordinates and the 3d pose of the surgical instrument TMs in camera coordinates.
0. 56. The method of claim 54, wherein the camera is calibrated such that image points expressed in pixel coordinates are mapped into image points expressed in distance units according to the projection function, the camera intrinsic parameters, and the camera distortion parameters.
0. 57. The method of claim 56, further comprising:
calibrating the camera by acquiring, with the camera, a calibration image of a known calibration grid and determining, based on the calibration image, the camera intrinsic parameters and the camera distortion parameters.
0. 58. The method of claim 57, further comprising:
updating the camera intrinsic parameters and the camera distortion parameters during the surgical procedure to compensate for one or more of differences in medium, rotation of lens scope with respect to camera-head, or variations in zoom.
0. 59. The method of claim 57, further comprising:
while acquiring the first intra-operative images, locating one or more points in the rigid anatomical part using a point location device, wherein a tool marker (TM) is rigidly coupled to the point location device, wherein the TM comprises a known pattern that defines a third system of coordinates in which a position of a portion of the point location device is known;
wherein receiving the 3d reconstruction comprises performing the 3d reconstruction according to the one or more located points and according to a location of the WM in the images, video, or images and video;
wherein the calibration image comprises the TM of the point location device and the portion of the point location device, the method further comprising:
calibrating the point location device by determining, based on the calibration image, a location of the portion of the point location device according to an inverse of a rigid transformation that encodes a 3d pose of the TM in camera coordinates.
0. 60. The method of claim 54, further comprising:
while acquiring the first intra-operative images, locating one or more points in the rigid anatomical part using a point location device, wherein a TM is rigidly coupled to the point location device, wherein the TM comprises a known pattern that defines a third system of coordinates in which a position of a portion of the point location device is known;
wherein receiving the 3d reconstruction comprises performing the 3d reconstruction according to the one or more located points and according to a location of the WM in the images, video, or images and video.
0. 61. The method of claim 60, wherein the point location device comprises one or more of:
a touch probe;
a laser pointer; or
a time-of-flight device.
0. 62. The method of claim 60, wherein the image processing system is configured to reconstruct the located points in the first coordinate system by, for at least one image:
determining a 3d pose of the WM in camera coordinates;
determining a 3d pose of the point location device TM in camera coordinates; and
reconstructing a point P located by the point location device according to the 3d pose of the WM and the 3d pose of the point location device TM.
0. 63. The method of claim 54, wherein the WM comprises one or more of a screw-like object with a flat head or facet, a nail-like object with a flat head or facet to be fixed by pressure, a needle-like object with a flat lateral facet for trans-dermic insertion, or a flat button-like object that is pulled by a thread or guide.
0. 64. The method of claim 54, wherein each pattern comprises recognizable fiducial points for estimating a plane-to-image homography that, given a camera calibration, can be factorized in a rotation and translation that map points in the coordinate system of the pattern into points in the coordinate system of the camera.
0. 65. The method of claim 54, wherein at least one of the known patterns is disposed on a planar surface.
0. 66. The method of claim 54, wherein at least one of the known patterns is disposed on a non-planar surface.
0. 67. The method of claim 54, further comprising:
determining a location of a portion of a further surgical instrument, the camera rigidly coupled to the further surgical instrument, according to an image that includes the WM and a known positional relationship between the portion of the further surgical instrument and the camera.
0. 68. The method of claim 54, wherein:
one of the surgical instruments is an end effector of a robotic tool; and
the image processing system is further configured to send a command to the robotic tool to control the end effector.
0. 69. The method of claim 68, wherein the camera is rigidly coupled to the end effector.
0. 70. The method of claim 54, wherein the image processing system and the display comprise a tablet computer.
0. 71. The method of claim 54, wherein receiving the 3d reconstruction comprises performing the 3d reconstruction according to one or more known points of the anatomical part and according to a location of the WM in the images, video, or images and video.
0. 72. The method of claim 54, wherein the WM or at least one of the TMs has multiple planar facets includes multiple known patterns, where the locations of the multiple known patterns are known in a common coordinate system of the WM or the at least one of the TMs.
0. 74. The system of claim 34, wherein the image processing system is further configured to determine the 3d reconstruction of the points of the anatomical part by determining locations of points of the anatomical part.
0. 75. The system of claim 34, wherein the image processing system is further configured to determine the 3d reconstruction of the points using one or more images captured by the free-moving camera.
0. 76. The system of claim 75, wherein the image processing system is further configured to determine the 3d reconstruction of the points using one or more images each comprising the WM and a point, of the anatomical part, to be located.
0. 77. The system of claim 76, wherein the image processing system is further configured to determine the 3d reconstruction of the points using one or more images each comprising the WM and an illumination by light of a light source of the point to be located.
0. 78. The system of claim 74, wherein the image processing system is further configured to determine the 3d reconstruction of the points using a light source and/or a time-of-flight device.
0. 79. The system of claim 78, further comprising:
the light source to illuminate points of the anatomical part for locating the points for 3d reconstruction.
0. 80. The system of claim 78, wherein:
the system further comprises a light source to emit structured light; and
the image processing system is further configured to determine a location of a point on the anatomical part by analyzing at least one image of a point of the anatomical part illuminated with the structured light.
0. 81. The system of claim 80, wherein the light source is arranged to emit collimated light.
0. 82. The system of claim 81, wherein the light source is configured to emit a collimated beam of visible light.
0. 83. The system of claim 80, wherein the light source is a laser.
0. 84. The system of claim 74, wherein the image processing system is further configured to determine the 3d reconstruction of the points using a touch probe.
0. 85. The system of claim 84, further comprising:
the touch probe, wherein the touch probe has a TM.
0. 86. The system of claim 34, further comprising:
a second surgical instrument, wherein the second surgical instrument is a guide.
0. 87. The system of claim 86, wherein the guide comprises a TM.
0. 88. The system of claim 34, wherein the image processing system is configured to monitor a location of a portion of a surgical instrument in a field of view of the free-moving camera.
0. 89. The system of claim 88, wherein the image processing system is configured to monitor a location of a distal end of the surgical instrument in the field of view of the free-moving camera.
0. 90. The system of claim 89, wherein the image processing system is configured to monitor a location of a tip of the surgical instrument in a field of view of the free-moving camera.
0. 91. The system of claim 90, wherein the surgical instrument is a touch probe and the tip of the surgical instrument is a tip of the touch probe.
0. 92. The system of claim 88, wherein the surgical instrument is rigidly coupled to the free-moving camera.
0. 94. The system of claim 93, wherein the WM is a planar object.
0. 95. The system of claim 93, wherein the WM comprises a known pattern.
0. 96. The system of claim 93, The system of claim 34, wherein the image processing system is further configured to determine the 3d reconstruction of the points of the anatomical part by determining locations of points of the anatomical part.
0. 97. The system of claim 93, wherein the image processing system is further configured to determine the 3d reconstruction of the points using one or more images captured by the free-moving camera.
0. 98. The system of claim 93, wherein the image processing system is configured to monitor a location of a portion of a surgical instrument in a field of view of the free-moving camera.
0. 99. The system of claim 98, wherein the surgical instrument is rigidly coupled to the free-moving camera.
0. 100. The system of claim 99, wherein the surgical instrument is a touch probe.
0. 101. The system of claim 93, further comprising:
a surgical instrument, wherein the surgical instrument is a guide.
0. 102. The system of claim 101, wherein the guide comprises a tool marker (TM).
|
This patent application
Let's now assume that the tool or instrument is a calibrated touch-probe such that PT is the vector of 3D coordinates of its tip in the TM reference frame. The surgeon can reconstruct a point of interest in the bone surface by touching it with the probe and acquiring a frame where both WM and TM are visible. This enables computation of the pose T of the probe and the obtaining of the point of interest P expressed in world coordinates (
The process above can be applied to successive frames in order to reconstruct a curve in the bone surface. In this embodiment the surgeon outlines the contour of interest while keeping both WM and TM in the Field-of-View (FOV) of the free-moving camera. This enables the obtaining of successive P estimates that define the desired 3D curve. Since 3D reconstruction results are stored in World Marker coordinates, the action of outlining can be stopped and resumed at any time. If the process is interrupted for any reason, it suffices for the camera to see again the WM for all the 3D information to be restored without having to repeat the tedious touching process (
The 3D reconstruction results, that can either be points, contours, or sparse surface meshes, can be used for the purpose of measuring, estimating shape, or overlying a pre-operative plan in the actual patient anatomy (3D registration). This pre-operative plan can be a set of rules using anatomical landmarks, a statistical 3D model of the anatomy of interest, or an actual 3D image of the organ (e.g. CT Scan) augmented with guidance information inserted by the surgeon (surgical plan). Let's assume the latter for illustrative purposes (
The clinical execution might require, in one embodiment, multiple different instruments—such as guides, drills, shavers, saws, burrs, etc.—that can either be used in sequence or simultaneously. Each one of these instruments is assumed to have a Tool Marker (TM) attached that defines a local system of coordinates where the instrument's relevant parts—such as tip, symmetry axis, or even complete CAD model—are represented. The system processes each frame with the objective of detecting, identifying, and estimating the 3D pose of every TM that is in the FOV of the camera. If the WM is also visible in image, then it is possible to determine the pose of the camera C, locate the instruments in the world coordinate system, relate their poses T with the 3D information stored in the WM reference frame, and ultimately provide real-time assistance to the surgeon (
Thus, the last stage of VTIAC consists of assisting the surgeon by performing continuous processing of the video for estimating in real-time the 3D pose of instruments with respect to patient anatomy and/or surgical plan represented in WM coordinates. The assistance can take multiple forms depending on a specific task and a preferred user interface. Possibilities include overlaying guidance information in video using Augmented Reality (AR), using computer graphics to animate the motion of instruments in a Virtual Reality (VR) environment showing the patient's anatomy and/or surgical plan, or controlling the action of actuators in the case of procedures assisted by robotic systems such as the Mako® or the Navio® robots.
3. Overview of Methods, Apparatus and Initial Calibration Requirements
This section overviews the methods and apparatus that are required to perform computer-aided surgery using VTIAC. The apparatus includes:
Since the VTIAC uses images for measurements and 3D inference, the free-moving camera must be calibrated at all times during the procedure such that 2D image points u, represented in pixel coordinates, can be mapped into 2D points x (or back-projection directions) represented in the metric system of coordinates of the camera. The calibration includes determining the vector of parameters k and ξ of the back-projection function f−1 (the inverse of the projection function f) where k comprises the so-called intrinsic parameters—focal length, principal point, aspect ratio, and skew—and ξ stands for the radial distortion parameters.
x=f−1(u; k, ξ) (equation 4)
The camera can either be pre-calibrated from factory, using any standard method in literature, or calibrated in the Operating Room (OR) just before starting the procedure. The latter is especially recommendable for the case of arthroscopic cameras, or any other camera with exchangeable optics. The calibration in the OR can be quickly accomplished by acquiring one image of a known calibration pattern from an arbitrary viewpoint, as described in U.S. Patent Publication No. 2014/0285676, which is incorporated by reference in its entirety. If the camera parameters change during operation because the surgeon rotates the lens scope and/or varies the optical zoom, then the initial calibration may be updated at every frame time using the techniques described in U.S. Patent Publication No. 2014/0285676 and Patent Publication WO2014054958, both of which are incorporated by reference in their entireties. The camera calibration must also take into account the medium of operation that, in the case of arthroscopy, is a wet medium. In this situation the initial single image calibration can either be carried in wet medium, or performed in air followed by compensating for the difference in the refractive index of air and water-based medium.
3.2 World Marker (WM) and Tool Markers (TMs).
The surgeon starts by fixing the World Marker (WM) to the bone surface. The WM can be any object comprising at least one planar facet with a known pattern that can be secured (e.g., glued), printed or engraved, and that can be recognized in images; that is small enough to be inserted into the anatomical cavity (e.g., up to 5 mm diameter in the case of arthroscopy); and that can be mechanically attached to the surface such that bone and marker do not move with respect to each other.
A non-exhaustive list of objects that can be used as WM includes: a screw-like object with a flat head or facet (
The touch-probe in (iii) and the surgical tools in (iv) are instrumented with a visual marker (the Tool Marker or TM), which can either be originally built-in at manufacturing time, or rigidly attached by the user (
3.3 Tool Calibration in the Operating-Room (OR)
If the tool calibration includes finding the coordinates PT of a particular tool point in the TM reference frame (e.g. the tip of the touch probe (iii)), then the operation can be quickly carried simultaneously with the initial calibration of the camera without requiring the acquisition of additional calibration frames. As described, e.g., in U.S. Patent Publication No. 2014/0285676, the camera calibration can be accomplished by acquiring a single image of a known grid or checkerboard pattern. This enables recovering the intrinsic parameters k, the radial distortion parameters ξ, and the rigid transformation Ĝ that maps coordinates in the grid reference frame into coordinates in the camera reference frame. Thus, if the tool tip is placed in a pre-defined point PG that is known in grid coordinates, and the calibration image is such that TM is visible, then it is possible to estimate the 3D pose {circumflex over (T)} of the tool marker from image information and obtain the TM coordinates of the tool tip by applying the formula below (
The tool calibration of the surgical instruments (iv) can either consist in determining the location of a point, a line or axis, or a CAD model in the coordinate system of the TM attached to the particular instrument. This can be accomplished with the help of the calibrated camera and touch-probe using a method similar to the one used for 3D reconstruction on the bone surface, but where the role of the WM is replaced by the TM of the instrument (
3.4 Alternatives and Extensions in the Physical Configuration of Visual Markers
The visual marker used in the WM of (ii) and in the TMs of (iii) and (iv) can comprise a single plane facet with a known pattern as assumed so far, or multiple plane facets with each facet having its own pattern that can be secured (e.g., glued), printed, or engraved, and where the location of each planar pattern is known in a common local coordinate system of the visual marker. The advantage of having multiple planar patterns facing different directions is to extend the range of viewing positions and orientations from which the marker can be observed by the camera for estimating the relative 3D pose (
Alternatively, the visual marker can be non-planar, in which case it should comprise n≥3 points with known coordinates in the local reference frame of the marker, with these points being such that they can be detected and identified in image in order to allow estimation of the relative pose by applying a Perspective-n-Point (PnP) method.
4. Estimation of Rotation and Translation (the 3D Pose) of a Known Planar Pattern from Image Information.
The small visual markers that are attached to instruments, tools, and anatomy of interest play a fundamental role in VTIAC being key-enablers for using the camera as a measuring device for determining 3D pose. As discussed, the visual marker can have different topological configurations but, for the sake of simplicity and without compromising generality, it will be assumed that the visual marker is a planar surface with a known pattern.
This planar pattern should be such that it has a local system of coordinates, it is amenable to be detected and uniquely identified from its image projection, and it has fiducial points that can be accurately detected in image for estimating the plane-to-image homography H from point correspondences. A point correspondence is the association between a point in the pattern p expressed in local coordinates and its projection x represented in camera coordinates. The homography H is a projective transformation that maps the former into the latter, and that can be linearly estimated from N≥4 point correspondences. The homography encodes the rotation and translation between pattern and camera coordinate systems, which means that the factorization of H provides the 3D pose of the pattern in the camera reference frame.
There are several pattern designs that meet the above mentioned conditions. It will be assumed, without compromising generality, that the planar patterns are similar to the CalTag checkerboard patterns, where the quadrilateral shape and high contrast enable fast detection, the sharp corners provide accurate point correspondences, and a bitmap binary code allows visual identification (
One possibility for improving accuracy and robustness of 3D pose estimation is to correct radial distortion via software, before running the processing pipeline for detection, identification, and homography/pose estimation. However, this has several drawbacks, such as the computational effort in warping the entire frame, and the fact that interpolation also introduces artifacts that degrade the accuracy of geometric estimation.
Since radial distortion has a relatively small impact in pattern detection, this disclosure provides an alternative approach based in photo-geometry. The approach includes using standard methods for detection, identification, and initial estimation of pattern rotation r0 and translation t0, followed by refining the 3D pose estimate by minimizing the photo-geometric error in aligning the current pattern image with its template using a warping function that takes into account the non-linear distortion.
Let C0 be the initial 3D pose estimate of the planar pattern in camera coordinates. The objective is to determine the pose update Δ, that encodes the increments in rotation δR and in translation δt, such that the photo-geometric error ε1 is minimized (
where T(u) is the pattern template, I(u) is the current frame, Ni is the image region comprising the pattern, and w is the image warping function (
w(u; r, t)=f(x: k, ξ)o h(x; r, t)o f−1 (u; k′, ξ′) (equation 7)
with h being the homography map that depends on the relative 3D pose r and t, and f denoting the projection function of the camera that encodes the effect of radial distortion, as described, e.g., in Patent Publication WO/2014054958. Since the template can be understood as a synthetic, fronto-parallel image of the planar pattern (
C=ΔC0 (equation 8)
The iterative minimization of the photo-geometric error εi can be carried using different optimization schemes available in literature such as forward composition, inverse composition, or efficient second order minimization, which requires some changes in formulation and parametrization in SE(3). The formulation can also be extended to be resilient to changes in illumination.
5. 3D Measurement and Reconstruction using VTIAC.
Section 4 describes a method for estimating the 3D pose of a planar visual marker in camera coordinates. Let's consider two of these markers such that one is attached to the anatomy of interest (WM), and the other is attached to a calibrated touch probe (TM).
For reconstructing an arbitrary point P in world coordinates the surgeon places the tip of the probe in the point, positions the camera such that both WM and TM are in the FOV, and commands the system to acquire an image that is processed as follows (
The approach can be extended to obtain a 3D contour or a sparse 3D reconstruction of a surface region, in which case the surgeon uses the touch probe to respectively outline the contour or randomly grasp the surface, while the camera acquires continuous video and steps above are executed for each frame (
The 3D reconstruction results are stored in memory in world coordinates, which means that they can be overlaid in images whenever the WM is in the camera FOV by performing the following steps at each frame time instant (
The ability of VTIAC to reconstruct and store in memory points, curves, and regions in the anatomy of interest (e.g. bone) has a multitude of purposes and/or possible clinical applications. A non-exhaustive list includes:
As stated, the reconstruction results can also be used as input in standard 3D registration methods for aligning or overlying a computational model with the current patient's anatomy. Such methods estimate the rigid transformation M that maps points PM in the model into corresponding points P in the intra-operative reconstruction obtained with VTIAC (
6. Assisted Execution of the Clinical Procedure using VTIAC.
So far we have shown how to obtain relevant 3D data in the common coordinate system of the WM that may consist in reconstruction results, measurements and other types of 3D inferences, or the registration of surgical plan against patient's anatomy. The term ‘surgical plan’ is employed in a broad sense and can mean, among other things, a set of rules based on anatomical landmarks, e.g. placing the femoral tunnel of the ACL at ⅓ the length of the notch ceiling measured from its posterior end; the fitting of a statistical model of an anatomy or pathology, e.g. the shape model of CAM femuro-acetabular impingement; or a pre-operative image of the targeted anatomy that can, or cannot, be augmented with guidance information, e.g. a CT scan annotated by the surgeon using a 3D planning software. This section describes how VTIAC can combine this 3D data with real-time 3D pose estimation of surgical instruments to provide intra-operative navigation features.
Let the surgical instrument—that can be a needle, guide, drill, shaver, saw, burr, or any other object required for proper clinical execution—have a TM attached. The marker defines a local reference frame where the position of a point, axis, or CAD model of the tool is known (calibrated tool). Navigation is accomplished by executing the following processing steps at every frame time instant:
VTIAC navigation also works for the case of multiple instruments being used in simultaneous, in which case each instrument has its own TM enabling parallel detection, identification, and estimation of 3D pose T.
The aiding features can take multiple forms depending on the particular task and/or surgical procedure. A non-exhaustive list of these features includes:
The disclosure has considered that camera and tool or instrument are two entities with independent motions. There are situations for which it might be advantageous to assemble the camera in the tool or instrument such that the two entities become a single rigid body. The assembly, that is henceforth referred as a Camera Tool or CamT, must be calibrated such that the position of the tool tip, axis of interest, or CAD model of the tool or instrument, is known in the reference frame of the camera. Depending on the particular clinical application the camera can be mounted in a multitude of possible tools ranging from a touch-probe to an impactor for cup placement during hip arthroplasty, passing by burrs and drills. In this setup where camera and tools are physically attached, their relative 3D pose is known, and as long as the camera sees the WM, it is possible to determine the 3D pose of the tool in the global system of coordinates of WM.
7.2 Single-image Calibration of CamT:
The CamT described above can either be pre-calibrated from factory, or calibrated in the OR from a single image of a known grid or checkerboard pattern. In this case the surgeon acquires the calibration frame by positioning the camera such that the pattern is visible in image and the tool tip touches a particular point PG whose coordinates are known in the coordinate system of the grid (
7.3 Contactless Probe using a Laser Pointer:
Section 5 discloses a method for 3D reconstruction where the surgeon uses a calibrated touch-probe to pinpoint points of interest while the camera observes both the WM and the TM of the tool. There might be situations for which touching a particular location in the anatomy is difficult or even unfeasible. Examples include situations of limited access or poor maneuverability where the touch-probe cannot reach a particular location without occluding the WM. It is now disclosed an alternative probe that can replace the conventional touch-probe in the task of performing 3D reconstruction using VTIAC, and that has the advantage of avoiding the need of physical contact.
This alternative probe, henceforth referred as contactless probe, consists in a laser pointer that emits a collimated beam of visible light. The pointer has a visual marker attached—the Tool Marker or TM—and it is assumed to be calibrated such that the position of the line LT defined by the beam is known in TM coordinates.
For reconstruction the surgeon directs the laser pointer such that the beam becomes incident on the point of interest, and uses the camera to acquire an image where WM, TM, and point of light incidence are visible. The point is reconstructed in 3D by intersecting the line LT of the beam with the back-projection line Bx of the image point x where the point of light incidence is projected (
Contactless 3D reconstruction can also be accomplished using an Active Contactless Probe consisting in a Laser Rangefinder (LRF), or other equivalent device or technology relying on Time-of-Flight (ToF) principles, that is able to measure distances λ along the direction of the beam line LT. The LRF has a visual marker attached and it is assumed to be calibrated such that the origin and unit direction of measurement, that are respectively ST and dT, are known in the local reference frame of TM. For 3D reconstruction the surgeon orients the LRF such that the beam becomes incident with the point of interest in the anatomy, and acquires in a synchronous manner the distance measurement λ and an image where both WM and TM are visible. The point of interest can be outside the camera FOV (
PT=ST+λdT
7.5 Using Multiple WMs to Extend the Range of Operation:
The World Marker or WM works as a global reference, which means that it must be viewed by the camera whenever the surgeon wants to use VTIAC for reconstruction or guidance purposes. There might be situations for which keeping the WM in the camera FOV can be difficult to accomplish in practice, either because the camera has a limited FOV, or because the region to cover is simply too broad or wide. This problem is solved by using multiple markers as shown in
W′=C−1C′
Since W′ enables to map information from WM into WM′ and vice-versa, it suffices for the camera to see one of the markers for the reconstruction and guidance functionalities of VTIAC to be readily available. The region of operation can be further extended by placing additional markers and repeating the step above to register them in world coordinates.
7.6 Using VTIAC with a Surgical Robot
Section 6 discloses a method for using VTIAC to assist the execution of a clinical procedure where the guidance information is provided by either overlying info in the images or video (AR), or by animating a VR model of anatomy and tools. In addition, VTIAC can also be used to guide or control the action of a surgical robot (
A surgical system like the Navio® robot relies on conventional OT for determining in real-time the 3D pose between the robotized tool and patient's anatomy and/or surgical plan. VTIAC can be used as an alternative to conventional OT for providing the kinematic feedback required to control the robot in closed loop (
8. Example of Application of VTIAC for Arthroscopic Reconstruction of Anterior Cruciate Ligament (ACL) in the Knee
This section discloses an embodiment of VTIAC based-navigation for Reconstruction of Anterior Cruciate Ligament (ACL) in the Knee, which can also be generalized for other arthroscopic procedures such as in the shoulder or hip.
ACL tear is a common pathology for which arthroscopy is the standard treatment (e.g., >300000 cases per year worldwide). The procedure includes replacing the torn ACL by a substitution graft that is pulled into the joint through a tunnel opened with a drill. Placing this tunnel in the correct anatomical position is crucial for the knee to fully recover its functionality. One technique is the transtibial (TT) approach that opens the tunnel in a single step by drilling from the bottom of the tibia plate till entering into the femur notch. Recent studies show that in about 39% of the cases TT fails in positioning the tunnel at the femoral end, and that much better results can be accomplished using the anteromedial (AM) approach. Unfortunately, AM is used in about 15% of the cases because it is more difficult to execute and increases the risk of critically short tunnel or blowout of the posterior femur wall. Intra-operative navigation can help in disseminating the AM approach by dramatically decreasing the execution risk and complexity. VTIAC may be applied to accomplish this intra-operative navigation by indicating the location in the femur notch where to open the tunnel (the ligament footprint) and by guiding the angular orientation of drilling.
In a possible design of the navigated procedure the surgeon starts by calibrating the arthroscopic camera and by attaching the WM in the medial side of the inter-condyle region (
The orientation for opening the tunnel may be determined by registering a statistical model of the femur bone. For this purpose, the surgeon uses the touch probe to reconstruct the boundary contours of the inter-condyle region (
9. Example of Application of VTIAC for Guiding the Placement of Pedicle Screws (PPS) During Open Surgery of Spine.
This section discloses an embodiment of VTIAC based-navigation for Placing Pedicle Screws (PPS) during spine surgery, which can also be generalized to other open procedures where a rigid surface is exposed, such as total hip replacement, total knee replacement, open shoulder surgery and implant placement in dentistry.
Although VTIAC always requires a video input, its use is not limited to arthroscopy. The framework can also be applied to open orthopedic procedures, such as knee/hip arthroplasty or spine surgery, as far as a camera is employed to observe incision and relevant anatomy. The camera can either be a generic handheld camera (
There are several traumas and pathologies of the spine whose treatment passes by a surgery for vertebra fusion. The procedure includes placing screws in two consecutive vertebras for keeping in position a metallic rod that prevents intervertebral motion. Each screw must be carefully inserted along the vertebra pedicle otherwise it can irremediably damage the spinal medulla or a vital blood vessel. The dominant technique for Placing Pedicle Screws (PPS) is the so-called “free-hand” approach, in which the surgeon relies in his experience and knowledge to insert the screw while occasionally using fluoroscopy to confirm the correct positioning. Since this process is risky and error prone, several manufacturers developed navigation systems for PPS where a pre-operative 3D plan is overlaid with the patient anatomy in the OR using opto-tracking. In this case the surgeon uses a pre-operative model of the vertebra (e.g. CT-Scan or MRI) to specify the 3D line along which the screw must be inserted, as well as the depth of insertion. The model and the surgeon specifications are henceforth referred as the pre-operative 3D plan. This section describes how VTIAC can be applied to accomplish intra-operative navigation after planning.
In the OR, and after opening an incision for partial or total exposition of the vertebra, the surgeon starts by rigidly attaching a visual marker (WM) to the bone surface. This marker plays the role of World Marker (WM) and is placed in an arbitrary position decided by the surgeon. The next step is to overlay the pre-operative plan with patient's anatomy in the OR, which passes by reconstructing points and/or curves on the vertebra surface to be used as input in a suitable 3D registration algorithm.
One possibility is to perform the 3D registration using a set of fiducial points or landmarks in the anatomy. In this case the system indicates a succession of landmark points to be reconstructed that are pin-pointed in by the surgeon using the touch-probe (
After registration, the VTIAC is able to overlay the 3D pre-operative plan in the intra-operative video, as well as the tip, axis, or CAD model of the tool, whenever WM and TM are respectively in the FOV of the camera (
The VTIAC can then project the guidance information into the AR view, such as the angle of the tool relatively to the planned direction (
10. Application of VTIAC for Intra-operative Guidance in Other Clinical Procedures
VTIAC can be applied for intra-operative navigation in several other clinical procedures. A non-exhaustive list of possibilities include:
Arthroscopic reconstruction of Posterior Cruciate Ligament (PCL): The PCL is a ligament in the knee joint that connects the posterior intercondylar area of the tibia to the medial condyle of the femur. In a similar manner to the ACL, the PCL reconstruction consists in replacing the torn ligament by a substitution graft that is pulled inside the joint through a tunnel opened with a drill. VTIAC can be applied to guide the placement of these tunnels both in tibial and femoral sides.
Arthroscopic Resection of Femuro-Acetabular Impingement (FAI): FAI occurs when the ball shaped femoral head rubs abnormally in the acetabular socket, which in about 91% of the cases is caused by an excess of bone tissue in the femur head-neck that creates a bump known as CAM impingement. The treatment is surgical and consists in removing the CAM to restore the ball shape to the femur-head. To accomplish this objective the surgeon uses a CT-scan of the femur to study the CAM position and plan the extension of resection. This plan is then mentally transposed for the execution in the OR, which is a very error prone process. VTIAC can be applied to enforce the pre-planning by overlying the annotated 3D model with the patient's femur in order to safely guide the surgeon. After model registration the CAM footprint can be overlaid in the arthroscopic video using AR techniques and the system can inform the surgeon about the quantity of the bone tissue to remove at every instant.
Arthroscopic assessment and diagnosis of confocal defects in cartilage: Confocal defects are damages in the articular cartilage that can be repaired by filling the holes or craters with a bio-compatible material. This operation often requires placing in the hole or crater an rigid support structure called scaffolder. VTIAC can be used for measuring and determining the shape of confocal defects, as well as to guide the placement of these scaffolds.
Total hip replacement (THR): THR is an open surgical procedure for replacing the hip joint by an implant. The implant consists in a cup, that replaces acetabulum in the pelvic bone, and in a stem with a sphere that replaces the femural head. VTIAC can be applied to guide the placement of the cup such that it is inserted with optimal angular orientation, as well as to define the cut plane in the femural neck to remove the head and insert the stem with sphere.
Total Knee Replacement and Unicompartmental Knee Replacement: Knee arthroplasty is an open surgical procedure for replacing total or part of the knee joint by an implant (total or unicompartmental knee replacement). VTIAC can be applied to guide the surgeon in cutting the femural condyle and placing the implant.
Shoulder Joint Replacement: This is another open surgical procedure for replacing in total or in part the shoulder joint by an implant. VTIAC can be applied in assisting the surgeon in several steps of the execution such as indicating the plane of cut to remove humeral head, or guiding the reaming of humeral shaft and/or glenoid.
Placement of dental implants in Prosthodontics: VTIAC can be applied in dental surgery for placing an implant in the maxilar bone as planned in a pre-operative Cone Beam CT (CBCT) of the patient. In this case the WM is rigidly attached to a tooth, the CBCT is overlaid with patient's anatomy by using VTIAC features for 3D reconstruction, and the system provides intra-operative guidance for inserting the implant through any of the AR and VR features that have been described in the ACL and PPS examples.
11. Additional Notes and Remarks
As shown in
Programming and/or loading executable instructions onto memory 508 and processor 502 in order to transform the image processing system 500 into a non-generic particular machine or apparatus that applies VTIAC to surgical procedures is well-known in the art. Implementing instructions, real-time monitoring, and other functions by loading executable software into a computer and/or processor can be converted to a hardware implementation by well-known design rules and/or transform a general-purpose processor to a processor programmed for a specific application. For example, decisions between implementing a concept in software versus hardware may depend on a number of design choices that include stability of the design and numbers of units to be produced and issues involved in translating from the software domain to the hardware domain. Often a design may be developed and tested in a software form and subsequently transformed, by well-known design rules, to an equivalent hardware implementation in an ASIC or application specific hardware that hardwires the instructions of the software. In the same manner as a machine controlled by a new ASIC is a particular machine or apparatus, likewise a computer that has been programmed and/or loaded with executable instructions may be viewed as a non-generic particular machine or apparatus.
In addition,
As shown in
Memory 904 interfaces with computer bus 902 so as to provide information stored in memory 904 to CPU 912 during execution of software programs such as an operating system, application programs, device drivers, and software modules that comprise program code, and/or computer executable process steps, incorporating functionality described herein, e.g., one or more of process flows described herein. CPU 912 first loads computer executable process steps from storage, e.g., memory 904, computer readable storage medium/media 906, removable media drive, and/or other storage device. CPU 912 can then execute the stored process steps in order to execute the loaded computer-executable process steps. Stored data, e.g., data stored by a storage device, can be accessed by CPU 912 during the execution of computer-executable process steps.
Persistent storage, e.g., medium/media 906, can be used to store an operating system and one or more application programs. Persistent storage can also be used to store device drivers, such as one or more of a digital camera driver, monitor driver, printer driver, scanner driver, or other device drivers, web pages, content files, playlists and other files. Persistent storage can further include program modules and data files used to implement one or more embodiments of the present disclosure.
A network link typically provides information communication using transmission media through one or more networks to other devices that use or process the information. For example, the network link may provide a connection through a local network to a host computer or to equipment operated by a Network or Internet Service Provider (ISP). ISP equipment in turn provides data communication services through the public, worldwide packet-switching communication network of networks now commonly referred to as the Internet.
A computer called a server host connected to the Internet hosts a process that provides a service in response to information received over the Internet. For example, server host hosts a process that provides information representing video data for presentation at display 910. It is contemplated that the components of system 900 can be deployed in various configurations within other computer systems, e.g., host and server.
At least some embodiments of the present disclosure are related to the use of computer system 900 for implementing some or all of the techniques described herein. According to one embodiment, those techniques are performed by computer system 900 in response to processing unit 912 executing one or more sequences of one or more processor instructions contained in memory 904. Such instructions, also called computer instructions, software and program code, may be read into memory 904 from another computer-readable medium 906 such as storage device or network link. Execution of the sequences of instructions contained in memory 904 causes processing unit 912 to perform one or more of the method steps described herein. In alternative embodiments, hardware, such as ASIC, may be used in place of or in combination with software. Thus, embodiments of the present disclosure are not limited to any specific combination of hardware and software, unless otherwise explicitly stated herein.
The signals transmitted over network link and other networks through communications interface, carry information to and from computer system 900. Computer system 900 can send and receive information, including program code, through the networks, among others, through network link and communications interface. In an example using the Internet, a server host transmits program code for a particular application, requested by a message sent from computer, through Internet, ISP equipment, local network and communications interface. The received code may be executed by processor 902 as it is received, or may be stored in memory 904 or in storage device or other non-volatile storage for later execution, or both.
For the purposes of this disclosure a module is a software, hardware, or firmware (or combinations thereof) system, process or functionality, or component thereof, that performs or facilitates the processes, features, and/or functions described herein (with or without human interaction or augmentation). A module can include sub-modules. A module, or software components of a module, may be stored on a computer readable medium for execution by a processor. Modules may be integral to one or more servers, or be loaded and executed by one or more servers. One or more modules may be grouped into an engine or an application.
At least one embodiment is disclosed and variations, combinations, and/or modifications of the embodiment(s) and/or features of the embodiment(s) made by a person having ordinary skill in the art are within the scope of the disclosure. Alternative embodiments that result from combining, integrating, and/or omitting features of the embodiment(s) are also within the scope of the disclosure. Where numerical ranges or limitations are expressly stated, such express ranges or limitations may be understood to include iterative ranges or limitations of like magnitude falling within the expressly stated ranges or limitations (e.g., from about 1 to about 10 includes, 2, 3, 4, etc.; greater than 0.10 includes 0.11, 0.12, 0.13, etc.). The use of the term “about” means ±10% of the subsequent number, unless otherwise stated.
Use of the term “optionally” with respect to any element of a claim means that the element is required, or alternatively, the element is not required, both alternatives being within the scope of the claim. Use of broader terms such as comprises, includes, and having may be understood to provide support for narrower terms such as consisting of, consisting essentially of, and comprised substantially of Accordingly, the scope of protection is not limited by the description set out above but is defined by the claims that follow, that scope including all equivalents of the subject matter of the claims. Each and every claim is incorporated as further disclosure into the specification and the claims are embodiment(s) of the present disclosure.
While several embodiments have been provided in the present disclosure, it may be understood that the disclosed embodiments might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.
In addition, the various embodiments described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.
In closing, it should be noted that the discussion of any reference is not an admission that it is prior art to the presently disclosed embodiments, especially any reference that may have a publication date after the priority date of this application. At the same time, each and every claim below is hereby incorporated into this detailed description or specification as additional embodiments of the presently disclosed embodiments.
De Almeida Barreto, João Pedro
Patent | Priority | Assignee | Title |
Patent | Priority | Assignee | Title |
10499996, | Mar 26 2015 | UNIVERSIDADE DE COIMBRA | Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera |
10504239, | Apr 13 2015 | UNIVERSIDADE DE COIMBRA | Methods and systems for camera characterization in terms of response function, color, and vignetting under non-uniform illumination |
11335075, | Mar 14 2017 | UNIVERSIDADE DE COIMBRA | Systems and methods for 3D registration of curves and surfaces using local differential information |
5313306, | Feb 08 1993 | Sony Corporation | Omniview motionless camera endoscopy system |
5767980, | Jun 20 1995 | SHANGHAI ELECTRIC GROUP CORPORATION | Video based color sensing device for a printing press control system |
5792147, | Mar 17 1994 | KING S COLLEGE LONDON | Video-based systems for computer assisted surgery and localisation |
5836869, | Dec 13 1994 | Olympus Optical Co., Ltd. | Image tracking endoscope system |
6047088, | Dec 16 1996 | RAKUTEN, INC | 2D mesh geometry and motion vector compression |
6072496, | Jun 08 1998 | Microsoft Technology Licensing, LLC | Method and system for capturing and representing 3D geometry, color and shading of facial expressions and other animated objects |
6449103, | Apr 16 1997 | Solid catadioptric omnidirectional optical system having central coverage means which is associated with a camera, projector, medical instrument, or similar article | |
7429999, | May 24 2004 | Centre de Recherche Industrielle du Quebec | Camera calibrating apparatus and method |
7750969, | Jul 21 2004 | SPACEVISION, INC | Camera calibration system and three-dimensional measuring system |
7751865, | Oct 17 2003 | Medtronic Navigation, Inc. | Method and apparatus for surgical navigation |
7808525, | Jun 18 2004 | JAPAN AEROSPACE EXPLORATION AGENCY | Transparent camera calibration tool for camera calibration and calibration method thereof |
7892165, | Oct 23 2006 | Hoya Corporation | Camera calibration for endoscope navigation system |
7970174, | Jun 27 2006 | Brainlab AG | Medical marker tracking with marker property determination |
8223193, | Mar 31 2009 | Intuitive Surgical Operations, Inc | Targets, fixtures, and workflows for calibrating an endoscopic camera |
8771177, | Jul 08 2008 | KARL STORZ IMAGING, INC | Wide angle flexible endoscope |
8902232, | Jan 31 2008 | University of Southern California | Facial performance synthesis using deformation driven polynomial displacement maps |
9307892, | Nov 22 2012 | KARL STORZ SE & CO KG | Endoscope with an adjustable viewing direction |
9367928, | Oct 05 2012 | UNIVERSIDADE DE COIMBRA | Method for aligning and tracking point regions in images with radial distortion that outputs motion model parameters, distortion calibration, and variation in zoom |
9398840, | Apr 20 2011 | Henke-Sass, Wolf GmbH | Endoscope with variable direction of view |
9438897, | Jul 25 2011 | UNIVERSIDADE DE COIMBRA | Method and apparatus for automatic camera calibration using one or more images of a checkerboard pattern |
9888831, | Apr 11 2011 | Olympus Corporation | Imaging device and imaging method |
9986183, | Jun 11 2014 | Samsung Electronics Co., Ltd.; Chung-Ang University Industry—Academy Cooperation Foundation | Lens distortion correction device and application processor having the same |
20040019274, | |||
20040070565, | |||
20040181149, | |||
20050047676, | |||
20050085720, | |||
20050089199, | |||
20050245821, | |||
20050270375, | |||
20050280709, | |||
20050281465, | |||
20060082546, | |||
20060239345, | |||
20060250300, | |||
20070236514, | |||
20080075324, | |||
20080097156, | |||
20080202509, | |||
20080239327, | |||
20080269596, | |||
20080306490, | |||
20100009314, | |||
20100039506, | |||
20100168562, | |||
20100245541, | |||
20100256504, | |||
20110075922, | |||
20110115798, | |||
20110130761, | |||
20120078049, | |||
20120120255, | |||
20130034203, | |||
20130123801, | |||
20130150863, | |||
20130281821, | |||
20140022248, | |||
20140031668, | |||
20140035893, | |||
20140107471, | |||
20140236159, | |||
20140285676, | |||
20140327796, | |||
20150065793, | |||
20150065799, | |||
20150254872, | |||
20150297177, | |||
20160000518, | |||
20160048953, | |||
20160148435, | |||
20160161602, | |||
20160183841, | |||
20160191887, | |||
20160220099, | |||
20160324580, | |||
20170325669, | |||
20180049622, | |||
20180089855, | |||
EP1759629, | |||
EP2153794, | |||
EP3130276, | |||
WO2013015699, | |||
WO2014054958, | |||
WO2016154557, | |||
WO2016168307, | |||
WO2018232322, | |||
WO2009042644, | |||
WO2013052187, | |||
WO2014122301, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Dec 15 2017 | DE ALMEIDA BARRETO, JOÃO PEDRO | UNIVERSIDADE DE COIMBRA | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 066401 | /0221 | |
Dec 10 2021 | UNIVERSIDADE DE COIMBRA | (assignment on the face of the patent) | / |
Date | Maintenance Fee Events |
Dec 10 2021 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
Date | Maintenance Schedule |
Apr 23 2027 | 4 years fee payment window open |
Oct 23 2027 | 6 months grace period start (w surcharge) |
Apr 23 2028 | patent expiry (for year 4) |
Apr 23 2030 | 2 years to revive unintentionally abandoned end. (for year 4) |
Apr 23 2031 | 8 years fee payment window open |
Oct 23 2031 | 6 months grace period start (w surcharge) |
Apr 23 2032 | patent expiry (for year 8) |
Apr 23 2034 | 2 years to revive unintentionally abandoned end. (for year 8) |
Apr 23 2035 | 12 years fee payment window open |
Oct 23 2035 | 6 months grace period start (w surcharge) |
Apr 23 2036 | patent expiry (for year 12) |
Apr 23 2038 | 2 years to revive unintentionally abandoned end. (for year 12) |