Abstract

We demonstrated GPU accelerated real-time confocal fluorescence lifetime imaging microscopy (FLIM) based on the analog mean-delay (AMD) method. Our algorithm was verified for various fluorescence lifetimes and photon numbers. The GPU processing time was faster than the physical scanning time for images up to 800 × 800, and more than 149 times faster than a single core CPU. The frame rate of our system was demonstrated to be 13 fps for a 200 × 200 pixel image when observing maize vascular tissue. This system can be utilized for observing dynamic biological reactions, medical diagnosis, and real-time industrial inspection.

© 2016 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
High-speed confocal fluorescence lifetime imaging microscopy (FLIM) with the analog mean delay (AMD) method

Youngjae Won, Sucbei Moon, Wenzhong Yang, Donguk Kim, Won-Taek Han, and Dug Young Kim
Opt. Express 19(4) 3396-3405 (2011)

Real-time visualization of two-photon fluorescence lifetime imaging microscopy using a wavelength-tunable femtosecond pulsed laser

Jiheun Ryu, Ungyo Kang, Jayul Kim, Hyunjun Kim, Jue Hyung Kang, Hyunjin Kim, Dae Kyung Sohn, Jae-heon Jeong, Hongki Yoo, and Bomi Gweon
Biomed. Opt. Express 9(7) 3449-3463 (2018)

Referencing techniques for the analog mean-delay method in fluorescence lifetime imaging

Young Jae Won, Sucbei Moon, Won-Taek Han, and Dug Young Kim
J. Opt. Soc. Am. A 27(11) 2402-2410 (2010)

References

  • View by:
  • |
  • |
  • |

  1. H. C. Gerristen, A. Draaijer, D. J. van den Heuvel, and A. V. Agronskaia, “Fluorescence lifetime imaging in scanning microscopy,” in Handbook of Biological Confocal Microscopy, 3rd Ed., J. B. Pawley, ed. (Springer, 2006).
  2. J. Neefjes and N. P. Dantuma, “Fluorescent probes for proteolysis: tools for drug discovery,” Nat. Rev. Drug Discov. 3(1), 58–69 (2004).
    [Crossref] [PubMed]
  3. A. Esposito, T. Tiffert, J. M. A. Mauritz, S. Schlachter, L. H. Bannister, C. F. Kaminski, and V. L. Lew, “FRET imaging of hemoglobin concentration in Plasmodium falciparum-infected red cells,” PLoS One 3(11), e3780 (2008).
    [Crossref] [PubMed]
  4. H. Wallrabe and A. Periasamy, “Imaging protein molecules using FRET and FLIM microscopy,” Curr. Opin. Biotechnol. 16(1), 19–27 (2005).
    [Crossref] [PubMed]
  5. J. McGinty, N. P. Galletly, C. Dunsby, I. Munro, D. S. Elson, J. Requejo-Isidro, P. Cohen, R. Ahmad, A. Forsyth, A. V. Thillainayagam, M. A. Neil, P. M. French, and G. W. Stamp, “Wide-field fluorescence lifetime imaging of cancer,” Biomed. Opt. Express 1(2), 627–640 (2010).
    [Crossref] [PubMed]
  6. W. Becker, A. Bergmann, M. A. Hink, K. König, K. Benndorf, and C. Biskup, “Fluorescence lifetime imaging by time-correlated single-photon counting,” Microsc. Res. Tech. 63(1), 58–66 (2004).
    [Crossref] [PubMed]
  7. T. H. Chia, A. Williamson, D. D. Spencer, and M. J. Levene, “Multiphoton fluorescence lifetime imaging of intrinsic fluorescence in human and rat brain tissue reveals spatially distinct NADH binding,” Opt. Express 16(6), 4237–4249 (2008).
    [Crossref] [PubMed]
  8. W. Becker, A. Bergmann, H. Wabnitz, D. Grosenick, and A. Liebert, “High count rate multichannel TCSPC for optical tomography,” Proc. SPIE 4431, 249–254 (2001).
    [Crossref]
  9. S. Moon, Y. Won, and D. Y. Kim, “Analog mean-delay method for high-speed fluorescence lifetime measurement,” Opt. Express 17(4), 2834–2849 (2009).
    [Crossref] [PubMed]
  10. Y. Won, S. Moon, W. Yang, D. Kim, W. T. Han, and D. Y. Kim, “High-speed confocal fluorescence lifetime imaging microscopy (FLIM) with the analog mean delay (AMD) method,” Opt. Express 19(4), 3396–3405 (2011).
    [Crossref] [PubMed]
  11. Y. J. Won, S. Moon, W. T. Han, and D. Y. Kim, “Referencing techniques for the analog mean-delay method in fluorescence lifetime imaging,” J. Opt. Soc. Am. A 27(11), 2402–2410 (2010).
    [Crossref] [PubMed]
  12. Y. J. Won, W. T. Han, and D. Y. Kim, “Precision and accuracy of the analog mean-delay method for high-speed fluorescence lifetime measurement,” J. Opt. Soc. Am. A 28(10), 2026–2032 (2011).
    [Crossref] [PubMed]
  13. K. Zhang and J. U. Kang, “Graphics processing unit-based ultrahigh speed real-time Fourier domain optical coherence tomography,” IEEE J. Sel. Top. Quantum Electron. 18(4), 1270–1279 (2012).
    [Crossref]
  14. K. K. C. Lee, A. Mariampillai, J. X. Z. Yu, D. W. Cadotte, B. C. Wilson, B. A. Standish, and V. X. D. Yang, “Real-time speckle variance swept-source optical coherence tomography using a graphics processing unit,” Biomed. Opt. Express 3(7), 1557–1564 (2012).
    [Crossref] [PubMed]
  15. J. Xu, K. Wong, Y. Jian, and M. V. Sarunic, “Real-time acquisition and display of flow contrast using speckle variance optical coherence tomography in a graphics processing unit,” J. Biomed. Opt. 19(2), 026001 (2014).
    [Crossref] [PubMed]
  16. NVIDIA, “NVIDIA CUDA C Programming Guide Version 7.0,” 2015.
  17. J. D. Mauseth, Plant Anatomy (The Blackburn Press, 2008)

2014 (1)

J. Xu, K. Wong, Y. Jian, and M. V. Sarunic, “Real-time acquisition and display of flow contrast using speckle variance optical coherence tomography in a graphics processing unit,” J. Biomed. Opt. 19(2), 026001 (2014).
[Crossref] [PubMed]

2012 (2)

K. Zhang and J. U. Kang, “Graphics processing unit-based ultrahigh speed real-time Fourier domain optical coherence tomography,” IEEE J. Sel. Top. Quantum Electron. 18(4), 1270–1279 (2012).
[Crossref]

K. K. C. Lee, A. Mariampillai, J. X. Z. Yu, D. W. Cadotte, B. C. Wilson, B. A. Standish, and V. X. D. Yang, “Real-time speckle variance swept-source optical coherence tomography using a graphics processing unit,” Biomed. Opt. Express 3(7), 1557–1564 (2012).
[Crossref] [PubMed]

2011 (2)

2010 (2)

2009 (1)

2008 (2)

T. H. Chia, A. Williamson, D. D. Spencer, and M. J. Levene, “Multiphoton fluorescence lifetime imaging of intrinsic fluorescence in human and rat brain tissue reveals spatially distinct NADH binding,” Opt. Express 16(6), 4237–4249 (2008).
[Crossref] [PubMed]

A. Esposito, T. Tiffert, J. M. A. Mauritz, S. Schlachter, L. H. Bannister, C. F. Kaminski, and V. L. Lew, “FRET imaging of hemoglobin concentration in Plasmodium falciparum-infected red cells,” PLoS One 3(11), e3780 (2008).
[Crossref] [PubMed]

2005 (1)

H. Wallrabe and A. Periasamy, “Imaging protein molecules using FRET and FLIM microscopy,” Curr. Opin. Biotechnol. 16(1), 19–27 (2005).
[Crossref] [PubMed]

2004 (2)

W. Becker, A. Bergmann, M. A. Hink, K. König, K. Benndorf, and C. Biskup, “Fluorescence lifetime imaging by time-correlated single-photon counting,” Microsc. Res. Tech. 63(1), 58–66 (2004).
[Crossref] [PubMed]

J. Neefjes and N. P. Dantuma, “Fluorescent probes for proteolysis: tools for drug discovery,” Nat. Rev. Drug Discov. 3(1), 58–69 (2004).
[Crossref] [PubMed]

2001 (1)

W. Becker, A. Bergmann, H. Wabnitz, D. Grosenick, and A. Liebert, “High count rate multichannel TCSPC for optical tomography,” Proc. SPIE 4431, 249–254 (2001).
[Crossref]

Ahmad, R.

Bannister, L. H.

A. Esposito, T. Tiffert, J. M. A. Mauritz, S. Schlachter, L. H. Bannister, C. F. Kaminski, and V. L. Lew, “FRET imaging of hemoglobin concentration in Plasmodium falciparum-infected red cells,” PLoS One 3(11), e3780 (2008).
[Crossref] [PubMed]

Becker, W.

W. Becker, A. Bergmann, M. A. Hink, K. König, K. Benndorf, and C. Biskup, “Fluorescence lifetime imaging by time-correlated single-photon counting,” Microsc. Res. Tech. 63(1), 58–66 (2004).
[Crossref] [PubMed]

W. Becker, A. Bergmann, H. Wabnitz, D. Grosenick, and A. Liebert, “High count rate multichannel TCSPC for optical tomography,” Proc. SPIE 4431, 249–254 (2001).
[Crossref]

Benndorf, K.

W. Becker, A. Bergmann, M. A. Hink, K. König, K. Benndorf, and C. Biskup, “Fluorescence lifetime imaging by time-correlated single-photon counting,” Microsc. Res. Tech. 63(1), 58–66 (2004).
[Crossref] [PubMed]

Bergmann, A.

W. Becker, A. Bergmann, M. A. Hink, K. König, K. Benndorf, and C. Biskup, “Fluorescence lifetime imaging by time-correlated single-photon counting,” Microsc. Res. Tech. 63(1), 58–66 (2004).
[Crossref] [PubMed]

W. Becker, A. Bergmann, H. Wabnitz, D. Grosenick, and A. Liebert, “High count rate multichannel TCSPC for optical tomography,” Proc. SPIE 4431, 249–254 (2001).
[Crossref]

Biskup, C.

W. Becker, A. Bergmann, M. A. Hink, K. König, K. Benndorf, and C. Biskup, “Fluorescence lifetime imaging by time-correlated single-photon counting,” Microsc. Res. Tech. 63(1), 58–66 (2004).
[Crossref] [PubMed]

Cadotte, D. W.

Chia, T. H.

Cohen, P.

Dantuma, N. P.

J. Neefjes and N. P. Dantuma, “Fluorescent probes for proteolysis: tools for drug discovery,” Nat. Rev. Drug Discov. 3(1), 58–69 (2004).
[Crossref] [PubMed]

Dunsby, C.

Elson, D. S.

Esposito, A.

A. Esposito, T. Tiffert, J. M. A. Mauritz, S. Schlachter, L. H. Bannister, C. F. Kaminski, and V. L. Lew, “FRET imaging of hemoglobin concentration in Plasmodium falciparum-infected red cells,” PLoS One 3(11), e3780 (2008).
[Crossref] [PubMed]

Forsyth, A.

French, P. M.

Galletly, N. P.

Grosenick, D.

W. Becker, A. Bergmann, H. Wabnitz, D. Grosenick, and A. Liebert, “High count rate multichannel TCSPC for optical tomography,” Proc. SPIE 4431, 249–254 (2001).
[Crossref]

Han, W. T.

Hink, M. A.

W. Becker, A. Bergmann, M. A. Hink, K. König, K. Benndorf, and C. Biskup, “Fluorescence lifetime imaging by time-correlated single-photon counting,” Microsc. Res. Tech. 63(1), 58–66 (2004).
[Crossref] [PubMed]

Jian, Y.

J. Xu, K. Wong, Y. Jian, and M. V. Sarunic, “Real-time acquisition and display of flow contrast using speckle variance optical coherence tomography in a graphics processing unit,” J. Biomed. Opt. 19(2), 026001 (2014).
[Crossref] [PubMed]

Kaminski, C. F.

A. Esposito, T. Tiffert, J. M. A. Mauritz, S. Schlachter, L. H. Bannister, C. F. Kaminski, and V. L. Lew, “FRET imaging of hemoglobin concentration in Plasmodium falciparum-infected red cells,” PLoS One 3(11), e3780 (2008).
[Crossref] [PubMed]

Kang, J. U.

K. Zhang and J. U. Kang, “Graphics processing unit-based ultrahigh speed real-time Fourier domain optical coherence tomography,” IEEE J. Sel. Top. Quantum Electron. 18(4), 1270–1279 (2012).
[Crossref]

Kim, D.

Kim, D. Y.

König, K.

W. Becker, A. Bergmann, M. A. Hink, K. König, K. Benndorf, and C. Biskup, “Fluorescence lifetime imaging by time-correlated single-photon counting,” Microsc. Res. Tech. 63(1), 58–66 (2004).
[Crossref] [PubMed]

Lee, K. K. C.

Levene, M. J.

Lew, V. L.

A. Esposito, T. Tiffert, J. M. A. Mauritz, S. Schlachter, L. H. Bannister, C. F. Kaminski, and V. L. Lew, “FRET imaging of hemoglobin concentration in Plasmodium falciparum-infected red cells,” PLoS One 3(11), e3780 (2008).
[Crossref] [PubMed]

Liebert, A.

W. Becker, A. Bergmann, H. Wabnitz, D. Grosenick, and A. Liebert, “High count rate multichannel TCSPC for optical tomography,” Proc. SPIE 4431, 249–254 (2001).
[Crossref]

Mariampillai, A.

Mauritz, J. M. A.

A. Esposito, T. Tiffert, J. M. A. Mauritz, S. Schlachter, L. H. Bannister, C. F. Kaminski, and V. L. Lew, “FRET imaging of hemoglobin concentration in Plasmodium falciparum-infected red cells,” PLoS One 3(11), e3780 (2008).
[Crossref] [PubMed]

McGinty, J.

Moon, S.

Munro, I.

Neefjes, J.

J. Neefjes and N. P. Dantuma, “Fluorescent probes for proteolysis: tools for drug discovery,” Nat. Rev. Drug Discov. 3(1), 58–69 (2004).
[Crossref] [PubMed]

Neil, M. A.

Periasamy, A.

H. Wallrabe and A. Periasamy, “Imaging protein molecules using FRET and FLIM microscopy,” Curr. Opin. Biotechnol. 16(1), 19–27 (2005).
[Crossref] [PubMed]

Requejo-Isidro, J.

Sarunic, M. V.

J. Xu, K. Wong, Y. Jian, and M. V. Sarunic, “Real-time acquisition and display of flow contrast using speckle variance optical coherence tomography in a graphics processing unit,” J. Biomed. Opt. 19(2), 026001 (2014).
[Crossref] [PubMed]

Schlachter, S.

A. Esposito, T. Tiffert, J. M. A. Mauritz, S. Schlachter, L. H. Bannister, C. F. Kaminski, and V. L. Lew, “FRET imaging of hemoglobin concentration in Plasmodium falciparum-infected red cells,” PLoS One 3(11), e3780 (2008).
[Crossref] [PubMed]

Spencer, D. D.

Stamp, G. W.

Standish, B. A.

Thillainayagam, A. V.

Tiffert, T.

A. Esposito, T. Tiffert, J. M. A. Mauritz, S. Schlachter, L. H. Bannister, C. F. Kaminski, and V. L. Lew, “FRET imaging of hemoglobin concentration in Plasmodium falciparum-infected red cells,” PLoS One 3(11), e3780 (2008).
[Crossref] [PubMed]

Wabnitz, H.

W. Becker, A. Bergmann, H. Wabnitz, D. Grosenick, and A. Liebert, “High count rate multichannel TCSPC for optical tomography,” Proc. SPIE 4431, 249–254 (2001).
[Crossref]

Wallrabe, H.

H. Wallrabe and A. Periasamy, “Imaging protein molecules using FRET and FLIM microscopy,” Curr. Opin. Biotechnol. 16(1), 19–27 (2005).
[Crossref] [PubMed]

Williamson, A.

Wilson, B. C.

Won, Y.

Won, Y. J.

Wong, K.

J. Xu, K. Wong, Y. Jian, and M. V. Sarunic, “Real-time acquisition and display of flow contrast using speckle variance optical coherence tomography in a graphics processing unit,” J. Biomed. Opt. 19(2), 026001 (2014).
[Crossref] [PubMed]

Xu, J.

J. Xu, K. Wong, Y. Jian, and M. V. Sarunic, “Real-time acquisition and display of flow contrast using speckle variance optical coherence tomography in a graphics processing unit,” J. Biomed. Opt. 19(2), 026001 (2014).
[Crossref] [PubMed]

Yang, V. X. D.

Yang, W.

Yu, J. X. Z.

Zhang, K.

K. Zhang and J. U. Kang, “Graphics processing unit-based ultrahigh speed real-time Fourier domain optical coherence tomography,” IEEE J. Sel. Top. Quantum Electron. 18(4), 1270–1279 (2012).
[Crossref]

Biomed. Opt. Express (2)

Curr. Opin. Biotechnol. (1)

H. Wallrabe and A. Periasamy, “Imaging protein molecules using FRET and FLIM microscopy,” Curr. Opin. Biotechnol. 16(1), 19–27 (2005).
[Crossref] [PubMed]

IEEE J. Sel. Top. Quantum Electron. (1)

K. Zhang and J. U. Kang, “Graphics processing unit-based ultrahigh speed real-time Fourier domain optical coherence tomography,” IEEE J. Sel. Top. Quantum Electron. 18(4), 1270–1279 (2012).
[Crossref]

J. Biomed. Opt. (1)

J. Xu, K. Wong, Y. Jian, and M. V. Sarunic, “Real-time acquisition and display of flow contrast using speckle variance optical coherence tomography in a graphics processing unit,” J. Biomed. Opt. 19(2), 026001 (2014).
[Crossref] [PubMed]

J. Opt. Soc. Am. A (2)

Microsc. Res. Tech. (1)

W. Becker, A. Bergmann, M. A. Hink, K. König, K. Benndorf, and C. Biskup, “Fluorescence lifetime imaging by time-correlated single-photon counting,” Microsc. Res. Tech. 63(1), 58–66 (2004).
[Crossref] [PubMed]

Nat. Rev. Drug Discov. (1)

J. Neefjes and N. P. Dantuma, “Fluorescent probes for proteolysis: tools for drug discovery,” Nat. Rev. Drug Discov. 3(1), 58–69 (2004).
[Crossref] [PubMed]

Opt. Express (3)

PLoS One (1)

A. Esposito, T. Tiffert, J. M. A. Mauritz, S. Schlachter, L. H. Bannister, C. F. Kaminski, and V. L. Lew, “FRET imaging of hemoglobin concentration in Plasmodium falciparum-infected red cells,” PLoS One 3(11), e3780 (2008).
[Crossref] [PubMed]

Proc. SPIE (1)

W. Becker, A. Bergmann, H. Wabnitz, D. Grosenick, and A. Liebert, “High count rate multichannel TCSPC for optical tomography,” Proc. SPIE 4431, 249–254 (2001).
[Crossref]

Other (3)

NVIDIA, “NVIDIA CUDA C Programming Guide Version 7.0,” 2015.

J. D. Mauseth, Plant Anatomy (The Blackburn Press, 2008)

H. C. Gerristen, A. Draaijer, D. J. van den Heuvel, and A. V. Agronskaia, “Fluorescence lifetime imaging in scanning microscopy,” in Handbook of Biological Confocal Microscopy, 3rd Ed., J. B. Pawley, ed. (Springer, 2006).

Supplementary Material (2)

NameDescription
» Visualization 1: MP4 (9532 KB)      AMD based real-time fluorescence lifetime imaging of Maize Vascular Tissue
» Visualization 2: MP4 (9786 KB)      AMD based real-time fluorescence lifetime imaging of Alexa Fluor 488

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (8)

Fig. 1
Fig. 1 Schematic diagram of the GPU accelerated real-time confocal AMD-FLIM.
Fig. 2
Fig. 2 System sequence flowchart for the GPU-accelerated real-time confocal AMD-FLIM.
Fig. 3
Fig. 3 Process flow in GPU cores.
Fig. 4
Fig. 4 Fluorescence intensity and lifetime image extracted from ideally generated fluorescence and IRF data set.
Fig. 5
Fig. 5 NVIDIA visual profiler to evaluate GPU processing time. (The results for 200 × 200 fluorescence lifetime imaging with the AMD method).
Fig. 6
Fig. 6 Visual Studio profiler evaluation of CPU processing time. (The results show 200 × 200 fluorescence lifetime imaging with the AMD method.).
Fig. 7
Fig. 7 GPU accelerated real-time fluorescence lifetime imaging of maize vascular tissue. (200☓200 pixels) (a) fluorescence intensity image (b) fluorescence lifetime image. (see Visualization 1).
Fig. 8
Fig. 8 GPU accelerated real-time fluorescence lifetime imaging of Alexa Fluor 488. (200☓200 pixels) (a) fluorescence intensity image (b) fluorescence lifetime image. (see Visualization 2)

Tables (2)

Tables Icon

Table 1 Simulation results for lifetime of 1 ns and 3 ns.

Tables Icon

Table 2 Processing time of CPU (Single core) and GPU.

Equations (1)

Equations on this page are rendered with MathJax. Learn more.

τ=< T e >< T e 0 >( t· i e ( t )dt i e ( t )dt )( t· i irf ( t )dt i irf ( t )dt )

Metrics