Abstract

We developed a novel real-time motion blur compensation system for the blur caused by high-speed one-dimensional motion between a camera and a target. The system consists of a galvanometer mirror and a high-speed color camera, without the need for any additional sensors. We controlled the galvanometer mirror with continuous back-and-forth oscillating motion synchronized to a high-speed camera. The angular speed of the mirror is given in real time within 10 ms based on the concept of background tracking and rapid raw Bayer block matching. Experiments demonstrated that our system captures motion-invariant images of objects moving at speeds up to 30 km/h.

© 2015 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Real-time eye motion compensation for OCT imaging with tracking SLO

Kari V. Vienola, Boy Braaf, Christy K. Sheehy, Qiang Yang, Pavan Tiruveedhula, David W. Arathorn, Johannes F. de Boer, and Austin Roorda
Biomed. Opt. Express 3(11) 2950-2963 (2012)

Real-time visual sensing system achieving high-speed 3D particle tracking with nanometer resolution

Peng Cheng, Sissy M. Jhiang, and Chia-Hsiang Menq
Appl. Opt. 52(31) 7530-7539 (2013)

References

  • View by:
  • |
  • |
  • |

  1. E. Bodenstorfer, J. Furtler, J. Brodersen, K. J. Mayer, C. Eckel, K. Gravogl, and H. Nachtnebel, “High-speed line-scan camera with digital time delay integration,” Proc. SPIE 6496, 64960I (2007).
    [Crossref]
  2. B. Golik and D. Wueller, “Measurement method for image stabilizing systems,” Proc. SPIE 6502, 65020O (2007).
    [Crossref]
  3. C. W. Chiu, P. C. P. Chao, and D. Y. Wu, “Optimal design of magnetically actuated optical image stabilizer mechanism for cameras in mobile phones via genetic algorithm,” IEEE Trans. Magn. 43(6), 2582–2584 (2007).
    [Crossref]
  4. Y. Yitzhaky, R. Milberg, S. Yohaev, and N. S. Kopeika, “Comparison of direct blind deconvolution methods for motion-blurred images,” Appl. Opt. 38(20), 4325–4332 (1999).
    [Crossref]
  5. J. Zhang, Q. Zhang, and G. He, “Blind deconvolution of a noisy degraded image,” Appl. Opt. 48(12), 2350–2355 (2009).
    [Crossref] [PubMed]
  6. A. Levin, P. Sand, T. S. Cho, F. Durand, and W. T. Freeman, “Motion-invariant photography,” ACM Trans. Graph. 27(3), 71 (2008).
    [Crossref]
  7. R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006).
    [Crossref]
  8. Y. Qian, Y. Li, J. Shao, and H. Miao, “Real-time image stabilization for arbitrary motion blurred image based on opto-electronic hybrid joint transform correlator,” Opt. Express 19(11), 10762–10768 (2011).
    [Crossref] [PubMed]
  9. K. Daniilidis, C. Krauss, M. Hansen, and G. Sommer, “Real-time tracking of moving objects with an active camera,” Real-Time Imag. 4(1), 3–20 (1998).
    [Crossref]
  10. H. Oike, H. Wu, C. Hua, and T. Wada, “Clear image capture-active cameras system for tracking a high-speed moving object,” in Proceedings of the Fourth International Conference on Informatics in Control (2007), pp. 94–102.
  11. K. Okumura, H. Oku, and M. Ishikawa, “High-speed gaze controller for millisecond-order pan/tilt camera,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2011), pp. 6186–6191.
  12. M. Ito, “Cerebellar control of the vestibulo-ocular reflex - around the flocculus hypothesis,” Annu. Rev. Neurosci. 1(5), 275–296 (1982).
    [Crossref]
  13. M. Davis and P. Green, “Head-bobbing during walking, running and flying: relative motion perception in the pigeon,” J. Exp. Biol. 138(1), 71–91 (1988).
  14. J. Heo, J. Kim, and D. Lee, “Real-time digital image stabilization using motion sensors for search range reduction,” in SoC Design Conference (ISOCC, 2012), pp. 363–366.
  15. I. V. Romanenko, E. A. Edirisinghe, and D. Larkin, “Block matching noise reduction method for photographic images applied in Bayer RAW domain and optimized for real-time implementation,” Proc. SPIE8437, 84370F (2012).
  16. O. Yang and B. Choi, “Laser speckle imaging using a consumer-grade color camera,” Opt. Lett. 37(19), 3957–3959 (2012).
    [Crossref] [PubMed]
  17. V. Duma, J. P. Rolland, O. Group, A. Vlaicu, and R. Ave, “Advancements on galvanometer scanners for high-end applications,” Proc. SPIE 8936, 893612 (2014).
    [Crossref]

2014 (1)

V. Duma, J. P. Rolland, O. Group, A. Vlaicu, and R. Ave, “Advancements on galvanometer scanners for high-end applications,” Proc. SPIE 8936, 893612 (2014).
[Crossref]

2012 (1)

2011 (1)

2009 (1)

2008 (1)

A. Levin, P. Sand, T. S. Cho, F. Durand, and W. T. Freeman, “Motion-invariant photography,” ACM Trans. Graph. 27(3), 71 (2008).
[Crossref]

2007 (3)

E. Bodenstorfer, J. Furtler, J. Brodersen, K. J. Mayer, C. Eckel, K. Gravogl, and H. Nachtnebel, “High-speed line-scan camera with digital time delay integration,” Proc. SPIE 6496, 64960I (2007).
[Crossref]

B. Golik and D. Wueller, “Measurement method for image stabilizing systems,” Proc. SPIE 6502, 65020O (2007).
[Crossref]

C. W. Chiu, P. C. P. Chao, and D. Y. Wu, “Optimal design of magnetically actuated optical image stabilizer mechanism for cameras in mobile phones via genetic algorithm,” IEEE Trans. Magn. 43(6), 2582–2584 (2007).
[Crossref]

2006 (1)

R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006).
[Crossref]

1999 (1)

1998 (1)

K. Daniilidis, C. Krauss, M. Hansen, and G. Sommer, “Real-time tracking of moving objects with an active camera,” Real-Time Imag. 4(1), 3–20 (1998).
[Crossref]

1988 (1)

M. Davis and P. Green, “Head-bobbing during walking, running and flying: relative motion perception in the pigeon,” J. Exp. Biol. 138(1), 71–91 (1988).

1982 (1)

M. Ito, “Cerebellar control of the vestibulo-ocular reflex - around the flocculus hypothesis,” Annu. Rev. Neurosci. 1(5), 275–296 (1982).
[Crossref]

Agrawal, A.

R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006).
[Crossref]

Ave, R.

V. Duma, J. P. Rolland, O. Group, A. Vlaicu, and R. Ave, “Advancements on galvanometer scanners for high-end applications,” Proc. SPIE 8936, 893612 (2014).
[Crossref]

Bodenstorfer, E.

E. Bodenstorfer, J. Furtler, J. Brodersen, K. J. Mayer, C. Eckel, K. Gravogl, and H. Nachtnebel, “High-speed line-scan camera with digital time delay integration,” Proc. SPIE 6496, 64960I (2007).
[Crossref]

Brodersen, J.

E. Bodenstorfer, J. Furtler, J. Brodersen, K. J. Mayer, C. Eckel, K. Gravogl, and H. Nachtnebel, “High-speed line-scan camera with digital time delay integration,” Proc. SPIE 6496, 64960I (2007).
[Crossref]

Chao, P. C. P.

C. W. Chiu, P. C. P. Chao, and D. Y. Wu, “Optimal design of magnetically actuated optical image stabilizer mechanism for cameras in mobile phones via genetic algorithm,” IEEE Trans. Magn. 43(6), 2582–2584 (2007).
[Crossref]

Chiu, C. W.

C. W. Chiu, P. C. P. Chao, and D. Y. Wu, “Optimal design of magnetically actuated optical image stabilizer mechanism for cameras in mobile phones via genetic algorithm,” IEEE Trans. Magn. 43(6), 2582–2584 (2007).
[Crossref]

Cho, T. S.

A. Levin, P. Sand, T. S. Cho, F. Durand, and W. T. Freeman, “Motion-invariant photography,” ACM Trans. Graph. 27(3), 71 (2008).
[Crossref]

Choi, B.

Daniilidis, K.

K. Daniilidis, C. Krauss, M. Hansen, and G. Sommer, “Real-time tracking of moving objects with an active camera,” Real-Time Imag. 4(1), 3–20 (1998).
[Crossref]

Davis, M.

M. Davis and P. Green, “Head-bobbing during walking, running and flying: relative motion perception in the pigeon,” J. Exp. Biol. 138(1), 71–91 (1988).

Duma, V.

V. Duma, J. P. Rolland, O. Group, A. Vlaicu, and R. Ave, “Advancements on galvanometer scanners for high-end applications,” Proc. SPIE 8936, 893612 (2014).
[Crossref]

Durand, F.

A. Levin, P. Sand, T. S. Cho, F. Durand, and W. T. Freeman, “Motion-invariant photography,” ACM Trans. Graph. 27(3), 71 (2008).
[Crossref]

Eckel, C.

E. Bodenstorfer, J. Furtler, J. Brodersen, K. J. Mayer, C. Eckel, K. Gravogl, and H. Nachtnebel, “High-speed line-scan camera with digital time delay integration,” Proc. SPIE 6496, 64960I (2007).
[Crossref]

Edirisinghe, E. A.

I. V. Romanenko, E. A. Edirisinghe, and D. Larkin, “Block matching noise reduction method for photographic images applied in Bayer RAW domain and optimized for real-time implementation,” Proc. SPIE8437, 84370F (2012).

Freeman, W. T.

A. Levin, P. Sand, T. S. Cho, F. Durand, and W. T. Freeman, “Motion-invariant photography,” ACM Trans. Graph. 27(3), 71 (2008).
[Crossref]

Furtler, J.

E. Bodenstorfer, J. Furtler, J. Brodersen, K. J. Mayer, C. Eckel, K. Gravogl, and H. Nachtnebel, “High-speed line-scan camera with digital time delay integration,” Proc. SPIE 6496, 64960I (2007).
[Crossref]

Golik, B.

B. Golik and D. Wueller, “Measurement method for image stabilizing systems,” Proc. SPIE 6502, 65020O (2007).
[Crossref]

Gravogl, K.

E. Bodenstorfer, J. Furtler, J. Brodersen, K. J. Mayer, C. Eckel, K. Gravogl, and H. Nachtnebel, “High-speed line-scan camera with digital time delay integration,” Proc. SPIE 6496, 64960I (2007).
[Crossref]

Green, P.

M. Davis and P. Green, “Head-bobbing during walking, running and flying: relative motion perception in the pigeon,” J. Exp. Biol. 138(1), 71–91 (1988).

Group, O.

V. Duma, J. P. Rolland, O. Group, A. Vlaicu, and R. Ave, “Advancements on galvanometer scanners for high-end applications,” Proc. SPIE 8936, 893612 (2014).
[Crossref]

Hansen, M.

K. Daniilidis, C. Krauss, M. Hansen, and G. Sommer, “Real-time tracking of moving objects with an active camera,” Real-Time Imag. 4(1), 3–20 (1998).
[Crossref]

He, G.

Heo, J.

J. Heo, J. Kim, and D. Lee, “Real-time digital image stabilization using motion sensors for search range reduction,” in SoC Design Conference (ISOCC, 2012), pp. 363–366.

Hua, C.

H. Oike, H. Wu, C. Hua, and T. Wada, “Clear image capture-active cameras system for tracking a high-speed moving object,” in Proceedings of the Fourth International Conference on Informatics in Control (2007), pp. 94–102.

Ishikawa, M.

K. Okumura, H. Oku, and M. Ishikawa, “High-speed gaze controller for millisecond-order pan/tilt camera,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2011), pp. 6186–6191.

Ito, M.

M. Ito, “Cerebellar control of the vestibulo-ocular reflex - around the flocculus hypothesis,” Annu. Rev. Neurosci. 1(5), 275–296 (1982).
[Crossref]

Kim, J.

J. Heo, J. Kim, and D. Lee, “Real-time digital image stabilization using motion sensors for search range reduction,” in SoC Design Conference (ISOCC, 2012), pp. 363–366.

Kopeika, N. S.

Krauss, C.

K. Daniilidis, C. Krauss, M. Hansen, and G. Sommer, “Real-time tracking of moving objects with an active camera,” Real-Time Imag. 4(1), 3–20 (1998).
[Crossref]

Larkin, D.

I. V. Romanenko, E. A. Edirisinghe, and D. Larkin, “Block matching noise reduction method for photographic images applied in Bayer RAW domain and optimized for real-time implementation,” Proc. SPIE8437, 84370F (2012).

Lee, D.

J. Heo, J. Kim, and D. Lee, “Real-time digital image stabilization using motion sensors for search range reduction,” in SoC Design Conference (ISOCC, 2012), pp. 363–366.

Levin, A.

A. Levin, P. Sand, T. S. Cho, F. Durand, and W. T. Freeman, “Motion-invariant photography,” ACM Trans. Graph. 27(3), 71 (2008).
[Crossref]

Li, Y.

Mayer, K. J.

E. Bodenstorfer, J. Furtler, J. Brodersen, K. J. Mayer, C. Eckel, K. Gravogl, and H. Nachtnebel, “High-speed line-scan camera with digital time delay integration,” Proc. SPIE 6496, 64960I (2007).
[Crossref]

Miao, H.

Milberg, R.

Nachtnebel, H.

E. Bodenstorfer, J. Furtler, J. Brodersen, K. J. Mayer, C. Eckel, K. Gravogl, and H. Nachtnebel, “High-speed line-scan camera with digital time delay integration,” Proc. SPIE 6496, 64960I (2007).
[Crossref]

Oike, H.

H. Oike, H. Wu, C. Hua, and T. Wada, “Clear image capture-active cameras system for tracking a high-speed moving object,” in Proceedings of the Fourth International Conference on Informatics in Control (2007), pp. 94–102.

Oku, H.

K. Okumura, H. Oku, and M. Ishikawa, “High-speed gaze controller for millisecond-order pan/tilt camera,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2011), pp. 6186–6191.

Okumura, K.

K. Okumura, H. Oku, and M. Ishikawa, “High-speed gaze controller for millisecond-order pan/tilt camera,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2011), pp. 6186–6191.

Qian, Y.

Raskar, R.

R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006).
[Crossref]

Rolland, J. P.

V. Duma, J. P. Rolland, O. Group, A. Vlaicu, and R. Ave, “Advancements on galvanometer scanners for high-end applications,” Proc. SPIE 8936, 893612 (2014).
[Crossref]

Romanenko, I. V.

I. V. Romanenko, E. A. Edirisinghe, and D. Larkin, “Block matching noise reduction method for photographic images applied in Bayer RAW domain and optimized for real-time implementation,” Proc. SPIE8437, 84370F (2012).

Sand, P.

A. Levin, P. Sand, T. S. Cho, F. Durand, and W. T. Freeman, “Motion-invariant photography,” ACM Trans. Graph. 27(3), 71 (2008).
[Crossref]

Shao, J.

Sommer, G.

K. Daniilidis, C. Krauss, M. Hansen, and G. Sommer, “Real-time tracking of moving objects with an active camera,” Real-Time Imag. 4(1), 3–20 (1998).
[Crossref]

Tumblin, J.

R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006).
[Crossref]

Vlaicu, A.

V. Duma, J. P. Rolland, O. Group, A. Vlaicu, and R. Ave, “Advancements on galvanometer scanners for high-end applications,” Proc. SPIE 8936, 893612 (2014).
[Crossref]

Wada, T.

H. Oike, H. Wu, C. Hua, and T. Wada, “Clear image capture-active cameras system for tracking a high-speed moving object,” in Proceedings of the Fourth International Conference on Informatics in Control (2007), pp. 94–102.

Wu, D. Y.

C. W. Chiu, P. C. P. Chao, and D. Y. Wu, “Optimal design of magnetically actuated optical image stabilizer mechanism for cameras in mobile phones via genetic algorithm,” IEEE Trans. Magn. 43(6), 2582–2584 (2007).
[Crossref]

Wu, H.

H. Oike, H. Wu, C. Hua, and T. Wada, “Clear image capture-active cameras system for tracking a high-speed moving object,” in Proceedings of the Fourth International Conference on Informatics in Control (2007), pp. 94–102.

Wueller, D.

B. Golik and D. Wueller, “Measurement method for image stabilizing systems,” Proc. SPIE 6502, 65020O (2007).
[Crossref]

Yang, O.

Yitzhaky, Y.

Yohaev, S.

Zhang, J.

Zhang, Q.

ACM Trans. Graph. (2)

A. Levin, P. Sand, T. S. Cho, F. Durand, and W. T. Freeman, “Motion-invariant photography,” ACM Trans. Graph. 27(3), 71 (2008).
[Crossref]

R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: motion deblurring using fluttered shutter,” ACM Trans. Graph. 25(3), 795–804 (2006).
[Crossref]

Annu. Rev. Neurosci. (1)

M. Ito, “Cerebellar control of the vestibulo-ocular reflex - around the flocculus hypothesis,” Annu. Rev. Neurosci. 1(5), 275–296 (1982).
[Crossref]

Appl. Opt. (2)

IEEE Trans. Magn. (1)

C. W. Chiu, P. C. P. Chao, and D. Y. Wu, “Optimal design of magnetically actuated optical image stabilizer mechanism for cameras in mobile phones via genetic algorithm,” IEEE Trans. Magn. 43(6), 2582–2584 (2007).
[Crossref]

J. Exp. Biol. (1)

M. Davis and P. Green, “Head-bobbing during walking, running and flying: relative motion perception in the pigeon,” J. Exp. Biol. 138(1), 71–91 (1988).

Opt. Express (1)

Opt. Lett. (1)

Proc. SPIE (3)

V. Duma, J. P. Rolland, O. Group, A. Vlaicu, and R. Ave, “Advancements on galvanometer scanners for high-end applications,” Proc. SPIE 8936, 893612 (2014).
[Crossref]

E. Bodenstorfer, J. Furtler, J. Brodersen, K. J. Mayer, C. Eckel, K. Gravogl, and H. Nachtnebel, “High-speed line-scan camera with digital time delay integration,” Proc. SPIE 6496, 64960I (2007).
[Crossref]

B. Golik and D. Wueller, “Measurement method for image stabilizing systems,” Proc. SPIE 6502, 65020O (2007).
[Crossref]

Real-Time Imag. (1)

K. Daniilidis, C. Krauss, M. Hansen, and G. Sommer, “Real-time tracking of moving objects with an active camera,” Real-Time Imag. 4(1), 3–20 (1998).
[Crossref]

Other (4)

H. Oike, H. Wu, C. Hua, and T. Wada, “Clear image capture-active cameras system for tracking a high-speed moving object,” in Proceedings of the Fourth International Conference on Informatics in Control (2007), pp. 94–102.

K. Okumura, H. Oku, and M. Ishikawa, “High-speed gaze controller for millisecond-order pan/tilt camera,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2011), pp. 6186–6191.

J. Heo, J. Kim, and D. Lee, “Real-time digital image stabilization using motion sensors for search range reduction,” in SoC Design Conference (ISOCC, 2012), pp. 363–366.

I. V. Romanenko, E. A. Edirisinghe, and D. Larkin, “Block matching noise reduction method for photographic images applied in Bayer RAW domain and optimized for real-time implementation,” Proc. SPIE8437, 84370F (2012).

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1
Fig. 1 Concept of back-and-forth motion control of the galvanometer mirror.
Fig. 2
Fig. 2 Rapid block matching method in the Bayer raw domain (BGR). (a) Background tracking for the rapid block matching method. (b) Block matching between two Bayer raw images.
Fig. 3
Fig. 3 Control flow of real-time motion blur compensation system.
Fig. 4
Fig. 4 Mirror angle waveform and exposure timing.
Fig. 5
Fig. 5 Schematic diagram of the experimental system.
Fig. 6
Fig. 6 Optical components of the prototype real-time high-speed motion blur compensation system.
Fig. 7
Fig. 7 Response characteristics of the galvanometer mirror. (a) Input signal [mV] and output signal [mV] (with noise removed to smooth the averaging). (b) Input signal [mV] and gain [dB].
Fig. 8
Fig. 8 Horizontally separated still Bayer-array images to be processed by the straightforward block matching method (green) and the rapid block matching method (red). (a) Previous image. (b) Current image.
Fig. 9
Fig. 9 Fundamental result obtained with our system when vr was 30 km/h vertically, and vertical profiles at the position of the blue lines (with images trimmed for aligned display). (a) Still image. (b) Image during vr =30 km/h with motion blur compensation off). (c) Image during vr = 30 km/h with motion blur compensation on.
Fig. 10
Fig. 10 Peak-to-peak intensity of the initial vertical black-and-white pair at each vr.
Fig. 11
Fig. 11 Applications of our system in practical situations (with images trimmed for aligned display). The first row ((a), (b), and (c)) shows cracked roads, the second row ((d), (e), and (f)) shows printed boards, and the third row ((g), (h), and (i)) shows helicopter shots. The first column ((a), (d), and (g)) shows still images, the second column ((b), (e), and (h)) shows images during vr = 30 km/h with motion blur compensation off, and the third column ((c), (f), and (i)) shows images during vr = 30 km/h with motion blur compensation on.

Tables (1)

Tables Icon

Table 1 Performance Comparison of Block Matching Methods for Color Images

Equations (8)

Equations on this page are rendered with MathJax. Learn more.

s w 2 l = tan α 2 ,
x d 2 l = tan ω r 2 .
ω r = 2 tan 1 ( x d s w tan α 2 ) .
ω m = { ω r ( t 1 t t 3 ) , ω r ( e . g . t = t 4 ) .
R S S D = j = 0 1 i = 0 W w 1 ( Img p ( i , j ) Img c ( i + 2 , j ) ) 2 .
θ = ω m t { ( t 1 t t 3 ) } .
θ = A sin ( 2 π f t ) .
A = ω m 4 f .

Metrics