Improving Image Inpainting based on Structure and Texture Information Using Quadtree

Document Type : Original Article


Computer and Electrical Engineering Department, Semnan University, Semnan, Iran


In this paper, we present a novel and efficient algorithm for image inpainting based on the structure and texture components. In our method, after decomposing the image into its texture and structure components using PCA (Principal Component Analysis), these components are inpainted separately using the proposed algorithm. Finally, the inpainted image is simply acquired by adding the two inpainted images. For structure inpainting we used quadtree concept to identify the importance of each pixel located on the boundary of the target region. Subsequently, we detect the correct path for filling so that this path demonstrates an orientation for the better structure inpainting. It is noteworthy that structure inpainting is more important because human vision is sensitive to the coherence of structure. For texture inpainting, we use Euclidean distance in the texture component for patch selection. Also, the geometric feature is considered by LSK (Local Steering Kernel) in the original image to assist chooing a better patch candidate. The experimental results of our algorithm demonstrate the effectiveness of the proposed method.


1. Bertalmio, M., Sapiro, G., Caselles, V. and Ballester, C., “Image
inpainting”, 27th annual conference on Computer graphics and
interactive techniques, (2000), 417-424. 
2. Chan, T. F. and Shen, J., “Nontexture Inpainting by CurvatureDriven
Diffusions”, Journal of Visual Communication and
Image Representation, Vol. 12, No. 4, (2001), 436-449. 
3. Shen, J., Kang, S. H. and Chan,  T. F., “Euler's Elastica and
Curvature-Based Inpainting”, SIAM Journal on Applied
Mathematics, Vol. 63, No. 2, (2003), 564-592. 
4. Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G. and Verdera,
J., “Filling-in by joint interpolation of vector fields and gray 
levels”, IEEE Transactions on Image Processing, Vol. 10, No.
8, (2001). 
5. Pu, Y., Wang, W., Zhou, J. and Wang, Y., “Fractional Derivative
Detection of Digital Image Texture Details and Implementation
of Fractional Derivative Filter”, Science in China Series F:
Information Sciences, Vol. 38, No. 12, (2008), 1319-1339. 
6. Ruzic, T. and Pizurica, A., “Context-aware patch-based image
inpainting using Markov random field modeling”, IEEE
Transactions on Image Processing, Vol. 24, No. 1, (2015), 444456.
7. Wang, H., Jiang, L., Liang, R. and Li, X. X., “Exemplar based
image inpainting using structure consistent patch matching”,
Neuro Computing, Vol. 269, (2017), 90-96. 
8. Criminisi, A., Pérez, P. and Toyama, K., “Region filling and
object removal by exemplar-based image inpainting,” IEEE
Transactions on Image Processing, Vol. 13, (2004), 1200-1212. 
9. Chen, Z., et al, “Structure aware image inpainting using patch
scale optimization,” Journal of Visual Communication and
Image Representation, Vol. 40, (2016), 312-323. 
10. Isogawa, M., Mikami, D., Takahashi, K. and Kojima, A., “Image
and video completion via feature reduction and compensation,”
Multimedia Tools and Applications, Vol. 76, (2017), 9443-9462. 
11. Bi, X., Liu, H., Lu, G., Wei, J., Chao, Y., “Exemplar-Based Image
Inpainting using Automatic Patch Optimization”, International
Conference on Video and Image Processing, (2018), 128-133.  
12. Zhang, N., Ji, H., Liu, L. and Wang, G., “Exemplar-based image
inpainting using angle-aware patch matching”, EURASIP
Journal on Image and Video Processing, Vol. 70, (2019). 
13. Chen, Z., Dai, C., Jiang, L., Sheng, B., Zhang, J., Lin, W., &
Yuan, Y. “Structure aware image inpainting using patch scale
optimization”, Journal of Visual Communication and Image
Representation, Vol. 40, (2016), 312-323.  
14. Liu, J., Musialski, P., Wonka, P. and Ye, J., “Tensor completion
for estimating missing  values in visual data,” IEEE Transactions
on Pattern Analysis and Machine Intelligence, Vol. 35, No. 1,
(2013), 208–220. 
15. Liang, X., Ren, X., Zhang, Z. and Ma, Y., “Texture repairing by
unified low rank optimization,” Journal of Computer Science
and Technology, Vol. 31, No. 3, (2016), 525-546. 
16. Calvi, G. G., Dees, B. S. and Mandic, D. P., “Tight lower bound
on the tensor rank based on the maximally square unfolding,”
17. Fazel, M., “Matrix rank minimization with applications,” PhD
thesis, Stanford University, (2002). 
18. Bengua, J. A., Phien, H. N., Tuan, H. D. and Do, M. N., “Efficient
tensor completion for color image and video recovery: low-rank
tensor train”, IEEE Transactions on Image Processing, Vol. 26,
(2017), 2466-2479. 
19. Guo, Q., Gao, S., Zhang, X., Yin, Y. and Zhang, C., “Patch-based
image inpainting via two-stage low rank approximation”, IEEE
Transactions on Visualization and Computer Graphics, Vol. 24,
(2018), 2023-2036. 
20. Jin, X., et al, “Sparsity-based image inpainting detection via
canonical correlation analysis with low-rank constraints”, IEEE
Access, Vol. 6, (2018), 49967-49978. 
21. Xu, Z. and Sun, J., “Image inpainting by patch propagation using
patch sparsity”, IEEE Transactions on Image Processing, Vol.
19, No. 5, (2010), 1153-1165. 
22. Jin, D. and Bai, X., “Patch-sparsity-based image inpainting
through a facet deduced directional derivative”, IEEE
Transactions on Circuits and Systems for Video Technology,
Vol. 29, (2019), 1310-1324.