Computer Vision Seminar – Forensic Imaging
Paper reference List
Course Homepage:
http://cs.haifa.ac.il/~hagit/courses/seminars/ImageForgery/ImageForgery.html
General References
Tutorial by Hany Farid
http://www.cs.dartmouth.edu/farid/downloads/tutorials/digitalimageforensics.pdf
Media Forensics and DeepFakes: An Overview by Luisa Verdoliva
https://ieeexplore-ieee-org.ezproxy.haifa.ac.il/stamp/stamp.jsp?tp=&arnumber=9115874&tag=1
Image forgery detection: a survey of recent deep-learning approaches by Zanardelli,etal
Multimedia Tools and Applications, Volume 82, pages 17521–17566, (2023)
https://link.springer.com/article/10.1007/s11042-022-13797-w
How to find papers online:
· Google Scholar
· Search Google for Title.
· Search Authors home page (look under Publications in author's home page)
· http://citeseerx.ist.psu.edu/
PART 1. Image Forgery Detection - OLD SCHOOL
In this talk we will cover classic Image Processing based forgery detection methods.
These detection methods fall into several categories:
* Physics of the scene
* Statistics of Natural Images
* Camera Based
Lecture 1 - Inconsistencies in the physics of the scene.
Size Inconsistencies
W. Zhang, X. Cao, Y. Qu , Y. Hou, H. Zhao and C. Zhang
‘Detecting and extracting the photo composites using planar homography and graph cut’,
IEEE Transactions on Information Forensics and Security, vol. 5, pp. 544–555, 2010.
(can skip the graph-cut part)
L. Wu, X. Cao, W. Zhang, and Y. Wang
‘Detecting image forgeries using metrology’,
Machine Vision and Applications, vol. 23, no. 2, pp. 363–373, 2010.
Reflection Inconsistencies
H. Farid and M. Bravo,
‘Image forensic analyses that elude the human visual system,’
SPIE Symposium on Electronic Imaging, San Jose, CA, 2010.
J. O’Brien and H. Farid,
‘Exposing photo manipulation with inconsistent reflections’,
ACMTransactions on Graphics, vol. 31, no. 1 pp. 4:1–4:11, 2012.
Inconsistent Lighting
M. K. Johnson and H. Farid,
‘Exposing digital forgeries by detecting inconsistencies in lighting’,
ACM Multimedia and Security Workshop, pp. 1–10, 2005.
M. K. Johnson and H. Farid,
‘Exposing digital forgeries in complex lighting environments’,
IEEE Transactions on Information Forensics and Security, vol. 3, no. 2, pp. 450–461, 2007.
E. Kee and H. Farid,
‘Exposing digital forgeries from 3-D lighting environments’,
Workshop on Information Forensics and Security, 2010.
W. Fan, K. Wang, F. Cayre and Z. Xiong,
‘3D lighting-based image forgery detection using shape from shading,’
20th European Signal Processing Conference, Bucharest, Romania, pp. 1777–1781, 2012.
Shadow inconsistencies
W. Zhang, X. Cao, J. Zhang, J. Zhu, and P. Wang,
‘Detecting photographic composites using shadows’,
Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’09), pp. 1042–1045, July 2009.
Q. Liu, X. Cao, C. Deng, and X. Guo,
‘Identifying image composites through shadow matte consistency’,
IEEE Transactions on Information Forensics and Security, vol. 6, no. 3, pp. 1111–1122, 2011.
E. Kee, J. O. ’Brien, and H. Farid,
‘Exposing photo manipulation with inconsistent shadows’, ACM
Transaction on Graphic, vol. 32, no. 3, pp. 1–12, 2013.
Morteza Nasiri, Alireza Behrad
Using Expectation-Maximization for exposing image forgeries by revealing inconsistencies in shadow geometry
Journal of Visual Communication and Image Representation Volume 58, January 2019, Pages 323-333
Detecting image forgeries using metrology
Lin Wu · Xiaochun Cao · Wei Zhang · Yang Wang
Machine Vision and Applications, Volume 23, Issue 2 Pages 363 - 373, 2012
Lecture 2 - Statistics of natural images.
Uses PCA
A. Popescu and H. Farid,
Exposing digital forgeries by detecting duplicated image regions,
Computer Science, Dartmouth College, Tech. Rep. TR2004-515, 2004.
Uses DCT
J.Fridrich, D. Soukal, and J.
Lukas.
Detection of Copy-Move Forgery in digital Images. Proc. Of Digital Forensic
Research Workshop, Aug. 2003.
Uses Sift
H. Huang, W. Guo, and Y. Zhang,
“Detection of Copy-Move Forgery in Digital Images Using SIFT Algorithm,”
Proceedings of IEEE Pacific-Asia Workshop on Computational Intelligence and Industrial Application, Vol. 2,2008, pp. 272-276.
I. Amerini, L. Ballan, R. Caldelli, A. Del Bimbo, G. Serra,
"A SIFT-based forensic method for copy move attack detection and transformation recovery",
IEEE Transactions on Information Forensics and Security 6 (3) (2011) 1099–1110.
Uses Moments
Vivek Kumar Singh and R. C. Tripathi
"Fast Rotation Invariant Detection of Region Duplication Attacks Even on Uniform Background Containing Digital Images"
Eleventh International Multi-Conference on Information Processing-2015 (IMCIP-2015)
Uses Patch-Match
D Cozzolino, G Poggi, L Verdoliva
Copy-move forgery detection based on patchmatch
2014 IEEE international conference on image processing (ICIP), 5312-5316
L D'Amiano, D Cozzolino, G Poggi, L Verdoliva
Video forgery detection and localization based on 3D patchmatch
IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 1-6, 2015
L D’Amiano, D Cozzolino, G Poggi, L Verdoliva
A patchmatch-based dense-field algorithm for video copy–move detection and localization
IEEE Transactions on Circuits and Systems for Video Technology 29 (3), 669-682, 2018
Uses Wavelets
Wavelet coefs - sort lexicographically
Vivek Kumar Singh and R.C. Tripathi
Fast and Efficient Region Duplication Detection in Digital Images Using Sub-Blocking Method
International Journal of Advanced Science and Technology Vol. 35, October, 2011
M. Bashar, K. Noda, N. Ohnishi, K. Mori,
Exploring duplicated regions in natural images
IEEE Transactions on Image Processing PP(99) March 2010
Ghulam Muhammad, Muhammad Hussain , Khalid Khawaji, and George Bebis
Blind Copy Move Image Forgery Detection Using Dyadic Undecimated Wavelet Transform
17th International Conference on Digital Signal Processing (DSP), 2011 July 2011
Discrete Wavelet Trans coefs are correlated to find repeated regions.
Er. Saiqa Khan Er. Arun Kulkarni
An Efficient Method for Detection of Copy-Move Forgery Using Discrete Wavelet Transform
(IJCSE) International Journal on Computer Science and Engineering Vol. 02, No. 05, 2010, 1801-1806
Uses Machine Learning (SVM) – copy-paste and splicing det
D. Cozzolino, D. Gragnaniello and L. Verdoliva,
"Image forgery detection through residual-based local descriptors and block-matching,"
2014 IEEE International Conference on Image Processing (ICIP),
Takes many filters and gets residual images – takes patches and tests for co-ocurrences and uses in
SVM
Lecture 3 - Camera Based Forgery Detection.
Optics based Forgery detection
Radial Distortion
K.S. Choi, E.Y. Lam, and K.K.Y. Wong, “Automatic source camera identification using intrinsic lens
radial distortion, ” Optics Express, vol. 14, no. 24, pp. 11551–11565, 2006a.
K.S. Choi, E.Y. Lam, and K.K.Y. Wong,
‘Source camera identification using footprints from lens aberration’,
SPIE Conference on Digital Photography (N. Sampat, J.M. DiCarlo, and R.A. Martin,eds.), vol. 6069, 2006.
H.R. Chennamma and L. Rangarajan,
‘Image splicing detection using inherent lens radial distortion,’
International Journal of Computer Science Issues, vol. 7, no. 6, pp. 149–158, 2010.
Vignetting
S. Lyu,
‘Estimating vignetting function from a single image for image authentication’,
ACMWorkshop on Multimedia and Security, pp. 3–12, 2010.
Chromatic Aberration
M.K. Johnson and H. Farid, ‘Exposing digital forgeries through chromatic aberration’,
ACM Multimedia and Security Workshop, pp. 48–55, 2006.
T.V. Lanh, S. Emmanuel, and M.S. Kankanhalli,
‘Identifying source cell phone using chromatic aberration’,
IEEE International Conference on Multimedia and Expo, July 2007
I. Yerushalmy and H. Hel-Or,
‘Digital image forgery detection based on lens and sensor aberration’,
International Journal of Computer Vision, vol. 92, no. 1, pp. 71–91, 2011
T. Gloe, K. Borowk, and A. Winkler,
“Efficient estimation and large scale evaluation of lateral chromatic aberration for digital image forensics,”
in Proc. SPIE, 2010, vol. 7541, pp. 1–13.
O. Mayer and M. Stamm, “Accurate and efficient image forgery detection using lateral chromatic aberration,”
IEEE Trans. Inf. Forensics Secur., vol. 13, no. 7, pp. 1762–1777, Jul. 2018
Sensor Based Forgery detection
PRNU Based Camera Signature
J. Lukas, J. Fridrich, and M. Goljan,
‘Digital camera identification from sensor noise’,
IEEE Transactions on Information Forensics and Security, vol. 1, no. 2, pp. 205–214, 2006.
A. Swaminathan, M. Wu, and K. J. R. Liu,
“Digital image forensics via intrinsic fingerprints,
” IEEE Trans. Inf. Forensics Security, vol. 3, no. 1, pp. 101–117, Mar. 2008.
S. Lyu, X. Pan, and X. Zhang,
“Exposing region splicing forgeries with blind local noise estimation,”
Int. J. Comput. Vision, vol. 110, no. 2, pp. 202–221, 2014.
Extending the above: Context dependent PRNU
C.T. Li,
‘Source camera linking using enhanced sensor pattern noise extracted from images’,
International Conference on Imaging for Crime Detection and Prevention, 2009.
M. Chen, J. Fridrich, M. Goljan,
and J. Lukas,
‘Source digital camcorder identification using sensor photo-response
non-uniformity,’
in SPIE Conference on Security, Steganography, and Watermarking of Multimedia
Contents (E.J. Delp and P.W. Wong, eds.), vol. 6505, 2007b.
Without source camera:
M. Goljan, M. Chen, and J. Fridrich,
‘Identifying common source digital camera from image pairs’,
IEEE International Conference on Image Processing, 2007.
Xunyu Pan, Xing Zhang, and Siwei Lyu. Exposing image splicing with inconsistent local noise variances. In 2012 IEEE International Conference on Computational Photography (ICCP), pages 1–10. IEEE, 2012.
Extension to above using machine learning:
L.H. Chan, N.F. Law, and W.C. Siu,
“A two dimensional camera identification method based on image sensor noise,”
IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1741–1744, 2012.
T. Filler, J. Fridrich, and M. Goljan,
‘Using sensor pattern noise for camera model identification’,
IEEE International Conference on Image Processing, pp. 1296–1299, 2008.
(SKIP)Camera Response Function (CRF) Based Forgery Detection
Y.F. Hsu and S.F. Chang,
‘Detecting image splicing using geometry invariants and camera characteristics consistency’,
IEEE International Conference on Multimedia and Expo, July 2006.
Y.F. Hsu and S.F. Chang,
‘Image splicing detection using camera response function consistency and automatic segmentation’,
IEEE International Conference on Multimedia and Expo, July 2007.
Z. Lin, R.Wang, X. Tang, and H.Y. Shum,
‘Detecting doctored images using camera response normality and consistency’,
IEEE Conference on Computer Vision and Pattern Recognition, 2005
T.-T. Ng, S.-F. Chang and Q. Sun,
“Blind Detection of Photomontage Using Higher Order Statistics”,
IEEE ISCAS, May 2004.
Demosaicing and Color
Filter Array (CFA) based detection
A.C. Popescu and H. Farid,
“Exposing digital forgeries in colour filter array interpolated images,”
IEEE Transactions on Signal Processing, vol. 53, no. 10, pp. 3948–3959, 2005
A.C. Gallagher and T.H. Chen,
‘Image authentication by detecting traces of demosaicing’,
IEEE Workshop on Vision of the Unseen (in conjunction with CVPR), pp. 1–8, 2008
S. Bayram, H.T. Sencar, N. Memon, and I. Avciba,
‘Source camera identification based on CFA interpolation’,
IEEE International Conference on Image Processing. Genoa, Italy: IEEE, 2005.
S. Bayram, H.T. Sencar, and N. Memon,
‘Improvements on source camera model identification based on CFA interpolation’,
International Conference on Digital Forensics, 2006
H. Cao and A. Kot,
“Accurate detection of demosaicing regularity for digital image forensics,”
IEEE Trans. Inf. Forensics Secur., vol. 4, no. 5, pp. 899–910, Dec. 2009
A. Dirik and N. Memon, “Image tamper detection based on demosaicing artifacts,”
Proc. IEEE Int. Conf. Image Process., 2009, pp. 1497–1500.
J. Ho, O. Au, J. Zhou, and Y. Guo, “Inter-channel demosaicing traces
for digital image forensics,” in Proc. IEEE Int. Conf. Multimedia Expo,
2010, pp. 1475–1480.
Pasquale Ferrara, Tiziano Bianchi, Alessia De Rosa, and Alessandro Piva.
Image forgery localization via fine-grained analysis of CFA artifacts.
IEEE Transactions on Information Forensics and Security, 7(5):1566–1577, 2012. 2
(SKIP) JPEG Compression based forgery detection
This lecture detects forgery based on JPEG image compression. Due to the characteristics of JPEG compression, Tampering of an image can be detected from misalignment of the blocks or artifacts of double compression.
Based on the visual JPEG artifacts in the image (Blockiness)
W. Luo, Z. Qu, J. Huang, and G. Qiu,
‘A novel method for detecting cropped and recompressed image block’,
IEEE Conference on Acoustics, Speech and Signal Processing, pp. 217–220, 2007
W. Li, Y. Yuan, and N. Yu,
‘Passive detection of doctored JPEG image via block artifact grid extraction’,
IEEE Transactions on Signal Processing, vol. 89, no. 9, pp. 1821–1829, 2009
C. Iakovidou, M. Zampoglou, S. Papadopoulos, and Y. Kompatsiaris,
“Content-aware detection of JPEG grid inconsistencies for intuitive image forensics,”
J. Visual Commun. Image Representation, vol. 54, pp. 155–170, 2018.
Based on DCT Coefficients
Z. Fan and R. de Queiroz,
‘Identification of bitmap compression history: JPEG detection and quantizer estimation’,
IEEE Transactions on Image Processing, vol. 12, no. 2, pp. 230–235, 2003.
J. Fridrich, M. Goljanb, and R. Du,
‘Steganalysis based on JPEG compatibility’,
SPIE Multimedia Systems and Applications IV, pp. 275–280, 2001
S. Ye, Q. Sun, and E.C. Chang,
‘Detecting digital image forgeries by measuring inconsistencies of blocking artifact’,
IEEE International Conference on Multimedia and Expo, pp. 12–15, 2007
Based on recompression
W. Luo, J. Huang, and G. Qiu,
‘JPEG error analysis and its applications to digital image forensics’,
IEEE Transactions on Information Forensics and Security, vol. 5, no. 3, pp. 480–491, 2010.
Y.Q. Zhao, F.Y. Shih, and Y.Q. Shi,
‘Passive detection of paint-doctored JPEG images’,
International Work shop on Digital Watermarking, pp. 1–11, 2010.
A. Popescu and H. Farid,
“Statistical tools for digital forensics,”
Proc. Int. Workshop Inf. Hiding, 2004, pp. 128–147.
Xinghao Jiang; Peisong He; Tanfeng Sun; Feng Xie; Shilin Wang
Detection of Double Compression with the Same Coding Parameters Based on Quality Degradation Mechanism Analysis (video)
IEEE Transactions on Information Forensics and Security , Vol 13, Issue: 1, 2018
Based on Benford's Law
D. Fu, Y.Q. Shi, and W. Su,
‘A generalized Benford’s law for JPEG coefficients and its applications in image forensics’,
SPIE Conference on Security, Steganography, and Watermarking of Multimedia Contents (E.J. Delp and P.W. Wong, eds.), vol. 6505, 2007.
J. Lukas and J. Fridrich,
‘Estimation of primary quantization matrix in double compressed JPEG images’,
Digital Forensic Research Workshop, Auguat 2003.
Based on the Double Quantization (DQ) Effect
B. Mahdian and S. Saic,
‘Detecting double compressed JPEG images’,
International Conference on Imaging for Crime Detection and Prevention, 2009.
Z. Lin, J. He, X. Tang, and C.K. Tang,
‘Fast, automatic and fine-grained tampered JPEG image detection via DCT coefficient analysis’,
Pattern Recognition, vol. 42, no. 11, pp. 2492–2501, 2009
Using ML
Statistics of JPEG compressed images
Histograms of DCT coefficients are used as input to the CNN
Q. Wang and R. Zhang,
Double JPEG compression forensics based on a convolutional neural network
EURASIP J. Inf. Secur., vol. 23, pp. 1–12, 2016.
J. Park, D. Cho, W. Ahn, and H.-K. Lee,
Double JPEG detection in mixed JPEG quality factors using deep convolutional neural network
Proc. Eur. Conf. Comput. Vision, 2018, pp. 1–17.
statistical features are
extracted, so as to also enable localization.
Works on compressed images with QF greater than the trained ones.
M. Barni et al.
Aligned and non-aligned double JPEG detection using convolutional neural
networks
J. Visual Commun. Image Representation, vol. 49, pp. 153–163, 2017
Uses both spatial domain and frequency domain inputs.
I. Amerini, T. Uricchio, L. Ballan, and R. Caldelli,
Localization of JPEG double compression through multi-domain convolutional neural networks,
Proc. IEEE Comput. Vision Pattern Recognit. Workshops, 2017, pp. 1865–1871.
Double compression in video. analyze separately intra-coded frames and predictive frames
S.-H. Nam, J. Park, D. Kim, I.-J. Yu, T.-Y. Kim, and H.-K. Lee,
TwoStream Network for Detecting Double Compression of H.264 Videos,
Proc. IEEE Int. Conf. Image Process., 2019, pp. 111–115.
PART 2. Image Forgery Detection - NEW SCHOOL
Lecture 4 – Machine Learning for Copy-Paste and Splicing detection
An end-to-end CNN- implements the three main steps of a classic copy-move solution, i.e. feature
extraction, matching and post-processing to reduce false alarms. Outputs forgery location map
Y. Wu, W. Abd-Almageed, and P. Natarajan, “Image copy-move forgery detection via an end-to-end deep neural network,” in Proc. IEEE Winter Conf. Appl. Comput. Vision, 2018, pp. 1907–1915.
Uses CNN
Y. Rao and J. Ni.
A deep learning approach to detection of splicing and copy-move forgeries in images.
Information Forensics and Security (WIFS), 2016 IEEE International Workshop on, pages 1–6. IEEE, 2016
Uses a CNN that includes a branch
to detect boundaries between inserted regions and background and another
branch for the surface of the manipulation
R. Salloum, Y. Ren, and C. C. J. Kuo, “Image splicing localization using a multi-task fully convolutional network (MFCN),”
J. Visual Commun. Image Representation, vol. 51, pp. 201–209,
2018.
Rich feature extraction using Deep , then SVM for classification.
B. Bayar and M. Stamm, “A deep learning approach to universal image manipulation detection using a new convolutional layer,” in Proc. ACM Workshop Inf. Hiding Multimedia Secur., 2016, pp. 5–10.
constrained filter weights are learnt during training, and the CNN is used also for classification
Source-target disambiguation . Determine
which is original and which is copy
Y. Wu, W. Abd-Almageed, and P. Natarajan, “BusterNet: Detecting copymove image
forgery with source/target localization,” in Proc. Eur. Conf. Comput. Vision,
2018, pp. 170–186. [141] M. Barni, Q.-T. Phan, and B. Tondi, “Copy move
source-target disambiguation through multi-branch CNNs,” 2019,
arXiv:1912.12640v1
A segmentation network based on U-Net is proposed
X. Bi, Y. Wei, B. Xiao, and W. Li,
RRU-Net: The Ringed ResidualU-Net
for Image Splicing Forgery Detection,
Proc. IEEE Comput. Vision Pattern Recognit. Workshops, 2019, pp. 30–39.
Use non-overlapping image patches as input to CNNs.
Y. Wei, X. Bi, and B. Xiao. C2r net: The coarse to refined network for image forgery detection. In 2018 17th
IEEE International Conference On Trust, Security And Privacy In Computing And Communications/12th IEEE International Conference On Big Data Science And Engineering
(TrustCom/BigDataSE), pages 1656–1659. IEEE, 2018.
CNN is used to extract camera-model features from image patches, followed by clustering to detect anomalies.
L. Bondi, S. Lameri, D. Güera, P. Bestagini, E. Delp, and S. Tubaro,
Tampering detection and localization through clustering of camera-based CNN features,
Proc. IEEE CVPR Workshops, 2017, pp. 1855–1864.
Uses Patches in a CNN-LSTM network
H. Bappy, A. K. Roy-Chowdhury, J. Bunk, L. Nataraj, and B. Manjunath.
Exploiting spatial structure for localizing manipulated image regions.
Proceedings of the IEEE International Conference on Computer Vision, pages 4970–4979,
Camera-model features are compared between regions using 2 networks
O. Mayer and M. Stamm,
Learned forensic source similarity for unknown camera models,
Proc. IEEE Int. Conf. Acoust., Speech Signal Process., Apr. 2018, pp. 2012–2016.
Uses camera fingerprint based on noise. Use Siamese CNN to detect copy-paste and splicing
D. Cozzolino and L. Verdoliva, “Noiseprint: A CNN-based camera model fingerprint,” IEEE Trans. Inf. Forensics Secur., vol. 15, no. 1, pp. 144–159, 2020.
D. Cozzolino and L. Verdoliva, “Camera-based image forgery localization using convolutional neural networks,” in Proc. Eur. Signal Process. Conf., Sep. 2018, pp. 1386–1390.
D. Cozzolino and L. Verdoliva,
Single-image splicing localization through autoencoder-based anomaly detection,
Proc. IEEE Workshop Inf. Forensics Secur., 2016, pp. 1–6.
Uses stacked Autoencoder model.
Y. Zhang, J. Goh, L. L. Win, and V. L. Thing.
Image region forgery detection: A deep learning approach.
SG-CRC, pages 1–11, 2016.
Lecture 5 - Camera/Source Identification
M. Kharrazi, H. T. Sencar, and N. Memon,
“Blind source camera identification,”
International Conference on Image Processing, pp. 709–712, 2004.
Features extracted from picxel values – SVM used to classify
Extends the above
S. Bayram, H. T. Sencar, and N. Memon,
“Improvements on source camera-model identification based on CFA interpolation,”
Proc. Working Group 11.9 Int. Conf. Digital Forensics, FL, 2006.
M. J. Tsai, C. L. Lai, and J. Liu,
“Camera/mobile phone source identification for digital forensics,”
Proc. IEEE Int. Conf. Acoustic, Speech Signal Processing, 2007.
Using Machine Learning
G.M. Farinella, M.V. Giuffrida(B) , V. Digiacomo, and S. Battiato
On Blind Source Camera Identification
Sensor noise – correlation with cameras, feature extracted from images – SVM
Advanced Concepts for Intelligent Vision Systems (ACIVS 2015)
S Mandelli, D Cozzolino, P Bestagini, L Verdoliva, S Tubaro
CNN-based fast source device identification
IEEE Signal Processing Letters 27, 1285-1289, 2020
L. Bondi, L. Baroffio, D. Guera, P. Bestagini, E. J. Delp, and ¨ S. Tubaro.
First Steps Toward Camera Model Identification With Convolutional Neural Networks.
IEEE Signal Processing Letters (SPL), 24:259–263, 2017.
Lecture 6 – Inpainting Detection
Inpainting explained and Methods
Diffusion based methods (PDE & total variation inpainting)
Bertalmio, M.; Sapiro, G.; Caselles, V.; Ballester, C. Image inpainting. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques—SIGGRAPH ’00, New Orleans, LA, USA, 23–28 July 2000; ACM Press: New York, NY, USA, 2000; pp. 417–424.
Papafitsoros, K.; Schoenlieb, C.B.; Sengul, B. Combined First and Second Order Total Variation Inpainting using Split Bregman. Image Process. Line 2013, 3, 112–136
Exemplar-based image inpainting
Criminisi, A.; Pérez, P.; Toyama, K. Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 2004, 13, 1200–1212.
Patch based
Ružić, T.; Pižurica, A. Context-aware patch-based image inpainting using Markov random field modeling. IEEE Trans. Image Process. 2015, 24, 444–456
Xu, Z.; Sun, J. Image inpainting by patch propagation using patch sparsity. IEEE Trans. Image Process. 2010, 19, 1153–1165.
Machine Learning-Based Methods
Pathak, D.; Krahenbuhl, P.; Donahue, J.; Darrell, T.; Efros, A.A.
Context Encoders: Feature
Learning by Inpainting.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), 27–30 June 2016; pp. 2536–2544.
CNN
Cho, R.K.; Sood, K.; Channapragada, C.S.C.
Image Repair and Restoration Using Deep Learning. In Proceedings of the 2022 4th International Conference on Artificial Intelligence and Speech Technology (AIST), Delhi, India, 9–10 December 2022.
Diffusion based
Lugmayr, A.; Danelljan, M.; Romero, A.; Yu, F.; Timofte, R.; Van Gool, L.
RePaint: Inpainting using Denoising Diffusion Probabilistic Models.
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 11451–11461.
Inpainting Detection
Image Inpainting Forgery Detection: A Review
Adrian-Alin Barglazan, Remus Brad and Constantin Constantinescu
J. Imaging 2024, 10(2), 42;
https://www.mdpi.com/2313-433X/10/2/42#B56-jimaging-10-00042
Forensic Methods for Image Inpainting https://encyclopedia.pub/entry/45989
Y. Wu, W. AbdAlmageed, and P. Natarajan,
ManTra-Net: Manipulation tracing network for detection and localization of image forgeries with anomalous features,
Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2019, pp. 9543–9552.
Li, H.; Huang, J. Localization of deep inpainting using high-pass fully convolutional network. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 8301–8310.
Li, H.; Luo, W.; Huang, J.
Localization of Diffusion-Based Inpainting in Digital Images.
IEEE Trans. Inf. Forensics Secur. 2017, 12, 3050–3064.
Zhang, Y.; Liu, T.; Cattani, C.; Cui, Q.; Liu, S.
Diffusion-based image inpainting forensics via weighted least squares filtering enhancement. Multimed. Tools Appl. 2021, 80
Liang, Z.; Yang, G.; Ding, X.; Li, L. An efficient forgery detection algorithm for object removal by exemplar-based image inpainting.
J. Vis. Commun. Image R. 2015, 30, 75–85.
Wu, Q.; Sun, S.; Zhu, W.; Li, G.H.; Tu, D.
Detection of digital doctoring in exemplar-based inpainted images.
Proceedings of the 2008 International Conference on Machine Learning and Cybernetics, Kunming, China, 12–15 July 2008; Volume 3, pp. 1222–1226.
Zhang, D.; Liang, Z.; Yang, G.; Li, Q.; Li, L.; Sun, X.
A robust forgery detection algorithm for object removal by exemplar-based image inpainting.
Multimed. Tools Appl. 2018, 77, 11823–11842.
Lu, M.; Liu, S. A detection approach using LSTM-CNN for object removal caused by exemplar-based image inpainting. Electronics 2020, 9, 858.
Lecture 7 – Computer Graphics (CG) vs Natural image detection
Train CNN to extract compact features at the patch level, and then perform external aggregation to make image-level decisions.
N. Rahmouni, V. Nozick, J. Yamagishi, and I. Echizen, “
Distinguishing computer graphics from natural images using convolution neural networks,”
Proc. IEEE Workshop Inf. Forensics Secur., 2017, pp. 1–6.
Eric Tokuda, Helio Pedrini, Anderson Rocha
Computer generated images vs. digital photographs: A synergetic feature and classifier combination approach
Journal of Visual Communication and Image Representation 24(8):1276–1292, 2013
Lecture 8– Generative Adversarial Nets (GAN) image detection
GAN Explanation:
Generative Adversarial Nets
Part of Advances in Neural Information Processing Systems 27 (NIPS 2014)
Many good tutorials on internet.
GAN detection (GAN vs natural images) (NOT Faces):
Pixel-level detection of errors produced by GAN-.
X. Zhu, F. Che, T. Yang, T.-Y. Yu, D. Meger, and G. Dudek,
Detecting GAN generated errors, 2019, arXiv:1912.00527v1.
Based on Gan color artifacts
S. McCloskey and M. Albright,
Detecting GAN-generated imagery using saturation cues,”
Proc. IEEE Int. Conf. Image Process., 2019, pp. 4584–4588.
H. Li, B. Li, S. Tan, and J. Huang,
Detection of deep network generated
images using disparities in color components,
Sig. Process., vol. 174,pp. 1–12, Sep. 2020.
Classic supervised learning classification of GAN vs real images
F. Marra, D. Gragnaniello, D. Cozzolino, and L. Verdoliva,
Detection of GAN-generated fake images over social networks,
Proc. 1st IEEE Int. Workshop Fake MultiMedia, Apr. 2018, pp. 384–389.
Don’t work on low res. Features are co-occurrence matrices
L. Nataraj, T. Mohammed, B. Manjunath, S. Chandrasekaran, A. Flenner,
J. Bappy, and A. Roy-Chowdhury,
Detecting GAN generated fake images using co-occurrence matrices,
in Proc. IS&T Electron. Imag.,Media Watermarking, Secur. Forensics, 2019, pp. 532-1–532-7
Features are co-occurrence matrices
Lecture
9– Gan source identification
F. Marra, D. Gragnaniello, L. Verdoliva, and G. Poggi, “Do GANs
leave artificial fingerprints?” in Proc. 2nd IEEE Int. Workshop Fake
MultiMedia, Mar. 2019, pp. 506–511.
N. Yu, L. Davis, and M. Fritz, “Attributing fake images to GANs: learning
and analyzing GAN fingerprints,” in Proc. Int. Conf. Comput. Vision,
2019, pp. 7556–7566.
Salama, Mohammad, and Hagit Hel-Or.
Face-image source generator identification."
European Conference on Computer Vision, pp. 511-527. Cham: Springer International Publishing, 2020.
M. Albright and S. McCloskey,
Source generator attribution via inversion,
Proc. CVPR Workshops, 2019, pp. 96–103.
Lecture 10 - Faces (Still Images)
Face manipulation warping etc
Detecting Photoshopped Faces by Scripting Photoshop
[142] a CNN based solution is devised to detect artifacts introduced by a specific Photoshop tool, Face-Aware Liquify,
Image composition (splicing) – Detection from the geometry of eye shape
M.K. Johnson and H. Farid,
“Detecting photographic composites of people’,
in Proceedings of the International. Workshop on Digital Watermarking, 2007.
Face morphing detection
C. Seibold, W. Samek, A.
Hilsmann, P. Eisert
Accurate and Robust Neural Networks for Face Morphing Attack Detection,
Journal of Information Security and Applications, vol. 53, August 2020
Face morphing attack detection based on high-frequency features and progressive enhancement
Cheng-kun Jia, Yong-chao Liu,Ya-ling Chen
Frontiers in Neurorobotics, 05 June 2023 Volume 17 - 2023 |
GAN for face image generation – how it works
A Style-Based Generator Architecture for Generative Adversarial Networks
Tero Karras, Samuli Laine, Timo Aila
X[176] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of StyleGAN,” 2019, arXiv:1912.04958. [
Gan Generated Face detection
Based on Face landmark locations –
Yang, Y. Li, H. Qi, and S. Lyu, “
Exposing GAN-synthesized faces using landmark locations,
Proc. ACM Workshop Inf. Hiding Multimedia Secur., Jun. 2019, pp. 113–118.
Deep Learning Based Computer Generated Face Identification Using Convolutional Neural Network
L. Minh Dang 1,Syed Ibrahim Hassan 1,Suhyeon Im 1,Jaecheol Lee 2,Sujin Lee 1 andHyeonjoon Moon
Appl. Sci. 2018, 8(12), 2610
Diffusion Facial Forgery Detection
Harry Cheng, Yangyang Guo, Tianyi Wang, Liqiang Nie, Mohan Kankanhalli
MM '24: Proceedings of the 32nd
ACM International Conference on Multimedia
Computer Vision and Pattern Recognition
Abdul Qadir Rabbia Mahum Mohammed A. El-Meligy Adham E. Ragab Abdulmalik AlSalman Muhammad Awais
An efficient deepfake video detection using robust deep learning
Science direct Volume 10, Issue 5, 15 March 2024,
FakeSpotter: A Simple yet Robust Baseline for Spotting AI-Synthesized Fake Faces
Run Wang, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Yihao Huang, Jian Wang, Yang Liu
Lecture 11 - Deepfake: FaceSwap + Face2Face
How to create:
Faceswap. History of early work in: FaceForensics++: Learning to Detect Manipulated Facial Images
Also: https://github.com/MarekKowalski/FaceSwap/
Facke: a Survey on Generative
Models for Face Swapping
Wei Jiang, Wentao Dong arXiv:2206.11203 https://arxiv.org/abs/2206.11203
Face2Face- transfer of facial expressions of one person to another
Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. Face2Face: Real-Time Face Capture and Reenactment of RGB Videos.
In IEEE Conference on Computer Vision and Pattern Recognition, pages 2387–2395, June 2016.
M. R. Koujan, M. C. Doukas, A. Roussos and S. Zafeiriou,
Head2Head: Video-based Neural Head Synthesis,
2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)
History of early work in FaceForensics++: Learning to Detect Manipulated Facial Images
Also: https://github.com/MarekKowalski/FaceSwap/
Facke: a Survey on Generative
Models for Face Swapping
Wei Jiang, Wentao Dong arXiv:2206.11203 https://arxiv.org/abs/2206.11203
NeuralTextures
Justus Thies, Michael Zollhofer, and Matthias Nießner.
Deferred neural rendering: Image synthesis using neural textures.
ACM Transactions on Graphics 2019 (TOG), 2019
Deepfakes github. https://github.com/deepfakes/faceswap.
Fakeapp. https://www.fakeapp.com/.
Detect Deepfake images
Uses both low and high features in a two-stream network. patch based triplet network
P. Zhou, X. Han, V. Morariu, and L. Davis,
Two-stream neural networks for tampered face detection,
Proc. IEEE CVPR Workshops, 2017, pp. 1831–1839.
Baojin Huang, Zhongyuan Wang, Jifan Yang, Jiaxin Ai, Qin Zou, Qian Wang, Dengpan Ye;
Implicit Identity Driven Deepfake Face Swapping Detection
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 4490-4499
Xinyi Ding, Zohreh Raziei, Eric C. Larson, Eli V. Olinick, Paul Krueger & Michael Hahsler
Swapped face detection using deep learning and subjective assessment
EURASIP Journal on Information Security volume 2020, Article number: 6 (2020)
Deep faces including Face retargeting Swap-Face CGI detection and others
A. Khodabakhsh, R. Ramachandra, K. Raja, P. Wasnik, and C. Busch,
Fake face detection methods: Can they be generalized?” in
Biometrics and Electronic Sig and Proc. Int. Conf. Biometrics Special Interest Group, Sep. 2018.
Kaede Shiohara, Toshihiko Yamasaki
Detecting Deepfakes With Self-Blended Images
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 18720-18729
Lecture 12 - Moving Portraits, Facial Motion Retargeting
How to create:
ChangAn Zhu, Chris Joslin
A review of motion retargeting
techniques for 3D character facial animation
Computers & Graphics Volume 123, October 2024, 104037
Yeonsoo Choi, Inyup Lee, Sihun Cha, Seonghyeon Kim, Sunjin Jung, Junyong Noh
Deep-Learning-Based Facial Retargeting Using Local Patches
Computer Graphics Forum 2024
Few-Shot Adversarial Learning of Realistic Neural Talking Head Models
Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, Victor Lempitsky
In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9459-9468. 2019.
Deep video portraits
Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, Christian Theobalt
ACM Transactions on Graphics (TOG), Volume 37, Issue 4
Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu.
Depthaware generative adversarial network for talking head video generation.
In CVPR, 2022
Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, DIngzeyu Li
MakeItTalk: Speaker-Aware Talking-Head Animation
27 Apr 2020 ·
Lecture 13 – Forgery detection in Face Videos
Video Face Manipulations
Fit 3D model. Natural images do not follow model as well as CG
D.-T. Dang-Nguyen, G. Boato, and F. De Natale, “3D-modelbased video analysis for computer generated faces identification,” IEEE Trans. Inf. Forensics Secur., vol. 10, no. 8, pp. 1752–1763,
Visual Artifacts based (different
eye color, bad teeth synthesis [note these are in older versions. New generation
methods
do MUCH better])
F. Matern, C. Riess, and M. Stamminger, “Exploiting visual artifacts to expose deepfakes and face manipulations,” in Proc. IEEE WACV Workshop Image Video Forensics, 2019, pp. 83–92.
Eye blink rates
Y. Li, M.-C. Chang, and S. Lyu, “In Ictu Oculi: Exposing AI created fake videos by detecting eye,” in Proc. IEEE Workshop Inf. Forensics Secur., 2018.
Heart beat
U. Ciftci, I. Demir, and L. Yin, “FakeCatcher: Detection of synthetic
portrait videos using biological signals,” 2019, arXiv:1901.02212v2.
S. Fernandes et al., “Predicting heart rate variations of deepfake videos
using neural ODE,” in Proc. ICCV Workshops, 2019, pp. 1721–1729.
Head pose inconsistencies (extends the landmark based detection in stills [193])
X. Yang, Y. Li, and S. Lyu, “Exposing deep fakes using inconsistent head pose,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2019, pp. 8261–8265. [
Older non deep:
V. Conotter, E. Bodnari, G. Boato, and H. Farid,
Physiologically-based detection of computer generated faces in video,
Proc. IEEE Int. Conf. Image Process., Oct. 2014, pp. 1–5.
Face warping artifacts (to fit original need some warping)
Y. Li and S. Lyu,
Exposing deepfake videos by detecting face warping artifacts,
Proc. IEEE CVPR Workshops, 2019, pp. 46–52.
Facial expression and head movements are correlated. (when deepfake becomes very good)
S. Agarwal and H. Farid,
Protecting world leaders against deep fakes,
in Proc. IEEE CVPR Workshops, 2018, pp. 38–45.
Frame by frame based (still images)
D. Afchar, V. Nozick, J. Yamagishi, and I. Echizen,
“MesoNet: A compact facial video forgery detection network,”
Proc. IEEE Int. Workshop Inf. Forensics Secur., 2018, pp. 1–7.
MesoNet to directly classify real faces and fake faces generated by DeepFake and Face2face
Compare many different approaches but on STILLS taken from the videos
A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner,
FaceForensics++: Learning to detect manipulated facial images, Proc. Int. Conf. Comput. Vision, 2019, pp. 1–11.
Localization
viewing the problem from a segmentation perspective.
J. Li, T. Shen, W. Zhang, H. Ren, D. Zeng, and T. Mei,
Zooming into Face Forensics: A Pixel-level Analysis, 2019,
arXiv:1912.05790v1.
J. Stehouwer, H. Dang, F. Liu, X. Liu, and A. Jain,
On the detection of digital face manipulation,
IEEE Conf. Comput. Vision Pattern Recognit., pp. 5781–5790, 2020
Difficulty in low res images as in soc media (also above papers)
Uses triplet network:
Kumar and A. Bhavsar, Detecting deepfakes with metric learning,
Int. Workshop Biometrics Forensics, 2020.
Video Forgery detection with temporal consideration (not only frame by frame)
Frame by frame extracts features that are sent to LSTM across many frames (spatial + temporal)
D. Güera and E. Delp,
“Deepfake video detection using recurrent neural networks,
Proc. IEEE Int. Conf. Adv. Video Signal Based Surveillance, (AVSS) 2018.
S. Sohrawardi et al.,
Towards robust open-world detection of deepfakes,
Proc. ACM SIGSAC Conf. Comput. Commun. Secur., 2019, pp. 2613–2615.
E. Sabir, J. Cheng, A. Jaiswal, W. Abd-Almageed, I. Masi, and P. Natarajan, “Recurrent convolutional strategies for face manipulation detection in videos,” in Proc. CVPR Workshops, 2019, pp. 80–87.
Use optical flow
I. Amerini, L. Galteri, R. Caldelli, and A. D. Bimbo, “Deepfake Video Detection through Optical Flow based CNN,” in Proc. ICCV Workshops, 2019, pp. 1–3.
Look at boundaries (assume blending in forgery )
L. Li et al., “Face X-ray for more general face forgery detection,” 2019, arXiv:1912.13458v1.