5th COST 276 Workshop (2003), pp. 1-5 B. Kovář, J. Přikryl, and M. Vlček (Editors)

Removal of Semi-Transparent Blotches in Old Photographic Prints F. Stanco, G. Ramponi, and L. Tenze DEEI - University of Trieste, via A. Valerio 10, Trieste, Italy {fstanco, ramponi}@units.it, [email protected] Abstract

In this paper a new algorithm to restore semi-transparent blotches (or water blotches) in vintage photographic prints is proposed. It works in two steps: the first stage is based on an additive/multiplicative model the optimal parameters of which are searched for around the area to be restored. The second one performs an interpolation along the gradient direction on the contour. The results obtained show that the stains are completely removed without producing visible artifacts. Semi-transparent blotches, cultural heritage, image restoration, defect detection.

1.

Introduction

Old photographic prints represent a huge reservoir of documents of high cultural value that must be preserved with care. Unfortunately, they are based on fragile materials, which are easily affected by bad environmental conditions. It is possible to classify the defects of a print into three sets depending on their origin [15]. Mechanical damage: defects are originated by an inaccurate handling of the photographic prints (cracks, scratches, holes, added stamps or text). Chemical damage: defects often are caused by water, humidity (semi-transparent blotches) or chemical reactions between the print and some microorganisms (foxing). Deposited matters: extraneous matter may adhere to the surface creating small spots that cover the original image. With classical, physical restoration (filling the gaps, consolidating the emulsion, fixing the gelatin, and repainting some parts by hand) we can produce cleaner images but only in an extremely expensive way, and often the damage still remains visible. However the diffusion of scanners and of software to manipulate the image opens a new way to the recovery of photos. The old image, after the digitization, can be virtually restored and, if necessary, reprinted. For this type of digital restoration, the goal is that a viewer should not be able to detect the areas where the image has been retouched. It has to be noted that this case is different from the one of the physical restoration of originals, where the original artwork needs to be preserved. Commercial software exists, which proposes heavy user-guided restoration procedures, where the defects are not automatically traced, and also all the corrections have to be user-suggested. Virtual restoration becomes complex and expensive, and it can be performed only by skilled personnel. Automated restoration is hence required to obtain quick, less expensive, simple and effective results.

In this paper we propose a technique to restore photographic prints affected by semi-transparent blotches (water blotches). The peculiarity of this type of damage is that it preserves a part of the original data; in fact each pixel in the blotch contains both valid information and noise. Our method exploits an additive/multiplicative model to eliminate the defect; values for the model parameters are searched for in a suitable area outside the blotch. Then, we perform an interpolation to reduce the artifacts close to the contour of the blotch. The rest of the paper is organized as follows: Section II provides a description of the semi-transparent blotches; Section III describes the algorithm; Section IV shows the experimental results. A conclusions section ends the paper. 2.

Semi-Transparent Blotches

Semi-transparent blotches (also called water blotches) are typical chemical damages due to the careless conservation of photographic prints: some examples are reported in Fig. 1. Humidity and water create blotches in the image that do not completely mask the original information, but add “noise” to such information. Usually, this kind of defect does not present a regular structure or a typical size. These depend on the intensity and diffusion of the humidity that affected the print. Visually they can be easily identified as stains with darker colors compared with the rest of the image. In particular, the contour of the blotch is usually darker than the blotch. This happens because the water droplets, while diffusing in the paper, drag dust and dirt which tend to accumulate close to the borders of the stain. The amount of original information still available in the border region of the blotch changes in each case. To decide how to work in these zones in order to reduce possible artifacts is a part of the problem treated in this paper.

Stanco et al / Removal of Semi-Transparent Blotches in Old Photographic Prints

The general restoration problem, like other similar problems, can be tackled in two phases [15]: detection and actual restoration. The defects in photographic prints (blotches, cracks, scratches) are very difficult to automatically detect [4, 9, 13, 14, 16, 18], since there are no simple rules to distinguish them from real image features. In particular, specific algorithms to treat water blotches do not exist in the scientific literature: we found that the best way is to use a semi-automatic algorithm. In this case, the users have to manually select one or more points in the damaged area, and automatically the remaining region of the damage is detected. Barni et al. [2], analyse an analogous but different problem, considering that the damage is darker than the background and is characterized by approximately uniform gray levels. The system works iteratively: it analyzes if the gray level of a pixel differs significantly from the uniform range, and hence it decides if this pixel can be classified as belonging to the crack. In [5] the classification is performed optimizing a function based on the luminance uniformity and on the magnitude of the gradient values in the region. More precisely, a pixel is deemed to belong to the damaged area if the new area obtained adding this pixel to the damaged area increases the value of a suitable function. Despite the accuracy of the results locating cracks or gaps in artworks, these algorithms do not provide good results with water blotches, because the blotch is not characterized by a uniform gray level, due to the above-mentioned presence of some image information.

the blotch stay out of the selection and vice-versa. Appropriate measures will be adopted and presented in the next subsections in order to avoid that this disadvantage could compromise the accuracy of the results.

(a)

(b)

(c)

It can be helpful to note that scanners with new technologies like Digital ICE (Image Correction and Enhancement) [6] or FARE (Film Automatic Retouching and Enhancement) [7] cannot be used because they can find scratches and dust only in transparent films and not in photographic prints. The second step suggested in [15] is the actual restoration of the damaged portion of the print. The goal is to eliminate the defect from the image automatically, and without any visible mark in the final image. Since the semi-transparent blotches contain residual information that we want to utilize, classical image inpainting algorithms [1, 3, 11, 12] or other methods that replace the original information [8, 10, 16, 18] can not be used. These algorithms indeed have been created to replace a part of the image rather than to recover possible residual information. 3.

The Proposed Algorithm

3.1.

Detection

To remove the semi-transparent blotches, first we need to find the pixels that belong to the damaged zone. In the previous section some detection techniques have been analyzed and we concluded that methods for the automatic detection of water blotches do not exist. For our experiments we manually select a pixel in the stain and then we label as part of the blotch all the neighbor pixels whose gray level is close to the selected pixel. This selection/labeling procedure is repeated several times, until all the pixels in the blotch are selected. It should be observed that the number of iterations necessary to obtain a good result can be very high and the final accuracy is affected by subjective factors. It can happen that pixels of

(d)

(e)

Figure 1: Semi-transparent blotches or water blotches.

3.2.

Restoration

The algorithm we propose to remove semi-transparent blotches can be summarized in two steps. The former restores all the pixels in the interior of the blotch, while the latter refines the contour area. Step 1. As mentioned above the pixels belonging to the water blotch have residual information. In the first step we recover this information using an additive-multiplicative model. Values for the model parameters are automatically extracted by examining a suitable area around the blotch. Similarly to [17] we want to approximate the uncorrupted image I(x,y) starting from the actual available image. If we denote with J(x,y) the actual image data, and with Ω x , y the zone to restore, we can write our model as:

J (Ω x, y ) = α * I (Ω x , y ) + β

(1)

where α and β are the parameters to determine. We should be able to obtain an estimate of the error-free image by inverting Eq. (1). Therefore, the problem is reduced to determine suitable values for the α and β parameters. To

Stanco et al / Removal of Semi-Transparent Blotches in Old Photographic Prints

this purpose, denoting with var[.] and E[.] respectively the variance and the mean in [.] and using Eq. (1), we can state that

[

]

[

var J (Ω x, y ) = α 2 var I (Ω x, y )

[

]

[

]

]

(2)

E J (Ω x, y ) = αE I (Ω x, y ) + β

Figure 3: Image 1a after Step 1.

However, the variance and mean of the uncorrupted image I are unknown; we circumvent this problem as ' follows. We denote with Ω x , y an area around the blotch

Ω x , y with width W. To avoid using pixels that are too close to the stain, and hence are unreliable, Ω x , y is automatically '

determined as a strip around the blotch shifted by S pixels away from the contour. This shift allows that erroneous portions of the border (as expressed in the previous section) do not compromise the accuracy of the final result. In Fig. 2 the area of the blotch Ω x , y is shown in black, and

Step 2. After Step 1 we have a good-looking image but a still apparent delimitation of the blotch. This is due to the structure of the defect, i.e. the fact that the contour is darker than the rest of the blotch. Indeed, if we choose a line in the image and plot it, it is possible to notice that the blotch area starts and ends with a negative luminance peak (Fig. 4a). This peak is still present after the first restoration step (Fig. 4b) and determines the visible contour in the restored image. To avoid this artifact is the main task of Step 2. We want to eliminate the peak using a linear interpolation across the contour.

Ω 'x , y is shown in white.

Figure 2: In black are reported all the pixesl into semi-transparent blotch, while in white are painted the good zone where extracted the parameters.

In order to solve Eq, (2), we approximate I (Ω x , y ) with

I (Ω'x , y ) . This replacement permits to estimate the values of α and β:

var[J (Ω x , y )] α~ = var[I (Ω 'x , y )] (3) ~ β = E [J (Ω x , y )] − α * E [I (Ω 'x , y )] Now, inverting Eq. (1) and using the estimated

~

parameters α~ and β , it is possible to compute the restored value for each pixel of the blotch. More precisely:

~

~ ~ I (Ω x , y ) = ( J (Ω x , y ) − β ) α~ .

Where I is the restored image. Fig. 3 shows an example of the restored image. It is possible to notice that the part inside the blotch is perfectly restored and it is similar to the rest of the image. However, the contour of the blotch is still apparent.

Figure 4: (a) The original gray level in the row 225. (b) The restored line is plotted in blue. (c) In black is reported the restored gray level after the interpolation.

Stanco et al / Removal of Semi-Transparent Blotches in Old Photographic Prints

Step 2a. To compute a linear interpolation across the border of the stain, first of all the luminance gradient is calculated for each pixel p in the contour. Then, we consider an array of 2L+1 pixels centred on p. This array contains L pixels belonging to the blotch and L outside the blotch chosen along the gradient direction indicated in p. If we denote with Pstart the first pixel of the array and with

the original. This feature allows to restore large damaged areas with details without introducing false information. In particular the images in (c) and (e) have details like the decoration of the columns that have been preserved eliminating only the water drop effect.

Pend the last, a linear interpolation is performed between these two values depending on the distance of the pixel. If d ( Pi ) =

Pi − Pend is the normalized distance of each Pstart − Pend

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(l)

pixel in the array from the Pstart position, the new intensity values are:

~ ~ ~ I ( Pi ) = d ( Pi ) * I ( Pstart ) + (1 − d ( Pi )) * I ( Pend ) It is important to notice that the new values in the vector depend only on the intensities in Pstart and Pend , and the original values are not considered. Step 2b. After Step 2a some pixels are not adjusted. This happens because the gradient directions can be similar for neighbor pixels, or they can be very different. In the latter case some pixels are interpolated several times, and other pixels are left unchanged. For this reason, we search all the pixels unchanged in a border strip of width 2L+1, and assign each pixel a gray level value corresponding to the average of its already interpolated neighbors in a 3×3 area. Fig. 5a shows in white all the pixels of the blotch that have been processed during Step 2a. It is possible to notice that the white strip is not continuous. In the interpolation step the unmarked pixels are processed, so that the strip is now continuous (Fig. 5b) and the artifact reduced.

Figure 5: (a) the pixel of the image 1a processed in the Step 1; (2) the pixel processed in image 1b after the Step 2.

4.

Experimental results

This section shows some results obtained applying the algorithm to old photographic prints. The method works over the user-selected areas and does not process the entire image if it is not necessary. The algorithm parameters have been set experimentally as follows: the distance S between the stain and the uncorrupted area (Step 1) is chosen equal ' to 7; the width W of the area Ω x ,y (Step 1) is 10; and in Step 2b it is set L = 3. Fig. 6 reports some results before and after Step 2. It is possible to notice that the algorithm enhances the information in the blotch and does not replace it with a texture or with new gray levels values not correlated with

Figure 6: (a) The image 1a after Step 1; (b) image 1b after Step 2; (c) image 1b after Step 1; (d) image 1b after Step 2; (e) image 1c after Step 1; (f) image 1c after Step 2; (g) image 1d after Step1; (h) image 1d after Step 2; (i) image 1e after Step 1; (l) image 1e after Step 2.

We have reported both the images before and after the second step to show that the last step eliminates the contour effects improving the final visual quality of the algorithm.

Stanco et al / Removal of Semi-Transparent Blotches in Old Photographic Prints

Sometimes (Fig. 6l) the contour is still visible. This happens because the original contour (strongly dark) is particularly wide; the area of interpolation does not cover this region. Larger areas would improve the performance in this respect but the interpolation would become more visible. Figs. 6c-e show that the non textured details in the images are preserved. Similarly all the other kinds of defects that are inside the blotch are not modified. The crack in Figs. 6i-l and the black small spots in Figs. 6e-f are examples of this fact. They can be later processed with a suitable algorithm for that defect. 5.

In order to further reduce the remains of the water blotch after the restoration, we are testing a quadratic additive/multiplicative model and a non linear interpolation algorithm to eliminate the artifacts within the contour. We are also working to create a new algorithm to automatically detect the blotch. Acknowledgment

We thank F.lli Alinari s.p.a for providing all the pictures used in our experiments. This work has been partially supported by a grant of the Regione Friuli Venezia Giulia and by the University of Trieste. References

[2] [3] [4]

[5]

[6] [7]

[9]

[10] [11] [12]

Conclusions

We have proposed a new algorithm to restore the semitransparent blotches also called water blotches. It is based on two steps: the first one restores the entire blotch using an additive/multiplicative model that finds its parameters in a zone outside the blotch. The second step interpolates along the gradient direction all the pixels in the contour to avoid artifacts. The experimental results show that the stains are removed without any visible artifacts enhancing the residual information in the area.

[1]

[8]

C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, J. Verdera, “Filling-in by Joint Interpolation of Vector Fields and Gray Levels”, IEEE Transactions on Image Processing, vol. 10 (8), pp. 1200- 1211, 2001. M. Barni, F. Bartolini, V. Cappellini, “Image Processing for Virtual Restoration of Artworks”, IEEE MultiMedia, 7(2), pp. 34-37, 2000. M. Bertalmio, G. Sapiro, V. Caselles, C. Ballester, “Image Inpainting”, In Proceedings of SIGGRAPH 2000, pp. 417-424, 2000. J. Chen, A.K. Jain, “A Structural Approach to Identify Defects in Textured Images”, In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, pp. 29-32, Beijng, 1988. A. De Rosa, A.M. Bonacchi, V. Cappellini, M. Barni, “Image Segmentation and Region Filling for Virtual Restoration of Art-Works”, In Proceedings of ICIP 2001, vol. 1. pp. 562-565, 2001. http://www.asf.com/products/ http://www.canon.com.sg/index.cfm?fuseaction=scan ner&prod_type=fare

[13] [14]

[15] [16] [17] [18]

http://www.polaroid.com/service/software/poladsr/po ladsr.html N.D. Kim, S. Udpa, “Nonlinear Operators for Edge Detection and Line Scratch removal”, In Proceedings of IEEE International Conference on SMC, pp. 44014404, 1998. A. Kokaram, Motion Picture Restoration, Springer, 1998. S. Masnou, J.M. Morel, “Level lines based disocclusion”, In Proceedings of ICIP 98, pp. 259263, 1998. M.M. Oliveira, B. Bowen, R. McKenna, Y.S. Chang, “Fast Digital Image Inpainting”, In Proceedings of the International Conference on Visualization, Imaging and Image Processing (VIIP 2001), Marbella, Spain, pp. 261-266, 2001. F. Roli, “Measure of texture anisotropy for crack detection on textured surfaces”, Electronics Letters, 36(14), pp. 1274-1275, 1996. K.Y. Song, M. Petron, J. Kitter, “Wigner based Crack Detection in Textured Images”, In Fourth IEEE International Conference on Image Processing and its Applications, pp. 315-318, 1992. F. Stanco, G. Ramponi, A. de Polo, “Towards the Automated Restoration of Old Photographic Prints: A Survey”, In Proceedings of IEEE-EURACON, 2003. D. Tegolo, F. Isgrò, “A Genetic Algorithm for Scratch Removal in Static Images”, In Proceedings of ICIAP 2001, pp. 507- 511, 2001. L. Tenze, G. Ramponi, “Line Scratch Removal in Vintage Film Based on an Additive/Multiplicative Model”, In Proceedings of IEEE-NSIP, 2003. L. Tenze, G. Ramponi, S. Carrato, “Robust detection and Correction of Blotches in Old Films using SpatioTemporal Information”, In Proceedings of SPIE Intern. Symp. Electronic Imaging 2002, 2002.