A New Color Image Database TID2013: Innovations and Results Nikolay Ponomarenko1, Oleg Ieremeiev1, Vladimir Lukin1, Lina Jin2, Karen Egiazarian2, Jaakko Astola2, Benoit Vozel3, Kacem Chehdi3, Marco Carli4, Federica Battisti4, and C.-C. Jay Kuo5 1

National Aerospace University, Dept of Transmitters, Receivers and Signal Processing, 17 Chkalova St, 61070 Kharkov, Ukraine [email protected], [email protected], [email protected] 2 Tampere University of Technology, Institute of Signal Processing, P.O.Box-553, FIN-33101 Tampere, Finland {lina.jin,karen.egiazarian,jaakko.astola}@tut.fi 3 University of Rennes 1 - IETR, CS 80518, 22305 Lannion Cedex, France {benoit.vozel,kacem.chehdi}@univ-rennes1.fr 4 University of Rome III, via Ostiense , 161, Rome, Italy {marco.carli,federica.battisti}@uniroma3.it 5 Media Communications Lab, USC Viterbi School of Engineering, SAL 300, USA [email protected]

Abstract. A new database of distorted color images called TID2013 is designed and described. In opposite to its predecessor, TID2008, this database contains images with five levels of distortions instead of four used earlier and a larger number of distortion types (24 instead of 17). The need for these modifications is motivated and new types of distortions are briefly considered. Information on experiments already carried out in five countries with the purpose of obtaining mean opinion score (MOS) is presented. Preliminary results of these experiments are given and discussed. Several popular metrics are considered and Spearman rank order correlation coefficients between these metrics and MOS are presented and discussed. Analysis of the obtained results is performed and distortion types difficult for assessment by existing metrics are noted. Keywords: full reference metrics, image visual quality, mean opinion score, color image database, subjective experiments.

1

Introduction

Image quality assessment (IQA) is a hot research topic nowadays [1,2]. IQA plays a key role in many applications of digital image and video processing such as lossy compression of still images and video [3,4], watermarking [5], multimedia [6], image denoising [7], image printing, etc. All these applications require quality, in the first order, visual quality metrics able to adequately characterize images. Full-reference metrics for which a reference image (or frame sequence) is available with respect to which an impact of distortions is evaluated are studied better till the moment [6,8,9]. A large number of full-reference metrics has been proposed and tested. However, tests J. Blanc-Talon et al. (Eds.): ACIVS 2013, LNCS 8192, pp. 402–413, 2013. © Springer International Publishing Switzerland 2013

A New Color Image Database TID2013: Innovations and Results

403

have demonstrated that there is no universal metric corresponding to human perception perfectly and applicable equally well in different areas [10]. D. Chandler in his excellent work [9] stresses that creation of databases for metric testing and verification is one of challenges in current research. In fact, more than twenty image databases have been already created [9] including such databases as LIVE [11,12], Toyama [13], TID2008 [14], etc. A good database has to meet a set of requirements and to provide multiple opportunities for its designers and users (note that making such databases freely available has become a good tradition in community of IQA metric and database designers). First of all, MOS determined with a high accuracy and in a reliable manner is to be provided. Second, quite many types and levels of distortions, in particular, those ones typical for practice and emerging applications are to be taken into account and simulated in a proper manner. Third, different means for convenience of analysis and comparisons have to be offered. For almost five years, TID2008 [14] was the world largest freely available database according to the number of distorted images, number of distortion types and the number of volunteers who participated in experiments thus ensuring high accuracy of MOS estimation. Although this database was originally created and intended for design and verification of full-reference IQA metrics, it has been intensively exploited by image processing community for other purposes as testing and efficiency analysis of blind methods for noise variance estimation [15], color image denoising techniques [16], verification of no-reference metrics [17], etc., due to availability of already distorted color images. Concerning its main intention, TID2008 has been intensively used and, actually, has become a standard mean for metric verification and performance analysis. In particular, TID2008 has been exploited to increase Spearman rank order correlation coefficient (SROCC) [18] between metrics and MOS. Whilst in 2009 the largest SROCC for TID2008 was observed for the metric MSSIM [19] and it was approximately equal to 0.85, the SROCC has reached 0.95 for the combined metric BMMF [20] in 2012. This has been, in particular, achieved due to aggregating advantages of the metrics FSIM [21] and PSNR-H(M)A [22] proposed in 2011. Thus, near-optimum performance of the metrics that take into account peculiarities of human visual system (HVS) has been gained for TID2008 (maximal attainable SROCC approaches to unity). This means that the most advanced modern metrics are able to manage types and levels of distortions present in TID2008 quite well. This was one motivation for creating a new database of distorted color images that we have called TID2013 to show the year of its creation and that TID2008 has served as the basis. Compared to TID2008, TID2013 contains seven new types of distortions. They have been added to account for new emerging applications and specific interest to color distortions (see Section 2 for more details). Besides, TID2008 was criticized [23] for having mainly distorted images for which distortions can be easily noticed (observed) compared to reference (distortion-free) images. To get around this shortcoming, we have also added images with one more (fifth) level of distortions to TID2013. For these images, distortions are not apparent (can be noticed not always). Under these modifications, TID2013 now has 3000 distorted images. Below we give its more detailed description and present some preliminary results of experiments carried out using TID2013. In particular, we give SROCC values for several known HVS-metrics and pay a special attention to IQA for new types of distortions.

404

2

N. Ponomarenko et al.

TID2013 Description

TID2013 (available at http://ponomarenko.info/tid2013.htm) includes the same 25 reference color images as TID2008 where 24 distortion-free images are obtained (by cropping) from the Kodak database available at http://r0k.us/graphics/kodak/ and one more reference image (25th) is an artificially created (synthetic) image. All the images (reference and distorted) are of the same fixed size equal to 512x384 pixels. This size is chosen to simultaneously display three images (a reference image below and two distorted ones created on basis of the given reference in the upper part) at the monitor screen for performing pair-wise comparisons. The database TID2013 contains 3000 distorted images where 120 distorted images (five levels for each of twenty four types of distortions) have been obtained for each reference image. As already said, there are five levels of distortions. They have been mainly simulated in such a way that the first level (added to TID2013 compared to TID2008) approximately corresponded to peak signal-to-noise ratio (PSNR) equal to 33 dB and four remainder levels related to 30, 27, 24, and 21 dB, respectively. Note that PSNR was successfully exploited for level setting in TID2008 and the use of PSNR for this purpose does not give benefit to any visual quality metric. TID2013 contains images with all seventeen distortion types earlier present in TID2008, namely: additive white Gaussian noise (#1), additive white Gaussian noise which is more intensive in color components than in the luminance component (#2), additive Gaussian spatially correlated noise (#3), masked noise (#4), high frequency noise (#5), impulse noise (#6), quantization noise (#7), Gaussian blur (#8), image denoising (residual noise, #9), JPEG lossy compression (#10), JPEG2000 lossy compression (#11), JPEG transmission errors (#12), JPEG2000 transmission errors (#13), non-eccentricity pattern noise (#14), local block-wise distortions of different intensity (#15), mean shift (#16), contrast change (#17). In addition, the following seven types of distortions have been added after thorough discussions between teams of authors of this paper: Change of color saturation (#18), Multiplicative Gaussian noise (#19), Comfort noise (#20), Lossy compression of noisy images (#21), Image color quantization with dither (#22), Chromatic aberrations (#23), Sparse sampling and reconstruction (#24). The motivations for including just these types of distortions were the following. First of all, three types of introduced distortions (## 18, 22, 23) somehow relate to color. Note that color information and color distortions are paid sufficient attention by humans in IQA. However, TID2008 as well as other databases do not contain many distorted images that relate to possible distortions of color. Besides, color distortions are valuable for such application as color image printing [24] and others. Meanwhile, many HVS metrics are not adapted to accounting for color distortions and recent results clearly demonstrate [21, 25] that it is worth doing so. Second, existing databases contain images corrupted by additive or impulse noise but, to the best of our knowledge, none database has images corrupted by signaldependent noise. Different types of signal-dependent noise might be met in practice. We have chosen multiplicative Gaussian spatially uncorrelated noise (#19) as a marginal (specific) representative of signal-dependent noise. Note that the recent results [26] show that existing HVS-metrics are unable to adequately characterize visual quality of images corrupted by signal-dependent noise.

A New Color Image Database TID2013: Innovations and Results

405

Comfort noise (#20) takes into consideration a specific HVS feature that humans often do not distinguish images corrupted by different realizations of the noise added to the same reference image. Besides, human often do not recognize changes in textural regions. These properties are, e.g., exploited in lossy compression of color images and video when noise with the same characteristics as in original image or video is added after decompression to provide natural appearance [27]. TID2008 and other databases have been criticized for considering only “pure” types of distortions while distortions of several types can be present in real-life images jointly. To partly fill this gap, we introduced distortion type #21 – images with noise are subject to lossy compression [4, 28]. Obviously, analysis for such type of multiple distortions is important for several modern applications. The last introduced distortion type (#24) relates to sparse sampling and reconstruction (also called compressive sensing) of images. This is a hot topic nowadays [29, 30]. Although visual quality of reconstructed images is of great importance, HVS-metrics have not been yet used in this application. And we expect that they will be exploited soon. Note that there are many techniques of compressive sensing. For generating distorted images, the method [29] of compressive sensing available at our disposal has been used. Due to a limited space, we do not present details of distorted image generation here. Certainly, a larger number of distortion types could be included in the new database. However, we have to take into account several obstacles. First, we needed to have even number of distortion types to have equal number of pair-wise comparisons for each distorted image. Second, by increasing the number of distorted images for a given reference one, we make larger the time needed for carrying out each subjective experiment. Note that it is desirable and recommended to have a limited time for performing one experiment to avoid tiredness of observers (volunteers, experiment participants). In our case, average time spent for one experiment was about 17 minutes. Similarly to the methodology used for TID2008, the experiments have been performed in tristimulus manner. A reference image and two distorted images have been displayed simultaneously. An observer had to choose a higher visual quality image between two distorted ones by clicking on it. The preferred image got one point. Nine comparisons were done for each distorted image, the winning points were summed-up. In fact, “quality competition” was organized in a manner of Swiss system in chess competitions where “approximately the same strength players” compete. In other words, after a few starting rounds, images of approximately the same visual quality were displayed for comparing them. After getting the results for all observers, they were processed in a robust manner to reject abnormal ones. Such outliers occurred with probability about 2%. Then, the results were averaged for each tested image. Therefore, the obtained MOS varies from 0 to 9 where greater values of MOS relate to better visual quality assessed. Note that protocol (results of pair-wise comparisons) for each experiment has been documented and saved. This allows carrying out additional studies for determining and analyzing MOS. Before starting the experiment, the participants were instructed concerning preferred conditions and a methodology of experiments. For TID2013, the experiments were conducted in five countries (Ukraine, Finland, Italy, France, USA).

406

N. Ponomarenko et al.

The obtained results were in good agreement. The experiments were carried out both in laboratory conditions under tutor’s control and distantly via Internet. Approximately equal number of experiments was performed for each reference image. Three persons outside aforementioned countries took part in experiments. Some other data concerning experiments and accuracy of the obtained MOS can be found in Table 1. Table 1. Comparison characteristics of Databases LIVE, TID2008 and TID2013 Test image database

N

Main characteristics

1

Number of distorted images Number of different types of distortions

2

3

4

5 6 7 8 9

3

Number of experiments carried out

LIVE Database

TID2008

TID2013

779

1700

3000

5

17

24

161 (all USA)

Totally 838 (437 - Ukraine, 251 - Finland, 150 - Italy)

Totally 971 (602 - Ukraine, 116 - Finland, 101 - USA, 80 - Italy, 72 - France)

Evaluation using Methodology of visual quality five level scale (Excellent, Good, evaluation Fair, Poor, Bad) Number of elementary evaluations of image visual 25000 quality in experiments 0..100 Scale of obtained estimates of (stretched from MOS the scale 1..5) Variance of estimates of 250* MOS Normalized variance of 0.083* estimates of MOS Variance of MOS -

Pair-wise sorting (choosing the best that visually differs less from original between two considered) 256428

524340

0..9

0..9

0.63

0.69

0.031

0.035

0.019

0.018

Preliminary Results for a Limited Set of HVS-Metrics

Since the values of MOS are available and they have been obtained for a large number of conducted experiments (i.e., MOS values are accurate enough), it is possible to determine SROCC and Kendall rank order correlation coefficient (KROCC) for some known HVS-metrics. We have done this for the following metrics: FSIM [21] that has both component-wise and color versions, the latter is further denoted as FSIMc; MSSIM [19], NQM [31], SSIM [32], VIFP [33], VSNR [34], and WSNR [35]. All the latter metrics have been calculated for intensity component and computed using the software tool [36]. We have calculated SROCC and KROCC for the metrics PSNR-HVS [37], PSNR-HVS-M [38] and conventional PSNR computed for color image intensity as well. Finally, rank order correlation

A New Color Image Database TID2013: Innovations and Results

407

coefficients have been determined for the metrics PSNR-HA and PSNR-HMA [22]. These metrics are able to take into consideration different sensitivity of HVS to distortions in different color components. The metric PSNRc has been analyzed as well which is the color version of PSNR. This version is adapted to color images in the same manner as the aforementioned metrics PSNR-HA and PSNR-HMA [22]. To analyze the obtained results, the values of SROCC and KROCC are presented for all 24 types of distortions as well as for a subset of distortions that includes only the 7 newly introduced types of distortions (see data in Table 2). This is done to evaluate how “difficult” are the new types of distortions for the existing metrics. Recall that maximally possible SROCC values approach to unity and they are commonly larger than corresponding KROCC values. Meanwhile, as it follows from analysis of data in Table 2, a smaller SROCC usually corresponds to a smaller KROCC. Thus, conclusions drawn from analysis of these rank order correlation coefficients are in proper agreement. Due to this, we will basically analyze SROCC. As it is seen, even for the best HVS-metrics (the first three best results are indicated by bold) SROCC and KROCC are smaller than for the database TID2008 (e.g., SROCC for TID2008 is equal to 0.884 for FSIMc [21]). Moreover, the best (largest) SROCC values for new distortions subset are even smaller than for all distorted images in TID2013. This shows that, as the result, we have reached our main goal – created the database which is “difficult” for existing HVS-metrics. To our opinion, this will serve the goal of designing new metrics or modifying existing ones to provide better rank correlation, i.e. better adequateness of metrics to MOS. Table 2. Rank order correlation coefficients of HVS-metrics and MOS Metric Analyzed FSIM FSIMc MSSIM NQM PSNR PSNRc PSNR-HA PSVR-HMA PSNR-HVS PSNR-HVS-M SSIM VIFP VSNR WSNR

Color or not +

+ + +

All distorted images

SROCC

KROCC

0.8007 0.8510 0.7872 0.6349 0.6395 0.6869 0.8187 0.8128 0.6536 0.6246 0.6370 0.6084 0.6809 0.5796

0.6300 0.6669 0.6079 0.4662 0.4700 0.4958 0.6433 0.6316 0.5077 0.4818 0.4636 0.4567 0.5077 0.4463

New distortion subset SROCC KROCC 0.6494 0.7878 0.6314 0.6258 0.6190 0.7772 0.7008 0.7382 0.6471 0.6474 0.5801 0.5921 0.5888 0.6471

0.5236 0.6120 0.4952 0.4831 0.4728 0.5761 0.5416 0.5723 0.5169 0.5179 0.4226 0.4512 0.4374 0.5150

It is also interesting that there are many HVS-metrics for which SROCC and KROCC for all distorted images in TID2013 are smaller than for standard PSNR and PSNRc. This confirms the fact that it is difficult to create a universal metric applicable to various types of distortions. One more valuable observation is that all metrics that are adapted to color (sign + in the column “Color or not”) perform considerably better than other their intensity counterparts. Success of FSIMc can be

408

N. Ponomarenko et al.

explained by its ability to react to color distortions as well as by the fact that it takes into account human’s attention to edges and details in images. Fig. 1 presents the histogram of MOS for the created database. As it is seen, there are no images the quality of which has been perceived as perfect by observers (for which MOS approaches to 9). Meanwhile, there is certain percentage of images the visual quality of which is very poor (that possess MOS smaller than 1). 250

200

150

100

50

0

0

1

2

3

4

5

6

7

8

Fig. 1. Histogram of MOS values for TID2013

4

Analysis for Different Types of Distortions in TID2013

Distorted image databases can also serve several particular purposes (in additional to the main purpose of HVS-metrics verification). One of them is to analyze what types of distortions are perceived as more annoying than others. In this sense, it is interesting to consider dependence of MOS on distortion type and level. Such dependence is presented in Fig. 2. 7

Averaged MOS

6 5 4 3 2 1 0

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 Number of distortion type

16

17

18

19

20

21

22

23

24

Fig. 2. Dependence of MOS on distortion type and level

For each number (index) of distortion type (horizontal axis), there are five points indicating MOS values for five levels of distortions starting from the smallest level (that corresponds to PSNR=33 dB and is indicated by the leftmost point in each group). For almost all types of distortions, MOS decreases if distortion level becomes

A New Color Image Database TID2013: Innovations and Results

409

larger. The exceptions are distortion type #14 (non-eccentricity pattern noise) for which MOS values for first two levels are approximately the same, distortion type #15 (local block-wise distortions of different intensity) for which MOS is mainly determined by the number of inserted homogeneous blocks but not by their intensities (see [14] for more details), and distortion type #17 (contrast change) for which contrast enhancement (stretching dynamic range of image representation) leads to better perception of images. Note that for all introduced types of distortions (## 18…24) MOS values are smaller for larger level of distortions. The plot in Fig. 2 also allows determining what are the most annoying types of distortions (MOS for them is the smallest). For the first level, the most annoying are distortion types #3 (additive Gaussian spatially correlated noise). #6 (impulse noise), and #15 (local block-wise distortions of different intensity). On the contrary, for large levels of distortions, the most annoying are distortion types #11 (JPEG2000 lossy compression) and #24 (Sparse sampling and reconstruction). Fig. 3 presents MOS root mean square errors (RMSEs) for different types and levels of distortions in the same manner as Fig. 2. It is seen that for most types and levels the RMSEs are about 0.3, i.e. opinions of observers concerning visual quality coincide well. Meanwhile, there are several types and levels of distortions for which RMSE values are sufficiently larger. For example, this happens for distortion types #11 (JPEG2000 lossy compression), #12 (JPEG transmission errors), and #23 (Chromatic aberrations), especially if distortion level is high. In these cases, observers mainly judge are they able to retrieve the image content from distorted images [9] and their opinions in this sense vary a lot. 1.5

RMSE of MOS

1.2 0.9 0.6 0.3 0

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 Number of distortion type

16

17

18

19

20

21

22

23

24

Fig. 3. RMSEs of MOS for different types and levels of distortions

An opportunity to analyze data in subsets allows determining what types of distortions are “the hardest” for a given metric. For this purpose, a subset is formed of two types of distortions where additive white Gaussian noise (distortion type #1) is the first type and the second type is a considered one. Then, SROCC is calculated and analyzed. The values of SROCC for such subsets are presented in Table 3 for the database TID2013 for the metric FSIMc which has been found the best according to data in Table 2. In each cell of Table 3, we first give index of distortion type and then present the obtained SROCC. It follows from analysis that the hardest distortion type for FSIMc is #18 (Change of color saturation) for which SROCC is the smallest. Distortion types # 17 (Contrast change) and #15 (Local block-wise distortions of different intensity) are quite hard as well. These results show that either FSIMc should not be applied if these types of distortions are observed (are predicted to appear) in the processed images or this metric should be modified to better cope with these types of distortions.

410

N. Ponomarenko et al. Table 3. SROCC for FSIMc and MOS for subsets containing two types of distortions

2) 0.913 3) 0.935 4) 0.875 5) 0.921

6) 0.791 7) 0.907 8) 0.932 9) 0.900 10) 0.946

11) 0.949 12) 0.870 13) 0.924 14) 0.81 15) 0.721

16) 0.876 17) 0.668 18) 0.535 19) 0.886 20) 0.917

21) 0.926 22) 0.892 23) 0.894 24) 0.934

There is also one more way to present the metric values upon MOS. For this purpose, scatter-plots can be used and points of different colors represent estimates for different distorted images. If these points form a common cluster for the two considered types of distortions, a metric describes visual quality for both types similarly. If clusters are separate, there is a problem for a metric for one (or both) types. Since FSIMc is adequate for additive white Gaussian noise (AWGN), we can make scatter-plots for AWGN and any other type of distortions. As an example, a scatter-plot for AWGN (blue circles) Change of color saturation (distortion type # 18, red diagonal crosses) is presented in Fig. 4. Scatterplot of FSIMc values vs MOS 6

Additive noise Change of color saturation

5.5

MOS

5

4.5

4

3.5

3

0.88

0.9

0.92

0.94 FSIMc values

0.96

0.98

1

Fig. 4. Scatter-plots of MOS vs FSIMc for two types of distortions

As it is seen, the clusters do not coincide and this is the reason why SROCC for these pair of distortions is so small (see data in Table 3). FSIMc overestimates visual quality of images distorted by color saturations. Finally, it is also possible to analyze SROCC (and KROCC) between a given metric and MOS for distorted images generated on basis of a given reference image. The corresponding plot is given in Fig. 5 where horizontal axis corresponds to index of the reference image. The smallest SROCC is observed for image #13 which is the most textural. Quite small SROCC values also take place for images ## 1 and 5 which are textural as well. Meanwhile, the largest SROCC are observed for images ## 3 and 12 that are quite simple. This shows that the most complicated task, even for a good HVS-metric, is to adequately characterize visual quality of textural images.

A New Color Image Database TID2013: Innovations and Results

411

Spearman correlation for reference images 0.92 0.9

Spearman correlation

0.88 0.86 0.84 0.82 0.8 0.78 0.76 0.74 1

2

3

4

5

6

7

8

9

10

11 12 13 14 15 Reference image

16 17

18

19 20

21

22 23

24

25

Fig. 5. SROCC vs image index in database for FSIMc

5

Conclusions

The paper presents the recently created database TID2013 that contains images with seven new types and one additional (fifth) level of distortions. The motivations for introducing these distorted images are given. It is shown that the introduced distortion types make IQA a more difficult task for most of known HVS-metrics. To our opinion, this can stimulate further design of HVS-metrics and/or improving their performance. Several ways of analyzing the obtained data are described and illustrated. These ways show distortion types for which IQA is a difficult task for observers, i.e. their assessments vary a lot. Besides, analysis allows determining what types of distortions is difficult for a given HVS-metric and, thus, what are possible directions of its modifying to provide better performance. Analysis has also demonstrated that quality evaluation for the metric FSIMc is a more difficult task for highly textural images. Similar analysis can be done for other HVS-metrics and it might clarify ways of their further improvement.

References 1. Keelan, B.W.: Handbook of Image Quality. Marcel Dekker, Inc., New York (2002) 2. Wu, H.R., Lin, W., Karam, L.: An Overview of Perceptual Processing for Digital Pictures. In: Proceedings of International Conference on Multimedia and Expo Workshops, Melbourne, pp. 113–120 (2012) 3. Larson, E.C., Chandler, D.M.: Most apparent distortion: full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging 19(1), 011006 (2010) 4. Ponomarenko, N., Krivenko, S., Lukin, V., Egiazarian, K.: Lossy Compression of Noisy Images Based on Visual Quality: A Comprehensive Study. EURASIP Journal on Advances in Signal Processing, 13 (2010), doi:10.1155/2010/976436 5. Carli, M.: Perceptual Aspects in Data Hiding. Thesis for the degree of Doctor of Technology, Tampere University of Technology (2008)

412

N. Ponomarenko et al.

6. Moorthy, A.K., Bovik, A.C.: Visual Quality Assessment Algorithms: What Does the Future Hold? Multimedia Tools and Applications 51(2), 675–696 (2011) 7. Fevralev, D., Lukin, V., Ponomarenko, N., Abramov, S., Egiazarian, K., Astola, J.: Efficiency analysis of DCT-based filters for color image database. In: Proceedings of SPIE Conference Image Processing: Algorithms and Systems VII, San Francisco, vol. 7870 (2011) 8. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13(4), 600–612 (2004) 9. Chandler, D.M.: Seven Challenges in Image Quality Assessment: Past, Present and Future Research. In: ISNR Signal Processing, vol. 2913, pp. 1–53 (2013) 10. Jin, L., Egiazarian, K., Jay Kuo, C.-C.: Perceptual Image Quality Assessment Using BlockBased Milti-Metric Fusion (BMMF). In: Proceedings of ICASSP, Kyoto, pp. 1145–1148 (2012) 11. Sheikh, H.R., Wang, Z., Cormack, L., Bovik, A.C.: LIVE Image Quality Assessment Database Release 2, http://live.ece.utexas.edu/research/ quality/subjective.htm 12. Sheikh, H.R., Sabir, M.F., Bovik, A.C.: A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms. IEEE Transactions on Image Processing 15(11), 3441–3452 (2006) 13. Horita, Y., Shibata, K., Parvez Saddad, Z.M.: Subjective quality assessment toyama database, http://mict.eng.u-toyama.ac.jp/mict/ 14. Ponomarenko, N., Lukin, V., Zelensky, A., Egiazarian, K., Carli, M., Battisti, F.: TID2008 - A Database for Evaluation of Full-Reference Visual Quality Assessment Metrics. Advances of Modern Radioelectronics 10, 30–45 (2009) 15. Uss, M., Vozel, B., Lukin, V., Abramov, S., Baryshev, I., Chehdi, K.: Image Informative Maps for Estimating Noise Standard Deviation and Texture Parameters. EURASIP Journal on Advances in Signal Processing, 961–964 (2011) 16. Lukin, V., Ponomarenko, N., Egiazarian, K.: HVS-Metric-Based Performance Analysis of Image Denoising Algorithms. In: Proceedings of EUVIP, pp. 156–161 (2011) 17. Vu, C.T., Phan, T.D., Chandler, D.M.: S3: a Spectral and Spatial Measure of Local Perceived Sharpness in Natural Images. IEEE Transactions on Image Processing 21(3), 934–945 (2012) 18. Kendall, M.G.: The advanced theory of statistics, vol. 1. Charles Griffin & Company limited, London (1945) 19. Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multi-scale structural similarity for image quality assessment. In: IEEE Asilomar Conference on Signals, Systems and Computers, pp. 1398– 1402 (2003) 20. Jin, L., Egiazarian, K., Jay Kuo, C.-C.: Performance comparison of decision fusion strategies in BMMF-based image quality assessment. In: Proceedings of APSIPA, Hollywood, pp. 1–4 (2012) 21. Zhang, L., Mou, X., Zhang, D.: FSIM: a feature similarity index for image quality assessment. IEEE Transactions on Image Processing 20(5), 2378–2386 (2011) 22. Ponomarenko, N., Eremeev, O., Lukin, V., Egiazarian, K., Carli, M.: Modified image visual quality metrics for contrast change and mean shift accounting. In: Proceedings of CADSM, Polyana-Svalyava, pp. 305–311 (2011) 23. Larson, E.C., Chandler, D.M.: Most apparent distortion: full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging 19(1), 011006 (2010)

A New Color Image Database TID2013: Innovations and Results

413

24. Pedersen, M., Bonnier, N., Hardeberg, J.Y., Albregtsen, F.: Attributes of Image Quality for Color Prints. Journal of Electronic Imaging 19(1), 011016-1–011016-13 (2010) 25. Hassan, M., Bhagvati, C.: Structural Similarity Measure for Color Images. International Journal of Computer Applications 43(14), 7–12 (2012) 26. Ponomarenko, N.N., Lukin, V.V., Ieremeyev, O.I., Egiazarian, K., Astola, J.: Visual quality analysis for images degraded by different types of noise. In: Proceedings of SPIE Symposium on Electronic Imaging, San Francisco, vol. 8655, p. 12. SPIE (2013) 27. Oh, B.T., Jay Kuo, C.-C., Sun, S., Lei, S.: Film Grain Noise Modeling in Advanced Video Coding, SPIE Proceedings, Vol. In: SPIE Proceedings, San Jose, vol. 6508, p. 12 (2007) 28. Petrescu, D., Pincenti, J.: Quality and noise measurements in mobile phone video capture. In: SPIE Proceedings, San Francisco, vol. 7881, p. 14 (2011) 29. Danielyan, A., Foi, A., Katkovnik, V., Egiazarian, K.: Spatially adaptive filtering as regularization in inverse imaging: compressive sensing, upsampling, and super-resolution. In: Milanfar, P. (ed.) Super-Resolution Imaging, CRC Press / Taylor & Francis (2010) 30. Paredes, J.L., Arce, G.R.: Compressive Sensing Signal Reconstruction by Weighted Median Regression Estimate. IEEE Transactions on Signal Processing 59(6), 2585–2601 (2011) 31. Damera-Venkata, N., Kite, T., Geisler, W., Evans, B., Bovik, A.: Image Quality Assessment Based on a Degradation Model. IEEE Transactions on Image Processing 9, 636–650 (2000) 32. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13(4), 600–612 (2004) 33. Sheikh, H.R., Bovik, A.C.: Image Information and Visual Quality. IEEE Transactions on Image Processing 15, 430–444 (2006) 34. Chandler, D.M., Hemami, S.S.: VSNR: A Wavelet-Based Visual Signal-to-Noise Ratio for Natural Images. IEEE Transactions on Image Processing 16(9), 2284–2298 (2007) 35. Mitsa, T., Varkur, K.: Evaluation of contrast sensitivity functions for the formulation of quality measures incorporated in halftoning algorithms. In: IEEE International Conference on Acoustic, Speech, and Signal Processing, Minneapolis, vol. 5, pp. 301–304 (1993) 36. Gaubatz, M.: Metrix MUX Visual Quality Assessment Package, http://foulard.ece.cornell.edu/gaubatz/metrix_mux 37. Egiazarian, K., Astola, J., Ponomarenko, N., Lukin, V., Battisti, F., Carli, M.: New fullreference quality metrics based on HVS. In: Proceedings of the Second International Workshop on Video Processing and Quality Metrics, Scottsdale, p. 4 (2006) 38. Ponomarenko, N., Silvestri, F., Egiazarian, K., Carli, M., Astola, J., Lukin, V.: On between-coefficient contrast masking of DCT basis functions. In: Proc. of the Third International Workshop on Video Processing and Quality Metrics, USA, p. 4 (2007)