Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
The JPEG XT Suite of Standards: Status and Future Plans
Thomas Richter (
[email protected]) Tim Bruylants (
[email protected]) Peter Schelkens (
[email protected]) Touradj Ebrahimi (
[email protected])
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
Overview ●
Status of the JPEG XT Standard – – –
●
JPEG XT Profiles for HDR Encoding –
● ● ●
Structure of the Standard Architecture of JPEG XT Transport and Encoding Performance Evaluation
Lossless Encoding Encoding of Opacity Information Future Plans – – –
JPEG Privacy Interactive Image Browsing Recording of Editing Information
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
JPEG-XT – Structure of the Standard JPEG 10918-1
Part 1: Baseline (8bpp, lossy) + JFIF
Part 2: Dolby JPEG HDR JPEG 10918-5 JFIF
Part 3: Box-Based File Format Part 6: Coding of IDR Images
Part 7: Coding of HDR Images
Part 8: Lossless Coding
Part 9: Opacity Coding/Alpha Channels
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
JPEG XT Status ● ● ● ●
Part 3 (file format) becomes IS Parts 6 & 7 (IDR & HDR) become FDIS Parts 8 & 9 (lossless, alpha) become DIS Part 5 (reference software) goes CD
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
Architecture of the JPEG XT Decoder ●
One common decoder architecture for all parts
JPEG Decoder JPEG Image Decoder Base Base Image
Inverse Inverse QNT QNT
FDCT FDCT or or IDCT IDCT
Chroma Chroma UpUpSamp. Samp.
Ref'ment Scan Ref'ment Scan
Base Image
YcbCr To RGB Base Base TransTransformation formation
Oi
HDR Image Base NLT J Color Base Color PointNLT i TransPoint TransTrafo formation Trafo formation
Prescaling Prescaling TransTransformation formation
Prescaling Prescaling NLT NLT Point Point Trafo Trafo
Res. Ref'ment Res. Ref'ment
JPEG Decoder JPEG Decoder Residual Image Residual Image
Inv. Inv. QNT QNT Noise Noise Shaping Shaping
DCT DCT Or Or bypass bypass
Chroma Chroma UpUpsamp. samp.
Residual Residual Point Point Ri Trafo Trafo
ν
Pi
∙∙
Πi
Base Φi2nd Hi 2nd Base Point Point Trafo Trafo
Qi
∙∙ μ
2nd 2nd Residual Residual Point Point Trafo Trafo
PostPostscaling scaling Point Point Trafo Trafo
InterResidual Intermediate Residual Transmediate Point Transformation Ψ Point formation i Trafo Trafo
Fi
++
Ωi
Ti
Residual Residual Color Color TransTransformation formation
Output Output Conv. Conv.
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
Architecture of the JPEG XT Decoder ●
Architecture is constructed from five elements:
JPEG Decoder JPEG Image Decoder Base Base Image
Residual Residual Point Point Trafo Trafo
Residual Residual TransTransformation formation
JPEG Entropy decoder (defined in ISO/IEC 10918-1), plus Inverse QNT and DCT Scalar non-linearities, either as LUTs or parametrized curves, one per component
Linear transformations (3x3 matrices)
++
Vector addition
∙∙
Scalar multiplication
Order of the operations is fixed, but parts/profiles only use a subset of the operations. Unused operations are replaced by defaults (identities).
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
JPEG XT – Transport (Part 3) ● ●
● ●
Based on 10918-1 (is backwards compatible!) Additional syntax elements “boxes” are embedded into APP11 markers ignored by legacy implementations Residual codestream is also included in a box. Defined in 18477-3 (18477-2 uses a “legay syntax”)
SOP
DHT
BOX HDR
DQT
APP11
APP11
BOX Contents
APP11
SOF
…
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
Discussion of Parts 7 and 2: Encoding of HDR Data camera camera Scene
camera multiple exposures
“radiance” formats, Sample value is proportional to the physical radiance merging
Radiance Image tonemapping
JPEG XT compressor
8 bit Data
Radiance image
JPEG XT decoder
JPEG XT codestream
8-bit TMO'd image
JPEG decoder
JPEG base-Layer
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
JPEG XT - Decoder ●
Parts 2 and 7, Profile A
JPEG Decoder JPEG Image Decoder Base Base Image
FDCT FDCT Or Or IDCT IDCT
Inverse Inverse QNT QNT
Chroma Chroma UpUpSamp. Samp.
Ref'ment Scan Ref'ment Scan
Base Base TransTransformation formation
Oi
Reconstruction: R0
HDRi =μ (e )⋅(γ−1 ( LDRi )+χ)
HDR Image Base NLT J Color Base Color PointNLT i TransPoint TransTrafo formation Trafo formation
Prescaling Prescaling TransTransformation formation
Prescaling Prescaling NLT NLT Point Point Trafo Trafo
Res. Ref'ment Res. Ref'ment
JPEG Decoder JPEG Decoder Residual Image Residual Image
Base Image
YcbCr To RGB
DCT DCT Or Or bypass bypass
Inv. Inv. QNT QNT Noise Noise Shaping Shaping
Chroma Chroma UpUpsamp. samp.
Residual Residual Point Point Ri Trafo Trafo
ν
Pi
∙∙
Πi
Base Φi2nd Hi 2nd Base Point Point Trafo Trafo
Qi
∙∙ μ
2nd 2nd Residual Residual Point Point Trafo Trafo
PostPostscaling scaling Point Point Trafo Trafo
InterResidual Intermediate Residual Transmediate Point Transformation Ψ Point formation i Trafo Trafo
Fi
++
Ωi
Ti
Residual Residual Color Color TransTransformation formation
Output Output Conv. Conv.
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
JPEG XT - Decoder ●
Part 7, Profile B
JPEG Decoder JPEG Image Decoder Base Base Image
FDCT FDCT Or Or IDCT IDCT
Inverse Inverse QNT QNT
Chroma Chroma UpUpSamp. Samp.
Ref'ment Scan Ref'ment Scan
Base Image
YcbCr To RGB Base Base TransTransformation formation
Oi
Reconstruction:
HDR Image Base NLT J Color Base Color PointNLT i TransPoint TransTrafo formation Trafo formation
Base Φi2nd Hi 2nd Base Point Point Trafo Trafo
−1
γ ( LDRi ) Prescaling HDRi = Prescaling TransTransformation Ψ ( Ri )+ϵ formation =exp(log γ−1 ( LDRi )−log(Ψ (Ri )+ϵi )) Prescaling Prescaling NLT NLT Point Point Trafo Trafo
Res. Ref'ment Res. Ref'ment
JPEG Decoder JPEG Decoder Residual Image Residual Image
DCT DCT Or Or bypass bypass
Inv. Inv. QNT QNT Noise Noise Shaping Shaping
Chroma Chroma UpUpsamp. samp.
Residual Residual Point Point Ri Trafo Trafo
ν
Pi
∙∙
Πi
Qi
∙∙ μ
2nd 2nd Residual Residual Point Point Trafo Trafo
PostPostscaling scaling Point Point Trafo Trafo
InterResidual Intermediate Residual Transmediate Point Transformation Ψ Point formation i Trafo Trafo
Fi
++
Ωi
Ti
Residual Residual Color Color TransTransformation formation
Output Output Conv. Conv.
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
JPEG XT - Decoder ●
Parts 6, Part 7 Profile C, Part 8:
only integer operations used here, see also last year's talk on SPIE JPEG Decoder JPEG Image Decoder Base Base Image
FDCT FDCT Or Or IDCT IDCT
Inverse Inverse QNT QNT
Chroma Chroma UpUpSamp. Samp.
Ref'ment Scan Ref'ment Scan
Base Base TransTransformation formation
Oi
Reconstruction:
̂ ∘Φ( LDRi )+Ri +Oi ) HDRi =̂ exp( log ̂ LDRi )+ Ri +O i ) =ê xp( Φ( ≈Φ( LDRi )⋅̂ exp( Ri +Oi )
HDR Image Base NLT J Color Base Color PointNLT i TransPoint TransTrafo formation Trafo formation
Prescaling Prescaling TransTransformation formation
Prescaling Prescaling NLT NLT Point Point Trafo Trafo
Res. Ref'ment Res. Ref'ment
JPEG Decoder JPEG Decoder Residual Image Residual Image
Base Image
YcbCr To RGB
DCT DCT Or Or bypass bypass
Inv. Inv. QNT QNT Noise Noise Shaping Shaping
Chroma Chroma UpUpsamp. samp.
Residual Residual Point Point Ri Trafo Trafo
ν
Pi
∙∙
Πi
Base Φi2nd Hi 2nd Base Point Point Trafo Trafo
Qi
∙∙ μ
2nd 2nd Residual Residual Point Point Trafo Trafo
PostPostscaling scaling Point Point Trafo Trafo
InterResidual Intermediate Residual Transmediate Point Transformation Ψ Point formation i Trafo Trafo
Fi
++
Ωi
Ti
Residual Residual Color Color TransTransformation formation
Output Output Conv. Conv.
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
Evaluation of JPEG XT ●
●
●
●
Quality indices for HDR: Still under research ● Mapping into perceptually uniform space + LDR metric ● MRSE (essentially: Weber's law, PSNR on log-space) ● HDR-VDP 2.2 (Bangor University, Mantiuk) – used here, works best Dependency on the tone mapping of the base image ● Here: TMO by Photoshop + manual adjustment Parameter space is (at least) two-dimensiona ● Base plus extension layer “quality” at least Images here coming from the HDR Survey (Mark Fairchild) PU2-MSSIM over Subjective quality
HDR-VDP over Subjective quality
Subjective quality over quality index from another source, 20 images, 4 pts per Image (to be published)
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
Image Examples
WillyDesk
BloomingGorse2
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
3D-Plots HDR-VDP over base/extension layer quality for WillyDesk
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
3D-Plots HDR-VDP over base/extension layer quality for BloomingGorse2
WillyDesk
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
Classical RD-Plots ● ● ● ●
Post-processing algorithm on the 3D plots Unconstrained: Best possible quality for maximum rate LDR image may have a very low quality JPEG XR and JPEG 2000 included (single-layer)
WillyDesk
unconstrained
BloomingGorse2
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
Classical RD-Plots ● ●
Constrained plots: Base quality at least 75 or better Base quality is good, overhead is larger
WillyDesk
constrained
BloomingGorse2
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
More Resultats: (to be published)
Average over >100 images, various tonemappers (global, local)
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
Overhead (base layer ratio vs. total rate) Ratio of the base layer in the overall bitstream for a minimum base quality of 75 over the overall bitrate ●Staircases are an artifact of the method (R/D surfaces of A/B are very flat!) ●
WillyDesk
BloomingGorse2
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
Future Topics (more in the paper) JPEG Privacy
●
–
Privacy protection of images or image regions by scrambling them and encrypting the unscrambled part
JPIP
●
–
Interactive image browsing based on JPEG and JPEG XT
Encoding of editing information
●
– –
Lossless reversal of image editing steps (“contrast enhancement”, “emboss”, “rotate”) Editing steps stored as differential images relative to standardized steps as JPEG XT “residual image”
Richter, Bruylants, Schelkens, Ebrahimi
The JPEG XT Suite of Standards: Status and Future Plans
Conclusions ●
●
● ●
●
JPEG XT is a backwards and forwards compatible image format that allows encoding of HDR images Image quality in HDR regime is comparable to JPEG 2000, sometimes even better (surprise!) Lossy and lossless coding is supported Alpha channels are supported JPEG XT updates JPEG to the 21st century!
Thank you! (much more in the paper, really!)
