Skip to content
Snippets Groups Projects

Fix #1431: Avoid clipping after inverse transform when RExt__HIGH_BIT_DEPTH_SUPPORT is enabled

Merged Fix #1431: Avoid clipping after inverse transform when RExt__HIGH_BIT_DEPTH_SUPPORT is enabled
Merged Adrian Browne requested to merge adybrowne/VVCSoftware_VTM:fix_ticket_1431 into master

The specification defines no clipping after the inverse transform but in practice the output of the inverse transform may be clipped to the range -(2^bitdepth - 1)..(2^bitdepth - 1) for memory reduction purposes without affecting the end result. When the bitdepth is set to 16 and ExtendedPrecision is disabled, clipping to the range -(2^15)..(2^15 - 1) occurs, which is incorrect. The clipping range is modified to the numeric limits of the Pel data type, i.e. -(2^15)..(2^15 - 1) when RExt__HIGH_BIT_DEPTH_SUPPORT is disabled (default, range is unchanged) and -(2^31)..(2^31 - 1) otherwise.

Edited by Frank Bossen

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Frank Bossen changed title from Fix ticket #1431: Aligned clipping after inverse transform with VVC specification to Fix #1431: Avoid clipping after inverse transform when RExt__HIGH_BIT_DEPTH_SUPPORT is enabled

    changed title from Fix ticket #1431: Aligned clipping after inverse transform with VVC specification to Fix #1431: Avoid clipping after inverse transform when RExt__HIGH_BIT_DEPTH_SUPPORT is enabled

  • Frank Bossen changed the description

    changed the description

  • merged

  • Frank Bossen mentioned in commit 34509dc6

    mentioned in commit 34509dc6

Please register or sign in to reply
Loading