Reconfigurable video coding on multicore

The current monolithic and lengthy scheme behind the standardization and the design of new video coding standards is becoming inappropriate to satisfy the dynamism and changing needs of the video coding community. Such scheme and specification formalism does not allow the clear commonalities between the different codecs to be shown, at the level of the specification nor at the level of the implementation. Such a problem is one of the main reasons for the typically long interval elapsing between the time a new idea is validated until it is implemented in consumer products as part of a worldwide standard. The analysis of this problem originated a new standard initiative within the International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) Moving Pictures Experts Group (MPEG) committee, namely Reconfigurable Video Coding (RVC). The main idea is to develop a video coding standard that overcomes many shortcomings of the current standardization and specification process by updating and progressively incrementing a modular library of components. As the name implies, flexibility and reconfigurability are new attractive features of the RVC standard. Besides allowing for the definition of new codec algorithms, such features, as well as the dataflow-based specification formalism, open the way to define video coding standards that expressly target implementations on platforms with multiple cores. This article provides an overview of the main objectives of the new RVC standard, with an emphasis on the features that enable efficient implementation on platforms with multiple cores. A brief introduction to the methodologies that efficiently map RVC codec specifications to multicore platforms is accompanied with an example of the possible breakthroughs that are expected to occur in the design and deployment of multimedia services on multicore platforms.     

Reconfigurable media coding: An overview

This paper provides an overview of the rationale of the Reconfigurable Media Coding framework developed by MPEG standardization committee to overcome the limits of traditional ways of providing decoder specifications. Such framework is an extension of the Reconfigurable Video coding framework now encompassing also 3D Graphics coding standard. The idea of this approach is to specify decoders using an actor dataflow based representation consisting of self-contained processing units (coding tools) connected altogether and communicating by explicitly exchanging data. Such representation provides a specification for which several properties of the algorithms interesting for codec implementations are explicitly exposed and can be used for exploring different implementation objectives.     

Synthesizing Hardware from Dataflow Programs

The MPEG Reconfigurable Video Coding working group is developing a new library-based process for building the reference codecs of future MPEG standards, which is based on dataflow and uses an actor language called Cal. The paper presents a code generator producing RTL targeting FPGAs for Cal, outlines its structure, and demonstrates its performance on an MPEG-4 Simple Profile decoder. The resulting implementation is smaller and faster than a comparable RTL reference design, and the second half of the paper discusses some of the reasons for this counter-intuitive result.     

Quasi-Static Scheduling of CAL Actor Networks for Reconfigurable Video Coding

The upcoming Reconfigurable Video Coding (RVC) standard from MPEG (ISO / IEC SC29WG11) defines a library of coding tools to specify existing or new compressed video formats and decoders. The coding tool library has been written in a dataflow/actor-oriented language named CAL. Each coding tool (actor) can be represented with an extended finite state machine and the data communication between the tools are described as dataflow graphs. This paper proposes an approach to model the CAL actor network with Parameterized Synchronous Data Flow and to derive a quasi-static multiprocessor execution schedule for the system. In addition to proposing a scheduling approach for RVC, an extension to the well-known permutation flow shop scheduling problem that enables rapid run-time scheduling of RVC tasks, is introduced.     

Configurable Media Codec Framework： A Stepping Stone for Fast and Stable Codec Development

Recent advances in reconfigurable computing have led to new ways of implementing complex algorithms while maintaining reasonable throughput.Video codecs are becoming more complex in order to provide efficient compression for video with ever-increasing resolution.This problem is compounded by the fact that spectra of video decoding devices has become wider in the move from traditional TV to cable and satellite TV,IPTV,mobile TV,and Internet media.MPEG is tackling this problem with a reconfigurable video coding(RVC) framework and is standardizing a modular definition of tools and connections.MPEG’s work started with video coding and has recently extended to graphics data coding.RVC will be supported by non-MPEG standards such as the Chinese audio-video standard(AVS).This article gives a brief background to the reconfigurable codec framework.The key to this framework is reconfigurability and reducing granularity to find commonality between different standards.     

MPEG Reconfigurable Video Coding: From specification to a reconfigurable implementation

This paper demonstrates that it is possible to produce automatic, reconfigurable, and portable implementations of multimedia decoders onto platforms with the help of the MPEG Reconfigurable Video Coding (RVC) standard. MPEG RVC is a new formalism standardized by the MPEG consortium used to specify multimedia decoders. It produces visual representations of decoder reference software, with the help of graphs that connect several coding tools from MPEG standards. The approach developed in this paper draws on Dataflow Process Networks to produce a Minimal and Canonical Representation (MCR) of MPEG RVC specifications. The MCR makes it possible to form automatic and reconfigurable implementations of decoders which can match any actual platforms. The contribution is demonstrated on one case study where a generic decoder needs to process a multimedia content with the help of the RVC specification of the decoder required to process it. The overall approach is tested on two decoders from MPEG, namely MPEG-4 part 2 Simple Profile and MPEG-4 part 10 Constrained Baseline Profile. The results validate the following benefits on the MCR of decoders: compact representation, low overhead induced by its compilation, reconfiguration and multi-core abilities.     

Secure computing with the MPEG RVC framework

Recently, ISO/IEC standardized a dataflow-programming framework called Reconfigurable Video Coding (RVC) for the specification of video codecs. The RVC framework aims at providing the specification of a system at a high abstraction level so that the functionality (or behavior) of the system become independent of implementation details. The idea is to specify a system so that only intrinsic features of the algorithms are explicitly expressed, whereas implementation choices can then be made only once specific target platforms have been chosen. With this system design approach, one abstract design can be used to automatically create implementations towards multiple target platforms. In this paper, we report our investigations on applying the methodology standardized by the MPEG RVC framework to develop secure computing in the domains of cryptography and multimedia security, leading to the conclusion that the RVC framework can successfully be applied as a general-purpose framework to other fields beyond multimedia coding. This paper also highlights the challenges we faced in conducting our study, and how our study helped the RVC and the secure computing communities benefited from each other. Our investigations started with the development of a Crypto Tools Library (CTL) based on RVC, which covers a number of widely used ciphers and cryptographic hash functions such as AES, Triple DES, ARC4 and SHA-2. Performance benchmarking results on the RVC-based AES and SHA-2 implementations in both C and Java revealed that the automatically generated implementations can achieve a comparable performance to some manually written reference implementations. We also demonstrated that the RVC framework can easily produce implementations with multi-core support without any change to the RVC code. A security protocol for mutual authentication was also implemented to demonstrate how one can build heterogeneous systems easily with RVC. By combining CTL with Video Tool Library (a standard library defined by the RVC standard), a non-standard RVC-based H.264/AVC encoder and a non-standard RVC-based JPEG codec, we further demonstrated the benefits of using RVC to develop different kinds of multimedia security applications, which include joint multimedia encryption-compression schemes, digital watermarking and image steganography in JPEG compressed domain. Our study has shown that RVC can be used as a general-purpose implementation-independent development framework for diverse data-driven applications with different complexities.     

Optimized dynamic compilation of dataflow representations for multimedia applications

This paper proposes two optimization methods based on dataflow representations and dynamic compilation that enhance flexibility and performance of multimedia applications. These optimization methods are intended to be used in an adaptive decoding context, or, in other terms, where decoders have the ability to adapt their decoding processes according to a bitstream. This adaptation is made possible by coupling the decoding information to process a stream inside a coded stream. In this paper, we use dataflow representations from the upcoming MPEG Reconfigurable Media Coding (RMC) standard to supply the decoding information to adaptive decoders. The benefits claimed by MPEG RMC are a reuse of coding tools between different specifications of decoder and an execution scalability on different processing units with a single specification, which can target either hardware and/or software platforms. These benefits are not yet achievable in practice as these specifications are not used at the receiver side in MPEG RMC. We valid these benefits and propose two optimizations for the generation and the execution of dataflow models: the first optimization takes benefits of the reuse of coding tools to reduce the time to obtain—configure—enforceable decoders. The second provides an efficient, dynamic, and scalable execution according to the features of the execution platform. We show the practical impact of these two optimizations on two decoder representations compliant with the MPEG-4 part 2 Simple Profile standard and the MPEG-4 Advanced Video Coding standard. The results shows that configuration time can be reduced by 3 and the performance of decoders can be increased by 50 %.     

Methods to explore design space for MPEG RMC codec specifications

The recent MPEG Reconfigurable Media Coding (RMC) standard aims at defining media processing specifications (e.g. video codecs) in a form that abstracts from the implementation platform, but at the same time is an appropriate starting point for implementation on specific targets. To this end, the RMC framework has standardized both an asynchronous dataflow model of computation and an associated specification language. Either are providing the formalism and the theoretical foundation for multimedia specifications. Even though these specifications are abstract and platform-independent the new approach of developing implementations from such initial specifications presents obvious advantages over the approaches based on classical sequential specifications. The advantages appear particularly appealing when targeting the current and emerging homogeneous and heterogeneous manycore or multicore processing platforms. These highly parallel computing machines are gradually replacing single-core processors, particularly when the system design aims at reducing power dissipation or at increasing throughput. However, a straightforward mapping of an abstract dataflow specification onto a concurrent and heterogeneous platform does often not produce an efficient result. Before an abstract specification can be translated into an efficient implementation in software and hardware, the dataflow networks need to be partitioned and then mapped to individual processing elements. Moreover, system performance requirements need to be accounted for in the design optimization process. This paper discusses the state of the art of the combinatorial problems that need to be faced at this design space exploration step. Some recent developments and experimental results for image and video coding applications are illustrated. Both well-known and novel heuristics for problems such as mapping, scheduling and buffer minimization are investigated in the specific context of exploring the design space of dataflow program implementations.     

Automated Design Flow for Multi-Functional Dataflow-Based Platforms

The implementation of processing platforms supporting multiple applications by runtime reconfigurations on dedicated hardware modules requires the solution of different problems. These problems are notably not-trivial since both platform and application complexities increase year after year. As a consequence, the design process is both time and resource demanding. System configuration along with resources management and mapping remain one of the most challenging problem, particularly when runtime adaptation is required. In this direction, the ISO/IEC SC29WG11 committee (MPEG) has developed the so called MPEG-RVC standards ISO/IEC 23001-4 and 23002-4. This standard provides specifications of video codecs in the form of dataflow programs. In this paper, an integrated design flow to derive optimized multi-functional platforms directly from disjoined high-level specifications is presented. To the authors’ best of knowledge, such an optimization, synthesis and mapping methodology for coarse-grained reconfigurable systems design does not exist within the MPEG-RVC framework. The design flow presented in this paper leverages on an integrated set of independently designed tools, all supporting the RVC standard. Results assessment has been carried out on three different scenarios: an MPEG-RVC decoder, a standard baseline MPEG-RVC JPEG codec and a generalized reconfigurable multi-quality JPEG encoder. For all these scenarios, the proposed design flow has been targeted for a Xilinx Virtex 5 FPGA. Results show how this approach is capable of yielding a reconfigurable design that preserves the original performance of the stand alone non-reconfigurable platform providing, at the same time, considerable area savings featuring a larger set of functionalities. Moreover, platforms programmability, on the basis of the required functionality ID, is automatically handled at runtime without any designer effort.     

Early Stage Automatic Strategy for Power-Aware Signal Processing Systems Design

Complexity management, portability and long term adaptivity are common challenges in different fields of embedded systems, normally colliding with the needs of efficient resource utilization and power balance. Image/signal processing systems, though required to offer a large variety of complex functions, have also to deal with battery-life limitations. Wearable signal processing systems, for example, should provide high performance and support new generation standards without compromising their portability and their long-term usability. These constraints challenge hardware designers: early stage trade-off analysis and power management automated techniques are helpful to guarantee a reasonable time-to-market. In the field of video codec specifications, the MPEG standard known as Reconfigurable Video Coding (RVC) framework addresses functional complexity and adaptivity leveraging on the intrinsic modularity of the dataflow model of computation, but it still lacks in offering power management support. The main contribution of this work is providing an automatic early-stage power management methodology to be adopted within the MPEG-RVC context. Starting from different high-level specifications, our mapping methodology identifies directly on the high-level models disjointed homogeneous logic clock regions, where the platform resources can be enabled/disabled together without affecting the overall system performance. To extend its usability to the RVC community, we have integrated this methodology within the Multi-Dataflow Composer (MDC) tool. MDC is a tool for on-the-fly reconfigurable signal processing platforms deployment. In this paper, we extended MDC to address power-aware multi-context systems. To prove the effectiveness of our work, a coprocessor for image and video processing acceleration has been assembled. This latter has been synthesized on a 90 nm ASIC technology, where demonstrated up to 90 % of reduction in the dynamic power consumption on different dataflow-intensive applications. The coprocessor has been implemented also on FPGA, confirming, partially, the benefits of adopting the proposed methodology.     

AVS视频码流解析软件QtAVS的设计与实现

码流分析对音视频编码标准产业化和应用十分重要.针对我国组织制订的AVS视频编解码标准GB/T 20090.2,利用新一代面向对象、跨平台的图形用户界面语言Qt设计实现了AVS视频码流解析软件QtAVS,该软件能够正确解析AVS视频码流,并可在序列、帧、块3个层次上对码流元素进行可视化显示.QtAVS的所有源代码已在AV...     

Wavelet-based scalable L-infinity-oriented compression.

Among the different classes of coding techniques proposed in literature, predictive schemes have proven their outstanding performance in near-lossless compression. However, these schemes are incapable of providing embedded L(infinity)-oriented compression, or, at most, provide a very limited number of potential L(infinity) bit-stream truncation points. We propose a new multidimensional wavelet-based L(infinity)-constrained scalable coding framework that generates a fully embedded L(infinity)-oriented bit stream and that retains the coding performance and all the scalability options of state-of-the-art L2-oriented wavelet codecs. Moreover, our codec instantiation of the proposed framework clearly outperforms JPEG2000 in L(infinity) coding sense.     

Side information creation for efficient Wyner–Ziv video coding: Classifying and reviewing

Video coding technologies have played a major role in the explosion of large market digital video applications and services. In this context, the very popular MPEG-x and H-26x video coding standards adopted a predictive coding paradigm, where complex encoders exploit the data redundancy and irrelevancy to ‘control’ much simpler decoders. This codec paradigm fits well applications and services such as digital television and video storage where the decoder complexity is critical, but does not match well the requirements of emerging applications such as visual sensor networks where the encoder complexity is more critical. The Slepian–Wolf and Wyner–Ziv theorems brought the possibility to develop the so-called Wyner–Ziv video codecs, following a different coding paradigm where it is the task of the decoder, and not anymore of the encoder, to (fully or partly) exploit the video redundancy. Theoretically, Wyner–Ziv video coding does not incur in any compression performance penalty regarding the more traditional predictive coding paradigm (at least for certain conditions). In the context of Wyner–Ziv video codecs, the so-called side information, which is a decoder estimate of the original frame to code, plays a critical role in the overall compression performance. For this reason, much research effort has been invested in the past decade to develop increasingly more efficient side information creation methods. This paper has the main objective to review and evaluate the available side information methods after proposing a classification taxonomy to guide this review, allowing to achieve more solid conclusions and better identify the next relevant research challenges. After classifying the side information creation methods into four classes, notably guess, try, hint and learn, the review of the most important techniques in each class and the evaluation of some of them leads to the important conclusion that the side information creation methods provide better rate-distortion (RD) performance depending on the amount of temporal correlation in each video sequence. It became also clear that the best available Wyner–Ziv video coding solutions are almost systematically based on the learn approach. The best solutions are already able to systematically outperform the H.264/AVC Intra, and also the H.264/AVC zero-motion standard solutions for specific types of content.     

DSP implementations of sophisticated speech codecs

Hardware implementation aspects of sophisticated speech codecs are addressed. The major points discussed are approaches to implementation of the sophisticated speech codecs, requirements for DSP (digital signal-processing) implementation of the codecs, such as the type of arithmetic processing and the necessity of bit-level specifications, and codec implementation and DSP programming techniques for three specific coding algorithms: 32-kb/s ADPCM (adaptive digital pulse code modulation), 64-kb/s (7 kHz) SB (subband)-ADPCM, and 16-kb/s APC-AB (adaptive predictive coding with adaptive but allocation) codecs     

Low delay distributed video coding with refined side information

Distributed video coding (DVC) is a new video coding paradigm based upon two fundamental theoretical results: the Slepian–Wolf and Wyner–Ziv theorems. Among other benefits, this new coding paradigm may allow a flexible complexity allocation between the encoder and the decoder. Several DVC codecs have been developed over the years addressing the specific requirements of emerging applications such as wireless video surveillance and sensor networks. While state-of-the-art DVC codecs, such as the DISCOVER DVC codec, have shown promising RD performance, most DVC codecs in the literature do not consider low delay requirements which are relevant for some of the addressed applications. In this context, the main objective and novelty of this paper is to propose an efficient, low delay and fully practical DVC codec based on the Stanford DVC architecture adopting a side information iterative refinement approach. The obtained performance results show that the developed DVC solution fulfils the objectives regarding relevant benchmarks, notably due to the novel side information creation and correlation noise modeling tools integrated in a side information iterative refinement framework.