The video codec standard, H.264, was first released in 2003 ¹ as a way to compress raw mjpeg video files. "H.264 or MPEG-4 Part 10 Advanced Video Coding (MPEG-4 AVC), is a block-oriented motion-compensation-based video compression standard that is currently one of the most commonly used formats for the recording, compression, and distribution of video content."². The H.264 has been widely adapted for the encoding and decoding of blu-ray discs, online internet streaming such as Youtube, Vimeo, Adobe Flash Player and recently has found its way into security surveillance cameras. Just like any other codec, H.264 is used to compress/encode the video for efficient storage, and needs to be decompressed/decoded by the player in order to be viewed. To compress and decompress the video requires a great deal more processing power than raw mjpeg. "Network cameras and video encoders will most likely use a profile called the baseline profile, which is intended primarily for applications with limited computing resources."³

"Video compression is about reducing and removing redundant video data so that a digital video file can be effectively sent and stored. The process involves applying an algorithm to the source video to create a compressed file that is ready for transmission or storage. To play the compressed file, an inverse algorithm is applied to produce a video that shows virtually the same content as the original source video. The time it takes to compress, send, decompress and display a file is called latency. The more advanced the compression algorithm, the higher the latency, given the same processing power."⁴ One of the main benefits is that "an H.264 encoder can reduce the size of a digital video file by more than 80% compared with the Motion JPEG format"⁵. By compressing the video, significant reductions in bandwidth and storage can be achieved.

The problem with H.264 compression, when it comes to Security Surveillance Cameras, is that it is a Predictive codec. It is defined as a block-oriented motion-compensation based video compression standard. "Motion-compensation is is an algorithmic technique used to predict a frame in a video, given the previous and/or future frames by accounting for motion of the camera and/or objects in the video."⁶, and it "describes a picture in terms of the transformation of a reference picture to the current picture. The reference picture may be previous in time or even from the future. When images can be accurately synthesised from previously transmitted/stored images"⁶. So effectively, it has now altered or tampered with the original image.

An mjpeg video file, is a motion combination of many pictures, or jpeg's. The more frames per second in the video, the more frames there will be. The jpeg images, when combined, form a motion jpeg or mjpeg video file. With mjpeg, each frame "is compressed separately as a jpeg image."⁷. H.264 also works with frames, but not all of them are actual real images. H.264 is made up of I-frames, P-frames, and B-frames.⁸

An I-frame, or intra frame, is a self-contained frame that can be independently decoded without any reference to other images. The first image in a video sequence is always an I-frame. I-frames are needed as starting points for new viewers or resynchronization points if the transmitted bit stream is damaged. I-frames can be used to implement fast-forward, rewind and other random access functions."⁹ "The drawback of I-frames is that they consume much more bits, but on the other hand, they do not generate many artifacts."⁹. Therefore the i-frame is as close as it gets to the real image and it is the only frame that can be viewed in full by itself. All the other non-I-frames, are predicted. "A P-frame, which stands for predictive inter frame, makes references to parts of earlier I and/or P frame(s) to code the frame. P-frames usually require fewer bits than I-frames, but a drawback is that they are very sensitive to transmission errors because of the complex dependency on earlier P and I reference frames."¹⁰. The B-frames are not normally used for surveillance cameras because they reference FUTURE frames to complete the image. "A B-frame, or bi-predictive inter frame, is a frame that makes references to both an earlier reference frame and a future frame."¹¹. "In the H.264 baseline profile, only I- and P-frames are used. This profile is ideal for network cameras and video encoders since low latency is achieved because B-frames are not used."¹² The various frames that make up an H.264 video file are shown below:

The most basic form of video compression that is used by most standards and also by H.264 is Difference Coding. This type of compression looks at the image, the background and the moving objects. For example a man moving toward a house that is stationary, can achieve significant compression, by only transmitting the moving objects and only referencing back to the stationary objects. This form of compression is acceptable as there is no change to the stationary objects and that would just be redundant information, the moving object is not tampered with, as can be seen in the figure below:

This sort of compression (difference coding), only reduces the video size when most of the image is stationary. In order to reduce the video size when most if the image is in motion, H.264 uses block-based motion compensation, and this is where the predictive technology is applied. "Block-based motion compensation takes into account that much of what makes up a new frame in a video sequence can be found in an earlier frame, but perhaps in a different location. This technique divides a frame into a series of macroblocks. Block by block, a new frame—for instance, a P-frame—can be composed or ‘predicted’ by looking for a matching block in a reference frame. If a match is found, the encoder simply codes the position where the matching block is to be found in the reference frame."¹⁵ Further reduction of the video size is therefore achieved by "enabling the successive prediction of smaller blocks of pixels within each macroblock in a frame. This is done by trying to find matching pixels among the earlier encoded pixels that border a new 4x4 pixel block to be intra-coded. By reusing pixel values that have already been encoded, the bit size can be drastically reduced."¹⁶

An example of H.264 predictive compression can be seen in the table below, taken from the movie Elephants Dream.

Table 1. Re-used frames are predicted and shifted to the predicted position using the motion vector.¹⁸

Type	Example Frame	Description
Original		Full original frame, as shown on screen.
Difference		Differences between the original frame and the next frame.
Motion compensated difference		Differences between the original frame and the next frame, shifted right by 2 pixels. Shifting the frame compensates for the panning of the camera, thus there is greater overlap between the two frames.

To counter the problem of large video sizes, while not compromising the video with predictions, Mobotix developed its own, patented codec, the mxpeg. The mxpeg is an open source and free codec, available to use by all that wish to do so. Below you can see the difference between an H.264 compression and an mxpeg compression:

As can be seen from the comparison, it is not possible to achieve frame-for-frame playback with H.264 compression, a critical part of the security surveillance recordings for evidence. Only Fast Forward and Rewind are possible. Even if you could pause on each individual frame, there is data missing, only the I-frames contain the full image.

In conclusion, H.264 is a great way of reducing storage requirements of video data, when plenty of processing power is available to code and encode the footage, and when it is not a concern that portions of the video have been predicted. Therefore, for Security Applications, H.264 compression should not be used. If compression is required, mxpeg is ideal. If mxpeg is not available, it is possible to use the mjpeg format, although it requires more storage, is not predictive and will not modify the evidence from the security camera.

References:

1. https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC, paragraph 9.

2. https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC, paragraph 4.

3. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 5, paragraph 3.

4. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 4, paragraph 1.

5. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 3, paragraph 2.

6. https://en.wikipedia.org/wiki/Motion_compensation, paragraph 1.

7. https://en.wikipedia.org/wiki/Motion_JPEG, paragraph 1.

8. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 5, paragraph 5.

9. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 5, paragraph 6.

10. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 5, paragraph 7.

11. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 5, paragraph 8.

12. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 6, paragraph 2.

13. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 5, figure 2.

14. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 6, figure 4.

15. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 7, paragraph 1.

16. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 7, paragraph 2.

17. AXIS WHITE PAPER, H.264 Compression Standard, New Possibilities within Video Surveillance. - wp_h264_31669_en_0803, http://www.axis.com/files/whitepaper/wp_h264_31669_en_0803_lo.pdf page 7, figure 5.

18. https://en.wikipedia.org/wiki/Motion_compensation, Table 1.

19. Mobotix AG presentation: mx_IPC_Spring_2015_Keynote20Points, slide 27.