XBMC + fglrx + Xvba + H264 High@5.1 (Playing MKV in XBMC on AMD/ATI Radeon cards)

So recently I’ve been playing around with my XBMC-setup, trying to get it to run MKV files smoothly. For some reason the HD MKV-files didn’t play well with my AMD/ATI Radeon card.

After some research I found out how incredible lack of support the card has under linux. Fortunately they guys over at XBMC-team are doing a terrific job on the field.

So whats the problem?

Apparently both fglrx (older versions) and XBMC need to be Xvba enabled. Xvba stands for X-Video Bitstream Acceleration, and is AMD/ATIs API for hardware accelerated video playback.

More specific, the problem seems to evolve around HD H264 movies (Matroska). Here’s an example of a file I had trouble playing:

ascii@localhost:~$ mediainfo <file>

Unique ID : 240935207499337039514996497945984877723 (0xB5426EDE335C708595074FEE13FDC49B)
Complete name : <file>
Format : Matroska
Format version : Version 2
File size : 13.4 GiB
Duration : 2h 33mn
Overall bit rate : 12.5 Mbps
Movie name : <file>
Encoded date : UTC 2012-01-22 22:25:19
Writing application : DVDFab
Writing library : libebml v0.7.8 + libmatroska v0.8.1

ID : 8
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L5.1
Format settings, CABAC : Yes
Format settings, ReFrames : 2 frames
Duration : 2h 33mn
Bit rate : 11.8 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.237
Stream size : 12.7 GiB (94%)
Writing library : x264 core
Default : No
Forced : No
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

So what we’re looking for in the mediainfo is actually 3 things..

Format profile : High@L5.1
Format settings, ReFrames : 2 frames
Bit rate : 11.8 Mbps

Format profile tells us what profile/level is used for encoding the file. Fglrx previously only supported <= 4.1, while this one is High@L5.1. Also the ReFrames could be trouble, if it reached a high number. Finally Bit rate above 5-6 Mbps also ment trouble.

What changed?

Well, not much really. Most guides would tell you to use built inn fglrx in 12.10 Quantal, or upgrade to fglrx 9.0 if you use older versions.

Quick fglrx setup

sudo apt-get install fglrx
sudo aticonfig --initial -f
sudo aticonfig --sync-vsync=on
#This next one enables Level 5.1 support
sudo aticonfig --set-pcs-u32=MCIL,HWUVD_H264Level51Support,1

Optionally to disable underscan

sudo aticonfig --set-pcs-val=MCIL,DigitalHDTVDefaultUnderscan,0

This didn’t work for me!
After using and setup the whole thing with the correct settings, I still hade problems running level 5.1 HD movies. Since everyone told me not to upgrade, and that new Catalyst drivers were broken, I didn’t try that until last resort. Of cours it worked.

So heres what you do to get xvba working with “latest” AMD/ATI Catalyst

#IMPORTANT: Get signature file from old fglrx installation!
cp /etc/ati/signature ~/

wget http://www2.ati.com/drivers/beta/amd-catalyst-
unzip amd-catalyst-

#IMPORTANT: As usually we stop X before doing any changes to driver or driver-config
sudo service lightdm stop
sudo apt-get install dh-make build-essential dh-modaliases libqtgui4 execstack dkms linux-headers-`uname -r`
sudo sh amd-catalyst- –buildpkg
sudo dpkg -i –force-all fglrx*.deb
sudo aticonfig –initial -f
sudo aticonfig –sync-vsync=on
sudo cp ~/signature /etc/ati/
sudo reboot

Now install XBMC and your H264 High@L5.1 movies should run just fine at LOW CPU cost on xvba :)

Install XBMC

Choose your PPA:

Latest build
sudo add-apt-repository ppa:wsnipex/xbmc-xvba

Frodo build
sudo add-apt-repository ppa:wsnipex/xbmc-xvba-frodo

Eden build
sudo add-apt-repository ppa:wsnipex/xbmc-xvba-eden

As usually run

sudo apt-get update
sudo apt-get install xbmc xbmc-bin

Recommende settings for XBMC

System->Settings->System->Video Settings:
Vertical Sync: Always on

System ->Video->Playback:
Adjust Refreshrate to match video: On
Sync Playback to Display: On
Sync Method: Video Clock (Drop/Dupe)


Of course I can take little credit for this job, since it is pretty much a recap of this (very) long forum thread.

They’re doing an excellent job over at XBMC-team! Thanks!