tungsten-replicator: Provisioning generates different column-index than a normal insert event

Time for another post about tungsten replicator. It has worked well since my last post, we thought. Until we suddenly found out that table-columns were shifted right for some reason.

After some testing, we found this:

*A blob-column was left out by us (defined in tungsten.tables and chunks.csv) * The missing column generated a leap in the column index by provisioning-inserts

* The missing column DID NOT generate a leap in the column index by normal inserts and updates

This would trigger a column-count error when inserting during provisioning, and when a “dummy”-field was added, the same column would make all columns afterwards shift right.

The solution

What we ended up with, was a custom javascript filter which reset the column index on every insert.


// Prepare the filter and setup structures
function prepare()
// Perform the filter process; function is called for each event in the THL
function filter(event)
    // Get the array of DBMSData objects
    data = event.getData();
    // Iterate over the individual DBMSData objects
    for(i=0;i<data.size();i++) {
        // Get a single DBMSData object
        d = data.get(i);
        if (d instanceof com.continuent.tungsten.replicator.dbms.RowChangeData) {
            // Get an array of all the row changes
            rows = d.getRowChanges();
            // Iterate over row changes
            for(j=0;j<rows.size();j++) {
                // Get the single row change
                rowchange = rows.get(j);
                if (rowchange.getAction() == 'INSERT') {
                    var colSpec = rowchange.getColumnSpec();
                    var isOutOfIndex = false;
                    var lastIndex = null;
                    for (x = 0; x &lt; colSpec.size(); ++x) {
                        var cs = colSpec.get(x);
                        lastIndex = cs.getIndex();
                        //if index diverts from a sequetial numbering, correct it!
                        if (cs.getIndex() != (x+1)) {
                            isOutOfIndex = true;
                            //set correct index
                    if (isOutOfIndex) {
                        logger.info('Correcting index on '+rowchange.getTableName()+' size/lastindex: '+colSpec.size()+'/'+lastIndex);

What we didn’t try

* Removing columns from chunks.csv to see if that made an impact.

tungsten-replicator: Oracle to MySQL schema converter using ddlscan [PATCH]

The last couple of weeks we’ve been playing around with tungsten for the purpose of a full scale Oracle to MySQL replication. The newest provision feature in 2.2.1 was especially helpful – great work!

We now have a full scale replication running, and the latest “minor” obstacle we met was converting existing scheme to MySQL scheme as automatically as possible. Since this might be updated more often than first thought, we were looking for an automatic approach. Thanks to ddlscan and its features – it solved the issue for us.

The only problem with ddlscan was the inability to convert decimal precision to MySQL, hence we made som modification to the database layer of tungsten to accommodate our needs. What we essentially lacked, was the ability to convert NUMBER(14,2) to DECIMAL(14,2), and be sure that NUMBER(10, 0) could be converted to INT(11).

This is the first draft of a patch, which adds decimal precision to the Column-object, and adds the appropriate information in AbstractDatabase. I also attached a patch to the ddl-oracle-mysql Velocity-script to illustrate the use of the new feature.

See attached patch:

diff '--exclude=.svn' '--exclude=build' -ur replicator/samples/extensions/velocity/includes/ddl-oracle-mysql-table.vm sources/replicator/samples/extensions/velocity/includes/ddl-oracle-mysql-table.vm
--- replicator/samples/extensions/velocity/includes/ddl-oracle-mysql-table.vm	2014-05-06 13:55:14.000000000 +0200
+++ sources/replicator/samples/extensions/velocity/includes/ddl-oracle-mysql-table.vm	2014-05-06 15:24:26.000000000 +0200
@@ -37,7 +37,7 @@
- /* $col.getTypeDescription()($col.getLength(), ?) */##
+ /* $col.getTypeDescription()($col.getLength(), $col.getDecimalPrecision()) */##
 #elseif ( $col.getTypeDescription().startsWith("FLOAT") )
 #if( $col.getLength() > 53 )
@@ -67,4 +67,4 @@
 #if( $colReserved ) /* WARN: changed column name because of reserved word ($col.getName()) */#end##
 #if( $foreach.hasNext ),
\ No newline at end of file
diff '--exclude=.svn' '--exclude=build' -ur replicator/src/java/com/continuent/tungsten/replicator/database/AbstractDatabase.java sources/replicator/src/java/com/continuent/tungsten/replicator/database/AbstractDatabase.java
--- replicator/src/java/com/continuent/tungsten/replicator/database/AbstractDatabase.java	2014-05-06 13:55:12.000000000 +0200
+++ sources/replicator/src/java/com/continuent/tungsten/replicator/database/AbstractDatabase.java	2014-05-06 15:19:59.000000000 +0200
@@ -986,12 +986,14 @@
                 String typeDesc = rsc.getString("TYPE_NAME").toUpperCase();
                 // Issue 798. Mimicking MySQLApplier.
                 boolean isSigned = !typeDesc.contains("UNSIGNED");
+		int decimalPrec = rsc.getInt("DECIMAL_DIGITS");
                 Column column = new Column(colName, colType, colLength,
                         isNotNull, valueString);
+		column.setDecimalPrecision(decimalPrec);
                 cm.put(column.getName(), column);
diff '--exclude=.svn' '--exclude=build' -ur replicator/src/java/com/continuent/tungsten/replicator/database/Column.java sources/replicator/src/java/com/continuent/tungsten/replicator/database/Column.java
--- replicator/src/java/com/continuent/tungsten/replicator/database/Column.java	2014-05-06 13:55:12.000000000 +0200
+++ sources/replicator/src/java/com/continuent/tungsten/replicator/database/Column.java	2014-05-06 14:45:24.000000000 +0200
@@ -45,6 +45,7 @@
                                                        // NULL column
     Serializable              value;
     int                       valueInputStreamLength;
+    int decimalPrecision;
     private int               position;
     private boolean           blob;
     private String            typeDescription;
@@ -109,6 +110,16 @@
         return this.type;
+    public int getDecimalPrecision()
+    {
+	return this.decimalPrecision;
+    }
+    public void setDecimalPrecision(int precision)
+    {
+	this.decimalPrecision = precision;
+    }
     public long getLength()
         return this.length;

XBMC + fglrx + Xvba + H264 High@5.1 (Playing MKV in XBMC on AMD/ATI Radeon cards)

So recently I’ve been playing around with my XBMC-setup, trying to get it to run MKV files smoothly. For some reason the HD MKV-files didn’t play well with my AMD/ATI Radeon card.

After some research I found out how incredible lack of support the card has under linux. Fortunately they guys over at XBMC-team are doing a terrific job on the field.

So whats the problem?

Apparently both fglrx (older versions) and XBMC need to be Xvba enabled. Xvba stands for X-Video Bitstream Acceleration, and is AMD/ATIs API for hardware accelerated video playback.

More specific, the problem seems to evolve around HD H264 movies (Matroska). Here’s an example of a file I had trouble playing:

ascii@localhost:~$ mediainfo <file>

Unique ID : 240935207499337039514996497945984877723 (0xB5426EDE335C708595074FEE13FDC49B)
Complete name : <file>
Format : Matroska
Format version : Version 2
File size : 13.4 GiB
Duration : 2h 33mn
Overall bit rate : 12.5 Mbps
Movie name : <file>
Encoded date : UTC 2012-01-22 22:25:19
Writing application : DVDFab
Writing library : libebml v0.7.8 + libmatroska v0.8.1

ID : 8
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L5.1
Format settings, CABAC : Yes
Format settings, ReFrames : 2 frames
Duration : 2h 33mn
Bit rate : 11.8 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.237
Stream size : 12.7 GiB (94%)
Writing library : x264 core
Default : No
Forced : No
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

So what we’re looking for in the mediainfo is actually 3 things..

Format profile : High@L5.1
Format settings, ReFrames : 2 frames
Bit rate : 11.8 Mbps

Format profile tells us what profile/level is used for encoding the file. Fglrx previously only supported <= 4.1, while this one is High@L5.1. Also the ReFrames could be trouble, if it reached a high number. Finally Bit rate above 5-6 Mbps also ment trouble.

What changed?

Well, not much really. Most guides would tell you to use built inn fglrx in 12.10 Quantal, or upgrade to fglrx 9.0 if you use older versions.

Quick fglrx setup

sudo apt-get install fglrx
sudo aticonfig --initial -f
sudo aticonfig --sync-vsync=on
#This next one enables Level 5.1 support
sudo aticonfig --set-pcs-u32=MCIL,HWUVD_H264Level51Support,1

Optionally to disable underscan

sudo aticonfig --set-pcs-val=MCIL,DigitalHDTVDefaultUnderscan,0

This didn’t work for me!
After using and setup the whole thing with the correct settings, I still hade problems running level 5.1 HD movies. Since everyone told me not to upgrade, and that new Catalyst drivers were broken, I didn’t try that until last resort. Of cours it worked.

So heres what you do to get xvba working with “latest” AMD/ATI Catalyst

#IMPORTANT: Get signature file from old fglrx installation!
cp /etc/ati/signature ~/

wget http://www2.ati.com/drivers/beta/amd-catalyst-
unzip amd-catalyst-

#IMPORTANT: As usually we stop X before doing any changes to driver or driver-config
sudo service lightdm stop
sudo apt-get install dh-make build-essential dh-modaliases libqtgui4 execstack dkms linux-headers-`uname -r`
sudo sh amd-catalyst- –buildpkg
sudo dpkg -i –force-all fglrx*.deb
sudo aticonfig –initial -f
sudo aticonfig –sync-vsync=on
sudo cp ~/signature /etc/ati/
sudo reboot

Now install XBMC and your H264 High@L5.1 movies should run just fine at LOW CPU cost on xvba :)

Install XBMC

Choose your PPA:

Latest build
sudo add-apt-repository ppa:wsnipex/xbmc-xvba

Frodo build
sudo add-apt-repository ppa:wsnipex/xbmc-xvba-frodo

Eden build
sudo add-apt-repository ppa:wsnipex/xbmc-xvba-eden

As usually run

sudo apt-get update
sudo apt-get install xbmc xbmc-bin

Recommende settings for XBMC

System->Settings->System->Video Settings:
Vertical Sync: Always on

System ->Video->Playback:
Adjust Refreshrate to match video: On
Sync Playback to Display: On
Sync Method: Video Clock (Drop/Dupe)


Of course I can take little credit for this job, since it is pretty much a recap of this (very) long forum thread.

They’re doing an excellent job over at XBMC-team! Thanks!

gnome-shell Macbook/applesmc keyboard brightness extension

So, since this needs a bit of hacking – because of permission and such, I decided to put up a small howto on how to add an extension to gnome-shell where you can set the keyboard brightness.

First off we need to setup a bash-script who takes care of the keyboard-backlight settings.

The following script will do the trick:

BACKLIGHT=$(cat /sys/class/leds/smc::kbd_backlight/brightness)
echo $BACKLIGHT | tee -a /sys/class/leds/smc::kbd_backlight/brightness
exit 0

Put that into /usr/local/bin/applekb-backlight with your favorite editor(emacs, vim, nano).

After you’ve done this(which of course needs to be sudo`ed) you need to add execution permissions to the script.

sudo chmod +x /usr/local/bin/applekb-backlight

Now the script is ready to work – you can test it by entering:

applekb-backlight [0-255]

After this i put together a gnome-shell extension who uses this script to set the brightness, just like the screen-brightness extension.

The extension can be downloaded here.

This is just a quick workup and need modification and error-handling to work properly, but it gives a general idea on how to solve this issue.

Of course the best solution would be to be able to use the fn + F5/F6 keys!

applesmc read fail on mid-2012 Macbooks

The error is simple “applesmc: read arg fail” – this happens when trying to modprobe the applesmc module.

[   46.584969] applesmc: LSSB: read arg fail
[   46.672745] applesmc: LSSB: read arg fail
[   46.759992] applesmc: : read arg fail
[   46.847830] applesmc: : read arg fail
[   46.935599] applesmc: : read arg fail
[   47.023427] applesmc: : read arg fail
[   47.111244] applesmc: : read arg fail
[   47.199101] applesmc: : read arg fail

So I’ve been hacking around and reading forum threads when this genius over at Ubuntu forums came up with the solution – really simple.

The solution is a patch which actually only modifies one line of code – the constant telling the read from SMC to wait for feedback.

static int __wait_status(u8 val)
    int us;
    val = val & APPLESMC_STATUS_MASK;
    for (us = APPLESMC_MIN_WAIT; us &lt; APPLESMC_MAX_WAIT; us &lt;&lt;= 1) {
        if ((inb(APPLESMC_CMD_PORT) &amp; APPLESMC_STATUS_MASK) == val)
            return 0;
    return -EIO;

The constant to be changed is the APPLESMC_MIN_WAIT.

#define APPLESMC_MIN_WAIT     0x0040

is changed to

#define APPLESMC_MIN_WAIT     0x0010

This should take care of your problems!

Compiling the new module on Ubunutu

On Ubuntu systems you can get the kernel source by typing

sudo apt-get install linux-source

You can find the source in /usr/src. Uncompress it!

When you’ve applied the patch

index 0162f55..f41585e 100644
--- a/drivers/hwmon/applesmc.c
+++ b/drivers/hwmon/applesmc.c
@@ -54,7 +54,7 @@
/* wait up to 32 ms for a status change. */
-#define APPLESMC_MIN_WAIT	0x0040
+#define APPLESMC_MIN_WAIT	0x0010
#define APPLESMC_MAX_WAIT	0x8000

Then simply run

cd /usr/src/linux-source-x.x.x/linux-source-x-x-x/drivers/hwmon/
sudo make -C /lib/modules/$(uname -r)/build M=$(pwd) applesmc.ko
sudo insmod applesmc.ko
dmesg | grep applesmc

`dmesg | grep applesmc` should read something like
[ 70.969188] applesmc: key=375 fan=1 temp=22 acc=1 lux=2 kbd=1

Hope this works for you! :)

Recover a broken/partitioned/whatever harddrive(external or internal)

So, who HAS NOT experienced a harddrive crash?! The answer is likely no one(either they know it or not).

The Problem
Since I experienced this lately I thought I write a tiny howto, to easy the pain and suffering for you. What happened to me was that I got delivered an external USB harddrive(WD 500GB Essential) and got told that “I NEED MY PICTURES BACK”….apparently after some digging into why it stopped working I found out it “might have been dropped or bumped while connected”. So this means theres a pretty good chance of errors! The harddrive was NTFS and were unable to mount at all – seemed there were no chance to get onto it.

What you need:
Computer running linux(I used Ubuntu Lucid 10.04)
The faulty harddrive connected
Software: testdisk/photorec
Note: You can also use a software called “foremost”, but on my drive it did a quick recover and skipped too much data. Ended up with about 3GB of pictures.

Recover as much data(pictures) as possible!
Continue reading Recover a broken/partitioned/whatever harddrive(external or internal)

How to get system-information with dmidecode

Ever wondered what kind of RAM your system supports? Or maybe you can’t remember your motherboard brand or model? Well, luck you, linux has a perfect tool for finding this information.

First, let’s have a look at what we actually can find out:

Type Information
1 System
2 Base Board
3 Chassis
4 Processor
5 Memory Controller
6 Memory Module
7 Cache
8 Port Connector
9 System Slots
10 On Board Devices
11 OEM Strings
12 System Configuration Options
13 BIOS Language
14 Group Associations
15 System Event Log
16 Physical Memory Array
17 Memory Device
18 32-bit Memory Error
19 Memory Array Mapped Address
20 Memory Device Mapped Address
21 Built-in Pointing Device
22 Portable Battery
23 System Reset
24 Hardware Security
25 System Power Controls
26 Voltage Probe
27 Cooling Device
28 Temperature Probe
29 Electrical Current Probe
30 Out-of-band Remote Access
31 Boot Integrity Services
32 System Boot
33 64-bit Memory Error
34 Management Device
35 Management Device Component
36 Management Device Threshold Data
37 Memory Channel
38 IPMI Device
39 Power Supply

Continue reading How to get system-information with dmidecode

Apple iPad – Problems And Solutions

So, received my new iPad and thought I would post some guides on how to work around different problems I encountered.

Air Video on Ubuntu
So I found this lovely app that enabled me to live convert and stream my videos from my computer(s) to my iPad! So excited I took a closer look and found out that the server-software was only made for OSX and Windows. Luckily, the server software was made mostly in Java, which means a port to linux could easily be made.

Here is a guide on how to comile and install the required packages in Ubuntu: http://wiki.birth-online.de/know-how/hardware/apple-iphone/airvideo-server-linux
And here is a forum-post about the issue on providers website: http://www.inmethod.com/forum/posts/list/34.page

Another issue I encountered, was that the ahavi-daemon setup didn’t work – so I found a working one in the forum post. Create a file called /etc/avahi/services/airvideoserver.service – and add the following lines:

<?xml version=”1.0″ standalone=’no’?><!–*-nxml-*–>
<!DOCTYPE service-group SYSTEM “avahi-service.dtd”>
<!– $Id: airvideoserver.service 995 2010-04-12 15:06:53Z sam $ –>
<name replace-wildcards=”yes”>%h</name>

Continue reading Apple iPad – Problems And Solutions

Ubuntu on ASUS UL30VT

So, lately theres been a fair few problems with the UL30VT. This is a post to compile the general solutions to especially the hybrid graphics cards.

This guide is intended to the new Ubuntu release Lucid Lynx(10.04 LTS), but would probably work for the Karmic(9.10) release too.

Hybrid Graphics – The problem
First off, the hybrid drivers found in Windows 7 seems to be developed by Asus themselves. Another problem is that to be able to “hotswitch” between the Nvidia and Intel graphics card it need to be implemented into the Xorg core – and this I would imagine we will not see in a long time. As it stand in Windows, only Windows 7 is able to the “hotswitch”, earlier versions like Vista and XP need to same solution as Xorg – a service restart(re-login).

Hybrid Graphics – Solution
So there are several solutions depending on the outcome you want.

  • Running only Intel
  • Running only Nvidia
  • Running “hotswitch” between Intel and Nvidia

Hybrid Graphics – Solution “Running only Intel”
So – this solution pretty much work out of the box. Since the Lucid install utilize the Intel card by default using the Intel drivers, this is the easiest solution. The only thing you need to do is pretty much install Ubuntu as you would on any other computer.
Continue reading Ubuntu on ASUS UL30VT

Chrom(ium) vs Firefox – personal opinion!

So, the news about Chromium(Google Chrome for linux/mac) is out. Personal I’ve been using the development build for a month already…and theres a reason WHY!

If you’re anything like me, you pretty much have Facebook up all the time – perfect place to procrastinate from whatever you are doing. So hey, whats wrong with that? I’ll tell you whats wrong – firefox memory build up! This does apply to several version of Firefox too, I’m using the newest stable release.

Firefox and memory

What happens to firefox after having facebook up for a while, is that it’s claiming all my memory, often we’re talking about up til 40% of my 2GB memory. No this is of course after my firefox have been open for about 2 days straight!(Why would you ever need to close your browser?!). So my main thought is that the javascript engine over at mozilla labs is not doing a proper enough job freeing up memory, which sucks!

Chromium the saviour?

So, can Chromium do anything about it? Hope so, at least looking promising now. Chromium is running the V8 javascript enginge coupled with the Webkit rendering engine, which apparently go well together in a bundle;) What happens to my memory consumption when running Chromium?

I’LL TELL YOU WHAT! After running it for several days straight(without crashing or anything) it still only claim about 4% of my memory, that is 10 times as little as Firefox would claim!

Why is this? Chromium on steroids?

I still havent figured out why! The question is more; what is wrong with who? Is Firefox javascript engined really flawed, or is V8 THAT good?…or is it something else?

I’ve been planning to investigate into this at a later stage, but for now I don’t have an answer for you. As soon as Chromium develops into a full blown and stable browser I’ll do new tests with proper data and see if it still is as fast and memory efficient as it is now.

Stay tuned..