tungsten-replicator: Oracle to MySQL schema converter using ddlscan [PATCH]

The last couple of weeks we’ve been playing around with tungsten for the purpose of a full scale Oracle to MySQL replication. The newest provision feature in 2.2.1 was especially helpful – great work!

We now have a full scale replication running, and the latest “minor” obstacle we met was converting existing scheme to MySQL scheme as automatically as possible. Since this might be updated more often than first thought, we were looking for an automatic approach. Thanks to ddlscan and its features – it solved the issue for us.

The only problem with ddlscan was the inability to convert decimal precision to MySQL, hence we made som modification to the database layer of tungsten to accommodate our needs. What we essentially lacked, was the ability to convert NUMBER(14,2) to DECIMAL(14,2), and be sure that NUMBER(10, 0) could be converted to INT(11).

This is the first draft of a patch, which adds decimal precision to the Column-object, and adds the appropriate information in AbstractDatabase. I also attached a patch to the ddl-oracle-mysql Velocity-script to illustrate the use of the new feature.

See attached patch:


diff '--exclude=.svn' '--exclude=build' -ur replicator/samples/extensions/velocity/includes/ddl-oracle-mysql-table.vm sources/replicator/samples/extensions/velocity/includes/ddl-oracle-mysql-table.vm
--- replicator/samples/extensions/velocity/includes/ddl-oracle-mysql-table.vm	2014-05-06 13:55:14.000000000 +0200
+++ sources/replicator/samples/extensions/velocity/includes/ddl-oracle-mysql-table.vm	2014-05-06 15:24:26.000000000 +0200
@@ -37,7 +37,7 @@
 #else
 DECIMAL($col.getLength())##
 #end
- /* $col.getTypeDescription()($col.getLength(), ?) */##
+ /* $col.getTypeDescription()($col.getLength(), $col.getDecimalPrecision()) */##
 #elseif ( $col.getTypeDescription().startsWith("FLOAT") )
 $col.getTypeDescription()($col.getLength())##
 #if( $col.getLength() > 53 )
@@ -67,4 +67,4 @@
 #if( $colReserved ) /* WARN: changed column name because of reserved word ($col.getName()) */#end##
 #if( $foreach.hasNext ),
 #end
-#end
\ No newline at end of file
+#end
diff '--exclude=.svn' '--exclude=build' -ur replicator/src/java/com/continuent/tungsten/replicator/database/AbstractDatabase.java sources/replicator/src/java/com/continuent/tungsten/replicator/database/AbstractDatabase.java
--- replicator/src/java/com/continuent/tungsten/replicator/database/AbstractDatabase.java	2014-05-06 13:55:12.000000000 +0200
+++ sources/replicator/src/java/com/continuent/tungsten/replicator/database/AbstractDatabase.java	2014-05-06 15:19:59.000000000 +0200
@@ -986,12 +986,14 @@
                 String typeDesc = rsc.getString("TYPE_NAME").toUpperCase();
                 // Issue 798. Mimicking MySQLApplier.
                 boolean isSigned = !typeDesc.contains("UNSIGNED");
+		int decimalPrec = rsc.getInt("DECIMAL_DIGITS");
 
                 Column column = new Column(colName, colType, colLength,
                         isNotNull, valueString);
                 column.setPosition(rsc.getInt("ORDINAL_POSITION"));
                 column.setTypeDescription(typeDesc);
                 column.setSigned(isSigned);
+		column.setDecimalPrecision(decimalPrec);
                 table.AddColumn(column);
                 cm.put(column.getName(), column);
             }
diff '--exclude=.svn' '--exclude=build' -ur replicator/src/java/com/continuent/tungsten/replicator/database/Column.java sources/replicator/src/java/com/continuent/tungsten/replicator/database/Column.java
--- replicator/src/java/com/continuent/tungsten/replicator/database/Column.java	2014-05-06 13:55:12.000000000 +0200
+++ sources/replicator/src/java/com/continuent/tungsten/replicator/database/Column.java	2014-05-06 14:45:24.000000000 +0200
@@ -45,6 +45,7 @@
                                                        // NULL column
     Serializable              value;
     int                       valueInputStreamLength;
+    int decimalPrecision;
     private int               position;
     private boolean           blob;
     private String            typeDescription;
@@ -109,6 +110,16 @@
         return this.type;
     }
 
+    public int getDecimalPrecision()
+    {
+	return this.decimalPrecision;
+    }
+
+    public void setDecimalPrecision(int precision)
+    {
+	this.decimalPrecision = precision;
+    }
+
     public long getLength()
     {
         return this.length;

XBMC + fglrx + Xvba + H264 High@5.1 (Playing MKV in XBMC on AMD/ATI Radeon cards)

So recently I’ve been playing around with my XBMC-setup, trying to get it to run MKV files smoothly. For some reason the HD MKV-files didn’t play well with my AMD/ATI Radeon card.

After some research I found out how incredible lack of support the card has under linux. Fortunately they guys over at XBMC-team are doing a terrific job on the field.

So whats the problem?

Apparently both fglrx (older versions) and XBMC need to be Xvba enabled. Xvba stands for X-Video Bitstream Acceleration, and is AMD/ATIs API for hardware accelerated video playback.

More specific, the problem seems to evolve around HD H264 movies (Matroska). Here’s an example of a file I had trouble playing:


ascii@localhost:~$ mediainfo <file>

General
Unique ID : 240935207499337039514996497945984877723 (0xB5426EDE335C708595074FEE13FDC49B)
Complete name : <file>
Format : Matroska
Format version : Version 2
File size : 13.4 GiB
Duration : 2h 33mn
Overall bit rate : 12.5 Mbps
Movie name : <file>
Encoded date : UTC 2012-01-22 22:25:19
Writing application : DVDFab
Writing library : libebml v0.7.8 + libmatroska v0.8.1

Video
ID : 8
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L5.1
Format settings, CABAC : Yes
Format settings, ReFrames : 2 frames
Codec ID : V_MPEG4/ISO/AVC
Duration : 2h 33mn
Bit rate : 11.8 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.237
Stream size : 12.7 GiB (94%)
Writing library : x264 core
Default : No
Forced : No
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

So what we’re looking for in the mediainfo is actually 3 things..


Format profile : High@L5.1
Format settings, ReFrames : 2 frames
Bit rate : 11.8 Mbps

Format profile tells us what profile/level is used for encoding the file. Fglrx previously only supported <= 4.1, while this one is High@L5.1. Also the ReFrames could be trouble, if it reached a high number. Finally Bit rate above 5-6 Mbps also ment trouble.

What changed?

Well, not much really. Most guides would tell you to use built inn fglrx in 12.10 Quantal, or upgrade to fglrx 9.0 if you use older versions.

Quick fglrx setup

sudo apt-get install fglrx
sudo aticonfig --initial -f
sudo aticonfig --sync-vsync=on
#This next one enables Level 5.1 support
sudo aticonfig --set-pcs-u32=MCIL,HWUVD_H264Level51Support,1

Optionally to disable underscan

sudo aticonfig --set-pcs-val=MCIL,DigitalHDTVDefaultUnderscan,0

This didn’t work for me!
After using and setup the whole thing with the correct settings, I still hade problems running level 5.1 HD movies. Since everyone told me not to upgrade, and that new Catalyst drivers were broken, I didn’t try that until last resort. Of cours it worked.

So heres what you do to get xvba working with “latest” AMD/ATI Catalyst

#IMPORTANT: Get signature file from old fglrx installation!
cp /etc/ati/signature ~/

wget http://www2.ati.com/drivers/beta/amd-catalyst-13.15.100.1-linux-x86.x86_64.zip
unzip amd-catalyst-13.15.100.1-linux-x86.x86_64.zip

#IMPORTANT: As usually we stop X before doing any changes to driver or driver-config
sudo service lightdm stop
sudo apt-get install dh-make build-essential dh-modaliases libqtgui4 execstack dkms linux-headers-`uname -r`
sudo sh amd-catalyst-13.15.100.1-linux-x86.x86_64.zip –buildpkg
sudo dpkg -i –force-all fglrx*.deb
sudo aticonfig –initial -f
sudo aticonfig –sync-vsync=on
sudo cp ~/signature /etc/ati/
sudo reboot

Now install XBMC and your H264 High@L5.1 movies should run just fine at LOW CPU cost on xvba :)

Install XBMC

Choose your PPA:

Latest build
sudo add-apt-repository ppa:wsnipex/xbmc-xvba

Frodo build
sudo add-apt-repository ppa:wsnipex/xbmc-xvba-frodo

Eden build
sudo add-apt-repository ppa:wsnipex/xbmc-xvba-eden

As usually run

sudo apt-get update
sudo apt-get install xbmc xbmc-bin

Recommende settings for XBMC

System->Settings->System->Video Settings:
Vertical Sync: Always on

System ->Video->Playback:
XVBA: On
Adjust Refreshrate to match video: On
Sync Playback to Display: On
Sync Method: Video Clock (Drop/Dupe)

Summary

Of course I can take little credit for this job, since it is pretty much a recap of this (very) long forum thread.

They’re doing an excellent job over at XBMC-team! Thanks!

gnome-shell Macbook/applesmc keyboard brightness extension

So, since this needs a bit of hacking – because of permission and such, I decided to put up a small howto on how to add an extension to gnome-shell where you can set the keyboard brightness.

First off we need to setup a bash-script who takes care of the keyboard-backlight settings.

The following script will do the trick:

#!/bin/bash

BACKLIGHT=$(cat /sys/class/leds/smc::kbd_backlight/brightness)

BACKLIGHT=$1

echo $BACKLIGHT | tee -a /sys/class/leds/smc::kbd_backlight/brightness
exit 0

Put that into /usr/local/bin/applekb-backlight with your favorite editor(emacs, vim, nano).

After you’ve done this(which of course needs to be sudo`ed) you need to add execution permissions to the script.

sudo chmod +x /usr/local/bin/applekb-backlight

Now the script is ready to work – you can test it by entering:

applekb-backlight [0-255]

After this i put together a gnome-shell extension who uses this script to set the brightness, just like the screen-brightness extension.

The extension can be downloaded here.

NOTE!
This is just a quick workup and need modification and error-handling to work properly, but it gives a general idea on how to solve this issue.

Of course the best solution would be to be able to use the fn + F5/F6 keys!

applesmc read fail on mid-2012 Macbooks

The error is simple “applesmc: read arg fail” – this happens when trying to modprobe the applesmc module.

[   46.584969] applesmc: LSSB: read arg fail
[   46.672745] applesmc: LSSB: read arg fail
[   46.759992] applesmc: : read arg fail
[   46.847830] applesmc: : read arg fail
[   46.935599] applesmc: : read arg fail
[   47.023427] applesmc: : read arg fail
[   47.111244] applesmc: : read arg fail
[   47.199101] applesmc: : read arg fail

So I’ve been hacking around and reading forum threads when this genius over at Ubuntu forums came up with the solution – really simple.

The solution is a patch which actually only modifies one line of code – the constant telling the read from SMC to wait for feedback.

static int __wait_status(u8 val)
{
    int us;
    val = val & APPLESMC_STATUS_MASK;

    for (us = APPLESMC_MIN_WAIT; us < APPLESMC_MAX_WAIT; us <<= 1) {
        udelay(us);
        if ((inb(APPLESMC_CMD_PORT) & APPLESMC_STATUS_MASK) == val)
            return 0;
    }

    return -EIO;
}

The constant to be changed is the APPLESMC_MIN_WAIT.

#define APPLESMC_MIN_WAIT     0x0040

is changed to

#define APPLESMC_MIN_WAIT     0x0010

This should take care of your problems!

Compiling the new module on Ubunutu

On Ubuntu systems you can get the kernel source by typing

sudo apt-get install linux-source

You can find the source in /usr/src. Uncompress it!

When you’ve applied the patch

index 0162f55..f41585e 100644
--- a/drivers/hwmon/applesmc.c
+++ b/drivers/hwmon/applesmc.c
@@ -54,7 +54,7 @@
#define APPLESMC_MAX_DATA_LENGTH 32
/* wait up to 32 ms for a status change. */
-#define APPLESMC_MIN_WAIT	0x0040
+#define APPLESMC_MIN_WAIT	0x0010
#define APPLESMC_MAX_WAIT	0x8000

#define APPLESMC_STATUS_MASK	0x0f
--
1.7.11.1

Then simply run

cd /usr/src/linux-source-x.x.x/linux-source-x-x-x/drivers/hwmon/
sudo make -C /lib/modules/$(uname -r)/build M=$(pwd) applesmc.ko
sudo insmod applesmc.ko
dmesg | grep applesmc

`dmesg | grep applesmc` should read something like
[ 70.969188] applesmc: key=375 fan=1 temp=22 acc=1 lux=2 kbd=1

Hope this works for you! :)

Recover a broken/partitioned/whatever harddrive(external or internal)

So, who HAS NOT experienced a harddrive crash?! The answer is likely no one(either they know it or not).

The Problem
Since I experienced this lately I thought I write a tiny howto, to easy the pain and suffering for you. What happened to me was that I got delivered an external USB harddrive(WD 500GB Essential) and got told that “I NEED MY PICTURES BACK”….apparently after some digging into why it stopped working I found out it “might have been dropped or bumped while connected”. So this means theres a pretty good chance of errors! The harddrive was NTFS and were unable to mount at all – seemed there were no chance to get onto it.

What you need:
Computer running linux(I used Ubuntu Lucid 10.04)
The faulty harddrive connected
Software: testdisk/photorec
Note: You can also use a software called “foremost”, but on my drive it did a quick recover and skipped too much data. Ended up with about 3GB of pictures.

Target
Recover as much data(pictures) as possible!
Continue reading

How to get system-information with dmidecode

Ever wondered what kind of RAM your system supports? Or maybe you can’t remember your motherboard brand or model? Well, luck you, linux has a perfect tool for finding this information.

First, let’s have a look at what we actually can find out:

Type Information
────────────────────────────────────────
0 BIOS
1 System
2 Base Board
3 Chassis
4 Processor
5 Memory Controller
6 Memory Module
7 Cache
8 Port Connector
9 System Slots
10 On Board Devices
11 OEM Strings
12 System Configuration Options
13 BIOS Language
14 Group Associations
15 System Event Log
16 Physical Memory Array
17 Memory Device
18 32-bit Memory Error
19 Memory Array Mapped Address
20 Memory Device Mapped Address
21 Built-in Pointing Device
22 Portable Battery
23 System Reset
24 Hardware Security
25 System Power Controls
26 Voltage Probe
27 Cooling Device
28 Temperature Probe
29 Electrical Current Probe
30 Out-of-band Remote Access
31 Boot Integrity Services
32 System Boot
33 64-bit Memory Error
34 Management Device
35 Management Device Component
36 Management Device Threshold Data
37 Memory Channel
38 IPMI Device
39 Power Supply

Continue reading

Apple iPad – Problems And Solutions

So, received my new iPad and thought I would post some guides on how to work around different problems I encountered.

Air Video on Ubuntu
So I found this lovely app that enabled me to live convert and stream my videos from my computer(s) to my iPad! So excited I took a closer look and found out that the server-software was only made for OSX and Windows. Luckily, the server software was made mostly in Java, which means a port to linux could easily be made.

Here is a guide on how to comile and install the required packages in Ubuntu: http://wiki.birth-online.de/know-how/hardware/apple-iphone/airvideo-server-linux
And here is a forum-post about the issue on providers website: http://www.inmethod.com/forum/posts/list/34.page

Another issue I encountered, was that the ahavi-daemon setup didn’t work – so I found a working one in the forum post. Create a file called /etc/avahi/services/airvideoserver.service – and add the following lines:

<?xml version=”1.0″ standalone=’no’?><!–*-nxml-*–>
<!DOCTYPE service-group SYSTEM “avahi-service.dtd”>
<!– $Id: airvideoserver.service 995 2010-04-12 15:06:53Z sam $ –>
<service-group>
<name replace-wildcards=”yes”>%h</name>
<service>
<type>_airvideoserver._tcp</type>
<port>45631</port>
</service>
</service-group>

Continue reading

Microsoft kills “Courier”

As most of you read on both Gizmodo and Engadget, Microsoft has terminated the “Courier” project. To be really honest, the “Courier” and “Nadal” are pretty much the only exciting and innovative projects Microsoft had going for them, at least from a gadget-geek-perspective. So I’m not trying to hide my disappointment when they terminated the “Courier”.

http://gizmodo.com/5527442/microsoft-cancels-innovative-courier-tablet-project
http://www.engadget.com/2010/04/29/microsoft-confirms-kills-courier-in-one-fell-swoop/

So why is it, that they suddenly kill this project? Did anyone ever see or touch a prototype? Was this really all a concept-video?

There’s a lot of questions to ask regarding this sudden cancellation, but what I take personally from this is that the road from a concept to prototype to product is not a walk in the park. Another punch in the face could be the cancellation of the HP Slate tablet. If you listen to the sources who tried the Slate – it was buggy and laggy, pretty much unresponsive. How would you then do a double-screen keeping a low-cost AND have touchscreen gestures work smoothly? This isn’t possible if you need to keep the cost down. But what I really want to know the answer to is: was this a decision based on the fact that Microsoft couldnt make it fast enough vs cheap enough? Or weren’t they able to make the product in a reasonable timeframe on a reasonable platform at all?

My first thought that struck me was the comparison to Apple and the way they *never* show prototypes like that, but still manage to hype their products like no one else. I think this is were Microsoft has much to learn. Either way, maybe next time at least have a prototype before you go out and hype it – or else you will look like the loser fools you do this time – because lets face it – there were a lot of interest around this product!

Anyhow, it’s a shame that it got terminated, but maybe they’ll pick up the thread at a later stage when they(maybe) have a better and lighter framework to work on. The Windows platform just don’t cut it when it comes to lighter tablets!

Ubuntu on ASUS UL30VT

So, lately theres been a fair few problems with the UL30VT. This is a post to compile the general solutions to especially the hybrid graphics cards.

This guide is intended to the new Ubuntu release Lucid Lynx(10.04 LTS), but would probably work for the Karmic(9.10) release too.

Hybrid Graphics – The problem
First off, the hybrid drivers found in Windows 7 seems to be developed by Asus themselves. Another problem is that to be able to “hotswitch” between the Nvidia and Intel graphics card it need to be implemented into the Xorg core – and this I would imagine we will not see in a long time. As it stand in Windows, only Windows 7 is able to the “hotswitch”, earlier versions like Vista and XP need to same solution as Xorg – a service restart(re-login).

Hybrid Graphics – Solution
So there are several solutions depending on the outcome you want.

  • Running only Intel
  • Running only Nvidia
  • Running “hotswitch” between Intel and Nvidia

Hybrid Graphics – Solution “Running only Intel”
So – this solution pretty much work out of the box. Since the Lucid install utilize the Intel card by default using the Intel drivers, this is the easiest solution. The only thing you need to do is pretty much install Ubuntu as you would on any other computer.
Continue reading

OpenGL and CG Shaders – introduction

I wanted to write a little introduction on how to setup and use the nVidia CG Toolkit and start creating some shaders for first and most OpenGL, who cares about D3D anyway?!?

Getting the toolkit

Head over to nVidias CG site and fetch the toolkit from there. Remember to get latest version!

What else do you need? You need, or preferrable want to update your graphics drivers to latest version in case you’re running on some old shit that doesn’t support different OpenGL extensions.

Getting started

Well, first I’m gonna give you an address to a beginners tutorial to CG, just so you have something to look up in, in case you were to, not that I’m saying you ever are, get stuck somewhere.

nVidia CG Tutorial book

It’s a good place to start, maybe not go through all the chapters straight away, but have a look at the first chapters and you get a general idea. That is pretty much what I’m gonna go through here too.

I already asume you are familiar with how to add library to your Visual Studio, cause we do need some of those cg librarys loaded. I will also use function oriented programming to demonstrate how this works, but theres no reason why you cant access it FROM objects so don’t be scred if you’re a OOP guru.

First, lets have a look at the CG shaders we’re gonna use. This is the simplest of the simplest ones, just to get you started. Lets have a look at the vertex shader:
Continue reading