tungsten-replicator: Provisioning generates different column-index than a normal insert event

Time for another post about tungsten replicator. It has worked well since my last post, we thought. Until we suddenly found out that table-columns were shifted right for some reason.

After some testing, we found this:

*A blob-column was left out by us (defined in tungsten.tables and chunks.csv) * The missing column generated a leap in the column index by provisioning-inserts

* The missing column DID NOT generate a leap in the column index by normal inserts and updates

This would trigger a column-count error when inserting during provisioning, and when a “dummy”-field was added, the same column would make all columns afterwards shift right.

The solution

What we ended up with, was a custom javascript filter which reset the column index on every insert.


// Prepare the filter and setup structures
function prepare()
// Perform the filter process; function is called for each event in the THL
function filter(event)
    // Get the array of DBMSData objects
    data = event.getData();
    // Iterate over the individual DBMSData objects
    for(i=0;i<data.size();i++) {
        // Get a single DBMSData object
        d = data.get(i);
        if (d instanceof com.continuent.tungsten.replicator.dbms.RowChangeData) {
            // Get an array of all the row changes
            rows = d.getRowChanges();
            // Iterate over row changes
            for(j=0;j<rows.size();j++) {
                // Get the single row change
                rowchange = rows.get(j);
                if (rowchange.getAction() == 'INSERT') {
                    var colSpec = rowchange.getColumnSpec();
                    var isOutOfIndex = false;
                    var lastIndex = null;
                    for (x = 0; x &lt; colSpec.size(); ++x) {
                        var cs = colSpec.get(x);
                        lastIndex = cs.getIndex();
                        //if index diverts from a sequetial numbering, correct it!
                        if (cs.getIndex() != (x+1)) {
                            isOutOfIndex = true;
                            //set correct index
                    if (isOutOfIndex) {
                        logger.info('Correcting index on '+rowchange.getTableName()+' size/lastindex: '+colSpec.size()+'/'+lastIndex);

What we didn’t try

* Removing columns from chunks.csv to see if that made an impact.

tungsten-replicator: Oracle to MySQL schema converter using ddlscan [PATCH]

The last couple of weeks we’ve been playing around with tungsten for the purpose of a full scale Oracle to MySQL replication. The newest provision feature in 2.2.1 was especially helpful – great work!

We now have a full scale replication running, and the latest “minor” obstacle we met was converting existing scheme to MySQL scheme as automatically as possible. Since this might be updated more often than first thought, we were looking for an automatic approach. Thanks to ddlscan and its features – it solved the issue for us.

The only problem with ddlscan was the inability to convert decimal precision to MySQL, hence we made som modification to the database layer of tungsten to accommodate our needs. What we essentially lacked, was the ability to convert NUMBER(14,2) to DECIMAL(14,2), and be sure that NUMBER(10, 0) could be converted to INT(11).

This is the first draft of a patch, which adds decimal precision to the Column-object, and adds the appropriate information in AbstractDatabase. I also attached a patch to the ddl-oracle-mysql Velocity-script to illustrate the use of the new feature.

See attached patch:

diff '--exclude=.svn' '--exclude=build' -ur replicator/samples/extensions/velocity/includes/ddl-oracle-mysql-table.vm sources/replicator/samples/extensions/velocity/includes/ddl-oracle-mysql-table.vm
--- replicator/samples/extensions/velocity/includes/ddl-oracle-mysql-table.vm	2014-05-06 13:55:14.000000000 +0200
+++ sources/replicator/samples/extensions/velocity/includes/ddl-oracle-mysql-table.vm	2014-05-06 15:24:26.000000000 +0200
@@ -37,7 +37,7 @@
- /* $col.getTypeDescription()($col.getLength(), ?) */##
+ /* $col.getTypeDescription()($col.getLength(), $col.getDecimalPrecision()) */##
 #elseif ( $col.getTypeDescription().startsWith("FLOAT") )
 #if( $col.getLength() > 53 )
@@ -67,4 +67,4 @@
 #if( $colReserved ) /* WARN: changed column name because of reserved word ($col.getName()) */#end##
 #if( $foreach.hasNext ),
\ No newline at end of file
diff '--exclude=.svn' '--exclude=build' -ur replicator/src/java/com/continuent/tungsten/replicator/database/AbstractDatabase.java sources/replicator/src/java/com/continuent/tungsten/replicator/database/AbstractDatabase.java
--- replicator/src/java/com/continuent/tungsten/replicator/database/AbstractDatabase.java	2014-05-06 13:55:12.000000000 +0200
+++ sources/replicator/src/java/com/continuent/tungsten/replicator/database/AbstractDatabase.java	2014-05-06 15:19:59.000000000 +0200
@@ -986,12 +986,14 @@
                 String typeDesc = rsc.getString("TYPE_NAME").toUpperCase();
                 // Issue 798. Mimicking MySQLApplier.
                 boolean isSigned = !typeDesc.contains("UNSIGNED");
+		int decimalPrec = rsc.getInt("DECIMAL_DIGITS");
                 Column column = new Column(colName, colType, colLength,
                         isNotNull, valueString);
+		column.setDecimalPrecision(decimalPrec);
                 cm.put(column.getName(), column);
diff '--exclude=.svn' '--exclude=build' -ur replicator/src/java/com/continuent/tungsten/replicator/database/Column.java sources/replicator/src/java/com/continuent/tungsten/replicator/database/Column.java
--- replicator/src/java/com/continuent/tungsten/replicator/database/Column.java	2014-05-06 13:55:12.000000000 +0200
+++ sources/replicator/src/java/com/continuent/tungsten/replicator/database/Column.java	2014-05-06 14:45:24.000000000 +0200
@@ -45,6 +45,7 @@
                                                        // NULL column
     Serializable              value;
     int                       valueInputStreamLength;
+    int decimalPrecision;
     private int               position;
     private boolean           blob;
     private String            typeDescription;
@@ -109,6 +110,16 @@
         return this.type;
+    public int getDecimalPrecision()
+    {
+	return this.decimalPrecision;
+    }
+    public void setDecimalPrecision(int precision)
+    {
+	this.decimalPrecision = precision;
+    }
     public long getLength()
         return this.length;

OpenGL and CG Shaders – introduction

I wanted to write a little introduction on how to setup and use the nVidia CG Toolkit and start creating some shaders for first and most OpenGL, who cares about D3D anyway?!?

Getting the toolkit

Head over to nVidias CG site and fetch the toolkit from there. Remember to get latest version!

What else do you need? You need, or preferrable want to update your graphics drivers to latest version in case you’re running on some old shit that doesn’t support different OpenGL extensions.

Getting started

Well, first I’m gonna give you an address to a beginners tutorial to CG, just so you have something to look up in, in case you were to, not that I’m saying you ever are, get stuck somewhere.

nVidia CG Tutorial book

It’s a good place to start, maybe not go through all the chapters straight away, but have a look at the first chapters and you get a general idea. That is pretty much what I’m gonna go through here too.

I already asume you are familiar with how to add library to your Visual Studio, cause we do need some of those cg librarys loaded. I will also use function oriented programming to demonstrate how this works, but theres no reason why you cant access it FROM objects so don’t be scred if you’re a OOP guru.

First, lets have a look at the CG shaders we’re gonna use. This is the simplest of the simplest ones, just to get you started. Lets have a look at the vertex shader:
Continue reading OpenGL and CG Shaders – introduction

Zend PHP Certifications

As I hope you all know Zend is a big contributer within the PHP community. They provide debugging, server enhancement and framework to support your PHP-development. Since it has stated such a huge reputation within the PHP-circle, I figured I’ll go ahead and take a PHP5 certification. Also I don’t have any education underlying my 4 years of PHP-experience, so I thought this might be a good way to get some papers on what I already know.

So I went ahead and bought a Zend PHP5 Certification bundle package, which includes free test-exams, a book in pdf and paperback and the examination itself. The package came to about $165 USD.

I went ahead and fired up one of the test-exams too see if I actually were something of a PHP-developer at all, and luckily I passed on my first attempt. Although I did miss a fair questions, I’m not too worried since it was all related to the Soap-extension(I’ve used the nusoap library for my soap purposes, but might actually change back soon).

So there it goes, my exam is book for the 28th of August(Friday so I can hopefully go out celebrate afterwards, or drown my sorrows!). I was actually surprised of how much the certification exam coverd. Everything from design patterns, databases, security to OOP – sweet!

Have a look over at Zend Certification if you’re interested in using their services. It’s highly recommended to take a look at the framework they provide. It’s an easy and fast way for creating web-sites and services.

OpenGL without the glaux library on Visual Studio(Replacement code)

I’ve been doing a lot of 3D programming lately, mostly due to assignments at school…yey! I started googling a fair bit for the glaux library to work under Visual Studio, but found soon enough out that the library was obsolete at this stage. So I thought I would post some replacement code for the many tutorials still relaying on the glaux library.

Thanks to the boys over at NeHe Productions – give them a visit – awesome tutorials regardgin OpenGL and 3D programming. You can even download all the tutorials in PDF! Kudos boys, spent many hours on your site.

Back to the point, go fetch the library from their site or download here:

GLaux replacement code

At the moment I’m having a crack at bump mapping, fun when you realize you’ve done the wrong aproach for half a day. Gotta love CG shaders…