SAP middleware (XI/PI) tool for Java geeks.

One area that I have been working in recently is SAP middleware (XI/PI). This is basically software that pushes and pulls data between different computers. There is a whole load of GUI tools and monitoring clients for mapping data and monitoring process flow and errors. However quantity does not always mean quality and I personally find it very tedious to click through the ABAP GUI and monitoring runtime web application to get to the actual error which is often hidden, or not even visible. Understanding the user interface is often harder to do than understanding the error. You have to spend a long time defining and maintaining data structures so that the messages can be serialized for the GUI tools. It is a complex environment and maybe I am just too stupid to understand it but I deiced to do something about it.

Basically XI follows the normalized message model. You have communication channels that send and receive data using technical protocols like HTTP, FTP, SOAP, File... and then a Process runtime (a non standard BPEL) engine. This is a nice easy to understand concept and similar to JBI (Java Business Integration). It is possible to do a Java mapping program, which gets a very low level binary stream and returns a low level binary stream. This is great because I understand 0010010011111 stuff. Including libraries and developing in XI is hard, but that is OK I have a cool set of tools for programing in Java on my laptop.

So what do I want?
- A cool test driven development environment.
- Direct access to the data streams for sample data.
- Java Stack traces when and if there is an error.
- To use my laptop and tools not some random computer.
- Something like tcpmon for seeing exactly what is going on.

How shall I do this?
- I need a kind of server running on my laptop that can collect data.
- A java mapping that will post data out of the XI environment.
- An error handler that posts stack traces out of the XI environment.
- Maven, JUnit, XSTL, Eclipse... all my favorite Java development tools ;-)

XI is normally in a very secure environment and the only way to get data out is often with HTTP and a proxy (there is no way can you access the server hard disk). I want the exact 01001001 stream and not have to debug data transformations so I will use Base64 to encode and decode. I can not really include lots of jar libs so I should use low level standard Java rather than Apache's HttpClient etc... I can import a jar into XI, so I want to be able to deliver jars (like JBI tools deliver jbi files).

So I have created a Maven project that builds a jar for import into XI. In the Maven project I have my test sample data, JUnit tested mappings, my debugging webserver, my error handler, and stream posting mapping, and cut down version of the Apache Codec Base64 encoder/decoder.

Time for some code.

Parent Mapping class.

package com.snapconsult.xi.mapping.common;

import java.io.InputStream;
import java.io.OutputStream;
import java.util.HashMap;
import java.util.Map;

import com.sap.aii.mapping.api.MappingTrace;
import com.sap.aii.mapping.api.StreamTransformation;
import com.sap.aii.mapping.api.StreamTransformationConstants;
import com.sap.aii.mapping.api.StreamTransformationException;
import com.snapconsult.xi.mapping.common.HttpConnectionUsingProxy;

/**
 * XI Mapping super class.
 * 
 * @author Doug Culnane
 */
public abstract class XIMapping implements StreamTransformation {

    /**
     * Map containing XI runtime environment variables.
     */
    private Map param = null;

    /**
     * Method used by XI to set map containing XI runtime environment variables.
     */
    public void setParameter(Map param) {
        this.param = param;
        if (param == null) {
            this.param = new HashMap();
        }
    }
    
    /**
     * Execute the mapping.  Called by XI.
     */
    public void execute(InputStream inputStream, OutputStream outputStream)
            throws StreamTransformationException {

        logStartTransform(param);
        
        try {
            
            doTransFormation(inputStream, outputStream);
            
        } catch (Exception e) {
            logTransformationException(param, e);            
            throw new StreamTransformationException(e.getMessage());
        }
        logEndTransform(param);
    }
    
    
    /**
     * Method to do the real implementation.
     *
     * @param inputStream
     * @param outputStream
     * @throws Exception
     */
    public abstract void doTransFormation(InputStream inputStream, OutputStream outputStream)
        throws Exception;

    
    
    /**
     * Utility method for logging start of transformation.
     * 
     * @param paramMap
     *            Map containing XI runtime environment variables.
     */
    public void logStartTransform(Map paramMap) {
        if (paramMap != null 
                && paramMap.get(StreamTransformationConstants.MAPPING_TRACE) != null) {
            MappingTrace trace = (MappingTrace) paramMap
                    .get(StreamTransformationConstants.MAPPING_TRACE);
            trace.addInfo("Start Transformation of "
                    + StreamTransformationConstants.MESSAGE_ID + ": "
                    + paramMap.get(StreamTransformationConstants.MESSAGE_ID));
        }
    }
    
    /**
     * Utility method for logging successful end of Transformation.
     * 
     * @param paramMap
     *            Map containing XI runtime environment variables.
     */
    public void logEndTransform(Map paramMap) {
        if (paramMap != null
                && paramMap.get(StreamTransformationConstants.MAPPING_TRACE) != null) {
            MappingTrace trace = (MappingTrace) paramMap
                    .get(StreamTransformationConstants.MAPPING_TRACE);
            trace.addInfo("End Transformation of "
                    + StreamTransformationConstants.MESSAGE_ID + ": "
                    + paramMap.get(StreamTransformationConstants.MESSAGE_ID));

        }
    }
    
    /**
     * Utility method for logging errors.
     * 
     * @param paramMap
     *            Map containing XI runtime environment variables.
     * @param e
     *            Exception thrown during transformation.
     */
    public void logTransformationException(Map paramMap, Exception e) {
        if (paramMap != null
                && paramMap.get(StreamTransformationConstants.MAPPING_TRACE) != null) {
            MappingTrace trace = (MappingTrace) paramMap
                    .get(StreamTransformationConstants.MAPPING_TRACE);
            trace.addWarning("Error during Transformation of "
                    + StreamTransformationConstants.MESSAGE_ID + ": "
                    + paramMap.get(StreamTransformationConstants.MESSAGE_ID)
                    + ", " + e.getMessage());
        }
        
        try {
            ConfigurationSingleton conf = ConfigurationSingleton.getInstance();
            
            if (conf.getPostMessages()) {
                
                HttpConnectionUsingProxy con = new HttpConnectionUsingProxy(
                    conf.getConfigValue(ConfigurationSingleton.CONF_PARAM_NAME_LOGGING_WEBSERVER_URL), 
                    conf.getConfigValue(ConfigurationSingleton.CONF_PARAM_NAME_PROXY_HOST), 
                    conf.getConfigValue(ConfigurationSingleton.CONF_PARAM_NAME_PROXY_PORT));
            
                StringBuffer buf = new StringBuffer();
                buf.append(e.getMessage()  + "\r\n");
                StackTraceElement[] trace = e.getStackTrace();
                for (int i=0; i < trace.length; i++) {
                    buf.append(trace[i].toString() + "\r\n");
                }
                con.post(buf.toString().getBytes());
            }
        } catch (Exception e1) {
           // so what can you do.
        }
    }
}

The HTTP Post class. Note if you need proxy authentication it is easy to implement just ask google.

package com.snapconsult.xi.mapping.common;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.net.Socket;
import java.net.URL;

import org.apache.commons.codec.binary.Base64;

/**
 * This is a HTTP connection programmed at very low level.
 *
 * @author Doug Culnane
 *
 */
public class HttpConnectionUsingProxy {
        
        private static final String ENCODING = "US-ASCII";
    
        String url;
        String proxyHost;
        int proxyPort;
        
        public HttpConnectionUsingProxy(String url , String proxyHost, String proxyPort){
            this.url = url;
            this.proxyHost = proxyHost;
            
            try {
                this.proxyPort = new Integer(proxyPort).intValue();
            } catch (Exception ex) {
                this.proxyPort = 8080;
            }
        }
        
        public void post(byte[] dataArray) throws Exception{

            Socket socket = null;
            URL server = new URL(url);
            
            server.getPort();
            
            if (proxyHost.equals("")) {
                socket = new Socket(server.getHost(), server.getPort());
            } else {
                socket = new Socket(proxyHost, proxyPort);
            }
            socket.setKeepAlive(true);

            Writer writer = new OutputStreamWriter(
                    socket.getOutputStream(), ENCODING);
            
            String content = new String(Base64.encodeBase64(dataArray, true), "UTF-8");
            
            writer.write("POST " + server.toExternalForm() + " HTTP/1.1\r\n" +
                         "Host: " + server.getHost() + "\r\n" +
                         "Cache-Control: no-cache\r\n" +
                         "Pragma: no-cache\r\n" +
                         "User-Agent: Java\r\n" +
                         "Accept: text/html\r\n" +
                         "Accept-Language: en-us,en;q=0.5\r\n" +
                         "Accept-Encoding: gzip,deflate\r\n" +
                         "Accept-Charset: utf-8\r\n" +
                         "Keep-Alive: 300\r\n" +
                         "Proxy-Connection: keep-alive\r\n" +
                         "Content-Type: application/x-www-form-urlencoded\r\n" +
                         "Content-Length: " + (content.length() + 4) + "\r\n" +
                         "\r\n" + content + "\r\n\r\n");   
            writer.flush();
            
            BufferedReader reader = new BufferedReader(new InputStreamReader(
                         socket.getInputStream(), ENCODING));
            
            
            while (reader.read() != -1);
            
            writer.close();
            reader.close();
            socket.close();
        }

    }

My post stream mapping.

package mapping;

import java.io.InputStream;
import java.io.OutputStream;

import com.snapconsult.xi.mapping.common.ConfigurationSingleton;
import com.snapconsult.xi.mapping.common.HttpConnectionUsingProxy;
import com.snapconsult.xi.mapping.common.XIMapping;

/**
 * XI Mapping that posts the stream to the logging webserver.  The mapping then 
 * passes the data on to the output stream unaltered.
 *
 * @author Doug Culnane
 */
public class PostMessageStreamToWebserver extends XIMapping {

    public void doTransFormation(InputStream inputStream,
            OutputStream outputStream) throws Exception {

        // Read bytes to array.
        byte[] dataArray = new byte[inputStream.available()];
        for (int i = 0; i < dataArray.length; i++) {
            dataArray[i] = (byte) inputStream.read();
        }

        ConfigurationSingleton conf = ConfigurationSingleton.getInstance();

        if (conf.getPostMessages()) {
            HttpConnectionUsingProxy con = new HttpConnectionUsingProxy(
                    conf.getConfigValue(ConfigurationSingleton.CONF_PARAM_NAME_LOGGING_WEBSERVER_URL),
                    conf.getConfigValue(ConfigurationSingleton.CONF_PARAM_NAME_PROXY_HOST),
                    conf.getConfigValue(ConfigurationSingleton.CONF_PARAM_NAME_PROXY_PORT));

            con.post(dataArray);
        }
        for (int i = 0; i < dataArray.length; i++) {
            outputStream.write(dataArray[i]);
        }

    }
}

The server.

package com.snapconsult.xi.loggingwebserver;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.PrintStream;
import java.net.ServerSocket;
import java.net.Socket;

import org.apache.commons.codec.binary.Base64;

public class Server {

    private static int requestCounter = 0;
    private static int port = 8888;
    private static boolean stopServer = false;
    
    public static final String DATA_FILE_FOLDER = "target/request-data";
    
    /**
     * @param args
     * @throws IOException 
     */
    public static void main(String[] args) throws IOException {
        
        if (args.length > 0) {
            try {
                port = Integer.valueOf(args[0]).intValue();
            } catch (NumberFormatException nfe) {
                System.out.println("Error setting port from command " +
                  "line argument: " + nfe.getMessage());
            }
        }
        
        ServerSocket serverSocket = new ServerSocket(port);
        
        while (!stopServer) {
            
            requestCounter++;
            Socket socket = serverSocket.accept();
            InputStream is = socket.getInputStream();
            BufferedReader in = new BufferedReader(new InputStreamReader(is));
            String line = null;
            
            while (!stopServer && (line = in.readLine()) != null) {
                
                System.out.println(line);
                
                if (line.equals("")){
                    
                    StringBuffer base64Data = new StringBuffer();
                    while (!(line = in.readLine()).equals("") ) {
                        base64Data.append(line);
                    }
                    
                    byte[] data = Base64.decodeBase64(base64Data.toString().getBytes());
                    System.out.println(new String(data));
                    System.out.println();
                    
                    writeFileToDisk(data, requestCounter);
                    
                    break;
                }
            } 
            
            String response = ";-)";
            PrintStream out = new PrintStream(socket.getOutputStream());
            
            out.println("HTTP/1.1 200 OK");
            out.println("Server: Server");
            out.println("Content-Type: text/plain");
            out.println("Content-Length: " + response.length() + "\r\n");
            
            out.println(response);
            out.println();
            out.println();
            out.flush();
            out.close();
        
        }

        
    }

    private static void writeFileToDisk(byte[] data, int requestCounter) {
        
        File outDir = new File(DATA_FILE_FOLDER);
        if(!outDir.exists()) {
            outDir.mkdirs();
        }
        
        File dataFile = null;
        FileOutputStream fos = null;
        try {
            dataFile = new File(DATA_FILE_FOLDER + "/request-" + requestCounter + ".txt");
            fos = new FileOutputStream(dataFile , false);
            fos.write(data);
            fos.close();
            
            System.out.println("Written Data file: " + 
                    dataFile.getAbsolutePath()); 
            System.out.println();
            
        } catch (FileNotFoundException e) {
            System.out.println("FileNotFoundException: " + 
                    dataFile.getAbsolutePath());
        } catch (IOException e) {
            System.out.println(e.getMessage());
            if (fos != null) {
                try {
                    fos.close();
                } catch (IOException e1) {
                    System.out.println(e1.getMessage());
                }
            }
        }
    }

    public static int getRequestCounter() {
        return requestCounter;
    }

    public static void setStopServer(boolean stopServer) {
        Server.stopServer = stopServer;
    }

}

I think that is about all the important bits of code you need to understand and build this or something similar yourself. So if you are a Java hacker you should be able to get this working, however this software is to be used at your own risk, so if it kills your cat or has some other undesired effect on your life it is not my fault. Here is some screen shots to show the results.

This shows the full archive contents. This jar file is built using Maven so I have all the advantages of Maven like source code management, quality reporting, test scope, project site.... However you should be able to use ANT or Eclipse or NetBeans to build a jar.


Here is some config. Note the PostToWebserver mapping sandwiches the real mapping so we get the before and after streams posted.


Here is the server console output running on my laptop. The server saves the 101010 streams to files in the target\request-data\ folder. The console displays the data but due to conole encoding you can not be sure that was is displayed is correct so always use the binary files for test samples.


If I try and do something stupid in a mapping like:


You get a nice stack trace on the server console.


XSLT is cool so I use this a lot for mappings, however I use a Java mapping to load the XSLT file and perform the transformation. This means I can JUnit test the XSLT mapping easily and catch any errors to my debug web server. I never use graphical mappings so I do not have to waste time defining and maintaining data structures. (This may sound stupid but data structures should be defined and documented at source as far as I am concerned. Is it really the end uses job to document and define a technical implementation provided by a supplier?) I use BPEL for process flow control, and the necessary XI Configuration etc...

The result of this methodic is I can do all my programming in my favorite environment and use test driven development for mappings. I can collect sample data from XI for my tests. I immediately see errors when they happen. I know exactly what is going on in XI and can fix it. I have a nice manageable maven project with a project website documenting project details. XI is used for what it does best which is piping messages around and controlling processes. Once the project if finished I can switch off the post with a simple config parameter. If I need to debug the productive environment I can switch it on temporarily on the productive server without affecting anything. This has transformed my work with XI from finding the stupid mistake to fixing the stupid mistake. Unfortunately I make a lot of stupid mistakes so this saves a lot of time.

So I think this is cool and it works great for me. The question is should I evolve this tool from a cool tool that works for me, to a tool that works for others? There is a lot of work to make a "Hello World" bit of code into a mature high quality, flexible tool that works in many environments and in many different ways but I think this could be worth doing.

So if I am the only Java geek that likes binary but hates clicky clicky then this will stay a quirky tool for a quirky programmer. However if you found this and got it working or would like to use this in your XI projects leave a comment. If you would like to contribute to an Open source project to develop this tool then please also leave a comment. If you have any feedback including negative comments please leave them. If you are an XI Programmer with an ABAP background and think this is all wrong and Java geeks should get the hell out of your domain then you are probably right so please also comment.

If you are a Moth sailor and are still reading this rubbish what the hell are you doing? I promise to post something interesting soon as I plan to go ice yachting for the first time tomorrow.

All the best,

Doug