Sunday 18 January 2009

Thinking alloud

Just got back from an afternoon at the workshop. Flashheart now has a Fastacraft main foil and a rudder with an Aardvark vertical and repaired Bladerider horizontal... It all looks like it will work but I put lots of forward rake on the rudder (bit too much as the thing moved before the glue set) so I hope I can get enough lift from it. The gantry bottle screw has gone from one extreme of spending the summer at the very end of it thread trying to lose lift to the other of tightly screwed up in the hope it will generate some lift. Still the forward rake on the rudder looks mad, and should reduce rudder ventilation. It also fits with the Mach 2 trend if the computer graphics of the gantry are to be believed.

The Bladerider horizontal was easier to repair than the Aardvark one although the word on the net is that small rudders are in. The Aardvark rudder was definitely small and was great. I never adjusted my twist tiller (because it did not work and) because I always had too much lift and so it would not adjust any more. However the small rudder seamed a good thing, so it is a shame I have maybe taken a step back.

However the tiller twist adjustment is much better now. It is a Bladerider mechanism (without the bloody great chunk of stainless tube, as I cut most of this away and replaced it with a carbon/glass tube. This saved about 100g). This Bladerider mechanism sits in an Aardvark tiller with and Doug Culnane rudder stock.

Flashheart is starting to look like a jumble of acquired Moth parts which is exactly what she is. However I am moving on to the sanding and painting phase now so hopefully the end result will all look like parts of the same team.

It is hard to sit behind a computer or in a workshop and work out which developments are joking banter and which are real developments. It seams that people are experimenting and there is some strange stuff groining on. Check out Chris's post about the new North Sails (http://usfoilingmoths.blogspot.com/2009/01/arnauds-sail.html). Arnaud is hiking very far forward..? Rohan has indicated that sitting at the back is fast downwind. Small rudder horizontals are being used by a group of Aussies. Warm water is fast (How do I know which side of the beat is warmer?). There was an interesting paper that attributed speed to sharing induced drag equally between the 2 foils. Snoring is fast but cutting of the genitals of your competition is faster. The aerodynamic drag of the wand mechanism is very important to upwind speed. Small bows do not cause a nose dive when they crash down like big buoyant ones do (this I believe because I have the smallest bow in the fleet, so I know. However there is a problem with this which will become notorious. If your foils no longer work and you have to lowride home and home is downwind then you are fucked). So next week at work will be interesting as I hope the hoaxes and the real developments become clearer, or at least there is even more to chat and think about.

I however have 2 boats I want to get sorted and that is enough work for this winter (and maybe a bit too much of spring). Next summer is to be the summer of completing races without alphabet and my target from 2008 to beat is an impressive 4 (out of 13 races). I want to find a good home for Tomahawk, so that we have another sailor and I have less Moth bits. I also want to help any local Moth sailors get in to this amazing game.

Moth sailing is still in its infancy in Austria (like many other countries) and this stuff is very hard to do in isolation. So if there are moth guys out there that do not have a new widget that is guaranteed to win the $1,000,000 prize in the local Moth open, I hope they will chat openly and not subscribe to the secrecy is success vibe. In my line of work it is important to have fun, and at the moment this stuff is fun.

Saturday 10 January 2009

Holy Shit that is fast.


Niki invited Kati and me to have a go at Ice Sailing, and wow.... If anyone thinks that wind is not a viable energy source they should have a go in one of these.

These DN things and ice surfers just generate their own power like a chain reaction perpetual motion machine that just goes faster and faster..... Unbelievable...



This DN was built 20 years ago by Niki's father and is a very nice toy.



The lake was full of people on various ice sailing toys from kites to DN yachts. It really is a great playground for those that do not go through the ice...

It was a great day out apart from me losing my wallet. This however gives me a new game. I tell people I have lost my wallet then they ask me "where?" and I just stare at them for about 4 seconds until they say "sorry that was a stupid question".

Monday 5 January 2009

SAP middleware (XI/PI) tool for Java geeks.

One area that I have been working in recently is SAP middleware (XI/PI). This is basically software that pushes and pulls data between different computers. There is a whole load of GUI tools and monitoring clients for mapping data and monitoring process flow and errors. However quantity does not always mean quality and I personally find it very tedious to click through the ABAP GUI and monitoring runtime web application to get to the actual error which is often hidden, or not even visible. Understanding the user interface is often harder to do than understanding the error. You have to spend a long time defining and maintaining data structures so that the messages can be serialized for the GUI tools. It is a complex environment and maybe I am just too stupid to understand it but I deiced to do something about it.

Basically XI follows the normalized message model. You have communication channels that send and receive data using technical protocols like HTTP, FTP, SOAP, File... and then a Process runtime (a non standard BPEL) engine. This is a nice easy to understand concept and similar to JBI (Java Business Integration). It is possible to do a Java mapping program, which gets a very low level binary stream and returns a low level binary stream. This is great because I understand 0010010011111 stuff. Including libraries and developing in XI is hard, but that is OK I have a cool set of tools for programing in Java on my laptop.

So what do I want?
- A cool test driven development environment.
- Direct access to the data streams for sample data.
- Java Stack traces when and if there is an error.
- To use my laptop and tools not some random computer.
- Something like tcpmon for seeing exactly what is going on.

How shall I do this?
- I need a kind of server running on my laptop that can collect data.
- A java mapping that will post data out of the XI environment.
- An error handler that posts stack traces out of the XI environment.
- Maven, JUnit, XSTL, Eclipse... all my favorite Java development tools ;-)

XI is normally in a very secure environment and the only way to get data out is often with HTTP and a proxy (there is no way can you access the server hard disk). I want the exact 01001001 stream and not have to debug data transformations so I will use Base64 to encode and decode. I can not really include lots of jar libs so I should use low level standard Java rather than Apache's HttpClient etc... I can import a jar into XI, so I want to be able to deliver jars (like JBI tools deliver jbi files).

So I have created a Maven project that builds a jar for import into XI. In the Maven project I have my test sample data, JUnit tested mappings, my debugging webserver, my error handler, and stream posting mapping, and cut down version of the Apache Codec Base64 encoder/decoder.

Time for some code.

Parent Mapping class.

package com.snapconsult.xi.mapping.common;

import java.io.InputStream;
import java.io.OutputStream;
import java.util.HashMap;
import java.util.Map;

import com.sap.aii.mapping.api.MappingTrace;
import com.sap.aii.mapping.api.StreamTransformation;
import com.sap.aii.mapping.api.StreamTransformationConstants;
import com.sap.aii.mapping.api.StreamTransformationException;
import com.snapconsult.xi.mapping.common.HttpConnectionUsingProxy;

/**
* XI Mapping super class.
*
* @author Doug Culnane
*/
public abstract class XIMapping implements StreamTransformation {

/**
* Map containing XI runtime environment variables.
*/
private Map param = null;

/**
* Method used by XI to set map containing XI runtime environment variables.
*/
public void setParameter(Map param) {
this.param = param;
if (param == null) {
this.param = new HashMap();
}
}

/**
* Execute the mapping. Called by XI.
*/
public void execute(InputStream inputStream, OutputStream outputStream)
throws StreamTransformationException {

logStartTransform(param);

try {

doTransFormation(inputStream, outputStream);

} catch (Exception e) {
logTransformationException(param, e);
throw new StreamTransformationException(e.getMessage());
}
logEndTransform(param);
}


/**
* Method to do the real implementation.
*
* @param inputStream
* @param outputStream
* @throws Exception
*/
public abstract void doTransFormation(InputStream inputStream, OutputStream outputStream)
throws Exception;



/**
* Utility method for logging start of transformation.
*
* @param paramMap
* Map containing XI runtime environment variables.
*/
public void logStartTransform(Map paramMap) {
if (paramMap != null
&& paramMap.get(StreamTransformationConstants.MAPPING_TRACE) != null) {
MappingTrace trace = (MappingTrace) paramMap
.get(StreamTransformationConstants.MAPPING_TRACE);
trace.addInfo("Start Transformation of "
+ StreamTransformationConstants.MESSAGE_ID + ": "
+ paramMap.get(StreamTransformationConstants.MESSAGE_ID));
}
}

/**
* Utility method for logging successful end of Transformation.
*
* @param paramMap
* Map containing XI runtime environment variables.
*/
public void logEndTransform(Map paramMap) {
if (paramMap != null
&& paramMap.get(StreamTransformationConstants.MAPPING_TRACE) != null) {
MappingTrace trace = (MappingTrace) paramMap
.get(StreamTransformationConstants.MAPPING_TRACE);
trace.addInfo("End Transformation of "
+ StreamTransformationConstants.MESSAGE_ID + ": "
+ paramMap.get(StreamTransformationConstants.MESSAGE_ID));

}
}

/**
* Utility method for logging errors.
*
* @param paramMap
* Map containing XI runtime environment variables.
* @param e
* Exception thrown during transformation.
*/
public void logTransformationException(Map paramMap, Exception e) {
if (paramMap != null
&& paramMap.get(StreamTransformationConstants.MAPPING_TRACE) != null) {
MappingTrace trace = (MappingTrace) paramMap
.get(StreamTransformationConstants.MAPPING_TRACE);
trace.addWarning("Error during Transformation of "
+ StreamTransformationConstants.MESSAGE_ID + ": "
+ paramMap.get(StreamTransformationConstants.MESSAGE_ID)
+ ", " + e.getMessage());
}

try {
ConfigurationSingleton conf = ConfigurationSingleton.getInstance();

if (conf.getPostMessages()) {

HttpConnectionUsingProxy con = new HttpConnectionUsingProxy(
conf.getConfigValue(ConfigurationSingleton.CONF_PARAM_NAME_LOGGING_WEBSERVER_URL),
conf.getConfigValue(ConfigurationSingleton.CONF_PARAM_NAME_PROXY_HOST),
conf.getConfigValue(ConfigurationSingleton.CONF_PARAM_NAME_PROXY_PORT));

StringBuffer buf = new StringBuffer();
buf.append(e.getMessage() + "\r\n");
StackTraceElement[] trace = e.getStackTrace();
for (int i=0; i < trace.length; i++) {
buf.append(trace[i].toString() + "\r\n");
}
con.post(buf.toString().getBytes());
}
} catch (Exception e1) {
// so what can you do.
}
}
}


The HTTP Post class. Note if you need proxy authentication it is easy to implement just ask google.


package com.snapconsult.xi.mapping.common;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.net.Socket;
import java.net.URL;

import org.apache.commons.codec.binary.Base64;

/**
* This is a HTTP connection programmed at very low level.
*
* @author Doug Culnane
*
*/
public class HttpConnectionUsingProxy {

private static final String ENCODING = "US-ASCII";

String url;
String proxyHost;
int proxyPort;

public HttpConnectionUsingProxy(String url , String proxyHost, String proxyPort){
this.url = url;
this.proxyHost = proxyHost;

try {
this.proxyPort = new Integer(proxyPort).intValue();
} catch (Exception ex) {
this.proxyPort = 8080;
}
}

public void post(byte[] dataArray) throws Exception{

Socket socket = null;
URL server = new URL(url);

server.getPort();

if (proxyHost.equals("")) {
socket = new Socket(server.getHost(), server.getPort());
} else {
socket = new Socket(proxyHost, proxyPort);
}
socket.setKeepAlive(true);

Writer writer = new OutputStreamWriter(
socket.getOutputStream(), ENCODING);

String content = new String(Base64.encodeBase64(dataArray, true), "UTF-8");

writer.write("POST " + server.toExternalForm() + " HTTP/1.1\r\n" +
"Host: " + server.getHost() + "\r\n" +
"Cache-Control: no-cache\r\n" +
"Pragma: no-cache\r\n" +
"User-Agent: Java\r\n" +
"Accept: text/html\r\n" +
"Accept-Language: en-us,en;q=0.5\r\n" +
"Accept-Encoding: gzip,deflate\r\n" +
"Accept-Charset: utf-8\r\n" +
"Keep-Alive: 300\r\n" +
"Proxy-Connection: keep-alive\r\n" +
"Content-Type: application/x-www-form-urlencoded\r\n" +
"Content-Length: " + (content.length() + 4) + "\r\n" +
"\r\n" + content + "\r\n\r\n");
writer.flush();

BufferedReader reader = new BufferedReader(new InputStreamReader(
socket.getInputStream(), ENCODING));


while (reader.read() != -1);

writer.close();
reader.close();
socket.close();
}

}


My post stream mapping.


package mapping;

import java.io.InputStream;
import java.io.OutputStream;

import com.snapconsult.xi.mapping.common.ConfigurationSingleton;
import com.snapconsult.xi.mapping.common.HttpConnectionUsingProxy;
import com.snapconsult.xi.mapping.common.XIMapping;

/**
* XI Mapping that posts the stream to the logging webserver. The mapping then
* passes the data on to the output stream unaltered.
*
* @author Doug Culnane
*/
public class PostMessageStreamToWebserver extends XIMapping {

public void doTransFormation(InputStream inputStream,
OutputStream outputStream) throws Exception {

// Read bytes to array.
byte[] dataArray = new byte[inputStream.available()];
for (int i = 0; i < dataArray.length; i++) {
dataArray[i] = (byte) inputStream.read();
}

ConfigurationSingleton conf = ConfigurationSingleton.getInstance();

if (conf.getPostMessages()) {
HttpConnectionUsingProxy con = new HttpConnectionUsingProxy(
conf.getConfigValue(ConfigurationSingleton.CONF_PARAM_NAME_LOGGING_WEBSERVER_URL),
conf.getConfigValue(ConfigurationSingleton.CONF_PARAM_NAME_PROXY_HOST),
conf.getConfigValue(ConfigurationSingleton.CONF_PARAM_NAME_PROXY_PORT));

con.post(dataArray);
}
for (int i = 0; i < dataArray.length; i++) {
outputStream.write(dataArray[i]);
}

}
}


The server.


package com.snapconsult.xi.loggingwebserver;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.PrintStream;
import java.net.ServerSocket;
import java.net.Socket;

import org.apache.commons.codec.binary.Base64;

public class Server {

private static int requestCounter = 0;
private static int port = 8888;
private static boolean stopServer = false;

public static final String DATA_FILE_FOLDER = "target/request-data";

/**
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException {

if (args.length > 0) {
try {
port = Integer.valueOf(args[0]).intValue();
} catch (NumberFormatException nfe) {
System.out.println("Error setting port from command " +
"line argument: " + nfe.getMessage());
}
}

ServerSocket serverSocket = new ServerSocket(port);

while (!stopServer) {

requestCounter++;
Socket socket = serverSocket.accept();
InputStream is = socket.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(is));
String line = null;

while (!stopServer && (line = in.readLine()) != null) {

System.out.println(line);

if (line.equals("")){

StringBuffer base64Data = new StringBuffer();
while (!(line = in.readLine()).equals("") ) {
base64Data.append(line);
}

byte[] data = Base64.decodeBase64(base64Data.toString().getBytes());
System.out.println(new String(data));
System.out.println();

writeFileToDisk(data, requestCounter);

break;
}
}

String response = ";-)";
PrintStream out = new PrintStream(socket.getOutputStream());

out.println("HTTP/1.1 200 OK");
out.println("Server: Server");
out.println("Content-Type: text/plain");
out.println("Content-Length: " + response.length() + "\r\n");

out.println(response);
out.println();
out.println();
out.flush();
out.close();

}


}

private static void writeFileToDisk(byte[] data, int requestCounter) {

File outDir = new File(DATA_FILE_FOLDER);
if(!outDir.exists()) {
outDir.mkdirs();
}

File dataFile = null;
FileOutputStream fos = null;
try {
dataFile = new File(DATA_FILE_FOLDER + "/request-" + requestCounter + ".txt");
fos = new FileOutputStream(dataFile , false);
fos.write(data);
fos.close();

System.out.println("Written Data file: " +
dataFile.getAbsolutePath());
System.out.println();

} catch (FileNotFoundException e) {
System.out.println("FileNotFoundException: " +
dataFile.getAbsolutePath());
} catch (IOException e) {
System.out.println(e.getMessage());
if (fos != null) {
try {
fos.close();
} catch (IOException e1) {
System.out.println(e1.getMessage());
}
}
}
}

public static int getRequestCounter() {
return requestCounter;
}

public static void setStopServer(boolean stopServer) {
Server.stopServer = stopServer;
}

}


I think that is about all the important bits of code you need to understand and build this or something similar yourself. So if you are a Java hacker you should be able to get this working, however this software is to be used at your own risk, so if it kills your cat or has some other undesired effect on your life it is not my fault. Here is some screen shots to show the results.

This shows the full archive contents. This jar file is built using Maven so I have all the advantages of Maven like source code management, quality reporting, test scope, project site.... However you should be able to use ANT or Eclipse or NetBeans to build a jar.


Here is some config. Note the PostToWebserver mapping sandwiches the real mapping so we get the before and after streams posted.


Here is the server console output running on my laptop. The server saves the 101010 streams to files in the target\request-data\ folder. The console displays the data but due to conole encoding you can not be sure that was is displayed is correct so always use the binary files for test samples.


If I try and do something stupid in a mapping like:


You get a nice stack trace on the server console.


XSLT is cool so I use this a lot for mappings, however I use a Java mapping to load the XSLT file and perform the transformation. This means I can JUnit test the XSLT mapping easily and catch any errors to my debug web server. I never use graphical mappings so I do not have to waste time defining and maintaining data structures. (This may sound stupid but data structures should be defined and documented at source as far as I am concerned. Is it really the end uses job to document and define a technical implementation provided by a supplier?) I use BPEL for process flow control, and the necessary XI Configuration etc...

The result of this methodic is I can do all my programming in my favorite environment and use test driven development for mappings. I can collect sample data from XI for my tests. I immediately see errors when they happen. I know exactly what is going on in XI and can fix it. I have a nice manageable maven project with a project website documenting project details. XI is used for what it does best which is piping messages around and controlling processes. Once the project if finished I can switch off the post with a simple config parameter. If I need to debug the productive environment I can switch it on temporarily on the productive server without affecting anything. This has transformed my work with XI from finding the stupid mistake to fixing the stupid mistake. Unfortunately I make a lot of stupid mistakes so this saves a lot of time.

So I think this is cool and it works great for me. The question is should I evolve this tool from a cool tool that works for me, to a tool that works for others? There is a lot of work to make a "Hello World" bit of code into a mature high quality, flexible tool that works in many environments and in many different ways but I think this could be worth doing.

So if I am the only Java geek that likes binary but hates clicky clicky then this will stay a quirky tool for a quirky programmer. However if you found this and got it working or would like to use this in your XI projects leave a comment. If you would like to contribute to an Open source project to develop this tool then please also leave a comment. If you have any feedback including negative comments please leave them. If you are an XI Programmer with an ABAP background and think this is all wrong and Java geeks should get the hell out of your domain then you are probably right so please also comment.

If you are a Moth sailor and are still reading this rubbish what the hell are you doing? I promise to post something interesting soon as I plan to go ice yachting for the first time tomorrow.

All the best,

Doug

Friday 2 January 2009

Happy New Year.

So far my New Year has been great. Kati and I spent the new years eve on the the top of a 12 story building in the middle of Vienna. I took the opportunity to pop the question, and luckily she said yes! The whole of Vienna then lit up with fireworks in celebration of the news.





So last year I bought the Pad, launched the Lord Flashheart, and got the girl. Lord Flash would be proud. Woof Woooof.