Mavenizing my project

I started working on a Google App Engine project the usual way: using the Eclipse plugin. However, unfortunately the update to the latest Eclipse plugin broke it and I haven’t found a fix yet. So I decided to try the maven-gae-plugin, once again; with Maven everything runs builds better anyway, right? So far I had only made some minor tests with the plugin.

Setup

For mavenizing the project I went the safe route: I created a new project with the maven archetype plugin:


mvn archetype:create\
 -DarchetypeGroupId=net.kindleit\
 -DarchetypeArtifactId=gae-archetype-gwt\
 -DarchetypeVersion=0.5.0\
 -DgroupId=your.groupId\
 -DartifactId=your-artifactId\
 -DremoteRepositories=http://maven-gae-plugin.googlecode.com/svn/repository

The project itself was only temporary, I was interested in the pom.xml file. I updated that file accordingly (e.g. the plugin version of the file was not the latest), removed GWT sections as I don’t use GWT in my project and changed some minor settings and added all required dependencies. Finally I copied the pom.xml into my project directory.

Subversion trouble

Then I made a lot of modifications within Eclipse – moved sources to standard Maven locations, removed JAR files etc. and was about to commit everything – but unfortunately Subversion detected a bunch of tree conflicts. Bummer. While resolving these conflicts seem to be hard, I decided to check out the project into another directory and start changing the structure from scratch – this time not using Eclipse but Subversion command line tools. That worked perfectly.

Running

With running mvn gae:run I started the development server. Startup was really smooth. However, by default the server is started on localhost only, but in my project I need a specific IP address as I need to access the server from the iPhone, too. Therefore I had to set the gae.address property. Of course it can be defined in pom.xml, but then it is the same for all development machines. I don’t want that. Therefore it must be defined in the users’s ~/.m2/settings.xml:


  <profiles>
    <profile>
      <id>gae</id>
      <properties>
        <gae.address>192.168.178.24</gae.address>
      </properties>
    </profile>
  </profiles>
  <activeProfiles>
    <activeProfile>gae</activeProfile>
  </activeProfiles>

Now the server runs on the right IP address.

Debugging

Now with using Maven I did not want to get rid of the ability to debug my application. How can this be done? Easy. First of all, the plugin provides the mvn gae:debug goal. When running this goal, the development server starts in debug mode.

In Eclipse a new run debug configuration has to be created.

  1. In the Console run mvn gae:debug; Maven will compile, execute tests and start the development server in debug mode.
  2. Wait until the server waits for the remote debugger; you will see the following output: “Listening for transport dt_socket at address: 8000″. The address is in this case the port, which needs to be set in the Eclipse debug configurations.
  3. In Eclipse go to Run > Debug Configurations …
  4. Create a new “Remote Java Application” configuration
  5. Give it a good name, choose Connection Type “Standard (Socket Attach)” and set the Connection Properties (in my case Host: 192.168.178.24, Port: 8000)
  6. Click on “Debug”. Now you see that the development server in the console continues.

Reloading webpages

Now everything works as good as when using the Eclipse plugin, right? Not quite. There is one thing that doesn’t work: dynamic reload of webpages, such as JSP or CSS files. Jetty allows dynamic reload, but when using Maven, Jetty does not use the src/main/webapp folder as working directory; instead it uses its own directory somewhere in target/…

To avoid long edit / build / deploy / run cycles, the recommended way at the moment is to run mvn gae:run in one console window and mvn cli:execute in another window. The command line interface allows you to quickly execute Maven goals. Run compile war to update the webpages in the development server’s working directory.

While this is not quite as simple as with Eclipse, it is a workaround that speeds up the development process significantly.

jQuery dynamic height of an element

This article describes how to set the height of an element dynamically based on the children of the element.

Normally, you would use CSS to set the height of an element, and you would expect that the element will automatically resize based on the content of the element. However, if the element’s height should automatically adapt to the height of one or many elements outside of this element, the dynaheight jQuery plugin might be handy.

Here is simple jQuery code to achieve that easily:


$.fn.dynaHeight = function(elements) {
  $(this).each(function() {
	var top = $(this).offset().top;
	var maxBottom = 0;
	$(elements).each(function() {
	  var pos = $(this).offset();
	  var height = $(this).outerHeight();
	  var bottom = pos.top + height;
	  if (bottom > maxBottom) {
	    maxBottom = bottom;
	  }
	});
    $(this).css({"height": (maxBottom - top) + "px"});
  });
  return this;
};

Example:

$(element).dynaHeight(“#item”);

As argument you pass the selector for the element that are used for determining the height.

What is it doing?

This piece of code first captures the top coordinate and then iterates through all given elements to find the max bottom coordinate. Finally it changes the CSS attribute “height”.

Download dynaheight.js.zip

Cache strategies for Google App Engine

Google App Engine (GAE) provides distributed in-memory cache, called Memcache. Due to quite rigid quotas, it might be necessary to extensively use the cache.

In my example, a web based photo gallery, a lot of image scaling is performed. For example, when an album is loaded, all images are shown as thumbnails. These thumbnails are generated with the Images service and more sooner than later an OverQuotaException is thrown. Although a few minutes after an OverQuotaException image transformation can be resumed, it is still annoying for the user. My application catches the exception and shows a “sorry, over quota” default image.

In order to reduce the number of image transformations, all transformed data is put into the cache. For this I am using a very simple (custom) API consisting of a CacheProducer, which creates the data and a CacheService, which checks if the data is already in cache or needs to be created.


public interface CacheProducer<T> {
    T produce();
}
public interface CacheService {
    <T> T cache(CacheProducer<T> producer, Serializable... params);
    void invalidate();
}

The Serializable... params parameter represents the components of the cache key.

Now let’s have a look at how to use this API. First we look at the PhotoDataService (which is used to get the actual image data from the datastore):


public class PhotoDataService {
...
    public byte[] getPhotoData(final Long photoId, final ImageFormat format, final ContentType type) {
        return cacheService.cache(new CacheProducer<byte[]>() {
            public byte[] produce() {
                PhotoData photoData = photoDataRepository.find(photoId);
                byte[] input = photoData.getDataAsArray();
                return imageService.process(input, format, type);
            }
        }, photoId, format, type);
    }
...
}

For this example I removed all code that is not related to caching. What you don’t see here: the CacheService first looks into the cache whether the data has been cached already. If not, the produce() method is called to produce the data, the CacheService puts it into the cache and returns the data. With this approach, I don’t need to care about the actual cache mechanisms, and I reduce clutter in the actual business logic. The cache API is reusable and can be used for any other data and key parameters.

However, the first run still produces a lot of requests to the Images service and here OverQuotaExceptions are still thrown. Next I’ll explain how to use a background task to prepopulate the cache in order to reduce the exceptions even further.

First run Maven GAE Plugin

Until now I used the Eclipse plugin to develop and test Google App Engine applications. Today I tried the Maven GAE Plugin, in order to become more flexible and independent from Eclipse. My first observations were:

  • As I started with the Maven GAE Archetype, the package declaration of the generated code did not match the actual file location. That needed to get corrected first.
  • I changed the compiler settings from 1.6 (default settings of the archetype) to 1.5 and had to remove some @Override annotations.
  • The gae.home property is undefined (which makes sense), but I had to define it to run the development server.

Overall, I’m quite pleased with the first run. Although Maven adds a lot of overhead, and the actual build process took a bit too long initially to download all dependencies, the overall impression is very good.

Multiple return values as class?

Recently I struggled again with the issue, that Java does not support multiple return values of a function. Usually, this is not a problem, but sometimes it is just not very elegant.

In this case, I was writing a method that computes the bounds of a crop rectangle (in percent of the original image size). Please note, that I was not working in the java.awt.* context, thus I did not want to use Rectangle or any other AWT class to store the return value.

The most simple solution is to use an array.

float[] bounds = computeCrop(...)

While this works in this example, it becomes crude when the return values are not all of the same type. Additionally, you have to guess which index represents which coordinate (top? left? bottom? right? width? …).

An alternative is to use a dedicated value class, such as Rectangle:

public class Rectangle {
    public float x;
    public float y;
    ...
}

Rectangle bounds = computeCrop(...);

However while implementing I thought that this approach could be optimized further. Why not let the class’ constructor do the job of computing the values? This has the advantage, that the member fields can be final, and the code for computing the crop bounds is in one place. It would look like this:

public class Crop {
    public final float x;
    public final float y;
    public final float width;
    public final float height;

    public Crop(float w1, float h1, float w2, float h2) {
        float r1 = w1 / h1;
        float r2 = w2 / h2;
        if (r1 < r2) {
            width = 1;
            height = r1 / r2;
        }
        else {
            width = r2 / r1;
            height = 1;
        }
        x = (1f - width) / 2;
        y = (1f - height) / 2;
    }
}

Apart from coupling the logic tightly with the data (logic = constructor, data = member fields), is there any other drawback for using this approach?

First Google App Engine application

As a coding exercise, I started writing a small application for the Google App Engine environment. It is a very simple web photo gallery, managing photos in albums. The main purpose of the application was to get my hands on some new frameworks, and also learn about deployment on the Google App Engine Environment.

You can look at the application here: http://photos.moritzpetersen.de and browse the source code here.

The application’s architecture is a simple 3-tier architecture:

  • The repository tier is responsible for persistence. I chose JDO as ORM technology.
  • The service tier is responsible for handling business logic. I tried to encapsulate as much logic as possible here. However, there are some exceptions, but I wouldn’t consider it business logic. For some features I implemented JSP EL functions (e.g. showing the right copyright date at the footer of each page).
  • The web tier is manages the web requests and interaction with the forms. As web framework I have chosen Stripes, which was a good choice. From the Stripes framework, I used templating, validation and error handling and file upload.

All layers are stitched together using Spring 3.0. Here I used mostly annotation based configuration of the container. I also used the JdoTemplate to easily implement the JDO repositories. Using Spring was a lot of fun, as it allowed me to implement some parts of the application very generic first: the service tier was completely generic for the first CRUD use cases, and I could easily extend the generic services later. Spring simply stitched together the right classes (emmitting a lot of warnings as I use(d) autowiring by type).

For security I used the GAE specific application security configured in the deployment descriptor. That caused the first problem when deploying the application to the App Engine: I had to use my Gmail user, not my Google Apps user for administration of the application – and I’m still not sure what the actual problem was (but maybe I should just RTFM).

The next problem was also caused by my laziness: I didn’t create any index definitions – and promptly the application threw a lot of errors because of missing indexes. Again: I should better RTFM.

But after a while, GAE automatically generated indexes and the application ran without throwing exceptions. But I immediately noticed – confirmed by a quick look at the administration dashboard: the application is too slow. It consumes a lot of CPU time for scaling images (e.g. creating thumbnails). As a quick next step I implemented a cache layer. Architecturally, the cache layer itself is a service used by the PhotoService. I am not quite confident about this implementation, but it does its job well. My original idea was to implement the cache layer as a layer between repository tier and service tier, or implement it directly into the repository tier. But during development I figured out that the cache is quite closely related to the business logic (scaling of images) and not to the persistence.

Finally I am quite satisfied with web application development on the GAE platform. The development environment is very easy to use and deployment is almost trivial. It is very easy to bring an application in production. On the other hand, my initial impression of the performance is quite mixed; loading pages takes very long and measuring the performance indicates that the GAE platform itself is not very responsive.

Handling multipart form data with Stripes on GAE

Stripes is a simple and elegant web framework that aims at simplifying Java web development. File uploads are handled elegantly and simply for the developer. A MultipartWrapper wraps the request and provides easy access to the uploaded files. Stripes provides two different implementations, but both share the same problem: they write the file to a temporary directory on the server. This is not supported by the Google App Engine. Therefore, a better solution is necessary.

Luckily, the default CommonsMultipartWrapper implementation is almost right. The biggest issue is hidden in this line:

List<FileItem> items = upload.parseRequest(request);

Replacing this with upload.getItemIterator(request) changes the game completely. getItemIterator() contains FileItemStream objects, objects, which refer to the HTTP stream directly without saving data to disk. The stream needs to be buffered in memory until it gets processed by the application.

Here is the complete code of that class:


/**
 * An implementation of MultipartWrapper that uses Jakarta Commons FileUpload (from apache)
 * to parse the request parts. This implementation requires that both commons-fileupload and
 * commons-io be present in the classpath.  While this implementation does introduce additional
 * dependencies, it's licensing (ASL 2.0) is significantly less restrictive than the licensing
 * for COS - the other alternative provided by Stripes. This implementation allows handling
 * uploads on the Google App Engine, as it does not rely on storing the files temporarily
 * on the local file system.
 * 
 * @author Moritz Petersen
 */
public class GaeMultipartWrapper implements MultipartWrapper {
    /**
     * Ensure this class will not load unless Commons FileUpload is on the
     * classpath.
     */
    static {
        FileUploadException.class.getName();
    }

    private final Hashtable<String, FileBean> files = new Hashtable<String, FileBean>();
    private final Hashtable<String, String[]> parameters = new Hashtable<String, String[]>();

    /**
     * Pseudo-constructor that allows the class to perform any initialization
     * necessary.
     * 
     * @param request
     *            an HttpServletRequest that has a content-type of multipart.
     * @param tempDir
     *            a File representing the temporary directory that can be used
     *            to store file parts as they are uploaded if this is desirable
     * @param maxPostSize
     *            the size in bytes beyond which the request should not be read,
     *            and a FileUploadLimitExceeded exception should be thrown
     * @throws IOException
     *             if a problem occurs processing the request of storing
     *             temporary files
     * @throws FileUploadLimitExceededException
     *             if the POST content is longer than the maxPostSize supplied.
     */
    public void build(HttpServletRequest request, File tempDir, long maxPostSize) throws IOException, FileUploadLimitExceededException {
        try {
            String charset = request.getCharacterEncoding();
            DiskFileItemFactory factory = new DiskFileItemFactory();
            factory.setRepository(tempDir);
            ServletFileUpload upload = new ServletFileUpload(factory);
            upload.setSizeMax(maxPostSize);
            Map<String, List<String>> params = new HashMap<String, List<String>>();

            for (FileItemIterator it = upload.getItemIterator(request); it.hasNext();) {
                FileItemStream item = it.next();
                // If it's a form field, add the string value to the list
                final byte[] buffer = IOUtils.toByteArray(item.openStream());
                if (item.isFormField()) {
                    List<String> values = params.get(item.getFieldName());
                    if (values == null) {
                        values = new ArrayList<String>();
                        params.put(item.getFieldName(), values);
                    }
                    values.add(charset == null ? new String(buffer) : new String(buffer, charset));
                }
                // Else store the file param
                else {
                    files.put(item.getFieldName(), new FileBean(null, item.getContentType(), item.getName() ) {
                        @Override
                        public long getSize() {
                            return buffer.length;
                        }
                        
                        @Override
                        public InputStream getInputStream() throws IOException {
                            return new ByteArrayInputStream(buffer);
                        }
                        
                        @Override
                        public void save(File toFile) throws IOException {
                            throw new UnsupportedOperationException();
                        }
                        
                        @Override
                        public void delete() throws IOException {
                            throw new UnsupportedOperationException();
                        }
                    });
                }
            }

            // Now convert them down into the usual map of String->String[]
            for (Map.Entry<String, List<String>> entry : params.entrySet()) {
                List<String> values = entry.getValue();
                parameters.put(entry.getKey(), values.toArray(new String[values.size()]));
            }
        }
        catch (FileUploadBase.SizeLimitExceededException slee) {
            throw new FileUploadLimitExceededException(maxPostSize, slee.getActualSize());
        }
        catch (FileUploadException fue) {
            IOException ioe = new IOException("Could not parse and cache file upload data.");
            ioe.initCause(fue);
            throw ioe;
        }

    }

    /**
     * Fetches the names of all non-file parameters in the request. Directly
     * analogous to the method of the same name in HttpServletRequest when the
     * request is non-multipart.
     * 
     * @return an Enumeration of all non-file parameter names in the request
     */
    public Enumeration<String> getParameterNames() {
        return parameters.keys();
    }

    /**
     * Fetches all values of a specific parameter in the request. To simulate
     * the HTTP request style, the array should be null for non-present
     * parameters, and values in the array should never be null - the empty
     * String should be used when there is value.
     * 
     * @param name
     *            the name of the request parameter
     * @return an array of non-null parameters or null
     */
    public String[] getParameterValues(String name) {
        return parameters.get(name);
    }

    /**
     * Fetches the names of all file parameters in the request. Note that these
     * are not the file names, but the names given to the form fields in which
     * the files are specified.
     * 
     * @return the names of all file parameters in the request.
     */
    public Enumeration<String> getFileParameterNames() {
        return files.keys();
    }

    /**
     * Responsible for constructing a FileBean object for the named file
     * parameter. If there is no file parameter with the specified name this
     * method should return null.
     * 
     * @param name
     *            the name of the file parameter
     * @return a FileBean object wrapping the uploaded file
     */
    public FileBean getFileParameterValue(String name) {
        return files.get(name);
    }
}

To use the GaeMultipartWrapper, it needs to be added to the StripesFilter in web.xml:


    <init-param>
      <param-name>MultipartWrapper.Class</param-name>
      <param-value>org.cloudme.stripes.GaeMultipartWrapper</param-value>
    </init-param>

Some ODF links

A collection of links related to the OpenDocument Format (ODF) and Java development:

Snow Leopard, Java 6, GWT and Eclipse

After upgrading to Snow Leopard, I experienced some well known issues in Java development: a) Java 5 has been removed from the computer and b) GWT doesn’t work anymore. While a) is not too bad (but not nice either), b) is a real pain. The reason why GWT doesn’t work is not caused by Apple, but the crappy implementation for a 32-bit runtime by Google. Luckily there is a way to work around this problem. The Lombardi blog has an entry explaining the reasons and what needs to be done.

In short: download the GWT source, patch the BootStrapPlatform class and make sure GWT is started in 32-bit mode.

In this solution I assume, you are using the Google Eclipse Plugin as described in the documentation.

Download the latest source from SVN: svn checkout http://google-web-toolkit.googlecode.com/svn/trunk/ trunk.
Find the class BootStrapPlatform (make sure it is the Mac OS X relevant class!).
BootStrapPlatform Class
Now, drag the class into your Eclipse project. Eclipse will show two errors: The package will be incorrect and the classes GraphicsEnvironment and Toolkit will be not validated. Fix both errors (“quick fix”, for the second error choose “Exclude ‘BootStrapPlatform.java’ from App Engine validation”.
Now change the method isJava5 to return true:


  private static boolean isJava5() {
      return true;
      // return System.getProperty("java.version").startsWith("1.5");
  }

Now change the run configuration of the project and add the VM argument -d32.
That’s it. But hopefully, Google will patch the GAE SDK soon.