The first question for the budding gadget developer is then which gadget
technology to choose. In an ideal world, there would only be a single standardised package format and only a single standardised API. This would allow gadgets to be used on any platform and the question of choosing a format would not even arise. Alas, we don't live in an ideal world and therefore different platforms and markets have produced different gadget formats. For example, there are Windows desktop gadgets, Linux desktop gadgets, Google gadgets, and gadgets designed to be integrated into web portals. We will look at the different types of gadgets and their use in brief.

Windows Gadgets

Formerly known as Windows sidebar, Windows gadgets are based on the widget engine for Microsoft gadgets, which runs on the Windows platform only. A minimal gadget contains an XML configuration file (gadget.xml) and an HTML file (main.html). Other web files can be added. These are zipped for distribution and the resulting file is renamed to *.gadget. Windows gadgets have access to a special API divided into three parts. 1. Gadget objects provide gadget state and event handling. 2. System objects provide access to files, network and OS functions. 3. Presentation objects provide visual functionality, namely background, image, and text handling. Many Windows desktop gadgets can also be run (with slight modifications) inside a Windows Live homepage. The latter don't have access to the system API and cannot modify the page's DOM object tree.

Apple Dashboard

Dashboard is an application that hosts widgets on a Mac Computer. The widgets are contained in an invisible layer that is activated by clicking on a dock icon, or by pressing a key. Like Windows gadgets, Dashboard widgets are based on standard web technologies. A typical dashboard gadget contains six files: a property list and a JavaScript containing the interactive functionality, and HTML and CSS files, a background image and an icon for the visual design. Dashboard implements a client server architecture with widgets running as clients. There are three classes of Dashboard widgets: Accessory widgets that are self-contained mini-applications like clocks, calculators, etc., application widgets that interact with an existing Mac application, and information widgets that retrieve information from the Internet.

Google Gadgets

As is the case with Windows gadgets, Google gadgets come in different flavours. They are based on the Google Gadget API and run inside an iGoogle page or can be embedded into any web page, usually by loading content from a remote server. Google gadgets can also be run on the desktop if the Google Desktop product is installed, which is a bit of a downer, because Google Desktop also contains desktop search functionality that constantly indexes your PC's filesystem and allows text searches on all of your files. The good news is that the latter functionality can be disabled. Furthermore, there are Google gadgets with enhanced capabilities for the (recently decommissioned) Google Wave application. Like Windows gadgets, Google gadgets consist of XML, HTML, JavaScript (lots of it) and other web files. The advantage over their Windows cousins is that Google gadgets are more platform-independent, since Google Desktop is available for Windows, Linux and Mac. Reusing web gadgets for the desktop (or vice versa) is also easier. The Java-like Google Gadgets API provides methods in the gadgets.* namespace for IO, string and JSON processing, skinning, and other functions. Developers can use the iGoogle gadget editor and gadget testing environment for creating gadgets.

Yahoo Widgets

Google's competitor Yahoo also offers a gadget technology called Yahoo widgets based on the Konfabulator product. Yahoo widgets are primarily intended to run on the desktop rather than inside a web page and to that end, users must install the Yahoo widget engine. Unfortunately, this product is closed source and only available for Windows and Mac. Like their cousins, Yahoo widgets are comprised of XML, HTML, JavaScript, CSS (and optionally Flash) and are zipped into a single *.widget file for distribution. The comprehensive Yahoo Widgets API includes functions for event-driven GUI programming, DOM processing, downloading web pages, and access to Yahoo services. It is even possible to create and use an SQLite database with Yahoo widgets or access OS-specific functions by running shell scripts on Windows or AppleScript on the Mac.

Linux/Unix Gadgets

There is a variety of widget engines available for Linux and the market seems to be highly fragmented. For the already mentioned Google gadgets, Linux users can download the open source Google-Gadgets-For-Linux software that allows Google gadgets to be run without Google Desktop. In addition, there are the following widget engines, among others, for which a limited choice of existing widgets is available:

Gdesklet - is a Gnome program for running gadgets on a Linux desktop. Despite its name, it can be also be used with other Desktop managers other than Gnome, like KDE or Xfce. Desklets are applets programmed in the Python language.

SuperKaramba - is a widget engine for the KDE desktop. The visual aspects of a SuperKaramba widget are specified in a text file, while its functionality can be programmed in either Python, Ruby, or JavaScript.

Screenlets - is a X11/Compiz-based widget engine that is independent of the desktop environment. It supports Python applets with skins drawn in SVG and -more recently- web widgets written in HTML, CSS and JavaScript.

Although I guess that Android will grab the larger market share in the long run, the Apple iPhone has presently the edge. Apple is clearly the leader, not only in sales, but more importantly in design and technology, as well as in being the most expensive. And what do the other competitors do? Follow the leader, of course.

As a consequence, one can find a growing number of "i-phony" applications that make your device look like an iPhone. The must have ingredients are: 1) jewel case look - shiny multi-coloured icons on a matt black background, 2) finger-tip sized GUI elements (retire your stylus), and 3) sliding dynamics that allow users to pan and scroll virtual sceens across the display without scroll bars.

The old adage says: "If you can't make it, fake it." If you happen to own a device with an aging operating system, such as Symbian or Windows Mobile, there are ways to jazz up your phone without shelling out half a month's salary for a new smart phone. SPB's Mobile Shell 3.5, for example, will give your Pocket PC or phone a new coat of paint. It can't magically transform your device into an iPhone, but it's as "i-phony" as it gets without actually copying the Apple look.
 

For those of you old enough to remember Dbase, the NoSQL moniker may not be much of an attention grabber, because after all, products like Dbase, FoxPro, Clipper and similar DB systems never had SQL support either. With these systems, relations had to be expressed implicitly in the application and “queries” had to be coded as retrieval sequences. By contrast, modern NoSQL systems depart from the relational model and in many cases also from the tabular data structure, in order to serve use cases where traditional RDBMS fail in one or another way. A typical example would be a sparsely populated table that contains very few data in rows and columns. Such a table -if it grows to a large size- presents an efficiency problem to most RDBMS with resulting performance loss. In the remainder of this article, we will look at a few selected NoSQL databases and see which use cases they cater to.

CouchDB

Apache CouchDB is a document-oriented database that represents documents as JSON objects. CouchDB supports all data types supported by JSON, or respectively by Javascript. The JSON objects are not required to comply with schemas and can therefore be defined freely, which means that each JSON object can have a different structure. CouchDB supports queries by views. Views are aggregate functions and filters programmed in Javascript that follow the MapReduce algorithm. Views are stored and indexed in the database. CouchDB provides a RESTful API where every object (and any other item) in the database can be retrieved by an URL. It uses the HTTP POST, GET, PUT, and DELETE methods for CRUD operations. Other features include ACID semantics on basis of multi-version concurrency control, similar to RDBMS, which is optimised for a high number of concurrent reads, and a distributed architecture that allows for easy bidirectional replication and offline usage. CouchDB is thus designed from ground up for Internet use.

Neo4J

Neo4J is a graph database. As the name suggests, it is intended for use with the Java platform, which includes any language that runs on the JVM. Neo4J stores information in nodes and edges; the latter are called relationships in case of Neo4J. Relationships are always of a defined type. Both nodes and relationships can store properties, i.e. data. The Neo4J database is thus optimised for representing complex graph and network structures, such as a hierarchical object repository or a social network. It offers high-performance graph traversal operations for data access. Nodes can also be indexed and retrieved by key which enables more conventional style queries. Additional features include ACID transactions and transaction recovery, based on the Java Transaction API (JTA). Optional libraries can expose a Neo4J database as an RDF store where the node space can be queried using SPARQL. Neo4J is an embedded database with a small footprint that runs in the same JVM as the application.

Redis

Redis is a modern implementation of a persistent key-value store for general purpose use. Key-value store is a name for a simple key-based access mechanism that basically implements a dictionary (or map) data structure. Traditionally, such systems were used for caching and Redis holds its entire database in memory, which makes it ideal for applications that require ultra-fast data access. Redis allows not just plain string data but also allows sets and lists of strings in the data space. The system offers a number of special commands, such as atomic push/pop and add/remove operations for lists and set operations such as building union, intersection, and difference. Redis persists data either by asynchronously writing memory to disk, or by appending to a journalling file as data is written by clients. Additional features include easy master-slave replication and rudimentary sharding. Redis offers support for various languages, such as C/C++, Java, Scala, PHP and others through native drivers and APIs.

HBase

HBase is a free implementation of Google’s BigTable written in Java. It is not the type of database you would use for a blog or a forum software. HBase is a tabular data storage designed for massive tables in the Petabyte range with billions of rows distributed over a number of physical machines and thus optimised for horizontal scaling. HBase is part of the Apache Hadoop project, a framework for data-intensive distributed applications, inspired by Google’s MapReduce and GFS technologies. Hadoop supports the database through its distributed filesystem HDFS which provides built-in replication and MapReduce traversal for HBase tables of arbitrary size. Features include optimised query push down via server-side scan and get filters, a high performance Thrift gateway, an XLM-based RESTful Webservice gateway, Hadoop cascading, per-column probabilistic Bloom filters, as well as data warehousing and data analysis modules. Since HBase saves column families rather than columns and since empty columns are not stored, it is ideal for sparse tables with semi-structured data. Typical use cases are cloud computing and applications that require massive storage using cheap commodity hardware.

Db4o

Db4o is an open-source object-oriented database system targeted at OOP developers. The idea behind Db4o is to enable programmers to create and persist a representation of the application object model directly in the database without the need for an object-relational mapping software layer. Object instances can then be stored and retrieved with a single line of code. Db4o provides a query mechanism called Native Query (NQ). This allows querying data with native OOP language constructs thus offering type safety for query expressions while eliminating the need for building query strings. Db4o is available for the Java and .NET platforms. If used with .NET languages, data can alternatively be queried with LINQ (language integrated query). The Db4o database is embeddable with a small footprint suitable to be deployed on mobile devices. Additional features include semi-automatic schema versioning, transaction support with ACID semantics, and synchronisation/replication mechanisms that allow synchronisation between different Db4o instances and data export into SQL databases.

The only thing that turned out to be slightly trickier was the Tomcat upgrade to 6.0.24. A surreptitious installation of Apache 2 (the purpose of which eluded me) took possession of port 80 which on my machine was previously occupied by the system-wide Tomcat installation. This was rather easy to solve with the command: sudo update-rc.d -f apache2 remove to disable Apache on boot. It turned out, however, that the application launcher jsvc was removed in Ubuntu 10.04. Since Tomcat previously used jsvc to launch Tomcat on privileged ports, Tomcat was not able to bind to port 80 any longer. I was able to solve this by setting the AUTHBIND variable in /etc/default/tomcat to ‘yes’. After that Tomcat started up on port 80 without complaints.

During the upgrade, the system politely asked whether to replace or keep manually changed system configuration files. I have chosen to replace most files, because, the upgrade manager is kind enough to create a copy of the existing configuration using the *.dpkg-old extension during the upgrade. That way I was able to diff configuration files later and incorporate any customisations into the new files. This method is superior to keeping the old files, because it allows for upgrading the configuration files in sync with the latest program versions, though, of course it takes a bit of work manually diffing and patching those files if you happen to have numerous customisations. You can alternatively keep the old files and then diff and patch the new files created by the upgrade manager with the *.dist extension. In summary, the upgrade was painless and took less than 90 minutes per machine.

http://www.thomasknierim.com/scala-pages-web-framework/

The source code is included in the distribution zip file. Comments, suggestions, and constructive criticism are always welcome.

Addendum: I was asked to make a brief comparison to the Lift framework, so here’s my answer:

The approach that SCP takes is different. To put it in a nutshell, Lift is Rails-inspired, XML-oriented, and it abstracts from the Servlet API and the request/response model. SCP is inspired by traditional Java MVC frameworks; it is text-oriented, and it builds directly on the request/response model and the Servlet API.

Lift processes templates using the SAX-parser and presents Scala XML data structures to the programmer. By contrast, SCP reads templates as plain text from top to bottom, performs variable replacement and executes embedded instructions. Template processing with SCP thus consumes fewer resources. Lift uses prefixed XML tags; SCP uses processing instructions.

Lift wraps much of the Servlet API and presents a number of abstracted objects to the programmer, such as the S object, the SHtml object, LiftRules, and so on. SCP doesn’t do that. With SCP you get the familiar JEE objects such as HttpServletRequest, ServletContext, etc. and you use these directly from the Scala code. So, it’s easy to use if you’ve done Java web development.

The consequence is that you deal with the request/response model directly, which I personally prefer, because it keeps the control flow simple and clear. The only disadvantage of the MVC request/response model is that controllers tend to become bulky and difficult to reuse. The SCP solution to this problem is the same that Lift offers: snippets.

In SCP, snippets are processing instructions powered by a custom Scala class that the programmer provides. This means you can encapsulate application logic and/or complicated display logic into a snippet class which -if properly coded- is perfectly reusable. SCP also keeps template expressions very simple, because the more complex a template language becomes (many are even Turing-complete), the more likely it is for application logic to sneak into the view. Currently, iteration is the only supported type of control-flow.

Last but not least, Lift is huge compared to SCP. It has a lot of functionality that SCP does not offer, at least not at this time.

Running two servers is a bit of a nuisance. Two servers consume more resources than one and you cannot run both on the same port. This problem can be solved in three different ways: you can only run one server at a time, you can use a different port number for one server which has to be included in the URLs, or you can integrate the two servers. There are again at least three different ways to accomplish the latter: you can proxy requests from Apache to Tomcat, you can proxy request from Tomcat to Apache, or you can use a connector module, such as mod_jk. Of course, maintaining two servers is is more complicated than maintaining one, and the integration adds additional complexity. Fortunately, there is an easier way to integrate PHP and Java web applications.

PHP/Java Bridge is a free open source product for the integration of the native PHP interpreter with the Java VM. It is designed with web applications in mind: Java servlets can “talk” to PHP scripts and vice versa. The official website describes it as an “implementation of a streaming, XML-based network protocol which is up to 50 times faster than local RPC via SOAP.” PHP/Java Bride requires no additional components to invoke Java procedures from PHP or vice versa. Although there are a number of different use cases, I am going to describe a particular one in this article, namely how to configure Tomcat with PHP/Java Bridge in order to have Tomcat serve PHP web pages. Let’s start with software requirements. We need the following software packages:

• A Java virtual machine (JRE is sufficient, but JDK recommended for developers)
• A recent version of Apache Tomcat (http://tomcat.apache.org)
• A recent version of PHP (standalone, without Webserver: http://php.net)
• PHP/Java Bridge (http://php-java-bridge.sourceforge.net/pjb/)

Follow the standard installation procedures for the JVM, Tomcat, and PHP. On Linux, you can use the standard packages for your distribution and on Windows you can use the regular installers. Make sure that both Tomcat and PHP are installed properly, which means that you should see Tomcat’s welcome web page at http://localhost:8080 and you should be able to execute a PHP script via the command line by invoking the standalone “php” command. The PHP/Java Bridge product does not use the regular executable, however, but fast CGI. The fast CGI executable is called php-cgi (or Php.cgi.exe on Windows), so you must make sure that your PHP installation contains it. Then you are all set to install and configure the PHP/Java Bridge.

The PHP/Java Bridge package comes with a sample web application named JavaBridge.war. Deploy the application in Tomcat, point your browser to http://localhost:8080/JavaBridge and try out the examples. If this works, you are half-finished. To provide the capability to execute PHP scripts server-wide, not just in a single web application, you need to make some changes to the Tomcat configuration. Find the three jar files named JavaBridge.jar, php-servlet.jar and php-script.jar (look in WEB-INF/lib) and move them to Tomcat’s shared library directory. This is usually found in $CATALINA_HOME/lib (or $CATALINA_HOME/shared in older Tomcat installations). Then edit Tomcat’s conf/web.xml configuration file and add the following lines:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

<listener>
  <listener-class>
    php.java.servlet.ContextLoaderListener
  </listener-class>
</listener>
 
<servlet>
  <servlet-name>PhpJavaServlet</servlet-name>
  <servlet-class>
    php.java.servlet.PhpJavaServlet
  </servlet-class>
</servlet>
 
<servlet>
  <servlet-name>PhpCGIServlet</servlet-name>
  <servlet-class>
    php.java.servlet.PhpCGIServlet
  </servlet-class>
  <init-param>
<param-name>prefer_system_php_exec</param-name>
<param-value>On</param-value>
  </init-param>
  <init-param>
<param-name>php_include_java</param-name>
<param-value>On</param-value>
  </init-param>
</servlet>
 
<servlet-mapping>
  <servlet-name>PhpJavaServlet</servlet-name>
  <url-pattern>*.phpjavabridge</url-pattern>
</servlet-mapping>
 
<servlet-mapping>
  <servlet-name>PhpCGIServlet</servlet-name>
  <url-pattern>*.php</url-pattern>
</servlet-mapping>

This adds the listeners and servlets required for PHP script execution to all web applications. While you are at it, you might also want to enable index.php files to display when a directory URL is requested. Simply add it to the list of welcome files in conf/web.xml. My list looks like this:

1
2
3
4
5
6

<welcome-file-list>
    <welcome-file>index.html</welcome-file>
    <welcome-file>index.htm</welcome-file>
    <welcome-file>index.jsp</welcome-file>
    <welcome-file>index.php</welcome-file>
</welcome-file-list>

Now you can copy PHP scripts into the context root directory of any web application and type the script URL into your browser. I suggest you try a script with phpinfo(). It gives you plenty of useful configuration info. If this doesn’t work and you are on Unix, the problem might be file permissions. On my machine, I had to copy the contents of “java” directory in the JavaBridge webapp manually to the context root directory where PHP applications were installed. This directory contains two files Java.inc and JavaProxy.php. Normally, the PHP/Java Bridge software copies it automatically, but it might not be able to do so if it does not have proper permissions:

~$ ls -lh /var/lib/tomcat6/webapps/ROOT/java
total 136K
-rw-rw-r-- 1 root root 64K 2009-12-30 14:14 Java.inc
-rw-rw-r-- 1 root root 64K 2009-12-30 14:14 JavaProxy.php

Now try calling a PHP script. For example, a script containing the phpinfo() command displays information about the server:

I have configured my machine to host all PHP web applications in Tomcat’s ROOT context. This eliminates the extra path component of the webapp context, since the ROOT’s context path is “/”. Then I softlinked the folder that contains all my PHP projects into the ROOT webapp directory, so that the actual source files are kept separate from the Tomcat installation. In order to enable Tomcat to follow symlinks, you need to edit the context.xml of the respective web application -in this case ROOT- and add the line: <Context path=”/” allowLinking=”true” /> .

Another possible gotcha is Tomcat’s security manager, which is enabled by default on Ubuntu, but not on Windows. Although a security manager is not necessary for most development scenarios, it is highly recommended for production. I consider it good practice to enable the security manager on the development machine, because it allows me to recognise security problems early during development, before the application is deployed on the production server. The downside is that additional configuration may be required, for PHP applications to function properly. The respective configuration files are located in $CATALINA_BASE/conf/policy.d. Most likely, you need to grant PHP web applications write access to files in the document root and possibly other permissions, such as opening sockets, etc. It’s probably safest to do this on a per-application basis.

Most of the key applications like Eclipse, Firefox, Thunderbird, and OpenOffice work exactly the same under Linux as they do under Windows. The only major change was replacing Notepad++ (which only runs on Windows) by vi/vim. These editors are suitable for programming in situations where you don’t want to fire up an IDE. Furthermore, I have made some customisations to ease the transition, which I’d like to share with you. If you are new to Linux, you might find one or another useful for your own work. The following list is by no means exhaustive or even comprehensive, just a number of things I stumbled across during my first two weeks with desktop Ubuntu.

Repositories and download servers
Ubuntu maintains software packages with the Synaptic package manager. Because as a new user you are likely to make frequent use of this tool, one of the most useful things to do is to optimise its usage. This involves defining the repositories and the download server. Choose System/Administration/Software Sources from the main menu. In the first tab “Ubuntu Software”, select the four items marked with “main”, “universe”, “restricted” and “multiverse” for the widest choice of software packages. Next, optimise the download server. I wasted a whole day with downloading the 9.04->9.10 update, because of a slow server. Ubuntu can find the fastest server for you. Select “Other…” in the “Download from” dropdown-box. A dialogue with a list of servers shows on screen. Click on “Select Best Server” to let Ubuntu test all available servers for their response time and select the fastest one.

Keyboard and language customisations
If you are -like me- frequently typing text in different languages, chances are that the default language and keyboard settings will not suit you. Fortunately, Ubuntu is easy to configure for international use, possibly even superior to Windows in this regard. First, I added Thai language support in System/Administration/Language Support. Then I configured two additional keyboard layouts, German and Thai, in System/Preferences/Keyboard/Layout. As I am using a Thai/English keyboard, I have to remember the German key mapping by heart which is only of limited use. On Windows I got used to producing international characters by typing ALT+num key sequences. On Linux, this is even easier thanks to the concept of the compose key. In the keyboard layout dialogue, click on “Layout Options” which will show you a number of intricate keyboard customisation options. Click on “Compose key position” and pick a key, for instance “Right Alt”. Now you can use this key to compose international characters. For example, type right Alt, double quotation marks, and letter ‘u’ to produce the German Umlaut ‘ü’. Type right Alt, backtick and the letter ‘a’ to produce the accent grave ‘à’. Voilà!

Customising Nautilus
Nautilus is the Linux/Gnome equivalent to the Windows Explorer. In fact, I find it to be superior to the latter, because it supports protocols for remote access (such as ftp/sftp); it offers better search capability and better support for compressed files. If you prefer to work with a GUI rather than the command line, you would probably want to customise Nautilus in some way. The most obvious candidates for customisation are probably file associations. These can be defined by right-clicking on a file, selecting “Properties” from the context menu and switching to the “Open With” tab in the property dialogue. Here you can define alternative applications to use for opening a file, as well as the default application that is started upon double-click. If you need even more customisation options, install the package named “nautilus-actions”. This package lets you define custom actions for file entries in Nautilus which can be incorporated into the context menu. Predefined Nautilus extensions (aka shell extensions) for various file display and transformation purposes are also available.

Command line and terminal customisations
Ubuntu comes with the bash (Bourne again shell) and the Gnome-Terminal as command line defaults. These are fine for me. However, there is one feature which I found missing in the terminal application. It is not possible to search the output buffer. For example, when I run applications that produce a large amount of diagnostic output, there is no intuitive way to search trough this data, other than piping it into a command like “less”. I have found a little program named “screen” which appears to solve this problem. After “screen” is started, virtual sessions can be created within the same terminal window, each with its own searchable buffer. “Screen” involves remembering some arcane keyboard commands, but that’s the best I could find so far. Another command line annoyance is that the “vi” editor runs in compatible mode by default. This will let the cursors keys produce character output in insert mode; in other words, the cursor keys are broken. There is an easy fix for this, however. Put a file named .vimrc in your home directory that contains a single line saying “set nocompatible” and the cursor keys will work again.

Backup and antivirus software
Surprisingly, neither backup nor antivirus software packages are included in the default Ubuntu installation. Although viruses are probably not an immediate threat on a Linux system, I would rather not breed any of them on my machine. There is the open source software clamAV as well as a number of free-for-private-use commercial offerings for Linux. I am still evaluating antivirus software. So far I found clamAV and AVG quite usable, but not quite as convenient as under Windows. Backup software is an absolute necessity in my opinion, and I am surprised that it isn’t integrated in the original Ubuntu installation. Of course, individual backup needs differ, but a simple mirroring and archiving facility is probably required for even the most basic usage. Initially, I planned to hack a script based on rsync together for that purpose, but I have found something much nicer. The “backintime” package lets you create incremental backups with great ease and minimal storage requirements. Backintime revolves around the concept of snapshots; it is a GUI framework for rsync, diff, and cron. I highly recommend it.

Personally, I’ve begun the new computing year with a major upgrade, though still far away from Kip’s 2010 vision. I have replaced my 32-bit Windows OS with a 64-bit Linux OS, doubled RAM from 4 GB to 8 GB and added another external USB hard drive for backups. The Ubuntu installation turned out a little difficult, because Linux did not want to cooperate with the BIOS RAID-1 configuration, so I had to switch to SATA mode and wipe out the Windows installation. The rest was easy, however. I used to worry about not being able to make my 3G USB modem work with Linux, but our maid has solved this problem for me. She obliterated the device by putting it into the washing machine. Can’t really blame her for that. I probably shouldn’t have carried the modem in the pockets of my shorts.

Back to the topic. What are the computing trends in 2010? Just off the top of my head: cloud computing is becoming a mainstream technology (or perhaps a mainstream buzz; time will tell). Along with that, virtualisation is now widely used. Supercomputers have broken the petaflop mark and now operate in the range of large clouds (> 1 PFlop). CRT monitors are quickly becoming relics of a past epoch. Single-core CPUs are headed the same way. Functional programming languages are beginning to catch on. 64-bit hardware and software are overtaking 32-bit systems in mainstream IT. Java 7 is announced for 2010. It surely looks like an interesting year.

Before we are going into that, let’s briefly look at the bright side. JSP technology is tempting for several reasons. First, it is mainstream and it bears the official stamp of approval by Sun. Second, it is an integral part of JEE and widely supported by application servers and tools. Third, it it is relatively easy to whip up dynamic content with JSP. So, why not use Java Server Pages? Well, there are several problems, the biggest of which are scriptlets. Scriptlets are embedded Java code. Because JSP allows you to mix Java code with markup, you get hybrid files which tend to become messy and difficult to maintain. Few HTML designers understand what’s going on in a JSP file. Few programmers are comfortable with mixing their code with reams of markup.

The typical production process for a JSP-based application looks like this: the UI designers finish their protoypes and hand over the resulting markup to the programmers. The programmers tear apart the HTML to make it easier to insert code, repeat  sequences, and insert display logic. Once the programmers are finished, there is absolutely no chance that the designers will recognise the markup they produced. It is therefore also unlikely that they will ever touch it again. Even if the finished product contains no scriptlets, JSTL, expression language and (God help us) custom tags will confuse the hell out of designers. This means that the entire process has to be repeated,  everytime a change of visuals is requested. This is however not the worst problem.

The biggest drawback is that there is absolutely no way to prevent application logic from creeping into the JSPs which ultimately leads to spaghetti code. You might think, “Oh well, I know how to code my JSPs properly.” At this point you should remember Murphy’s law, especially the part that says: “if something can go wrong it will go wrong.” Perhaps you are a disciplined individual who wouldn’t even dream about putting application logic into a JSP file. But can you say the same about the other programmers in your team? What if work needs to be completed under time pressure? Can you resist the temptation to solve a problem quick-and-dirty by putting a hack into a JSP file along with a “will fix this later” note? I have seen way too many “will fix this later” notes in JSPs and most of them were several years old.

My project consisted of an application with roughly 1.5 million LOC where about half of the code was implemented in JSPs. That’s about 1500-2000 JSP files. Most files were less than 500 lines, but some were in excess of 2000 lines. Once you hit upon an XL-sized JSP, it’s a safe bet that some crucial functionality is buried in it. Reading and understanding a 2000 line JSP can take several hours. Modifying and maintaining it is quite another deal. The project did not attempt to separate business logic from display logic in JSPs, thus XML was liberally mixed with scriptlets. While this approach works OK for narrow functionality that can be coded  in a single JSP, it becomes very unwieldy for functionality that spans a larger problem space and, hence, many JSPs.

The “solution” for this was to use scriptlet fragments which are shared by multiple JSPs. This isn’t a real solution, however, because it forgoes almost all advantages of using an OOP language, such as abstraction, encapsulation, inheritance, and as a side effect it produces “ultra-tight coupling” between JSPs that use the same fragments. In theory, this can be made slightly less painful by defining inner classes in fragments for shared functionality. These class fragments are then included in the consuming JSPs at compile time. However, inner classes have their own limitations and compile time includes bulk up the resulting byte code. Since inner classes can access variables and methods in the outer scope, they lack proper encapsulation. Tying them to multiple consumers can lead to some weird dependencies and logical errors. Finally, there is no way to produce unit tests for scriptlets. The only way to test scriptlets is by inserting the test code directly into the JSP, which is obviously insane. In summary, there is no way to code scriptlets cleanly, so it’s best to avoid them.

But even without scriptlets, there’s plenty of trouble. For example, there is the Java standard tag library (JSTL) and the expression language (EL) which are supposed to replace scriptlets as coding instruments. In particular, the EL has been praised as enabling clean coding for MVC applications with Java Server Pages as view component. - Well, I disagree. - JSTL+EL are neither very clean nor very concise. What is worse, they are too powerful for their own good. JSTL+EL are Turing-complete, just like XSLT which they resemble, which means that replacing scriptlets with JSTL+EL is like jumping from the frying pan into the fire. In addition, JSTL provides tags that allow programmers to access a database and execute queries. If you see MVC going out of the window at this point, you have recognised the problem. In summary, JSTL+EL produce the same problems as scriptlets, but with an XML syntax.

In conclusion, Java Server Pages can lead to maintenance nightmares, especially when used in a large project. While it is possible to code JSPs cleanly, it is apparently not a widespread practice, which is probably Murphy’s law taking its toll. Hence, if you are building a new Java web application, think twice about using JSP. If you have a legacy application, you might want to replace JSPs by something more appropriate. In most cases, this can be done gradually whereas legacy JSPs can run side by side with an alternative view technology.

I hope this will be useful for people interested in tinkering with Lift who want to avoid the aggravation of having to deal with Maven. As usual, you can download the project sekeleton and Ant build file right here. I have packaged the Lift 1.0 jars as well as its dependencies (commons-fileupload.jar and log4j.ar), so that you can get the helloworld example to run without additional installation steps. If you plan to use this for your work, you should replace the Lift jars with the most recent versions from GitHub. The Ant script has the same target definitions as the one I presented for general Web development including deploy, undeploy and reload. The Lift dependencies are likewise handled in the same way, as the script uses everything it finds in the WEB-INF/lib directory. Your only responsibility is to put the right jars in there. The main difference consists of a different directory tree structure that reflects the layout of a standard Lift project. This means you won’t have to rearrange things if you move a project from Maven to Ant. In fact all of the Lift examples should run simply by dropping in the sources (I’ve tried this for a few but not all examples). So, here’s the Ant script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235

<?xml version="1.0"?>
<project name="hellolift" default="build">
 
  <!-- root directory of this project -->
  <property name="project.dir" value="."/>
 
  <!-- root directory of Scala installation -->
  <property name="scala.home" 
    value="C:\\Program Files\\Scala"/>
 
  <!-- root directory of Tomcat installation -->
  <property name="server.home" 
    value="C:\\path\\to\\Tomcat\\installation"/>
 
  <!-- URL for Tomcat's manager application -->  
  <property name="server.manager.url" 
    value="http://dev:8080/manager"/>
 
  <!-- account name for Tomcat's admin account -->  
  <property name="server.manager.username" value="admin"/>
 
  <!-- password name for Tomcat's admin account -->  
  <property name="server.manager.password" value="admin"/>
 
  <!-- location of scalatest.jar for unit testing -->
  <property name="scalatest.jar" 
    value="C:\\path\\to\\scalatest-1.0\\scalatest-1.0.jar"/>
 
  <target name="init">
 
    <!-- derived path names -->
    <property name="source.dir" value="${project.dir}/src"/>
    <property name="web.dir" value="${project.dir}/web"/>
    <property name="classes.dir" 
      value="${web.dir}/WEB-INF/classes"/>
    <property name="lib.dir" 
      value="${web.dir}/WEB-INF/lib"/>
    <property name="webapp.dir" 
      value="${server.home}/webapps/${ant.project.name}"/>
    <property name="test.dir" value="${project.dir}/test"/>
 
    <!-- scala libraries for classpath definitions -->
    <property name="scala-library.jar" 
      value="${scala.home}/lib/scala-library.jar"/>
    <property name="scala-compiler.jar" 
      value="${scala.home}/lib/scala-compiler.jar"/>
 
    <!-- classpath for the compiler task definition -->   
    <path id="scala.classpath">
      <pathelement location="${scala-compiler.jar}"/>
      <pathelement location="${scala-library.jar}"/>
    </path>
 
    <!-- classpath for project build -->
    <path id="build.classpath">
      <pathelement 
        location="${server.home}/lib/servlet-api.jar"/>
      <pathelement location="${scala-library.jar}"/>
      <fileset dir="${lib.dir}">
        <include name="*.jar"/>
      </fileset>
      <pathelement location="${classes.dir}"/>
    </path>
 
    <!-- classpath for unit test build  -->
    <path id="test.classpath">
      <path refid="build.classpath"/>
      <pathelement location="${scalatest.jar}"/>
    </path>
 
    <!-- definition for the 
      "scalac" and "scaladoc" ant tasks -->
    <taskdef resource="scala/tools/ant/antlib.xml">
      <classpath refid="scala.classpath"/>
    </taskdef>
 
    <!-- definition for the "scalatest" ant task -->
    <taskdef name="scalatest" 
      classname="org.scalatest.tools.ScalaTestTask">
      <classpath refid="test.classpath"/>
    </taskdef>
 
    <!-- definition for the "reload", "deploy" 
      and "undeploy" Tomcat tasks -->
    <taskdef name="deploy" 
      classname="org.apache.catalina.ant.DeployTask">
      <classpath 
        path="${server.home}/lib/catalina-ant.jar"/>
    </taskdef>
    <taskdef name="reload" 
      classname="org.apache.catalina.ant.ReloadTask">
      <classpath 
        path="${server.home}/lib/catalina-ant.jar"/>
    </taskdef>
    <taskdef name="undeploy" 
      classname="org.apache.catalina.ant.UndeployTask">
      <classpath 
        path="${server.home}/lib/catalina-ant.jar"/>
    </taskdef>
 
  </target>
 
  <!-- compile project -->
  <target name="build" depends="init" description="build">
    <buildnumber/>
    <tstamp/>
    <mkdir dir="${classes.dir}"/>
    <mkdir dir="${lib.dir}"/>
    <copy file="${scala-library.jar}" todir="${lib.dir}"/>
    <scalac 
      srcdir="${source.dir}" 
      destdir="${classes.dir}"
      classpathref="build.classpath"
      force="never"
      deprecation="on"
    >
      <include name="**/*.scala"/>
    </scalac>
  </target>
 
  <!-- create a deployable web archive -->
  <target name="war" depends="build" description="war">
    <war destfile="${project.dir}/${ant.project.name}.war"
      webxml="${web.dir}/WEB-INF/web.xml"
      basedir="${web.dir}"
    />
  </target>
 
  <!-- creates a deployable web archive with 
    all classes packed into a single jar file -->
  <target name="packedwar" 
    depends="build" description="packedwar">
    <jar destfile="${lib.dir}/${ant.project.name}.jar" 
      basedir="${classes.dir}"
      duplicate="preserve">
      <manifest>
        <section name="Program">
          <attribute name="Title" 
            value="${ant.project.name}"/>
          <attribute name="Build" value="${build.number}"/>
          <attribute name="Date" value="${TODAY}"/>
        </section>
      </manifest>
    </jar>
    <delete dir="${classes.dir}"/>
    <war destfile="${project.dir}/${ant.project.name}.war"
      webxml="${web.dir}/WEB-INF/web.xml"
      basedir="${web.dir}"
    />
    <delete file="${lib.dir}/${ant.project.name}.jar"/>
  </target>
 
  <!-- deploy project on Tomcat server -->
  <target name="deploy" depends="war" description="deploy">
    <mkdir dir="${webapp.dir}"/>
    <copy todir="${webapp.dir}">
      <fileset dir="${web.dir}"/>
    </copy>
    <deploy url="${server.manager.url}"
      username="${server.manager.username}"
      password="${server.manager.password}"
      path="/${ant.project.name}"
      localWar="${project.dir}/${ant.project.name}.war"
      />
  </target>
 
  <!-- update and reload project on Tomcat server -->
  <target name="reload" 
    depends="build" description="reload">
    <copy file="${scala-library.jar}" todir="${lib.dir}"/>
    <copy todir="${webapp.dir}">
      <fileset dir="${web.dir}"/>
    </copy>
    <reload url="${server.manager.url}"
      username="${server.manager.username}"
      password="${server.manager.password}"
      path="/${ant.project.name}"/>
  </target>
 
  <!-- remove project from Tomcat server -->
  <target name="undeploy" 
    depends="init" description="undeploy">
    <undeploy url="${server.manager.url}"
      username="${server.manager.username}"
      password="${server.manager.password}"
      path="/${ant.project.name}"/>
  </target>
 
  <!-- build unit tests -->
  <target name="buildtest" depends="build">
    <mkdir dir="${test.dir}/build"/>
    <scalac 
      srcdir="${test.dir}" 
      destdir="${test.dir}/build" 
      classpathref="test.classpath"
      force="never"
      deprecation="on"
    >
      <include name="**/*.scala"/>
    </scalac>
  </target>
 
  <!-- run unit tests -->
  <target name="test" 
    depends="buildtest" description="test">
    <scalatest runpath="${test.dir}/build">
      <reporter type="stdout" config="YFABRT"/>
      <membersonly package="suite"/>
      <!-- <reporter type="graphic" config="YFABRT"/> -->
      <!-- <suite classname="suite.TestSuite"/> --> 
    </scalatest> 
  </target>
 
  <!-- delete all build files -->
  <target name="clean" depends="init" description="clean">
    <delete dir="${classes.dir}"/>
    <delete dir="${project.dir}/doc"/>
    <delete dir="${test.dir}/build"/>
    <delete file="${lib.dir}/scala-library.jar"/>
    <delete file="${project.dir}/${ant.project.name}.war"/>
  </target>
 
  <!-- create API documentation in doc folder -->
  <target name="scaladoc" 
    depends="build" description="scaladoc">
    <mkdir dir="${project.dir}/doc"/>
    <scaladoc 
      srcdir="${source.dir}" 
      destdir="${project.dir}/doc" 
      classpathref="build.classpath" 
      doctitle="${ant.project.name}" 
      windowtitle="${ant.project.name}"/>
  </target>
 
</project>