Towards web engineering

2007-05-11

Perhaps you have never heard of the term web engineering. That is not surprising, because it is not commonly used. The fact that it is not commonly used is, however, surprising. Very surprising actually. The process of software creation resembles that of website creation and engineering principles are readily applied to the former. A website can be considered an information system. However, there is one peculiarity: the creation of a website is as much a technical as an artistic process.

The design of graphics and text content (presentation) goes hand in hand with the design of data structures, interactive features, and user interface (application). Despite some crucial differences, web development and software development have many features in common. Since we are familiar with software engineering, and since we understand its principles and advantages, it seems sensible to apply similar principles to web development. Thus web engineering.

Before expanding on this thought delving into the details of the engineering process, let’s try to define the term “web engineering”. Here is the definition that Prof. San Murugesan suggested in his 2005 paper Web Engineering: Introduction and Perspectives: “Web engineering uses scientific, engineering, and management principles and systematic approaches to successfully develop, deploy, and maintain high-quality web systems and applications.”

Web Engineering

Maturation Stages

It seems that current web development practices rarely conform with this definition. Most websites are still implemented in an uncontrolled code-and-fix style without codified procedures and heuristics. The person responsible for implementation is often a webmaster who is basically a skilled craftsman. The process of crafted web development is quite different from an engineering process. Generally speaking, there is a progression that all fields of industrial production go through from craftsmanship to engineering. The following figure illustrates this maturation process (Steve McConnel, 2003):

Maturity Stages

At the craft stage, production is carried out by accomplished craftsmen, who rely on artistic skill, rule of thumb, and experience to create a product. Craftsmen tend to make extravagant use of resources and production is often time consuming. Therefore, production cost is high.

The commercial stage is marked by a stronger economic orientation. Growing demand, increased competition, and companies instead of craftsmen carrying out work are hallmarks of the commercial phase. At this stage, the production process is systematically refined and codified. Commercial suppliers provide standardised products at competitive prices.

Some of the technical challenges encountered at the commercial stage cannot be solved, because the research and development costs are too high for individual manufacturers. If the economic stakes are high enough, a corresponding science will emerge. As the science matures, it develops theories that contribute to commercial practice. This is the point at which production reaches the professional engineering stage.

On a global level, software production was in the craft stage until the 1970s and has progressed to the commercial stage since then. Though the level of professional engineering is already on the horizon, the software industry has not reached it yet. Not all software producers make full use of available methods, while other more advanced methods are still being researched.

Recent developments in methodology

Web development became a new field of production with the universal breakthrough of the Internet in the 1990s. During the subsequent decade, web development has largely remained in the craft stage. This is now changing, albeit slowly. The move from hand-coded websites to template-based websites, content management systems, and standard application packages signals the transition from craft to commercial stage. Nonetheless, web development has not yet drawn level with the state of art in software development.

Until recently, web development was like a blast from the past. Scripting a web application involved the use of arcane languages for producing non-reusable, non-object-oriented, non-componentised, non-modularised, and in the worst case non-procedural code. HTML, JavaScript, and messy CGI scripts were glued together to create so-called dynamic web pages. In other words, web development was a primitive craft. Ironically, all principles of software engineering were either forgotten or ignored. Thus, in terms of work practices, developers found themselves thrown back twenty years. However, the rapid growth of the Internet quickly made these methods appear outmoded.

During the past 15 years, the World Wide Web underwent a transformation from a linked information repository (for which it was originally designed) to a universal vehicle for worldwide communication and transactions. It has become a major delivery platform for a wide range of applications, such as e-commerce, online banking, community sites, e-government, and others. This transition has created demand for new and advanced web development technologies.

Today, we have some of these technologies. We have unravelled the more obscure aspects of HTML. We have style sheets to encapsulate presentation logic. We have OOP scripting languages for web programming. We have multi-tier architectures. We have more capable componentised browsers. We have specialised protocols and applications for a great variety of purposes.

However, we don’t yet have established methodologies to integrate all these technologies and build robust, maintainable, high-quality web applications. We don’t yet have a universally applicable set of best practices that tells us how to build, say a financial web application, from a server language, a client language, and a database backend. Consequently, there is still a good deal of black magic involved in web development.

If you build a house, you can choose from a number of construction techniques and building materials, such as brick, wood, stone, or concrete. For any of these materials, established methods exist that describe how to join them to create buildings. Likewise, there are recognised procedures to install plumbing, electricity, drainage, and so on. When you build a house, you fit ready-made components together. Normally you don’t fabricate your own bricks, pipes, or sockets.

Unfortunately, this is not so in the field of web development. Web developers do not ubiquitously rely on standard components and established methods. On occasion, they still manufacture the equivalent of bricks, pipes, and sockets for their own purposes. And they fit them together at their own discretion, rather than by following standard procedures. Unsurprisingly, this results in a more time consuming development process and in less predictable product quality.

Having recognised the need for web engineering, a number of question arises. What does web engineering have in common with software engineering? What are the differences? Which software engineering methods lend themselves best to web development? Which methods must be defined from scratch? A detailed examination of all these questions is unfortunately beyond the scope of this article. However, we can briefly outline the span of the field and name those aspects that are most crucial to the Web engineering process.

Web applications are different

Web applications are inherently different from traditional software applications, because they combine multimedia content (text, graphics, animation, audio, video) with procedural processing. Web development comprises software development as well as the discipline of publishing. This includes, for instance, authoring, proofing, editing, graphic design, layout, etc.

Web applications evolve faster and in smaller increments than conventional software. Installing, fixing, and updating a website is easier than distributing and installing a large number of applications on individual computers. Web applications can be used by anyone with Internet access. Hence, the user community may be vast and may have different cultural and educational backgrounds. Security and privacy requirements of web applications are more demanding. Web applications grow in an environment of rapid technological change. Developers must constantly cope with new standards, tools, and languages.

Multidisciplinary approach

Building a large website involves dissimilar tasks such as photo editing, graphics design, user interface design, copy writing, and programming, which in turn requires a palette of dissimilar skills. It is therefore likely that a number of specialists are involved in the creation of a website, each one working on a different aspect of it. For example, there may be a writer, a graphic designer, a Flash specialist and a programmer in the team. Hence, web development calls for a multidisciplinary approach and team work techniques. “Web development is a mixture between print publishing and software development, between marketing and computing, between internal communications and external relations, and between art and technology.” (Powell, 2000)

The website lifecycle

The concept of the website lifecycle is analogous to that of the software lifecycle. Since the field of software engineering knows several competing lifecycle models, there are likewise different approaches to website design. For example, the waterfall model can be used for relatively small web sites with mainly static content:

  1. Requirements Analysis
  2. Design
  3. Implementation
  4. Integration
  5. Testing and Debugging
  6. Installation
  7. Maintenance

This methodology obviously fails for larger websites and web applications for the same reason it fails for larger software projects: the development process of large scale projects is incremental and requirements typically evolve with the project. But there is another argument that speaks for a more incremental/iterative methodology. Web applications are much easier to rollout and update than traditional software applications. Shorter lifecycles therefore make good practical sense. The “release early, release often” philosophy of the open source community certainly applies to web development. Frequent releases increase customer confidence, feedback, and avoid early design mistakes.

Categories of web applications

Different types of web applications can be distinguished by functionality (Murugesan, 2005):

Function Examples
Informational Online newspapers, product catalogues, newsletters, manuals, reports, online classifieds, online books.
Interactive Registration forms, customised information presentation, online games.
Transactional Online shopping (ordering goods and services), online banking, online airline reservation, online payment of bills
Workflow Oriented Online planning and scheduling, inventory management, status monitoring, supply chain management
Collaborative work environments Distributed authoring systems, collaborative design tools
Online communities, market places Discussion groups, recommender systems, online market places, e-malls (electronic shopping malls), online auctions, intermediaries

Maintainability

Maintainability is an absolutely crucial aspect in the design of a website. Even small to medium sites can quickly become difficult to maintain. Many developers find it tempting to use static HTML for presentation logic, because it allows for quick prototyping, and because script code can be mixed seamlessly with plain HTML. What is more, presentation logic is notoriously difficult to separate from business and application logic in web applications. However, this approach is only suitable for very small sites. In most other cases, a HTML generator, a template engine, or a content management system will be more appropriate.

Dependency reduction also contributes to maintainability. Web applications have multiple levels of dependencies: internal script code dependencies, HTML and template dependencies, and style sheet dependencies. These issues need to be addressed individually. The reduction of dependencies usually comes at the price of increasing redundancy/complexity. For example, it is more complex to use individual style sheets for different templates, than to use one master style sheet for all. Internal dependencies and complexity levels need to be balanced by an experienced developer.

As in conventional software development, code reusability is paramount to maintainability. Modern object-oriented scripting languages allow for adequate encapsulation and componentisation of web application parts. The problem is that developers do not always make use of these programming techniques, because they require more upfront planning effort. In environments with high economical pressure, there is a tendency to ad-hoc coding which yields quick results, but lacks maintainability.

Scalability

Scalability is one of the friendlier facets of web development, because the web platform is -at least in theory- inherently scalable. Increasing traffic can often be handled by simply upgrading the server system. Nonetheless, developers still need to consider scalability when designing software systems. Two important issues are session management and database management. Memory resource usage is proportional to the amount of session management data. I/O and CPU load is proportional to concurrent database access, thus developers do well to anticipate peak loads and optimise links between different tiers in an n-tier system.

Ergonomics and aesthetics
The look-and-feel of a website, its layout, navigation, colour schemes, menus, etc. make up the ergonomics and aesthetics of a website. This is a more artistic aspect of web development and it should be left to a professional with relevant skills and a good understanding of usability aspects. Software ergonomics and aesthetics should not be an afterthought, but an integral aspect of the development process. These factors are strongly influenced by culture. A website that appeals to a global audience is more difficult to build than a website targeted at a specific group and culture.

Website testing

Website testing comprises all aspects of conventional software testing. In addition, it involves special fields of testing which are specific to web development:

  • Page display
  • Semantic clarity and cohesion
  • Browser compatibility
  • Navigation
  • Usability
  • User Interaction
  • Performance
  • Security
  • Code standards compliance

Compatibility and interoperability

Web applications are generally intended to run on a large variety of client computers. Users might use different operating systems, different font sets, different languages, different monitors, different screen sizes, and different browsers to display web pages. In some cases, web applications do not only run on standard PCs, but also on pocket computers, PDAs, mobile phones, and other devices. While it is nearly impossible to guarantee proper page display on all of these devices, there needs to be a clearly defined scope of interoperability. This scope should spell out at least the browsers, browser versions, languages, and screen sizes that are to be supported.

Bibliography

Steve McConnell (2003). Professional Software Development
San Murugesan (2005). Web Engineering: Introduction and Perspectives
T.A. Powell (1998). Web site engineering: Beyond Web page design