org.melati Requirements Analysis (document $Revision: 1.1 $)

# `org.melati` Requirements Analysis (document $Revision: 1.1 $)

This document provides an analysis of the requirements which the system is intended to meet. See also the system's master QA document.

## Background

### The customer

Melati is being developed to provide the infrastructure for the all future Paneris sites, and will hopefully be used by others when it is open-sourced.

### Project genesis

The Melati project came to life out of discussions on the JAL messageboard.

## Business need

JAL (Java Application Layer) is a class library layer which mediates between a relational database---currently Postgres---and webmacro controllers and templates, encapsulating database table rows as objects which handle tasks such as resolving linked records and generating HTML input fields semi-transparently. It also offers a comprehensive administration system for any database whose schema adheres to certain idioms, without further customisation. The present implementation of JAL is held to suffer from

- lack of clarity, both in its overall structure and in its line-by-line coding, arising from its status as a prototype which was developed quickly as a way of discovering what functionality would turn out to be useful
- failure to take advantage of the Java type system: table rows look like (and in fact are) `Hashtable`s
- inability to support cacheing of data in the Java process in a way which guarantees consistency when used in combination with SQL transactions

It is felt that rewriting JAL in a way which preserves its strengths while eliminating these weaknesses will yield a more maintainable product, and shorten the development cycle of projects which use it.

## Users

The users who are going to interact with the system fall into the following classes:

Paneris programmers. Paneris programmers will use Melati as the core infrastructure for constructing most if not all of their customers' web sites.

The open source community. It is hoped that Melati will become popular among open-source developers outside Paneris. This will bring Paneris the familiar benefits of `free' debugging and further development, and, to Paneris's customers, the reassurance of (free) dual-sourcing.

Customers' end users. End users will interact directly with Melati, as they do with JAL, through its login screen (part of the access control facility).

Customers' site administrators. Melati will preserve JAL's very powerful generic database administration system, which allows all but the most domain-specific administration tasks to be carried out remotely through a web interface.

Customers' installers. These are the people who install Melati-based software on a fresh machine.

## Services

The services the system will offer its users are:

### Paneris programmers

#### Data structure definition

A facility for defining the database schema, including linked fields (`one to one' and `one to many' relationships) and field display preferences, in a convenient notation perhaps similar to that of Java class definitions. It is important that this should be treated as a minimum definition: administrators should be able to add other tables and fields, so that they can be used in templates and the generic administration system, directly over the web interface.

In the long run, it might be nice to allow inheritance relationships between tables to be supported automatically (but not by exploiting the Postgres inheritance facility, for portability reasons).

#### `Persistent object' mechanism

Database objects returned to the programmer via the Melati API should appear to her as Java objects with type-safe get/set method pairs corresponding to the fields given in the data structure definition. All fields, including extra ones defined dynamically by administrators etc., should also be accessible by their (string) names, to facilitate the writing of generic code; in particular, it must be possible to iterate over a persistent object's fields in a webmacro template (e.g. that for a generic data entry screen).

#### Object specialisation

The programmer should be able to add domain-specific `convenience methods' to the bare persistent objects.

#### Retrieval and searching

Explicit retrieval of linked records should not be necessary: the programmer should be able to treat the records as normally linked Java objects.

The API should provide for any kind of search to be performed, if necessary by means of arbitrary `WHERE` clauses. The special case of retrieval by a canonical ID field should be made trivial (but it must not be a necessary that this field is actually called `id`).

In general, the basic unit of retrieval should be the record/object. However, it must be possible to execute arbitrary SQL `SELECT`s returning a selective join of fields---on the understanding that this will bypass the cache.

It may also be worth thinking about providing an API for retrieving only some of only some of an object's fields (the remainder being fetched on demand), and, conversely, for pre-fetching linked objects using a single call to the backend.

#### Transactions

At least a simple kind of `transactional' capability must be provided: e.g. commit/rollback of all changes made to the database during the processing of a particular HTTP request.

#### Cacheing

The programmer should be able to control the manner in which Melati caches data from the RDBMS: provisionally, the maximum size of the cache for each table, and perhaps whether the results of each individual operation which might trigger a `SELECT` should be cached.

The cacheing mechanism must interact consistently with SQL transactions. If a cached row is changed through a non-autocommitting connection, the change must be visible through that connection only, and must revert correctly if the transaction is rolled back.

#### Internationalisation

Melati should provide support for convenient internationalisation: language-specific table columns should be localised transparently or semi-transparently.

#### Access control

It must be possible to enforce restrictions which ensure that only groups of named users (as determined by knowledge of a password to go with their username) are able to access certain services, defined on a per-record basis, on a Melati-based site. For the programmer's convenience and to avoid security bugs creeping in due to programmer oversight, the checks implementing access control should be made as close to the data as possible, so that in principle all she has to worry about is passing in the right `user' object or authentication token (once only). There should also be a way for her to perform specific checks of her own if she wishes to catch authentication-failure situations before wasting any of the user's time.

For similar reasons of convenience and robustness, Melati should allow the programmer to define access control policies---for instance, one to the effect that an invoice should be visible both to its issuing party and to its receiving party. (But arbitrary, per-row permissions setting must still be possible for the administrator.)

The system should support the definition of user groups (and of group-groups, etc.) for use in authentication rules.

Basic access levels to table records should include `none', `read only' and `read-write'.

#### Markup generation for templates

There should be a simple and unintrusive way of placing markup implementing views of and data entry fields for objects in a webmacro template, and of reading fields back into objects when they return from the user.

This facility should be markup-language neutral, so that we can as a minimum write templates in any of HTML, XML and WML.

Javascript validation checks should be added automatically or semi-automatically to input fields where appropriate.

### The open source community

The services provided for Paneris programmers must be sufficiently accessible---easy to grasp, convenient to use, well documented---for people from the wider open source community to want to use them too.

### End users

#### Login

End users must be able to log in to Melati-based sites for authentication purposes, either on demand (when they attempt to do something for which authentication is required) or explicitly. They should also be able to logout or re-login under a different username at any time. It would be nice if they had the option of storing their logged-in-ness across browser sessions in (e.g.) a cookie (but cookies must not be forced on them).

#### Password management

Melati must provide a facility for reminding users of their password, by some reasonably secure means, if they have forgotten it, and allow them to change it at will.

### Administrators

#### Database viewing/editing

Melati must support the functionality of JAL's current generic administration system: briefly,

- list tables
- add/delete tables
- list records in a table matching a template
- add/edit/delete a record
- list records linked to (or linking to) a record
- change the capabilities required to read/write a record

For an example of JAL's admin system in action, see: `http://paneris.org/webmacro/org.paneris.jal.controller.Admin?db=paneris&action=tables`

#### Localisation

The admin system should support localisation of an internationalised database in some convenient form.

### Installers

#### Database creation/validation

At the request of the installer, a Melati-based site should be able to (re-)create and initialise a fresh database so that this doesn't have to be done manually.

When a Melati-based site starts up, it should be able to verify that the database schema in the RDBMS is safely consistent with the program's idea of what should be there, and fail with an error message if it isn't.

## Constraints

### Platform

Melati is targetted to run on a Linux machine (2.2 and up) as a Java process (1.1.7 and up) using Postgres as a backend RDBMS (6.5 and up) and webmacro for the templates. We should feel confident that we can replace Postgres with any other full-featured SQL RDBMS for all Melati-based sites inside a week at most. For the benefit of the wider open-source community, it's important that Melati should ultimately be portable to NT4/W2K plus SQL Server (for developers as well as for running live sites).

It is acceptable that no other processes should be allowed write access to the database in the backend RDBMS while Melati is using it (apart from those run by hacker-administrators who know what they are doing; without this restriction, cacheing makes no sense, and Melati will have continually to check that the database schema is not changing under its feet).

### Performance

Melati-based sites should be able to handle HTTP requests for simple templates including linked records at a rate of twenty hits per second. This should suffice for all applications except heavy-duty ones for which the tradeoff point between development effort/maintainability and implementation simplicity/speed falls unusually far towards the latter.

### Security

Melati-based sites should offer reasonable security: we should feel happy about allowing it to control access to user preferences, messageboards etc., but facilities for spending money and deeply private information will have to be protected by an external mechanism.

### Robustness

Given the consistency of Melati's cacheing mechanism, and low-level data integrity checks, the site's robustness should be limited only by that of the RDBMS, and of course the site-specific code.

We may want to look into running a watchdog process which restarts `httpd` and everything else every minute or two if the system fails a test transaction, and sends a warning email or whatever.

### Stability

It should be very surprising and worrying for a Melati/Postgres/Linux site to go down.

## Risk analysis

It's unlikely that the requirements proposed in the present document, once reviewed, will turn out to be seriously wrong, because JAL and the sites developed using it provide an excellent case study of what is needed and what can in practice be supported.

The main doubt relates to the data structure definition: it may be that a simple `language' turns out to be too inflexible to map onto the kind of `CREATE` `TABLE` we need in complicated cases.

## About this document

### Authors

William Chesters <williamc@paneris.org>

Most recent CVS $Author: timp $ @paneris.org

### Quality

The current quality level of this document is: Beta.

### Readership and purpose

- The customer should feel confident that the work will be useful, and understand its scope, if this is considered relevant: No, it's us.
- The project leader should feel happy about taking responsibility for leading the spec of the system.
- The developers should feel informed about the broad picture surrounding what each of them is doing.
- Future maintainers should be able to understand the motivation behind the system so that it falls into place easily for them.

This document is also intended to give the wider open source community an insight into the motivation behind Melati, so that they can give us feedback pre-release and understand the finished product.

### History

The important points in the life of this document are listed below (for everyday update information, consult its CVS log).

Date | Event
--- | ---
(not yet) | Certified at release quality level by ...

The CVS log for this document is:
$Log: Requirements.html,v $
Revision 1.1 2005/11/21 22:01:49 timp
Moved from site/doc
Revision 1.12 2000/02/29 09:53:02 williamc
Finish recovering from disaster; point out in the docs that you can 'add methods to table rows'
Revision 1.2 2000/02/04 18:28:34 williamc
Add QA stub; explain DSD-admin interactino a little better