Cheshire is a client/server information retrieval system that brings
modern information retrieval techniques to a wide array of data domains.
Cheshire provides uniform document storage in the form of SGML, supports
probabilistic search, and supports Z39.50 interoperability with dozens
of library information systems around the world.
Cheshire is now several years old. Its program logic is coded entirely
in C and most of its user interface is done in Tcl/Tk. Further development
is being inhibited by a system complexity that has outgrown its original
design and its dated software technologies. New technologies are now available
that can dramatically reduce coding effort and enhance robustness, maintainability,
and interoperability. This paper proposes to re-engineer Cheshire into
a modern software system, with the hope of ensuring its future viability
as a platform for information retrieval research. The next section outlines
the system objectives of a next generation system. Section 3 discusses
the technologies available for meeting those objectives. Section 4 proposes
a modular reorganization of Cheshire functionality into packages. Section
5 discusses the directory structure for deployment. Section 6 examines
issues in migrating the existing system to the new design. Section 7 concludes.
2. Design Objectives
To continue Cheshire's viability as a research platform, the system
must appeal to users and developers alike. It must satisfy the information
needs of users, and it must also make it easy for developers to modify
and experiment with the system. I identify below seven sets of features
I would like to see in a next generation Cheshire.
Distributed Queries. We want Cheshire to be able to satisfy a user's
cross-domain information need. To do that, a Cheshire server needs to look
at not only the database it maintains, but also other information sources
reachable over the Internet. Each server needs access to meta information
about other information sources, from which it can decide whether to query
that particular source to satisfy a particular information need.
Interoperability. To maximize its domain reach, Cheshire systems
need to not only interoperate with each other, but with as many different
types of systems as possible. Cheshire needs to support international standards
such as Z39.50, emerging protocols such as SDLIP, and be ready to adopt
Concurrency, Scalability and Robustness. A research system is most
useful if the results of good research can be immediately deployed to serve
a large number of users. Cheshire should be able to efficiently use machine
resources to serve concurrent requests, grow linearly in throughput as
more hardware resources are added, and offer reliable operations suitable
for academic environments.
Web-Based System Adminitration. The system should be easy to deploy
and easy to administer. The administrator should be presented with
a unified view of system operations and given easy means to customize them.
A web-based administration tool will give administrators the most flexibility
in accessing the system.
Dynamic Databases. An administrator should be able to incrementally
grow the database without interrupting user services. The database should
be automatically indexed as it is grown.
Simplified, Structured, Maintainable Code. A research platform exists
to serve innovation. It is constantly evolving as new ideas are tested
and old ideas discarded. Developers come and go. This process becomes more
vibrant if the system is accessible to new developers and is amenable to
Unprivileged Deployment. A research system is a toy for everyone.
Its users will not always have privileged access to the operating system.
Cheshire should not require such access for deployment.
3. New Technologies
A number of new software technologies exist today that can help achieve
the objectives outlined in the previous section.
All of the technologies listed above are free. Source code is available
for most of them.
Java Programming Language. Java has
taken the computing world by storm. It is elegant in design, easy to learn,
and a pleasure to use. A developer writes Java code once, and it
is instantly portable to almost every platform. Java is strongly typed
and object oriented from the ground up, giving us programs that are more
robust, more maintainable, and more expandable. Networking is at the core
of Java. Everything from naming, to its execution environment, to API design
is tailored to networked computing. A rich set of reusable Java components
is available to provide infrastructure support, and Java integration is
available for nearly every major language. Java performance is more than
adequate to provide the systems glue, with performance critical components
written in other languages. Java 1.3 promises to provide significant improvements
in client-side performance. Java tools are mostly free. Java programmers
are easy to find. And as one of the most successful computing platforms
of our era, users can expect continued expansions, upgrades, and community
Remote Method Invocation makes it extremely simple to implement client/server
communications. A method on a remote object is called exactly the same
as a local method. The network is nearly invisible.
Java HotSpot Server
(V2.0 available for Solaris and Linux in fall, 2000). High performance
multithreaded Virtual Machine for server applications. Can be used in Cheshire
for high performance concurrency.
Berkeley DB 3.0.55. Better concurrency
support. Java API included.
A Java architecture for generating dynamic content, to be delivered through
standard web servers. We can use this to build a web interface for Cheshire.
More and more thin clients such as PDA's will have web browsing capability.
Cheshire may see radically different kinds of use in the future. For example,
a book store patron may want to check on his PalmPilot to find out if an
expensive book is available at a public library and if yes, immediately
make a time limited reservation.
DOM Compliant XML Parsers. Available
with Java API's. Conform to the DOM standard for efficient (fewer passes)
parsing of XML.
Forte Integrated Development
Environment. A Java IDE that includes an integrated debugger, GUI access
to object tree, etc.
Java Naming and
Directory Interface Comes with a reference directory service. Cheshire
servers connected to the Internet can discover each other through this
service, allowing them to cooperate both in the local area and in the wide
A simple information retrieval protocol available with Java transport and
CORBA and HTTP bindings. SDLIP is simple to understand and elegant in design.
SDLIP is far simpler to implement than Z39.50 and may find more support
from a wider array of information sources, particularly from the fast evolving
web search engines. The simplicity of SDLIP can encourage experimentation
in user interface, server architecture, and retrieval algorithms. SDLIP
may reduce the barrier to entry for distributed library information systems
the way the web reduced barriers to publishing. The web paradigm has been
a move away from structure, away from careful selection and long range
planning, and a move toward universal, low barrier access while powerful
computers mine structure from information where its human makers were spared
from the effort.
After some examination, I propose to not adopt the following technologies.
Java Enterprise Edition Emphasizes
support for enterprise computing needs such as reliable operations and
security. Includes Enterprise Java Beans component infrastructure. EJB
provides component support services such as life cycle management, persistance,
and load balancing. I decided against using this technology in Cheshire.
Most of its power is not needed for Cheshire and EJB adds unnecessary complexity
to the development process, which may in the end inhibit experimentation
in a research system rather than promote it.
CORBA CORBA is a distributed objects
architecture used for building distributed systems. In the foreseeable
future, SDLIP already provides all the server-to-server interoperability
Cheshire will need. In fact, SDLIP itself in a distributed information
systems architecture that sits on top of CORBA. For our purposes, SDLIP
interoperability subsumes CORBA interoperability. At the present, there
isn't a clear need to write distributed objects that sit below SDLIP. Even
if that need arises in the future, it may be simpler to convert the object
interfaces to RMI, which provides a distributed architecture native to
the Java language.
4. Source Code Organization
Java provides a "package" based object naming system in which the location
of a file in the directory tree, in either source or object form, corresponds
to its package name. The idea is that objects with like functionality would
be grouped together in a package. These objects have privileged access
to each other's non-private and non-protected fields and methods, whereas
only public fields and methods are accessible to objects in different packages.
Thus, the "package" provides a level of encapsulation higher than the "class".
A package is a collection of classes that do similar work. Packages themselves
can be grouped into hierarchies.
For a system as complex as Cheshire, a package oriented organization
can be very useful. It makes the broader purpose of a class self documenting,
while at the same time encouraging the developer to not straddle functionality
that should belong to different classes with a single class. The diagram
below gives the proposed modular view of the new Cheshire, with the modules
named by the packages that will implement them. A module contained in another
module is a subpackage of the container package. Arrows indicate data flow.
Cheshire packages should be rooted at edu.berkeley.cheshire
to ensure their uniqueness across the Internet. Symbolic links and aliases
can be used to get around any inconveniences caused by long package names.
edu.berkeley prefix was omitted in the above diagram for display
considerations. The packages are described below.
5. Deployment Organization
edu.berkeley.cheshire.document Contains access classes
to the document collection. Uses javax.xml.parsers routines for
XML parsing and various utility functions in edu.berkeley.cheshire.util
for format conversions and parsing of other data types.
edu.berkeley.cheshire.document.index Contains record index
classes for the document collection. Since a file may contain multiple
records (or documents), and records may be of varying length, a record
index allows random access to a named record.
edu.berkeley.cheshire.index Contains indexing classes that
provide key-based access to document collection. Uses com.sleepycat.db
for the underlying database that stores the index.
edu.berkeley.cheshire.index.util Various utilities for
examining the contents of an index, recovering from an incomplete index
edu.berkeley.cheshire.retrieval Contains classes used for
query planning, query traversal, result set merging, result set storage,
edu.berkeley.cheshire.boolean Contains classes used only
during boolean search.
edu.berkeley.cheshire.probabilistic Contains classes used
only during probabilistic search.
edu.berkeley.cheshire.interface Contains the multi-threaded
Cheshire server. Its threads are used to execute user queries as well as
administrative updates. Uses classes in SDLIP, Z39.50 and JSP subpackages
to translate protocol requests into an internal query plan representation.
In this context, a user may be a Cheshire client or another Cheshire server
executing a distributed query. Indexes are also built using this interface.
is the only access point into the Cheshire system for both users and administrators.
edu.berkeley.cheshire.interface.sdlip Contains classes
used by the Cheshire server to provide SDLIP service. Uses the edu.stanford.sdlip
edu.berkeley.cheshire.interface.z3950 Contains classes
used by the Cheshire server to provide Z39.50 service. May use legacy C
modules for Z39.50 functionality.
edu.berkeley.cheshire.interface.jsp Contains Java Servlet
classes that create dynamic JSP content to be delivered by an external
edu.berkeley.cheshire.interface.admin Contains classes
used by the Cheshire server to provide the administrative interface. Uses
a private RMI based protocol.
edu.berkeley.cheshire.interface.jndi Interface to JNDI
services for discovering other Cheshire servers on the Internet. Also used
to make its own services known to other servers.
edu.berkeley.cheshire.client.user A Java Applet user interface
for submitting search requests. May locate servers from both a configuration
file and JNDI services
edu.berkeley.cheshire.client.user.sdlip Implements the
SDLIP protocol, using edu.stanford.sdlip.
edu.berkeley.cheshire.client.user.z3950 Implements the
Z39.50 protocol. May use legacy C modules.
edu.berkeley.cheshire.client.user.jndi Interface for discovering
Cheshire servers on the Internet.
edu.berkeley.cheshire.client.admin A Java Applet for system
adminitration. A command line interface is also available. Uses a private
RMI based protocol. Indexing is done through this interface. If the adminitrative
client shares a file system with the Cheshire server, document collections
may be specified as local files. Otherwise, document URL's may be given.
edu.berkeley.cheshire.util Contains data structures and
utilities shared across packages that do not have a natural primary home.
For example, classes in the retrieval package uses document
package classes. The classes used to access document databases, even though
shared between two packages, have a natural home in document and
thus do not belong here.
edu.berkeley.cheshire.util.log Utility classes for logging
edu.berkeley.cheshire.util.sgml Parsing or conversion routines.
edu.berkeley.cheshire.util.marc Parsing or conversion routines.
edu.berkeley.cheshire.util.z3950 Z39.50 routines shared
by server and client
javax.xml.parsers DOM compliant XML parsing package.
edu.stanford.sdlip SDLIP Java transport from stanford.
com.sleepycat.db Java API for Berkeley DB embedded database
Using standard UNIX naming conventions for directories,I propose the
following directory tree for deployment.
This deployment scheme does not require root access.
$CHESHIRE_HOME/bin Scripts for starting servers or clients.
$CHESHIRE_HOME/lib JAR files and native libraries.
$CHESHIRE_HOME/src Root of source tree.
$CHESHIRE_HOME/doc System documentation. javadoc source code documentation.
$CHESHIRE_HOME/etc Default configurations files for servers and
clients. Configuration files in users' home directories may override the
options set here.
$CHESHIRE_HOME/var/log System log files.
$CHESHIRE_HOME/var/pool Berkeley DB shared buffer pool backing
6. Migration Path
Having outlined a redesign of the Cheshire system using completely new
technologies, some important questions now need to be answered. What is
the right implementation strategy? Do we integrate the new technologies
and new source code with the existing system one piece at a time, or do
we build the system completely from scratch all at once?
The integration approach would be done from the top down. The top down
approach says first we implement the glue that holds the system together
by building interfaces to legacy components, then we replace those components
one at a time. The opposite, bottoms up approach makes little sense for
our situation; we would see none of the benefits of the new technology
at the frontend until the entire system is complete. The integration approach
has the benefit of quickly realizing the benefits of modern server technologies
in a deployable system. It also forces us to consider the full set of supporting
interfaces as each new interface is implemented, reducing the chance that
important functionalities become incompatible with the new interface.
The integration approach has two disadvantages. The first is that integrating
legacy components at each step of the way forces us to deal with the old
architecture, and will not fully benefit from the cleanness and consistency
of a new design. The second is the enormous amount of programming effort
that will be required to devise, implement and test these interfaces. This
author is of the opinion that these two disadvantages far outweight the
benefits of the integration approach. The existing Cheshire system is usable
and there isn't an immediate need for a new server. An interface that doesn't
work can always be redesigned. And such a redesigned interface will still
be more self-consistent and would have required less implementation effort
than an interface built on integration with legacy components at every
step of the way.
Rebuilding the system from scratch does seem to be the more sensible
approach. Due to the complexity of the Cheshire system, however, it will
be difficult to sustain a happy development effort if we go on to build
a massive system and see nothing working whatsoever for three months. I
propose that we take a mini-system-evolves-to-full-system approach. We
start by building a simple multi-threaded, boolean search, SDLIP service,
written all from scratch. From there we expand to where Cheshire is today,
and beyond. Under this approach, it is important to have a carefully designed
skeleton structure established up-front, as I believe I have done in the
previous sections. Each piece of new code may be solving a tiny piece of
the puzzle, but with the skeleton for reference, we always know where we
are and what our purpose is.
I have proposed a re-design of the Cheshire information retrieval system.
While the design does not in itself achieve the systems objectives outlined
in section 2, it does achieve the architectural objective of accommodating
all seven of those system objectives. We enable distributed queries through
a clean inter-server discovery and search interface. We style toward multiple
interfaces and extensibility from the grounded up, with clean plug-ins
for interoperability. We take advantage of Java and Java tools to write
maintainable, concurrent, high performance code. Java web technologies
enable a broad base of clients and allow Cheshire to be searched and administered
from anywhere, without having to install software locally. The groundwork
has been laid for a next generation platform for information retrieval.