PsiStack

Technical Manual

DRAFT

Version 0.06a

© 1997-1998 PsiStack Development Group

Table Of Contents


1 Introduction

This manual is intended for programmers working on the stack software. It contains design and implementation notes on the stack. You may also want to read the SDK guide.

The material in here is not for the average user. For people who just want to turn on, tune in and surf, read the Owner's Manual. That document is intended to answer all your questions about the stack - you shouldn't have to come to this document to have your queries answered. If you do find something lacking from these documents, please email me, matt@gumbley.demon.co.uk and I'll see if I can improve them.

The stack is based on the model given in Doug Comer's "Internetworking with TCP/IP" Vol II. Details are in the references and bibliography.

For more up-to-date information on where we're currently at with development, please see the project web pages, at http://www.gumbley.demon.co.uk/psistack.html


2 Analysis

Initially, we thought we could base this on Phil Karn's KA9Q networking package. This is a monolithic program which provides a TCP/IP stack, and many popular applications. It has been ported to many different platforms, and supports most of the popular TCP/IP based protocols, providing both clients and servers for many of them. Unfortunately, there is no single version of KA9Q we could use as a base. We needed to find a version which supports many protocols and platforms to ensure we make a timely and portable product. Unfortunately, the monolithic nature of KA9Q does present difficulties, and makes the product more complex.
In a single program, it can be difficult to handle events when a timer for one protocol expires while the program is executing code for another protocol. If the system uses a separate process to implement each protocol that requires a timeout, the process only needs to handle timeout events related to its protocol. Thus, the code in each process is easier to understand and less prone to errors." [Comer 1994]
So, we have started developing a new TCP/IP stack, taking the multitasking nature of EPOC into account from the start. We are also examining some of the routines from KA9Q and [Comer 1994] to see how certain problems were tackled. Our stack takes a similar structure, incorporating several 'processes' in one real process. This makes the code more complex, but as noted in RFC 817, the scheduling of the various processes which may comprise a fully modular TCP/IP implementation would cause a performance hit. So, there's just one process, which handles all the TCP/IP traffic.

The networking subsystem is split into separate processes: there's a process for the TCP/IP system, one providing debugging support, a configuration program, and a database manager. In this way, the TCP/IP stack gets the maximum amount of memory available for handling network data, and the database can have a data segment all to itself. The console and setup programs can be removed when the system is fully working.


3 Statement of Purpose

These are the requirements and specific exclusions for the TCP/IP stack only. We'll fulfil them gradually.

RFC Compliance PsiStack will be compliant with the appropriate RFC's. We diverge from them in the following ways:
IP options
Option processing is unimplemented yet, although RFC 1122 says we MUST support at least source routing. This is not normally used, and most firewalls would ignore source routed frames for security reasons, so I'm not spending time doing this..
More slight exclusions are expected as the development progresses.
Routing Routing is straightforward: there is a single network interface. Data destined for the outside world goes out from this interface, and only datagrams addressed to this interface (and broadcasts) are accepted by it. The original idea was to allow the Psion to be used as a multi-homed host, so you could attach a modem to the Psion, an IrDA device to your PC, and use your Psion as a gateway to the Internet via IrDA. Both devices could be available simultaneously. There would be no gateway code - to use the Psion to pass data "through", the two interfaces must be on separate networks. However, due to the nature of the hardware in the Psion, it is not possible to do this. Therefore, PsiStack (as of version 0.05) provides only a single network interface.
Interfaces The local pseudo-interface, serial port and IrDA devices will be the only network interfaces. Only one serial device can be used at a time.
Data rate Serial handling will provide up to 57600 baud on the RS232 port and 115200 baud on the IrDA device. 19200 is the highest speed on a Series 3a. 19200 is the fastest speed on a Siena's IrDA link.
RS232 Control Either XON/XOFF or hardware (RTS/CTS) handshaking, with flexible control for detecting or ignoring other RS232 signals such as DTR/DSR/DCD.
Link protocols SLIP, CSLIP and PPP will be the only data link protocols. We will use these over the IrDA device, treating it as a serial device, and let IP handle any errors. These protocols will be provided in the order:
  • SLIP
  • PPP
  • CSLIP
SLIP/PPP "devices" do not have addresses (they use IP addressing); there is no need for ARP.
Other protocols ICMP, IPv4 with dynamic and static addressing, TCP, UDP, DNS name resolution. Later releases may have IP with provision for mobile connections and IPv6. (Volunteers for IPv6?) We're not booting the Psion over the network, so there is no need for RARP or BOOTP. Multicast IP will not be supported.
Programming interface The top layer of the stack will be exposed via the Sockets IPC messages, which give access to the entry points for the stack itself. This is very low-level, and so several APIs will be provided, in the following order of importance:
  • Berkeley (BSD) sockets
  • An API for the Psion-Java project's java.net classes
  • Psion's version of BSD sockets (for compatibility)
It may not be possible to add a PsiMail Internet compatibility layer - this is nowhere near the top of our priority list though :-)
APIs are also available for accessing the database manager, diagnostic console, and low-level stack control.
Modem & ISP support Connection to your ISP will be aided by a powerful dial-up connection script language. Scripts for connection to popular ISPs HAYES modems will be provided, although the package will work with non-HAYES modems, and will connect to "strange" ISPs.
Memory & disk usage The software will be pure small-model, and adhere to Psion's register conventions. It will fit in 64KB code, 64KB data. It will fit on the internal RAM disk of a 1MB Series 3c (providing no large applications are installed, and the user has modest application data stored there.)
Configuration storage All user settings (IP addresses, port settings, ISP details) and stack internal settings will be managed by a database management program. The format of the database files are thus easily changeable, if the system needs to be more flexible than we originaly envisage. ISPs and hardware vendors will be able to and provide configuration files and dial-up scripts which are tailored to access their systems. The configuration will be lockable to prevent users tampering.


4 Design

4.1 TCP/IP stack kernel design

The basic design of the TCP/IP kernel comes from the model in [Comer]. This has been modified to place all the TCP/IP functionality into a single process - against Comer's advice, but giving us a slight performance benefit. On a small, low-powered system such as EPOC 16, we need all the power we can get. To this end, those compnents which may not be needed on a "live" system, such as debugging support and setup may be removed when finished setting up.

We also extend Comer's model to provide a dial-up scripting language, a database for storing configuration information, and to make it more Psion-like.

The diagram below shows the basic design of the stack so far. The stack currently only has these "lower layers" of the TCP/IP protocol implemented. In terms of the ISO seven-layer model, we have the physical layer (RS232/IrDA), data link layer (SLIP/PPP), network layer (IP) almost complete. (IP datagram reassembly is not finished). The ICMP and UDP routines are done as well. TCP is coming along slowly.

4.1.1 An illustration of stack data flows

To explain how the system works so far, let us consider a stack running on a Psion - connected to a Linux box (or other Internet host). This is how I do my testing: run the native Linux TCP/IP software, with /dev/ttyS0 running SLIP. The Psion stack code talks to TTY:A, and the two ports are wired together using the PsiWin cable. In essence, the stack is connected to another host, and its network interface is "up".

The Linux machine pings the stack. This means that an ICMP Echo Request datagram is created on the Linux host, fragmented such that each fragment is less than the MTU for the SLIP interface, and sent to the SLIP device, on /dev/ttyS0. These SLIP-encoded frames go over the PsiWin cable, to the stack.

The network interface receives data from its input device. The stack receives bytes on its serial port, TTY:A and adds them to its buildup buffer. Function slip_in_process in slip.c does this. When a frame has been received, it is passed to the network interface, which queues it on its IP input queue, ni_ipinq. Defined in nif.h.

The IP layer detects a datagram on the network interface. Assuming the datagram is valid, IP sees that its destination IP address matches that of this network interface, and so routes the datagram by placing it on the Pseudo-Interface's output queue. NB: for the pseudo-interface, input and output are swapped. So, a datagram placed on the output queue is actually an input to the higher layers of the stack. Similarly, a datagram placed on the input queue is actually output from the higher layers of the stack.

That's all IP does - route datagrams between "up" interfaces. The pseudo-interface is always "up". IP routes incoming datagrams to the pseudo-interface, and retrieves the higher layer's output from the pseudo-interface, and routes it out to the network interface.

The network interface generates a signal when it has data to process. The main event loop wakes up when it sees data, and processes it. The network interface is processed as described above, but the pseudo-interface is special. Datagrams are placed on its input queue by higher layers when they output. IP takes care of these datagrams as described above. When the pseudo-interface is polled, any datagrams on its output queue are checked to see if they are fragments, and if so, sent to the reassembly routines. Any non-fragments (and reassembled fragments) are checked to determine which higher layer protocol should receive them, and they are dispatched directly to that protocol handler.

The ICMP echo request fragments (if there are any fragments) are reassembled, and passed on to the ICMP layer.

ICMP sees the echo request, and "turns the datagram around" - swaps the source and destination IP addresses, changes the datagram to an ICMP echo reply, and places it on the pseudo-interface's input queue.

IP will eventually discover this, and route the datagram back to the network interface from whence it came.

The network interface will detect this, and send the datagram out, fragmenting it as necessary to fit below the MTU.

The Linux box will see the echo reply, and that's all there is to it!

4.2 Diagnostic facilities

The console is a separate process. It provides a screen and file logging for debugging messages. It isn't necessary to provide this in final user installations. The stack and database will use it if it is loaded, otherwise, their debugging messages vanish into the ether. The advantages of using a separate console is that it cuts down the code in the stack and database, moves the burden of file logging and logging verbosity off the stack and database, cuts down on the size of an installation, and removes one or two nasty deadlocks. For example, if the setup program, or your client application were providing console facilities, and you cause the stack or database to do some operation which generates console messages, if you are not in a position to receive and process console messages at that time (say you've presented a dialog, and you're out of your normal message loop), then everything could halt.

The console alleviates this latter problem. However, applications must still be aware of stack events - there is still some onus on the application writer to always be in a position to process stack event notification messages.

To use the console, link your client application with the console link library, and use the functions outlined in psistack\console.h. The console can handle several clients simultaneously.

4.3 Configuration managment

Similarly, the management of PsiStack's configuration is handled by a separate process. It manages several tables for the stack kernel and setup program, and is accessed by a simple API. The stack kernel starts the DBMS process as soon as it starts, and stops it just before it stops itself.

4.3.1 The database model

Note: As of version 0.05, we're still using flat binary files.

4.3.1.1 Some false starts...

4.3.1.1.1 Flat binary files
Bad Idea I: The database used in early versions of the software used flat binary files, and the stack user had to manually configure everything. OK for the casual hacker, but not suitable for a scalable, powerful system for the professional who must be connected anywhere in the world.
4.3.1.1.2 Name-Value pairs
Bad Idea II: Next came Matt's idea of storing all the configuration information in a single file, in name=value pair format. This was a really, really terrible idea. It means you have to address each cell of your table individually. In the Windows .ini file format, each tuple has a different [section], allowing one tuple in each [section], and each attribute has to be explicitly referenced. Each tuple has to have a [section], and each attribute in every tuple is referenced explicitly. When putting all such tables into a single file, you have to add a table reference to each cell, by encoding the table name with the tuple identifiers. Essentially, it was an attempt to compress several tables, each with arbitrary rows and columns, into one three-column, zillion-row table. We really need proper database tables.

4.3.1.2 Multiple managed tables: Methods, Host Connects, Virtual Tables

The Host Connect Table / Method Table system was proposed by Greg Smith. The use of virtual tables for ISP, and devices is intended to ease installation of such information. (Users merely copy the relevant files onto their Psion, and start the system) Repeating groups (as could be found in the ISP services and method phone numbers) have been replaced by separate tables (ISV and PHN).

See the implementation section for an explanation of virtual tables - think of them as normal relational database tables. They aren't, well, they are… sort of…

4.3.1.2.1 List of tables in the PsiStack database
There are 11 tables in the PsiStack database:
Table NameFunction
ISP (Virtual)Holds general information about each ISP, and default account information. ISPs can be direct connections to TCP/IP systems, not just modem links to a commercial ISP.
PHN (Virtual)Holds all phone numbers for ISPs. Direct connections don't have any.
SRV/etc/services
PRT/etc/protocols
HST/etc/hosts
ISV (Virtual)Services and port numbers provided by each ISP.
MPHLists of phone numbers used by each method. Actually, foreign keys into the method table and phone numbers tables.
HCTThe host connect table; two ordered lists of foreign keys into the method table. One list is the auto-connect list; the other is the prompt-user list.
METThe method table; a list of possible ways of connecting to the Internet (or an internet), linking ISPs and devices. Each method has an optional list of phone numbers associated with it, stored in the MPH table.
DEV (Virtual)Devices. All the serial characteristics of the direct connections and modems in use.
SYSVarious name/value pairs of miscellaneous configuration information used by the system.
4.3.1.2.2 Entity-Relationship diagram
The database schema is shown here in an Entity-Relationship diagram:

Key to attributes:

Underlined: Primary Key for this table

Italicised: Foreign Key for some other table (TABLE.Key-Name)

(May add per-method IP addresses, DNS addresses, GW addresses?)

4.3.1.2.3 An illustration of the Host Connect Table / Method Table
The currently proposed system for setting up, and changing access number is as follows.
  1. You install your ISP's data files. There are three: a .ISP file, containing general details of your ISP. A .PHN file, containing all their access numbers, and a .ISV file, containing a list of all the services they provide (in the /etc/services sense of the word 'services'). These three files are all DBF files, and all start with the name of the ISP. E.g. DEMON.ISP, DEMON.PHN, DEMON.ISV. (These form virtual tables… see later)
  2. You install your device configuration files. These are .DEV files, and are DBF format. These may also be created manually using the setup program.
  3. You create one or methods for accessing your ISP. To do this, you select a) which device to use b) which ISP and c) any phone numbers if necessary. You may also change your user name, password, data link protocol and script for each method. Defaults for these are taken from the ISP configuration files.
  4. Add the methods you want to either the auto-connect list or the prompt-user list.

When PsiStack needs to connect to your ISP, it will cycle through the host connect entries for that ISP (either automatically, using the auto-connect list, if it exists, or under user control, using the prompt-user list, if it exists), attempting to connect with each one, in the order you specify, until a connection is successful. (Does auto-connect take priority over prompt-user? Should the user be able to select which to use in the case that both lists exist?)

If you want to change access number frequently, or want more control over which method table entry is used to connect, add a number of 'popular' methods to the 'prompt user' list. You'll then be prompted to select a method table entry when you want to connect.

This requires you to do all the setup of your 'prompt user' list before you fly off to the far corners of the earth. Once it's set up, you just select which method table entry you want (give them names, like "Connection to Demon Internet via the Stoke-on-Trent POP, using SLIP" and "Connection to Demon Internet via the London POP, using PPP, with an MTU of 76 bytes")

4.3.1.3 Summary


5 Implementation

Perhaps this information should go into the SDK? A client writer would need to know this kinda thang...

5.1 How the multiple processes fit together

The stack takes the form of a single process, which handles everything from the RS232 signals through the TCP/UDP layer to the sockets and stack IPC message layer. This is shown in the picture as STACK.IMG.

To set-up the stack (for instance, setting static IP addresses, ISP phone numbers, dial-up script settings, etc.), there is a user interface, shown here as SETUP.APP.

User applications are statically linked with the sockets library (which in turn uses functions in the stack library). The stack is loaded and initialised by the first sockets library call, or by launching the stack setup user interface program. This is done by the stack loader calls of the stack link library.

Configuration and user settings are managed by the database manager, shown here as DBMS.IMG. Communication with the DBMS is achieved by use of calls from the DBMS link library.

Logging of diagnostic messages is performed by the diagnostic console program, CONSOLE.APP. Communication with the console is achieved by use of calls from the console link library.

5.2 IPC messaging between the processes

All communication between the stack, user interface, database and console is performed using IPC messages. The programmer of client applications does not need to know this - the format of these messages is only of interest to programmers wishing to communicate with the stack from languages other than those currently supported. The format of the IPC messages is convered in an appendix. The library calls which wrap around the IPC messages are described in subsequent sections.

5.3 Supported APIs

To interface your client software with PsiStack, include the relevant C header files, and link with the single PSISTACK.LIB library. Details of this are provided in the SDK. The link library comprises several other libraries:

High level functions

Low level functions The use of these libraries will now be discussed.

5.3.1 BSD Sockets Library

For client software development, to gain the maximum effect, we should adhere to existing interface standards. The standard networking primitives provided by PsiStack adhere to the BSD sockets model, provided by the sockets link library. However, Psion's PsiMail Internet TCP/IP stack has an API different to that of the 'traditional' BSD sockets system. There are also other stack APIs out there such as the System V Transport Layer Interface (TLI). It is not envisaged that there will be much call for this, although if you have a need for it, sufficient material will be provided in the technical documentation to allow you to develop one.

Should you want to develop an application which makes use of PsiStack, please refer to [Comer 1994] Vol III, and the PsiStack SDK (Software Development Kit) which will be relased when the stack is more complete. This document only covers the sockets layer where it interfaces with the stack.

The sockets library provides calls such as socket(), accept(), bind() etc. for application use. Authors of user applications should write their networking code to use the sockets model explained in Volume III of Douglas Comer's "Internetworking with TCP/IP" series. Please note that this library provides functions with names such as open(), close(). These names also exist in the standard C library, CLIB. I know of no method of qualifying function names with library names, so if you want to build sockets applications, you'll have to use PLIB. The API used is taken from Linux, and is a superset of that given in Comer's book. Although many functions are provided, several are simply stubs. For instance, sethostfile() is a stub - you cannot change the location of the hosts file from within an application.

5.3.2 Network Database Library

Gives access to the /etc/services, /etc/protocols, /etc/hosts, /etc/hosts.conf files (their equivalents in the database system) and provides access to DNS name lookup. Provides getprotobyname() i.e. the 'databasey' lookup functions. Some of these calls use a mixture of stack messages (i.e. handled by the stack, via DNS) and database messages (first, do a database lookup) calls. See the SDK for more info.
Note: DNS is yet to be implemented. This library will be tricky to implement.

5.3.3 Error returns in the high level libraries

The Sockets and Network Database libraries handle errors differently than standard EPOC function calls, and calls in the other, low level libraries. It is usual in EPOC to return a negative code on receipt of an error. BSD Sockets calls do not work in this way, and usually retuen -1, with an extended error code stored in a global variable, usually called errno. This error code is usually small, and positive. The BSD Sockets library and Network Database library both provide error reporting in this manner. When linking with these libraries, you will have access to a global errno variable. Calls to these libraries will return the same codes as their BSD counterparts, to ease the porting of applications to the PsiStack system.

5.3.4 Native asynchronous functionality

BSD Sockets applications typically do things differently than the corresponding "correctly-written" EPOC program. For instance, in EPOC, asynchronous completion of I/O requests is common. "Under the hood", sockets operations in PsiStack are asynchronous. In the BSD world, special things have to be done to the file descriptor to make the I/O operations available through it asynchronous or synchronous. The BSD sockets layer in PsiStack takes care of this for you, converting the operation to synchronous operation when a fcntl(fd, O_NOBLOCK) (or equivalent) call is made.

It is possible to use the underlying sockets calls in your PsiStack application - it will look more like an EPOC application if you do. It will also be smaller, since you won't have the BSD translation layer linked with your code. However, you will have to handle all the standard I/O stuff, as described in the SIBO C SDK. PsiStack does not (yet) provide proper EPOC device-based access to the stack. This means that you cannot (yet) open the "NET::TCP://mymailserver.mycompany:25" device.

5.3.5 The Stack Library

This is used by all client applications including the Control and Configuration Application. Functions in the stack library fall into distinct categories:

5.3.5.1 Load / Unload

Functions for loading, interrogating, unloading the stack. The stack kernel and database are dynamically loadable, when an application needs to do network operations. An application can either load the PsiStack kernel itself, or rely on the sockets library to do this. If the application loads the stack, it could do so in a low-priority thread while it is initialising itself. When the application then needs to 'go live' to the net, the stack is already loaded. This is attractive from a user-interface point of view, since there is no perceived delay while the stack is loaded. But it does require extra work. If the application relies on the sockets library to load the stack, there will be a slight delay when the first sockets call is made, as the stack is loaded and initialised. Other than that, there is no special code needed to start the stack from your application.

Applications actually only load the TCP/IP kernel - the DBMS is loaded by the kernel, since it is required by the kernel for its operation. Similarly, it is the kernel's responsibility to close the RDBMS when it closes itself.

5.3.5.2 Event notification

Functions and definitions for stack events, and how to monitor them. All quality applications should use the event monitoring and reporting facilities of the stack library to keep the user informed about network activity.

Allows the setup program to get inside the stack and work out what's happening, and for client programs to monitor the status of a connection. The client posts a request to the kernel, requesting to be notified if particular events occur. Then, the kernel then replies to the client, completing the request. (In the official Client/Server paradigm, Servers don't usually spontaneously send messages to Clients, so this makes applications into Servers, and the kernel into a Client, if you see what I mean).

Multiple client apps can register as listeners of events. There are several types of event that can be listened for - a possible subset of which might be: terminal mode message, transport layer event, ICMP error generated, interface state change, data link layer event, hardware layer event, etc. For example, your whizzo app might want to intercept terminal mode messages (as does the setup program). You call the appropriate function in the stack library and this informs the kernel that you want to be notified when a block of terminal mode data is received. Your client application supplies the address of a status word, and a buffer address. When the terminal mode data arrives, the kernel copies this data to all clients that expressed an interest, and sets their status words to something other than E_FILE_PENDING. It then signals them. It is then up to the application to process the data, and resubmit a request for event notification.

5.3.5.3 Stack control

There are several functions whose purpose is to copy the stack's run-time structures to client memory, and to update these. There are also functions to trigger certain stack activities. These functions are mainly used by the setup program.

5.3.6 The Database Library

Provides access to the database. Wraps a simple API around the IPC messages used to communicate with the database process. Applications and the stack kernel know nothing of the actual structure - they merely use API calls to the DBMS.

Note: The Database process, its API and file structures are yet to be implemented. Since what I've written so far to implement the DBMS access is basically a set of stubs, and the API might change radically, I won't be documenting this yet.

5.4 The script environment

When a script starts executing on the interface, certain script-language variables are automatically assigned from the contents of the netif structure and data model. For example, the IP address, subnet mask, data-link protocol, phone number to dial, user name and password are all passed into the script interpreter as variables. When the interface is brought up, the values of these variables are stored back in the netif structure and data model. This allows the script to modify parameters as it executes. A full list of these variables is given in the owner's manual, in the chapter on dial-up scripting.

Note: Insert ideas about variable as environment…

5.5 The Script Interpreter

The script interpreter is a complex module - it's big, and there is quite a lot of duplicate code in there at the moment. ATM, I've got the language details sorted, and it's now powerful enough to automate connection to an ISP whose PAD has a complex login sequence. The commands are described in detail in the owner's manual. Here, I'll explain a little of the overall structure.

The main script information for each interface is stored in the scriptcontext structure, defined in kernel\script.h. When a script starts, script_start() will read the script into a memory block, note where in that block each line starts, and remembers the location of all the labels, since the goto commands need these. Various elements of the network interface structure are copied into the scriptcontext structure, as variables. Script_step() handles the actual interpretation of the script; this function is called as a result of an ATTN_SCRIPT event. When a line has been interpreted, another ATTN_SCRIPT event is generated, this kicks the script interpreter next time it gets the CPU.

Central to the script interpreter are the sleepstate and inputstate settings. A script will only run if its sleepstate is SSS_ACTIVE. Otherwise, it will be sleeping. script_sleep() is called every half-second, and processes any sleeping scripts. Depending on the instruction which caused the script to sleep, either a timeout, or incoming serial data will cause the script to wake up, and it will be further processed by script_step().

The actual statements of the language are processed by the cmd_* routines.

5.6 The Database File Structures

5.6.1 Some desirable goals for the database structures

Desirable Goal I: Use the available database system if it works. Build in custom file compression (i.e. tries, for storing dictionary data) as needed.

Desirable Goal II: Don't impact the stack's data area with database read-ahead buffers.

Desirable Goal III: Make it easy for users to add or remove devices, ISP configurations, phone numbers, etc.

Goals I and II are satisfied by decoupling the actual implementation of the database from the stack and other applications. Create a database access abstraction complete enough for our requirements, and hide the implementation in a separate process, with its own address space. The database schema we need is sufficiently complex to require a proper relational database management system. But we don't have time to do that fully, so we'll provide a simple random-access and key-access system. No SQL, no selections, cursors or views.

Goal III is satisfied by the use of virtual tables.

5.6.2 Virtual Tables

The ISP, PHN, DEV, and ISV tables are not stored as single tables. The ISP table might seem as though it could contain the following example data:
NameDescription Link ProtocolUser Name PasswordScript Name
DEMONDemon Internet PPPdwarfie f00barDEMON.SCP
PUBNETPPublic Internet PPP access PPPc00ld00dcafebabe PUBP.SCP
PUBNETSPublic Internet SLIP access SLIPc00ld00dcafebabe PUBS.SCP.
LINUXMy Linux Box PPPfrodo xygorfLINUX.SCP

However, no such table exists. Instead, there are four DBF-format files: DEMON.ISP, PUBNETP.ISP, PUBNETS.ISP and LINUX.ISP. Each one contains the relevant tuple from the above 'virtual' table - except the first Name attribute.

When the user requests a tuple from the virtual table by key 'DEMON', the DBMS reads the DEMON.ISP file, and returns the singel tuple stored there, with 'DEMON' added in as the first attribute.

Similarly for the PHN 'table' - it's made up from all the ISP's .PHN files. So, DEMON.PHN, PUBNETP.PHN, PUBNETS.PHN and LINUX.PHN are used to form the PHN table. (LINUX.PHN might be a direct cable connection, in which case, no .PHN file need be supplied).

The above example also illustrates the use of defaults. There are two ISP entries for Public Internet, one using SLIP access, the other using PPP access. Since the primary key (base name of the .ISP file) is different, there will be two .ISP files, and two .PHN phone number databases - these will possibly be identical. To circumvent this, only create one entry for the ISP, and override the default data link protocol when you define methods for that ISP.

Similarly for the DEV 'table'. You might have HAYES288.DEV, HAYES336.DEV, GS18.DEV, NULLMDM.DEV and IRDA.DEV files, which join to create the DEV virtual table. To use a modem with two different sets of serial port settings, duplicate the .DEV file, and modify the duplicate using the setup program.

5.6.3 Misc. Configuration

Note: This section is slightly out of date
The stack has some default settings built-in. For example, the number of memory buffers available; the MTU of the pseudo-interface; the baud rate and other serial parameters of the serial port. (There will be plenty more of these parameters)

Some of these parameters can be changed by the user interface of the stack program. E.g. the serial port characteristics, which script to be used to establish communications, MTU. When these values are changed, the new values are stored in a configuration file. When the stack is subsequently reloaded, this file is interrogated, and any parameters given there override the built-in defaults. This file is called

\APP\PSISTACK\STACK.CFG.

Some of the parameters can be changed by use of the scripting language "on the fly". For instance, when dialling a SLIP connection, it is sometimes necessary to change between 8N1 and 7E1 on the port. The IP address of the network interface being connected may be assigned dynamically.

The parameters changed by a script are not saved, but they are used until the stack is terminated. If the user uses the user interface to change the parameters after they have been changed by a script, the values they will be initially shown are those changed by the script - saving the parameters from the user interface (without modifying them) will save the values set by the script. These values will then be used when the stack is reloaded.

The configuration file can be made read-only to prevent changes by the user interface or by a script being saved.

The stack user interface also has a configuration file:

\APP\PSISTACK\PSISTACK.CFG.

The kernel module config.[ch] handles saving and loading of this configuration information. The setup program config file handling is done in setup.c. The custom binary format of the files will be superseded by the database system described previously in later releases.

5.7 Network Interfaces and their states

There are two network interfaces in the stack: (See earlier diagram)

Interfaces can be in one of four main states: DOWN, TERMINAL, SCRIPT, and UP, as described in the section on interface states in the owner's guide. There are two sub-states, reserved for PPP usage, PPP_UP and PPP_DOWN.

When loaded, the stack is passive, and only the pseudo-interface is up. The pseudo-interface is always up, and can never be brought down. The other interface can be brought up and down as required, by a script, by the null script in response to the user interface command to bring the interface up, by the sockets library, or from the terminal state. The stack changes an interface from down to up by the following method:

The transition to up is complicated by the PPP protocol. When a physical connection is achieved, the interface enters an intermediate phase of negotiation with the remote PPP layer, known as ppp_up. If a successful link can be negotiated, the interface is then brought up. If negotiation fails, the interface is brought down. Similarly, when bringing the interface down, an intermediate phase is entered, where the PPP layers at the local and remote ends of the link inform each other that they are closing (ppp_down). Then the interface goes down properly.

These five changes in state (up, down, term, ppp_up, ppp_down) may be listened for, using the event notification mechanism.

The following state diagram describes the permissible changes of state of a non-PPP interface, and what triggers them:

Note: The control-key sequences have not been implemented yet. This diagram is out of date.

When the interface is up, none of its characteristics can be changed. It can only be brought down. For instance, changing the serial port settings while the interface is active is a rather silly thing to do!

Historical note: Much of the data relevant to each layer of the stack, and each subsystem was placed in the network interface structure (for want of somewhere else to store it). Each interface (there were three in the first versions of PsiStack) had separate storage for its individual ISP data, script context, serial data handling, and data link layer storage. Starting with alpha version 0.05, the third interface has been removed - a limitation in Psion's hardware means that it is not possible to have more than one serial device open simultaneously, and therefore, you couldn't have an IrDA link active at the same time as a PPP/SLIP link via the RS232 port. So, the third interface has been removed. There are now only two interfaces: a local loopback interface, which routes data to/from the upper transport layers of the stack, and the network interface, which can route SLIP/PPP-encapsulated IP datagrams over one of the Psion's devices. Since it is not necessary to keep script, serial, ISP data separate for each interface now (only one interface requires this information), and especially since the local loopback interface does not hold any of this information anyway, the non-IP context has been removed from the network interface structure, and now resides in the modules that deal with it. This reduces the coupling between the various layers of the stack, and makes everything much cleaner. Also, in previous versions, each module needed to access its context information when doing script, ISP, serial operations. Since their context information was coupled to the network interface structure, a pointer to the correct interface had to be passed round everywhere. Now, the context information for each subsystem has now been made into global data, and the subsystems can get at it immediately. This increases performance, since the netif pointer is now no longer passed around. No doubt the purists will moan about the use of globals - tough!

5.8 Event sources within the stack

The correct way of building Psion programs is to make them asynchronous. That is, instead of waiting for a keypress, or timer completion, or serial data, a request is queued for all input sources, then the program goes to sleep, waiting for one of the requests to be satisfied. The program interrogates each input source to see which one caused it to be woken up. That input device is then serviced, and another read queued, and the program goes to sleep. The stack is quite "event-rich" - there are plenty of different event sources:

There are various other sources within the stack. For example, going "up" the stack, when serial data arrives at an interface, the appropriate data link layer process routine, ppp_handler, or slip_handler will build it up into an IP datagram. When the datagram is complete, it is placed on the interface's IP input queue, ni_ipinq. (Refer to diagram in the section entitled "The design of PsiStack") Similarly, when a datagram is queued on an interface's output queue, ni_devoutq, it needs to be sent to the port. So, each access to the IP or nif queues results in a signal being generated, and an appropriate flag being set in the 'internal attention mechanism' flag, attn, in kernel\attn.c. Enqueueing a datagram causes an event to be generated, which will wakes up the stack. The event will eventually get checked for, and processed.

When an interface is brought up, a read request is queued, which will hopefully get serviced, generating an input event. After processing, another read is queued. And so it goes, until the interface is brought down. At that point, the outstanding read request is cancelled. (See SIBO C SDK, v1, General Programming Manual, p32).

When the stack closes, all up interfaces are brought down (cancelling their outstanding read requests) and all other input requests are cancelled.

5.9 Some efficiency tricks

kernel\asy.c\asy_sethandler() sets a function pointer in the asynchronous device (asy) structure, depending on the state of the interface, and the data link protocol in use. Serial reads are queued a byte at a time. Whenever one of these one-byte serial reads completes, the main loop checks to see if there's any more data which could be read. If so, it reads it (concatenating it to the single byte which was already there). The function pointer is then dereferenced, calling the appropriate handler. By doing it this way, I don't have to check which routine gets the serial data every time data arrives. I just have to make sure that whenever the state changes, I call asy_sethandler().

kernel\attn.c is the internal event mechanism, a cheap form of notification semaphore. When, say, the SLIP handler enqueues a datagram for the IP process, it 'wakes up' the IP process by calling attn_signal(ATTN_IP), which sets a bit in the attention variable, attn. Various other 'processes' may set bits in attn, and when all the processes have been run, attn is checked. If it is non-zero, p_iosignal() is called, which generates an asynchronous event. Next time round the main loop, attn is checked, and the processes that need running are run.

5.10 Byte ordering issues

The Internet Protocols are sent with their words arranged in big-endian format, a.k.a. network byte-order. However, the Psion is little-endian, and so care must be taken when dealing with words.

When a datagram arrives, convert all its words to little-endian, and when a datagram is sent, convert all its words to big-endian before sending. As datagrams flow around the stack, they are held in the native format of the Psion, namely, little-endian, or host-order. This scheme has the advantage that datagrams are processed faster, and the disadvantage that conversions have to be done at each protocol-layer interface. I.e. When SLIP passes raw data to IP, IP converts the words in the IP header to host-order, decides which higher-level protocol the datagram is destined for, and forwards it. That layer then converts its words to host-order, and so on, up the stack to the application. When datagrams flow the other way, the application converts its payload to network byte-order, then TCP converts its headers to network byte-order, then IP, etc. Conversions at every protocol interface. Conversions take place even if they're not needed. For instance, none of my code checks the ICMP id or sequence in an echo request/reply, but a conversion is still necessary, because a higher layer might.


6 Development Infrastructure

6.1 The source archive

The source is distributed in a PKZIP .ZIP archive, SRCxxx.ZIP, where xxx is the version number. E.g. Version 0.01 is SRC001.ZIP, and version 2.34 is SRC234.ZIP. If we ever get past version 9, we'll go hexadecimal, so version 15.98 will be SRCF98.ZIP. Documentation in HTML is available in the DOCHTxxx.ZIP archive. A release of the installation files is in BINxxx.ZIP. All files are available from our FTP server, ftp://ftp.inet.alsutton.com.

The archives are produced using the RELEASE.BAT file, by the way.

All files in the source archive are prefixed with PSISTACK\, so uncompressing will always place the files in a directory of their own. You should uncompress this with:

C:

CD \

PKUNZIP -d SRCxxx.ZIP

For more information about the contents of the source archive, and to discover how to build PsiStack, read the README.TXT file in the C:\PsiStack directory.

6.2 Hierarchy of the PsiStack source

Note: The hierarchical layout of the PsiStack source has undergone some heavy changes for the 0.05 release. I haven't got round to documenting all this yet.
The SOURCE directory holds:

6.3 Some notes on how I develop

All development is done on a 486DX4 100MHz, 16MB RAM, with two 1.2MB hard disks, one containing Windows 95, PsiWin, TopSpeed C and the SIBO C SDK, v2.20. The other drive contains Slackware Linux '96. Testing is done with the PsiWin cable between my two computers, with Linux providing SLIP services on one port, with the stack talking to it. I also use the 3a emulator, in DOS, for basic testing to see if the stack will crash a Psion, before letting it loose on my 3c! I also test with my 3c cabled up to my Linux system, and cabled up to the 3a emulator, so I can see two Psion's pinging each other.

The development archive sits on my Win95 drive, in

C:\PsiStack\…

The three Psion emulators are located in C:\S3AEMUL, C:\SIEMUL and C:\WRKAEMUL. In the source package's root directory, the 3A.BAT, SIENA.BAT and WRKA.BAT files all invoke EMUL.BAT which copies the EPOC executables from the INSTALL directory (see the README.TXT file) into the emulator's "M:\APP\", the sample script files (.SCP) into "M:\PSISTACK\" and then launches the emulator.

The emulators are not distributed as part of this package, however, they can be downloaded from your favourite Psion archive. Look for S3AEMUL.ZIP, SIEMUL.ZIP and WRKAEMUL.ZIP.

The SIBO C SDK is installed under C:\SIBOSDK, and the TopSpeed compiler under C:\TS. All installation files and header files assume that this is how your installation will look. Consult the README.TXT file to see how to customise the locations of the package.

6.4 Coding style

Should you wish to help with the development of the stack code, please adhere to my coding style: If you don't stick to these guidelines, expect to see your code reformatted in the next release. They may seem trivial and petty, but they're the way I try to discipline myself, please respect this!

7 References & Bibliography

[Comer 1994]
Internetworking with TCP/IP Volume I: Principles, Protocols and Architecture (third edition), ISBN 0-13-468505-9, £29.95
Internetworking with TCP/IP Volume II: Design, Implementation and Internals (2e), ISBN 0-13-125527-4, £23.95
Internetworking with TCP/IP Volume III: Client/Server Programming and Applications (BSD Sockets version, 2e), ISBN 0-13-260969-X, £29.95
Douglas E. Comer, David L. Stevens
Prentice Hall International Editions.

[Tanenbaum 1996]
Computer Networks (fifth edition)
Andrew S. Tanenbaum
Prentice Hall International Editions.

[Philips 1997]
TCP/IP Explained, ISBN 1-55558-166-8
Philip Miller
Digital Press