CS161 Programming Assignment 3: IMAP Client

Preliminaries

Questions: cs161tas@cs.brown.edu
Help sessions:
    Chris - XXX
    Ron - XXX

Due date: Wednesday, October 19th, 10pm.

Add this line to your shell startup scripts:

    source /course/cs161/startups/cs161student

Introduction

Your third programming task is to extend you login client into a full-fledged IMAP client. Again, IMAP is described in RFC 3501. This is a long document, but you should read it with care.

Your goal in this assignment is to develop a client that will be useful when you develop and test your (future) IMAP server. As such, your focus should be on flexibility, rather than performance. However, since one use of your client will be to simulate high loads, performance does matter, and you will, once again, use an event-based architecture to support large numbers of concurrent connections. To do so in Java, you'll use the non-blocking sockets (SocketChannel) from Java's NIO package and an event loop implented over these sockets (by a Selector). Have a look at basic information about NIO below.

Your client will run commands from a file, as in Assignment 1, with a couple changes. First, you will support many new commands (see below). Second, you will support some level of concurrency even when issuing commands to a single server.

Your client must accept, as its sole argument, a filename of a file we'll use to test your client. The file will describe a list of servers and associated commands to run, and will be in the following format:

anonymous.cs.brown.edu 143
CAPABILITY
LOGIN user pass
LOGOUT

server1.cs.brown.edu 143
CAPABILITY
NOOP
LOGIN user badpass

server2.cs.brown.edu 1043
LOGIN user pass
a1 CAPABILITY
a2:a1 NOOP
a3:a1 NOOP
a4:a2,a3 LOGOUT

The only addition, compared to the format for Assignment 1, is the optional addition of an ordering specification at the front of a command. If the first token contains a digit, it is an ordering specification. The ordering specification is a label and, optionally, a commana separated list of predecessors. If there are no predecessors, then the previous command should implicitly be considered a predecessor. You should run all commands in parallel for a given server, except that commands should not be run until their predecessors have completed.

In the example for server2.cs.brown.edu, LOGIN must be run first, to completion. Then CAPABILITY may be run (it implicitly depends on the LOGIN command). Next, both NOOPs may be run in parallel because they each depend only on the CAPABILITY command. When both complete, LOGOUT may be run.

Output should be written to a standard output in a format that you can interpret unambiguiously (see below) for conformance testing. You will want to log timestamps with your results.

Assignment Specification

  1. Do not use a thread per connection.
  2. You should implement support for the following IMAP4 commands: CAPABILITY, NOOP, LOGOUT, LOGIN, SELECT, EXAMINE, CREATE, DELETE, RENAME, LIST, STATUS, APPEND, CLOSE, EXPUNGE, SEARCH, FETCH, STORE, COPY
  3. You will turn in a pair of files, conformance.cmd and a script in your favorite programming langage conformance that, when run on the output producing by running your imap client with conformance.cmd will produce a human-readable report that indicates how the server responded to various "strange" cases that you have tested.
  4. You have considerable leeway in what you test for. You should deomnstrate that you have considered where the RFC allows for alternate implementations. For example, try to determine if the server will ever execute commands out of order.
  5. Your explanations of your conformance testing should be placed in a README file.

Testing

We have set up an IMAP server for you to test against on pythagoras.ilab.cs.brown.edu, port 143. We hope to add some test accounts, but for now you can begin testing by logging in as yourself, with your normal departmental password

Be aware that you can use telnet as a quick way to explore from the command line. For example, this sequence shows a LOGIN command:

cslab5b:~> telnet pythagoras.ilab.cs.brown.edu 143
Trying 10.116.20.7...
Connected to pythagoras.
Escape character is '^]'.
* OK [CAPABILITY IMAP4REV1 LITERAL+ SASL-IR LOGIN-REFERRALS STARTTLS AUTH=LOGIN] pythagoras.ilab.cs.brown.edu IMAP4rev1 2004.352 at Wed, 6 Oct 2004 23:52:05 -0400 (EDT)
A0001 LOGIN jj "mypassword"
A0001 OK [CAPABILITY IMAP4REV1 LITERAL+ IDLE NAMESPACE MAILBOX-REFERRALS BINARY UNSELECT SCAN SORT THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND] User jj authenticated

You should also read RFC 2683 which contains advice for IMAP developers (clients and servers). It will help you write a better client, but more importantly, it will give you ideas for variant behavior that your client can provide in order to test your server.

Java NIO Basics

For this assignment, there are a few things you will need to understand well in Java NIO. In a similar fashion to Sockets in Java's "Old I/O", SocketChannels create channels enabling communication with a remote host. The number of SocketChannels correspond to how many hosts you are talking to that that instance.

SocketChannels

Here's a basic recipe for using a SocketChannel object:

// Open the channel
SocketChannel sc = SocketChannel.open();
// Set socketchannel to be non-blocking (default blocking)
sc.configureBlocking(false);
// Connect
sc.connect(remotehost);
...
int n = sc.read(Buffer);
int w = sc.write(Buffer);
sc.close();

But we said you should use asynchronous connections, right? So how we can use many SocketChannels at once? This is accomplished with a Selector, which was multiplexes (selects) among many SelectableChannels.

For event registration, we use the call SocketChannel.register(Selector, int) or SocketChannel.register(Selector, int, Object) to register what event we want from that specific SocketChannel.

SelectionKeys (Events)

There are 4 types of events:

 SelectionKey.OP_CONNECT;
 SelectionKey.OP_READ;
 SelectionKey.OP_WRITE;
 SelectionKey.OP_ACCEPT;
From JavaDocs:

Selectors

Here's a basic recipe for using a Selector object:

// Create one selector for all channels
private Selector sel;
...
sel = Selector.open(); // might throw IOException
...
// Register so that when connect is done/failed select returns
sc1.register(sel, SelectionKey.OP_CONNECT);
...
// after selector.select() returns selkey from connect,
if (selkey.isConnectable && selKey.isValid())
    Attachment a = selkey.attachment();
    SocketChannel sc1 = selkey.channel();
    ...
    selkey.cancel();
}
The following tips may help in using a Selector:
  1. The SelectionKey is an int, so we can combine events with a bitwise-or, e.g. "SelectionKey.OP_READ | SelectionKey.OP_WRITE".
  2. In this assignment we do not have to mingle with OP_ACCEPT. We're only writing a client for now, and we do not have to accept connections. Recall a little note that we learnt from class: OP_WRITE would always be TRUE when the SocketChannel is newly created because the network buffer must have space for our data to be written to, so be careful only register for an event if you REALLY need it. Only register for OP_WRITE if you have lots and lots of data to write out to (eg. large .plan files).
  3. The call SocketChannel.register(Selector, int, Object) takes an Object argument that is an attachment. When this channel is selected, due to an event specified in this register call, you can retrieve the Object using the SelectionKey.attachment() function of the appropriate SelectionKey. It allows more versatile code and less lists/tables to keep for us programmers.
  4. One thing to notice, is that the only way to know that the remote host has closed the connection is to issue a read/write method, and if the return value is -1, it was gracefully closed on the other side. If it was not gracefully closed (i.e terminated) on the remote host, it would throw an Exception. Then afterwards you have to issue a SocketChannel.close() method to close our socket.
  5. Above, we use Select.select() to block. You can also call Select.select(int TIMEOUT) which returns after select() is done, or TIMEOUT is exceeded, whichever comes first. Once our call to select returns, several SelectionKey objects may be "selected". You can determine these with the Selector.selectedKeys() function.
  6. SelectionKey has a few interesting functions that you can make use of:
    SelectionKey.isValid(): checks if the key is still valid, because during the process of selection the key might die.
    SelectionKey.isConnectable(): checks if the key is connectable.
    SelectionKey.isAcceptable(): checks if the key is acceptable.
    SelectionKey.isWritable(): checks if the key is writable.
    SelectionKey.isReadable(): checks if the key is readable.
    SelectionKey.finishConnect(): checks if tcp syn connection is finished, or just started or already dead.
    SelectionKey.cancel(): Once the event you requested is no longer needed, issue this to unregister.
    SelectionKey.wakeup(): When select() is blocked on one thread, use this to wake it up on another thread.
    SelectionKey.shutdown(): Shuts the selector down when it is not needed anymore.
    

From our assignment specification, you should have understood that you need to connect, then write, then read some data. For this, you'll want to register for one event first, and after that event is done, cancel and register for another one.

Buffers

In NIO there is another crucial element called Buffers, which are just some Objectified arrays with lots of helpful functions. SocketChannel.read/write both take Buffers as argument. There are many types of buffers, including CharBuffers, ByteBuffers, etc.

Buffer basics:

Charset/Charset Decoders

A last part of NIO Basics is the Charset and its decoder. They are used to convert the above Buffers (CharBuffer/ByteBuffers etc.) to human readable format for a specific Charset. If you use Buffer.toString(), the output string might be decoded in some other format such as unicode, which normally differs from what is transmitted on the network as 8-bit chars, so you might see some Asian/European/Symbols characters coming out. Instead, you can convert a ByteBuffer to CharBuffer, then you can easily use CharBuffer.toString() to display 8-bit chars.

Here's a recipe to read from a Buffer:

Charset charset = Charset.forName("US_ASCII");
CharsetDecoder decoder = charset.newDecoder();
...
ByteBuffer buf = ByteBuffer.allocateDirect(1024);
buf.putChar('A');
CharBuffer cb = decoder.decode(buf);
System.out.println("My CharBuffer: " + cb);

Submitting your code

To submit your code, run the following script:

/course/cs161/bin/cs161_handin imapc [dirname]

where [dirname] is the location of your code (defaults to the current directory).

Please contact the TAs if you have problems submitting. For your own sanity, don't leave your submission until the minute before the deadline.

Grading scheme

Your code will be graded on the following factors (and their weights).

Documentation and code legibility: 20%
Functionality (non-blocking, error reporting, etc): 50%
Robustness (malicious input, many connections, etc): 30%