Interface and Human Computer Interaction Issues

The spreadsheet interface model presents many human computer interaction problems not easily solved using standard HCI methods. Having several different interaction modes generally leads to bad interfacing, however in the tried and tested spreadsheet model this is not the case. The challenge is made all the greater when spreadsheet calculations involve processing of large images. Such operations generally take a long time, making it difficult to adhere to the HCI principle that users should receive regular and fast feedback.

As a group we spent many sessions discussing the various problems and any solutions we could devise. Our thoughts were tested by use of prototyping and walk-throughs of user operations. To begin with we worked using pen and paper deciding on the basic layout and essential features. Once this was done we were able to produce a first prototype, this was done using HyperCard. The use of a prototype allowed us to easily identify problem areas in our design and to test out ideas quickly. Later prototypes were made using Tcl/Tk and were an investigation of what was possible as well as a demonstration of menu entries, pop-up windows and displaying images.

For the purposes of presenting our findings we have broken the interface down into different areas:

Image Display and Calculation

The user requires many images to be displayed clearly, unambiguously and quickly. The problems faced when trying to achieve this are numerous. In general only one image will be viewable in full dimensions, including colour depth, at any time. This is in conflict with the goal to view all those of interest. Feedback of images will not be immediate due to calculation times. In a spreadsheet formulas may reference back through several cells each of which might need recalculating, it is therefore very easy to introduce long delays for the user.

Our proposed solution to the problem is a compromise. By displaying thumbnails of the images many can be visible on one screen. Feedback delays are also greatly reduces if all calculations are done at thumbnail size. Leaving full images to be calculated only when required or during idle processor time, of which there is an abundance between user interactions. Some problems of image clarity can be avoided by allowing the thumbnail size to be user specified and, with respect to colour depth, by showing images in grey scale if the choice of colours does not give adequate information.

The Spreadsheet Grid

We decided that a spreadsheet of images would require all the features of the grid in standard spreadsheets. These would include selection of cell(s) which can be cut, pasted, copied or cleared, insertion and deletion of rows and columns, and also filling down and filling across of cell formulae.

Additional options were also proposed such as being able to choose the orientation of the spreadsheet, it is convention to label rows numerically and columns alphabetically. However in some scenarios such as editing film sequences a choice would be desirable. Properties of the grid display were discussed asking: should all images be equalised so they are proportionally sized, or should they simply fill as much cell space as possible for easier viewing? Again this would depend on the particular application the spreadsheet was being put to, giving an option would be fairest.

The Grid Cells

Each cell in the spreadsheet will display some form of information for user consumption. When displaying image data an important idea relating to feedback time is to allow cells to be disabled, in this way the user could choose which calculations are of interest and which can be ignored.

Standard features should be included to allow attributes of the cell to be specified such as: left, right and centre alignment, selectable type styles and the ability of cell data to overrun into empty neighbouring cells. Another property specific to spreadsheets of images would be the type of clipping or resizing applied to images when the spreadsheet is resized.

Since each cell holding image data would only be displaying as miniature we though of having a floating window to display the image in full, since at some point the user is likely to want to view the output. It was decided that double clicking on a cell or some other operation would make its image appear in a large window.

The Formula Line

It was suggested that formulas in an image spreadsheet might be quite large, covering at least a few lines. For this reason a separate formula window was suggested rather than the normal single line entry. This presented some questions: which cell would the formula window belong to or should each cell have an individual associated formula window? It was later found however, after some thought on the formula language, that a single line would suffice and could be provided in the standard way.

From our experience of traditional image processing packages the idea to have a menu of image processing functions next to the formula line was brought up. Common functions could be quickly selected from a menu and their inputs chosen from the grid in the regular fashion. Along similar lines we concluded that formulas such as for loading images could be duplicated in menu entries which would provide the more familiar file requester method for loading.

Sequences of Images as Stacks

A useful feature, it was established, when dealing with sequences of images would be to allow some concept of stacks. A sequence of related images such as an animation could be placed in a single cell as a stack, operations on stacks would work as on single images with calculations affecting each of the images in the stack. A formulae operator could be added for creating stacks from a selected of cells.

When displayed in a cell stacks would need to be distinguishable from single image cells. A graphical representation, looking much like a deck of cards with the first image on top, would be used.

This stacking concept would clearly improve interactions for any work carried out on sequences. However, many subtle HCI issues need consideration. We proposed that some tool would allow stacks to be flattened onto the grid for viewing. Any flattening of sequences would ideally be reversible using the stack creation tool. The behaviour of these tools and their effect on cell formulae was cause for some debate. For displaying the flattened stacks a suggested method was to show the sequence running from left to right, beginning at the stack cell. What would happen in the situation where cells to the right were already occupied was difficult to decide, sequences could stop at the first occupied cell or overrun to the end. The effect on cell formulae also needed to be defined, if flattening was regarded merely as a viewing operation then cell formulae would not be inserted, this could cause inconsistencies in the grids normal use. A better idea was to allow stacks to be flattened into rectangular selected areas with formulae being set as references to the stack cell. Never the less there would still remain the problem of what should be done when the area was too small for a sequence, or how ordering would be organised for areas covering multiple rows. Answers to these questions, we felt, would only be found after some controlled prototype testing.

Modal Interaction

Although generally considered bad HCI practice the spreadsheet model is laden with different modes of interaction. Over the years these have been proven very effective.

In a spreadsheet of images there are many other areas which would suggest use of modes. Especially given certain scenarios of use. A browsing mode was decided on for easy viewing of cell images, in the case of sequences this mode would be extended to allow film clips to be viewed by passing the mouse cursor over the cells at the desired speed. Another suggested mode was for dragging and dropping cell contents, such a mode would allow the user to quickly reorganise the grid display.

Modal interactions such as these would provide well defined short cuts to repetitive tasks. For this reason they would improve the systems easy of use and increase the number of possible applications.

Menu Entries

Much time was spent considering what menu items would be appropriate. It was decided in the end that a spreadsheet of images would have provision for all standard spreadsheet menu items along with some additions such as for loading and saving of cell contents, typically meaning images.