We invited Peter Manger from the Fenner School at the ANU to be one
of our test users. Peter has a background in simulation, modelling and
software. He is a good example of someone proficient with data
processing and analysis but with no previous exposure to SWISH tools.
How well did the user use SWISH?
Peter made his way through the tutorial, quicker than I expected for someone who had never used the software before.
During the testing two mistakes occurred
- A missing an input link to an actor. When run the workflow reported an error, Peter quickly identified the omission himself and corrected it.
- The identifier "Date" was used with a capital letters instead of "date". When run Kepler reported an error message; however this problem was too subtle. Even after I pointed it out Peter was unclear as to why the error was occurring.
The images in the tutorial provided the necessary details to continue
through the tutorial without encouraging Peter to notice and understand
the accompanying text. The tutorial explains that all the links need to
be connected for the workflow to run and the identifiers are the names
of columns in the input data.
Did the user complete the analysis they intended to do?
Yes!, Peter successfully assembled an operational workflow that ran and produced the correct result by following the tutorial.
Peter was then curious and started to 'play' with values in the heat
index calculation. He lowered the maximum temperature limit and
increased the min - max threshold to generate a greater number of 'heat
waves'.
What features did they find useful and what features did they have difficulty with or wish to see (and are subject to future improvement if there are available resources)?
Peter was able to quickly use the installer and update the software
to the latest version of SWISH actors. Peter found the drag, drop and
link nature of Kepler intuitive and usable. He was able to easily find
all the actors needed by the tutorial. The actors used where clearly
named and labelled, Peter found them simple to use.
The biggest problem is the handling of errors. The SWISH actors that
use Stata report their own error messages. The SWISH actors that use R
do not report any error messages other than the fact an error has
occurred. General Kepler actors report the Java stack trace from where
the error occurred in Kepler source code.
None of these are useful to the user and could be improved.
The error reported by the Stata based actors indicate the error that
Stata had, but it is always a consequence of a problem located somewhere
in the lead up to running Stata. The error messages require an intimate
understanding of how SWISH operates to decode.
The R errors are completely opaque and provide no useful information.
General Kepler errors are also cryptic and not helpful.
Error handling for general Kepler actors is the responsibility of the
developers of the actors and Kepler to manage. The SWISH Stata actors,
although they perform their own error checking and reporting would
improve the user experience dramatically by reporting more meaningful
messages and solutions to the user. At present the SWISH R actors are completely
lacking in any form of error handling and need to implement it within
the R code itself.
Observations
Peter had trouble linking some actors together as the links 'snap' to
other nearby links or port. Peter worked around this when necessary by
dragging links the long way round to avoid connecting to the wrong
place.
Peter did not run the workflow until the very end. This made finding errors more difficult.
Peter did not realise that by creating and using the workflow in the tutorial he was using the statistical software Stata.
The understanding Peter gained from the tutorial was mainly operational details of how to use Kepler.
Peters feedback
Peter knew that Kepler was existing software but was unclear what
part of the exercise corresponded to SWISH project. His suggestion was
to package all the actors available in a SWISH subgroup of some kind.
Peter liked the images, and found the progression from individual
small steps at the start of the tutorial to boarder instructions at the
end made sense.