What is it all about?

Usability is an important quality attribute for APIs, but measuring it is not an easy task. It requires a lot of time and resources, as well as experienced evaluators. With the API Concepts Framework, we want to provide a possibility for measuring API usability in an easy and automated way.

API Concepts Framework aims to provide a way for
  • API developers, to get fast initial response to the designs of their APIs, and to avoid costly changes in late development stages
  • API users, to be able to objectively compare the usability of different APIs, which is not possible with existing methods, since they require too much effort, do not provide quantifiable results, are not comprehensive enough and/or do not take the context of use into account
The API Concepts Framework is part of an ongoing research in the area of API usability, done by the Space Based Computing Group of the Institute of Computer Languages of Vienna University of Technology.

How to use it

When evaluating the usability of an API (or actually any kind of software product), an important factor is the context of use (i.e. who is the user and what does the user want to do with the API). A unique trait of the API Concepts Framework is that it takes this into account. It therefore doesn't only take the API itself as input for the evaluation, but additionally the code that needs to be written when doing a specific task with an API (can e.g. be an API usage example).

In the current version of the API Concepts Framework the measurement is half-automated, i.e. it is not (yet) possible to start measuring with only a single button click. But it is still quite simple by just writing a few lines of code. Let's have a look at a very small example:

Say we have an API for creating ZIP files, like the very good DotNetZip library, and we want to measure the usability for the simple task of creating a zip file that contains a single file. The required code would be something like this:

using (ZipFile zip = new ZipFile())

To measure the usability of the API concerning this task you now need to feed information of the task into the API Concepts Framework. Therefore the API Concepts Framework itself has an API that you can use. In case of the API and code above, you would simply do the following:

//initilization of ApiEvaluator
var eval = new ApiEvaluator(typeof(ZipFile).Assembly);

//definition of the task for which the usability is evaluated
eval.Instantiation(() => new ZipFile());
eval.MethodCall((ZipFile z) => z.AddFile("ReadMe.txt"), returnParameterIsUsed: false);
eval.MethodCall((ZipFile z) => z.Save("MyZipFile.zip"));

//start the evaluation and print the results to the console

You can see that every statement that involves the API is fed into the evaluator. For every class that is instantiated the method "Instantiation" needs to be called, for every method call the method "MethodCall", and so on. So it is pretty simple to feed code into the evaluator.

The output from calling "EvaluateAndPrintToConsole" in the end is as follows:

Cost S Cost U Cost Sum Concept Usages Prev Usages Description
30.0 21.0 51.0 ClassUsage 1 0 ZipFile
2.0 6.5 8.5 Instantiation 1 0 new ZipFile(String)
15.7 7.5 23.2 MethodCall 1 0 ZipFile.AddFile(String) (return value not used)
13.7 5.0 18.7 MethodCall 1 0 ZipFile.Save()
101.3 Overall

The evaluation is based on the idea that, when using a certain API for a certain task, specific concepts need to be used. Each concept represents a single action of what a programmer needs to do witch a certain API. The evaluation shows a list of all used concepts, and rates their complexity. The complexity rating is split into search and usage complexity ("Cost S" and "Cost U"):
  • Search complexity depicts how hard it is for a user to find out which concept to use for a certain task (e.g. which class needs to be used to create a zip file, which method to add a file). For finding a class member, it depends e.g. on the number of other members with the same prefix (e.g. having many members starting with "Add" would make the method "AddFile" harder to find), or the number of overloads.
  • Usage complexity depicts how hard it is to actually use the concept. E.g. for a method, using it is harder the more parameters it has.
The complexity values are summed up into a single end result, which gives a basic idea of the overall complexity and allows a comparison to other APIs.

The calculation of complexity additionally depends on how often a concept has been used. In the evaluation this is shown as "Usages" and "Prev Usages". The former defines how often a concept is used within the evaluated task. E.g. if the task is to add 3 files to the zip file, the AddFile method would need to be called 3 times, making 3 usages. The latter defines how often a programmer has previously used the concept - the more knowledge a programmer already has about a concept, the easier the task will be for him. E.g. a programmer who knows an API very well won't need to search for the concepts any more, so the search complexity will be greatly reduced. This allows taking the developer's experience into account - e.g. for handling the question: "Is it worth the effort to learn how to use the new API B, or should I stay with API A which I already know very well?".

By the way, the evaluation output can be generated in 3 different formats (you can change it using the ApiEvaluator.OutputFormat property): Console (fixed width separation), Latex table, and wiki table (the table shown above has been generated using that).

For a more advanced example, please take a look here: WCF vs XcoAppSpace

Using the Results to Improve Usability

The results can generally be interpreted as: the lower the complexity, the better the usability. The number and type of concepts of course play a big role - the list of concepts allows you to identify the most complex concepts and thereby think about ways to reduce or eliminate them. ClassUsage is one of the most complex concepts, meaning you should try to keep the number of different classes that need to be instantiated, inherited, or otherwise used as low as possible.

Other than that, the evaluation output also shows suggestions if there are any specific details in the API that need improvement. E.g. it shows a suggestion when the number of method parameters should be reduced (when the number is > 5), or when there are too many members with the same prefix.

Extensibility: Adding new Concepts

The API Concepts Framework is an ongoing work, and there will of course be concepts and properties that our framework doesn't yet take into account. For that reason, it was our goal to to make the framework highly extensible. It is easily possible to add new concepts (or measurable properties to an existing concept) by implementing the interfaces {ILowLevelConcept}, {IHighLevelConcept} and {IMeasurableProperty}.

If you want to know more, please don't hesitate to contact us.

More information coming soon!

Last edited Sep 7, 2014 at 6:14 PM by thomass, version 14