Neil Ernst

Browsing Posts published by Neil Ernst

How do Software Architects Consider Non-functional Requirements: An Exploratory Study

November 6, 2012 Neil Ernst No comments

David Ameller, Claudia Ayala, Jordi Cabot, and Xavier Franch, How do Software Architects Consider Non-functional Requirements: An Exploratory Study, RE 2012, Chicago.

Dealing with non-functional requirements (NFRs) has posed a challenge onto software engineers for many years. Over the years, many methods and techniques have been proposed to improve their elicitation, documentation, and validation. Knowing more about the state of the practice on these topics may benefit both practitioners’ and researchers’ daily work. A few empirical studies have been conducted in the past, but none under the perspective of software architects, in spite of the great influence that NFRs have on daily architects’ practices. This paper presents some of the findings of an empirical study based on 13 interviews with software architects. It addresses questions such as: who decides the NFRs, what types of NFRs matter to architects, how are NFRs documented, and how are NFRs validated. The results are contextualized with existing previous work.

In this work, Ameller et al. consider the contention that NFRs ought to be driving concerns for software architects. They conducted a study with Spanish software architects in a variety of domains to understand how they thought of NFRs. Their first finding was that no one held a formal “architect” role, although that was what their work entailed. The job position was based on skills and knowledge rather than training. Their second finding was that NFRs were not of primary importance, which contradicts other research findings. Instead, they found it was more important to consider project-wide constraints like licencing and overall cost. This suggests some interesting directions for new research in the role architecture plays in the software development process.

A related blog post with more detail can be found here.

Qualitative Studies, Reproducibility, Uncategorized

Using topic modelling to understand requirements

August 23, 2012 Neil Ernst No comments

Abram Hindle and Thomas Zimmerman, “Do Topics Extracted from Requirements Make Sense to Managers and Developers?“, International Conference on Software Maintenance, 2012.

Disclosure: Abram and I have collaborated on a somewhat related paper.

Large organizations like Microsoft tend to rely on formal requirements documentation in order to specify and design the software products that they develop. These documents are meant to be tightly coupled with the actual implementation of the features they describe. In this paper we evaluate the value of high-level topic-based requirements traceability in the version control system, using Latent Dirichlet Allocation (LDA). We evaluate LDA topics on practitioners and check if the topics and trends extracted matches the perception that Program Managers and Developers have about the effort put into addressing certain topics. We found that effort extracted from version control that was relevant to a topic often matched the perception of the managers and developers of what occurred at the time. Furthermore we found evidence that many of the identified topics made sense to practitioners and matched their perception of what occurred. But for some topics, we found that practitioners had difficulty interpreting and labelling them. In summary, we investigate the high-level traceability of requirements topics to version control commits via topic analysis and validate with the actual stakeholders the relevance of these topics extracted from requirements.

A holy grail of software research is to (automatically) relate the business value of the software feature to the code implementing that feature, known as requirements traceability. All sorts of benefits are posited to result from this, including the ability to tell whether your customer’s needs are met.

One approach to this is to use an information retrieval technique called topic modelling. Topic modelling generates word distributions for a set of documents, like requirements specifications. One of the problems with topic modelling is that the topics are presented as lists of seemingly unrelated words, and the content of these topics must be captured with a descriptive label. In this paper the authors assess whether developers at Microsoft find the topics easy to label and understand.

What they discovered was that the study participants agreed with the proposed linkages between requirements topics and commits, but that the topics were difficult to label without being customized to the individual developer. Program managers seemed to find the topics more comprehensible, possibly because they deal with a wider array of features in their work. Further use of topic modelling in this area seems to require labelling by domain experts before being widely applicable to the traceability problem.

Organizational Studies, Quantitative Studies

Usability Analysis of Visual Programming Environments: a ‘Cognitive Dimensions’ Framework

June 25, 2012 Neil Ernst 1 comment

Thomas Green and Marian Petre, “Usability Analysis of Visual Programming Environments: a ‘cognitive dimensions’ framework”, Visual Languages and Computing, 7:131—174, 1996.

The cognitive dimensions framework is a broad-brush evaluation technique for interactive devices and for non-interactive notations. It sets out a small vocabulary of terms designed to capture the cognitively-relevant aspects of structure, and shows how they can be traded off against each other. The purpose of this paper is to propose the framework as an evaluation technique for visual programming environments. We apply it to two commercially-available dataflow languages (with further examples from other systems) and conclude that it is effective and insightful; other HCI-based evaluation techniques focus on different aspects and would make good complements. Insofar as the examples we used are representative, current VPLs are successful in achieving a good ‘closeness of match’, but designers need to consider the ‘viscosity’ (resistance to local change) and the ‘secondary notation’ (possibility of conveying extra meaning by choice of layout, colour, etc.).

It must be a major coup as a researcher for your research to make in onto Wikipedia (and not get deleted for “notability”). The reason the Cognitive Dimensions framework has had such impact is that it is a great way to assess user interfaces without conducting user studies (which are costly and often of questionable value). Green’s Cognitive Dimensions framework proposes a set of heuristic criteria which evaluate how well the interface supports certain tasks. These criteria include consistency, hard mental operations and error-proneness.

In this paper Green and Petre apply the framework to visual programming environments (e.g., Simulink). The research question they investigate is the degree to which these tools help in writing software compared to a text editor for Basic programming, using a yardstick problem of calculating rocket trajectory.

Some of their findings:

Visual programming languages can easily grow incomprehensible, placing “severe demands on working memory” as concepts become intertwined.
Abstraction is easier to manage in visual languages, since low-level items can literally be hidden away.
Adding new code to existing visual programs was very slow: from 508 seconds for LabView, 194 seconds for Prograph and only 63 seconds in Basic, which they note is “an astonishing ratio of 8:1 between extremes”.
These languages, like spreadsheets, make long-range data dependencies difficult to track down.
Textual languages (like Pascal or Basic) support secondary notations such as the use of whitespace or statement ordering to communicate other information. In the two visual languages they studied, these notations are meagre. Visual layout is not terribly useful for communicating information. For example,

“I quite often spend an hour or two just moving boxes and wires around, with no change in functionality, to make it that much more comprehensible when I come back to it.” It is hard to imagine a Pascal programmer having to spend an hour or two doing nothing but re-arranging the components of the program to make it comprehensible.

Many of these findings are even more relevant today in the model-driven design (MDD) field, and show the power of a simple set of heuristics in uncovering potential difficulties. Designers of modern visual language tools (like Metacase) should be able to explain how they overcome these challenges.

Programming Languages, Qualitative Studies, Tools, Usability

Software Requirements Change Taxonomy: Evaluation by Case Study

October 28, 2011 Neil Ernst No comments

Sharon McGee and Des Greer, “Software Requirements Change Taxonomy: Evaluation by Case Study“, International Conference on Requirements Engineering, Trento, Italy, September 2011.

Although a number of requirements change classifications have been proposed in the literature, there is no empirical assessment of their practical value in terms of their capacity to inform change monitoring and management. This paper describes an investigation of the informative efficacy of a taxonomy of requirements change sources which distinguishes between changes arising from ‘market’, ‘organisation’, ‘project vision’, ‘specification’ and ‘solution’. This investigation was effected through a case study where change data was recorded over a 16 month period covering the development lifecycle of a government sector software application. While insufficiency of data precluded an investigation of changes arising due to the change source of ‘market’, for the remainder of the change sources, results indicate a significant difference in cost, value to the customer and management considerations. Findings show that higher cost and value changes arose more often from ‘organisation’ and ‘vision’ sources; these changes also generally involved the co-operation of more stakeholder groups and were considered to be less controllable than changes arising from the ‘specification’ or ‘solution‘ sources. Overall, the results suggest that monitoring and measuring change using this classification is a practical means to support change management, understanding and risk visibility.

Many people have considered how best to classify requirements changes: for example, Harker (pdf) or Zowghi (pdf). In this paper, the authors conducted a single case study to understand whether their taxonomy could not only capture the various changes which occurred during an industrial software development project, but also whether such a classification could help with project management concerns.

It is a well-worn truth that changes in requirements can be very expensive to fix later in the project. However, one of the things that is typically not considered is the opportunity that a change affords. We tend to focus on the negative, but as McGee demonstrates during her case study, these changes are a key part of business strategy.

In particular, the highest-value/highest-cost changes came from the strategic, organization level, and the lowest-value/lowest-cost changes to the system from the detail-oriented, solution implementation level. The classification of change origins provides evidence that the context of the change is important in understanding how to manage that change.

During her research presentation, Sharon McGee also commented on the challenges of this type of research. While valuable, the organization she embedded with found the research process time-consuming. She doubted they would be willing to undergo a follow-up study. This is a major barrier to obtaining case study opportunities that go beyond the anecdotal.

Organizational Studies, Qualitative Studies, Uncategorized

Requirements Prioritization in Agile Projects: Insights from a Case Study

July 4, 2011 Neil Ernst 1 comment

Zornitza Racheva, Maya Daneva, Andrea Herrmann, Klaus Sikkel and Roel Wieringa, ”Do We Know Enough About Requirements Prioritization in Agile Projects: Insights from a Case Study“. RE10.

Requirements prioritization is an essential mechanism of agile software development approaches. It maximizes the value delivered to the clients and accommodates changing requirements. This paper presents results of an exploratory cross-case study on agile prioritization and business value delivery processes in eight software organizations. We found that some explicit and fundamental assumptions of agile requirement prioritization approaches, as described in the agile literature on best practices, do not hold in all agile project contexts in our study. These are (i) the driving role of the client in the value creation process, (ii) the prevailing position of business value as a main prioritization criterion, (iii) the role of the prioritization process for project goal achievement. This implies that these assumptions have to be reframed and that the approaches to requirements prioritization for value creation need to be extended.

Mention the phrase “requirements engineering” to many software developers and you’ll get a groan. For a long time, requirements engineering (elicitation, analysis, modeling, etc.) has been seen as something you do to satisfy the paper-pushers. There’s even a derogatory acronym: BRUF, Big Requirements Up Front. Nevertheless, the fact remains that we typically build software to satisfy a user, even if the user is ourselves. Doing so requires us to think about what we should build, and, hopefully, why we are building it. While requirements may take different forms (user stories, tasks, use cases), they remain fundamental to the process of building software.

This paper points out that the problem of prioritizing requirements is even more of a concern in agile methodologies than other approaches. This is because short cycle times require frequent prioritization of the backlog. To do so, both XP and Scrum, for instance, call for an involved customer. However, this study found that this was rarely possible, and that as a result prioritization was done by the developers. Customers either found planning meetings too technical, or were not aware of their own requirements. They also found a difference in the understanding of the “value” of a requirement: there is a distinct difference between the value for the customer and the value for the developer. For example, developers might prefer to re-use solutions from other projects. Racheva et al. did find, though, that the notion of frequent, short iterations with re-prioritization was highly useful, in particular for dealing with new information and unclear requirements.

Organizational Studies, Qualitative Studies

It Will Never Work in Theory

Subscribe by email

Contributors

Recent Tweets

Categories

Recent comments