Code coverage resources (MC/DC)

Avionics software development requires different levels of code coverage based on the assurance level of the software, which is based on what kind of aircraft failure condition would result from a software anomaly. For Level A software (catastrophic failure),
the required code coverage includes modified condition/decision coverage (MC/DC). Here are several good resources explaining various aspects of this type of code coverage:

NASA – A Practical Tutorial on Modified Condition/Decision Coverage
FAA/CAST – What is a “Decision” in Application of Modified Condition/Decision
Coverage (MC/DC) and Decision Coverage (DC)?
FAA – An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion

Personal data on software defects

Recently, I spent a week at a simulator facility supporting checkout of software changes. During the week, I created several prototype changes for new features. I had a number of issues, and decided to share some data on those issues.

1. Editing problems
I was asked to swap a parameter used in a calculation for another parameter. I was editing source code in a Subversion branch checkout. However, one of the files I had opened was from a trunk checkout of the same software baseline. This file had the main declaration of the new parameter. As a result, I was getting an “undefined parameter” linking error. I thought I had confirmed that all of the files I was using was in the branch checkout directory, but that was not the case. I had to get two of my peers to take a look at what I had done, one of them was able to successfully compile the code changes. That gave me the idea to recheck the location of the files, and I then saw the problem.

2. Compile errors – part 1
My next prototype was a significant modification of two functions. I had a set of redlined requirements, and decided to do a pair programming exercise with the flight control engineer who developed the requirement changes. We reviewed the code changes as we went, then started compiling and removing errors. Here is what I had:
a. Forgot to update function prototypes in include file
b. Two LOC had semicolons when they should have had no semicolons (function initial statements)
c. One incorrect variable declaration
It took about 5 minutes to correct those problems.

3. Compile errors – part 2
I got a set of draft requirements for a new algorithm from an engineer. The requirements were in a Word document, and included some pseudocode and actual code snippets. I created the initial prototype by myself, then reviewed it with the engineer. After our review, I then started compiling and removing errors. I cannot remember the early errors, but there was one that persisted. The compiler was telling me that there was a problem with a character on two line, but did not directly identify the character. The problem was when I copy/pasted the code snippets, the minus sign in two equations was not a dash, but some other character. I deleted that text and retyped the minus signs, and that corrected the final problem.

Summary
1. When in a hurry, take time to review your changes.
2. When copy/pasting from a Word document, be sure to check equation text.
3. Pair programming can be a useful development method.

 

The Software Quality Profile

Returning to the topic of metrics, Watts Humphrey discusses concrete ways of measuring software quality, and how to overcome
issues with data collection. From his introduction:

“There are four reasons why the software community has been slow to use numbers for software quality.
1. There is no generally recognized definition for quality measures.
2. Even when we know what numbers to use, the data are not easy to gather.
3. Even with data, it is not obvious how to interpret and use the numbers.
4. People are reluctant to measure the quality of their personal work.
This paper addresses these questions and gives examples of software quality data that can readily be gathered and used by properly trained engineers.”

Links:

 

Software development metrics that matter

Jim Bird has a post discussing software metrics (the good, the bad, the ugly). He advocates doing what makes sense for your organization:

“Only measure things to identify potential problems and negative trends, strong points and weak points in the code and your development process. If you want to set goals for the team to improve speed or cost of quality, let them decide what to measure and how to do it. And measure whatever you need to build a business case – to prove to others that the team is moving forward and doing a good job, or to make a case for change. Make metrics work for the team. You can do all of this without adding much to the cost of development, and without being evil.”

My perspective on metrics is as follows:
1. You can’t know how you are doing unless you collect some data and analyze it.
2. Only collect data on things you are interested in or want to doing something about.
3. If you aren’t going to do anything with the data, don’t collect it. (Do I care about the data? If no, then don’t collect it.)

Links:

 

Source code counters

I am working on metrics development for my software development group. To aid in this project, I have looked for tools to provide consistent counting of
source code (SLOC). The Understand tool provides this capability (commercial tool). There are also several open source tools available:

Capers Jones has published an evaluation of several software size metrics (function points, SLOC, other methods), and discusses their advanatages and disadvantages.

A Look Back: Space Shuttle software development process

The software development environments of today are experiencing an explosion of new development methodologies (Agile, Scrum, etc.). For a look back to see at one of the first documented development processes, this book provides insight into the development and verification practices used for development of the Space Shuttle flight software.

Links:

  • National Academies Press – Chapter 3 of the book “Assessment of Space Shuttle Flight Software Development Practices” is an overview of the complete flight software development and upgrade process.

Coding standards examples

Many companies have internal coding standards to help software teams be more consistent in their development work. Here are several coding standards available online to serve as examples for groups looking for a standard.

Links:

  • US Air Force – F-35 Joint Strike Fighter
  • JPL – C coding standard
  • MISRA – Main site, C coding standard available for purchase
  • Wikipedia – Overview of MISRA coding standard/rules
  • Embedded.com – Overview of MISRA C coding standard
  • EmbeddedGurus – Discussion of whether it is worth demonstrating full compliance to MISRA standard
  • EmbeddedGurus – Collection of articles on coding standards topics

Tips for safety critical code from JPL

Gerard J. Holzmann with the Laboratory for Reliable Software at the Jet Propulsion Laboratory has been a frequent contributor to IEEE Software on software development topics. His paper from 2006 on
rules for developing safety-critical code are good advice for anyone working on avionics or medical software applications (via Ward Cunnningham). Denis Radin (@PixelCommander) has also applied these rules to Javascript development.

Links:

Apollo software engineer gets Medal of Freedom

In November 2016, President Barack Obama gave the Presidential Medal of Freedom to Margaret Hamilton, who was director of the Software Engineering Division of the MIT Instrumentation Lab in the 1960s. Her team was responsible for the development of the Apollo Guidance Computer software. Margaret is a pioneer in software development/engineering, this is a well-deserved recognition.

Quote from Margaret Hamilton:
“From my own perspective, the software experience itself (designing it, developing it, evolving it, watching it perform and learning from it for future systems) was at least as exciting as the events surrounding the mission. … There was no second chance. We knew that. We took our work seriously, many of us beginning this journey while still in our 20s. Coming up with solutions and new ideas was an adventure. Dedication and commitment were a given. Mutual respect was across the board. Because software was a mystery, a black box, upper management gave us total freedom and trust. We had to find a way and we did. Looking back, we were the luckiest people in the world; there was no choice but to be pioneers.”

Links:

  • ABC News – video of presentation
  • MIT News – Announcement of award, short bio
  • MIT News – Brief history of photo of Margaret Hamilton standing next to printouts of Apollo Guidance Computer software
  • MIT News – Reminiscences 40 years after Apollo 11
  • Quartz – Article on Apollo Guidance Computer software
  • Github – Original Apollo 11 Guidance Computer (AGC) source code for the command and lunar modules.