A Software Cost Estimation Tool for the European Space Agency (ESA): SOCRAT

Graduating from a Defined Software Development Process to a Managed Process - A Case History.

Jon Fairclough, Logica Space and Communications Ltd

Rick Blake, Logica Space and Communications Ltd

lan Alexander, Logica Space and Communications Ltd

Joe Wheadon, European Space Operations Centre

Carlo Mazza, European Space Operations Centre

Keywords: Software Cost Estimation Tool, Reusable Components, Project Cost History

Abstract

The Software Cost and Resources Assessment Tool (SOCRAT') project was initiated by the European Space Operations Centre (ESOC) in October 1991 to investigate the factors that affect Dedicated Control System (DCS) costs and to construct a prototype cost estimation tool. Logica were contracted to perform the study and delivered the first SOCRAT in April 1992.

The SOCRAT project is important because it shows what organisations can do once they have established a software development process model, and design their system around reusable software components. Humphreys has characterised the transition from the 'defined process' to the 'managed process' by the attempt to cost the steps in the development process. This paper gives a practical demonstration of how this can be done.

Areas: Databases of Project Histories

1 Introduction

One of the ways to improve the software development process is to increase the speed and accuracy of costing software projects. Even when software projects aim to follow standards, eventual success can be thwarted by poor estimating of the costs of each step in the process. When impossible schedules are set, standard techniques for producing quality software are often abandoned in the rush to meet deadlines. Project managers need to be able to predict accurately the cost of each work package if the software is to be delivered on-time, within budget, and with the required quality. The effective management of a project depends upon good cost estimates.

The European Space Operations Centre (ESOC) is responsible for the ground control and monitoring of several satellites. The software for the Dedicated Control Systems (DCS) is developed according to the European Space Agency's (ESA's) PSS-05-0 Software Engineering Standards [11. Many of the software components in a DCS are reused from previous missions.

 The Cost-To-Completion (CTC) of a DCS software development project ranges from a few hundred man-months to several hundred man-months. The CTC must be defined as soon as the system requirements have been defined, well before the software has been defined precisely enough to allow the number of lines of code to be estimated. ESOC therefore decided that an approach tosoftware cost estimation was required that differed from the traditional algorithmic approaches such as COCOMO [2]. ESOC contracted Logica to [3]:

2 Analysis

When the SOCRAT project started, ESOC had built three dedicated control systems for the HIPPARCOS, EURECA and ERS-1 space missions. lie ISO control system was being coded and tested during the SOCRAT project, and analysis work for the CLUSTER system was starting.

All dedicated control systems provide the functions of:

The existence of these common functions implies that cost savings can be achieved through software reuse. The four dedicated control systems built so far use ORACLE software for database management and the Spacecraft Control and Operating System (SCOS) for telemetry processing. The SCOS development costs were outside the scope of SOCRAT. Differences in lower-level mission requirements limit the possibilities for software reuse. All the spacecraft have differing payloads and have distinctive payload data processing and distribution policies, for example.

All DCS projects are carried out according to ESA's Software Engineering Standards [11. These standards define a life cycle model of six phases:

The standards define the inputs, activities and outputs of each phase. All DCS projects adopt a incremental delivery life cycle approach towards software development. This means that the delivery of system functions is staggered to match the rest of the project schedule. The project plan will therefore consist of one UR, one SR and one AD phase and several DD, TR and OM phases.

Although the control systems are designed and implemented for ESOC by different contractors, the existence of a standard life cycle model facilitates comparison between different projects at the highest level. All projects will, for example, deliver a Software Requirements Document with a standard structure.

In planning a project, managers always develop a 'Work-Breakdown Structure' (WBS) that decomposes the project into separate jobs called 'Work Packages'. Early in a project work packages are all high level, like 'Write Software Requirements Document'. Later, when the architectural design is complete, the work breakdown structure follows the software component structure. The number of levels in the work breakdown structure increases.

Work packages are the cost elements of the project. Project plans contain estimates of work package costs, and project reports contain actuals. If the cost of each work package is known, the cost of the project can be calculated by summing the costs of the work packages. This microscopic approach to cost estimation is preferable to the algorithmic ones, such as COCOMO, when systems are:

While the macroscopic approach used by COCOMO is useful when these conditions are not met, using COCOMO throws away information when they are. For example if job 1 contains work packages A, B, C and D and job 2 contains only A, B and C, the approach to cost estimation cannot be to calibrate a formula to describe the cost of job I and then apply it to job 2. The simpler, more accurate, approach is to subtract the cost of work package D from the cost of job 1. Of course, this relies on knowing the individual costs of A, B, C and D, and not their sum. This means that projects must practice precise cost accounting. ESOC and its contractors have been doing this for many years.

Another reason why COCOMO was not suitable for use in costing DCS developments was because it relies on estimating the number of lines of code. Ibis usually cannot be done until the architectural design is complete, and this is too late for ESOC managers. The Albrecht function point analysis method could be applied earlier. Unfortunately the Albrecht approach was devised for information systems, not real-time systems.

The approach to cost estimation for SOCRAT was to define a cost model that assumed a generic work-breakdown structure. The DCS project work package exists at the top level (zero). This work package was decomposed into the six phases defined by the ESA software engineering standards. Each phase work package was decomposed into work packages covering the activities required for each phase, such as project management, analysis, design and coding and testing.

The detailed design, coding and unit testing work package was decomposed into work packages that follow the typical DCS architecture. Work packages were defined for the eight functions listed above. Each of these was further subdivided into common work packages and project-specific work packages.

There was considerable variety in the design of dedicated control systems between projects. This was due to variation in requirements, contractors and designers. The work breakdown structures differed as a result, making direct cost comparison impossible. To this problem the following procedure was adopted. For the DD and TR phases of each project:

Phase

Old Phase %

New Phase %

Range

Std. Deviation

SR

7

7

3.4 - 9.7

2.51

AD

13

10

7.5 - 14.8

2.92

DD

64

79

69.7 - 84.8

5.83

TR

16

4

2.6 - 5.8

1.20

 

100

100

Table 1: Refined Phase % model

WP

Title

ERS

EURECA

HIPPARCOS

ISO

4331

Manual Stack

21

12

6

13

Table 2: Work package 4331 costs

No.

Function

ERS

EURECA

HIPPARCOS

ISO

1

Basic manual stack task

6

6

6

6

2

Interface to command queues

6

3

 

3

3

Full command interpretation

7

 

 

 

4

Error

+1

+3

0

+4

 

 

21

12

6

13

Table 3: Level 4 work package cost model for WP 4331

During this process there was some refinement of definitions of the top-level functions. The result was detailed accounts of DCS projects. There were about 630 individual work packages. This was the raw data for further analysis.

After this step it was possible to cost the work packages for each project for the top three levels in the generic work breakdown structure. The total cost of each project, the Cost To Completion (CTC) could be then calculated. This varied from 509 man-months to 265 man-months.

The next step was to give the cost model predictive capability by trying to explain the variation in work package cost. At the phase level, the best way to explain the variation was the 'phase percentage' model. Ibis assumes that each development phase will assume a constant proportion of the effort. The results are shown in Table 1. One of the important outcomes of the SOCRAT project was to revise ESOC's assumes about the phase% values. More effort was being expended in the DD phase than previously thought.

Cost modelling of most level 2 work packages was not possible because of the lack of the cost data for the standard level 2 activities. Cost modelling of level 3 work packages in the DD phase was possible because of the data available on level 4 work packages. The approach was to:

This step in the analysis separated the level 4 common functions from the mission-specific level 4 functions. Figure 1 shows the generic work-breakdown structure.

There are 96 work packages.

The SOCRAT data allow the cost savings possible through software reuse to be estimated. The generic work packages include 77% to 87% of the total effort required to build the four dedicated control systems studied. This means that less than a quarter of the effort will be required to construct the novel functions of a new system. A related statistic is that the effort required to build the software for common functions for the average system was twice that required to build the mission-specific functions. These measurements quantify what ESOC already knows: more kernel software should be developed for reuse between missions.

Twenty-three work packages cost models were built to explain cost variations observed at levels three and four. For work packages that implemented a system function, the approach used was to examine the design documentation for differences in functionality. Tables 2 and 3 illustrate the approach. Typically the lowest cost work package would be used to define the cost of the basic function. Additional functions would be used to explain variations in the cost of the work package in other projects. Cost data on the subfunctions was not available.

A different approach was required for the management work packages, such as those concerned with project management and configuration management. Simple models assuming a constant level of effort (e.g. one full-time project manager) for the duration of the project explained the data accurately.

The work package cost models could not fully account for the actual costs (for example the errors in Table 3 are 1, 3, 0 and 4 man months). Work is currently under way to check if these differences reflect real differences in the volume and complexity of the code. Differences in productivity due to the difference in experience of different contractors, or even project teams from the same contractor, were only observed in one instance. The unexpected rarity of this effect probably reflects the fact that personnel differences even out in large projects.

As well as being at the heart of the cost model, the generic WBS can be used to organise the work of an DCS software project. Adherence to the standards is built in from the start. Activities will not get accidentally omitted in the transformation from the standards to a project schedule.

The output of the analysis work was a Software Requirements Document for the SOCRAT and auxiliary documents describing the cost model. These were input to the design and implementation phase of the SOCRAT project.

3 Design and implementation

BIS/Estimator is a dedicated fourth generation software project cost estimation tool produced by BIS Applied Systems [41]. The tool is written in LPA Prolog and runs under MS-DOS on an IBM-compatible PC. ESOC directed Logica to evaluate BIS/Estimator for use as the basis of SOCRAT. The evaluation resulted in the selection of BIS/Estimator for the SOCRAT project.

BIS/Estimator supports a variety of techniques for cost estimation, one of which is the 'framework model'. This is a hierarchy of 'stages', each of which represents an activity that must be costed. Stages are decomposed into substages, and the cost of a stage is the sum of the cost of the substages.

The generic work breakdown structure developed for costing dedicated control systems maps directly onto the framework model. Work packages correspond to stages. The top level stage is known as the 'project' stage, just as with the generic work breakdown structure. The similarity is repeated at level one and level two with the phase and activity stages. Levels three and four in the framework model are known as the task and subtask stages.

Each stage consists of a:

Stages may be costed in one of the following ways:

The higher level stages usually just calculate the sum of estimates from their substages and perform range checking. The cost models embedded in the lower-level stages use one of the other methods. At the project level, summing all substages gives the Cost-To-Completion (CTC) of a dedicated control system. The cost of a phase can be estimated in two ways:

The RSL rules embedded in each stage make use of libraries of 'questions' and 'functions'. Questions are used to obtain data from the user for the cost model. Several types of questions are possible, ranging from prompts to multiple-choice menus. Functions are written in RSL and codify operations common to several stages. SOCRAT's cost data is contained in a database that is accessed by a function. Stages that need the cost data for estimating and range checking call the function.

A user estimates with SOCRAT by cloning the generic work breakdown structure and working through the bottom-level work packages. Each stage employs one of several possible input mechanisms:

The initial cost of a stage is always an upper and a lower limit, rather than simply zero. As the user estimates each work package, the range of cost is narrowed down. After estimating a set of substages, the user is returned to the parent stage, whose cost is then summed. The time of an estimating session, from logon to CTC result, can last as little as twenty minutes.

The framework cost model is maintained by the SOCRAT Administrator, who can update the global database and the stage cost models. Ordinary users cannot alter the global database.

Figure 2 summarises SOCRAT inputs and outputs.

4 Socrat and software process improvement

The European Communities' ESPRIT programme has funded the Applicafion of Metrics in Industry (AMI) project to develop a practical approach towards the use of measurement in software development [5]. The fundamentals of the AMI approach are:

  1. assess your environment and define management goals;
  2. analyse the management goals into subgoals and identify metrics;
  3. metricate by writing a measurement plan, collecting the raw data and verifying it
  4. improve the process by relating the data to goals and implementing actions.
  5. The SOCRAT project fits in well with this approach.
  6. ESOC's goal was to improve its software process.

Step 1 of the AMI approach requires the environment to be estimated. The Software Engineering Institute's (SEI) software process maturity model is a commonly used technique for evaluating a software process [6].

The SEI model defines five levels:

  1. initial - the process is unrepeatable, unstable, and disorganised;
  2. repeatable - the process is repeatable with basic management practices in place, such as project management and accounting;
  3. defined - the process is defined (in standards), thereby ensuring consistent implementation and enabling computer aided software engineering;
  4. managed - the process is regularly and comprehensively measured and the results used to control it;
  5. optimised - the process is regularly modified to optimise its performance, using the results of an integrated measurement program.

Organisations such as ESA with a standardised process [1] can graduate from a defined process to a managed process by [5]:

Project development effort is the basic project metric. In the SOCRAT project, ESOC developed a database, in both electronic and paper form, of project development effort of dedicated control systems. ESOC has provided resources for the development and maintenance of the database by funding follow-on work by Logica, resulting in refinement of the generic work breakdown structure. ESOC plan to update the database as projects are completed. Analysis of the results has stimulated further research work to metricate software complexity, with the objective of gaining a better understanding of cost variations.

References

1. ESA Software Engineering Standards, ESA PSS-05-0 Issue 2, February 1991

2. Software Engineering Economics, B.W.Boehm, Prentice-Hall, 1981

3. Software Cost and Resources Assessment Tool, ESA Study Contract Report 8794/90/D/IM, J.Fairclough R.Blake, I.Alexander, Logica, 1992.

4. BIS/Estimator User Guide, BIS Applied Systems, 1990.

5. The Application of Metrics in Industry (AMI) Guide, ESPRIT project report, Centre for Systems and Software Engineering, 1992.

6. Managing the Software Process, W.S.Humphreys, Addison Wesley, 1989.

 

See also Guide to Software Project management, ESA PSS-05-08, Issue 1, Revision 1, March 1995

and Software Engineering Standards