Preface
The federal government depends heavily on a variety of
information technology products and services to serve the public.
Each year the government spends billions of dollars on computer
equipment, services, software, and telecommunications. The success
or failure of information system acquisitions affects executive
agencies' credibility with the Congress and the public as well as
their abilities to carry out their missions effectively and
efficiently.
The General Accounting Office (GAO) and offices of inspectors
general have consistently identified problems with information
technology acquisitions. Problems identified in numerous
evaluations include information systems that do not meet users'
needs, exceed cost estimates, or take significantly longer than
expected to complete.
This guide provides a logical framework for evaluating
information technology acquisitions. It incorporates a risk
assessment methodology intended to reduce audit planning time and
ensure that significant issues are included. It is based on a model
of the acquisition process developed by GAO in cooperation with a
wide range of federal and private sector officials. 1 The model
outlines the process used to acquire information technologies and
identifies elements of the process that are essential for planning
and carrying out acquisitions.
This guide is intended for use in planning and conducting risk
assessments of computer hardware and software, telecommunications,
and system development acquisitions. A risk assessment is the
process of identifying potential risks in a system under
development and then determining the significance of each risk in
terms of its likelihood and impact on the acquisition's cost,
schedule, and ability
1Information Technology: A Model to Help Managers Decrease
Acquisition Risks (GAO/IMTEC-8.1.6, August 1990).
to meet the agency's needs. Such assessments may have their
greatest impact if carried out early, when an agency can more
easily alter its acquisition plans and strategy to manage and
control the identified risks.
The audit guide consists of 10 chapters, with appendixes.
Chapter 1 introduces the purpose of the guide, explains its
essential concepts and techniques, and provides direction in
tailoring the guide for use on specific assignments. Chapters 2
through 10 address specific activities in the acquisition process.
The nine chapters present audit guidance on:
•
management and user support,
•
project staffing,
•
needs/requirements/ specifications,
•
alternatives,
•
acquisition planning,
•
solicitation document,
•
source selection,
•
contract management, and
•
test and acceptance.
Each chapter lists audit objectives, commonly expected
documentation, detailed audit questions, and references to federal
regulations and guidance. The chapters are intended to assist in
the identification of specific risk areas and to contribute to an
overall assessment of how well an agency is meeting its acquisition
objectives.
This audit guide is also available in a software format,
accompanied by reference materials such as the GAO model of the
acquisition process, relevant federal acquisition regulations,
General Services Administration (GSA) guidance, and Office of
Management and Budget (OMB) circulars. The software version
utilizes a hypertext software package to help auditors quickly and
flexibly review
Page 2 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
all the included documents. This software may be requested from
GAO by returning the card included with this guide.
This guide supplements and does not replace other GAO policies
or procedures. It was prepared under the direction of Jack L.
Brock, Jr., Director, Government Information and Financial
Management, who can be reached at (202) 512-6406. Other major
contributors to the guide are listed in appendix IV.
Ralph V. Carlone Assistant Comptroller General Information
Management and
Technology Division
Werner Grosshans
Assistant Comptroller General for Policy
Contents
Preface
Chapter 3 22
Audit Objective
22
Project Staffing
Documentation Required
22
Audit Steps: Project Management
22
Audit Steps: Project Staff
23
References
24
Chapter 4 25
Audit Objectives 26
Needs /
Documentation Required
26
Requirements /
Audit Steps: Needs Determination
26
Audit Steps: Requirements Analysis 27Specifications
Audit Steps: Specifications 30
Audit Steps: Test Plans
32
References
32
Chapter 6 40
Audit Objective
40
Acquisition
Documentation Required
40
Planning
Audit Steps 41
References
43
Chapter 7 44
Audit Objectives 45
Solicitation
Documentation Required
45
Document
Audit Steps 46
References
49
Chapter 8 50
Audit Objective
50
Source Selection
Documentation Required 51
Audit Steps
51
References
54
Chapter 9 55
Audit Objectives 56
Contract
Documentation Required
56
Management
Audit Steps
56
References
59
Chapter 10 60
Audit Objectives 61
Test and
Documentation Required
61
Acceptance
Audit Steps
61
References
63
Appendixes
Appendix I: Acquisition Profile
Appendix II: Management Metrics 68
Appendix III: Reference Materials 80
Appendix IV: Major Contributors to This 86Audit Guide
Glossary 87
Table I.1: Map to the Audit Guide
Table
Figures
Figure II.1: Problem Reports in Software 73Project
Figure II.2: Problems By Priority Levels and 74Number
of Months Unresolved
Figure II.3: Software Requirements Changes 75
Figure II.4: Development Progress 76
Figure II.5: Software Size Estimates 78
Abbreviations
Chapter 1
Introduction
This audit guide is based on and incorporates GAO's information
technology acquisition model. The model describes three phases in
the acquisition process: presolicitation, solicitation and award,
and postaward. It sets out essential activities in each phase along
with critical factors related to those activities. The model is
intended to give managers an overview of the acquisition process
and to help them decrease acquisition risks.
The model's critical factors were drawn from the judgment and
expertise of a wide range of knowledgeable individuals from
government and private industry. Compliance with these critical
factors can increase the likelihood that an acquisition will meet
an agency's needs at a reasonable cost and in a timely manner.
This guide is intended to help auditors conduct more
Objectives focused reviews of information technology
acquisitions by enabling them to quickly identify significant areas
of risk. Using this guide will help auditors identify critical
factors not addressed by management, make a general assessment of
any procurement risks, and provide rapid feedback to agency
officials so they can take corrective action in a timely and
efficient manner. Use of the guide should be selectively tailored
to the requirements of particular reviews and adapted to the status
of the acquisition.
Auditors will need to exercise professional judgment in
assessing the significance of audit results or findings. For
example, the guide assists auditors in determining how an agency
identified and defined its requirements. Professional judgment is
necessary to evaluate this information and determine if the agency
conducted an adequate requirements analysis.
Page 8 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Some areas of assessment will require the expertise of auditors
with specific technical skills and experience. These specific areas
include knowledge of solicitation procedures, benchmarking and
other performance or capability validation techniques, and
knowledge of technical areas such as database management and
telecommunications networks. These areas are identified within the
guide. The audit team should include experienced members with
enough knowledge of information technologies to satisfy the
Government Auditing Standards
requirement that auditors have appropriate skills and
knowledge.
The audit approach described in this guide is intended to result
in a risk assessment of an acquisition project at any point in its
development. The auditor will be trying to determine whether a
project will result in a specified product or level of performance
and will be delivered at a specified time and cost. An auditor
should use this guide to identify areas that are most likely to
result in technical failures, unmet user needs, cost overruns, or
schedule delays. Those risks should then be brought to the
attention of appropriate agency officials. The audit steps and
questions provided are directed toward assessing whether an agency
has sufficiently addressed critical factors, including support from
managers and users, adequate project staff, and controls over the
acquisition's scope before and after a contract is awarded.
Approach
Assignment Planning
When planning a risk assessment, the auditor should first review
the agency's acquisition policies and directives to identify the
organizations and individuals responsible for approving
procurements. Approval thresholds, for example, should show which
officials have the authority to review and approve
acquisitions.
Page 9 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Organization and Use of the Audit Guide
The agency's directives should also show the specific milestones
and documentation required for a procurement.
The auditor should also review previous studies or audits of the
acquisition project and of the agency's information resources
management functions. Reports by GAO or other auditing institutions
can provide valuable background information. The auditor should
also determine whether previous recommendations have been carried
out.
The chapters of this audit guide focus on logically distinct
steps of the acquisition process, as described on page 10 of GAO's
acquisition model. 1 The following table identifies the appropriate
chapters for reviewing the various steps of an acquisition. In
general, the auditor will want to concentrate on the steps that are
relevant to the phase of the acquisition being reviewed. However,
regardless of how far the acquisition has advanced, at a minimum
the auditor should always ascertain whether senior managers and
users were involved in the project's initiation. The auditor should
also verify that the agency has defined its needs and requirements
to support its mission, and that those requirements continue to be
valid as the acquisition progresses through contract award and
contract management.
1GAO/IMTEC-8.1.6, August 1990.
Each of the following chapters provides references to
regulations and other guidance relevant to material in the chapter.
In addition, each chapter identifies specific audit objectives and
documentation expected for the major activities at that point in
the acquisition process. The documents listed may exist with
different names than those used here. The auditor should refer to
agency-specific requirements for more information. Finally, each
chapter sets out audit steps to help plan and conduct the
assessment of an acquisition. The questions may need to be tailored
to
Page 11 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
the particular circumstances of an audit, using the subject
agency's requirements.
The appendixes provide further information for use in planning
and conducting an acquisition audit. Appendix I contains a
worksheet, called an acquisition profile, to summarize essential
information about the acquisition project. The worksheet should
contain such information as the names and locations of units
responsible for the acquisition, the project purpose, and the
expected cost and time frames. The completed profile should be kept
available for later reference. If a profile exists from an earlier
assignment, the auditor should review and update it.
Appendix II of this guide describes techniques to use in
identifying software development risks of delays and cost overruns,
known collectively as management metrics or indicators. These
techniques require information about the size of the system being
developed and about progress after development has begun. Thus, the
techniques are only appropriate for use after the agency has
completed its design and progressed into developing the system. An
auditor using this guide to review an acquisition that includes
significant software development and has a contract in place should
review appendix II for techniques to help identify variances from
the agency's cost and schedule estimates. In some cases, the
project team may be unable to complete a project on time due to an
unrealistic schedule. The cost models described in appendix II can
be used to make a rough estimate of how long a system development
may require. Such an estimate can be compared to agency plans to
see if the agency has committed itself to unrealistically short
time frames. In other cases, delays in completing scheduled
activities, such as system design, coding, and testing, can lead to
project
Page 12 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Application of Audit Guide to Prototyping Methodology
slippage later on. Auditors will need to tailor the use of the
tools described in appendix II to the circumstances of the
audit.
Appendix III provides a comprehensive list of references cited
throughout the guide, as well as publications that may be useful
for technical information. Finally, a glossary is attached that
defines a wide range of technical and procurement terminology.
This acquisition guide is intended for use in reviewing any
information technology acquisition, regardless of the system
development methodology being used. The guide is structured around
the federal acquisition process, and is independent of development
methods for information systems. An auditor should recognize that
there are different system development models that may be used when
a system development effort is acquired. Such models may include
the "waterfall" model, rapid prototyping, or evolutionary
development. Some of the documents required under different
development approaches may differ. Prototyping, for example, may
act as part of the requirements definition process, helping the
agency identify and control areas of high uncertainty and technical
risk. In this situation the auditor should
(1) focus on determining how one or more prototypesor
incremental versions function to define the agency's requirements
and (2) determine how the system development methodology used by
the agency controls the prototyping process.
One approach to using prototyping as part of the system
development process has been described as a "spiral" model of
system development. 2 The spiral
2For a description of prototyping and the spiral model, see
Roger S. Pressman, Software Engineering: A Practitioner's Approach,
3rd ed.
(New York: McGraw-Hill, Inc., 1992), pp. 26-34.
Presenting Conclusions and Recommendations
model portrays a process in which an agency iteratively (1)
determines its objectives, alternatives, and constraints; (2)
evaluates its alternatives and identifies and resolves risk issues;
(3) develops and verifies its next-level products; and (4) plans
its next phases. Prototypes are developed or modified as part of
the second phase. As part of this model, a prototype is developed
or revised whenever a risk analysis shows that significant areas of
uncertainty remain that pose substantial risks to project success.
When the system has been defined well enough to manage risks
effectively, the agency develops and tests a full-scale system.
In order to review an acquisition that is using prototypes, the
auditor should determine what regulations or guidance the agency
has to define a prototyping methodology. This methodology will
establish the documentation and approval points that agency
officials should meet. The auditor can then measure the progress of
the prototypes against the agency's criteria.
The auditor should conclude the review by identifying
outstanding areas of risk and the agency's actions to address those
risks. Note should be made of how well the agency has addressed the
critical factors in GAO's 1990 model of information technology
acquisitions as well as agency compliance with acquisition
regulations, standards, and other federal guidance. 3 Results of
the review should be communicated to agency officials for their
comment, in accordance with government auditing standards.
The auditor should also recommend changes to ensure that the
agency is properly addressing the critical factors in the GAO
model. For example, the
3GAO/IMTEC-8.1.6, August 1990.
auditor may recommend a greater role for users if they are not
involved in approving alternatives or validating system
requirements. Similarly, the auditor should recommend that the
agency take appropriate actions where senior management involvement
seems lacking, or where the project organization is unstable and
subject to high turnover. The recommendations should be reviewed
with agency officials and be appropriate to the probability and
significance of the risks identified.
Chapter 2
Management and User Support
This chapter focuses on levels of commitment and support for a
planned acquisition by senior managers and users, two key
stakeholders in an acquisition. Senior managers include those who
have overall agency responsibility for strategic, including
information-related, objectives. Users are those who operate or
rely on agency information resources and include the managers and
staff responsible for agency policies and programs supported by the
acquisition.
The involvement and support of senior management throughout an
acquisition is essential for success. Senior management should
envision the agency's acquisition goals, define strategic
objectives, and oversee the projects that implement the overall
vision. One or more senior managers should act as system sponsors,
with sufficient authority to ensure that applicable resources are
available for the project.
Users should also be involved and provide support throughout the
acquisition to ensure that their requirements are understood and
that the resulting system is both accepted and used. User
involvement should be sustained from the needs determination phase
through final acceptance and implementation. User involvement in
the acquisition process will help avoid the development of products
that ultimately do not meet agency requirements.
The audit steps in this section should be used to assess the
potential risks posed by the lack of management or user support. An
acquisition that lacks either or both of these elements is at risk
of incurring unnecessary cost overruns, not meeting its planned
delivery schedule, and not satisfying agency needs.
Audit Objectives
1.
To ensure that senior managers support and areactively
involved throughout the development and implementation of an
acquisition.
2.
To ensure that users support the acquisition byactively
participating in defining procurement requirements, developing the
solicitation document, and verifying that the equipment and/or
services contracted for meet the agency's needs.
• Decision papers, memoranda, or other records of
Documentation
senior management oversight and approval ofRequired acquisition
objectives and plans.
•
Program management directives or other written directives
from senior managers stating goals and objectives of the
acquisition and delegating authority to carry out the
acquisition.
•
Budget exhibits and plans showing that sufficient funding
is committed to the acquisition.
•
Project plans showing the role of users in planning and
overseeing the acquisition. Any documentation of the users' role in
validating acquisition requirements, alternatives, and the
solicitation document. Users' roles and responsibilities may be
detailed in a memorandum of understanding between user
organizations and the program office.
•
Agency policies or guidelines on the structure of
steering committees or other oversight bodies, with
responsibilities of project members and senior managers
delineated.
Audit Steps: Top Management Support
1. Identify senior management officials responsiblefor the
acquisition. Include senior program officials heading the user
organizations, information resource management officials, and
members of senior oversight or steering committees. Determine the
roles
Page 17 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
and responsibilities of any groups or committees of senior
managers, and the relationships between them.
2. Review the documentation related to theacquisition and
determine if the senior managers identified in the acquisition
profile:
a.
Approve the goals and objectives of the
acquisition.
b.
Designate a program sponsor who is responsibleand
accountable for the acquisition.
c.
Establish a formal process to keep concernedparties
appropriately informed.
d.
Participate in the specified reviews and
decisions.
•
Determine how promptly management reviews are conducted
and approvals given and if the approvals are given at the
appropriate levels.
•
Review documentation of such reviews, such as decision
memoranda and review-committee minutes.
•
Determine the frequency of reviews and the quality of
direction senior managers give to project personnel.
e.
Provide initial funding for the project and
establishnear- and long-term funding commitments, periodically
informing Congress of acquisition objectives and status.
f.
Secure any necessary support from key
externalorganizations, such as OMB, GSA, and relevant congressional
committees.
g.
Assign independent officials to ensure that securityand
internal controls needs are met.
3. On the basis of the above audit steps and oncontacts with
other project officials, determine
Page 18 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Audit Steps: User Involvement
whether senior managers fostered good working relations among
the sponsor, acquisition manager (program manager), other top
managers, the technical offices, and the contracting community.
Specifically, determine whether the managers:
a.
Coordinate agreement for developing theevaluation and
source selection plans and gain acceptance of the evaluation
process and criteria.
b.
Coordinate an agreement between the programstaff and
technical and contracting offices for managing the
contract.
c.
Obtain the support of important acquisitionofficials in
the department or agency, such as the official responsible for
source selection.
4. Obtain users' and project staff's evaluations ofmanagement's
support in the above steps. Find out how long it took to get
approvals from management and the direction management gave to
project personnel.
1.
Identify the population of users from projectdocuments
and the agency's organization charts. Review agency criteria
(regulations and procedures) to determine the roles the agency
assigns to users. Determine from the program manager and selected
users which user organizations are actively involved in the
acquisition. Identify significant user groups who are not involved
in the acquisition.
2.
Determine if users:
a. Are involved in periodic reviews, and if so, howfrequently
they are involved.
Page 19 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
b.
Sign off on needs and requirements statements orotherwise
validate the requirements and corresponding solutions when
developed. (For more information see ch. 4, step 3 under Needs
Determination, step 4 under Requirements Analysis, and step 1 under
Specifications.)
c.
Validate alternatives against original requirements.(See
step 1 of ch. 5 for related audit steps.)
d.
Approve the alternative selected (such as thechoice
between off-the-shelf technologies or custom development,
centralized or distributed processing, etc). (See step 1 of ch. 5
for related audit steps.)
e.
Validate the specifications against therequirements. (For
more information see step 4a under Specifications, in ch.
4.)
f.
Provide acceptance criteria.
g.
Assist in preparing the solicitation document andawarding
the contract (for example, user representatives may assist in
developing evaluation criteria and participate in the source
selection team evaluating alternative proposals). (See ch. 7, step
4, and ch. 8, step 1d, for related information.)
h.
Participate in postaward activities that may
includeinstallation, test, and acceptance of equipment.
i.
Participate in reviews of contractor-prepareddeliverable
documents such as design specifications, system analyses, and user
and training manuals.
j.
Participate in government/contractor
workinggroups.
Page 20 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
k. Participate in the postaward audit for assessing thedegree of
success of the acquisition.
3.
Determine if users allocate staff time and otherresources
to the project.
4.
Determine if the users participating in the programhave a
high, moderate, or low turnover rate.
5.
Assess the adequacy of users' funding support tothe
project.
a.
Obtain funding commitment from users.
b.
Identify appropriations to be used and theiravailability
to support contract award.
• Federal Information Resource Management
References
Regulation (FIRMR), Part 201-2: Designated Senior Officials.
• GAO, Information Technology: A Model to Help
Managers Decrease Acquisition Risks
(GAO/IMTEC-8.1.6, August 1990), Phase I, steps 2 and 3; Phase
II, step 1; Phase III, step 1.
Chapter 3
Project Staffing
Project management for an acquisition is accomplished primarily
by a program manager and staff responsible for carrying out project
activities. The program manager should have sufficient authority
and an appropriate mix of skills and experience to successfully
manage the project.
The acquisition project staff should be assigned clear roles and
responsibilities. The team should include members who are skilled
in the information technology procurement process, understand the
technology, and have experience in managing contracts. The team
should also have members knowledgeable about the programs that the
acquisition is to support.
To determine if the acquisition team has theAudit Objective
necessary skills and authority to effectively plan and execute the
acquisition.
• A list of key project team members showing their
Documentation
responsibilities, job titles, and experience. TheRequired
auditor may have to generate this list on the basis of interviews
if one is not available.
• The agency procurement request for a delegation of procurement
authority from GSA, showing names and experience of senior project
officials (as required by GSA guidance detailed in its FIRMR
Bulletin C-5).
Audit Steps: Project Management
1.
Review the agency's criteria for acquisition todetermine
the responsibility and accountability of the program manager and
his/her required qualifications.
2.
Review the documentation related to theacquisition to
determine how clearly the program
Audit Steps: Project Staff
manager's responsibility and accountability are defined.
Determine if the program manager has:
a.
A charter to establish authority, responsibility, and
accountability.
b.
A clearly defined relationship with the program,
technical, and procurement offices.
c.
A clearly defined relationship with the sponsor and
users.
d.
Access to senior agency officials.
e.
The authority to manage acquisition funds.
f.
Input to the budgetary process.
3.
Review the qualifications of the program managerto
determine if the program manager has the appropriate mix of skills
and experience. Has the program manager directed other projects of
similar size and complexity? Is he or she trained to manage complex
acquisitions (GSA's Trail Boss program may be an example of such
training).
4.
Review management continuity on the acquisitionas shown
by the turnover rate of program managers.
1.
Review the agency's acquisition criteria todetermine both
the responsibility and accountability of project personnel as well
as their required qualifications.
2.
Review the make-up of the project team to ensurethat a
mix of appropriate acquisition skills are represented. (See ch. 9,
steps 2 and 3, for related audit steps.)
Page 23 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
a.
Determine the authority and experience level of
theContracting Officer's Technical Representative.
b.
Identify key project staff and review theirexperience and
qualifications. Determine whether the Contracting Officer is
trained and experienced in information technology
acquisitions.
c.
Determine if the project staff is experienced inmanaging
contractors.
d.
Determine the extent of turnover within the
projectstaff.
3.
Determine if the agency trains project staff tomaintain
their skills and qualifications.
4.
Review the reasonableness of project milestonesand
schedule with project team members.
• OMB Circular A-109: Major System Acquisitions.
References
• GAO, Information Technology: A Model to Help
Managers Decrease Acquisition Risks
(GAO/IMTEC-8.1.6, August 1990), Phase I, step 5...i
•
GSA, Overview Guide, pp. 2-3 to 2-7.
•
GSA, Guide for Contracting Officers' Technical
Representatives, Chapter 2.
•
FIRMR Bulletin C-5: Delegation of Procurement Authority
for a Specific Acquisition.
Chapter 4
Needs / Requirements / Specifications
The purpose of this chapter is to guide the auditor in
determining whether the agency has developed an accurate
description of its information technology needs. The acquisition
should be clearly linked to program needs, to the agency's overall
strategies, and to governmentwide policies and standards. The
agency should expand on its basic description of needs to define
specific requirements so providers of information technology can
respond with meaningful solutions. In some cases, an agency may use
prototypes to help define or validate its requirements. The
requirements then form the basis for even more detailed
specifications.
In identifying its requirements, an agency should plan for
testing the information resources it needs. These plans should
cover acceptance, security, and certification requirements. The
test plans developed at this point form the basis for later
evaluations of contract performance.
Failure to clearly and accurately define information technology
requirements poses high risks for any agency. For instance,
improperly defined requirements may preclude alternatives, restrict
competition, increase the risk of cost and schedule overruns, and
lead to systems that are inconsistent with an agency's overall
architecture and incompatible with other agency systems. The
hardware or software purchased may also be inconsistent with
government standards. Designing and implementing a system is also
more difficult if input, output, and processing specifications are
incomplete or inaccurate. In addition, if security and internal
control requirements are not well defined, control over sensitive
information or other assets may be lost.
Page 25 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Chapter 4 Needs / Requirements / Specifications
Audit Objectives
1.
To ensure that the acquisition is based on
clearlyunderstood needs or opportunities and that it is consistent
with the overall strategy and architectures used by the
agency.
2.
To ensure that the agency defines its requirements,based
on the needs identified earlier and validated by functional users,
well enough to support the acquisition of hardware, software,
telecommunications, and system development services. These
requirements should primarily be expressed in functional terms in
accordance with FIRMR policy.
3.
To ensure that system specifications clearly
andaccurately summarize the agency's requirements.
Documentation • Needs statement.
Required • Requirements analysis or functional requirements
document.
•
System specifications, if prepared. Also, draft
specifications with industry comments if draft specifications were
released.
•
Test plan and requirements prepared before contract
award. Test requirements may be summarized in a test and evaluation
master plan.
Audit Steps: Needs Determination
1. Review the agency's stated needs, which may bedocumented in a
Mission Element Needs Statement, Statement of Operational Need, or
System Operational Concept. Determine whether the needs statement
clearly and accurately reflects the users' needs as indicated in
the mission statement and strategic objectives of the users'
organization, the strategic information plan, or the computer
security plan.
Chapter 4 Needs / Requirements / Specifications
Audit Steps: Requirements Analysis
2. Check the needs statement for:
a.
Existing system architecture and functions to besupported
(i.e., description, cost, volume of work, projected
growth).
b.
Justification for changes, such as correctingdeficiencies
in existing capabilities, complying with new or changed program
requirements, or taking advantage of opportunities for increased
economy and efficiency.
3. Contact several users as well as project staff whoare not
users to determine if they generally agree that the needs analysis
presented in the needs statement adequately addresses actual
problems. (See ch. 2 for related questions.)
1.
Review the requirements analysis to determine if
itdescribes the current system. This description should include all
the functions of the existing system that any new system will have
to perform. The users, functions, work load, operating costs, and
components of the current system should also be
identified.
2.
Confirm that the agency has defined its
informationrequirements for the new information resources. These
requirements include:
a.
Information now being received or information thatis
needed but that is not being received.
b.
Information to be provided to or obtained fromother
agencies or the public.
c.
Sources available from which to obtain the
neededinformation.
Page 27 GAO/IMTEC-8.1.4 Assessing Acquisition Risks Chapter 4
Needs / Requirements / Specifications
d.
Information relationships.
e.
The degree of information validation, integrity,
accuracy, completeness, and reliability.
f.
The quantity of information to be processed and types of
output expected.
g.
The timeliness and format of the information.
h.
The security, accessibility, and privacy
requirements.
3. Determine if the agency has defined its functionaland support
requirements, including:
a.
Present and projected work loads and capacity analysis,
including peak load requirements and requirements for future
capacity management.
b.
Privacy and security requirements.
c.
Contingency requirements for resources whose losswould
either prevent or significantly impair the agency from performing
its mission or would have an adverse impact on the
nation.
d.
Records management factors relating to integrationof
electronic records with other agency records, records retention and
disposition, and safeguards against unauthorized use or destruction
of records.
e.
Space and environment factors, such as floorloading, heat
dissipation, and power supply.
f.
Federal standards with which the new technologymust
comply.
g.
Organizational training needs.
Page 28 GAO/IMTEC-8.1.4 Assessing Acquisition Risks Chapter 4
Needs / Requirements / Specifications
h.
Interfaces with other systems.
i.
User interface requirements.
j.
Compatibility limitation requirements.
k.
Capability or performance validation
requirements.
4. Determine whether the requirements analysisprovides for:
a.
A methodology for having users validate therequirements
analysis for both capability and performance. (See ch. 2 for
related questions).
b.
Measurable requirements that may be used later toverify
system effectiveness.
5.
Determine if the requirements are presented infunctional
or performance terms in consideration of full and open competition.
Functional requirements promote full and open competition, while
performance requirements may not.
6.
With regard to restrictive requirements (which donot lead
to full and open competition), determine:
a.
If brand-name-or-equal or specific make and
modelrestrictions are appropriately justified.
b.
Whether all required justifications are completed and
approved for other compatibility-limited requirements.
(See ch. 6, step 3, for more information on procurements that do
not promote full and open competition).
Page 29 GAO/IMTEC-8.1.4 Assessing Acquisition Risks Chapter 4
Needs / Requirements / Specifications
Audit Steps: Specifications
7. With regard to the process of revising requirementsduring the
acquisition, determine:
a.
Whether a core of basic requirements has beenidentified
in order to maintain project scope.
b.
If a formal change control process has beenestablished to
manage changes when necessitated by a changing
environment.
c.
Who is responsible for reviewing and approvingchanges to
requirements.
d.
How often requirements have been changed.
e.
Whether proposed new requirements are validatedagainst
mission needs.
f.
What process is used to analyze the impacts ofchanges on
the other elements of the requirements.
1.
Determine whether functional users and/or theprogram
manager confirmed that the specifications accurately reflect the
requirements and conform to the approved acquisition strategy
discussed in the acquisition strategy module. Did users or the
program manager sign off on the system specifications? (See ch. 2
for related questions.)
2.
Examine the specifications document for:
a.
A summary of the functional requirements to besatisfied
by the technology.
b.
Performance evaluation requirements.
c.
Performance requirements that addressinformation
accuracy; data integrity requirements;
Page 30 GAO/IMTEC-8.1.4 Assessing Acquisition Risks Chapter 4
Needs / Requirements / Specifications
timing for response, update processing, information transfer,
transmission, and throughput; and flexibility to changes in the
requirements.
d.
An identification of new types of equipmentrequired
(e.g., processors, input/output devices, or information
transmission devices).
e.
An identification of support and test
software.
f.
A description of interfaces.
g.
A description of overall security and
privacyrequirements.
h.
A description of the operational controls
needed.
i.
A description of the operating characteristics of theuser
and computer centers where the software will be used.
j.
A description of the logic flow of the entire
system.
k.
A specification of the functions to be satisfied bythe
software.
3.
Determine how restrictions to full and opencompetition in
the specifications (e.g., equipment characteristics and performance
elements) are handled. Ensure that all required justifications are
completed and properly approved for such restrictions. (See ch. 6,
step 3, for more information on procurements that do not promote
full and open competition.)
4.
Determine how changes to the originalspecifications are
handled.
Page 31 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Chapter 4 Needs / Requirements / Specifications
Audit Steps: Test Plans
a.
Establish responsibility for reviewing and approving
changes to specifications. Interview users to determine whether or
not they actually reviewed and approved changes to specifications.
(See ch. 2, step 2e, under User Involvement.)
b.
Review documentation on change requests.Determine how
often specifications are changed, whether new specifications are
validated against requirements, and what process is used to
identify impacts of changes on other elements of
specifications.
5. Determine if feedback (comments and questions)from users and
industry is accounted for, considered, and incorporated as
appropriate on a continual basis. (See ch. 7, step 6, for related
information.)
1.
Determine if the agency has developed test plansbased on
acceptance criteria furnished or validated by the users.
2.
Verify that test plans incorporate security
andcertification requirements.
3.
Determine if test plans adequately measure
systemperformance requirements to be specified in the request for
proposals (RFP).
(Note: Refer to ch. 10 for more information on test plans.)
• FIRMR 201-20.1: Requirements Analysis.
References
•
FIRMR 201-20.303: Standards.
•
Federal Acquisition Regulation (FAR) Part 6: Competition
Requirements.
Page 32 GAO/IMTEC-8.1.4 Assessing Acquisition Risks Chapter 4
Needs / Requirements / Specifications
•
FAR Part 10: Specifications, Standards, and Other
Purchase Descriptions.
•
GSA, Guide for Requirements Analysis and
Analysis
of Alternatives, Chapter 2: Requirements Analysis.
•
American National Standards Institute/Institute for
Electrical and Electronic Engineers (ANSI/IEEE) Standard 830: IEEE
Guide to Software Requirements
Specifications.
•
GAO, Information Technology: A Model to Help
Managers Decrease Acquisition Risks
(GAO/IMTEC-8.1.6, August 1990), Phase I, steps 1, 6, 11, 12, 13,
14.
• Federal Information Processing Standards (FIPS) Publication
64: Guidelines for Documentation of
Computer Programs and Automated Data Systems for
the Initiation Phase.
• FIPS Publication 101: Guideline for Lifecycle
Validation, Verification, and Testing of Computer
Software.
Chapter 5
Alternatives
Audit Objectives
After identifying its requirements, the agency should assess
alternatives for cost-effectively meeting those requirements. The
approach selected should reflect an understanding of what is
available in the commercial market as well as what is available
within the government. Approaching the acquisition this way will
lessen, but not eliminate, the risk that an agency may select an
alternative that does not fully meet user requirements or that is
unnecessarily complex and expensive.
1.
To determine if the agency has considered allreasonable
alternatives for meeting its needs.
2.
To determine if the agency identified the risks,costs,
and benefits of each alternative.
3.
To verify that the agency selected an
alternativebalancing expected benefits against costs, time, and
risks of failure.
Documentation Required
•
Record of alternatives analysis, such as a system
decision paper. Economic and risk analyses should accompany or be a
part of the decision paper.
•
Market survey research conducted to identify alternatives
for meeting user needs and to support cost estimates.
•
Findings and approvals statements to support restrictions
on specifications, such as compatibility-limited
requirements.
•
Cost/benefit analysis to justify the selection of the
alternative selected over other alternatives, in dollar terms or in
terms of some other criteria, such as effectiveness.
1. Assess the involvement of responsible parties andAudit Steps
verify whether:
a.
Users agreed with the range of alternativesconsidered and
were involved in validating those alternatives against the original
requirements. (For related information on this and the next point
see ch. 2, steps 2c and 2d, under User Involvement.)
b.
Users agreed with the alternative finally
selected.
c.
Appropriate senior management approved thealternative
selected. (See ch. 2, step 2, under Top-management Support, for
more information.)
d.
The Contracting Officer or other contracting personnel
participated in the alternatives analysis to ensure that a feasible
acquisition approach was selected.
e.
Project staff conducted market surveys to determine how
industry can best meet the agency's requirements.
2.
Ensure that the agency considered, as appropriate,the
alternatives included in GAO's acquisition model and FIRMR
201-20.203-1.
3.
Assess how the agency evaluated alternatives
bydetermining whether:
a.
The agency consistently analyzed alternatives usingthe
same criteria for each alternative.
b.
The alternatives are described in sufficient detail to
support time and cost estimates and cost/benefit
analyses.
c.
The alternatives considered fit within the agency's
information architecture.
d.
The range of alternatives considered was restricted by
resource assumptions (staff or funding limitations).
4. Determine whether the agency consistentlyanalyzed the costs
and benefits of each alternative. Ensure that the economic analysis
includes present values for costs and benefits and is updated
periodically. Identify the system life used as a basis for
evaluating alternatives and determine whether it appears realistic
in light of user needs, expected changes in the technology,
expected availability of maintenance and other support, and the
time needed to prepare subsequent acquisitions. Verify that the
economic analysis includes a sensitivity analysis to identify
factors that affect the choice of one alternative over another. The
costs and benefits considered for each alternative should
include:
a.
Conversion costs.
b.
Personnel costs.
c.
Operation and maintenance costs.
d.
Nonrecurring but quantifiable benefits in terms of
information processing, administration, and support (these may
include cost reductions resulting from improved system operations
or value enhancement through improved use of resources).
e.
Recurring and quantifiable benefits on a monthly and/or
quarterly basis over the system life from reductions in such items
as salaries, fringe benefits, supplies, utilities, and space
occupancy.
Page 36 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
f. Any nonquantifiable benefits, such as improved service and
enhanced organizational image.
5. Determine whether the agency analyzed thefollowing noncost
factors for each feasible alternative:
a.
Obsolescence: strategies for avoiding outdatedresources
over the system life. A technology upgrade clause is one way to
avoid obsolescence by allowing an agency to buy advanced versions
of equipment or software when they become available.
b.
Availability: to what extent the system will beavailable
to users.
c.
Reliability: how frequently the system requirescorrective
maintenance.
d.
Maintainability: the ease with which failed
systemcomponents can be repaired, taking into account the level of
service, personnel support, and supplies needed.
e.
Expandability: the ease with which the system canbe
enhanced to meet anticipated growth.
f.
Flexibility: the extent to which the alternative can
accommodate changes in the nature of the work load.
g.
Security: the ability to prevent unauthorized access and
tampering and consideration of national security and emergency
preparations.
h.
Privacy: the extent to which the privacy
ofpersonnel-related data can be maintained.
Page 37 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
i.
Affect on personnel: the impact on the level of support
personnel needed, including the skills required.
j.
User acceptance: the overall impact on the user
community, including the amount of change to user
procedures.
k.
Accountability: the ability of the alternative toallow
system activity to be tracked and measured.
6.
Review the risk analysis to see if it identifiessensitive
data and vulnerabilities. Verify that the magnitude of each
vulnerability has been stated. Determine whether or not the risk
analysis conforms with the standards identified in FIPS
Publications 65 and 73 and OMB Circular A-130.
7.
Verify that each alternative is evaluated forfinancial,
technical, and schedule risks. Financial risk refers to the extent
to which each alternative is subject to unexpected additional
costs. Technical risk indicates the probability that each
alternative's technical objectives will prove difficult to achieve
in whole or in part. Schedule risk is the extent to which each
alternative is subject to unexpected schedule delays and slippage
in meeting the system's technical objectives, regardless of
cost.
8.
Ensure that the agency has selected the mostadvantageous
and realistic alternative with respect to benefits, costs, and
risks (based on steps 4 and 7).
9.
Verify that users and senior managers approve anychanges
to the planned scope of the project.
• FIRMR Part 201-20.2: Analysis of Alternatives.
References
• GAO, Information Technology: A Model to Help
Managers Decrease Acquisition Risks
(GAO/IMTEC-8.1.6, August 1990), Phase I, steps 7, 8,
9.
•
GSA, Guide for Requirements Analysis and
Analysis
of Alternatives, Chapter 3: Analysis of Alternatives.
•
OMB Circular A-130: Management of Federal Information
Resources.
•
OMB Circular A-109: Major System Acquisitions.
•
OMB Circular A-76: Performance of Commercial
Activities.
•
FIPS Publication 64: Guidelines for Documentation
of
Computer Programs and Automated Data Systems for
the Initiation Phase.
•
FIPS Publication 65: Guidelines for Automatic
Data
Processing Risk Analysis.
•
FIPS Publication 73: Guidelines for Security
of
Computer Applications.
Chapter 6
Acquisition Planning
Acquisition planning is the process of coordinating and
integrating the efforts of personnel responsible for acquisitions.
A major objective of acquisition planning is to promote and provide
for full and open competition. To ensure that the planning is
accomplished in an effective, economical, and timely manner, the
agency should prepare an acquisition plan containing an overall
strategy for managing the preaward, acquisition, and postaward
phases.
An effective acquisition plan is critical to project success.
The plan sets out what the agency will do to complete a procurement
and how it will do it. The plan also specifies the type of contract
that will be awarded, how the agency will select a contractor, cost
and schedule goals, milestones, significant risk areas, and
contract management controls.
Auditors should assess the extent to which the agency's
acquisition planning is realistic and comprehensive. One part of
this assessment should be the review of the Agency Procurement
Request (APR) to ascertain whether it is complete and accurately
reflects the objectives and scope of the project.
To verify that the agency has defined an effectiveAudit
Objective strategy and plan for selecting a contractor and managing
contract performance.
• Acquisition plan and related documents as
Documentation
appropriate, such as a plan of action and milestones.Required •
Agency procurement request and other correspondence with GSA.
1. Determine whether the acquisition plan was
Audit Steps reviewed and approved in a timely manner by the
officials designated in the agency's acquisition regulations.
Ensure that the program manager periodically reviews the
acquisition plan and updates it when necessary.
2. Review the acquisition plan and determine if itcontains the
elements required by the FAR Section 7.1 and GAO's Information
Technology: A Model to Help
Managers Decrease Acquisition Risks. These elements
include acquisition objectives; cost goals; responsible
decision-makers; capability or performance characteristics; risks
associated with technical matters, scheduling, and costs; plan of
action; competition; source selection procedures; contract type and
special contract provisions; contract management procedures or
organization; budget and funding; information needed to monitor
contractor performance; test and evaluation; security and privacy;
and acquisition milestones. The plan should also identify the
acquisition method, key "go/no-go" points, a formal training plan,
and a contingency plan to minimize losses.
3.
Determine if the acquisition plan calls for full andopen
competition. If the plan calls for limiting the acquisition to
resources compatible with existing equipment, verify that
conversion cost studies have been completed to justify
compatibility restrictions. If the plan calls for other than full
and open competition, verify that restrictions on competition, such
as make and model restrictions or sole source requirements, have
been justified and approved by the designated authority. (See ch. 4
step 6, under Requirements and step 3, under
Specifications.)
4.
Determine whether the agency has planned a"grand design"
project or organized the acquisition
Page 41 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
into modules. Incremental purchasing may limit risks by
identifying problems earlier, which allows for easier change or
correction.
5.
Evaluate the project management tools andtechniques to
satisfy management information requirements for monitoring
contractor performance, tracking progress against the acquisition
plan, and taking action on cost or schedule slippage.
6.
Review the agency's schedule for developing
thesolicitation document and for source selection activities.
Assess the reasonableness of the schedule through discussions with
procurement officials and reviews of project progress.
7.
Identify the dollar limit of the agency's
generaldelegation of procurement authority from GSA. Confirm that
the agency receives a specific delegation of authority from GSA if
the value of the acquisition exceeds the agency's authority level.
Confirm that the new delegation of procurement authority was
received before a solicitation document is issued or a contract
awarded.
8.
Review the APR if the acquisition exceeds theagency's
delegated procurement authority level. Determine if the APR
identifies the officials responsible for managing the effort, in
accordance with the GSA guidance provided in FIRMR Bulletin C-5.
The APR should include:
a.
Names and titles of senior project officials, with
adescription of their roles in the organization. If the acquisition
exceeds $25 million, the APR should also describe the project
manager's experience in previous acquisitions, responsibilities,
and scope of authority, and the reporting structure for each
official as well as whether each official is assigned full- or
part-time to the acquisition.
b.
The project title and a brief description of the
acquisition.
c.
Information resources currently in use.
d.
Resources to be acquired.
e.
The contracting approach. The approach shouldinclude any
limitations on competition, the planned dates for release of the
solicitation and for contract award, and a strategy for a follow-on
implementation if a prototype is to be used.
f.
Estimated contract life and contract cost.
g.
Completion dates for key project documents.
• FAR Part 7: Acquisition Planning.
References
•
FAR Part 34: Major System Acquisition.
•
GSA, Overview Guide, p. 4-3.
•
OMB Circular A-109: Major System Acquisitions.
•
GAO, Information Technology: A Model to Help
Managers Decrease Acquisition Risks
(GAO/IMTEC-8.1.6, August 1990), Phase I, step 10.
• FIRMR Bulletin C-5: Delegation of Procurement Authority for a
Specific Acquisition.
Page 43 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Chapter 7
Solicitation Document
A solicitation document provides information necessary for
vendors to propose equipment, software, and services to meet the
agency's requirements. In most cases, information resources will be
purchased by issuing an RFP, which forms the basis for the
resulting contract. Less commonly, an agency may acquire
information resources using an invitation for bids. An RFP may be
preceded by a request for information or request for quotation.
An RFP should be clear and comprehensive and include the
elements described in GSA's guidance on standard solicitation
documents. 1 The areas of most interest to auditors include section
C: Description/Specifications/Work Statement, section
E: Inspection and Acceptance, and section M:Evaluation Factors
for Award. Section C describes the tasks to be performed by the
contractor and the products to be delivered. Section E sets out
government and contractor responsibilities in ensuring that
contract deliverables are acceptable to the agency. Section M
explains how the agency intends to select a winning contractor by
describing the importance of all factors to be considered in
evaluating proposals.
In developing the RFP an agency may hold presolicitation or
preproposal conferences in order to seek industry views on the
planned acquisition and to encourage companies to offer proposals.
Once the RFP is developed, it may be released in draft form in
order to obtain industry questions and reactions. Auditors should
determine what steps the agency has taken to get feedback on its
requirements, how the agency has handled comments or questions on a
proposed RFP, and whether the agency has acted to ensure that
contractor proposals are competitive.
1See, for example, U.S. General Services Administration,
Information Resources Management Service, Overview Guide:
Acquisition of Information Resources, (Jan. 1990).
A proper RFP is a critical element of a successful acquisition
because it becomes part of the binding contract once a proposal is
made and accepted. If the RFP does not accurately and clearly
describe the agency's requirements, or if the evaluation factors do
not accurately reflect the agency's priorities, then the resulting
acquisition may not meet user needs. If the RFP is unjustifiably
restrictive, favoring one contractor over others, the agency may be
unable to benefit from full and open competition.
Auditors should become familiar with the standard format for an
RFP. The audit team should include persons sufficiently
knowledgeable about the information technology being purchased to
judge how well the agency has defined its requirements in the RFP.
Team members should include or have access to people who can
identify areas where the RFP does not define the agency's needs
well enough to protect the government's interests. These persons
should also know enough about performance or capability validation
techniques to determine whether or not the agency's requirements
are reasonable and effective.
Audit Objectives To determine whether or not the solicitation
document is complete, clear, and consistent, verify that
requirements continue to reflect user needs, and determine if the
proposed evaluation process will result in an effective and
economical acquisition.
Documentation Required
•
Record of presolicitation or preproposal
conference.
•
Solicitation document: RFP or invitation for bid. Draft
RFPs and Requests for Information, if any were issued.
•
Report of a solicitation review panel or committee if
appropriate.
•
Source selection plan.
•
Benchmark materials or other capability and performance
validation requirements.
•
Vendor comments or questions on solicitation document. If
a vendor protests the solicitation determine the basis of the
protest and how it was resolved.
•
Proposal evaluation guide.
Audit Steps 1. Determine whether the solicitation document
contains:
a.
A statement of work or specifications statementthat
clearly and accurately describes the government's requirements,
including a clear definition of all deliverables and the conditions
of their acceptability.
b.
A clear definition of government and contractor
responsibilities.
c.
The relative importance of evaluation factors.
d.
A proposal format requiring that cost and
technicalelements be separated.
e.
Reasonable provisions that protect the agency(such as
liquidated damages provisions) or give incentives to the contractor
(such as bonuses for good performance). Identify option clauses
that create uncertainty in work-load projections.
2. Examine the evaluation criteria to ensure:
a.
They are consistent with the requirements
analysis,specifications, and proposal preparation
instructions.
b.
They provide all the factors and significantsubfactors to
be considered in evaluating offers and the relative importance of
different technical or cost factors, in accordance with FAR
15.605.
3. Evaluate the performance evaluation package theagency will
use. Determine whether the agency reimburses contractors for
participating in benchmarking or other testing, and whether the
cost of such performance evaluation efforts constitutes a barrier
to competition.
a.
If there is a benchmark, has the benchmark been
independently examined by some outside organization? Review the
benchmark plan to confirm that demonstration criteria are clearly
stated. Determine how the agency selected a representative mix of
programs for the benchmark. Determine if the complexity of programs
in the benchmark is representative of the projected work load.
Determine how the agency validated the benchmark as representative
of the agency's work load.
b.
If there is simulation or modeling, determine how the
agency selected parameters for the model. Review any concerns or
complaints raised by vendors.
c.
If a compute-off (demonstration of prototypes) is to be
used, confirm that the agency has established plans for prototype
and follow-on contracts.
4. Interview users and managers, if necessary, todetermine
whether they concur with the solicitation. Determine whether users
or managers identify new or changed requirements not included in
the RFP. Find out if there are any factors considered important for
selecting an offer that are not included in the
Page 47 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
evaluation criteria or if other included factors unfairly
restrict the competition. (For related information see ch. 2, step
2g, under User Involvement.)
5.
Review the agency's source selection plan to ensurethat
it clearly describes the source selection organization and
activities. Ensure that the source selection plan has been approved
by the source selection authority before presolicitation
conferences are held or the solicitation document is issued, in
accordance with FAR 15.612.
6.
Review the industry feedback process anddetermine
if:
a.
Comments received on the draft solicitationidentify any
need for clarification, restrictive specifications, or alternative
ways of satisfying user needs. (See ch. 4, step 5 under
Specifications, for a related question.)
b.
The agency responded promptly and thoroughly tothe
comments received. (Judgmental sampling techniques may be required
in this section if the number of vendors and vendor comments are
substantial.)
c.
The feedback provided adequate comments onproduct
availability in the market.
d.
The use of an ombudsman facilitated the process of
addressing vendor concerns, disputes, and grievances.
7.
Determine who has performed legal reviews of
thesolicitation document.
8.
Determine if any vendor submitted a protest, towhom (the
agency, GAO, or GSA's Board of Contract
Page 48 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Appeals), and how it was resolved. Determine the basis of the
protest and its resolution.
• GSA, Overview Guide, Chapter 6.
References
•
FAR, Part 15: Contracting by Negotiation,
Subpart
15.4: Solicitation and Receipt of Proposals andQuotations;
Subpart 15.612: Formal Source Selection; Subpart 15.605: Evaluation
Factors.
•
FAR Part 14: Sealed Bidding.
•
FAR Part 5: Publicizing Contract Actions.
•
FAR Part 34: Major System Acquisition.
•
GAO, Information Technology: A Model to Help
Managers Decrease Acquisition Risks
(GAO/IMTEC-8.1.6, August 1990), Phase II, steps 2 through 7.
• FIPS Publications 75 and 42-1.
Chapter 8
Source Selection
The source selection process is critical to securing the best
value for the government. All proposals must be evaluated in
accordance with the criteria published in the RFP. If the
evaluation process does not conform to the RFP the agency runs a
greater risk of a successful bid protest by losing vendors. The
agency should receive proposals, evaluate the technical and cost
merits of different proposals, negotiate with contractors, and
award a contract in accordance with a source selection plan
developed before release of the RFP.
The auditor should be aware of the agency's organization and
procedures for making a contract award. Agencies may use some or
all of the following positions:
•
Contracting Officer (CO). The Contracting Officer
publicizes the procurement, amends the RFP if necessary, and
conducts all negotiations with offerors.
•
Source Selection Authority (SSA). The SSA makes the final
decision on contract award. The Contracting Officer may be the SSA
for some procurements, while in other cases a more senior manager
may serve as SSA.
•
Source Selection Advisory Council (SSAC). The SSAC
reviews the evaluations of different proposals and makes a
recommendation to the SSA on contract award.
•
Source Selection Evaluation Board (SSEB). The SSEB
conducts technical and cost evaluations of vendor
proposals.
To ensure that the source selection process isAudit Objective
planned and carried out in order to successfully reach a contract
that gives the best value to the government.
Page 50 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
• Source selection plan, including source selection
Documentation
organization.
Required • Lessons learned report or other report by the
Contracting Officer describing negotiations and selection
activities.
•
Contracting Officer's contract file.
•
Records of debriefings, if any.
•
Results of benchmarks or other performance and capability
validation techniques used.
•
Correspondence between offerors and the agency regarding
questions or clarifications and any amendments to the
RFP.
•
Proposal evaluation guide.
•
Preaward survey reports.
Audit Steps
1. Examine the evaluation process by reviewingrecords of the
source selection procedures and determine:
a.
If evaluation personnel strictly adhered to andapplied
the publicized evaluation process and criteria in the
solicitation.
b.
If evaluation factors were applied that were notlisted in
the solicitation.
c.
Whether cost and technical evaluations were done
separately.
d.
The role users played in the evaluation process. (See ch.
2, step 2g, for related question.)
e.
If the process resulted in (1) the establishment of a
competitive range and (2) removal of offerors from further
consideration, in accordance with FAR 15.609.
2.
Determine how many vendors received thesolicitation and
how many submitted proposals.
3.
Determine if any vendor submitted a protest and, ifso,
whether the protest was handled by the agency, GAO, or GSA's Board
of Contract Appeals. What was the basis of the protest? How was the
protest resolved?
4.
Determine whether the Contracting Officerestablished
prenegotiation objectives for cost, profit and fee, and other
issues in accordance with FAR 15.807. These objectives should help
determine the overall reasonableness of proposed prices, and may be
based on an independent government cost estimate and other
information, such as a field pricing report on each contractor's
proposal.
5.
Determine if the agency obtained field pricingsupport in
accordance with FAR 15.805-5. Was a preaward audit of the cost
proposal obtained and used during negotiations? Were the offeror's
proposed rates compared with the direct, indirect, overhead, and
general and administrative rates recommended by the appropriate
contract audit activity?
6.
Examine the negotiation process.
a.
Confirm that discussions with all vendors in the
competitive range were held and the proceedings
documented.
b.
Determine from a review of documentation the controls
that existed to protect the security of sensitive
information.
c.
Contact responsible officials for their evaluationsof
security.
Page 52 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
d.
Determine if technical leveling or technicaltransfusions
occurred that would change vendors' proposals. FAR 15.610 describes
these and other prohibited actions, such as indicating a cost or
price that offerors must meet to be considered for contract award
or informing a vendor of its price standing relative to others
seeking the contract.
e.
Determine if estimated life cycle costs werereconciled
with each vendor's proposal to ensure that cost estimates appear
realistic.
7. Examine how the agency handled best and finaloffers.
a.
Determine if the agency made multiple calls forbest and
final offers, justified in accordance with FAR 15.611.
b.
Determine if the best and final offers were solicitedand
evaluated in accordance with the source selection plan.
8. Determine if all debriefings:
a.
Were scheduled as soon as possible whenrequested by the
vendor.
b.
Were based on a debriefing plan that addressed
andresolved issues likely to cause concern and complaints among the
losing vendors.
c.
Were documented.
d.
Adequately explained why the losing vendor(s) lostthe
contract. (Note: In the explanation, the agency cannot make
point-by-point comparisons with other proposals, but can point out
the government's evaluation of significantly weak or deficient
elements in the proposal of the vendor being debriefed. Refer to
FAR 15.1003 for more information.)
9. Examine the effort made to identify lessonslearned.
a.
Determine what policies or procedures the agencyusers
have to evaluate acquisition results and communicate "lessons
learned" to staff conducting future assignments and to other
agencies. Do these include a comparison of the preaward activities
to the acquisition plan?
b.
Review the lessons learned report, if one wascompleted,
to determine how agency officials assessed the contracting
process.
• FAR Part 15: Contracting by Negotiation:
References
Subpart 15.4: Proposals and Quotations. Subpart 15.6: Source
Selection. Subpart 15.8: Price Negotiations. Subpart 15.10:
Notifications, Protests, and Mistakes.
•
FIRMR Part 201-39: Acquisition of Federal Information
Processing Resources by Contracting.
•
GSA, Overview Guide, Chapter 7: Source
Selection.
•
GSA, Guide for Acquiring Commercial Software.
•
GAO, Information Technology: A Model to Help
Managers Decrease Acquisition Risks
(GAO/IMTEC-8.1.6, August 1990), Phase II, steps 8 through
13.
Page 54 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Chapter 9
Contract Management
Contract management includes the steps required to ensure that
the agency receives products and services within established costs
and time frames. An agency is required to monitor contractor
performance, ensuring that work done conforms to the agency's
requirements. The agency must also control changes to the contract
and accept or reject contract deliverables. Finally, an agency
should conduct postimplementation reviews to determine how well
acquisition goals were met and whether the information resources
acquired should be added to or replaced.
The agency's Contracting Officer and Contracting Officer's
Representative/Contracting Officer's Technical Representative hold
primary responsibility for administering the contract. The
Contracting Officer monitors costs as required by the contract type
(fixed-price or cost-reimbursable) and makes contract modifications
as needed. The program manager helps monitor contractor performance
to ensure that user requirements are met by the products or
services delivered and that senior officials provide support and
oversight.
A contract consists of the agency's RFP, as amended, and the
successful vendor's proposal. The contract should specify all
deliverables required from the vendor. The Contracting Officer and
Contracting Officer's Representative/Contracting Officer's
Technical Representative should ensure that deliverables are
received as required. Any status and cost reports required from the
contractor should be reviewed and action taken to correct problems,
if necessary. Training, documentation, and maintenance requirements
should be fulfilled.
Page 55 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Audit Objectives
To ensure that the agency:
1.
Oversees contractor performance.
2.
Ensures that contract requirements continue toaccurately
reflect user needs.
3.
Verifies that products and services delivered meetuser
needs.
4.
Implements configuration management.
5.
Modifies the contract only as needed.
6.
Enforces contract provisions intended to protectthe
agency, such as warranties or liquidated damages
clauses.
• Agency regulations or directives specifying
Documentation
requirements for periodic reviews, managementRequired oversight,
and configuration management.
•
The contract as awarded and with
modifications.
•
The agency's contract management organization and
structure.
•
Current status reports and cost or schedule
projections.
•
Current budget reports.
•
The configuration management plan for the
project.
Audit Steps
1.
Review agency directives to identify the
agency'srequirements for contract oversight. Department of Defense
Standard 2167A, for example, requires periodic reviews of contract
deliverables, with cost and status reports from the contractor.
Defense also has directives governing configuration management
activities to ensure that the contractor delivers the equipment or
services called for and that no changes
to the contract are made without consideration of their overall
impact.
2.
Identify the roles of users and senior managers
inmonitoring the contract, verifying that both users and senior
managers are involved in managing the contract and approving any
changes.
3.
Determine the authority and experience level of
theContracting Officer's Representative or Contracting Officer's
Technical Representative. (See ch. 3, step 2, for a related
step.)
4.
Evaluate the project staff.
a.
Identify key project staff and review theirexperience and
qualifications. Is the Contracting Officer trained and experienced
in information technology procurement? Does the project include
staff experienced in managing contractors?
b.
Determine how much turnover there has beenwithin the
project staff, including the project manager. (See ch. 3, step 2,
for a related step.)
5.
Assess changes to the agency requirements toensure that
the contract continues to reflect valid user needs. Review
configuration management activities to verify that changes to
requirements are recorded and controlled and that impacts of
changes to contract requirements are identified.
6.
Assess changes to the agency's cost and
scheduleestimates. Are variances in cost and schedule projections
tracked by the project manager or Contracting Officer's Technical
Representative? Are cost and schedule estimates changed
appropriately?
Page 57 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
7. Determine if agency monitoring of the contractor'sperformance
includes:
a.
Periodically reviewing both scheduled and completed
deliverables and effectively reacting to any delays. Determine how
the agency compares contractor progress with the contract work
schedule.
b.
Periodically reviewing contract reports. Review a sample
of status and cost reports to verify that they are regularly
submitted by the contractor as required by the contract. Discuss
their usefulness with the project staff.
c.
Assessing the adequacy of the contractor's
qualityassurance process.
8. Assess the effectiveness of the agency's workingrelationship
with the contractor.
a.
Verify that the agency has controlled changes to
thecontract and integrated the change process into the acquisition
management structure. Determine the impact of changes on contract
cost and schedule.
b.
Determine if corrections are made, awards are
implemented, and damages assessed, as appropriate.
9. Determine if the agency controls contractmodifications
by:
a.
Requiring the contracting office to approve allcontract
modifications.
b.
Establishing a review process to ensure thatproposed
engineering changes are within the scope of the
contract.
Page 58 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
c. Regularly comparing contract expenditures withthe delegation
of procurement authority to ensure that the agency does not exceed
its authorized level of total expenditures.
• GSA, Overview Guide, Chapter 8: Contract
References
Administration.
•
GSA, Guide for Contracting Officers' Technical
Representatives, Chapter 2.
•
GAO, Information Technology: A Model to Help
Managers Decrease Acquisition Risks
(GAO/IMTEC-8.1.6, August 1990), Phase III.
Chapter 10
Test and Acceptance
Testing provides the basis for making decisions on whether to
accept contract deliverables. For testing to be effective, it must
be addressed relatively early in the acquisition so it can be
properly included in planning. Test plans provide testing
procedures and the evaluation criteria to assess results of the
testing.
An agency should establish its initial test plans in the
presolicitation phase. These plans should show how the agency will
verify that the acquired equipment, software, or services meet user
needs and satisfy security requirements. After a contract is
awarded the agency will need to carry out test and acceptance
procedures. The auditor should ensure that test planning is
conducted early enough so that test requirements are included in
the contract.
In assessing the postaward phase, the auditor should ensure that
the agency has not accepted equipment or software that does not
meet its requirements. The contract should specify conditions for
acceptable performance. For example, the contract may require that
a computer operate successfully for 30 consecutive days out of a
90-day test period. Agency personnel should ensure that the
contractor fully meets the conditions for acceptable performance.
Internal auditors may be required to verify that the equipment or
software pass the specified tests.
Assessing the test and acceptance phase may require a high level
of technical skill on the part of auditors, such as when an agency
has contracted for software development services and must test the
quality of delivered software. The auditor should be able to
understand the system requirements, development methodologies, and
test tools being used.
Page 60 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
To confirm that the agency has:
Audit Objectives
1.
Fully defined its requirements for testing thetechnology
to be purchased.
2.
Effectively carried out test and acceptanceprocedures to
verify that the resources purchased meet the agency's
needs.
• Agency directives giving requirements for cost and
Documentation
status reporting, configuration management, andRequired
management oversight.
•
Records of configuration reviews or other progress
reports.
•
Trouble reports or other records of deficiencies found by
agency personnel.
•
Records of product acceptances.
•
Test plans for inspection and acceptance.
Audit Steps
1. Determine if test plans were developed todetermine whether
the mandatory:
a.
Functional requirements were satisfied.
b.
Security requirements arising from governmentalpolicy,
agency mission needs, and specific user needs were
satisfied.
2. Determine if the test plans include:
a. Types of testing.
•
Unit testing-e.g., in software, individual code modules
are tested by the programmer who wrote them.
•
Integrated testing-e.g., in software, aggregate functions
formed by groups of modules and intermodule communication links are
tested.
•
System testing-examines the operation of the system as an
entity in an actual or simulated operating environment.
b.
The locations for testing.
c.
A realistic testing schedule.
d.
The resource requirements.
•
Test equipment needed, including the specific period of
use, types, and quantities needed.
•
Software needed to support the testing.
•
Personnel from both user and acquisition groups with
their needed numbers and skills specified.
e. Testing materials to be used.
•
Documentation needed, such as source code and
manuals.
•
Software to be tested and its medium.
•
Test inputs and sample outputs.
•
Test control software and worksheets.
f. Training in testing to be given, personnel to betrained, and
the training staff.
3.
Determine whether criteria have been establishedfor
certifying that security requirements are met.
4.
Determine whether the appropriate userrepresentative has
formally acknowledged the completion of testing and acceptance of
the system. If not, determine the reasons and the potential
impact.
Page 62 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
a.
Determine if deficiencies discovered in
contractdeliverables are expeditiously resolved.
b.
Determine if any requirements that were not metby the
delivered hardware, software, and telecommunications are still
pending and why.
5. Interview system operators and users to determineif the
system has been successfully integrated into the existing
environment.
• GSA, Overview Guide, chapter 9: Installation and
References
Operation.
• GAO, Information Technology: A Model to Help
Managers Decrease Acquisition Risks
(GAO/IMTEC-8.1.6, August 1990), Phase I, step 14; Phase III,
steps 4 to 6.
• FIPS Publication 101: Guideline for Lifecycle Validation,
Verification, and Testing of Computer
Software.
Appendix I
Acquisition Profile
The acquisition profile, which is a mechanism for documenting
key information about an acquisition under review, is used to help
auditors plan and conduct assessments of an acquisition. The
profile, which is kept available for future reference, includes
the
• overall characteristics of the acquisition including its
objectives shown within the context of the mission(s) or
function(s) to be supported,
•
management organization and staffing of the acquisition
project, and
•
acquisition schedule and cost estimates.
1. What is the project's name?
Overall
Characteristics 2. What is the purpose of the information
technology acquisition? What missions or functions is the
acquisition to support?
3. What type of acquisition is it?
•
system integration,
•
commercial off-the-shelf applications,
•
software conversion,
•
software development,
•
hardware, or • other (specify).
4.
Are there any related in-house efforts?
5.
With what systems will this acquisition
interface?
6.
Is the acquisition based on
•
full and open competition,
•
competition restricted by compatibility
limitations,
Page 64 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Management Organization and Staffing
•
limited competition (such as make or model limited),
or
•
noncompetitive, sole source.
7. Is the contract based on
•
fixed prices (specify type of fixed-price contract),
or
•
cost reimbursement (specify type of cost-reimbursement
contract).
8.
What project management tools or techniques arein use to
oversee the project (such as Gantt charts or critical path
method/Program Evaluation and Review Technique).
9.
What, if any, development tools and techniques(such as
Computer-Aided Software Engineering-CASE) are in use?
10.
If a primary contractor has been selected for
theacquisition, list the contractor name, address, telephone
number, and point of contact.
11.
Identify key subcontractors, with company
names,addresses, telephone numbers, and points of
contact.
12.
How is the management of the acquisition projectorganized
and structured?
13.
What are the title, name, and phone numbers
ofthe
•
acquisition sponsor,
•
acquisition manager,
•
contracting officer,
•
user representatives, and
•
senior officials who approve acquisitions.
Page 65 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
14.
What are the responsibilities and duties of
theacquisition manager? What is his/her authority (for example,
planning, budget, staffing, or progress reporting)?
15.
Has an acquisition steering committee beenestablished?
What are the committee's responsibilities and duties?
16.
Who staffs the acquisition team and what are theteam
members' qualifications?
17.
What was the most recent milestone or key
Acquisition
Schedule and decision point approved?
Cost Estimates 18. What are the actual or estimated dates for
the following:
• original start/completion dates,
•
current start/completion dates,
•
requirements analysis completed and approved,
•
solicitation document completed for release,
•
contract awarded,
•
initial operation,
•
test and acceptance, and
•
full operation.
19.
Is the project on schedule or has there been
aslippage?
20.
Has the agency completed an estimate of life cyclecost?
If so, what are the original and current estimates?
21.
Identify the budget authority and outlays by yearfor the
project.
Page 66 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
22.
Identify the net benefits or cost savings projectedby the
cost/benefit analysis used to justify a chosen alternative, if the
acquisition has progressed to this point.
23.
Has the project received scheduled funds andresources or
have there been shortfalls? (Explain any variances and their
effects.)
Appendix II
Management Metrics
Purpose
This appendix describes tools and techniques for measuring the
status of acquisitions involving significant software development.
It describes techniques, known as software metrics, for
quantitatively measuring how closely a project conforms to
development plans and assessing whether an acquisition is at risk
of delay or cost increases. These techniques require considerable
information about the system being developed and can only be used
after system design. The metrics described here will generally be
used after contract award, in order to assess how well the agency
is managing the system development process.
Software metrics, which use mathematical models to measure
elements of the development process, are intended to help
organizations better understand and manage the relationships
between resource decisions, development schedules, and the cost of
software projects. By using software metric tools an auditor can
independently evaluate software development projects by analyzing
project budgets, requirements, schedules, and resources, and then
confirm or question cost, schedule, and resource estimates.
Different metrics may be useful for an audit, depending on the
objectives and status of an acquisition. Using cost models to
estimate the cost and length of time necessary to develop a new
software system, for example, is appropriate only after
requirements have been defined, a system design has been developed,
and the size of the new system has been estimated. The models
described here require estimates of either the lines of code or
number of function points that the new system will include. Cost
models project the time and cost to develop a system on the basis
of estimates of the system's size and other pertinent factors. They
may be used before a solicitation for software development is
Page 68 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
issued in order to assess the reasonableness of the agency's
schedule (if a system design has been prepared), or after a
contract has been awarded and software development has begun. The
other metrics described in this appendix require that system
development work be underway. A comparison of the actual number of
functions successfully tested to the number planned in the
acquisition schedule, for example, requires that system testing be
underway.
Cost Models Cost models are tools that estimate the effort
needed to develop software, based on assumed relationships between
the size of a system and the effort needed to design, code, and
test the software. These models can help the auditor assess whether
the acquisition's estimated cost and schedule are reasonable. Each
model uses cost drivers, which are parameters such as the level of
experience of the programmers, the reliability requirements of the
programs, and the complexity of the project, along with the
estimated project size, to derive overall cost and schedule
estimates for the acquisition. When a system involving software is
being developed, one or more cost models may be useful. A model
will be more reliable if it takes into account the agency's
historical experience in developing systems.
Cost models are available both commercially and from a few
agencies within the Department of Defense. Most of the tools were
developed initially for use with Defense department projects, but
can also be used with non-Defense systems. Many are based on
industry-recognized models such as the Constructive Cost Model
(COCOMO), PRICE, Putnam, and Jensen. The commercial tools range in
cost from about $500 to well over $20,000. Government-developed or
government-modified tools are available free of
Page 69 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
charge with a nominal charge for upgraded copies of the
software.
Because many different packages are available, and because more
than one can be used, auditors should determine what models, if
any, are used in their agencies. Typically cost models are derived
using data gathered from years of experience with a wide range of
software projects. Therefore, accuracy of predictions may improve
as further experience is accumulated in an agency.
Auditors should use cost models to look for discrepancies with
the cost and schedule estimates established for an acquisition. By
varying the estimated values for cost drivers, the auditor may also
be able to perform a sensitivity analysis illustrating where
project estimates are most susceptible to change. However, cost
models have significant limitations in accuracy unless their
underlying assumptions of system size and cost drivers are
carefully chosen and reflect the agency's previous experience in
system development. It is a matter of auditor judgment to decide
how discrepant project estimates and estimates provided by cost
models should be to raise concerns about risks of cost and schedule
overruns. In making this judgment, the auditor should take into
account the uncertainty of estimates and assumptions made in using
a cost model.
Auditors should use a cost model to provide general estimates
and not precise figures. Therefore, in applying software metrics to
audit work, care must be taken to follow generally accepted
government auditing standards when drawing conclusions based on the
results of software metrics. Specifically, all findings should be
qualified by the recognition that these tools are limited by the
accuracy of the
Page 70 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
estimated system size and other project data provided for the
model, the historical data from which the model was developed, and
the fact that all estimates are projections of an inherently
uncertain future. Procedures for minimizing the effect of these
limitations require, for example, proper technical advice,
qualifying language in audit reports, and a review of data and
assumptions by agency officials.
Auditors should also ensure that the model used is consistent
with the software development methodology of the project under
consideration. A system developed through rapid prototyping, for
example, should be evaluated with a model that takes prototyping
into account.
The following indicators are intended to help auditors
Other Indicators
assess how effectively an agency is managing a system
development contract. These indicators measure differences between
what an agency planned for its contractor to accomplish by certain
points in the systems development life cycle and the actual
results. In most cases this comparison is most easily done
graphically. Using more than one indicator can give a broader
picture of a project's status. Auditors should use indicators
appropriate to the project under review and for which data are
available.
Problem Reports This indicator involves tracking the number of
open and closed problems reported by a contractor as a system is
developed. A problem could be any anomaly discovered during design,
coding, testing, or implementation. Problems are distinguished from
failures of code, which represent defects discovered during
operation. Contracts should specify how problems are to be
identified and reported. By reviewing these reports and noting how
quickly
Page 71 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
problems are resolved, auditors can obtain an understanding of
how well the contractor is performing.
The project manager or Contracting Officer's Technical
Representative may be the best source for problem reports. Because
an agency may choose to prioritize problems in order of their
impact, the reports should distinguish between priority levels of
problems.
Figure II.1 shows one way to present this kind of analysis. It
shows the number of new problems reported and the number of
problems closed in order to show a trend over time and to show any
backlog that may be developing. The auditor may also choose to
report total problems reported and resolved or break them out by
level of priority. Figure II.2 shows the length of time that
problems have remained open in order to demonstrate how quickly
software problems are resolved once found. In this example, three
levels of priority are distinguished for reported problems.
Figure II.1: Problem Reports in Software Project
Figure II.2: Problems by Priority Levels and Number of Months
Unresolved
Software Volatility Software volatility is also measured
graphically. Figure II.3 shows changes in the total number of
approved system requirements. Cumulative changes are also tracked,
including additions, deletions, and modifications to requirements.
Steady increases in the number of requirements and changes to
requirements may indicate that the project is at risk for delays
and cost overruns.
Figure II.3: Software Requirements Changes
Development Progress
This indicator involves the comparison of actual and expected
progress in the design, coding, and integrating of system units.
Units can be measured in terms of computer software units or
computer software configuration items. If the project team does not
complete its design or programming and testing activities as
planned, this indicator can show schedule delays before major
milestones are reached. The progress indicator can show all
elements of design, coding, testing, and integrating, or it may
treat them as separate indicators. Figure II.4 shows how
Page 75 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
planned and actual progress in the design and coding of software
units can be displayed.
Figure II.4: Development Progress
Software Size This indicator records changes in the expected
magnitude of the software development effort. The size of the
system, measured in source lines of code as established by the
system design, may change as the system is coded. Changes in size
can be expressed
Page 76 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
as total lines of code or can be broken out into new, modified,
and reused lines of code. Changes in the estimated system size may
indicate that the project was overestimated or underestimated in
size and complexity. These changes may also indicate that
requirements are changing and may be related to the software
volatility issue described earlier. Increasing estimates of
software size should alert the auditor that the project's schedule
and expected cost may be underestimated. Changes in the expected
system size may necessitate reestimates of the development cost,
using the cost models described earlier in this appendix. Figure
II.5 shows how the estimates of software size can be tracked over
time.
Figure II.5: Software Size Estimates
Personnel Stability Tracking the total number of people assigned
to a system development effort compared to planned staffing levels
provides another indicator of potential problems. The auditor can
examine projected versus actual levels of total personnel or of
key, experienced personnel. Understaffing may result in schedule
slippage. Adding personnel late in a project can actually cause
further slippage.
Other metrics may be chosen as needed. Auditors could, for
example, compare expected to actual usages of hardware resources in
order to identify emerging computer capacity problems. Another
approach would involve tracking changes to the estimated completion
date for a system. The metrics described here offer suggestions to
be tailored for use as appropriate to particular projects.
Appendix III
Reference Materials
Federal Regulations
Each chapter of this guide lists applicable reference materials
including federal regulations and guidance published by GSA, OMB,
and other agencies. When using these references, summarized below,
auditors should ensure that the most current version available is
used.
Applicable regulations include both general rules for
procurement, the FAR, and the FIRMR regulations written by GSA
specifically for acquiring federal information processing
resources. GSA issues regulations under its Brooks Act authority.
In addition to the FIRMR, GSA issues bulletins that provide
guidance on a wide range of federal information processing
acquisition issues:
GSA • Federal Acquisition Regulation (FAR)
• Federal Information Resource Management Regulation (FIRMR)
OMB Circulars • A-109: Major System Acquisitions
•
A-130: Management of Federal Information Resources-under
proposed revision
•
A-76: Performance of Commercial Activities
•
A-11: Preparation and Submission of Budget
Estimates
GSA Acquisition Guide Series
•
Overview Guide
•
Guide for Requirements Analysis and Analysis of
Alternatives
•
Guide for Acquiring Maintenance Services
•
Guide for Acquiring Commercial Software
•
Guide for Contracting Officer's Technical
Representatives
•
Guide for Acquiring Systems Integration
Services
National Institute of • Federal Information Processing Standard
(FIPS) Publications. FIPS Publications describe federal
Science and
Technology standards for hardware, software, and the
management of information technologies. FIPS(NIST)-formerly
Publications relevant to the acquisition of informationthe National
Bureau technology include the following: of Standards
FIPS PUB 11-3-Guideline: American National Dictionary for
Information Processing Systems,
Feb. 1, 1991. FIPS PUB 38-Guidelines for Documentation of
Computer Programs and Automated Data Systems,
Feb. 15, 1976. FIPS PUB 42-1-Guidelines for Benchmarking ADP
Systems in the Competitive Procurement
Environment, May 15, 1977.
FIPS PUB 46-1-Data Encryption Standard, Jan. 22, 1988. FIPS PUB
64-Guidelines for Documentation of
Computer Programs and Automated Data Systems for
the Initiation Phase, Aug. 1, 1979. FIPS PUB 65-Guideline for
Automatic Data
Processing Risk Analysis, Aug. 1, 1979.
FIPS PUB 73-Guidelines for Security of Computer
Applications, June 30, 1980. FIPS PUB 75-Guideline on
Constructing
Benchmarks for ADP System Acquisitions,
Sept. 18, 1980. FIPS PUB 76-Guideline for Planning and Using
a
Data Dictionary System, Aug. 20, 1980.
FIPS PUB 87-Guidelines for ADP Contingency
Planning, Mar. 27, 1981. FIPS PUB 88-Guideline on Integrity
Assurance and
Control in Database Administration, Aug. 14, 1981.
FIPS PUB 96-Guideline for Developing and
Implementing A Charging System for Data Processing
Services, Dec. 6, 1982. FIPS PUB 99-Guideline: A Framework for
the
Evaluation and Comparison of Software Development
Tools, Mar. 31, 1983. FIPS PUB 101-Guideline for Lifecycle
Validation,
Verification, and Testing of Computer Software,
June 6, 1983. FIPS PUB 102-Guidelines for Computer Security
Certification and Accreditation, Sept. 27, 1983.
FIPS PUB 106-Guideline on Software Maintenance,
June 15, 1984. FIPS PUB 124-Guideline on Functional
Specifications for Database Management Systems,
Sept. 30, 1986. FIPS PUB 127-1-Database Language SQL,
Feb. 2, 1990. FIPS PUB 132-Guideline for Software
Verification
and Validation Plans, Nov. 19, 1987. FIPS PUB 146-1-Government
Open Systems
Interconnection Profile, Apr. 3, 1991. FIPS PUB 151-1-POSIX:
Portable Operating System
Interface for Computer Environments, Mar. 28, 1990.
FIPS PUB 158-The User Interface Component of the
Applications Portability Profile, May 29, 1990.
• Special publications. Publications relevant to information
technology acquisition include the following:
500-087-Management Guide for Software
Documentation, January 1982. 500-090-Guide to Contracting for
Software
Conversion Services, May 1982. 500-105-Guide to Software
Conversion Management.
500-106-Guide on Software Maintenance.
500-109-Overview of Computer Security
Certification and Accreditation. 500-120-Security of Personal
Computer Systems: A
Management Guide. 500-133-Technology Assessment: Methods for
Measuring the Level of Computer Security. 500-134-Guide on
Selecting ADP Backup Processing
Alternatives. 500-147-Guidance on Requirements Analysis for
Office Automation Systems (Update). 500-148-Application Software
Prototyping and
Fourth Generation Languages. 500-153-Guide to Auditing for
Controls and Security:
A System Development Life Cycle Approach. 500-154-Guide to
Distributed Database Management.
500-155-Management Guide to Software Reuse.
500-161-Software Configuration Management: An
Overview. 500-165-Software Verification and Validation: Its
Role in Computer Assurance and Its Relationship
with Software Product Management Standards. 500-172-Computer
Security Training Guidelines.
500-173-Guide to Data Administration.
500-174-Guide for Selecting Automated Risk
Analysis Tools. 500-175-Management of Networks Based on Open
Systems Interconnection (OSI) Standards: Functional
Requirements and Analysis. 500-180-Guide to Software
Acceptance.
500-183-Stable Implementation Agreements for
Open System Interconnection Protocols. 500-184-Functional
Benchmarks for Fourth
Generation Languages. 500-187-Application Portability Profile,
The U.S.
Government's Open System Environment Profile
OSE/1 Version 1.0. 500-192-Government Open Systems
Interconnection
Profile Users' Guide, Version 2. 500-193-Software Reengineering:
A Case Study and
Lessons Learned.
800-4-Computer Security Considerations in Federal
Procurements: A Guide for Procurement Initiators, Contracting
Officers, and Computer Security Officials,
March 1992.
GAO Audit Guidance • Information Technology: A Model to Help
Managers
Decrease Acquisition Risks (GAO/IMTEC-8.1.6,
August 1990).
• Government Auditing Standards, 1988 Revision.
Appendix IV
Major Contributors to This Audit Guide
Information Management and Technology Division, Washington,
D.C.
Mark E. Heatwole, Assistant Director David R. Turner,
Evaluator-in-Charge Bernard R. Anderson, Senior Evaluator Peter C.
Wade, Senior Evaluator Trinh N. Hoang, Computer Programmer David
Chao, Technical Reviewer Shane D. Hartzler, Editor
Glossary
Acquisition
The obtaining, by contract with appropriated funds, of supplies
or services (including construction) by and for the use of the
federal government through purchase or lease, whether the supplies
or services are already in existence or must be created, developed,
demonstrated, and evaluated. Acquisition begins at the point when
agency needs are established and includes the description of
requirements to satisfy agency needs, solicitation and selection of
sources, award of contracts, contract financing, contract
performance, contract administration, and those technical and
management functions directly related to the process of fulfilling
agency needs by contract.
Acquisition Planning
The process by which the efforts of all personnel responsible
for an acquisition are coordinated and integrated through a
comprehensive plan for fulfilling the agency need in a timely
manner and at a reasonable cost. It includes developing the overall
strategy for managing the acquisition.
A request by a federal agency for GSA to acquireAgency
information processing resources or for GSA toProcurement delegate
the authority to acquire these resources.
Request (APR)
The overall structure of a computer system including
Architecture
hardware and software.
The degree to which a system or component isAvailability
operational and accessible when required for use, often expressed
as a probability.
Page 87 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Baseline
(1) A specification or product that has been formallyreviewed
and agreed upon that thereafter serves as
the basis for further development and that can be changed only
through formal change control procedures. (2) A document or set of
such documents formally designated and fixed at a specific time
during the life cycle of a configuration item. Note: Baselines,
plus approved changes from those baselines, constitute the current
configuration identification. (3) Any agreement or result
designated and fixed at a given time from which changes require
justification and approval.
A test that uses a representative set of programs and
Benchmark Test
data designed to evaluate the performance of computer hardware
and software in a given configuration.
A final opportunity for offerors in the competitive
Best and Final
Offer range to revise proposals.
Capability Validation
The technical verification of the ability of a proposed system
configuration, replacement component, or the features or functions
of its software, to satisfy functional requirements. The intent is
to ensure that the proposed resources can provide the required
functions. Performance requirements are not implied or measured in
the validation. Examples of capability validation include:
a.
operational capability demonstrations of thefunctions of
the hardware, operating system, or support software;
b.
verification of conformance with informationprocessing
standards;
Page 88 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Certification
c.
expert examination of the technical literaturesupplied
with the offer;
d.
contacts with other users of the proposedinformation
processing resource; and
e.
vendor certification of conformance with thefunctional
requirements.
(1) A written guarantee that a system or componentcomplies with
its specified requirements and is acceptable for operational use.
For example, a written authorization that a computer system is
secure and is permitted to operate in a defined environment. (2) A
formal demonstration that a system or component complies with its
specified requirements and is acceptable for operational use. (3)
The process of confirming that a system or component complies with
its specified requirements and is acceptable for operational
use.
See configuration control.
Change Control
A daily publication that lists the government's
Commerce
procurement invitations, contract awards,
Business Daily subcontracting leads, sales, surplus property,
and foreign business opportunities. (GSA/IRMS, A Guide for
Acquiring Commercial Software, Jan. 1991, p. A-1.)
(1) The ability of two or more systems or components
Compatibility to perform their required functions while sharing
the same hardware or software environment. (2) The ability of two
or more systems or components to exchange information.
A statement of requirements expressed in terms
thatCompatibility-require items to be compatible with
existingLimited information processing resources.
Requirement
The group of offerors selected, after technical andCompetitive
cost evaluation, to whom award of a contract is aRange reasonable
possibility.
Configuration
(1) The arrangement of a computer system ornetwork as defined by
the nature, number, and chief characteristics of its functional
units. (2) The physical and logical elements of an information
processing system, the manner in which they are organized and
connected, or both. The term may refer to a hardware configuration
or a software configuration.
An element of configuration management, consistingConfiguration
of the evaluation, coordination, approval orControl disapproval,
and implementation of changes to
configuration items after formal establishment of
their configuration identification.
An aggregation of hardware and/or software that isConfiguration
designated for configuration management and treatedItem as a single
entity in the configuration management
process.
The continuous control of changes made to a
system'sConfiguration hardware, software, and documentation
throughoutManagement the development and operational life of the
system.
Page 90 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
A person with the authority to enter into,
administer,Contracting and/or terminate contracts and make
relatedOfficer (CO) determinations and findings.
An individual to whom the CO delegates certainContracting
contract responsibilities, usually related to technicalOfficer's
acceptance issues.
Technical Representative (COTR)
Modification of existing software to enable it to
Conversion
operate with similar functional capability in a different
environment; for example, converting a program from FORTRAN to Ada
or converting a program that runs on one computer to run on
another.
A contract in which the government reimburses the
Cost
contractor for expenses so long as the contractorReimbursement
provides its "best effort" to complete the work called Contract
for.
Authority to acquire information processing resourcesDelegation
of up to a specified limit, issued by GSA in response toProcurement
an agency procurement request.
Authority (DPA)
The regulation that codifies uniform acquisition
Federal
policies and procedures for executive agenciesAcquisition
governmentwide. Regulation (FAR) The regulation that sets forth
uniform policies and
Federal
procedures for acquiring information processingInformation
resources; used in conjunction with the FAR. Resources Management
Regulation (FIRMR)
A contract that provides for a firm price or in
Fixed-Price
appropriate cases, a firm price with fees or otherContract
adjustments.
A requirement that specifies a function that a system
Functional
Requirement or system component must be able to perform.
Verification and validation performed by anIndependent
organization that is technically, managerially, andVerification and
financially independent of the development Validation (IV&V)
organization. (See verification and validation, defined
separately as follows.)
The solicitation document used when contracting by
Invitation for Bid
(IFB) sealed bidding.
The ability of information technology resources to
Interoperability provide services to and accept services from
other resources and to use the services so exchanged to enable them
to operate effectively together.
Compensation to the government for a contractor'sLiquidated
failure to perform in a timely manner.
Damages
Page 92 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
Maintainability The ease with which maintenance of a functional
unit can be performed in accordance with prescribed
requirements.
Market Survey
Attempts to ascertain whether other qualified sources capable of
satisfying the government's requirement exist. This testing of the
marketplace may range from written or telephone contacts with
knowledgeable federal and non-federal experts regarding similar or
duplicate requirements and the results of any market test recently
undertaken, to the more formal sources-sought announcements in
pertinent publications (e.g., technical/scientific journals, the
Commerce Business Daily), or solicitations for
information or planning purposes.
The technical verification of the ability of a proposed
Performance
system configuration or replacement component to
Validation handle agency-specific work-load volumes (present and
expected) within agency-determined performance time
constraints.
The key management official who represents the
Program Manager program office in formulating resource
requirements and managing presolicitation activities. In some
organizations the program manager or another management official is
designated as the acquisition manager for a specific
acquisition.
A written objection by an interested party to (1) a
Protest
solicitation for a proposed contract, (2) a proposed award, or
(3) the award of a contract.
Page 93 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
A preliminary type, form, or instance of a system thatPrototype
serves as a model for later stages or for the final, complete
version of the system.
A hardware and software development technique in
Prototyping which a preliminary version of part or all of the
hardware or software is developed to permit user feedback,
determine feasibility, or investigate timing or other issues in
support of the development process.
A type of prototyping in which emphasis is placed onRapid
developing prototypes early in the developmentPrototyping process
to permit early feedback and analysis in
support of the development process.
The ability of a system or component to perform itsReliability
required functions under stated conditions for a stated period of
time.
An announcement in the Commerce Business Daily orRequest for
other publication requesting industry comment onComment draft
specifications for resources.
An announcement in the Commerce Business Daily orRequest for
other publication requesting information fromInformation industry
about a planned acquisition, and, in some
cases, corporate capability information.
The solicitation document used in negotiatedRequest for
procurements to communicate governmentProposals (RFP) requirements
and to solicit proposals.
Page 94 GAO/IMTEC-8.1.4 Assessing Acquisition Risks
An official government request for bids/proposals
Solicitation
generally publicized in the Commerce Business Daily in
accordance with federal regulations.
The government official in charge of selecting the
Source Selection
source for an acquisition. Most often the title is usedAuthority
(SSA) when the selection process is formal and the official is
other than the Contracting Officer.
A board composed of technical, contract, information
Source Selection
resources managers, and other government personnelEvaluation
Board whose primary function is to evaluate proposals (SSEB)
received in response to an RFP.
A document that describes the entire process for
Source Selection
awarding a contract-proposal evaluation criteria,Plan evaluation
methodology, evaluator's responsibilities, and final selection
procedures.
A description of the government's requirement forSpecific Make
resources which is so restrictive that only a particularand Model
manufacturer's products will satisfy the government's
Specification needs.
Technical Leveling
Helping an offeror to bring its proposal up to the level of
other proposals through successive rounds of discussion, such as by
pointing out weaknesses resulting from the offeror's lack of
diligence, competence, or inventiveness in preparing the
proposal.
The mix of tasks typically run on a given computer
Work Load
system. Major characteristics include input/output requirements,
amount and kinds of computation, and computer resources
required.
Ordering Information
The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made out
to the Superintendent of Documents, when necessary. Orders for 100
or more copies to be mailed to a single address are discounted 25
percent.
Orders by mail:
U.S. General Accounting Office
P.O. Box 6015Gaithersburg, MD 20884-6015
or visit:
Room 1100 700 4th St. NW (corner of 4th & G Sts. NW)
U.S. General Accounting OfficeWashington, DC
Orders may also be placed by calling (202) 512-6000 or by using
fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (301) 258-4097 using a
touchtone phone. A recorded menu will provide information on how to
obtain these lists.
United States General Accounting Office Washington, D.C.
20548-0001
Official Business Penalty for Private Use $300
Address Correction Requested
Bulk Mail Postage & Fees Paid GAO Permit No. G100
