United States General Accounting Office
Report to the Subcommittee on Readiness and
Management Support, Committee on Armed Services,
U.S. Senate
July 2002
BEST PRACTICES
Capturing Design and Manufacturing
Knowledge Early Improves Acquisition Outcomes
a
GAO-02-701
Contents
Letter
2
Executive Summary
Purpose2
Background3
Results in Brief4
Principal Findings6
Recommendations for Executive Action9
Agency Comments10
11
Chapter 1
Best Practices of Leading Commercial Companies12
DOD's Traditional Approach to Product Development15
DOD's Adoption of Best Practices16
Objectives, Scope, and Methodology17
22
Chapter 2
DOD Programs Had Better Outcomes When Design and
Manufacturing Knowledge Was Captured at Key Program
Manufacturing
Junctures22 Knowledge Is Critical to Program
Success
Chapter 3 Best Practices Enable Timely Capture of Design and
Manufacturing Knowledge
29
Leading Commercial Companies Use Evolutionary Product
Development Framework to Reduce Development Risks30
Leading Commercial Companies Use a Product Development
Process to Capture Design and Manufacturing Knowledge for
Decision Making32
When DOD Programs More Closely Approximated Best
Practices,
Outcomes Were Better43
Chapter 4 A Better Match of Policy and Incentives Is Needed
to Ensure Capture of Design and Manufacturing Knowledge
52
Acquisition Policy Lacks Specific Implementation
Criteria53 Incentives in the DOD Acquisition Environment Do
Not Favor
Capture of Design and Manufacturing Knowledge Early
Enough57
Related GAO Products
Figure 1:
Figures
Figure 2: Figure 3: Figure 4: Figure 5: Figure 6: Figure 7:
Figure 8: Figure 9:
Research, Development, Test and Evaluation, and
Procurement Funding for Fiscal Years 1995 to 200712
Knowledge-based Process for Applying Best Practices to
the Development of New Products13
Notional Illustration Showing the Different Paths That
a
Product's Development Can Take15
DOD's Concurrent Approach to Weapon System
Development16
Notional Single-Step and Evolutionary Approaches to
Developing New Products31
Achieving Stability on AIM-9X Missile Program by
Knowledge Point 244
History of Drawing Completion for the F-22 Program46
PAC-3 Design Knowledge at Critical Design Review49
Illustration to Show How the Best Practice Model Would
Apply to DOD's Acquisition Process56
A
United States General Accounting Office Washington, D.C.
20548
July 15, 2002
The Honorable Daniel Akaka Chairman The Honorable James Inhofe
Ranking Minority Member Subcommittee on Readiness and Management
Support Committee on Armed Services United States Senate
As you requested, this report examines how best practices offer
improvements to the way the Department of Defense develops new
weapon systems, primarily the design and manufacturing aspects of
the acquisition process. It examines the attainment of design and
manufacturing knowledge and its use at critical junctures to make
decisions about weapon systems' readiness to move forward in the
acquisition process. We make recommendations to the Secretary of
Defense for improvements to weapon system acquisition policy to
better align design and manufacturing activities with best
practices that have shown that the capture and use of key knowledge
can result in better cost, schedule, and performance outcomes.
We are sending copies of this report to the Secretary of
Defense; the Secretary of the Army; the Secretary of the Navy; the
Secretary of the Air Force; the Director of the Office of
Management and Budget; the Director, Missile Defense Agency; and
interested congressional committees. We will also make copies
available to others upon request. In addition, the report will be
available at no charge on the GAO Web site at
http://www.gao.gov.
If you have any questions regarding this report, please call me
at (202) 512-4841. Other contacts are listed in appendix II.
Katherine V. Schinasi Director Acquisition and Sourcing
Management
Executive Summary
Historically, the Department of Defense
(DOD) has taken much longer and
Purpose
spent much more than originally planned to develop and acquire
its weapon systems, significantly reducing the department's buying
power over the years. Clearly, it is critical to find better ways
of doing business and, in particular, to make sure that weapon
systems are delivered on time and cost-effectively. This is
especially true given the vast sums DOD is spending and is expected
to spend on weapons acquisition-$100 billion alone in 2002 and an
anticipated $700 billion over the next 5 years. DOD has recognized
the nature of this problem and has taken steps to address it,
including advocating the use of best practices for product
development from commercial companies. Leading commercial companies
have achieved more predictable outcomes from their product
development processes because they identify and control design and
manufacturing risks early and manage them effectively. While DOD
has made some progress in recent years, GAO's recent weapon system
reviews show that persistent problems continue to hinder
acquisition cost, schedule, and performance outcomes. For this
reason, GAO has continued a body of work to identify the lessons
learned by best commercial companies to see if they apply to weapon
system acquisitions.
This report addresses how DOD can manage its weapon system
acquisition process to ensure important knowledge about a system's
design, critical manufacturing processes, and reliability is
captured and used to make informed and timely decisions before
committing to substantial development and production investments.
It identifies best practices to facilitate this decision making at
two critical junctures-transition from system integration to system
demonstration during product development and then transition into
production. Ultimately, this should improve cost, schedule, and
quality outcomes of DOD major weapon system acquisitions. In
response to a request from the Chairman and the Ranking Minority
Member, Subcommittee on Readiness and Management Support, Senate
Committee on Armed Services, GAO (1) assessed the impact of design
and manufacturing knowledge on DOD program outcomes, (2) compared
best practices to those used in DOD programs, and (3) analyzed
current weapon system acquisition guidance for applicability of
best practices to obtain better program outcomes.
Background
Executive Summary
In any new product development program there are three critical
points that require the capture of specific knowledge to achieve
successful outcomes. The first knowledge point occurs when the
customer's requirements are clearly defined and resources-proven
technology, design, time, and money-exist to satisfy them.
Commercial companies insist that technology be mature at the outset
of a product development program and, therefore, separate
technology development from product development. The second
knowledge point is achieved when the product's design is determined
to be capable of meeting product requirements-the design is stable
and ready to begin initial manufacturing of prototypes. The third
knowledge point is achieved when a reliable product can be produced
repeatedly within established cost, schedule, and quality targets.
GAO's prior work on best practices covers achieving the first
knowledge point.1 This report examines best practices for achieving
the second and third knowledge points.
Commercial companies understand the importance of capturing
design and manufacturing knowledge early in product development,
when costs to identify problems and make design changes to the
product are significantly cheaper. In a knowledge-based process,
the achievement of each successive knowledge point builds on the
preceding one, giving decision makers the knowledge they need-when
they need it-to make decisions about whether to invest significant
additional funds to move forward with product development. Programs
that follow a knowledge-based approach typically have a higher
probability of successful cost and schedule outcomes. Problems
occur in programs when knowledge builds more slowly than
commitments to enter product development or production. The effects
of this delay in capturing knowledge can be debilitating. If a
decision is made to commit to develop and produce a design before
the critical technology, design, or manufacturing knowledge is
captured, problems will cascade and become magnified through the
product development and production phases. Outcomes from these
problems include increases in cost and schedule and degradations in
performance and quality.
1 U.S. General Accounting Office, Best Practices: Better
Matching of Needs and Resources Will Lead to Better Weapon System
Outcomes, GAO-01-288 (Washington, D.C.: Mar. 8, 2001) and Best
Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes,
GAO/NSIAD-99-162(Washington, D.C.: July 30, 1999).
Page 3 GAO-02-701 Best Practices
The success of any effort to develop a new
product hinges on having the
Results in Brief
right knowledge at the right time. Knowledge about a product's
design and producibility facilitates informed decisions about
whether to significantly increase investments and reduces the risk
of costly design changes later in the program. Every program
eventually achieves this knowledge; however, leading commercial
companies GAO visited have found that there is a much better
opportunity to meet predicted cost, schedule, and quality targets
when it is captured early, in preparation for critical investment
decisions. A product development process includes two phases
followed by production-integration phase and demonstration phase.
The commercial companies GAO visited achieved success in product
development by first achieving a mature, stable design supported by
completed engineering drawings during an integration phase and then
by demonstrating that the product's design was reliable and
critical manufacturing processes required to build it were in
control before committing to full production. The more successful
DOD programs GAO reviewed-the AIM-9X and the FA-18-E/F programs-had
achieved similar knowledge as the commercial companies, resulting
in good cost and schedule outcomes. In contrast, the DOD programs,
which had completed about one-quarter of their drawings when they
transitioned to the demonstration phase and had less than half of
their manufacturing processes in control when entering production,
experienced poor cost and schedule outcomes.
Leading commercial companies employed practices to capture
design and manufacturing knowledge in time for making key decisions
during product development. Two were most prominent. First, the
companies kept the degree of the design challenge manageable before
starting a new product development program by using an evolutionary
approach to develop a product. This minimized the amount of new
content and technologies on a product, making it easier to capture
the requisite knowledge about a product's design before investing
in manufacturing processes, tooling, and facilities. Second, the
companies captured design and manufacturing knowledge before the
two critical decision points in product development: when the
design was demonstrated to be stable-the second knowledge point-and
when the product was demonstrated to be producible at an affordable
cost-the third knowledge point. A key measure of design stability
was stakeholders' agreements that engineering drawings were
complete and supported by testing and prototyping when necessary. A
key measure of producibility was whether the companies' critical
manufacturing processes were in control and product reliability was
demonstrated. Most DOD programs GAO reviewed did not complete
engineering drawings prior to entering the demonstration phase, nor
did they bring critical manufacturing processes in control or
demonstrate reliability prior to making a production decision.
DOD has made changes to its acquisition policy2 in an attempt to
improve its framework for developing weapon systems, but the policy
does not require the capture of design or manufacturing knowledge
or sufficient criteria to enter the system demonstration and
production phases. In addition, it does not require a decision
review to enter the demonstration phase of product development.
Further, there is little incentive for DOD program managers to
capture knowledge early in the development process. Instead, the
acquisition environment emphasizes delaying knowledge capture and
problem identification since these events can have a negative
influence on obtaining annual program funding-a key to success for
DOD managers. In contrast, commercial companies encourage their
managers to capture product design and manufacturing knowledge to
identify and resolve problems early in development, before making
significant increases in their investment.
GAO is making recommendations to the Secretary of Defense on
ways to improve DOD's acquisition process to achieve better
outcomes by incorporating best practices to capture design and
manufacturing knowledge and then use this knowledge as a basis for
decisions to commit significant additional time and money as an
acquisition program progresses through system demonstration and
into production.
2 DOD Directive 5000.1, The Defense Acquisition System (Oct. 23,
2000), DOD Instruction 5000.2, Operation of the Defense Acquisition
System (Apr. 5, 2002), and DOD Regulation 5000.2-R, Mandatory
Procedures for Major Defense Acquisition Programs (MDAPS) and Major
Automated Information System (MAIS) Acquisition Programs (Apr. 5,
2002).
Page 5 GAO-02-701 Best Practices
Principal Findings
Timely Design and Manufacturing Knowledge
Is Critical to Program Success
Knowledge that a product's design is stable early in the program
facilitates informed decisions about whether to significantly
increase investments and reduces the risk of costly design changes
that can result from unknowns after initial manufacturing begins.
Likewise, later knowledge that the design can be manufactured
affordably and with consistent high quality prior to making a
production decision ensures that targets for cost and schedule
during production will be met. Leading commercial companies do not
make significant investments to continue a product development or
its production until they have knowledge that the product's design
works and it can be manufactured efficiently within cost and
schedule expectations.
DOD programs that captured knowledge similar to commercial
companies had more successful outcomes. For example, the AIM-9X and
the F/A-18E/F captured design and manufacturing knowledge by key
decision points and limited cost increases to 4 percent or less and
schedule growth to 3 months or less. In fact, the AIM-9X had 95
percent of its drawings completed at its critical design review.
The F/A-18E/F had 56 percent of its drawings completed and also had
over 90 percent of its higher level interface drawings completed,
adding confidence in the system design. Both took steps to ensure
that manufacturing processes were capable of producing an
affordable product by the time the programs made production
decisions.
On the other hand, the F-22, PAC-3, and Advanced Threat Infrared
Countermeasures/Common Missile Warning System (ATIRCM/CMWS)
programs did not capture sufficient knowledge before significant
investments to continue the programs and experienced cost growth
that ranged from 23 to 182 percent and schedule delays that ranged
from 18 months to over 3 years. None of these programs had
completed more than 26 percent of their engineering drawings for
their critical design reviews, and only the F-22 and PAC-3 programs
attempted to track the capability of their critical manufacturing
processes prior to production.
Best Practices Enable Timely Capture of
Design and Manufacturing Knowledge
Leading commercial companies developed practices that enabled
the timely capture of design and manufacturing knowledge. First,
they used an evolutionary approach to product development by
establishing timephased plans to develop a new product in
increments based on technologies and resources achievable now and
later. This approach reduced the amount of risk in the development
of each increment, facilitating greater success in meeting cost,
schedule, and performance requirements. The commercial companies
GAO visited used the evolutionary approach as their method for
product development. Each company had a plan for eventually
achieving a quantum leap in the performance of its products and had
established an orderly, phased process for getting there, by
undertaking continuous product improvements as resources became
available. For the most part, DOD programs try to achieve the same
leap in performance but in just one step, contributing to
development times that can take over 15 years to deliver a new
capability to the military user.
Second, each leading commercial company had a product
development process that was prominent and central to its success.
The process was championed by executive leadership and embraced by
product managers and development teams as an effective way to do
business. Critical to the product development process were
activities that enabled the capture of specific design and
manufacturing knowledge and decision reviews to determine if the
knowledge captured would support the increased investment necessary
to move to the next development phase or into production. These
activities provided knowledge that the product design was stable at
the decision point to start initial manufacturing (exiting the
integration phase) as demonstrated by the completion of 90 percent
of the engineering drawings. They also captured knowledge that a
product was ready to begin production (exiting the demonstration
phase) as demonstrated by proof that critical processes were in
control and product reliability was achievable. The activities that
enabled the capture and use of this knowledge to make decisions are
listed in table 1.
Table 1: Activities That Enable the Capture of Design and
Manufacturing Knowledge
Design is stable Product can be produced
•
Limit the design challenge. • Identify key system
characteristics and critical manufacturing processes.
•
Demonstrate, through prototyping or other means, that
product • Determine that processes are in control and stable.
works.
•
Complete design reviews of system and subsystems. •
Analyze potential failure modes and their effects on
performance.
•
Obtain stakeholder concurrence that the design is
complete and • Set reliability goals and growth plan and conduct
reliability producible. testing.
DOD programs that had more successful outcomes used key best
practices to a greater degree than others. For example, the AIM-9X
missile program completed 95 percent of its engineering drawings at
the critical design review because it made extensive use of
prototype testing to demonstrate the design met requirements
coupled with design reviews that included program stakeholders. The
F/A-18-E/F program eliminated over 40 percent of the parts used to
build predecessor aircraft to make the design more robust for
manufacturing and identified critical manufacturing processes,
bringing them under control before the start of production. Both
programs developed products that evolved from existing versions,
making the design challenge more manageable.
On the other hand, DOD programs with less successful outcomes
did not apply best practices to a great extent. At their initial
manufacturing decision reviews, the F-22, PAC-3, and ATIRCM/CMWS
had less than onethird of their engineering drawings, in part,
because they did not use prototypes to demonstrate the design met
requirements before starting initial manufacturing. On the F-22
program, it was almost 3 years after this review before 90 percent
of the drawings needed to build the F-22 were completed. Likewise,
at their production decision reviews, these programs did not
capture manufacturing and product reliability knowledge consistent
with best practices. For example, the PAC-3 missile program had
less than 40 percent of its processes in control and, as a result,
the missile seekers had to be built, tested, and reworked on
average 4 times before they were acceptable. The F-22 entered
production despite being substantially behind its plan to achieve
reliability goals. As a result, the F-22 is requiring significantly
more maintenance actions than planned.
A Better Match of Policy and Incentives Is
Needed to Ensure Capture of Design and Manufacturing Knowledge
DOD's acquisition policy establishes a good framework for
developing weapon systems; however, more specific criteria,
disciplined adherence, and stronger acquisition incentives are
needed to ensure the timely capture and use of knowledge and
decision making. DOD recently changed its acquisition policy to
emphasize evolutionary acquisition and establish separate
integration and demonstration phases in the product development
process. Its goal was to develop higher quality systems in less
time and for less cost. While similar to the leading commercial
companies' approach, the policy lacks detailed criteria for
capturing and using design and manufacturing knowledge to
facilitate better decisions and more successful acquisition program
outcomes. It also lacks a decision review to proceed from the
integration phase to the demonstration phase of product
development.
While the right policy and criteria are necessary to ensure a
disciplined, knowledge-based product development process, the
incentives that influence the key players in the acquisition
process will ultimately determine whether they will be used
effectively. In DOD, current incentives are geared toward delaying
knowledge so as not to jeopardize program funding. This undermines
a knowledge-based process for making product development decisions.
Instead, program managers and contractors push the capture of
design and manufacturing knowledge to later in the development
program to avoid the identification of problems that might stop or
limit funding. They focus more on meeting schedules than capturing
knowledge. On the other hand, commercial companies must develop
high-quality products quickly or they may not survive in the
marketplace. Because of this, they encourage their managers to
capture product design and manufacturing knowledge to identify and
resolve problems early in development, before making significant
increases in their investment. Instead of a schedule-driven
process, their process is driven by events that bring them
knowledge: critical design reviews that are supported by completed
engineering drawings and production decisions supported by
reliability testing and statistical process control data. They do
not move forward without the design and manufacturing knowledge
needed to make informed decisions.
GAO recommends that the Secretary of
Defense revise policy and guidance
Recommendations for
on the operation of the defense
acquisition system to include (1) a requirement to capture specific
design knowledge to be used as exit criteria for transitioning from
system integration to system demonstration
and (2) a requirement that the current optional interim progress
review between system integration and demonstration be a mandatory
decision review requiring the program manager to verify that design
is stable and that this be reported in the program's Defense
Acquisition Executive Summary and Selected Acquisition Report. The
policy and guidance should also be revised to include (1) a
requirement to capture and use specific manufacturing knowledge at
the production commitment point as exit criteria to transition from
system demonstration into production and (2) a requirement to
structure major weapon system contracts to ensure the capture and
use of knowledge for DOD to make investment decisions at critical
junctures when transitioning from system integration to system
demonstration and then into production.
DOD generally agreed with the report and
its recommendations. A detailed
Agency Comments
discussion of DOD's comments appears in appendix I.
Chapter 1
Introduction
The Department of Defense (DOD) spends close to $100 billion
annually to research, develop, and acquire weapon systems, and this
investment is expected to grow substantially. Over the next 5
years, starting in fiscal year 2003, DOD's request for weapon
system development and acquisition funds is estimated to be $700
billion (see fig. 1).
How effectively DOD manages these funds will determine whether
it receives a good return on its investment. Our reviews over the
past 20 years have consistently found that DOD's weapon system
acquisitions take much longer and cost much more than originally
anticipated, causing disruptions to the department's overall
investment strategy and significantly reducing its buying power.
Because such disruptions can limit DOD's ability to effectively
execute war-fighting operations, it is critical to find better ways
of doing business.
In view of the importance of DOD's investment in weapon systems,
we have undertaken an extensive body of work that examines DOD's
acquisition issues from a different, more cross-cutting
perspective-one that draws lessons learned from the best commercial
product development efforts to see if they apply to weapon system
acquisitions. This report looks at the core of the acquisition
process, specifically product development and ways to successfully
design and manufacture the product. Our previous reports looked at
such issues as how companies matched customer needs and resources,
tested products, assured quality, and managed suppliers and are
listed in related GAO products at the end of the report.
Best Practices of Leading Commercial
Companies
Figure 1: Research, Development, Test and Evaluation, and
Procurement Funding for Fiscal Years 1995 to 2007
Source: DOD.
Leading commercial companies expect their program managers to
deliver high-quality products on time and within budget. Doing
otherwise could result in the customer walking away. Thus, the
companies have created an environment and adopted practices that
put their program managers in a good position to succeed in meeting
these expectations. Collectively, these practices ensure that a
high level of knowledge exists about critical facets of the product
at key junctures during development. Such a knowledgebased process
enables decision makers to be reasonably certain about critical
facets of the product under development when they need this
knowledge.
To ensure the right level of knowledge at each key decision
point in product development, leading commercial companies separate
technology from product development and take steps to ensure the
product design is stabilized early so product performance and
producibility can be demonstrated before production. The process
followed by leading companies, illustrated in figure 2, can be
broken down into the following three knowledge points.
•
Knowledge point 1 occurs when a match is made between the
customer's needs and the available resources-technology, design,
time, and funding. To achieve this match, technologies needed to
meet essential product requirements must be demonstrated to work in
their intended environment. In addition, the product developer must
complete a preliminary product design using systems engineering to
balance customer desires with available resources.
•
Knowledge point 2 occurs when the product's design
demonstrates its ability to meet performance requirements. Program
officials are confident that the design is stable and will perform
acceptably when at least 90 percent of engineering drawings are
complete. Engineering drawings reflect the results of testing and
simulation and describe how the product should be built.
•
Knowledge point 3 occurs when the product can be
manufactured within cost, schedule, and quality targets and is
reliable. An important indicator of this is when critical
manufacturing processes are in control and consistently producing
items within quality standards and tolerances. Another indicator is
when a product's reliability is demonstrated through iterative
testing that identifies and corrects design problems.
Figure 2: Knowledge-based Process for Applying Best Practices to
the Development of New Products
Source: GAO's analysis.
This report focuses on best practices for achieving knowledge
points 2 and 3, particularly at how successful companies design and
manufacture a product within established cost, schedule, and
quality targets. The concepts discussed build on our previous
reports, which looked at the earlier phases of an acquisition,
including matching customer needs and available resources.
A key success factor evident in all our work is the ability to
obtain the right knowledge at the right time and to build knowledge
to the point that decision makers can make informed decisions about
moving ahead to the next phase. Programs that do this typically
have successful cost and schedule outcomes. Programs that do not
typically encounter problems that eventually cascade and become
magnified through the product development and production phases. As
shown in figure 3, the effects of not following a knowledge-based
process can be debilitating.
DOD's Traditional Approach to Product
Development
Figure 3: Notional Illustration Showing the Different Paths That
a Product's Development Can Take
Source: GAO's analysis.
DOD has historically developed new weapon systems in a highly
concurrent environment that usually forces acquisition programs to
manage technology, design, and manufacturing risk at the same time.
This environment has made it difficult for either DOD or
congressional decision makers to make informed decisions because
appropriate knowledge has not been available at key decision points
in product development. DOD's common practice for managing this
environment has been to create aggressive risk reduction efforts in
its programs. Cost reduction initiatives that typically arise after
a program is experiencing problems are common tools used to manage
these risks. Figure 4 shows the overlapping and concurrent approach
that DOD uses to develop its weapon systems. This figure shows that
DOD continues to capture technology, design, and manufacturing
knowledge long after a program passes through each of the three
knowledge points when this knowledge should have been available for
program decisions.
Figure 4: DOD's Concurrent Approach to Weapon System
Development
Source: GAO's analysis.
More important, the problems created by this concurrent approach
on individual programs can profoundly affect DOD's overall
modernization plans. It is difficult to prioritize and allocate
limited budgets among needed requirements when acquisition
programs' cost and schedule are always in question. Programs that
are managed without the knowledge-based process are more likely to
have surprises in the form of cost and schedule increases that are
accommodated by disrupting the funding of other programs. Because
of these disruptions, decision makers are not able to focus on a
balanced investment strategy.
DOD is taking steps to change the culture
of the acquisition community
DOD's Adoption of
with actions aimed at reducing
product development cycle times and improving the predictability of
cost and schedule outcomes. DOD recently made constructive changes
to its acquisition policy that embrace best
Objectives, Scope, and Methodology
practices. These changes focused primarily on (1) ensuring
technologies are demonstrated to a high level of maturity before
beginning a weapon system program and (2) taking an evolutionary,
or phased, approach to developing new weapon systems. Because these
changes occurred in 2000 and 2001, it is too early to determine how
effectively they will be put into practice. While these are good
first steps, further use of best practices in product development
would provide a greater opportunity to improve weapon system cost
and schedule outcomes.
Our overall objective was to determine whether best practices
offer methods to improve the way DOD ensures that the design is
stable early in the development process and whether having
manufacturing processes in control before production results in
better cost, schedule, and quality outcomes in DOD major
acquisition programs. Specifically, we identified best practices
that have led to more successful product development and production
outcomes, compared the best practices to those used in DOD
programs, and analyzed current weapon system acquisition guidance
for applicability of best practices.
To determine the best practices for ensuring product design and
manufacturing maturity from the commercial sector, we conducted
general literature searches. On the basis of our literature
searches and discussions with experts, we identified a number of
commercial companies as having innovative development processes and
practices that resulted in successful product development. We
visited the following commercial companies:
•
Caterpillar designs and manufactures construction and
mining equipment, diesel and natural gas engines, and industrial
gas turbines. In 2001, it reported sales and revenues totaling
$20.45 billion. We visited its offices in Peoria,
Illinois.
•
Cummins Inc. (Engine Business group) designs and
manufactures diesel and natural gas engines ranging in size from 60
to 3,500 horsepower for mining, construction, agriculture, rail,
oil and gas, heavy and mediumduty trucks, buses, and motor homes.
In 2001, the Engine Business Group reported sales of $3.1 billion.
We visited its offices in Columbus, Indiana.
•
General Electric Aircraft Engines designs and
manufactures jet engines for civil and military aircraft and gas
turbines, derived from its successful jet engine programs, for
marine and industrial applications.
In 2001, it reported earnings totaling $11.4 billion. We visited
its offices
in Evendale, Ohio.
•
Hewlett Packard designs and manufactures computing
systems and imaging and printing systems for individual and
business use. In 2001, it reported revenues totaling $45.2 billion.
We visited its offices involved in the design and manufacturing of
complex ink jet imaging equipment in Corvallis, Oregon.
•
Xerox Corporation designs and manufactures office
equipment, including color and black and white printers, digital
presses, multifunction devices, and digital copiers designed for
offices and production-printing environments. In 2001, it reported
revenues totaling $16.5 billion. We visited its offices in
Rochester, New York.
At each of the five companies, we conducted structured
interviews with representatives to gather uniform and consistent
information about each company's new product development processes
and best practices. During meetings with these representatives, we
obtained a detailed description of the processes and practices they
believed necessary and vital to mature a product design and get
manufacturing processes under control. We met with design
engineers, program managers, manufacturing and quality engineers,
and developers of the knowledge-based processes and policies.
During the past 5 years, we have gathered information on product
development practices from such companies as 3M, Boeing Commercial
Airplane Group, Chrysler Corporation, Bombardier Aerospace, Ford
Motor Company, Hughes Space and Communications, and Motorola
Corporation. This information enabled us to develop an overall
model to describe the general approach leading commercial companies
take to develop new products.
Our report highlights several best practices in product
development based on our fieldwork. As such, they are not intended
to describe all practices or suggest that commercial companies are
without flaws. Representatives from the commercial companies
visited told us that the development of their best practices has
evolved over many years and that the practices continue to be
improved based on lessons learned and new ideas and information.
They admit that the application and use of these have not always
been consistent or without error. However, they strongly suggested
that the probability of success in developing new products is
greatly enhanced by the use of these practices. Further, because of
the sensitivity to how data that would show the actual outcomes of
new product development efforts might affect their competitive
standing, we did not obtain specific cost, schedule, and
performance data. Most examples provided by these companies were
anecdotal. However, the continued success of these companies over
time in a competitive marketplace indicated that their practices
were important and key to their operations. Furthermore, based on
our observations during meetings at these companies, it was
apparent that because of the level of detailed process tools
developed for their managers and executive leadership these best
practices were a centerpiece of their operations.
Next, we compared and contrasted the best practices with product
development practices used in five DOD major acquisition programs.
Below is a brief description of each program we examined:
•
The F-22 fighter aircraft program. This aircraft is
designed with advanced features to allow it to be less detectable
to adversaries, capable of high speeds for long ranges, and able to
provide the pilot with improved awareness of the surrounding
situation through the use of integrated avionics. The F-22 program
began in 1986 and entered limited production in 2001. The Air Force
expects to buy 341 at a total acquisition cost (development and
procurement) estimated at $69.7 billion.
•
The Patriot Advanced Capability (PAC-3) missile program.
This program is intended to enhance the Patriot system, an
air-defense, guided missile system. PAC-3 is designed to enhance
the Patriot radar's ability to detect and identify targets,
increase system computer capabilities, improve communications,
increase the number of missiles in each launcher, and incorporate a
new "hit-to-kill" missile. The "hit-to-kill" missile capabilities
represent a major part of the development program, as these are not
capabilities included in prior versions of the Patriot system. The
missile program began in 1994 and entered limited production in
1999. The Army plans to buy 1,159 missiles at a total acquisition
cost estimated at $8.5 billion.
•
The Advanced Threat Infrared Countermeasures/Common
Missile Warning System (ATIRCM/CMWS) program. ATIRCM/CMWS is a
defensive countermeasure system for protection against infrared
guided missiles. The common missile warning system detects missiles
in flight, and the advanced threat infrared countermeasure defeats
the missile
with the use of a laser. The combined system is designed for
helicopter
aircraft. The common missile warning system is also designed
for
tactical aircraft such as fighters. The program began in 1995
and is
expected to start limited production in 2002. The Army and the
Special
Operations Command plan to buy 1,078 systems at a total
acquisition
cost estimated at $2.9 billion.
•
The AIM-9X missile program. AIM-9X is an infrared, short
range, air-to-air missile carried by Navy and Air Force fighter
aircraft. The AIM-9X is an extensive upgrade of the AIM-9M. The
AIM-9X is planned to have increased resistance to countermeasures
and improved target acquisition capability. A key feature is that
it will have the ability to acquire, track, and fire on targets
over a wider area than the AIM-9M. The AIM-9X program began in 1994
and entered limited production in 2000. DOD plans to buy 10,142
missiles at a total acquisition cost estimated at $3
billion.
•
The F/A-18 E/F fighter aircraft program. This aircraft is
intended to complement and eventually replace the current F/A-18
C/D aircraft and perform Navy fighter escort, strike, fleet air
defense, and close air support missions. It is the second major
model upgrade since the F/A-18 inception. The development program
began in 1992. The program entered limited production in 1997 and
full rate production in 2000. The Navy plans to buy 548 aircraft at
a total acquisition cost estimated at $48.8 billion.
We selected these programs for review based on cost, schedule,
and performance data presented in the Selected Acquisition Reports3
for each program. We also selected these programs because we
considered them to be in two basic categories-successful and
unsuccessful cost and schedule performance outcomes. This basis for
selection was to compare and contrast the development practices
used on each with best practices used by the commercial companies.
For each program, we interviewed key managers and design and
manufacturing engineering representatives. In some cases, we
discussed design and manufacturing issues with representatives of
the primary contractor for the specific program to obtain
information on the practices and procedures used by the program to
ready
3 The Selected Acquisition Report provides standard,
comprehensive summary reporting of cost, schedule, and performance
information for major defense acquisition programs to the
Congress.
Page 20 GAO-02-701 Best Practices
the product design for initial manufacturing and testing as well
as for production. We also discussed the use and potential
application of best practices that we identified. In addition to
discussions, we analyzed significant amounts of data on engineering
drawings, design changes, labor efficiencies, manufacturing
processes, quality indicators, testing, and schedules. We did not
verify the accuracy of the data but did correlate it to other
program indicators for reasonableness. Our analysis of the data was
used as a basis to develop indicators of each program's development
efficiencies and detailed questions to discuss product design and
manufacturing practices.
We conducted our review between May 2001 and April 2002 in
accordance with generally accepted government auditing
standards.
Chapter 2
Timely Design and Manufacturing Knowledge Is Critical to Program
Success
DOD Programs Had Better Outcomes When
Design and Manufacturing Knowledge Was Captured at Key Program
Junctures
The success of any effort to develop a new product hinges on
having the right knowledge at the right time. Every program
eventually achieves this knowledge; however, leading commercial
companies we visited have found that there is a much better
opportunity to meet predicted cost, schedule, and quality targets
when it is captured early, in preparation for critical decisions.
Specifically, knowledge that a product's design is stable early in
the program facilitates informed decisions about whether to
significantly increase investments and reduces the risk of costly
design changes that can result from unknowns after initial
manufacturing begins. This knowledge comes in the form of completed
engineering drawings before transitioning from the system
integration phase to the system demonstration phase of product
development. Best practices suggest that at least 90 percent of the
drawings for a product's design be completed before a decision to
commit additional resources is made. Likewise, later knowledge that
the design can be manufactured affordably and with consistent high
quality prior to making a production decision ensures that cost and
schedule targets will be met. This knowledge comes in the form of
evidence from data that shows manufacturing processes are in
control and system reliability is achievable. Leading commercial
companies rely on knowledge obtained about critical manufacturing
processes and product reliability to make their production
decisions.
The Department of Defense (DOD) programs we reviewed captured
varying amounts of design and manufacturing knowledge in the form
of completed engineering drawings and statistical process control
data. We found a correlation between the amount of knowledge each
captured and their cost and schedule outcomes. Programs that were
able to complete more engineering drawings and control their
critical manufacturing processes had more success in meeting cost
and schedule targets established when they began.
Conceptually, the product development process has two phases: a
system integration phase to stabilize the product's design and a
system demonstration phase to demonstrate the product can be
manufactured affordably and work reliably. The system integration
phase is used to stabilize the overall system design by integrating
components and subsystems into a product and by showing that the
design can meet product requirements. When this knowledge is
captured, knowledge point 2 has been achieved. It should be
demonstrated by the completion of at least 90 percent of
engineering drawings, which both DOD and leading commercial
companies consider to be the point when a product's design is
essentially complete. In the DOD process, this knowledge point
should happen by the critical design review, before system
demonstration and the initial manufacturing of production
representative products begins. The system demonstration phase is
then used to demonstrate that the product will work as required and
can be manufactured within targets. When this knowledge is
captured, knowledge point 3 has been achieved. Critical
manufacturing processes are in control and consistently producing
items within quality standards and tolerances for the overall
product. Also, product reliability has been demonstrated. In the
DOD process, like with the commercial process, this knowledge point
should happen by the production commitment milestone. Bypassing
critical knowledge at either knowledge point will usually result in
cost, schedule, and performance problems later in product
development and production.
We found that the most successful programs had taken steps to
gather knowledge that confirmed the product's design was stable
before the design was released to manufacturing organizations to
build products for demonstration. They had most of the detailed
design complete, supported by the completion of a large percentage
of engineering drawings to manufacturing. Again, engineering
drawings are critical because they include details on the parts and
work instructions needed to make the product and reflect the
results of testing. These drawings allowed manufacturing personnel
to effectively plan the fabrication process and efficiently build
production representative prototypes in the factory so
manufacturing processes and the product's performance could be
validated before committing to production. The most successful DOD
programs also captured the knowledge that manufacturing processes
needed to build the product would consistently produce a reliable
product by the end of system demonstration, before making a
production decision. On these programs, the initial phase of
production-sometimes known as low-rate initial production-was able
to focus on building operational test articles and improving the
production processes, instead of continuing the product's design
and development.
Problematic programs moved forward into system demonstration
without the same knowledge from engineering drawings that
successful cases had captured. They increased investments in
tooling, people, and materials before the design was stable. In
these programs, only a small percentage of the drawings needed to
make the products had been completed at the time the designs were
released to manufacturing organizations for building production
representative prototypes. In doing so, these programs undertook
the difficult challenge of stabilizing the designs at the same time
they were trying to build and test the products. This design
immaturity caused costly design changes and parts shortages that,
in turn, caused labor inefficiencies, schedule delays, and quality
problems. Consequently, these programs required significant
increases in resources--time and money-over what was estimated at
the point each program began the system demonstration phase.
The most problematic programs also started production before
design and manufacturing development work was concluded. In these
cases, programs were producing items for the customers while making
major product design and tooling changes, still establishing
manufacturing processes, and conducting development testing. These
programs encountered significant cost increases, schedule delays,
and performance problems during production.
Table 2 shows the relationship between design stability and
manufacturing knowledge at key junctures and the outcomes for the
DOD programs we reviewed. To measure design stability at the start
of the system demonstration phase, knowledge point 2, we determined
the percentage of the product's engineering drawings that had been
completed by the critical design review. In DOD programs, after the
critical design review, the system design is released to
manufacturing to begin building the production representative
prototypes for the system demonstration phase. To measure
producibility at the production decision, knowledge point 3, we
determined whether the critical manufacturing processes were in
statistical control at that time. We compared this information with
best practices. The cost and schedule experiences of the program
since the start of system demonstration are also shown.
Table 2: Attainment of Design and Manufacturing Knowledge in DOD
Programs and the Program Outcomes
aWhile AIM-9X used statistical process control on a limited
basis, we believe other factors contributed to a successful
production outcome to date. Other factors included early
achievement of design stability, early identification of key
characteristics and critical manufacturing processes, use of
established manufacturing processes for components common to other
weapon systems, design trade-offs to enhance manufacturing
capability, and a product design less vulnerable to variations in
manufacturing processes.
bF/A-18 E/F had 56 percent of drawings completed but also had
completed most of the higher-level assembly drawings. The
combination of these drawings with the fact the aircraft was a
variant of previously fielded F-18 aircraft models provided the
program a significant amount of knowledge that the design was
stable at the start of system demonstration.
Source: DOD program offices and Selected Acquisition
Reports.
As shown in the table, the AIM-9X and FA-18 E/F programs had
captured a significant amount of design knowledge at the start of
system demonstration and manufacturing knowledge by the start of
production. In each of those programs, product developers had the
advantage of prior versions of the systems. These programs came
very close to meeting their original cost and schedule estimates
for product development. The other three programs, F-22, PAC-3, and
ATIRCM/CMWS, had less knowledge at each key junctures. Their
development cost and schedule results significantly exceeded
estimates. Specific details on the AIM-9X, F-22, and ATIRCM/CMWS
program experiences follow.
AIM-9X Program Experience
The AIM-9X program began in 1994, continuing the long-term
evolution of the AIM-9 series of short-range air-to-air missiles.
In 1999, after developing and testing a number of engineering
prototype missiles, the program held a critical design review to
determine if the program was ready to begin initial manufacturing
of a production representative prototype for system demonstration.
At this review, about 95 percent of the eventual engineering
drawings were completed-a stable design by best practices. Because
AIM-9X was the next generation in this family of missiles, the
program had significant knowledge on how to produce the missile. At
the 1999 critical design review, the estimated development and
production costs totaled $2.82 billion. As of December 2001, the
estimate was $2.96 billion, less than a 5 percent increase.
F-22 Program Experience The F-22 program
began detailed design efforts in 1991 when it entered a planned
8-year product development phase. In 1995, about the expected
midpoint of the phase, the program held its critical design review
to determine if the design was stable and complete. Despite having
only about a quarter of the eventual design drawings completed for
the system, the program declared the design to be stable and ready
to begin initial manufacturing. At that time, the program office
had estimated the cost to complete the development program at $19.5
billion. However, the program did not complete 90 percent of its
drawings for the aircraft until 1998, 3 years into the system
demonstration phase. During the building of the initial aircraft,
several design and manufacturing problems surfaced that affected
the deliveries of major sections of the aircraft. Large sections
were delivered incomplete to final assembly and had to be built out
of the planned assembly sequence.
In 1997, an independent review team examined the program and
determined the product development effort was underestimated. The
team found that building the first three aircraft was taking
substantially more labor hours than planned. Between 1995 and 1998,
the development estimate for the F-22 increased by over $3.3
billion and the schedule slipped by a year. Achieving design
stability late has contributed to further cost increases. As of
December 2001, the estimated development cost was $26.1 billion, a
34 percent increase since the critical design review was held in
1995.
While the program attributes some production cost increases to a
reduction in F-22 quantities, it has been significantly affected by
design and
ATIRCM/CMWS Program Experience
manufacturing problems that started during development. The
independent review team evaluated the cost impact on the production
aircraft that would likely occur because of cost and schedule
problems in development and found that production aircraft would
have to begin later, at a slower pace, and cost more than expected.
The team estimated that production costs could increase by as much
as $13 billion if savings were not found. The Air Force
subsequently increased the estimate to more than $19 billion in
cost savings required to avoid cost increases. In 2001, when the
F-22 limited production decision was made, the program had less
knowledge about the aircraft's reliability and manufacturing
processes than more successful cases. For example, at its limited
production decision, it had only 44 percent of its critical
manufacturing processes in control. In September 2001, the program
reported that overall production cost would likely increase by more
than $5.4 billion. This estimate was based on the effort needed so
far to build the aircraft during product development.
Since it began in 1995, the ATIRCM/CMWS program has had
significant cost growth and schedule delays during product
development. The product developer held a major design review in
1997. Like the F-22, the review demanded less proof about the
product's design in the form of engineering drawings before
deciding to begin initial manufacturing. At that time, only 21
percent of the engineering drawings had been completed, and it was
still unknown whether the design would meet the requirements. In
fact, the program knew that a major redesign of a critical
component was needed. Despite this, the program office deemed the
risk acceptable for moving the program forward to begin
manufacturing prototypes. Over the next 2 years, the program
encountered numerous design and manufacturing problems. It was not
until 1999, about 2 years after the critical design review, that
program officials felt that the design had stabilized; however, by
this time, the product development cost had increased 160 percent
and production had been delayed by almost 3 years.
ATIRCM/CMWS is scheduled to begin limited production in early
2002, but without the same degree of assurance as the more
successful programs that the product can be manufactured within
cost, schedule, and quality targets. The program has not yet
determined if manufacturing processes needed to build the product
are in control. Many of the development units were built by hand,
in different facilities, and with different processes and
personnel. Program officials stated that because they did not
stabilize the design until late in development, manufacturing
issues were not adequately addressed. Since 1997, the estimated
unit cost for the system has increased by 182 percent.
Chapter 3
Best Practices Enable Timely Capture of Design and Manufacturing
Knowledge
Leading commercial companies have been successful in achieving
product development goals because they have found ways to enable
the capture of design and manufacturing knowledge about the
products they are developing in a timely way. We found two
practices that allowed leading commercial companies to capture
necessary knowledge for product development. First, they
established a framework of evolutionary product development that
limited the amount of design and manufacturing knowledge that had
to be captured. This framework limited the design challenge for any
one new product development by requiring risky technology, design,
or manufacturing requirements to be deferred until a future
generation of the product. Second, each company (1) employed a
disciplined product development process that brought together and
integrated all of the technologies, components, and subsystems
required for the product to ensure the design was stable before
entering product demonstration and (2) demonstrated the product was
reliable and producible using proven manufacturing processes before
entering production.
The product development process includes tools that both capture
knowledge and tie this knowledge to decisions about the product's
design and manufacturing processes before making commitments that
would significantly affect company resources. For example, during
system integration, each leading commercial company used various
forms of prototypes and information from predecessor products to
stabilize the product's design and identify critical processes,
then used a decision review that required agreements from key
stakeholders that the requisite design knowledge was captured in
making a decision to move into system demonstration. During system
demonstration, each company used statistical process control and
reliability testing to ensure the product could be produced
affordably and would be reliable, then used a similar decision
review that required agreements from key stakeholders that the
requisite knowledge was captured when deciding to move into
production.
The Department of Defense (DOD) programs that we reviewed used
some of these practices to varying degrees and experienced
predictable outcomes. For example, the AIM-9X and F/A-18 E/F
programs were evolutionary in nature, modifications of existing
products with a manageable amount of new technological or design
challenges. They also gathered design and manufacturing knowledge,
although not to the extent we found at commercial companies.
Finally, they held program reviews and ensured that the design and
manufacturing knowledge was captured before moving forward. They
had relatively successful outcomes. The other DOD
Leading Commercial Companies Use
Evolutionary Product Development Framework to Reduce Development
Risks
programs-the F-22, ATIRCMS, and PAC-3-did not closely
approximate best practices in capturing design or manufacturing
knowledge during product development. They took on greater design
challenges, had program reviews that were not supported by critical
design and manufacturing knowledge, and made decisions to advance
to the next phases of development without sufficient design and
manufacturing knowledge.
A key to the success of commercial companies was using an
evolutionary approach to develop a product. This approach permitted
companies to focus more on design and development with a limited
array of new content and technologies in a program. It also ensured
that each company had the requisite knowledge for a product's
design before investing in the development of manufacturing
processes and facilities. Companies have found that trying to
capture the knowledge required to stabilize the design of a product
that requires significant amounts of new content is an unmanageable
task, especially if the goal is to reduce cycle times and get the
product into the marketplace as quickly as possible. Design
elements not achievable in the initial development were planned for
subsequent development efforts in future generations of the
product, but only when technologies were proven to be mature and
other resources were available.
Commercial companies have implemented the evolutionary approach
by establishing time-phased plans to develop new products in
increments based on technologies and resources achievable now and
later. This approach reduces the amount of risk in the development
of each increment, facilitating greater success in meeting cost,
schedule, and performance requirements. In effect, these companies
evolve products, continuously improving their performance as new
technologies and methods allow. These evolutionary improvements to
products eventually result in the full desired capability, but in
multiple steps, delivering a series of enhanced interim
capabilities to the customer more quickly.
Historically, DOD's approach has been to develop new weapon
systems that often attempt to satisfy the full requirement in a
single step, regardless of the design challenge or the maturity of
technologies necessary to achieve the full capability. Under this
single-step approach, a war fighter can wait over 15 years to
receive any improved capability. Figure 5 shows a notional
comparison between the single-step and evolutionary approaches.
Figure 5: Notional Single-Step and Evolutionary Approaches to
Developing New Products
Source: GAO's analysis and DOD acquisition guidance.
Each commercial company we visited used the evolutionary
approach as the primary method of product development. General
Electric builds on the basic capability of a fielded product by
introducing proven improvements in capability from its advanced
engineering development team. General Electric considers the
introduction of immature technologies into fielded products or new
engine development programs as a significant cost and schedule
risk. Its new product development process is primarily focused on
reducing and managing risk for design changes and product
Leading Commercial Companies Use a Product
Development Process to Capture Design and Manufacturing Knowledge
for Decision Making
introductions. Cummins and Hewlett Packard managers indicated
that, in the past, their companies learned the hard way by trying
to make quantum leaps in product performance and by including
immature technologies. Now, both companies have new product
development processes that actively manage the amount of new
content that can be placed on a new product development effort.
Caterpillar also limits new content on its new products as a way to
more successfully and cost-effectively develop new, but
evolutionary, products. Even during the development of its 797
mining truck, which it considered a major design challenge, it did
not require the truck to achieve capabilities-such as prognostics
for better maintenance-that it could not demonstrate or validate in
the design in a timely manner.
Of the five DOD programs we reviewed, two-the F/A-18-E/F and the
AIM-9X-were variations of existing products-the F/A-18-C/D and the
AIM-9M-and the programs made a commitment to use existing
technologies and processes as much as possible. These two programs
had relatively successful cost and schedule outcomes. They
represented an exception to the usual practice in DOD. The
overwhelming majority of DOD's major acquisitions today require
major leaps in capability over their predecessors or any other
competing weapon systems, with little knowledge about the resources
that will be required to design and manufacture the systems.
Decisions are continually made throughout product development
without knowing the cost and schedule ramifications.
Leading commercial companies we visited had spent significant
amounts of time and resources to develop and evolve new product
development processes that ensured design and manufacturing
knowledge was captured at the two critical decision points in
product development: when the product's design was demonstrated to
be stable-knowledge point 2-and when the product was demonstrated
to be producible at an affordable cost-knowledge point 3. The
process established a disciplined framework to capture specific
design and manufacturing knowledge about new products. Companies
then used that knowledge to make informed decisions about moving
forward in a new product development program. Commercial companies
tied this knowledge to decisions about the products' design and
manufacturing processes before making commitments that would
significantly impact company resources. Each commercial firm we
visited had a new product development process that was prominent
and central to the firm's successes. It included three
Design Knowledge Should Be Captured before
Entering Product Demonstration
aspects: (1) activities that led to the capture of specific
design knowledge,
(2) activities that led to the capture of specific manufacturing
and product reliability knowledge, and (3) decision reviews to
determine if the appropriate knowledge was captured to move to the
next phase.
To ensure that the product's design was stable before deciding
to commit additional resources to product demonstration, commercial
companies demanded knowledge, either from existing product
information or by building engineering prototypes. They also used a
disciplined design review process to examine and verify the
knowledge that had culminated at the end of product integration,
This design review process required agreement from stakeholders
that the product design could be produced and would satisfy the
customer's requirements. Stakeholders included design engineers,
manufacturing or production personnel, and key supplier
representatives who used engineering drawings, supported by test
results and engineering data, as a key indicator of the design's
stability. Once the program achieved a stable design, the certainty
of their cost and schedule estimates was substantially increased,
allowing them to plan the balance of the product development
program with high confidence. Table 3 shows the activities required
to capture design knowledge that leads to executive decisions about
whether to transition to the next phase of development.
Table 3: Activities to Capture Design Knowledge and Make
Decisions
Activities to Achieve Stable Design Knowledge
•
Limit design challenge - The initial design challenge is
limited to a product that can be developed and delivered quickly
and provide the user with an improved capability. A time-phased
plan is used to develop improved products-future generations-in
increments as technologies and other resources become
available.
•
Demonstrate design meets requirements -The product's
design is demonstrated to meet the user's requirements. For a new
product that is not based on an existing product, prototypes are
built and tested. If the product is a variant of an existing
product, companies often used modeling and simulation or prototypes
at the component or subsystem level to demonstrate the new
product's design.
•
Complete critical design reviews - Critical design
reviews are used to assess whether a product's design meets
requirements and is ready to start initial manufacturing. They are
conducted for the system, subsystems, and components to assess
design maturity and technical risk.
•
Stakeholders agree drawings complete and producible - The
agreement by stakeholders (engineers, manufacturers, and other
organizations) is used to signify confidence that the design will
work and the product can be built.
•
Executive level review to begin initial manufacturing -
Corporate stakeholders meet and review relevant product knowledge,
including design stability, to determine whether a product is ready
to initiate manufacturing of production representative prototypes
used during system demonstrations. The decision is tied to the
capture of knowledge.
Demonstrating the Design Helped Achieve
Stability
A key tool used by each company to ensure that a product's
design was stable by the end of the product integration phase was a
demonstration that the design would meet requirements. The
companies visited indicated that prototypes at various system
levels were the best way to demonstrate that the product's design
would work. If the product under development was an incremental
improvement to existing products, such as the next generation of a
printer or engine, these companies used virtual prototypes for any
components that were being used for the first time. If the product
included more new content or invention, fully integrated prototypes
were frequently used to demonstrate that the design met
requirements. Prototypes at this stage in development were
typically not built in a manufacturing facility. This allowed
demonstrations of the design before the companies made more costly
investments in manufacturing equipment and tooling to build
production representative prototypes for the demonstration phase.
Table 4 shows an example of the types and purposes for various
kinds of prototypes used by Cummins Inc. depending on the amount of
knowledge it needed to capture and the point it was in the
development process. Prototypes were used by commercial companies
throughout the product development process and not just during
product integration.
Table 4: Examples of Prototypes Used by Cummins Inc. at Various
Stages of Product Development
Product integration Product demonstration Production
Prototype Engineering prototypes (virtual or Production
representative prototypes Initial products physical)
Purpose Demonstrate form, fit and function, and Demonstrate the
product is capable, Demonstrate ready for full rate a stable design
reliable, and manufacturing production processes in statistical
control
Build environment Engineering Manufacturing Production (all rate
tooling) (1st set of production tooling)
Cummins, the world sales leader in diesel engines over 200
horsepower, effectively uses prototypes to ensure that a design is
stable and believes in the value of prototyping throughout product
development. A Cummins representative stated that not using
prototypes becomes a matter of "pay me now or pay me later,"
meaning that it is far less costly to demonstrate a product's
design early in development with prototypes, concepts, and analyses
than to incur the cost of significant design changes after a
product has entered production-a much more costly environment to
make changes. Cummins built and tested 12 engineering concept
prototype engines for its Signature 600 engine, a new concept, 600
horsepower,
Disciplined Reviews and Stakeholder
Agreements Supported the Capture of Design Knowledge
overhead cam diesel engine that represented a quantum leap in
performance beyond Cummins' existing products. These prototypes
were built using production-like tooling and methods using
production workers. In addition to using engineering prototypes
during the product integration phase of product development,
Cummins and other companies we visited used other prototypes-such
as production representative prototypes-in the remaining product
development phases before production, as shown in table 4, to
demonstrate product reliability and process control. Prior to
reaching production for its Signature 600 engine, Cummins used many
prototypes to complete hundreds of thousands of test hours,
accumulating millions of test miles.
Caterpillar, a major manufacturer of heavy equipment, has a
continuous product improvement philosophy. That is, it tries to
develop new products that increase the capabilities of existing
product lines, but it limits the amount of new content on any one
product development because new content inherently increases design
risk. In evolving its products this way, Caterpillar is able to use
modeling and simulation prior to initial manufacturing because it
has existing products to provide a baseline of knowledge and a good
benchmark for assessing the simulated performance. In addition,
with knowledge of existing components, it can focus attention on
maturing the new content, the higher risk element of the new
product. When Caterpillar developed the 797 mining truck, a new
360ton payload truck design, it demonstrated design stability by
identifying the critical components and building engineering
prototypes of them for reliability testing and demonstration of the
design before beginning initial manufacturing. This knowledge,
coupled with vast experience in manufacturing trucks, ensured the
stability of the 797-truck design before initial manufacturing
started. Caterpillar was able to deliver this design in 18 months
after the product development was started.
The commercial companies we visited understood the importance of
having disciplined design reviews and getting agreement from the
stakeholders that the product's design had been demonstrated to
meet requirements before beginning initial manufacturing. Each
company had a design review process that began at the component
level, continued through the subsystem level, and culminated with a
critical design review of the integrated system to determine if the
product was ready to progress to the next phase of development. In
addition to design engineers, a crossfunctional team of
stakeholders in the process included key suppliers, manufacturing
representatives, and service and maintenance representatives. From
past experience, commercial companies have
Executive Level Reviews Were Required to
Begin Initial Manufacturing
discovered that cross-functional teams provide a complete
perspective of the product. While design engineers bring important
skills and experience to creating a product design, they may not be
aware of manufacturing issues, available technologies, or
manufacturing processes, and they may design a product that the
company cannot afford to produce or maintain.
The product's design is stable when all stakeholders agree that
engineering drawings are complete and that the design will work and
can be built. A commercial company considers engineering drawings4
to be a good measure of the demonstrated stability of the product's
design because they represent the language used by engineers to
communicate to the manufacturers the details of a new product
design-what it looks like, how its components interface, how it
functions, how to build it, and what critical materials and
processes are required to fabricate and test it. The engineering
drawing package released to manufacturing includes items such as
the schematic of the product's components, interface control
documents, a listing of materials, notations of critical
manufacturing processes, and testing requirements. It is this
package that allows a manufacturer to build the product in the
manufacturing facility.
In developing the Signature 600, Cummins used cross-functional
design teams that included stakeholders from suppliers, machine
tool manufacturers, foundry and pattern makers, purchasing,
finance, manufacturing engineering, design engineering, and other
technical disciplines. Signature 600 components were designed with
the key suppliers co-located at the Cummins design facility.
Likewise, Caterpillar said that early supplier and manufacturing
involvement was critical to success and that engineering drawings
were signed by design and manufacturing stakeholders. Caterpillar
representatives said that signing the drawings was a certification
that the design could be manufactured the next day, if
necessary.
Each commercial company, after capturing specific design
knowledge, had an executive level review at the decision point to
determine if the product design had sufficiently progressed to
permit a transition from product integration to product
demonstration. This decision point used the knowledge captured as
exit criteria for moving to the next phase of development. For
example, to demonstrate the product design was stable
4 Engineering drawings can include the standard two-dimensional
drawings or newer threedimensional drawings that are the product of
computer-aided design software systems.
Page 36 GAO-02-701 Best Practices
Manufacturing and Product Reliability
Knowledge Should Be Captured before Starting Production
and ready to move from integration to demonstration, the design
had to be demonstrated, at least 90 percent of the engineering
drawings had to be completed, design reviews had to be completed,
and stakeholders had to agree the design was complete and
producible. If the design team could not satisfy the exit criteria,
then other options had to be considered. Options included canceling
the development program, delaying the decision until all criteria
were met, or moving ahead with a detailed plan to achieve criteria
not met by a specific time when leadership would revisit the other
options. One company emphasized that if a major milestone is
delayed, an appropriate adjustment should be made to the end date
of the program, thereby avoiding compressing the time allotted for
the rest of product development and managing the risks that
subsequent milestones will be missed.
This decision point coincides with the companies' need to
increase investments in the product development and continue to the
next phase. For this reason, the decision point was considered
critical to achieving success in product development and could not
be taken lightly. For example, transitioning from the integration
to the demonstration phase requires a significant investment to
start building and testing production representative prototypes in
a manufacturing environment. This requires establishing a supplier
base and purchasing materials. In addition, establishing tooling
and manufacturing capability is also required. After a product
passes this decision point and added investments are made, the cost
of making changes to the product design also increases
significantly. Therefore, commercial companies strive to firm the
design as early in the process as possible when it is significantly
cheaper to make changes.
We found that leading commercial companies used two tools to
capture knowledge that a product's design was reliable and
producible within cost, schedule, and quality targets before making
a production decision. These tools are (1) a quality concept that
uses statistical process control to bring critical manufacturing
processes under control so they are repeatable, sustainable, and
consistently producing parts within the quality tolerances and
standards of the product and (2) product tests in operational
conditions that ensure the system would meet reliability goals-the
ability to work without failure or need of maintenance for
predictable intervals. Company officials told us that these two
tools enabled a smooth transition from product development to
production, resulting in better program outcomes. Companies
employed these tools on production representative prototypes,
making the prototypes a key ingredient to successful outcomes.
Table 5 shows the activities required to capture manufacturing
knowledge that leads to executive decisions about whether to
transition from product development into production.
Table 5: Activities to Capture Manufacturing Knowledge and Make
Decisions
Activities to Achieve Manufacturing Knowledge
•
Identify key system characteristics and critical
manufacturing processes - Key product characteristics and critical
manufacturing processes are identified. Because there can be
thousands of manufacturing processes required to build a product,
companies focus on the critical processes-those that build parts
that influence the product's key characteristics such as
performance, service life, or manufacturability.
•
Determine processes in control and capable - Statistical
process control is used to determine if the processes are
consistently producing parts. Once control is established, an
assessment is made to measure the process's ability to build a part
within specification limits as well as how close the part is to
that specification. A process is considered capable when it has a
defect rate of less than 1 out of every 15,152 parts
produced.
•
Conduct failure modes and effects analysis - Bottom-up
analysis is done to identify potential failures for product
reliability. It begins at the lowest level of the product design
and continues to each higher tier of the product until the entire
product has been analyzed. It allows early design changes to
correct potential problems before fabricating hardware.
•
Set reliability growth plan and goals - A product's
reliability is its ability to perform over an expected period of
time without failure, degradation, or need of repair. A growth plan
is developed to mature the product's reliability over time through
reliability growth testing so that it has been demonstrated by the
time production begins.
•
Conduct reliability growth testing -Reliability growth is
the result of an iterative design, build, test, analyze, and fix
process for a product's design with the aim of improving the
product's reliability over time. Design flaws are uncovered and the
design of the product is matured.
•
Conduct executive level review to begin production -
Corporate stakeholders meet and review relevant product knowledge,
including manufacturing and reliability knowledge, to determine
whether a product is ready to begin production. The decision is
tied to the capture of knowledge.
or manufacturability. Therefore, when design engineers are
designing the new product, they must identify its key
characteristics so that manufacturing engineers can identify and
control critical manufacturing processes. Key product
characteristics and critical manufacturing processes are noted on
the engineering drawings and work instructions that are released to
manufacturing.
Once critical processes are identified, companies perform
capability studies to ensure that a process will produce parts that
meet specifications. These studies yield a process capability index
(Cpk), a measure of the process's ability to build a part within
specified limits. The index can be translated into an expected
product defect rate. The industry standard is to have a Cpk of 1.33
or higher, which equates to a probability that 99.99 percent of the
parts built on that process will be within the specified limits.
Four of the five5 companies we visited wanted their critical
processes at a minimum of a 1.33 Cpk and many had goals of
achieving higher Cpks. Table 6 shows various Cpk values and the
defect rate associated with each value. The table also shows the
higher the Cpk, the lower the defect rate.
Table 6: Cpk Index and Probability of a Defective Part
Cpk values also have an additive effect on various individual
parts when each part is integrated into the final product. For
example, a product composed of 25 parts, where each part is
produced on a manufacturing process with a Cpk of 0.67, has a 95.5
percent probability that each part will be defect free. However,
when all 25 parts are assembled into the final product, the
probability that the final product will be defect free is only 32
percent. In comparison, if the same parts are produced with
manufacturing processes at a Cpk of 1.33, the probability of each
part being defect free is 99.99 percent. When these same 25 parts
are assembled into the final product, the probability that the
final product will be defect free is
5 The fifth company wanted its critical manufacturing processes
at a minimum of 1 Cpk.
99.8 percent. This comparison illustrates the impact that having
manufacturing processes in control has on the amount of rework and
repair that would be needed to correct defects and make the product
meet its specifications.
Cummins uses statistical process control data to measure a
product's readiness for production. In developing the new Signature
600 diesel engine, Cummins included manufacturing engineers and
machine tool and fixture suppliers in the design decision process
as the engine concept was first being defined. Cummins built
production representative prototypes of its engines to demonstrate
that the design and the engine hardware would perform to
requirements. These prototypes represented the first attempt to
build the product solely using manufacturing personnel, production
tooling, and production processes. Cummins used the knowledge
captured from these and subsequent prototypes to refine and
eventually validate the manufacturing processes for the engine.
This process of employing statistical process control techniques on
prototype engines verified that the manufacturing processes were
capable of manufacturing the product to high quality standards
within established cost and schedule targets.
Other companies we visited emphasized the importance of
controlling manufacturing processes before committing to
production. For example, Xerox captures knowledge about the
producibility of its product early in the design phase. By
production, it strives to have all critical manufacturing processes
for the product-including key suppliers' processes-in control with
a Cpk index of at least 1.33. Xerox achieves this by building
production representative prototypes and by requiring suppliers of
key components and subassemblies to produce an adequate sample of
parts to demonstrate the suppliers' processes can be controlled,
usually before the parts are incorporated into the prototypes.
General Electric Aircraft Engines has digitally captured, and made
available to design engineers, Cpk data on almost all of its
manufacturing processes and it strives to have critical processes
in control to a point where they will yield no more than 1 defect
in 500 million parts, a Cpk of 2.0. Other companies, such as
Caterpillar and Hewlett Packard, told us that getting manufacturing
processes in control prior to production is key to meeting cost,
schedule, and quality targets. Each of the companies visited used
this as an indicator of the product's readiness for production and
emphasized the importance of having critical manufacturing
processes under control by the start of production.
Demonstrating Product Reliability Indicates
the Product Is Ready for Production
A product is reliable when it can perform over a specified
period of time without failure, degradation, or need of repair.
Reliability is a function of the specific elements of a product's
design. Making design changes to achieve reliability requirements
after production begins is inefficient and costly. Reliability
growth testing provides visibility over how reliability is
improving and uncovers design problems so fixes can be incorporated
before production begins.
In general, reliability growth is the result of an iterative
design, build, test, analyze, and fix process. Prototype hardware
is key to testing for reliability growth. Initial prototypes for a
complex product with major technological advances have inherent
deficiencies. As the prototypes are tested, failures occur and, in
fact, are desired so that the product's design can be made more
reliable. Reliability improves over time with design changes or
manufacturing process improvements. The earlier this takes place,
the less impact it will have on the development and production
program. Companies we visited matured a product's reliability
through these tests and demanded proof that the product would meet
the customer's reliability expectations prior to making a
production decision.
Improvements in the reliability of a product's design can be
measured by tracking a key reliability metric over time. This
metric compares the product's actual reliability to a growth plan
and ultimately to the overall reliability goal. Several commercial
companies we visited began gathering this data very early in
development and tracked it throughout development. The goal was to
demonstrate the product would meet reliability requirements before
starting full rate production.
Caterpillar establishes a plan to grow and demonstrate the
product's reliability before fabrication of a production
representative prototype begins. Before Caterpillar starts making
parts, it estimates the product's reliability in its current stage
of development based on knowledge captured from failure modes and
effects analysis,6 component prototype testing, and past product
experience. This information marks the starting point for the
product's reliability growth plan and is the basis for assessing
whether the plan is achievable by production. If Caterpillar
believes the risks are too
6 Failure modes and effects analysis is a bottom-up approach to
failure identification. It should begin at the lowest level of the
product design. Through analysis potential failure modes are
identified allowing early design change to correct potential
problems before fabricating hardware-a more cost-effective time to
identify and fix problems.
Page 41 GAO-02-701 Best Practices
Executive Level Reviews Are Conducted to
Begin Production
high and the goal cannot be achieved on time, decision makers
assess trade-offs between new and existing components to reduce the
risks to a more manageable level. Trade-offs might be made if the
product's performance still fails to meet requirements. If
trade-offs are not possible, decision makers may decide not to go
forward with the development. Once Caterpillar has established this
plan, it tracks demonstrated reliability against it as a management
tool to measure progress. It sets an interim reliability milestone
and expects to be at least halfway toward the expected goal by the
time it begins to build production units. Caterpillar has learned
from experience that it will achieve the full reliability goal by
full production if it meets the interim goal by the time it
produces pilot production units. If the reliability is not growing
as expected, then decisions about changing or improving the design
must be addressed.
Caterpillar improves the product's reliability during
development by testing prototypes, uncovering failures, and
incorporating design changes. According to Caterpillar officials,
the production decision will be delayed if they are not on track to
meeting their reliability goal. These officials told us that
Caterpillar maintains the philosophy of first getting the design
right, then producing it as quickly and efficiently as possible.
They emphasized that demonstrating reliability before production
minimized the potential for costly design changes once the product
is fielded.
The commercial companies, after capturing specific manufacturing
knowledge, had executive level reviews to determine if the product
development had sufficiently progressed to permit a transition into
production. Executives used the knowledge captured as exit criteria
for the transition. For example, to demonstrate the product was
ready for production, critical processes had to be in control and
testing should have demonstrated the product reliability. If the
design team could not satisfy the exit criteria, then other options
had to be considered. The production decision led to increased
investments for materials and resources such as additional tooling
to build the product at a planned rate, facilities, people,
training and support.
When DOD Programs More Closely Approximated
Best Practices, Outcomes Were Better
Our analysis of DOD programs showed that those more closely
approximating best practices had better outcomes. The F/A-18 E/F
fighter and the AIM-9X missile were based extensively on
predecessor programs and employed similar tools to capture design
and manufacturing knowledge at critical program junctures. These
programs had demonstrated a significantly higher degree of design
stability prior to entering system demonstration and committing to
initial manufacturing when compared to other DOD weapon programs in
our review. They also gained control of most of their manufacturing
processes and demonstrated that the products were reliable before
entering production. The success of these programs is best
demonstrated by the fact that they have been close to meeting cost,
schedule, and performance objectives. On the other hand, the PAC-3
missile, F-22 fighter, and ATIRCM/CMWS programs did not use these
best practices. These programs were not based on predecessor
products or evolutionary in nature, and each product's full
capability was expected in one step, with the first product off the
production line. With this daunting task, these programs failed to
demonstrate a stable design before committing to initial
manufacturing, causing quality and labor problems. These programs
also had much less knowledge about the manufacturability of their
design when they entered production. As a result, they experienced
significant increases in development costs and production delays
usually at the expense of other DOD programs. Details on the five
DOD programs follow.
AIM-9X Missile Program The AIM-9X
development practices closely paralleled best practices used by the
commercial companies we visited. The program achieved design
stability before moving into system demonstration by incorporating
mature technologies and components from other missiles and
munitions, using engineering prototypes to demonstrate the design,
holding a series of design reviews prior to the system level
critical design review, and completing and releasing 95 percent of
the engineering drawings at that time. Figure 6 shows the building
of knowledge required to achieve a stable design on the AIM-9X.
Figure 6: Achieving Stability on AIM-9X Missile Program by
Knowledge Point 2
Source: GAO's analysis.
The AIM-9X program made extensive use of engineering prototypes
to stabilize the missile's design before building production
representative prototypes. Program officials stated that testing of
engineering prototypes uncovered problems with missile design and
manufacturing tooling early in the development, during system
integration, allowing time to re-design and re-test in follow-on
configurations. According to program officials, this not only
helped stabilize the design before entering initial manufacturing
but grew system reliability and reduced total ownership costs. The
program also held design reviews for each of the major subsystems,
allowing the program to achieve and demonstrate a stable design in
July 1999, before beginning initial manufacturing of production
representative prototypes.
While the AIM-9X used statistical process control only to a
limited extent, other factors have allowed it to have a more
successful production outcome to date. Program officials took steps
to ensure that manufacturing aspects of the product were included
in the design, including empowering a product leader with a
manufacturing background, identifying the key characteristics and
critical manufacturing processes early, making design trade-offs to
enhance manufacturing capability, and demonstrating a robust design
to make the product less vulnerable to variations in manufacturing
process. In addition, the ability to achieve design stability at
the critical design review allowed program officials to focus the
system demonstration phase on maturing the manufacturing processes.
Prior to committing to production, the program demonstrated that
the product could be efficiently
F/A-18 E/F Program
built using production processes, people, tools, and facilities
to build prototypes. According to the former program manager, these
steps gave the officials knowledge that a reliable product could be
produced within cost and schedule targets prior to entering
production. To date, the AIM-9X program has largely met its
production targets.
The F/A-18 E/F aircraft development program was able to take
advantage of knowledge captured in developing and manufacturing
prior versions of the aircraft. This evolutionary approach
significantly contributed to the cost and schedule successes of
this program. Because the F/A-18 E/F was a variant of the older
F/A-18 aircraft, the developer had prior knowledge of design and
manufacturing problems. This knowledge, coupled with the use of
modeling and computer-aided design software, helped create a design
that was easier to manufacture. While the program did not fully use
each of the best practices, it did embrace the concepts of
capturing design and manufacturing knowledge early in the
program.
During the program's critical design review, about 56 percent of
the drawings were completed and, while the program did not meet the
best practice of 90 percent complete, it did have additional
drawing data of the F/A-18 E/F assemblies available for review at
the critical design review. The Navy used early versions of the
F/A-18 aircraft to demonstrate new component designs and new
materials. In addition, the aircraft was designed to have 42
percent fewer parts than its predecessor, making its design more
robust. The program also identified the critical manufacturing
processes and collected statistical process control data early in
product development. At the start of production, 78 percent of
these critical processes were in control. Unit costs for the F/A-18
E/F program have not grown since the critical design review and its
schedule has been delayed by only 3 months.
F-22 Fighter Program
The F-22 program is structured to provide the product's full
capability with the first product off the production line-an
extreme design challenge. This required the product design to
include many new and unproven technologies, designs, and
manufacturing processes. It did not demonstrate design stability
until about 3 years after it held its critical design review. The
program completed 3,070 initial engineering drawings at its
critical design review in 1995, about 26 percent of the eventual
drawings needed. It did not complete 90 percent of the necessary
engineering drawings until 1998, after the first two development
aircraft were delivered. Figure 7 shows the drawing completion
history for the program.
Figure 7: History of Drawing Completion for the F-22 Program
Source: GAO's analysis.
After its critical design review, the F-22 program encountered
several design and manufacturing problems that resulted in design
changes, labor inefficiencies, cost increases, and schedule delays.
For example, delivery of the aft fuselage-the rear aircraft body
section-was late for several of the test aircraft and two ground
test articles because of late parts and difficulties with the
welding process. According to the F-22 program office, design
maturity and manufacturing problems caused a "rolling wave" effect
throughout system integration and final assembly. Late engineering
drawing releases to the factory floor resulted in parts shortages
and work performed out of sequence. These events contributed to
significant cost overruns and delays to aircraft deliveries to the
flight test program.
The F-22 program initially had taken steps to use statistical
process control data during development and gain control of
critical manufacturing processes by the full rate production
decision. In 1998,7 we reported that the program had identified 926
critical manufacturing processes and had almost 40 percent in
control 2 years before production was scheduled to begin. Although
this did not match the standard set by commercial companies, it
offered major improvements over what other DOD programs had
attempted or achieved. Unfortunately, citing budgetary constraints
and specific hardware quality problems that demanded attention, the
program abandoned this best practices approach in 2000 with less
than 50 percent of it critical manufacturing processes in control.
Currently, the program is using post-assembly inspection to
identify and fix defects rather than statistical process control
techniques to prevent them. In March 2002,8 we recommended that the
F-22 program office monitor the status of critical manufacturing
processes as the program proceeds toward high rate production. The
program stated that it would assess the processes status as the
program moves forward.
The program entered limited production despite being
substantially behind its plan to achieve reliability goals. A key
reliability requirement for the F-22 is mean time between
maintenance, defined as the number of operating hours for the
aircraft divided by the number of maintenance actions. The
reliability goal for the F-22 is a 3-hour mean time between
maintenance. The Air Force estimated that in late 2001, when the
F-22 entered limited production, it should have been able to
demonstrate almost 2 flying hours between maintenance actions.
However, when it actually began limited production it could only
fly an average of 0.44 hours between maintenance actions. In other
words, the F-22 is requiring significantly more maintenance actions
than planned. Additionally, the program has been slow to fix and
correct problems that have affected reliability. To date, the
program has identified about 260 different types of failures, such
as main landing gear tires wearing out more quickly than planned,
fasteners being damaged, and canopy delaminating. It has identified
fixes for less than 50 percent of these failures. Ideally, the
design fixes for the failures should be corrected prior to
manufacturing production units.
7 U.S. General Accounting Office, Best Practices: Successful
Application to Weapon Acquisition Requires Changes in DOD's
Environment
GAO/NSIAD-98-56(Washington, D.C.: Feb. 24, 1998).
8 U.S. General Accounting Office, Tactical Aircraft: F-22 Delays
Indicate Initial Production Rates Should Be Lower to Reduce Risks
GAO-02-298 (Washington, D.C.: Mar. 5, 2002).
Page 47 GAO-02-701 Best Practices
PAC-3 Missile Program The PAC-3 missile did
not achieve design stability until after the building of production
representative prototypes for system demonstration began. At the
program's critical design review, the PAC-3 program had completed
980 engineering drawings-21 percent of the eventual drawings needed
for the missile. Since then, almost 3,700 more drawings have been
completed. The total number of drawings expected to represent the
completed design grew from about 2,900 at the critical design
review to almost 4,700 as of July 2001. This uncertainty in the
expected drawings not only indicates that the design was not stable
when initial manufacturing began but also shows that there was a
significant lack of knowledge about the design. Figure 8 shows the
design knowledge at the critical design review, when the decision
was made to commit to initial manufacturing of the missile.
Figure 8: PAC-3 Design Knowledge at Critical Design Review
Source: GAO's analysis.
Prototypes of the product design were not built before the
critical design review or before initial manufacturing started to
show that the design would work. Therefore, because of the immature
design, initially manufactured development missiles were hand-made,
took longer to build than planned, and suffered from poor quality.
As a result, many design and manufacturing problems surfaced during
system demonstration. Subsystems did not fit together properly, and
many failed ground and environmental tests the first time. The
contractor attributed $100 million of additional cost to first time
manufacturing problems.
Prior to entering limited production in 1999, the program had
less than 40 percent of the critical manufacturing processes in
control for assembling the missile and the seeker. According to
program officials, there was little emphasis during development or
initial production on using statistical control on critical
manufacturing processes. Most of the development missiles were
built in specialty shops rather than in a manufacturing
environment. The result was a lack of knowledge about whether the
critical manufacturing processes could produce the product to
established cost, schedule, and quality targets. This uncertainty
is reflected in contractor estimates that more than 50 percent of
the time charged to build the initial production missiles will be
for engineering activities. Actual production labor is expected to
account for about 30 percent of the charged time.
To further understand the problems on the PAC-3 program, we
focused on its seeker subsystem, which is key to acquiring and
tracking targets and represents a large percentage of the missile's
cost. Currently, despite being in production, it is unclear whether
the supplier of the seeker can produce it within cost, schedule,
and quality targets. During development, the supplier had
difficulty in designing and manufacturing this subsystem. It was
not uncommon for seekers to be built, tested, and reworked seven or
eight times before they were acceptable. The program entered
production, despite these producibility issues. Now, even with 2
years of production experience, the supplier continues to have
difficulty producing the seeker with acceptable quality. Data
provided by the supplier in October 2001 showed that less than 25
percent of the seekers were being manufactured properly the first
time and the rest had to be reworked, on average, four times.
ATIRCM/CMWS Program According to program
officials, ATIRCM/CMWS did not have a stable design until about 2
years after the critical design review. A contributing factor to
this was a lack of understanding about the full requirements for
the new system at the critical design review in 1997. This led to a
major redesign of the common missile warning system's sensor. At
the critical design review, only 21 percent of a product's
engineering drawings had been completed. It did not complete 90
percent drawings-the best practice-until 1999. The immature design
caused inefficiencies in manufacturing, rework, and delayed
deliveries. In addition, between 1995 and 1999, the development
contract target price increased by 165 percent.
The ATIRCM/CMWS program did not begin reliability growth testing
until 4 years after its critical design review, leaving only 1 year
to test the system prior to scheduled production. Program officials
said that an immature design limited their ability to begin
reliability testing earlier in development. About one-third of the
way through the reliability growth test program, testing was halted
because too many failures occurred in components such as the power
supply, the high voltage electrical system, and the cooling system.
According to a program official, the inability to demonstrate
system reliability contributed to a production delay of about 1
year. The program plans to build, develop, and test six additional
development units during 2002 and 2003 that will incorporate design
changes to fix the system failures. ATIRCM/CMWS plans to enter
limited production in the early part of 2002 with significantly
less knowledge about the design's producibility than commercial
companies. The contractor does not use statistical process control
and has not identified critical manufacturing processes. A
production readiness review identified the lack of statistical
process control as a major weakness that needs to be corrected.
Chapter 4
A Better Match of Policy and Incentives Is Needed to Ensure
Capture of Design and Manufacturing Knowledge
The Department of Defense's (DOD) acquisition policy9
establishes a good framework for developing weapon systems;
however, disciplined adherence, more specific criteria, and
stronger acquisition incentives are needed to ensure the timely
capture and use of knowledge in decision making. DOD changed its
acquisition policy to emphasize evolutionary acquisition and
establish separate integration and demonstration phases in the
product development process. Its goal was to develop higher quality
systems in less time and for less cost. However, DOD's acquisition
policy lacks detailed criteria for capturing and using design and
manufacturing knowledge to facilitate better decisions and more
successful acquisition program outcomes. As demonstrated by
successful companies, using these criteria can help ensure that the
right knowledge is collected at the right time and that it will
provide the basis for key decisions to commit to significant
increases in investment as product development moves forward.
While the right policy and criteria are necessary to ensure a
disciplined, knowledge-based product development process, the
incentives that influence the key players in the acquisition
process will ultimately determine whether they will be used
effectively. In DOD, current incentives are geared toward delaying
knowledge so as not to jeopardize program funding. These incentives
undermine a knowledge-based process for making product development
decisions. Instead, program managers and contractors push the
capture of design and manufacturing knowledge to later in the
development program to avoid the identification of problems that
might stop or limit its funding. They focus more on meeting
schedules than capturing and having the knowledge necessary to make
the right decisions at those milestones. Such an approach
invariably leads to added costs because programs are forced to fix
problems late in development.
By contrast, commercial companies must develop high-quality
products quickly or they may not survive in the marketplace.
Because of this, they encourage their managers to capture product
design and manufacturing knowledge to identify and resolve problems
early in development, before making significant increases in their
investment. Instead of a scheduledriven process, their process is
driven by events that bring them
9 DOD Directive 5000.1, The Defense Acquisition System (Oct. 23,
2000), DOD Instruction 5000.2, Operation of the Defense Acquisition
System (Apr. 5, 2002), and DOD Regulation 5000.2-R, Mandatory
Procedures for Major Defense Acquisition Programs (MDAPS) and Major
Automated Information System (MAIS) Acquisition Programs (Apr. 5,
2002).
Page 52 GAO-02-701 Best Practices
Chapter 4 A Better Match of Policy and Incentives Is Needed to
Ensure Capture of Design and Manufacturing Knowledge
Acquisition Policy Lacks Specific
Implementation Criteria
knowledge: critical design reviews that are supported by
completed engineering drawings and production decisions that are
supported by reliability testing and statistical process control
data. They do not move forward without the design and manufacturing
knowledge needed to make informed decisions.
Greater emphasis on evolutionary acquisitions and structuring
the product development process into two phases-system integration
and system demonstration-were good first steps for DOD to achieve
its goals of buying higher quality systems in less time and for
lower costs. However, DOD policy still lacks criteria to be used to
capture specific design and manufacturing knowledge and does not
require the use of that knowledge as exit criteria at key decision
points to transition from system integration to system
demonstration and then into production. In three of the five DOD
program examples in chapter 3, managers decided to move forward in
development, even when developers had failed to capture design and
manufacturing knowledge to support increased investments. As a
result, these programs encountered significant increases in
acquisition costs as well as delays in delivering capabilities to
the war fighter.
Table 7 illustrates key criteria used by commercial companies
that are currently lacking in DOD's policy. The table shows the
design and manufacturing knowledge needed to make more informed
decisions. The capture of some of the important manufacturing and
reliability knowledge should begin in the integration phase in
order to have the full knowledge needed to make decisions at the
end of the demonstration phase for transitioning into
production.
Chapter 4 A Better Match of Policy and Incentives Is Needed to
Ensure Capture of Design and Manufacturing Knowledge
Table 7: Analysis of DOD Acquisition Policy for Inclusion of
Best Practices for Knowledge-based Design and Manufacturing
Decisions
According to DOD's current acquisition policy, the system
integration phase of an acquisition normally begins with the
decision to launch a program. The policy states that, during this
phase, a system's configuration should be documented and the system
should be demonstrated using prototypes in a relevant environment.
While these are noteworthy activities and resemble best practices,
the policy does not provide criteria for what constitutes the level
of knowledge required for completing this stage, nor does it
require a decision-based on those criteria-as to whether a
significant, additional investment should be made. Commercial
companies demand knowledge from virtual or engineering prototypes,
90 percent of required engineering drawings for the product
supported by test results, demonstration that the product meets
customer requirements, a series of disciplined design reviews, and
stakeholder agreement that the design is stable and ready for
product demonstration before a commitment is made to move forward
and invest in product demonstration. Under DOD's revised policy, it
is still difficult to determine if a product should enter product
demonstration with a stable design.
Chapter 4 A Better Match of Policy and Incentives Is Needed to
Ensure Capture of Design and Manufacturing Knowledge
DOD's current acquisition policy also states that the system
demonstration phase begins after prototypes have been built and
demonstrated in a relevant environment during system integration.
According to the policy, a system must be demonstrated before the
department will commit to production. The low-rate initial
production decision occurs after this phase of product development.
Like the end of system integration, the policy fails to provide
specific criteria for what constitutes the knowledge required to
support the decision to move into production. For example, the
policy states there should be "no significant manufacturing risks"
but does not define what this means or how it is measured. Without
criteria for building knowledge during the demonstration phase, the
production decision is often based on insufficient knowledge,
creating a higher probability of inconsistent results and cost and
schedule problems. On the other hand, commercial companies demand
proof that manufacturing processes are in control and product
reliability goals are attained before committing to production.
With more specific knowledge in hand at the end of development,
decision makers can make a more informed decision to move into
production with assurances that the product will achieve its cost,
schedule, and quality outcomes.
Finally, while DOD's policy separates product development into a
two-stage process-integration and demonstration-it does not require
a decision milestone to move from one stage to the next. The policy
states that an interim progress review should be held between the
two stages, but the review has no established agenda and no
required outputs of information unless specifically requested by
the decision maker. Its purpose is to confirm that the program is
progressing as planned. On the other hand, commercial companies
consider this review a critical decision point in their product
development process because it precedes a commitment to
significantly increase their investment. Therefore, they use
specific, knowledge-based standards and criteria to determine if
the product is ready to enter the next phase and they hold decision
makers accountable for their actions. These decision reviews are
mandatory and are typically held at the executive level of the
commercial firm.
Figure 9 illustrates the commercial model for knowledge to be
captured and delivered during product integration and product
demonstration and the possible application of that model to DOD's
acquisition process. Without a similar decision review to bring
accountability to the DOD process, acquisition programs can-and
do-continue to advance into system demonstration without a stable
design. As shown in our case
Chapter 4 A Better Match of Policy and Incentives Is Needed to
Ensure Capture of Design and Manufacturing Knowledge
studies, this provides for a high probability of cost growth and
schedule delays to occur.
Figure 9: Illustration to Show How the Best Practice Model Would
Apply to DOD's Acquisition Process
Source: GAO's analysis.
Chapter 4 A Better Match of Policy and Incentives Is Needed to
Ensure Capture of Design and Manufacturing Knowledge
Incentives in the DOD Acquisition
Environment Do Not Favor Capture of Design and Manufacturing
Knowledge Early Enough
The incentives for program managers and product developers to
gather knowledge and reduce risk are also critical to DOD's ability
to adopt best practices for product development. In DOD, incentives
are centered on obtaining scarce funding on an annual basis in a
competitive environment to meet predetermined and typically
optimistic program schedules. These incentives actually work
against the timely capture of knowledge, pushing it off until late
in the process to avoid problems that might keep a program from
being funded. Because design and manufacturing knowledge is not
captured, key decision points intended to measure and ensure that a
weapon system has sufficiently matured to move forward in the
process risk becoming unsupported by critical knowledge. In leading
commercial companies, the opposite is true. Because companies know
they have to deliver high-quality products quickly and affordably,
they limit the challenge for their program managers and provide
strong incentives to capture design and manufacturing knowledge
early in the process. Program managers are empowered to make
informed decisions before big investments in manufacturing
capability are required.
DOD's current acquisition environment is driven by incentives to
make decisions while significant unknowns about the system's design
and manufacturability persist. This environment results in higher
risks and a greater reliance on cost-reimbursement10 contracts for
longer periods of time during product development. Because events
that should drive key decisions, such as critical design reviews,
interim progress reviews, and production decision reviews, are
based on inadequate design and manufacturing knowledge, they do not
support decisions to invest more and move to the next phase of the
acquisition process. Nevertheless, this approach has proven
effective in securing funds year to year. For example, the F-22,
PAC-3, and ATIRCMS/CMWS programs had less than one-third of their
engineering drawings completed at their critical design review, but
each obtained the funding necessary to move onto the initial
manufacturing of production representative prototypes. That funding
allowed a significant increase in investment to develop a
manufacturing capability before critical
10 Cost-reimbursement contracts provide for payment of allowable
incurred costs, to the extent prescribed in the contracts. They are
suitable for use only when uncertainties involved in contract
performance, such as research and development work, do not permit
costs to be estimated with sufficient accuracy. In contrast,
fixed-priced contracts, except those subject to price adjustment,
provide for a preestablished firm price, place maximum risk and
full responsibility for all costs and resulting profit or loss on
the contractor, and provide maximum incentive for the contractor to
control costs and perform effectively.
Chapter 4 A Better Match of Policy and Incentives Is Needed to
Ensure Capture of Design and Manufacturing Knowledge
knowledge had been captured. The incentive to capture funding
for the program was greater than the incentive to wait, capture
knowledge, and reduce the risk of moving forward. Each of these
programs encountered significant cost increases and schedule
delays.
The incentives are quite different for leading commercial
companies. For them, the business case centers on the ability to
produce a product that the customer will buy and that will provide
an acceptable return on investment. If the firm has not made a
sound business case, or has been unable to deliver on one or more
of the business case factors, it faces a very real prospect of
failure-the customer may walk away. Also, if one product
development takes more time and money to complete than expected, it
denies the firm opportunities to invest those resources in other
products. For these reasons, commercial companies have strong
incentives to capture product knowledge early in the process to
assess the chances of making the business case and the need for
further investments.
Production is a dominant concern in commercial companies
throughout the product development process and forces discipline
and trade-offs in the design process. This environment encourages
realistic assessments of risks and costs since doing otherwise
would threaten the business case and invite failure. For the same
reasons, the environment places a high value on knowledge for
making decisions. Program managers have good reasons to identify
risks early, be intolerant of unknowns, and not rely on testing
late in the process as the main vehicle for discovering the
performance characteristics of the product. By adhering to the
business case as the key to success, program managers in leading
commercial companies are conservative in their estimates and
aggressive in risk reduction. Ultimately, adherence to the business
case strengthens the ability to say "no" to pressures to accept
high risks and unknowns. Practices such as prototyping, early
manufacturing and supplier involvement, completing 90 percent of
engineering drawings by critical design review, demonstrating
product reliability, and achieving statistical control of critical
manufacturing processes by production are adopted because they help
ensure success.
In DOD's current acquisition environment, the customer is
willing to trade time and money for the highest performing weapon
system possible. That willingness drives the business case. This
creates strong incentives for the program office to take
significant risks with technologies and designs to ensure it can
offer the customer a weapon system that is a quantum leap above the
competition. In addition, because funding is secured on an
Chapter 4 A Better Match of Policy and Incentives Is Needed to
Ensure Capture of Design and Manufacturing Knowledge
annual basis in DOD, strong incentives exist for the program
office to make optimistic assumptions about development cost and
schedule. Because the customer is willing to wait and funding is
never certain, an environment exists where program managers have
good reasons to avoid the capture of knowledge and delay testing.
Since the business case in DOD places very little premium on
meeting cost and schedule targets, but a very high premium on
performance, programs succeed at the point where sunk costs make it
difficult-if not prohibitive-for decision makers to cancel
them.
The practices commercial companies use to capture knowledge are
not currently used in this environment because the business case
does not favor them. Instead, DOD's product development environment
relies on cost-type contracting throughout the entire product
development process. Once in production, programs will cut
quantities to maintain funding or once fielded, they rely on the
operations and maintenance budget to pay for reliability problems
not solved in development.
Chapter 5
Conclusions and Recommendations
The Department of Defense's (DOD) planned
$700 billion investment in
Conclusions
weapon systems over the next 5 years requires an approach that
keeps cost, schedule, and performance risks to a minimum. This
approach means adopting and implementing an evolutionary approach
to developing new weapon systems, improving policy to more closely
approximate a knowledge-based product development process, and
creating incentives for capturing and using knowledge for decision
making. Without an evolutionary approach as its foundation, the
ability to capture design and manufacturing knowledge early in the
development process is significantly reduced. Programs, in turn,
take on too much new unproven content to meet their objectives and
risks invariably increase. DOD has made improvements in its
acquisition policy by incorporating guidance for evolutionary
acquisition, creating guidelines for the development of a basic
product that can be upgraded with additional capabilities as
technologies present themselves. However, evolutionary acquisition
has yet to be consistently implemented with success on individual
weapon system acquisitions.
Regardless of whether DOD emphasized greater use of evolutionary
acquisition, acquisition programs are not capturing sufficient
design and manufacturing knowledge to make good decisions at key
investment points. The current policy establishes a good framework
to develop a product, but the policy still lacks specific criteria
required to move a program forward and does not tie knowledge to
decisions for increasing investments in the program as it moves
from system integration to system demonstration. As a result,
programs often pass through each development phase and into
production with an unstable design and insufficient knowledge about
critical manufacturing processes and product reliability. This
results in greater likelihood for inconsistent and poor results and
cost and schedule problems later in the program.
Additionally, DOD does not provide the proper incentives to
encourage the use of best practices in capturing knowledge early in
its development programs. Currently, managers are focused more on
the annual exercise of obtaining funding needed to keep their
programs viable and alive. The importance of capturing design and
manufacturing knowledge early gives way to the pressures of
maintaining funding, often resulting in the acceptance of greater
risks. Raising problems on a program early because design and
manufacturing knowledge is discovered can cause extra oversight and
questions that threaten a system's survival. The prevailing
Recommendations for Executive Action
culture is to accept greater risks upfront and then fix problems
later in the development program.
We found that leading commercial companies over the years had
found ways to overcome these problems and had identified best
practices that resulted in the early capture of and use of design
and manufacturing knowledge. This was done by a combination of four
key elements. First, they established and used an evolutionary
approach to develop products that made the capture of design and
manufacturing knowledge a more manageable task. This framework
limited the design challenge for any one new product development by
allowing risky technology, design, or manufacturing requirements to
be deferred until a future generation of the product. DOD's current
policy addresses this; however, it has not had sufficient time to
show how this will be implemented.
Second, each company we visited used the same basic product
development process and criteria for bringing together and
integrating all of the technologies, components, and subsystems
required for the product to ensure the design was stable and then
demonstrating that the product was producible and reliable using
proven manufacturing processes. DOD's policy lacks the criteria to
measure design stability and process controls. Third, successful
companies used tools to capture design and manufacturing knowledge
about the product and decide about whether to invest further based
on that knowledge. Their new product development process included
key, high-level decision points before moving into product
demonstration, and again before making the production decision that
required specific, knowledge-based exit criteria. DOD's policy does
not require a decision to move from system integration to system
demonstration. Finally, leading companies created an environment
for their managers that emphasized capturing design and
manufacturing knowledge early, before committing substantial
investments in a product development that made cancellation a more
difficult decision to make. DOD's environment encourages meeting
schedule milestones instead of capturing design and manufacturing
knowledge to make decisions.
DOD should take steps to close the gaps between its current
acquisition environment and best practices. To do this, it should
ensure that its acquisition process captures specific design and
manufacturing knowledge, includes decisions at key junctures in the
development program, and provides incentives to use a
knowledge-based process. Such changes are necessary to obtain
greater predictability in weapon system
Page 61 GAO-02-701 Best Practices
programs' cost and schedule, to improve the quality of weapon
systems once fielded, and to deliver new capability to the war
fighter faster. More specifically, we recommend that the Secretary
of Defense:
• Require the capture of specific knowledge to be used as exit
criteria for decision making at two key points-when transitioning
from system integration to system demonstration and from system
demonstration into production. The knowledge to be captured when
moving from system integration into system demonstration should
include the following:
•
Completed subsystem and system design reviews.
•
Ninety percent of drawings completed.
•
Demonstration that design meets requirements-prototype or
variant testing.
•
Stakeholders' (cross functional design team that includes
design engineers, manufacturing, key supplier) assurance that
drawings are complete.
•
Completed failure modes and effects analysis.
•
Identification of key system characteristics.
•
Identification of critical manufacturing
processes.
•
Set reliability targets and growth plan.
The knowledge to be captured when moving from system
demonstration into production should include the following:
•
Demonstrated manufacturing processes.
•
Built production representative prototypes.
•
Tested prototypes to achieve reliability goal.
•
Tested prototypes to demonstrate product in operational
environment.
•
Collected statistical process control data.
•
Demonstration that critical processes are capable and in
control.
•
Require that the interim progress review, currently
identified in DOD's policy as that point in the process between
system integration and system demonstration, be a mandatory
decision review. At this point, the design should be demonstrated
to be stable so that during the next phase of development attention
can be focused on demonstrating manufacturing processes and product
reliability. The program manager should have proof-based on the
exit criteria for moving out of system integration in the above
recommendation-that the product design is stable. The exit criteria
should be demonstrated and verified by the program manager before
the program can make the substantial investments needed to begin
manufacturing production representative prototypes in the next
phase of development-system demonstration. To ensure visibility of
demonstrated exit criteria to decision makers, the criteria and the
program's status in achieving them should be included in each
program's Defense Acquisition Executive Summary and Selected
Acquisition Reports. If the program does not meet the exit
criteria, investments should be delayed until such time as the
criteria are satisfied. To proceed without completing the required
demonstrations should require approval by the decision
authority.
•
Expand exit criteria for the Milestone C decision to
include the knowledge to be captured during the system
demonstration phase as identified in recommendation one. This will
require that the program office demonstrate that the critical
manufacturing processes are under statistical control and that
product reliability has been demonstrated before entering
production of the new weapon system. These are best practices and
indicate that the product design is mature and the program is ready
to begin production of units for operational use that will meet the
cost, schedule, and quality goals of the program.
•
To ensure that contracts support a knowledge-based
process, we further recommend that DOD structure its contracts for
major weapon system acquisitions so that (a) the capture and use of
knowledge described in recommendation one for beginning system
demonstration is a basis for DOD's decision to invest in the
manufacturing capability to build initial prototypes and (b) the
capture and use of manufacturing and reliability knowledge
discussed in recommendation one for moving from system
Agency Comments and Our Evaluation
demonstration to production is a basis for DOD's decision to
invest in production.
DOD concurred with a draft of this report and agreed with the
benefits of using design and manufacturing knowledge to make
informed decisions at key points in a system acquisition program.
DOD had some comments with regard to the details contained in the
recommendations, which are summarized below. DOD concurred with our
recommendation to add exit criteria at two key points in the
acquisition process-when transitioning from system integration to
system demonstration and from system demonstration into production.
DOD believes, however, that the milestone decision authority needs
to retain flexibility in applying the knowledge requirement for
drawings. Flexibility and judgment are management prerogatives that
should exist in any decision process. We agree there may be
circumstances, such as in the development of software, when it
makes good sense to progress with less than the best practice
standard for drawings, but the DOD policy should maintain the
requirement to achieve 90 percent drawings by the completion of the
system integration phase.
DOD also concurred that critical manufacturing processes must be
demonstrated using statistical process control techniques before
production, but believes that achieving this at Milestone C, the
low rate production decision, is unlikely. It believes the criteria
would be better applied to the full rate production decision or
when low rate production quantities extend beyond 10 percent of the
planned weapon system buy. This is a reasonable approach when
processes are new or unique. However, not all critical processes
will be new or unique to a specific weapon system. Some will have
been used to manufacture parts or components for other systems or
products. At a minimum, it should be possible to demonstrate these
by Milestone C. For other critical processes that may require
additional production experience to bring under statistical process
control, a program manager should have a reasonable plan at the
Milestone C decision review to bring those processes into control
by the full rate production decision, but no later than completion
of 10 percent of the planned buy.
Appendix I
Comments from the Department of Defense
Appendix II
GAO Staff Acknowledgments
Cheryl Andrew, Cristina Chaplain, Michael
Hazard, Matthew Lea, Gary
Acknowledgments
Middleton, Michael Sullivan, Katrina Taylor, and Adam
Vodraska.
Related GAO Products
Defense Acquisitions: DOD Faces Challenges in Implementing Best
Practices.
GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead
to Better Weapon System Space Outcomes.
GAO-01-288. Washington, D.C.: March 8, 2001.
Best Practices: A More Constructive Test Approach Is Key to
Better Weapon System Outcomes.
GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000.
Defense Acquisition: Employing Best Practices Can Shape Better
Weapon System Decisions.
GAO/T-NSIAD-00-137. Washington, D.C.: April 26,
2000.
Best Practices: DOD Training Can Do More to Help Weapon System
Programs Implement Best Practices.
GAO/NSIAD-99-206. Washington,D.C.: August 16,
1999.
Best Practices: Better Management of Technology Development Can
Improve Weapon System Outcomes.
GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999.
Defense Acquisition: Best Commercial Practices Can Improve
Program Outcomes.
GAO/T-NSIAD-99-116. Washington, D.C.: March 17,
1999.
Defense Acquisition: Improved Program Outcomes Are Possible.
GAO/T-NSIAD-98-123. Washington, D.C.: March 18,
1998.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon
System Programs.
GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.
Best Practices: Successful Application to Weapon Acquisitions
Requires Changes in DOD's Environment.
GAO/NSIAD-98-56. Washington, D.C.:February 24,
1998.
Major Acquisitions: Significant Changes Underway in DOD's Earned
Value Management Process.
GAO/NSIAD-97-108. Washington, D.C.: May 5, 1997.
Best Practices: Commercial Quality Assurance Practices Offer
Improvements for DOD.
GAO/NSIAD-96-162. Washington, D.C.: August 26,
1996.
GAO's Mission
The General Accounting Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and
accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs
and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and
funding decisions. GAO's commitment to good government is reflected
in its core values of accountability, integrity, and
reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at
no cost is through the Internet. GAO'
s Web site (www.gao.gov) contains abstracts and
fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to
help you locate documents using key words and phrases. You can
print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports,
testimony, and correspondence. GAO posts this list, known as
"Today's Reports," on its Web site daily. The list contains links
to the full-text document files. To have GAO e-mail this list to
you every afternoon, go to
www.gao.govand select "Subscribe to daily E-mail alert
for newly released products" under the GAO Reports heading.
Order by Mail or Phone
The first copy of each printed report is free. Additional copies
are $2 each. A check or money order should be made out to the
Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are
discounted 25 percent. Orders should be sent to:
U.S. General Accounting Office 441 G Street NW, Room LM
Washington, D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537
Fax: (202) 512-6061
Contact:
To Report Fraud, Web site:
www.gao.gov/fraudnet/fraudnet.htm
E-mail:
[email protected]
Federal Programs Automated answering system: (800) 424-5454 or
(202) 512-7470
Jeff Nelligan, managing director,
[email protected](202) 512-4800
Public Affairs
U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
Presorted Standard Postage & Fees Paid GAO Permit No.
GI00
United States General Accounting Office Washington, D.C.
20548-0001
Official Business Penalty for Private Use $300
Address Service Requested
