How do software developers understand and measure success

How do software developers understand and measure success?
ABSTRACT
In this research project, I explored how software developers understand and measure the
success of the projects. Analyzing qualitative data collected through three semi-structured
interviews, I find that time, cost, project specifications, and customer satisfaction are the
common measures that software developers use to measure the success of the projects.
Moreover, one interviewee suggested that the financial performance of the project is also an
important component of measuring the success of the project along with cost, project
specifications, and customer satisfaction. Feedback from employees, management,
customers, and the market is an important source to measure the project’s success.
i
Contents
CHAPTER 1 ……………………………………………………………………………………………………………… 1
1.
Introduction ………………………………………………………………………………………………………… 1
CHAPTER 2 ……………………………………………………………………………………………………………… 2
2.
Literature Review………………………………………………………………………………………………… 2
2.1.
Introduction ………………………………………………………………………………………………….. 2
2.2.
Measuring the success of the project………………………………………………………………… 2
2.3.
Summary of the literature review …………………………………………………………………….. 4
CHAPTER 3 ……………………………………………………………………………………………………………… 5
3.
Research Methodology ………………………………………………………………………………………… 5
CHAPTER 4 ……………………………………………………………………………………………………………… 7
4.
Findings……………………………………………………………………………………………………………… 7
4.1.
Introduction ………………………………………………………………………………………………….. 7
4.2.
Measuring the success of the projects ………………………………………………………………. 7
4.2.1.
Time as success measure …………………………………………………………………………. 7
4.2.2.
Cost/budget as a success measure ……………………………………………………………… 7
4.2.3.
Meeting specifications as a measurement of the project success …………………… 8
4.2.4.
Customer satisfaction is a measurement of the project success ……………………… 9
4.3.
Sources of measuring success …………………………………………………………………………. 9
4.4.
Summary of chapter …………………………………………………………………………………….. 10
CHAPTER 5 ……………………………………………………………………………………………………………. 11
5.
Discussion and conclusion ………………………………………………………………………………….. 11
References ……………………………………………………………………………………………………………….. 12
Appendices ………………………………………………………………………………………………………………. 14
Appendix (A) ……………………………………………………………………………………………………….. 14
ii
CHAPTER 1
1. Introduction
Success has been understood and defined differently by different people and in different
contexts. The quest for how to measure the success of the project has the success of projects has
resulted in a plethora of publications (e.g., Pinto and Slevin, 1988; Larson and Gobeli, 1989;
Shenhar et al., 2001; Dolio, 2009; de Bakker et al., 2010) though without consensus on
conceptualizing the notion “how to measure the success”.
Given the lack of consensus on how to define and measure the success of a project and the
importance of measuring the success of the project in the overall performance of the teams and
organizations (Ahadzie et al., 2008; Yo et al., 2005), this research project goes some way to
explore and understand the way software developers understand and measure the success of a
project by focussing on answering the following research question.
Research Question: How do software developers understand and measure success?
The structure of this research report is presented in the following figure 1.
Chapter 1: Introduction
Chapter 2: Literature review
Chapter 3: Research Methodology
Chapter 4: Findings
Chapter 5: Discussion and
conclusion
Figure (1): Structure of the report
1
CHAPTER 2
2. Literature Review
2.1.
Introduction
A few of the previous studies about measuring the success of the projects have been reviewed in
this chapter. The purpose is to analyze and understand different views about the notion of
measuring the success of the projects.
2.2.
Measuring the success of the project
Measuring “success” is a matter of full of complexities as it is contextual and has different
meanings for different people (e.g., Glass, 2001; Chan and Chan, 2004). A project can be a
success for one person, team, or a stakeholder group but a failure for the others (Clarke, 1999).
There is no exception for information technology and information system projects (Procaccino
and Verner, 2006) in this regard. Literature on measuring the project success does not seem to
show consensus on what determines is the success of a project or how to measure the success of
the projects (e.g., Kappelman et al., 2006; Anda et al., 2009; Shenhar et al., 2001).
Many researchers suggest measures of project success based on the completion of the project in
terms of meeting budget, budget, and quality (Atkinson, 1999; Wateridge, 1995; Anda et al.,
2009). Such a criterion based on time, cost, and quality, termed as “Golden Triangle” or “Iron
Triangle’’ suggests that if a project is completed within the schedule and budget limits and meets
all the requirements set by the client is a successful project (e.g., Wateridge, 1998 Atkinson,
1999; Kappelman et al., 2006).
However, there exist many examples of the projects in the literature which have been considered
successful in terms of budget, time, and meeting requirements. But these projects have not been
a success from the client’s satisfaction perspective. Ahonen and Savolainen (2010) studied five
different software projects which were completed and delivered to the clients in time and were
completed within estimated budgets. Moreover, all the projects were according to the
specifications and clients made the payments as well. Seen in this way, these projects were
successful if seen from the supplier perspective and measured according to the Gold Triangle
criteria (Belout and Gauvreau, 2004). However, clients were not satisfied; they did not
implement these projects and never used them. Therefore, from clients’ perspectives, these
projects are failures. Hence, measuring the projects against time, cost and requirement is not
adequate. Rather these projects suggest, as proposed by Ahonen and Savolainen (2010), client
2
satisfaction is another dimension, vital to measuring the success of the project. In other words,
even if the projects are successful in terms of developers’ perspectives measured against the
Golden Triangle, the projects are not successful from the clients’ perspective.
The situation can be the other way round too. The projects can be categorized as failures in terms
of time and cost overruns, a process failure (Lyytinen and Hirschheim, 1987). These projects can
be successful in terms of business point of view. Shenhar et al (2001) discuss a construction
project, the Sydney Opera house construction project, to explain such a project. Shenhar et al
(2001) state that this project was a failure if measured through the basic criteria of time and cost
as this project were time over-run and cost over-run. This project was completed in three times
more than the estimated time and the cost was approximately five times more than the budgeted
cost. However, this project is one of the famous landmarks in Australia and tourists do not leave
the country without vesting this landmark. Hence, the business perspective of the project can be
argued as a key factor in measuring the success of the projects.
Therefore, as argued by Shenhar et al (2001) and empirically examined by Ittner and Larcker
(1997), business perspective and performance is an important component of measuring the
success of the project instead of relying on time, cost, and specification. de Wit’s (1988)
definition of project success offers a broader perspective of the success of the project as this
definition sums up the entire discussion.
“The project is considered an overall success if the project meets the technical
performance specifications and/or missions to be performed, and if there is a high level
of satisfaction concerning the project outcome among key people in the parent
organization, key people in the project team, and key users or clientele of the project
effort” (de Wit, 1988: 165).
3
2.3.
Summary of the literature review
Following figure 2 offers the summary of the literature review and has been used as a framework
to analyze the data, collected through semi-structured interviews.
Time
Specifica
tions
Budget
Project
Customer
Success
satisfaction
Business
Performance
Project
team
satisfaction
Figure 2: Criteria project success
4
CHAPTER 3
3. Research Methodology
Qualitative research methodology has been used to explore the research question “How do
software developers understand and measure success?” as the purpose is has been not to test
existing theories or hypotheses about how to measure the success of the project. The purpose has
been to offer a theoretical perspective about the ways software developers understand and
measure the success of the project. Hence, as suggested by Easterby-Smith et al (2008),
qualitative research methodology has been used for the purpose building theory about how do
software developers understand and measure success instead of testing any previous theories.
Data has been collected using semi-structured interviews. As Creswell (2003), semi-structured
interviews offer in-depth exploration of a social phenomenon and as the purpose of this research
project has been to get an in-depth understanding of how software developers understand and
measure the success of the project, semi-structured interviews have been used to collect data.
I conducted one interview with an associate research professor who is involved in the design and
development of different information technology-related projects in a team of 6 members. The
other two interviews which I included in this research project were conducted by two other
students. The detail of the interviewees is as under:
I conducted the interview with Z.L. who is for Shenzhen Institutes of Advanced Technology, a
subsidiary of the Chinese Academy of Sciences. The interviewee is working in the department
which is in charge of the development and production of the integrated, intelligent, and
automatic and network medical system and equipment. The interview has experience of 4 years
in this industry and his current job title is associate research professor, in charge of a team
consisting of six members and the team is responsible for the modular design and development
such as medical image systems and equipment including X-rays, three-dimensional ultrasound
scans, computer graphics, tiny cameras, and other technologies.
The other interviewee, P.S. is working as a Senior Analyst in an IT company which is called IT
One. It is a company that is based in Thailand and the interviewee’s main responsibility is to
design and sometimes code applications for the clients.
The third interviewee whose interview I included in this research project is D.K., who is working
in IMB Korea. D.K. works in Global Process Services (GPS). The clients of the company in
5
whi* D.K works include Samsung electronics and Amor Pacific, a leading company in the
Korean cosmetic industry.
I analyzed the data using coding following Turner (1981). Using open coding main themes, I
identified and labeled the main themes. I used the axial coding concept to link different themes.
Moreover, I used the concept of selective coding to integrate and refine theories. Following table
(1), themes/codes/interviewee tick table provides a summary of themes and codes identified and
labeled in the three interviews of P.S., D.K., and Z.L.
Themes
Codes
Interviewees
P.S
D.K
Z.L
Specifications
√
√
√
Time
√
√
√
Measuring
Cost
√
√
×
success
Customer satisfaction
√
√
√
Business performance
×
√
×
Sources of
Internal evaluation
√
√
√
measurement
Customer feedback
√
√
√
Market feedback
×
×
√
These themes have been discussed in the next chapter 4.
6
CHAPTER 4
4. Findings
4.1.
Introduction
This chapter discusses the main themes identified from the data analysis. The detail of the
themes has been presented in Appendix (A), themes/codes/quotations table. Each theme has
been discussed separately. For each theme, the significance, implications, and relationship with
the literature and to the research question have been discussed. Verbatim quotations have been
used to support the themes.
4.2.
Measuring the success of the projects
The project success is measured using different measures including time, cost, specifications,
customer satisfaction, and business performance. Time, cost, specifications, and customer
satisfaction have been found as common measures of the success of the project.
4.2.1. Time as a success measure
Time is one of the key measures of the success of the project and has been a common theme in
all three interviewees. For instance, according to Z.L,
“The schedule is more important to us. It directly affects the following process if failed to
accomplish the task on time. The whole project will then get in trouble” (Z.L)
This shows that Z.L includes time as one of the key measures of the success of the project.
Similarly, another interviewee, D.K. says that time is an important measure of the project’s
success.
“If we don’t finish on time, there will be a penalty usually 1/1000 of total project cost per
day”. (D.K)
P.S also indicates that time delays have negative effects on the budget and therefore, time is an
important measure in this regard. Hence, it is concluded that time is understood as a key measure
of the success of the projects by the software developers. This is keeping with previous theories
and empirical studies which suggest that time is one of the key measures of the success of the
project (e.g., Atkinson, 1999; Kappelman et al., 2006).
4.2.2. Cost/budget as a success measure
The cost of the project has been found another important measure of the project’s success.
7
“For IBM, we need to make a profit. For example, we planed make 30% profit, but we
cost more. Finally, we only make 5% profit. We call it to fail” (D.K)
Another interviewee has a similar opinion that budget is amongst the key elements of measuring
the success criteria of the projects.
“Yes, the contract of fixed schedule will be based on the man-day system to estimate the
man-day that use for each project. The man-day system can help us to arrange the
resource and to prevent the over-budget issues as well” (P.S).
In this way, this is line with the previous studies such as Kappelman et al (2006) and Anda et al
(2009). These studies show that completing the projects within the estimated budget is a key
element in measuring the success of the project.
However, Z.L. suggests that though budget is important but not the key success criteria.
“We usually get sufficient financial support from government funding and cooperative
organizations, thus the budget effect to be small despite there is indeed budget limitation
in every project” (Z.L).
Hence, this contrasts with the findings of the studies which include cost as the key measuring
criteria of the project’s success (Kappelman et al., 2006; Anda et al., 2009).
4.2.3. Meeting specifications as a measurement of the project success
Meeting specifications and requirements of the project have been found one of the main
components of the project success measuring criteria. All three interviewees suggest that
meeting projects specification is vital in terms of measuring the projects’ success.
Other indicators, actually we do not have but for me, we should focus on the design. If the
design can respond to the need of the customer and give them satisfaction, I think that one can
be the key indicator to measure the quality of the project.
“Amorepacific project, I think it is a successful one. It is a 3 years project. For the
project, it is clear to say whether it is successful or not. If we meet the requirements, it is
successful” (D.K)
Z.L also has the similar views
“All projects approval and establishment come with contracts which contain particular
acceptance criteria. The acceptance criteria have two parts. First is technical indicators
8
such as production precision, stability, safety, functionality, etc. Second is the indicator of
industrialization. …..All productions will carry out the acceptance tests by these
organizations” (Z.L)
This agrees with the findings of Anda et al (2009), who find that meeting project’s specification
is key in measuring the success of the project.
4.2.4. Customer satisfaction is a measurement of the project success
Customer satisfaction is understood by software developers as one of the important measures of
the success of the projects.
“Sometimes we can rank the quality of each project by the number of incoming calls
from the customers. If some projects have too much request from customers, we can say
that project may have a problem” (P.S).
Similarly, D.K also suggests customer satisfaction as a major component of measuring project
success.
“Yeah, customers. At the end of the project, customers will sign a project conclusion
document. We have a client satisfaction survey, which we can check (D.K).
Z.L also has a similar point of view. Therefore, software developers understand that customer
satisfaction is an important component of measuring the project success and they measure the
project success using customer satisfaction amongst the measuring criteria of the project success.
Hence, this is keeping with Ahonen and Savolainen (2010), who find that customer satisfaction,
is an important measure of the project’s success.
Business performance is also a vital element of measuring the success of the project. However,
only D.K suggests that business performance is a key measure of success. According to D.K
“For IBM, we need to make a profit. For example, we planed make 30% profit, but we
cost more. Finally, we only make 5% profit. We call it to fail” (D.K).
This is keeping with Shenhar et al (2001) who argue that other than the traditional measurement,
business performance is an important measurement of the success of the projects.
4.3.
Sources of measuring success
Who offers the feedback or evaluates the projects has been one of the themes that emerged from
the data. Feedback from the customers, managers, and, in some instances, from the market is the
major source of evaluating success. According to Z.L,
9
“Yes. There are three groups in charge of acceptance tests during each timeline. First is
the intra-team acceptance test, including my boss and all team members. Second, the
acceptance test will be performed by a qualified third-party assessment entity. The third
is the end-user (Z.L).
P.S states that
“Apart from my boss, there is my team lead of the project which we do the face-to-face
evaluation to discuss all the skills we need to focus which involve for that project” (P.S)
Hence, both internal and external feedbacks are important in evaluating the success of the
project during the development and after the development.
4.4.
Summary of the chapter
From the data analysis, it has been found that software developers use time, cost, project
specifications, customer satisfaction to measure the success of the project. Moreover, only in
IBM, software developers also measure the success against the financial performance of the
project. The projects are evaluated both from internal and external sources. Internally,
management and other team members evaluate the success of the project and the performance of
the individuals. On the other hand, market and customer feedbacks also contribute to the
projects’ success measurement.
10
CHAPTER 5
5. Discussion and conclusion
Literature shows no consensus on how to measure the success of the projects. However,
measuring the success of the projects is important because it helps to improve the performance
of the individuals and teams (Doloi, 2009). Given the lack of consensus on how to define and
measure the success of a project and the importance of measuring the success of the project, this
research project explored “how do software developers understand and measure success”?
By analyzing the qualitative data, collected using semi-structured interviews, this research
project reveals that software developers understand that time, cost, and project specifications are
important measures of the project success. They are more focused on meeting specifications of
the project, time and cost, Golden Triangle of measuring project success (Kappelman et al.,
2006). Moreover, software developers also include customer satisfaction as an important
element in measuring project success. However, only D.K of IBM suggests that the business
perspective and performance is an important criteria to measure the success of the project
(Shenhar et al., 2001). Feedback from customers and market, and internal feedback play
important role in evaluating the success of the projects.
However, these findings are based on a study that has certain limitations, mainly due to data
collection. Only three interviews have been used to reach these conclusions. Moreover,
interviews would offer better and deep insights into the ways software developers understand
and measure success.
To meet ethical considerations, the respondents were informed about the purpose and nature of
the research. Written consent was obtained. Moreover, anonymity has been kept and the data
will only be used for the purpose. i.e. academic purpose, as stated to the respondents
11
References
Anda, B.C.D., Sjoberg, D.I.K., and Mockus, A. (2009). Variability and reproducibility in
software engineering: a study of four companies that developed the same system. Journal of
Software Engineering, 35, 3, pp407-429
Ahonen, J.J., and Savolainen, P. (2010). Software engineering projects may fail before they are
started: post-mortem analysis of five canceled projects. Journal of System Software, 83, 11,
pp2175–2187
Atkinson, R. (1999). Project management: cost, time, and quality, two best guesses and a
phenomenon, it’s time to accept other success criteria. International Journal of Project
Management, 17, 6, pp337-342
Ahadzie, D.K., Proverbs, D.G., and Olomolaiye, P.O. (2008). Critical success criteria for mass
house building projects in developing countries. International Journal of Project Management,
26, pp675–687
Belout, A., and Gauvreau, C., (2004). Factors influencing project success: the impact of human
resource management. International Journal of Project Management, 22, pp1-11
Chan, A.P.C., and Chan, A.P.L. (2004). Key performance indicators for measuring construction
success. Benchmarking: An International Journal, 11, 2, pp203-221
Clarke, A. (1999). Practical use of key success factors to improve the effectiveness of project
management. International Journal of Project Management, 17, 3, pp139-145
Creswell, J.W. ( 2003). Research Design; Qualitative, Quantitative, and Mixed Methods
Approaches. London: Sage
de Wit, A. (1988). Measurement of project success. International Journal of Project
Management, 6, 3, pp164-70
de Bakker, K., Boonstra, A., Wortmann, H., 2010. Does risk management contribute to IT
project success? A meta-analysis of empirical evidence. Int. J. Proj. Manage. 28 (5), 493–503
Doloi, H., 2009. Analysis of pre-qualification criteria in contractor selection and their impacts on
project success. Construction Management and Economics 27, 1245–1263.
Easterby-Smith, M., Thrope, R., and Jackson P.R.,(2008). Management Research. London:
Sage
12
Ittner, C. D. and Larcker, D. F. (1997). Product development cycle time and organizational
performance, Journal of Marketing Research, 34, pp13–23
Glass, R.L. (2001). Frequently forgotten fundamental facts about software engineering.
Information System Journal, 18, 3, pp110-118
Kappelman, L.A., McKeeman, R., and Zhang, L. (2006). Early warning signs of IT project
failure: the dominant dozen. Information System Management, 23, 4, pp31-36
Larson, E. W. & Gobeli, D. H. (1989). Significance of project management structure on
development success. Engineering Management, 36, pp119-125
Lyytinen, K., and Hirschheim, R. (1987). Information systems failures: A survey and
classification of the empirical literature. Oxford Surveys in Information Technology, 4, pp257309
Pinto, J.K., and Slevin, D.P. (1988). Project success: definitions and measurement techniques.
Project Management Journal, 19, 1, pp67-71
Procaccino, J.D., and Verner, J.M. (2006). Software project managers and project success: an
exploratory study. Journal of System Software, 79, 11, pp1541-1551
Shenhar, A.J., Dvir, D., Levy, O., Maltz, A.C., (2001). Project success: a multidimensional
strategic concept. Long Range Planning, 34, 6, pp699–725
Yu, A.G., Flett, P.D., and Bowers, J.A. (2005). Developing a value-centered proposal for
assessing project success. International Journal of Project Management, 23, pp428–436
Wateridge, J. (1995). IT projects a basis for success. International Journal of Project
Management, 13, 3, pp169-72
Wateridge, J. (1998). How can IS/IT projects be measured for success? International Journal of
Project Management, 16, 1, pp59-63
13
Appendices
Appendix (A)
Theme
Quotes
Measuring
Meeting
success
requirements
scope
customer Other indicators, actually we do not have but for me, we should focus
and on the design. If the design can respond to the need of customers and
give them satisfaction, I think that one can be the key indicator to
measure the quality of the project.
Understanding the requirement is very important for the success of
the project… So, if you ask for the cause for unsuccessful. I can say
that there was a mistake. When we got the requirement because the
end-user and the top management had a conflict that they cannot get
understand each other. Then, the requirement and the scope were
quite ambiguous. And the end-user was also not willing to change
their nature because they had to leave the manual process and adapt
to the new process that they did not want to get involved to this new
one. So, the customer was not willing to use it so this project was
quite unsuccessful (P.S).
“Amorepacific project, I think it is a successful one. It is a 3 years
project. For the project, it is clear to say whether it is successful or
not. If we meet the requirements, it is successful” (D.K)
“Maybe clients come first. We have to follow their requirements. So
they come first” (D.K)
All projects approval and establishment come with contracts that
contain particular
acceptance criteria. The acceptance criteria have two parts. First is
technical indicators such as production precision, stability, safety,
functionality, etc. Second is the indicator of industrialization. We
signed a long-term cooperation contract with some medical
equipment companies and hospitals.
All productions will carry out the acceptance tests by these
organizations. After that, we will gather market feedback to finally
evaluate if it is a success. The IGSNS I mentioned above was a
14
successful production as both two parts were well done (Z.L)
Customer
satisfaction
For the criteria that I use to determine. As I said, the satisfaction of
the customer is the key indicator to determine success. And I can say
that if the business process of the customer can be more
systematically and they can work efficiently, I think that it can be the
factor for the success (P.S)
The first sign, as I said, should be the feeling of the customer. If the
customers feel uncomfortable with our design, it may not friendly
enough for their nature. That means we should improve our design
again (P.S)
Time
If we don’t finish on time, there will be a penalty usually 1/1000 of
total project cost per day (D.K)”
“If you want to me choose one unsuccessful project. I would say that
Samsungs.com is the one. We finished 1 stage of 3 because of the
client’s strategy change. It didn’t fail, but it is hard to finish. We
made a wrong prediction on the scoping of the project. So it is hard
to finish. I worked 18 hours a day for 3 weeks. I have to live hotel
next to my office” (D.K)
At some points, the customer may require to change their
requirements during the project. So, we need to respond to it by
revising the contract, and sometimes it can lead to a delay of the
timeline and may hurt the budget as well (P.S).
The schedule is more important to us. It directly affects the following
process if failed to accomplish the task on time. The whole project
will then get in trouble (Z.L).
Cost
For IBM, we need to make a profit. For example, we planned to
make a 30% profit, but we cost more. Finally, we only make 5%
profit. We call it to fail (D.K)
Yes, the contract of fixed schedule will be based on the man-day
system to estimate the man-day that uses for each project. The manday system can help us to arrange the resource and to prevent the
15
over-budget issues as well. (P.S).
We usually get sufficient financial support from government funding
and
cooperative organizations, thus the budget effect to be small despite
there are indeed budget
limitation in every project (Z.L)
Business
For IBM, we need to make a profit. For example, we planned to
performance
make a 30% profit, but we cost more. Finally, we only make 5%
profit. We call it to fail
(DK)
Sources of Customer feedback
For my company, IT One company, we do not have particular
evaluation
indicators but we set a team as the customer service teams to support
the customers. Sometimes we can rank the quality of each project by
the number of incoming calls from the customers. If some projects
have too many requests from customers, we can say that project may
have a problem (P.S).
Yeah, customers. At the end of the project, customers will sign a
project conclusion document. We have a client satisfaction survey,
which we can check (D.K).
Yes. There are three groups in charge of acceptance tests during each
timeline. First is the intra-team acceptance test, including my boss
and all team members. Second, the acceptance test will be performed
by a qualified third-party assessment entity. They have more
marketing experience than us, thus we invite them to evaluate
projects in both quantitative and qualitative measurement. The third
is the end-user, in terms of my projects being managed and experts of
hospitals will be the final test. In one word, acceptance tests follow
up with the whole progress of development. (Z.L)
Internal evaluation
Apart from my boss, there is my team lead of the project which we
do the face-to-face evaluation to discuss all the skills we need to
focus on which involve for that project….. The criteria of my boss
and I are quite the same because IT One has the standard for
checking the progress of the project such as weekly meetings. We
16
also follow that standard (P.S)
For me, the feedback of top management is more significant. As a
state research institute, the marketing benefit may not be required for
some kind of project. Thus, compared with the original contract, the
internal feedback could reflect the indicator of performance more
objectively. (Z.L)
Market feedback
There will be an intra-team meeting after the project, we collect all
info to a report to the top management. The government as the big
shareholder will give their feedback as well. More importantly, the
final aim of designing and developing is to bring the research
findings into the market and to accept its test until it makes success
and achieves motion benefits. The feedback report from the market
research company plays an important role in judging the production
success, and the content of feedback will be used for further
improvement (Z.L)
17
18
Name:
Description:
…