|
First Posted: 1998 Last edited:
23 September 2005 |
THE EVALUATION OF
TECHNICAL ASSISTANCE AS
A CONTRIBUTION TO
DEVELOPMENT
by Cyril
Belshaw (c)
Originally
published in the International Development Review June
1966
Author's Note:
The context has of course changed immeasurably. Have the principles changed?
If not, are
they being observed? If so, what are the changes?
THE EVALUATION
OF TECHNICAL ASSISTANCE has been of concern to international
officials
for
over fifteen
years [1996]. During that time there have been some attempts, more or
less
sporadic,
to
develop
techniques of evaluation, but very few of the attempts have been coordinated
with
others,
and
one cannot
describe a continuous evolution of thought about the matter. The symposium
on
techniques
of
evaluation
which took place in 1955 under TAB auspices, summarized in the
International
Social Science
Bulletin, might have taken place today. One is struck by the fact that there
is
in existence
no bibliography of evaluation reports, and that most evaluation teams or
studies
bilateral
agencies sometimes lose track of what they have done before. Another
impression
contact, so
that learning and the transfer of experience between them is minimal, and
the
growth of
technique and conceptualization is slow.
This article
is not to make up these deficiencies, but to consider a point which has been
lost
sight of
because of the considerable duplications of conceptual effort. The nature
of
evaluation
depends entirely upon the purposes of the evaluation, and so far very
few
attempts have
been systematically concerned with the effects of technical assistance
upon
the
development of a country (a polity, a society, an economy) as a working system.
Concern
with this
overall approach to technical assistance has been growing in recent years, and
is
reflected in
the debates and resolutions of the Economic and Social Council which have
led
to the
appointment of a number of evaluation missions. Such missions, however, must
work
out their own
technique in the short space of time available to them, and must
therefore
build very
largely upon the past with the minimum of innovation. This paper grows out
of
the experience
of one such mission and attempts to reflect further upon the possibilities. It
is
of course a
personal paper, with which the other members of the mission, and the
United
Nations, are
not associated.
LIMITATIONS ON
PROJECT EVALUATION
Most
approaches to technical assistance evaluation are concerned with the question,
to what
extent did
the
project
achieve its goals in the most efficient manner? Such a question can often
be
answered with
a great deal of accuracy, which varies according to whether we can
assume
that the goals
are specifically defined, and whether the analyst can obtain sufficient
data
about results,
timing, and the resources used. When the approach is coupled with
an
examination of
possible technical alternatives, it can lead to an improvement in method,
and
often
increased results with fewer resources. Evaluations of this kind are necessary
to
improve the
technical capacity and economy of the agencies providing
assistance.
Yet to
describe this as an evaluation of the contribution of technical assistance
to
development
would be to give a false emphasis: the operation is simply an extension
of
research into
the techniques and methodology of agricul tural, health,
educational,
industrial, or
other operations, with special emphasis on problems which are of
frequent
occurrence in
technical assistance.
Even within
this conception there is a major limitation to the approach. Frequently the
goals
of technical
assistance projects are not specified with a sufficient degree of precision
to
make such an
assessment possible, and the methods used involve skills and
intangibles
which weaken
attempts at quantification. This is occasionally a matter of careless
project
formulation,
but very often it is deliberate, since the objective is to probe, to test, to
explore,
to stimulate,
and since, if the goal were pre-defined, it might dictate an apparent
solution
Evaluating
Technical Assistance in Development
which would
otherwise not be appropriate. This is true of open-ended
scientific
exploration, of most institutional change, and of projects which involve
the
formation of
ideas and values.
Linked to this
limitation is another, namely that important effects of technical
assistance,
both positive
and negative, are side-effects. These are not only unforeseen results in
closely
allied fields
(the use of a school textbook offending mores and creating antipathy to
the
school), but
ramifying results over many areas (a road creating a demand for
markets,
transportation,
drainage, organized water supply). To concentrate solely on the specific
goal
of the project
might mean the setting aside of extremely significant
conditions.
Another
consideration is the analysis of the goal itself. To accept it as given may be
to avoid
asking the
question, was it the most appropriate goal under the circumstances, or
would
some other
uses have been more effective?
And finally
sometimes, though not always, the evaluation does not distinguish between
the
technical
assistance component of a project, and the total project itself. Thus, for
example, it
is fairly easy
to state that a tuberculosis eradication campaign was associated with
a
reduction of
disease incidence which in turn was associated with a sharp drop in
mortality
attributable
to tuberculosis. But it is not always so easy to decide how much of this result
is
attributable
to the skill of government action itself, to the W.H.O. key assistance, to
the
actions of
U.S. aid financed teams, or to alterations in the habits of the population in
turn
linked with
improved housing, water supply, education, nutrition or marketing. Perhaps
the
same result
would have been achieved without international technical assistance. Perhaps
it
would have
been achieved more quickly if there had been more of it. Perhaps it was
the
crucial
catalyst making the whole com-
Because of
factors such as these, some approaches to evaluation are more open-ended
and
discursive
than the tight technical ones which are superficially precise. Thus
Professor
Charles Madge,
for example, attempted, on behalf of UNESCO, to tap the experience
and
knowledge of a
limited number of technical assistance experts who had been working
in
Thailand in
fields where the rural human relations component was highly significant in
the
work. The
experts were asked to keep diaries indicative of their experience,
including
interaction
with counterparts and with village people, to answer questions about
their
experience and
its results, and to add their own analysis of the impact and significance
of
the
project
in a
free-flowing manner. Similarly, Professor Herbert Hyman and his associates at
the
United Nations
Research Institute for Social Development, have made a detailed
statistical
and analytical
study of the views of over four hundred experts in ten countries,
again
working in
areas where rural human relations are of great significance. The most
immediate
value of such
studies is to cast considerable light upon the attitudes of technical
assistance
experts who
are, of course, key elements in the chains of interaction which ultimately
bring
about a
technical assistance result. The views of Resident Representatives, as surveyed
by
Evaluating
Technical Assistance in Development
]
In a limited
way the experts are also used to reveal the processes at work during the
course
of their
project,
and since they
are intelligent commentators and observers they can record many events
and
interactions
which are extremely instructive, and can add an element of depth and
humanity
to otherwise
mechanical studies. Nevertheless, they cannot do the whole job of
analysis.
What one
observes depends on what questions one is interested in. Field experts are
very
seldom
economists or sociologists, and if they make observations of interest to
economists
or
sociologists, it is quite often because they have absorbed appropriate knowledge
as
intelligent
men and women stimulated by an overseas environment that has intrigued
them.
In the same
way, anthropologists made use of missionaries and other field observers
in
the early days
of the discipline, and now contact other anthropological colleagues
for
information of
a comparative kind. But anthropologists interested in generalizations
are
continuously
frustrated because their colleagues have not recorded information which
is
essential to
the new theory, and in many instances were not even aware of its
significance
when they were
in the field. The dimension added by the knowledge of the field
experts
must
eventually be supplemented by more systematic value-free
observation
which can go beyond the theories at present currently circulating among
the
professional
fraternity.
OVERALL
CONTRIBUTION OF TECHNICAL AID
These types of
investigation are leading into much broader fields than those indicated
by
the problem
as
to whether a
project is being handled in the most efficient or effective way possible.
They
are leading in
fact towards the question, what is the contribution of technical assistance
to
the overall
socio-economic development of a given country? At first sight, it would
seem
that this
question, being even broader, is less capable of being
answered.
To some extent
this is true, particularly if one expects quantitative precision in
one's
answers. It
would be
feasible to
select particular types of technical assistance where relations between inputs
and
outputs are
direct, and measure impact on the economy at least. But most
technical
assistance,
and perhaps the most significant, is not of this order at all. How would
one
translate the
effect of an adviser on national planning in such terms? How would one
put
together the
effects of a fellowship concerned with the administration of social welfare
and
the advice of
an expert who suggested that the time was not ripe for a national
standards
laboratory? To
add up, for example, the monetary value (in terms of salary and other
costs)
of the
fellowship and the expert project would not give a comparative indication
of
significance,
and would not take into account the ramifying effects of decisions which
flow
Evaluating
Technical Assistance in Development
from the two
enterprises.
One is
therefore forced back into a process of analysis which endeavors to place
the
technical
assistance project in an organic framework, and which endeavors to ascertain
what
the
implications of the projects are for the economic and social environment in
which it is
placed. One
must analyze what happens, and what the linkages of events might be.
This
implies the
selection of information according to judgments about its significance
for
organic
relationships rather than for its significance for quantitative measurement.
(The two
concepts are
not necessarily opposed: it is a question of starting point.
One method
starts with observations which are measurable now, and builds a pattern
of
significance
around them; the other starts with patterns of significance, and ultimately
tries
to measure
what one can within the pattern.)
Thus in the
analysis judgment is all important. Judgment in such a field is
systematized
around
propositions, explicit or not, about the crucial operation of significant
relationships.
There is in
fact a theory of the operation of a society-economy, and a theory about the way
in
which certain
factors and relationships operate to bring about development.
Technical
assistance is
judged according to the manner in which it fits, or contributes to, the
operation
of the assumed
system.
"Over-all
impact" evaluation teams do not normally have the time to elaborate on this
matter
systematically
and consciously, or to link their observations in more than a passing
manner
to current
developmental theories, of which the number is legion. Where
immediate
knowledge is
required, this is all to the good, since to do otherwise would be to carry out
a
prior
theoretical exer cise without the certainty of agreement, and to lose some of
the
exploratory
significance of the investigations.
But inherent
assumptions are there .
But as time
goes on there will be an increased need for greater systematization in
the
observations
to allow
for
comparability, both temporally and cross-culturally, and to enable the validity
of the
implied
propositions to be tested by further experience. The explicit linkage of
over-all
impact
evaluation to theories of development will therefore be an advantage. This paper
is
an endeavor to
push deliberate thinking on this topic a little
further.
A POSSIBLE
APPROACH
One
interesting suggestion, unfortunately in a paper not released for publication,
has been
to reduce the
whole of the development process to one basic factor, namely the increase
in
Evaluating
Technical Assistance in Development
skills. This
dramatic analytical step, cutting through the confusing intricacies of social
and
economic
theory to look for a prime mover which is not only empirically sound
but
eminently
subject to policy influence, has a great deal of appeal. Increase in skills
brings
together under
one heading processes which potentially reflect upon motivation,
technical
capacity,
efficiency, innovative power, and organizing ability. It is a factor linked not
merely
with economic
growth. in the sense of a (possibly temporary or fortuitous) alteration
in
output, but
more fundamentally with development in an organic sense, namely an
alteration
in the
organized characteristics of a culture, society, economy and polity upon
which
enduring and
self-sustained growth is based. One cannot envisage such
development
without an
alteration in skills.
This of course
does not end the matter. There is not just one indicator of skills, there
are
many.
Studies
must range
over such matters as technological skill, entrepreneurial ability, and
the
wide-ranging
questionanswering type of skill usually associated with research or
university
education.
They must inquire into the degree to which technical assistance has
contributed
to the growth
of such skills, and pay allention to the problem as to whether the spectrum
of
skills is
appropriate to the circumstances of the country at the time. There is no doubt
that
we have here a
basic factor in social development.
However, no
single criterion is likely to be good enough. One of the difficulties is that
an
increase in
skills is not an independent variable, and that it may be supported
or
counter-acted
by other variables, which ought therefore to be taken into account. It is
not
difficult to
think of instances in which a dramatic alteration in the availability of
physical
resources
called forth, to some degree, an increased supply of skills (mining linked with
the
supply of
engineers, roads with the development of marketing skills). On the other
hand,
examples of
the over supply of skills are legion, and there are countries where scientists
and
engineers are
available, but are not being used to capacity because of lack of
appropriate
forms
of
organization
or institutions. A single criterion which hides such factors will reduce
the
significance
of the
evaluation as
a diagnostic tool.
SOME
RECOMMENDED CRITERIA
I wish now to
set out a series of propositions and questions which in my view are
closely
linked with
the development process. Since there is no clear agreement among scholars
and
administrators
as to the theory, or the nature, of development, or as to the weight to
be
attached to
such factors as are considered to be significant, it is most unlikely that this
series
will be
generally accepted. Nevertheless, if it has merit as an approximation it will
lend
support to the
idea that evaluation exercises should endeavor to make explicit
the
Evaluating
Technical Assistance in Development
development
theorems on which they are based, and to relate these to the specific
technical
assistance
programs they are examining. It should be stressed that there is no
implication
that projects
should, to be effective, contribute in terms of all propositions, or that they
are
better or more
effective if they contribute to the first rather than the second. At this
stage,
evaluation of
the impact of technical assistance upon development is not to be
achieved
through
arithmetical summation but through qualitative analysis.
(1) A technical assistance project contributes to
development if the program of which it is
a
part
perma-
nently
alters the effective demand schedule, or consumption pattern, of the country in
such a
way that
an
increased
level of satisfaction is achieved, and the gap between effective demand and
the
pre-existing
potential demand is narrowed.
[Anthropologists note: does the change in
culture
increase
satisfaction or not?-1998] This is the most difficult proposition to
formulate
succinctly,
and also by far the most difficult to assess. Note first, the phrase is
"pre-existing"
potential
demand, because it is very often the case that
the
satisfaction of one series of wants at one level opens up a whole range of
other
unsatisfied
ambitions, or brings them into consciousness in such a way that they
can
constitute
goals for further action (see proposition two). Note secondly, no attempt has
been
made to
distinguish between economic, cultural or social wants, or to state which
have
prepotence or
greater importance. The distinction is unsatisfactory analytically, and in
any
case, within
the context of this proposition, the choice is for the people to make
themselves.
The difficulty
of assessment lies in the identification of the effective demand schedule,
and
even more in
judging potential demand. Work being done at the United Nations
Research
Institute for
Social Development on levels of living may contribute to the solution of
the
first problem,
and the second may be solved approximately or temporarily by adopting
the
convention
that the government's assessment of plans and aspirations is the one to be
used.
We cannot,
however, rest content with the mere acceptance of government
priorities,
particularly
since some governments may be out of touch with the aspirations of its
citizens,
or may not be
able to analyze the implications. It must be admitted that most
evaluation
teams will of
necessity judiciously and somewhat subjectively combine information
gleaned
from
government
plans and
statements with indications from market behavior, statements of opinion,
and
social and
economic analyses. [Anthropologists note: an ethnography is a rough estimate
of
effective
demand in all sectors: a study of the way in which people valuate potential
changes
contribvutes
to the second sechedule - 1998]
(2) A
technical assistance project contributes to development if the program of which
it is a
part
increases the
satisfaction of
wants in such a way that other unsatisfied wants, some of which may be
new,
alter their
position in the potential demand schedule to such an extent that they become
goals
of further
action. In other words, an increase in consumption (material or immaterial)
spurs
people to try
to obtain more, constituting a dynamic force for change. This by no
means
Evaluating
Technical Assistance in Development
always
happens, particularly in marginal communities, but unless it happens
development
associated
with self-sustained growth will not be present. It must be admitted that
neither
economic
theory nor empirical analysis have yet given adequate guide-lines for
the
assessment of
the multiplying or ramifying effects of particular forms of
consumption.
Nevertheless,
judgments about this are implicit in the evaluation of the impact of
technical
assistance.
(3) A
project will contribute to development if it assists a program to increase the
range of
indigenous
resources
utilized or increase the range of commodities produced (provided this is
justified
economically) or
to remove bottlenecks in the system of resource exploitation and
production,
thus
liberating further productive forces.
(4) A
similar result will occur, other things being equal, when technical assistance
contributes
to an
in-
creased
division of labor (social concomitant: increased diversification of social
role),
provided
that this con-
tributes
optimally to production or to the direct satisfaction of wants. It should be
stressed
here once
again that the wants will be material, cultural or in the nature of social
welfare
satisfactions,
and that a priori judgments of appropriateness or imbalance should, in
this
context, be
eschewed.
(5) A further
proposition linked with (4) is that development reflects innovation, and that
the
rate of
in-
novation,
other things being equal, is associated with the size of the pool of relevant
ideas and
information
on the one hand, and the ability of the potential innovators to question,
observe,
generalize
and apply knowledge. Therefore one should ask, does the form in which
technical
assistance is
given add to the pool of ideas permanently available to the society (the visit
of
an expert who
took his knowledge away with him would not qualify)? And does it
equip
personnel, not
merely to apply knowledge statically, but to develop it in the
circumstances
of the
country? li should be noted here that innovation is not
merely
technical in a physical sense, but also implies alterations in modes of
organization.
(6) Just as an
increase in the velocity of circulation of money may be deemed to increase
the
quantity
of
money, so
an increase in the velocity of circulation of ideas and information may he
deemed to
increase
the
effective
size of the pool of ideas. Thus communications are of vital significance
in
development.
But communications have a further effect, namely that of assisting
individuals
to adapt to
one another. They are a vital element in the articulation of a society, a
culture, a
polity, an
economy. Does technical assistance contribute to increased effectiveness
of
communications?
Here the evaluation must look for several types of indicators. The use
of
two-way
transmitter/receivers, the improvement of the stock exchange (or its
foundation),
the state of
the commodity market, the use of telephones and the mails, the extension
of
reliable
freight services, the use of computers, may be areas involving technical
assistance
Evaluating
Technical Assistance in Development
which have
profound social and economic consequences. The problem for assessment
may
link equally
with boundaries in social relations (for all communication flows along
lines
indicated by
social structure and social organization) or increasing the effectiveness
of
symbol systems
(the basis of the means of com-
munication),
whether these be conventions such as legal contracts, or language
elements
such as
literacy, mathematics, or weight symbols. What does technical assistance do,
both
directly and
indirectly, in these connections ?
(7) In most
countries to which technical assistance is given, the spectrum of
organized
institutions
is either
not as
complete as in developed countries, or does not constitute an articulated whole
in
such a way
that the society operates as a set of inter-acting organizations.
Technical
assistance
will make a contribution towards the establishment of a modern
articulated
society if it
meets some of the earlier criteria. But in addition, as an aspect of the
division of
labor, and as
an extension of the fourth proposition, it will make a contribution if it
assists
in the
creation of specific institutions. The functional emphasis of these is likely to
be the
following,
although functions need not be limited in this way:
(a) the
creation of institutions which produce skills and knowledge, (b) the
organization of
production
and services (including cultural and welfare services), (c) the organization of
units
of public
administration, (d) the organization of institutions to remedy societal ills
which are
frictional
to the operation of the system.
It should be
noted that the emphasis is upon the training of trainers", to use the
current
jargon,
and
the creation
of institutions. If a technical expert simply produces a new textbook, teaches
a
number of
children, succeeds himself in removing malaria from a village, cures a number
of
opium
addicts, secures the production of x quantity of a raw material, his impact is
limited
to an effect,
probably temporary, on growth rather than on development, unless in
addition
his action
meets the requirements of propositions one and two, Such wants or ills cannot
be
dealt with
permanently unless permanent institutions are available to assess and
act.
Technical
assistance thus tends to be geared to institution building, and rightly so, for
this
increases the
complexity of effective social organization, and the capacity of a society to
take
its own
responsibility for increasing its output, and its performance from the point of
view
of satisfying
wants.
(8)
Organizations themselves are small socio-economic systems usually capable of
improved
performance.
If the
operation of organizations is improved so that their contribution to the
over-all social
system
improves, they will be changed and developed and will be part of a development
of the
social
system. Technical assistance is likely to make a contribution in several
fields, which
would include
increasing the efficiency of operation, an improvement in
adjustment
mechanisms
(both internal and as an aspect of adjustment to external conditions and
to
other
institutions), and the creation of an orientation toward growth and
expansion.
Evaluating
Technical Assistance in Development
(9) Finally
(but this is implied throughout) technical assistance will make a
greater
contribution to
develop-
ment the
more it results in the internalization of the above factors, so that they are
not
dependent
on external artficial stimulus.
Any technical
assistance project can be linked with the above propositions, and
analyzed
according to
which propositions (if any) it fits. Presumably, if it does not fit any of
the
propositions
it does not qualify as technical assistance which contributes towards
over-all
development.
There are indeed technical assistance projects which can be ruled out on
these
grounds, or
on the grounds that their contribution is minimal. But beyond these
extreme
cases, the
propositions as they stand do not in themselves give a clear indication of
the
relative
merits of alternative technical assistance projects or proposals. I do not
believe that
any firm
agreement on the technique for achieving this next step is at this stage
probable;
indeed, at
this stage it may be undesirable, since in the present state of our knowledge
a
further
period of trial and error, of experiment, and of observation, is probably to
be
preferred
over a too hasty commitment to one line of
thought.
A STRATEGY OF
DEVELOPMENT
Insofar as
there are principles to be discerned or worked out, they will probably
crystallize
around
the
notion of a
strategy of development. This is simply a short way of saying that projects
vary
according to
the
weight of
their impact. Conceptually, such variation is in the degree to which projects
have
multiplying
or
ramifying
effects throughout a socioeconomic system, and it may be argued that the
greater the
multiplying or
ramifying effects the more strategic the project. This refers to the
objective of
the project,
and its indirect
consequences.
In addition, technical assistance may not be concerned with a total
project,
but only part
of it, and here again the possible alternatives may be judged according to
the
proliferation
of their effects.
Unfortunately,
this criterion, necessary though it is, is not only a matter of evaluative
and
analytic
judgment, but is also predicated to some extent on the existence, of a
working
socioeconomic
system which has a high degree of internal articulation. A project may
have
highly
ramifying effects in one country, because the effects travel, as it were,
through the
social and
communicative links of the system, and effects take place as various
institutions
adapt to the
new conditions. But in another country the same measures may have little or
no
effect beyond
the immediate implementation of a target, because communication does
not
exist or
because institutions in the linkage of social interaction are missing, or
because they
are
non-adaptive. This suggests that a strategy of development may need to place
greater or
prior
emphasis upon institution building and the communications system in some
countries,
Evaluating
Technical Assistance in Development
or for some
sectors.
In any event,
the contribution of technical assistance to over-all development cannot
be
judged
finally without some fairly specific assumptions about the strategy of
development
in the
circumstances of the country.
Evaluating
Technical Assistance in Development
|
