Computer performance

                                     Computer performance

·        High throughout (rate of processing work)

·        This is the amount of work accomplished by a computer system

·        Depending on the context, high computer performance may involve one or more of the ffg

·        Short response time for a given piece of work

·        Low utilization of computing resources

·        High availability of the computing system or application

·        Fact (or highly compact) data compression and decompression

·        High bandwidth

·        Short data transmission time

·        The performance of any cpt system can be evaluated in measurable, technical terms, using one / more of the metrics listed above

·        In a non-technical term, cpt performance simply means how well is the cpt doing the work it is supposed to do, according to arnoid Allen  

Computer performance metrics

Cpt performance metrics (things to measure) include:

1.     Availability: availability of a system is typically measured as a factor of its reliability, increase so does availability

2.     Response time: is the total amount of time it takes to response to a request for service (any unit of work ranging from a simple disk I/O to loading a complex web page

Response time is the sum of the three times below:

Service time – how long it takes to do the work requested

Wait time- how long the req

uest has to wait for request queued ahead of it before it gets to run

Transmission time: how long it takes to move the request to the cpt doing the work and the respond back to the requester

3.     Reliability- the probability that a piece of equipment or component will perform its intended satisfactorily for a prescribed time and under stipulated environmental conditions reliability is often quantified by MTTF- mean time to failure

4.     Channel capacity- is the tightest upper bound on the rate of info, that can be reliably transmitted over a communication channel

5.     Throughput- the rate of production or the rate at which can be processed

6.     Bandwidth-  in cpt networking, bandwidth is a measurement of bit- range of available or consumed data communication resources, expressed in bits’ parsec or multiples of it (bit, kbit, mbit, gbit, etc)

7.     Latency: is the time delayed between the cause and the effects of some physical change in the system being observed

It is a result of the limited velocity with it any physical interaction can take place  

8.     Scalability- the ability of a system, network or process to handle a growing amount of work in a capable accommodate that growth

9.     Power consumption- the amount of electricity used by the cpt. This becomes especially important for system with limited power sources solar, batteries, human power

10.                        Compression ratio- compression helps to reduce resources usage data storage space or transmission capacity

Compressed data must be decompressed to use hence it is subject to a space-time complexity trade off

 

 

Failure- a failure is the non-performance or instability of the system or component to perform its intended for a specified time under specified environment conditions. A failure is a behavior or an event

Error- an error is a designed flaw or deviation from a desired or intended state, an error might lead to a failure unless something constructive is done. A failure in turn might lead to a new erroneous state

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Fault- a fault is the adulated cause for an error generally a failure is a fault but not vice versa, since a fault does not necessarily leads to a failure

Primary fault (and failure)- is caused by an error in the software so that the cpt output does not means its specifications

Secondary fault (and failure)- occurs when the input to a cpt does not comply with the specifications. This can happen when the cpt and the software is used in an environment not designed for

Command fault- occurs when the cpt delivers the correct result but at the wrong time or in the wrong order. This is typically a real time fault

Performance environment- tells what a pgm did and how it did it

Evaluation-tells the program effect on the people families or communities it is serving, ie whether the pgm is producing results or having an impact

What is impact?

-refers to the net effect of a pgm relatives to what would have happened had the pgm not existed.

-changes in the outcomes produced by the pgm alone and not caused by other factors.

-typically reflects direct outcomes of a pgm ie persistent changes in participant‘s situation or behavior

-impact can be either intented [ie those meant to be caused by the program] or unintended [ie those incidental to the pgm’s original goals or objectives].

 

Performance measurement

-involves collecting and reporting data that can be used to summarize and access the way a pgm is being implemented

-performance measurement data to best improve pgm performance are collected and accessed while participants are being served rather than only after a pgm cycle has concluded

-questions that performance measurement data can answer include:

a. inputs

-what staff/volunteer are involved in the pgm?

-what is the pgm budget?

-what equipment does the pgm have?

-how were all the inputs used?

b. pgm participation

-who is participating [ie. participant xhcs]?

-how many participants are there?

-how much did each person participate (esp. relatives to a desired level of service)

c. outputs

- what services are delivered?

- who delivered the service?

-how well are services being delivered

d. outcomes

- what changes do we observe in participants?

Computer system performance measures?

1.     Responsiveness- these measure are intended to evaluate how quickly a given task can be accomplished by the system. Possible measures waiting time, queue length etc

2.     Usage level- these measures are intended to evaluate how well the various component of the system are being used. Possible measures are throughput and utilization of various resources

3.     Missionability- these measures indicate if the system would remain continuously operational for the duration of a particular period of time. Possibly missionability measures) the distribution of the work accomplished during the mission time interval availability (probability that the system will keep performing satisfactorily throughout the mission time and the life-time (time when the probability of unacceptable behavior increases beyond some threshold)

4.     Dependability- these measures indicate how reliable the system is over the long run. Possible measures are the number of failure per day, MTTF (mean time to failure), MTTR (mean time to repair), long-term availability, and the cost of failure.

5.     Productivity- these measures indicate how effectively a user can get his or her work accomplished. Possible measures are user-friendliness, maintainability and understandability.

Application domains

1.     General purpose computing

These systems are designed for general purpose problem solving. Relevant measures are responsiveness, usage level, and productivity dependability requirements are modest, especially for bearing failures (mild non threating, not malignant)

2.     High availability-

Such system is designed for transactions processing environment (banks, airlines, or telephone databases, switching system etc) the most important measures are responsiveness and dependability. Both of these requirements are mire server (separate, disunited) than for general purpose computing system.

Productivity is also an important factor

3.     Real time control

Such system must respond to both periodic and randomly occurring events within some (possibly hard) timing constraints. They require highly level of responsiveness and dependability for most workloads and failure and are therefore significantly over designed. Note that the utilization and throughput pay little role in such.

4.     Mission oriented

These systems require high level of reliability over a short period, called the mission time. Little or no repair and tuning is possible during the mission. Such system includes fly-by-wire airplanes, battle field systems and spacecraft’s. responsiveness is also Important but usually not difficult to achieve. Such systems may try to achieve high reliability during short term at the expense of poor reliability beyond mission period.

5.     Long life

Systems like the ones used in unnamed spaceships need long life without provision for manual diagnostics and repairs. Thus, in addition to being highly dependable, they should have considerable intelligence built into the diagnostics and repair either automatically or by remote control from a ground staton. Responsiveness is important but not difficult to achieve.

 

EVALUATION

“When the cook tastes the soup, that’s formative, when the guest tastes the soup, that’s summative”

Robert Stakes

         Through effective performance measurement, pgm staff can collect valuable data however, cannot directly answer questions about pgm impact not can performance measurement data necessity answer all question about pgm impact not can performance measurement data necessity answer all questions about how a system pgm is working or how results were achieved, to answer these questions evaluation method is needed

   Evaluation covers a variety of activities that provides evidence about what a system pgm did and whether (or how well) it achieved its aims. While these evaluations, if well done, can provide valuable information about a pgm system did and whether (or how well) it achieved its aims. While these evaluation methods, if done well, can provide valuable information about a pgm/system, not all measure provide generalizable conclusions about whether a pgm/system was effective.    

 

                              TYPES OF EVALUTION

a)    Formative Evaluative

_ planning study

_ process study

B) Summative Evaluative

_ Experimental study

_ Comparison study

 

 

Evaluation results depends on evaluation’s internal validity [how accurately an evaluation measures a pgm’s impact on outcomes for the population it is serving] and evaluation’s external validity [the extent to which  conclusions about pgm impact can be reliably generalized to other populations, pgms, geographies, or time spans than those studied].

 

Formative Evaluative

  The purpose of FE is to learn how a pgm is being designed or carried out, with the objective of providing information that can be used to improve implementation and results.

a)    Planning study: this is a this is a type of formative evaluation which takes place during the design or planning phase to clarify a pgm’s plan and to make improvements at an early stage. These are some of the questions that can be addressed by a planning study.

-         What are the goals and objectives of the system pgm

-         What population is the pgm and system intended to serve?

-         Is the pgm intervention appropriate for the identified goals and population?

-         What impact is the pgm expected to have?

-         Are the available resources (staff, facilities, equipment, findings) adequate to accomplish the pgms and system goals and objectives?

-         Is the pgm implementation timeline achievable?

 

b)    Process study: formative evaluation can also be undertaken throughout implementation as an implementation or process studying. A process study is important for pgms that are still developing, so that changes during the implementation phase may be clearly documented. A process study answer questions such as the following about the quality of pgm implementation

-         What intervention was implemented?

-         Did services get delivered as intended? Were the appropriate populations reached by the pgm? If not? Why not?

-         Were the resources (staff, facilities, equipment, funding) sufficient to accomplish the pgm goals and objectives?

-         Did staff encounter problem in setting up or running the pgm? Were they able to respond to and address all challenges?

-         A process study can help a nonprofit understand why a pgm understand why a pgm is getting or not getting particular results and provide information leadership and staff can use to change the pgm as its being Implemented

-         Formative evaluation uses a variety of research method to achieve their objectives such as performance management data, surveys, direct observation of client or staff interviews or focus group client and staff or other means, to supplement routine performance management data.

-         Data gathered can be analyzed by the organization using summary table or chat

Summative evaluation

-         Looks retrospectively at what a program accomplishes. a program has changed the condition described in the theory of change and logic model, specifically to determine the programs impact on those conditions

-         Summative evaluation can help funders determine whether additional resources to serve more people would be a good investment

-         On the other hand, nonprofit or funders might decide that a program failing to show positive impact should be redesigned or phased out

-         2 types of summative evaluations intended impact include:

a)    Experimental study (randomized control trial)

-         The most reliable ways to determine a pgm impact

-         Here, the pgm participants (the treatment group) and the nonparticipants (the control group) are selected at random from the same population

-         This eliminates the risk of self-selection bias (the possibility that people who choose to participate will differ in some way from those who do not)

-         If properly done, Experimental stud have a high degree of internal validity in measuring program impact.

b)    Comparison study (quasi-experimental study)

-         Here, changes in outcome from program participant (the treatment group) will be compared to changes for a comparison group that resembles the program participants group as much as possible

-         In both experimental and comparison study, a valid Summative evaluation can help determine a programs effect on the people, families, or communities it serves and its result or impact.

-         In other words, a Summative evaluation answers a fairly straight forward questions. Did the program work and if so, so how well?

-         The following criteria can be used to access a programs summative evaluation readiness:

a)    Relevance and importance of program impact question: a summative evaluation fundamentally addressed the question of program impact. Before proceeding, the nonprofit and its funder should establish that answering this question is relevant to the program and likely to be of high value to the community being served

b)    Applicability, scalability, and replicability: since summative evaluations have high costs, funding should be prioritized towards evaluating a program that would be wildly relevant to the field or applicable to other sites. Ideally, the program should be scalable and replicable

c)    Organizational capacity and scale: the program should be well implemented the evaluations. Further, the program should gave sufficient participant, or be capable of grouping large enough to support an evaluation

d)    Program stability:  should record stable service delivery for multiple year and should help evaluators access discrete changes in program design

e)    Adequate funding for program operations and evaluations

f)      Buy in and research planning participation from staff: both evaluators and nonprofit staff should see the evaluation as a partnership, have an equal commitment to the study fidelity and success, and be willing to listen and to contribute to making it work

 

 

DIFFERENCE BETWEEN PERFORMANCE MEASUREMENT AND EVALUATION

Though the dividing line between performance measurement and evaluation (formative) can be blurry, a few things can generally distinguish them

-         performance measurement is ongoing while evaluation is discrete.

performance measurement is part of a continuous improvement process, in it data are collected, analyzed and reported as close to real time as possible, giving staff immediate and actionable feedback on a programs status. Evaluation is not done continuously but rather during particular periods of a programs development or implementation and covers a specified time frame. For example, a formative evaluation may be carried out during the first 6 month of the programs planning and implementation while a summative evaluation might be done during the fifth year of an established program

-         performance measurement is responsive and adaptive, evaluation answers a predetermined set of questions

-         while performance measurement exploits program and outcome data, it can also be used in evaluation, evaluation usually involves other data collection and research methods

-          performance measurement is mostly done by program staff, whereas evaluation is typically carried out by people outside the program.

 

 

TECHNIQUE FOR PERFORMANCE MEASUREMENT

1.     Measurement: this is the most fundamental technique and is needed even in analysis and simulation to calibrate the models. Some measurements are best done in hardware, software or in hybrid manner

2.     Simulation modeling: this involves constructing a model for the behavior of the system and driving it with an appropriate abstraction of the work load. The major advantage of Simulation is it generality and flexibility, almost any behavior can be easily simulated

-         Both measurement and simulation involve careful experiment design, data gathering and data analysis only characterize the system behavior for the range of input parameter covered. Although exploration can be used to obtain result for the nearby parameters values, it is not possible to ask “what if” questions for ability values

3.     Analytic modeling: this involves constructing a mathematical model of the system behavior (at the desired level of details) and solving it. The main difficulty here is that the domain of tractable models is rather limited. Thus, analytic modeling will fail if the objective is to study the behavior in great details. However, for an overall behavior characterization, analytic modelling is an excellent tool

The major advantage of analytic modelling over measurement and simulation modelling are:

(a) It generates good insight into the working of the system that is valueable even if the model is too difficult to solve

(b) Simple analytic models can usually be solved easily , yet provide surprisingly accurate results

(c) Results from analysis have better predictive value than those obtained from measurement or simulation

4.     Hybrid modeling: a complex model may consist of several sub-model, each representing certain aspect of the system. Only some of these sub models may be analytically tractable the other may be simulated

 

APPLICATIONS OF PERFORMANCE EVALUATIONS

1.     System design: in designing a new system, one typically starts out with certain performance and reliability objectives and a basic system architecture, and then decides how to choose various parameters to achieve the objectives this involves constructing a model of the system behavior at the appropriate level of details, and evaluating it to choose the parameter. At higher levels of design, simple analytic reasoning may be adequate to eliminate bad choice, but simulation becomes an indispensable tool for making detailed design decisions and avoiding costly mistakes

2.     System selection: here the problem is to select the best system for among a group of system that are under consideration for reasons of cost, availability, compatibility etc

     Although direct measurement is the ideal technique to use here, there might be practical difficulty in doing so (e.g. not being able to use them under relish workloads, or not having the system available locally or not having the system evitable locally). Therefore, it may be necessary to make projections based on available data and some simple modeling

 

3.     System upgrade: this involve replacing the entire system or part thereof with a newer but compatible unit. The compatibility and cost consideration may dictate the vendor, so the only remaining problem is to choose quantity, speed and the like

       Often, analytic modeling is adequate here. However, in large system involving complex interactions subsystems, simulation modeling maybe essential. Note that a direct experimentation would require the new unit first, and thus is not practical

4.     System tuning: the purpose of tune up is to optimize the performance by appropriately changing the various resource management policies it is necessary to decide which parameters to consider changing and how to change them to get maximum potential benefits. Direct experimentation is the simplest technique to use here, but may not be fusible in a production tot environment. Since the tuning often involves changing aspect and cannot be easily represented in analytic models, simulation id indispensable in this application

5.     System analysis: suppose that we find a system to be unacceptably sluggish. The reason could be either inadequate hard ware resource or poor system management. In the former case we need system upgrade and in the later a system tune-up. Nevertheless, the first task is to determine which of the two cases applies. This involve monitoring the system and examining the behavior of various resource management policies unde different loading conditions

       Experimentation with simple analytic reasoning is usually adequate to identify the trouble spot, however, in some cases, complex interactions may make a simulation studt essential

System workload

This refers to a set of input generated by the environment in which the system is used, example, the inter-arrival times and service demands of incoming jobs and are usually not under control of the system designer and administrator. These inputs can be used for driving real system (as in measurement) or its simulation model, and for determining distributions for analytic and distribution modeling

Workload characterization

This is one of the central issues in performance evaluation because it is not always clear what aspect of the workload are important, in how much details the workload should be recorded, and how the workload should be represented and used.

     Workload characterization only builds a model of the real workload maybe captured or is relevant

    A workload model maybe executable or non-executable. For example, recording the arrival instant and service duration of the jobs creates an executable model, whereas only determining the distribution creates a non-executable model

-         An executable model need not be a record of inputs; it can also be a program that generates the inputs

-         Executable workloads are useful in direct measurement and trace driven simulations whereas non-executable workloads are useful for analytic modelling and and distribution driven simulations

The art of performance evaluations

-         Successful evaluations cannot be produced mechanically

-         Evaluation requires detailed knowledge of the system to be modeled

-         Careful selection of methodology, workload, and tools

-         Conversion from an abstract feeling or notion to a real problem which needs to be formalized in a way that can be handled by established tools

-         Analysts tend to have different styles

THE ART OF PERFORMANCE EVALUATION

-         Successful evaluation cannot be produced mechanically

-         Evaluation requires detailed knowledge of the system to be modeled

-         Careful selection of methodology, workload and tools

-         Conversion from an abstract feeling or notion to a real problem which needs to be formalized in a way that can be handled with established tools

-         Analysts tends to have different styles

SYSTEMATIC PERFORMANCE EVALUATION

Ten steps

1.     State goals of the study and define the system

-         Identical hard ware and software yet, the system may vary depending on goals

-         The chosen system boundaries affect the performance metrics as well as the workload used to compare the systems

-         Additionally, administrative control of the sponsors of the study. Sponsors May want to keep uncontrollable components out of the system boundaries

2.     List service and outcome

Network: send packets to a specified destination

Database: respond to queries

-         Also list the possible outcomes, example database query: correctly, incorrectly, not at all

3.     Select metrics

-         Criteria to measure the performance: usually speed, accuracy, and availability

-         Network: throughput, delay(speed), error rate (accuracy)

-         CPU: time to execute various instructions (speed)

4.      List parameters that affects performance

-         System parameters (both hardware and software) workload parameters (characteristics of user’s request) the list of parameters may not be added, always keep list as comprehensive as possible

5.     Select factors to study

-         Factors: parameters that varied during the evaluation

-         Levels: values of a factor

-         Limited resources: start with a short list and extend if the resources permit

-         Choose parameters expected to have high impact as factors

-         Also consider economic, political, technological constrains and decision maker

6.     Select technique for evaluation

-         Analytical modelling, simulation, measuring a real system

-         Depends on time, resources and the desired level of details

7.     Select workload

-         List of service request to the system

-         Depends on the evaluation technique: probability of various requests (analytical), trace of request from real system (simulation), user script (measurement)

-         Representative workload often require to measure and characterize the workload on existing systems

8.     Design experiment: maximize information with minimum efforts

Two phase:

1^st: many factors, only few levels… determine relative effect on factors

2^nd: few most significant factors, increase the numbers of levels

9.     Analyze and interpret data:

-         Consider the variability of simulation and measurement results. Use statistics

-         Interpretation is the key part of the analyst: analyst produces results but no conclusion or decision

-         Analyst conclusion may be given the same set of results

10.                                                                      Present results:

-         Communicate the result to the other member of the decision making team

-         Information needs to be easily understood

-         No statistical jargon

-         Choose graphic form with proper scaling of graphs

-         At this point: reconsider and question some of the decision made in the previous steps (example. System boundaries, factors, or metrics)   

-         The complete evaluation project consists of several cycles rather than a single sequential pass