ED814 – Evaluating the Effectiveness of the Educational Process -- Discussions
Writing up a Project Evaluation*

The following headings and subheadings collectively provide a possible structure for an evaluation report.  The detail of what is included in each section may vary but you should ensure that the main points are all covered.  

Title Page

(i)	Title of the service or project & its location
(ii)	Any additional headings or subheadings, e.g. the title of the study itself or the main research design used to evaluate
(iii)	Period covered by the report
(iv)	Date of submission
(v)	Authors

Section 1: Executive Summary/ Abstract

Purpose: to provide a succinct and brief overview of the evaluation

This should include:

(i)	Rationale for the evaluation (why it was conducted)
(ii)	Main findings/ conclusions
(iii)	Recommendations

Section 2: Background Information on the Project being evaluated

(i)	Origins of the project - Reasons behind project being set up - by whom (e.g what type of agencies/ funders?).  What was the need identified? Was there a needs assessment conducted? Where was the project implemented? In what communities?

(ii)	Aims of the project

(iii)	Service users involved in project - who is it aimed at?

(iv)	Description of the project - interventions involved (type of support, therapy etc), activities, procedures, staff involved and what they did, location/s of work done with service users, their qualifications, what resources were available to whom (e.g transport, practical support), how highly prescribed was the programme (do we know precisely what was measured/ evaluated?). 

(v)	Description of service users involved in project - characteristics of intended participants on the project (e.g. age, socioeconomic status, special needs); how were participants selected onto the project?; Did participants remain on the project for its duration? (if not who left and what were their characteristics?)

Section 3: Description of the Evaluation Study

(i)	Purposes of the Evaluation - who requested it? Who is the audience for the evaluation? What is the information required for and how will it be used? Overall purpose of evaluation e.g. to determine effectiveness of a service? To find out why people were or were not using it? What were the specific research questions the evaluation sought to answer?
(ii)	Evaluation Design & Methodology - overall design, i.e. qualitative or quantitative. Methodology e.g. RCT, quasi-experimental study, survey, interviews etc. What were the strengths and weaknesses of the design chosen?
(iii)	Sampling - who was included and excluded in the evaluation and how were people chosen (type of sampling method, e.g. random sample, representative sample, snowballing)
(iv)	Outcome Measures - what were the outcomes measures and how were they chosen?
(v)	Data collection procedures (instruments such as questionnaires, standardised measures, interview schedules, procedures such as where data was stored (tape recorded and transcribed etc), who collected the data and where e.g. computer packages such as Atlas Ti or Excel)
(vi)	Data analysis - how was the analysis conducted? E.g. using a computer package such as SPSS. Were association and correlations looked for in the data? (Mostly in quantitative Research) Coding/ Themising, Grounded theory, IPA, Constant Comparison Analysis (qualitative research). 

Section 4: Results/ Discussion of the Results

(i) Ensure you present all the results at each stage that tests were taken and for each measure used

(ii) Ensure statistical tests are used and reported (if appropriate)

(iii) Ensure most important results are highlighted

(iv) Present in tables and graphs or diagrams to make it more interesting for the reader

(v) How certain is it that the results were caused by the project? How good were the results?

Section 5: Conclusion, Recommendations, Options

(i) What are the main conclusions about the effectiveness of the project?

(ii) Can judgements be made about all aspects of the project based on this evaluation?

(iii) Did the evaluation overlook anything? 

(iv) Has the study highlighted the need for particular areas requiring further research?  

(v) Recommendations or options regarding the project - what are its strengths and weaknesses? Is it sufficient? Is it liked by service users? Should it be refunded? 

TIP SHEET
Writing an Evaluation Report
Three Reasons To Write a Report:
1. You’ll have a formal record.
What you learned in conducting an evaluation, both the process and
the results, may be applicable to future programs to be planned by
you or others. Staff may change and your memory may fade; an
evaluation report is assurance that lessons learned are available for
future application.
2. You can help others.
Sharing your evaluation report with peers who may be considering
the development of similar programs may help them to design their
programs more effectively, convince them to use (or modify) your
program instead, and establish your reputation for good program
design.
3. You’ll have a foundation for future evaluation efforts.
It’s much easier to design an evaluation based on former experience
than to start “from scratch”. A report outlining what you did, and
why, as well as what worked and what should be altered in the
future, provides a solid base from which to plan a new pretest or
evaluation. Be sure to include any questionnaire or other instrument
you used in your report so that you can find and review them later.
Writing the Report:
Consider including these sections in your report:
·  Background: Purpose and objectives of the program
·  Description: What was evaluated
·  Purpose: Why the evaluation was conducted
·  Methodology: How it was conducted (with whom, when, how many,
instruments used)
·  Obstacles: Problems in designing or conducting the evaluation
·  Results: What you found out, and what application it ahs to the
program (program recommendations)
Although the report should provide a clear record of what you did, it should
not be any longer or more formal than needed. Keep it short and easy to
read. Attach any questionnaires, tally sheets or other instruments you used
as appendices instead of describing them in narrative form. Don’t make it
any harder a task than necessary!

I. Abstract or Executive summary (p. 383, Program Evaluation)
II. Introduction
	A. Purpose of the evaluation
	B. Audiences for the evaluation report
	C. Limitations of the evaluation report
	D. Overview of the report contents
III. Focus of the evaluation
	A. Description of the evaluation report
	B. Evaluative questions or focus objectives of the study
	C. Information needed to complete the evaluation
IV. Brief overview of the evaluation plan and procedures
V.  Presentation of evaluation results
	A. Summary of evaluation findings
	B. Interpretation of evaluation findings
VI. Conclusions and Recommendations
	A. Criteria and standards used to judge evaluation object
	B. Strengths and weaknesses of the object of the evaluation
	C. Recommendations
VII. Minority reports or rejoiners
VIII. Appendixes
	A. Evaluation plan, instruments, data analysis and interpretation
	B. Tabulations, analysis of quantitative data, transcripts, summaries of qualitative data
	C. Other information
http://www.aypf.org/rmaa/pdfs/Calvert.pdf   (Calvert evaluation)
--
Dr. Jerry Giraud's response to U2A2
Richard, you have an interesting approach that works well in organizing your evaluation. I have a couple of commments/suggestions. First, in your description, include a description of the context in which you are evaluating the Calvert system. That is, describe the school context where the Calvert is used for this evaluation study. I am curious, was it originally designed or is it intended for use in schools where english is a second language? If not, then the evaluation would take on new demension of interest. Include in your desciption details of setting, and how the Calvert system is used in the particular setting. Second, the questions you list as guiding the evaluation are more like interview quesitons, in that they ask someone (the unidentified you) to respond. Guiding quesitons should be phrased and are conceptually broad questions that the evaluation seeks to answer about the program. For example, rather than What do you like/dilike about the program, a guiding question might be What are participants' perceptions of the program?  In terms of grade for the course so far, I would assign an A to your discussion posts. You do a good job of referencing the text and clearly answering the questions posed.  	
----
Please post a brief introduction, including a brief description of your current work, your background in research/statistics, and what you hope to get from this course.
Hello, my name is Richard Bloodworth. For the past several years I have been teaching English in in foreign countries ( Seoul, South Korea; Tokyo and Chiba, Japan; Prague, Czech Republic; Istanbul, Turkey; Xi'an and Shanghai, China; and now Taipei, Taiwan). Prior to the English teaching I was employed in mostly arts related activities having studied art and architecture. I received my degree BFA from the University of Georgia in drawing and painting.  I have lived mostly in Athens and Atlanta, GA in addition to the above mentioned cities and I have traveled around Europe and Asia. 

I have taken three online courses, one with a lab, previous to this course (ED5004, ED5005, ED5006, and ED8111).   I am interested in learning more in the field of education in order to be more effective in my career and decided to pursue the online approach since it seems to be the wave of the future and also it allows me to take the courses from anywhere in the world.  Most of my current experience in research and statistics involves the papers written for the above mentioned courses.  I hope to attain from this course the statistical skills that this course offers.
--
I have read the course outline and expectations ( I am still waiting for the texts but I think they should arrive by Monday).  I am looking forward to gaining new knowledge and skills in this course.
--
U1D1
The evaluative report I found is a Caltech/MIT Voting Technology Project entitled Voting: What is, What could be.

This is an intriguing evaluative study that I would suggest for everyone to read, especially for those interested in future of democracy and political institutions.  The evaluation was done by teams of researchers from Caltech and MIT and can be read by activating the link in the reference below.  

In Part One of the Report the authors say that:

"We estimate that between four and six million presidential votes were lost in the 2000 election. These are qualified voters who wanted to vote but could not or were not counted. (Losses occur for two reasons: first, some voters do not, or cannot, participate due to problems with voter registration or polling place practices; second, some votes that are cast are not counted due to problems with ballots.) 

Two million ballots, two percent of the 100 million ballots cast for president in 2000, were not counted because they were unmarked, damaged, or ambiguous.
Of this two percent it is estimated that 0.5 percent did not intend to vote for president, so 1.5 percent (or 1.5 million people) thought they voted for president but their votes were not counted". (Caltech/MIT, 2001, 8).

In this well planned and conducted 92 page evaluative report there are numerous statistics as well as suggestions as to how to remedy the problems associated with public voting and how to incorporate new available technology to implement their suggestions.

Reference:

Caltech/MIT Voting Technology Project, California Institute of Technology and The Massachusetts Institute of Technology Corporation.  (2001).  Voting: What is, What could be.  Retrieved January 8, 2004  from  http://www.vote.caltech.edu/Reports/july01/July01_VTP_%20Voting_Report_Entire.pdf

-

U1D1  response to Barbara:

I was as shocked as you (as well as most other people) when I read these statistics but, as I read further on, they seem to be well-supported.  When referring to large populations, large numbers could result.  For example, when calculated based on a population of 100 million, the + or - 4% margin of error mentioned in opinion polls (such as the Harris or Gallop polls) would translate to + or - 4 million.

Below are some supporting statistics of the 4 to 6 million estimate of lost votes:

"We lost between one-and-a-half and three million votes because of the registration process in 2000.  According to the U.S. Census, Current Population Survey, 7.4 percent of the forty million registered voters who did not vote stated that they did not vote because of registration problems." (Caltech/MIT, 2001, p.8).

We lost between 500,000 and 1.2 million votes because of polling place operations. According to the U.S. Census, Current Population Survey, 2.8 percent of the forty million registered voters who did not vote in 2000 stated that they did not vote because of problems with polling place operations such as lines, hours, or locations. The figure was 1.2 percent in 1996.  (Caltech/MIT, 2001, p.9).

Below is an explanation of the term "residual votes":

RESIDUAL VOTES = Uncounted ballots + Unmarked ballots + “Overvoted ballots”

BALLOTS THAT CONTRIBUTE TO THE RESIDUAL VOTES:

Uncounted ballots: Ballots that are cast by voters but uncounted by election officials for whatever reason.
Unmarked ballots: Sometimes termed the “undervote.” May occur because the voter abstained or the recording device did not register a mark.
Overvoted ballots: Ballots that record a vote in more than one place for a given office (unless the ballot explicitly allows for more than one choice to be made.) May occur because the voter clearly marked more names than allowed.  Often occurs when a voter places a legal mark next to a candidate’s name and then writes the same name on the “Write-in candidate” line on the ballot.  (Caltech/MIT, 2001, p.20).

"Over the past four presidential elections, the rate of residual votes in presidential elections was slightly over two percent. This means that in a typical presidential
election over two million voters did not have a presidential vote recorded for their ballots.  The presidential race is the “top of the ticket.” The rate of residual votes is even higher down the ballot- five percent for Senate and gubernatorial elections. In other words, almost five million votes are not recorded for other prominent statewide offices.  It is difficult to judge intentions,  but exit polls suggest approximately thirty percent of residual votes are intentional. This implies that 1.5 million presidential votes are lost each election; 3.5 million votes for governor and senator are lost each cycle."  (Caltech/MIT, 2001, pp.20, 21).

Here is a summary of the sources of lost votes mentioned:

"1.5 to 2 million lost because of faulty equipment and confusing ballots
1.5 to 3 million lost because of registration mix ups
Up to 1 million lost because of polling place operations
Unknown losses because of absentee ballot problems."   (Caltech/MIT, 2001, p.9).

Concerning recommendations below is a list of some of them made in the Caltech/MIT report:

RECOMMENDATIONS:

• Move away from complex, monolithic machines.
• Make source code for all vote recording and vote counting processes open source and source code for the user interface proprietary.
• Make recording software openly auditable in the same mode that is used to conduct the counts.
• Adapt equipment so that voters can create a record of the vote that they can examine directly, and that can be used to audit equipment and elections.
• Conduct audits of votes and equipment, even without a recount.
• Design equipment that logs all events (votes, maintenance, etc.) that occur on the machine.
• Train election officials in the interior workings of their voting equipment.
• Delay Internet voting until suitable criteria for security are put in place.  (Caltech/MIT, 2001, p.42).
• The federal government,working with state and local governments, needs to develop standard methods of accounting for election expenses and standard reports that are made publicly available.
• The federal and state governments should offer significant matching funds for upgrades to replace voting technologies-such as punch cards, lever machines, centrally counted optical scanning, paper, and some under-performing DREs-that are clearly dominated by existing equipment.
• The federal and state governments should pay for the maintenance of voter registration databases maintained at the state and local levels.
• The federal government needs to maintain a publicly available database of election expenditures.   (Caltech/MIT, 2001, p.48).
• The federal government should establish a National Elections Research Lab or program, along the lines of that in Brazil.    (Caltech/MIT, 2001, p.66).
• Include real voters in testing process    (Caltech/MIT, 2001, p.72).
• Test equipment as it is set up and used at the polling place
• Require that all non-interface software be open source
• Re-test systems after field use
• Perform random system audits
• Separate the certification process for ease of use and for security    (Caltech/MIT, 2001, p.73).
• Develop new testing protocols and guidelines for ease of use of ballots and user interfaces
• Develop a standard process for review of ballots and user interfaces
• Develop hardware and software specifications for vote casting-devices
• Create a new standard for redundant recordings
• Develop standards for voter educational materials
• Create a national elections standards commission    (Caltech/MIT, 2001, p.74).Richard

Reference:

Caltech/MIT Voting Technology Project, California Institute of Technology and The Massachusetts Institute of Technology Corporation.  (2001).  Voting: What is, What could be.  Retrieved January 8, 2004  from  http://www.vote.caltech.edu/Reports/july01/July01_VTP_%20Voting_Report_Entire.pdf

---
Barbara,

Below are answers to some of the many questions that you asked me. (I have only asked you one and it is embedded within one of the answers, can you find it?):
On what do you base your statement that this was a well-planned and well-conducted report, other than numerous statistics and suggestions?
By having read the report and the procedures for arriving at their conclusions.
What were some of their suggestions, and do you agree with them?
The suggestions or recommendations are mentioned in the previous response and yes I agree with them. 
Do you think the evaluators presented the data objectively - because it is possible to manipulate numbers so as to show almost any result you are looking for.
I think that it was presented relatively objectively considering it is about voting and politics which involves a collection of voting subjects and is therefore a subjective issue in natue.  I don't think they manipulated numbers but instead used existing statistics to illustrate their ideas.

You mentioned that between four and six million votes were lost, and go on to describe how 2 million ballots were not counted. Is this 8 million total then?
I believe the 2 million uncounted votes are included in the 4 to 6 million lost vote estimate.
Although these numbers are decidedly large, what percent of the total population does this represent? 
The U.S. population is estimated to be about 292, 357, 744  (http://www.census.gov/cgi-bin/popclock) so 4 million is 1.3% and 6 million is 2%.  To get the percentages based on the number of registered voters we could divide the 4 to 6 million by the number of registered voters which would be a higher percentage.
Would it be enough to make a difference in the election results?
Yes, Al Gore received about 230, 000 more counted votes in the 2000 election and George W. Bush received only a few hundred more counted votes in the Florida election so these sorts of numbers would make a difference and, besides, I was always taught that every vote counts, is that in fact not the case?
Is there any mention in the report about people who simply don't want or choose not to vote?
Yes.
For other details and answers to other questions please, as I am not one of the authors of the report, please read the article where I think you will find some of the information and explanations that you are asking about.
Thank you for your interest, Richard 
Reference:

Caltech/MIT Voting Technology Project, California Institute of Technology and The Massachusetts Institute of Technology Corporation.  (2001).  Voting: What is, What could be.  Retrieved January 8, 2004  from  http://www.vote.caltech.edu/Reports/july01/July01_VTP_%20Voting_Report_Entire.pdf

--

Response to everyone:

To answer your questions regarding the solutions to voting problems in the United States of America, I would invite you to read the report, the Caltech/MIT Voting Technology Project entitled Voting: What is, What could be.  Within this report you will find terms defined, explanations given, and the problems (the "What is" section) and recommendations for solutions (the "What could be" section) elaborated upon.  

You can read this report by entering the following address in your web browser address box and then clicking "Enter":
http://www.vote.caltech.edu/Reports/july01/July01_VTP_%20Voting_Report_Entire.pdf

The fact that the report was written in a rational, non-hysterical and non-contentious way by some of the highest level academics of two of the highest level academic institutions (and though we of course all know that just because of this it doesn't mean that it is dictated by God), California Institute of Technology and the Massachuchetts Institute of Technology, helps to support the fact that it is worth reading and considering.  

Thank you for your interest,  Richard Bloodworth
--
Professor Giraud,
I have been catching up on my reading as my books have just arrived.
I want to ask you something about the evaluation report we are to write and what of several ideas I have would be the best program to evaluate.  I wrote in the first discussion that I was thinking of evaluating the Calvert System since we are using that in the school where I am teaching. 
In my previous Capella courses I wrote on the subject of direct democracy (voting directly rather than through representatives).  I thought another possibility for a report would be to write an evaluation of some direct democracy procedures already in use, for example in Switzerland or I could write an evaluation of the report that I mentioned in the CourseRoom, Caltech/MIT Voting Technology Project entitled Voting: What is, What could be (but that might be an evaluation of an evaluation).  What do you think about those ideas?
To give you an idea of my line of thinking concerning direct democracy, I've included two attachments which are papers I wrote for previous courses.
Thank you,  Richard Bloodworth
--
Professor Giraud,

Many of these figures seem to have been derived from the U.S.Census through interviews, surveys, and questionnaires and some from voting method and equipment analysis and interviews with voters through exit polls, etc.  For some of the specific statistics mentioned, please read my U1D1 response to Barbara.  Richard
--

U1D2  
A. Which would be more appropriate for this program, a formative evaluation or a summative evaluation? Please explain your response. 

B. Which would be more appropriate to conduct this evaluation, an external evaluator or an internal evaluator? Please explain your response. (T 12a, L 6h.)

The program I am considering evaluating is one that I am involved with presently and that is teaching using the Calvert School system.  The Calvert School began over 100 years ago as a small private school, then a larger school was developed, and then textbooks were developed for home schooling using the Calvert System.
The system begins with pre-kindergarten and moves up through eighth grade plus advanced enrichments. Among other subjects, mathematics, writing, literature, and science are presented through a variety of colorful books that are all integrated into the Calvert system with suggested lesson plans that, if completed, would cover all of the material in the provided texts.

Even though the books were developed with home schooling in mind, many schools are using the books in their study curricula.  In the school where I am teaching the system of Calvert books is also used as a method for an ESL (English as a Second Language) total immersion technique.   From my perspective, an evaluation of the system (and also in its relation to its effectiveness as a method of foreign language instruction) would be formative since I am involved in teaching using that system.  However, I could read and report on evaluative studies that have been based on statistical studies of test results, student and teacher achievement and satisfaction, etc of people who have been involved in the program over a period of time.  In a formative study, I would be the internal evaluator but for summative conclusions I would rely on the efforts of external evaluators.
-
U1D2 Comment to Dar

It is unfortunate that the visual arts often seem to get put on the back burner in the schools' curricula.  Though I have always realized the importance of the visual arts, having studied it in school and done quite a bit of it myself, I have realized its importance even more after having taught ESL (English as a Second Language) since visual language is understood universally and globally whereas verbal language is not.  For example, the message of a photograph or painting of starving or malnurished child is understood by everyone regardless of their cultural background or verbal language.

--

U2D1

-- Post who the client is (you do not need to reveal his or her name) and how you established a client-evaluator role with him or her.

The client is the owner of the American English School in San Chung City (which is adjacent to Taipei), Taipei Hsien, Taiwan.  I established a client-evaluator role with her in order to satisfy the requirements for the course, Evaluating the Effectiveness of the Educational Process, and since I am one of the English language instructors at the school which is a privately owned high school preparatory school (with the purpose of ultimately being a college or university preparatory school);  the school is a sponsor and user of the Calvert system.  However, if the definition of client is "the specific agency or individual who requests the evaluation" (Fitzpatrick, Sanders & Worthen, 2004, p. 174) then it could be said that the client is Capella University since they requested the evaluation as part of the course in program evaluation.

A. What are the broad questions that will drive the evaluation, and how was the client involved in choosing or refining them? 

The very broad questions I asked the students, other teachers, and administrators are:

What do you like about the Calvert program?
What do you not like about the Calvert program?
How do you think that it could be improved?

Other questions are:

What do you think are the objectives or philosophy of the program?
How do you think the model of the program is intended to work?  
Do you think the model of the program does accomplish its intentions?
What concerns do you have about the program?
What do you hope to learn from the evaluation?
Why are these issues important to you?
How can you use the information provided by the evaluation?

The school's manager, as representative of the client/owner, and another teacher using the Calvert system were involved in determining what questions are to be answered and they have already given answers to some of the questions.

B. Do you think that involving the client in deciding what participants and methods will be involved in data collection will bias your evaluation?

I think to some extent that involving the client in deciding what participants and methods would be involved in data collection would bias the evaluation primarily for the reason that the participants, and especially the client, would want their evalutation to have a positive outcome in their favor to justify the expense in involved in investing in the program.  I think it is probably best to involve them in the project without letting them know all of the details as to the purpose or procedures of the study, as a sort of Candid Camera approach, otherwise the clients and stakeholders might only say what they think should or what they want to be true as this is the case everywhere but especially in asia where the concept of  "saving face" is paramount.
---
To Fredi and Dar

The "client" in this case is only paying me to teach the language course and not to perform an evaluation of the course.  Also, keep in mind that I am in the ROC (Republic of China) not the USA (the United States of America).  I think in the U.S., or in another situation, I would approach this project differently and I would discuss the project with the school administration.  The candid approach I was referring to would be to establish a situation that would create an environment where everyone would not feel "you are being watched, observed, and evaluated" in order to achieve a more natural result and is referring mostly to the students.  (The teachers are aware of this project).  This is a much different culture where, among other differences, English is not spoken so trying to explain the circumstances of a project is not as simple to accomplish.  For example, here, Christmas is not a holiday.  The big holiday for them that we are about to have is Chinese New Year.

Happy Chinese New Year,  Richard

Reference:

Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
--
to C Ball
I have one question about sign language. I am wondering if there is an international sign language so that if, for instance, a Japanese person uses sign language would he be understood by a French speaker or an English speaker using sign language?
Just wondering.
---

U2D2
Identify who the key stakeholders are for the program that you will be evaluating. Do not reveal their names, but relate how you identified them as key. Then respond to the following question: What role, if any, should the stakeholders have in planning and/or conducting the evaluation? 
Stakeholders can include a broad range of members including "sponsors, clients, program managers, staff, recipients of program services and their families, other agencies affiliated with the program, elected officials, and the public at large" (Fitzpatrick, Sanders & Worthen, 2004, p. 174) -- which sounds like almost everybody, since everybody is what the public at large is, or at least anyone who has the remotest connection to a particular program.   The key stakeholders for the Calvert System evaluation at our school are primarily the students, 14 in one class and 13 in another, who are the students at the school who are studying using the Calvert system and learning their academic subjects in addition to English as a second language, and the parents who have invested in the program -- and their childrens' educations -- and the school's staff (the owner, manager, administration, and teachers).
I think the owners, staff and management of the school could have some say in the planning and conducting of the evaluation.  I have already asked some of them (staff and teachers) some of the questions mentioned in my U2D1 posting such as what do you like or not like about the Calvert program and what improvements can be made and what programs can be eliminated or what new programs can be added.   I think the students, especially young students, could have some part in the participatory aspect of the evaluation in the sense that their achievements can be monitored and recorded and suggestions can be elicited from them.  Older, more mature or adult students could have almost the same questions asked of them as are asked to the clients, staff, and teachers.
To get the opinions of some of the young students, I distributed questions to them essentially asking them what they like, what they don't like, and what they would want to change (what to add to or remove from the Calvert program).  Most all of them understood and answered appropriately with subjects and aspects that they like or don't like about the program though some of them gave answers such as "I like dogs" or "I don't like frogs".  So when including people, students, or stakeholders in the evaluation process it is important to identify and recognize their particular levels of understanding.

Reference:

Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
--
U2A2    eracs@cox.net <mailto:eracs@cox.net>

Dr Gerry,

Here is an outline as derived from the text, Program Evaluation: Alternative Approaches and Practical Guidelines.  How about our including a Bibliography at the end of the evaluation?  Richard

An Evaluation of the Effectiveness of the Calvert School Program in Educating the Students in the Subjects Offered and Teaching English as a Second Language

A Paper Presented in Partial 

Fulfillment of the Requirements of

ED814

Evaluating the Effectiveness of the Educational Process

January, 2004

Richard Bloodworth
P.O. Box 78123
Atlanta, GA  30357
RBloodworth99@yahoo.com

I. Abstract

		In this formative and participatory evaluation, the effectiveness of the Calvert School program, originally a private school in Maryland, then later also a program for home schooling, and now a curriculum used by many schools within the educational institutions, is examined using classes at the American English School in San Chung City, Taipei Hsien, Taiwan.  The program is evaluated for its effectiveness in educating the children in the subjects offered (reading, writing, mathematics, science, geography, history, phonics, vocabulary building, language acquistion) with an emphasis on its effectiveness as a method of teaching English as a second language to young students, in this particular case, from Taiwan.  The Calvert program ranges from Pre-Kindergarten up through the eighth grade but this study will concentrate on its effectiveness on mostly the first and second grade levels.

II. Introduction

	A. Purpose of the evaluation -- to determine the effectiveness of the Calvert Program. I chose this program since I am one of the language instructors at the American English School using the Calvert School program.
	B. Audiences for the evaluation report -- The stakeholders and the course participants of ED814
	C. Limitations of the evaluation report  -- The evaluation being done by me functioning as an internal evaluator
	D. Overview of the report contents
III. Focus of the evaluation
	A. Description of the evaluation report and of the Calvert Program
The Calvert Program is an extensive program covering all subjects for students being educated at home or by schools using the Calvert system.  The program is comprised of a Lesson Manual accompanied by many supplementary books, all in colorful format and design to hold the interest of the young students.  The books for Grade One are listed later in this evaluation.  The program also includes in the educational package/kit sent to the home student or schools as one package for each students containing the many books (more than 20) and supplies including ruler, compass, Cousenaire rods, pens, pencils, art supplies, paint, crayons, erasers, glue, scissors, sharpeners, and even an inflatable Earth globe the size of a beach ball.  The students follow lessons, and if they spend one day on each lesson, each level lasts one school year.  Though this evaluation will concentrate on the first grade, all of the color-coded levels in the Calvert Program are listed below:

Pre-Kindergarten	Lavendar
Kindergarten		Light Green
First Grade		Light Blue
Second Grade		Purple
Third Grade 		Orange
Fourth Grade		Gold
Fifth Grade		Dark Green
Sixth Grade		Red
Seventh Grade		Blue
Eighth Grade		Brown
Enrichments		Navy Blue

 	B. Evaluative questions or focus objectives of the study
		1  .What do you like about the Calvert program?
		2.  What do you not like about the Calvert program?
		3.  How do you think that it could be improved?
		4.  What do you think are the objectives or philosophy of the program?
		5.  How do you think the model of the program is intended to work?  
		6.  Do you think the model of the program does accomplish its intentions?
		7.  What concerns do you have about the program?
		8.  What do you hope to learn from the evaluation?
		9.  Why are these issues important to you?
		10. How can you use the information provided by the evaluation?
	C. Information needed to complete the evaluation

Texts in the series:
Lesson Manual
Red Robin, Fly Up!
Here we Go
Treasures
Wonders
Cooper, J.D. (2001). Surprises. Boston, MA: Houghton Mifflin Company.
Let's Be Friends
Reading Work Pages
Reading Work Cards
Stories and Longer Verses
98 Stories for Children
Writing Fun
Phonics Library. (2000). Boston, MA: Houghton Mifflin Company.
Mathematics:
	Calvert Math
	Calvert Math: Practice and Enrichment Workbook
	Math Manual
Activity Pages
History Read-Alouds
Moyer, R. (2000). Science. New York: National Geographic/McGraw-Hill Co., Inc.
Activity Pages
Billings, H. (2000). Maps, Globes, Graphs. Austin, TX: Steck-Vaughn Co./Harcourt Brace.

IV. Brief overview of the evaluation plan and procedures
V.  Presentation of evaluation results
	A. Summary of evaluation findings
	B. Interpretation of evaluation findings
VI. Conclusions and Recommendations
	A. Criteria and standards used to judge the Calvert program
	B. Strengths and weaknesses of the object of the evaluation
	C. Recommendations
VII. Minority reports or rejoiners
VIII. Appendixes
	A. Evaluation plan, instruments, data analysis and interpretation
	B. Tabulations, analysis of quantitative data, transcripts, summaries of qualitative data
	C. Bibliography	
	D. Other information
Contact information for the Calvert School:
Education Services
10713 Gilroy Road, Suite B
Hunt Valley, Maryland 21031
1-888-487-4652
http://www.calvertschool.org

---
Dr. Jerry Giraud's response to U2A2:
Richard, you have an interesting approach that works well in organizing your evaluation. I have a couple of commments/suggestions. First, in your description, include a description of the context in which you are evaluating the Calvert system. That is, describe the school context where the Calvert is used for this evaluation study. I am curious, was it originally designed or is it intended for use in schools where english is a second language? If not, then the evaluation would take on new demension of interest. Include in your desciption details of setting, and how the Calvert system is used in the particular setting. Second, the questions you list as guiding the evaluation are more like interview quesitons, in that they ask someone (the unidentified you) to respond. Guiding quesitons should be phrased and are conceptually broad questions that the evaluation seeks to answer about the program. For example, rather than What do you like/dilike about the program, a guiding question might be What are participants' perceptions of the program?  In terms of grade for the course so far, I would assign an A to your discussion posts. You do a good job of referencing the text and clearly answering the questions posed. 	

My response to Dr. Jerry:
Thank you  for your comments.
One question: Do you have a sample project evaluation done for this course, or a similar one, that I could read?  (In the other courses, there were sample papers in the MediaCenter but I didn't find one for this course.)  
Since this is a new process to me (performing this sort of evaluation) I anticipated that my first draft would be somewhat sketchy and exploratory.  I think each such successive submission will add new information -- fill-in-the-blanks, as it were -- and changes and modifications based on the new information obtained, readings from the texts, and your suggestions.
I'll write a more complete description of the school context for the evaluation and rewrite the questions in third person based on your suggestion.  I should have made it more clear that the main purpose of the evaluation is to determine not only the effectiveness of the program in teaching the students the subject matter (mathematics, reading, science, history, etc.) but also to determine its effectiveness for learning English as a second language since this program was developed with native English speaking young students in mind who are for the most part being educated at home.  The teachers like using this method because it covers not only grammar, spelling, and vocabulary but it also introduces these skills within the context of the above-mentioned subjects thus making the teaching, and hopefully the learning, of the information more enjoyable, fulfilling, useful, and effective which is the intention of the school.
Thank you for your helpful suggestions,
Richard
---
U3D1
Consider the program that you are evaluating for this course. Post your thoughts on how and when you may need to take extra steps to show respect and sensitivity to any person or persons involved in the evaluation. Which stakeholders or consumers might be sensitive to questions in the data gathering part of the evaluation in the context of your study, and in what way? What information that you will gather might be potentially embarassing or even harmful to participants?
I think everywhere but especially in foreign countries, it is important to be aware of the cultural differences and of the foreign countries customs and beliefs  For example, it might be alright in some places but being blunt in Taiwan is considered bad form.   Modesty and humility are considered admirable but boastfulness and pushiness are considered low form on the Confucian scale.  Here, as in Japan, politeness is supreme and rudeness is taboo.  In expressing opinions, here one should gingerly beat around the bush rather than get to the point.   Respect to elders should be shown in all interactions.  Nothing should be done to embarass anyone, which is part of the idea of saving face I referred to in a previous posting.  Papers, especially paper money, should be handed to the recipient with two hands, not one.  In marking papers, using (too much, just some is OK) red ink is discouraged as it is considered an unfriendly, hostile gesture.  Also, one should never touch, or especially hit, a student's head -- after all, it is the Temple of the Mind. The schools here have an emphasis on science and technology, hence the importance they place on those subjects as well as mathematics and their educational system is an exam-based system so many of the student study in specialized schools to help them prepare for their difficult exams that will allow them to enter a school of their choice.   Another example of cultural differences and expectations: when entering someone's home in Taiwan, and in some schools in Japan, removing one's shoes is essential because to walk into someone's home with one's shoes on is considered crass and could be comparable in the west to someone walking on someone's bed with their shoes on.
While doing this evaluation (the results of which, by the way,  would probably of more interest to --rather than the school using the program -- the Calvert School, who could conceivably modify a part of their program, using the same material, with ESL students in mind, and would therefore be a potential "client" for the evaluation if they are interested in developing the ESL approach to their program) I should consider the above mentioned cultural differences and ask any questions or gather any information with these cultural differences in mind and keep in mind the rights of the human subjects involved, the stakeholders which are the students, staff, and administration of the school whose program I will be evaluating.
References:  
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications.
--
U3D2
Consider each of the four general standards of program evaluation: Utility, feasibility, propriety, and accuracy. How will you ensure that the evaluation you conduct for this course follows each of these four standards? Please be specific and provide examples that relate to the particular program that you are evaluating for each of the four standards. 
Utility -- fitness for some purpose or worth to some end (Merriam-Webster Dictionary, 2003). The extent to which an evaluation prodeuces and disseminates reports that inform relevant audiences and have beneficial impact on their work.  (Sanders, 1994).
The purpose of this report is to determine the effectiveness of the Calvert program in transmitting the curriculum of subjects offered but primarily to determine its effectiveness as a method for teaching English as a second (or foreign) language.  The results of this information could be useful to the stakeholders (the students, parents, teachers, staff, the school and its owners, and also possibly the Calvert school organization itself), mostly for the school itself to determine if this is a program that they should be using, and also to the Calvert School, who could conceivably modify their program, using the same material, with ESL students in mind.
Feasibility -- that which is capable of being done or carried out (Merriam-Webster Dictionary, 2003).  The extent to which resources allow an evaluation to be conducted. (Sanders, 1994).
I think the formative part of the evaluation can be done by me as an internal evaluator based on questionnaires, interviews, test results, and students' performances.  The summative aspect or external evaluation can be done by reporting the results of evaluations of the Calvert Program.  I have found some assessing its ability to transmit the information of the traditional curriculum of subjects and will look to see if there have been other any studies done about the program's effectiveness as a method for teaching English as a second language.
Propriety -- conformity to what is socially acceptable in conduct or speech (Merriam-Webster Dictionary, 2003). The extent to which the evaluation has been conducted in a manner that evidences uncompromising adherence to the highest principles and ideals (including professional ethics, civil law, moral code, and contractual agreements). (Sanders, 1994).
To satisfy the first (dictionary) definition I will try to abide by the social customs and expectations that I mentioned in the U3D1 posting.  To satisfy the (Sanders) text definition, I will try to adhere to ethical principles and treat the participants in the evaluation in a respectful way, obtain the information fairly and truthfully, and report  the information according to professional, moral, and ethical methods.
Accuracy -- freedom from mistake or error and conformity to truth or to a standard or model (Merriam-Webster Dictionary, 2003).  The extent to which an evaluation is truthful or valid in what it says about a program, project, or material.  (Sanders, 1994).
I will be as accurate as possible in reporting the the objective, quantitative aspects of the evaluation, such as test results and answers to questions, but I think the assessment and evaluation of this sort of report is qualitative in nature so the interpretation of the data could be considered subjective.
References:  
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications. 
--
U3D1 comment to Daphney
If a teacher is teaching and functioning well and dispensing the correct information then I don't think that the teacher can be blamed for poor performance by the student(s).  I think that the teacher is really a facilitator or a catalyst for the learning process but, beyond that point, success is up to the student(s).
U3D2 Comment to Ginger: Qualitative and quantitative
You certainly covered the four standards (utility, feasibility, propriety, and accuracy) very well.  One could get a lot of information about them just by reading your posting.  You say that you are "a little rusty with meanings of both qualitative and quantitative analysis".
Here are some definitions and explanations of the terms:
For qualitative just think "quality" or essence and for quantitative think "quantity" or number.
Qualitative research (postpositivist research). Inquiry that is grounded in the assumption that individuals construct social reality in the form of meanings and interpretations, and that these constructions tend to be transitory and situational.  The dominant methodology is to discover these meanings and interpretations by subjecting the resulting data to analytic induction. (Gall, 2003).
Qualitative research characteristics: variables, selective sampling of smaller populations, qualitative analysis, interpretive, transitory, naturalistic, subjective, localized.
---
Quantitative research (positivist research).  Inquiry that is grounded in the assumption that features of the social environment constitute an objective reality that is relatively constant across time and settings.  The dominant methodology is to describe and explain features of this reality by collecting numerical data on observable behaviors of samples and by subjecting these data to statistical analysis.  (Gall, 2003).
Quantitative research characteristics:  constants, random samplings of larger populations, quantitative analysis, statistical, stationary, analytical, objective, universal.
Reference:
Gall, M.D., Borg, W., & Gall, J.P. (2003). Educational Research: An Introduction. 7th Ed. Boston: Pearson Education, Inc.
--
U4D1
Do you think there are any two of the approaches that should never be used together in an eclectic model? Please justify your answer.
The management-oriented evaluation and the consumer-oriented (or the participant-oriented) evaluation represent opposite ends of evaluated population segments so they generally should not be used together even if someone wanted to add the two segments to create one whole unit and such a study would be somewhat analogous to trying to be two places at one time.  What could be done instead would be to have two separate evaluations done with one from the management's perspective and the other from the consumers' (or participants') perspective and then examine the results of the two separate studies to draw any conclusions.
The management-oriented evaluation has the advantages of being able to affect policy but the disadvantages of costliness in administering the findings and the narrow concerns of the organization's leaders.  The advantages of the consumer-oriented (or participant-oriented) are potentially better products and services but the disadvantages can be the costs involved, not being open to cross examination, and the suppression of creativity.
References:  
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications. 
U4D2
Which aspects of the various evaluation approaches will you use in the eclectic design of the program evaluation that you conduct for this course? Please explain how you came to make this decision.
In evaluating the effectiveness of the Calvert program for the dispensing and retention by the students of the academic subjects offered but especially the program's effectiveness as a means of teaching English as a second language, the evaluation methods I will use most are the objectives-oriented (to satisfy the above stated objectives) and the participants-oriented evaluation with the participation of the stakeholders of the program which are primarily the students enrolled in the language courses.
The objectives-oriented evaluation will involve logic models suggesting what improvements could be made and program theory by noting the causes of the observed outcomes.  Through the use of empirical information such as test results and student performance and language comprehension, conclusions can be drawn to assess the effectiveness of the program and the evaluation can, as formulated by M. Provus, go through the five stages of (1) definition, (2) installation, (3) process, (4) product, and (5) cost and benefit analysis.
The participant-oriented aspect of the evaluation could also involve the above mentioned use of empirical information such as test results, student performance, and language comprehension as well as perhaps cases studies of individual students.  This would be a participatory and responsive (pluralistic and process-focused) evalutation and it could also involve the use of utilization-focused evaluation to determine the usefulness of the program for its intended purposes.
References:  
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications
---
U4
Activity 4.1 
Read Chapters 4, 5, 6, 7, 8, 9, 10, and 11 of the Worthen, Sanders, and Fitzpatrick textbook. 
Activity 4.2 
E-mail the first draft (approximately two to three pages) of the Evaluation Design section of your course evaluation project to your instructor. This section describes the eclectic approach that will be taken and includes the methods that will be used to collect the evaluation data. Also included should be information about the participants and the instrumentation that will be used. The facilitator will provide comments for revision and inclusion in the course final project. (T 10b, 2d)
Dr. Gerry,
Here is the design plan for the evaluation.  Concerning information about the participants and the stakeholders, I have only used their first name in English (not their real Chinese names) so I don't think that any privacy will be invaded but since the assignment asked about information about the participants I included that information.  Even though it would remain anonymous, you can tell me if some of the information is too personal, or perhaps not relevant, (such as someone's father being killed in the line of duty), but it is only done to give a human and individual face to each of the students.  Also, the language acquisition level of each of the students would be individually and qualitatively assessed so that would also contain a personal aspect.
This will be an objectives-oriented and a participant-oriented evaluation primarily with a formative, qualitative approach for a utilization-focused design using mostly qualitative analysis.  Also, attention will be paid to conducting the evaluation in a way respectful to the Taiwanese culture and customs and familiarity of the Chinese culture, its history, its philosophy and its adherence to the Confucian hierarchal structure.
All of the students (male and female) are Taiwanese children, between  8 and 9 years old, who speak Mandarin Chinese (though some might also speak the local Taiwanese dialect to their grandparents but this seems to be a fading local language, whereas Mandarin Chinese, mostly as a result of the large population of China, is the most spoken native language in the world with over 1.4 billion speakers which is 3 times more than the number of native English speakers) who are studying English in order to increase their future opportunities so that they may be able to work, study, travel, or live abroad.  Also, to get in to the sort of universities and schools that they want to enter, or that now their parents want them to enter, they need to have a thorough knowledge of the English language.
The classes being evaluated are comprised of students who are between 8 and 9 years of age.  In their elementary schools in Taiwan, they would be in the third or fourth grades but the texts that they are using are intended for native English speaking first graders so that some of the subjects such as mathematics, science, history, and geography may already be somewhat familiar to them but they are being taught within the context of a total immersion English language environment.
The Calvert School

		In this formative and participatory evaluation, the effectiveness of the Calvert School program, originally a private school originating over 100 years ago in Maryland, then later also a program for home schooling, and now a curriculum used by many schools within the educational institutions, is examined using classes at the American English School in San Chung City (which is adjacent to Taipei), Taipei Hsien, Taiwan, Taipei Hsien, Taiwan which is a privately owned high school preparatory school (with the purpose of ultimately being a college or university preparatory school).  The textbooks that were developed for home schooling using the Calvert System are also being used for the purposes of educating the students primarily with the objective of teaching them English as a second language using a total immersion technique.   In the school where I am teaching the system of Calvert books is also used as a method for an ESL (English as a Second Language) total immersion technique..  The system begins with pre-kindergarten and moves up through eighth grade with additional advanced enrichments. Among other subjects, mathematics, writing, literature, and science are presented through a variety of colorful books that are all integrated into the Calvert system with suggested lesson plans that, if completed, would cover all of the material in the provided texts. The program is being evaluated for its effectiveness in educating the children in the subjects offered (reading, writing, mathematics, science, geography, history, phonics, vocabulary building, language acquistion) but especially with an emphasis on its effectiveness as a method of teaching English as a second language to young students, in this particular case, from Taiwan.  The teachers like using this method because it covers not only grammar, spelling, and vocabulary but it also introduces these skills within the context of the above-mentioned subjects thus making the teaching, and hopefully the learning, of the information more enjoyable, fulfilling, useful, and effective which is the intention of the school.  The Calvert program ranges from Pre-Kindergarten up through the eighth grade but this study will concentrate on its effectiveness on mostly the first and second grade levels.

	The Calvert Program is an extensive program covering all subjects for students being educated at home or by schools using the Calvert system.  The program is comprised of a Lesson Manual accompanied by many supplementary books, all in colorful format and design to hold the interest of the young students.  The books for Grade One are listed later in this evaluation.  The program also includes in the educational package/kit sent to the home student or schools as one package for each students containing the many books (more than 20) and supplies including ruler, compass, Cousenaire rods, pens, pencils, art supplies, paint, crayons, erasers, glue, scissors, sharpeners, and even an inflatable Earth globe the size of a beach ball.  The students follow lessons, and if they spend one day on each lesson, each level lasts one school year.  Though this evaluation will concentrate on the first grade, all of the color-coded levels in the Calvert Program are listed below:

Pre-Kindergarten	Lavendar
Kindergarten		Light Green
First Grade		Light Blue
Second Grade		Purple
Third Grade 		Orange
Fourth Grade		Gold
Fifth Grade		Dark Green
Sixth Grade		Red
Seventh Grade		Blue
Eighth Grade		Brown
Enrichments		Navy Blue
Texts in the First grade program:
Billings, H. (2000). Maps, Globes, Graphs. Austin, TX: Steck-Vaughn Co./Harcourt Brace
Calvert School.  (2003).  Activity Pages.  Baltimore: Calvert School, Inc
Calvert School.  (2003).  History Read-Alouds. Baltimore: Calvert School, Inc.
Calvert School. (2003).  Lesson Manual.  Baltimore: Calvert School, Inc.
Calvert School.  (1976).  98 Stories for Children.  Baltimore: Calvert School, Inc.
Calvert School.  (2003).  Reading Work Cards. Baltimore: Calvert School, Inc.
Calvert School. (2003). Reading Work Pages. Baltimore: Calvert School, Inc.
Calvert School.  (2001). Stories and Longer Verses. Baltimore: Calvert School, Inc.
Calvert School. (2003). Tests. Baltimore: Calvert School, Inc.
Calvert School.  (1995). Writing Fun.  Baltimore: Calvert School, Inc.
Cooper, J.D. (2001). Here we Go.  Boston, MA: Houghton Mifflin Company.
Cooper, J.D. (2001).Let's Be Friends. Boston, MA: Houghton Mifflin Company.
Cooper, J.D. (2001). Surprises. Boston, MA: Houghton Mifflin Company.
Cooper, J.D. (2001). Treasures.  Boston, MA: Houghton Mifflin Company.
Cooper, J.D. (2001). Wonders.  Boston, MA: Houghton Mifflin Company.
George, J. (1993). Red Robin, Fly Up!  Baltimore: Calvert School, Inc.
Mathematics:
	Buffington, A. (2000).  Calvert Math.  Baltimore: Calvert School, Inc.
	Buffington, A. (2000).  Calvert Math:  Practice and Enrichment Workbook. Baltimore: Calvert School, Inc.
	Calvert School.  (2000).  Math Manual.  Baltimore: Calvert School, Inc.
Moyer, R. (2000). Science. New York: National Geographic/McGraw-Hill Co., Inc.
Phonics Library. (2000). Boston, MA: Houghton Mifflin Company.
The Stakeholders:
Owner:		Sabrina   --  owns a preparatory school and, with the profits from that school, then opened this school.  Most of the kindergarten and elementary students who graduated from that school transferred and continued in this new school, that she named the American English School which uses a franchised text called "Bumble Bee" for the younger students while the older students are using the Calvert School program which is being evaluated for the purposes of this report.
Teachers:  	Richard   --      Began teaching first grade English using the Calvert program (in addition to other classes using another program) in September, 2003.  Has also taught ESL to adults and children using other programs and texts in South Korea, Japan, Czech Republic, Turkey, and China.
		Oliver    --       Began teaching second grade English in July, 2003.  He was born in France and raised in Quebec (so he has a very French accent.)  
Staff:		Patricia   --    manager, and the school disciplinarian,  a Taiwanese who was once an English teacher herself being a good English speaker who lived a few years in the USA.
		Tiffany   --     staff assistant and at times teaching assistant for the Calvert classes and a sometimes translator for the students
		Other staff and teachers    ---   there are several Taiwanese staff and teachers and some other foreign teachers as well
Students:
First grade:	
		Tina 1  --  she is a well behaved and interested student and is learning her oral English.
		Peter --  energetic, a good speller and a helpful student and his mother is an interested parent who studies with him.
		Steven --  the best speller and makes the highest scores on the tests but pouts and cries if he does not get his way.
		Jason  -- disruptive but is getting better,  I heard that his father who was a policeman was killed (murdered) in the line of duty which might help explain his displinary problems.
		Bruce --  energetic and one of the best students (he was Rudolf in the Christmas show) but sometimes hard to control like Peter who is his best friend
		Debbie  -- quiet and her mind wanders off during the class and she often talks to Tina 2 who sits next to her so she does not pay attention
		Kevin  -- just wants to have a good time but is learning some even though he leaves many items on his tests blank.
		Howard  -- is one of the best students as he is personable but has a violent streak that often surfaces
		Ken  -- is one of the best students and seems to be the most mature and also physically the largest of the students (he was Santa at the Christmas show)
		Joey --  he is a good student but refuses to do some of the activities and pouts a lot and his father is an interested parent.
		Mark -- is the quietest student who does not join in many activities and does not mix with the other "naughty" boys so is only tolerated by them but he has an involved mother and he does fairly well on the spelling tests.
		Cherry   a good English speaker who once had a private Australian tutor but she is not able to write or spell well at all so because of this, and because she was having difficulty in her other classes in public school,  she discontinued the English classes at this school
		Tina 2 --  she is a fairly good student though she also sometimes has difficulty with comprehension so she does not respond quickly to task assignments.
		Andy -- is the brother of Cherry (there father is a medical doctor, a physician) and is a fairly good English speaker.
Parents of the above students.
		
	B. Evaluative questions or focus objectives of the study

		1  .What the participants like about the Calvert program.
		2.  What the participants do not like about the Calvert program.
		3.  How the participants think that it could be improved:  certain subjects and activities that they would like to have added, extended, or eliminated.
		4.  The objectives or philosophy of the program.  To teach the subjects offered while teaching English as a second language in an interesting way.
		5.  How the model of the program is intended to work: by teaching English with a total immersion technique using useful and interesting information.
		6.  Assessing if the model of the program does accomplish its intentions.
		7.  Concerns do you have about the program.  Different subjects are presented at different levels and the material is presented from an American-centric cultural view.
		8.  What is anticipated to be learned from the evaluation: hopefully that the program is effective and how it can be improved.
		9.  Why these issues are considered important.   To be sure this is an effective program and a worthwhile investment for the school and students.
		10. How the information provided by the evaluation can be used.  To make suggestions for improvement to the American English School and the Calvert School.
		11. Criteria for judging the program.  Are the students improving in their reading, writing, speaking, listening skills and retaining grammar and vocabulary information?
		12. Lesson plan assessment, to determine its effectiveness for English as a second language
		13. Program documents,  primarily the texts and study plans
		14. Interviews with stakeholders (students, teachers, staff, parents)
		15. Surveys  written and telephone surveys:   To get the opinions of some of the young students, I distributed questions to them essentially asking them what they like, what they don't like, and what they would want to change (what to add to or remove from the Calvert program).  Most all of them understood and answered appropriately with subjects and aspects that they like or don't like about the program though some of them gave answers such as "I like dogs" or "I don't like frogs".  So when including people, students, or stakeholders in the evaluation process it is important to identify and recognize their particular levels of understanding.
		16. External analysis, review of other summative evaluations of the Calvert School program, examination of existing records and documents
			such as:  http://www.aypf.org/rmaa/pdfs/Calvert.pdf  
		17. Qualitative analysis of data:  test results, survey, interview, and questionnaire results
	The effectiveness will be assessed by interviews with the students, parents, teachers, and school staff.  Most of the data analysis will be a qualitative assessment of the student's performance as indicated by verbal responses and understanding, spelling and writing ability, communication and conversational skills, accumulation of vocabulary, precision and accuracy of pronunciation, and the understanding of grammatical concepts including the understand the grammatical structure of a sentence and identifying and understanding the parts of speech (nouns, verbs, adjectives, adverbs, prepositions, conjunctions, verbals, interjections), and grammatical elements such as auxilary verbs.  Also an understanding of number and tenses (since there are none in Chinese since time is indicated by grammatical context.  For example, in Chinese, the literally translated equivalents of  "I go, I went, and I will go" are "Now I go, Yesterday I go, Tomorrow I go") will be assessed.  Similarly, in Chinese there is no singular or plural as this is also determined by context so that in Chinese the literally translated English of "[one] apple -- [six] apples" is "one apple -- six apple".  Also articles (a, an, the) are not used in Chinese so that often one hears "I want apple" or "Give me banana, please" since articles are low informational content words and are therefore not considered important and a thus among the first words to be omitted or the last to be added to one's conversational speech patterns.
January, 2004

Richard Bloodworth
P.O. Box 78123
Atlanta, GA  30357
RBloodworth99@yahoo.com

Contact information for the Calvert School:
Education Services
10713 Gilroy Road, Suite B
Hunt Valley, Maryland 21031
1-888-487-4652
http://www.calvertschool.org
---
U4D1 res
I think the generalizations of the evaluation descriptions leaves room for exceptions in many cases.  For this reason, I think most of the evaluation approaches should be analyzed on a case by case basis to determine whether or not there are conflict of interests or conflict of philosophies.
--
U4D2 res
---
U5A3
Activity 5.3 seems to be missing from the unit study assignments. Here it is: Develop a management plan (outline form is fine) for your evaluation that includes completion dates and a budget. Make your plan to fit within the time available in the course. Acknowledging the time available for the evaluation will encourage you to plan an evaluation that is realistic. Be aware that you might want to continue your evaluation beyond the course, depending on your purpose. If you want, you can include an extended plan to cover activities beyond the end of the course. Email your plan to be by Saturday, February 7
Unit 5 -- Week of Feb 2
Unit 6 -- Week of Feb 9
Unit 7 -- Week of Feb 16
Unit 8 -- Week of Feb 23
Unit 9 -- Week of Mar 1
Unit 10 -- Weeks of Mar 8 and Mar 15
---
Dear Dr. Gerry,
Here is a general management plan outline for the execution of the evaluation:
Week of February 2, 2004 -- examine some summative Calvert evaluations
Week of February 9 -- conduct telephone surveys, questionnaires, and interviews
Week of February 16 -- Qualitative analyses of above surveys, questionaires,and interviews
Week of February 23 -- Assess, graph, and chart test results 
Week of  March 1 --   Quantitative and qualitative analyses of test results
Weeks of March 8 -- 15 Compose evaluation into a final form
Since this evaluation is being done for the purpose of satisfying the requirements of a course about program evaluation there is not really a budget to speak of.  However, if this were to be done on a large scale, rather than in the one class that I am myself evaluating (other than questions to the other teacher in the other class).   Then the costs could involve the fee of the evaluator(s) and staff, consultants, travel costs, communications costs, printing costs, data processing, printed matter, supplies and equipment, subcontracts and overhead.  I could envision something like this occurring only if the Calvert School decided to conduct an evaluation to research the feasibility of adapting its basic (home schooling) program into an ESL (English as a Second Language) program worldwide.  As it stands now, the only costs would be the wages I receive to teach the language classes (but not to conduct an evaluation of the course content). The only payment by me is the fee paid to Capella for the course in program evaluation.  All of the equipment, such as photocopiers, etc. are already a part of the school equipment but if someone were to have to buy all of those sorts of equipment themselves then it could become an expensive endeavor.  The telephone calls I would make would be local and therefore minimal in cost.  But, again, if this were to be done on an international scale then the costs could escalate.

These focus objectives of the study will be addressed throughout execution of the evaluation:

		1  .What the participants like about the Calvert program.
		2.  What the participants do not like about the Calvert program.
		3.  How the participants think that it could be improved:  certain subjects and activities that they would like to have added, extended, or eliminated.
		4.  The objectives or philosophy of the program.  To teach the subjects offered while teaching English as a second language in an interesting way.
		5.  How the model of the program is intended to work: by teaching English with a total immersion technique using useful and interesting information.
		6.  Assessing if the model of the program does accomplish its intentions.
		7.  Concerns do you have about the program.  Different subjects are presented at different levels and the material is presented from an American-centric cultural view.
		8.  What is anticipated to be learned from the evaluation (hopefully that the program is effective and how it can be improved).
		9.  Why these issues are considered important.   To be sure this is an effective program and a worthwhile investment for the school and students.
		10. How the information provided by the evaluation can be used.  (To make suggestions for improvement to the American English School and the Calvert School).
		11. Criteria for judging the program.  (Are the students improving in their reading, writing, speaking, listening skills and retaining grammar and vocabulary information?)
		12. Lesson plan assessment (to determine its effectiveness for English as a second language)
		13. Program documents  (primarily the texts and study plans)
		14. Interviews with stakeholders (students, teachers, staff, parents)
		15. Surveys  written and telephone surveys:   To get the opinions of some of the young students, I have distributed questions to them essentially asking them what they like, what they don't like, and what they would want to change (what to add to or remove from the Calvert program).  .To follow up there will be some telephone and interview questions to the students and their parents. 
		16. External analysis, review of other summative evaluations of the Calvert School program, examination of existing records and documents
			such as:  http://www.aypf.org/rmaa/pdfs/Calvert.pdf  
		17. Qualitative analysis of data:  test results, survey, interview, and questionnaire results

	C. Information needed to complete the evaluation
Evaluation of the texts used in the program

----  Most of the following information will be consolidated during the final three weeks of the writing of the evaluation:

IV. Brief overview of the evaluation plan and procedures
V.  Presentation of evaluation results
	A. Summary of evaluation findings
	B. Interpretation of evaluation findings
VI. Conclusions and Recommendations
	A. Criteria and standards used to judge the Calvert program
	B. Strengths and weaknesses of the object of the evaluation
	C. Recommendations
VII. Minority reports or rejoiners
VIII. Appendixes
	A. Evaluation plan, instruments, data analysis and interpretation
	B. Tabulations, analysis of quantitative data, transcripts, summaries of qualitative data
	C. Bibliography	
	D. Other information

Dear Dr. Gerry;
I was going to reply about the guidelines you sent but I have been having email problems recently. The guidelines are what I was referring to previously when I asked for the basic form of the evaluation and I think that outline will help immensely in compiling the written evaluation. Even with all of the information contained within the text, I think that the inclusion of those outlines would make the text that much more complete. 
Thank you,  Richard
---
U5D1
Discussion 5.1: 
Think about the management plan that you will submit to your instructor in Activity 5.3. What are your areas of concern in terms of completion in a timely manner? What compromises will you have to make because of the time available? How do other resource limitations affect your plan?
In a word, the number one concern in the completion of the evaluation is: Time.
Other concerns include:
Cooperation by the stakeholders
Being able to adequately represent the results of test scores and quantitative data presented through the use of graphs
Time to collect and then analyze the data
Time to review past studies
Time to review the literature on the topic
Time to write all drafts and the final corrected paper.
Time to submit data -- fortunately, submission by email is immediate, but a mailed, printed hard copy could add an extra two weeks to the time frame.
---
U5D2
Discussion 5.2: 
How did you arrive at the costs associated with the evaluation? Have you factored in time as a cost? What other resources will be expended, and how will you determine their cost?
As it stands now, the costs of the program or the evaluation are the costs of the program to the students and the salaries of the teachers to teach the program.  But if someone were to hire an evaluator the main cost would be the fee of the evaluator(s) and the amount of time the evaluator(s) would spend on the project.  In this situation, I am the internal evaluator of my class but if this were to be done thoroughly it would be done by contacting several schools, teachers, and the schools' administrations.  One could multiply the cost times the number of schools being evaluated plus the fees of the evaluator(s). Also, for the purposes of evaluating the effectiveness of the Calvert program as a method for learning ESL (English as a Second Language), the evaluation could be done on a worldwide scale and this would increase the cost factor of the evaluation considerably.    The evaluator(s) would administer, collect, and analyze the data and, depending on the scope of the project, could spend a full school year for each evaluation in order to get a complete assessment.  In order to assess the effectiveness over time, a study could be done to follow the progress of selected groups of students over a five year (or more) period.  The more time that is spent on the project, the more it would cost; as they say, time is money.
When arriviving at cost estimates, the text mentions that the resources and their costs are:
1.  Evaluation staff salary and benefits.
2.  Consultants
3.  Travel and per diem or allowance (for staff and consultants) 
4.  Communications (postage, telephone calls, etc.)
5.  Printing and duplication
6.  Data processing
7.  Printed matter
8.  Supplies and equipment
9.  Subcontracts
10. Overhead (facilities, utilities, etc.)
References:  
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications
---
U5D1
Even though it is necessary, I have always thought that planning for the future is one of the most difficult tasks that we as humans have to do.  Since I  can not predict the future, it is difficult to say how long it will take to do something, particularly a creative task.  
I have heard that even God can not change the past and likewise no one can stop the forward march of time.  We all have to go forward into the future and plan for it the best we can so I have made a general plan and timeline for the execution of this evaluation -- I just hope I am able to come up with the time needed to do it all. 
Here is the time plan in general form:
Here is a general management plan outline for the execution of the evaluation:
Week of Feb 2  -- examine some summative Calvert evaluations
Week of Feb 9 -- conduct telephone surveys, questionnaires, and interviews
Week of Feb 16 -- Qualitative analyses of above surveys, questionaires,and interviews
Week of Feb 23 -- Assess, graph, and chart test results 
Week of Mar 1 -- Quantitative and qualitative analyses of test results
Weeks of Mar 8 -- 15 Compose evaluation into a final form
U5D2 response  The answer is: I don't know
Since this evaluation is being done for the purpose of satisfying the requirements of a course about program evaluation there is not really a budget to speak of.  No one is paying me to do this and I am not paying anyone either (unless the cost of the Capella course in program evaluation is considered as part of the cost of the evaluation).  As it stands now, the only costs possibly could be the wages I receive to teach the language classes (but this is not for the purpose of conducting an evaluation of the course content) -- and would the cost of an evaluation by me include the cost of the airplane ticket to Taiwan (to teach English) and all of the visas and work permits that I have to get to teach here, my house rent, food, etc? All of the equipment, such as photocopiers, etc. are already a part of the school equipment but if someone were to have to buy all of those sorts of equipment themselves then it could become an expensive endeavor.  The telephone calls I would make would be local and therefore minimal in cost and I as the evaluator am not being paid so there is not any exchange of money at this point directly for the purpose of this evaluation.
However, if this project were to be done on a large scale, rather than in the one class that I am myself evaluating, such as on an international scale, then the costs could escalate.   Then the costs could involve the fee of the evaluator(s) and staff, consultants, travel costs, communications costs, printing costs, data processing, printed matter, supplies and equipment, subcontracts and overhead (the fee of the evaluator would be negotiated with the evaluator based on an hourly or daily charge times the number of anticipated days it would take to complete the evaluation).  I could envision something like this occurring only if the Calvert School decided to conduct an evaluation to research the feasibility of adapting its basic (home schooling) program into an ESL (English as a Second Language) program worldwide. Since this is not something I myself would do I'll let the Calvert school and their bidding evaluators come up with the estimates if that is something they would want to do (they would need to determine to what extent -- how many schools, how many students, and how much time -- the evaluation would be conducted) though they first would need to create additional texts and lesson plans intended for the purpose of  teaching ESL which itself would cost more than an evaluation of those texts. By the way, the text and school supplies in a package for one student for the Calvert School is about $550 for first grade up to about $670 for eighth grade per year, so for a foreign country then someone would add shipping and customs charge and then someone could add the extra charge for a school that is using the program in their curriculum.   Concerning estimates, I think estimates are just that, that is, they are not actual prices, they are negotiating tools so someone usually names a higher price and then bargains downward.  [If an evaluation, or any project, is conducted with volunteer labor and donated products and services, then the cost could be zero, but if everything is charged with inflated prices, adding together everything anyone can think of in a large scale project,  then it could cost, like it can with some military evaluations or studies, millions of dollars.]
----
U6
Activity 6.1 
Read pages 342 - 363, and pages 371 - 388 in the Worthen, Sanders, and Fitzpatrick textbook. 
Activity 6.2 
Read "Using Focus Groups for Evaluation <http://ag.arizona.edu/fcr/fs/cyfar/focus.htm>," by Mary Marczak and Meg Sewell. 
http://ag.arizona.edu/fcs/cyfernet/cyfar/focus.htm
Activity 6.3 
E-mail the first draft (approximately three to five pages) of the Evaluation Plan and Procedures section of your course evaluation project to your instructor. This section presents the management plan (with appropriate edits from the last unit's post), copies of instrumentation (protocols for interviews, or surveys, etc.), and a description of the participants. The instructor will provide comments for revision and inclusion in the course final project. (T 4a, L 6e)
--
U6D1 response to Bridgette
The performing arts themselves are all about qualitative evaluation and data concerning the emotions and perceptions, comedy and tragedy, motion and contemplation, etc.  The quantitative side involves the number of participants, performers, audience members, etc. Also, the economics of the arts would involve quantitative factors  -- Michelangelo could not have done what he did without the patronage of the de Medicis (or someone like them).
U6D1
Do you think that the audience for your evaluation will place more value on qualitative data or on quantitative data? Which of these seems most valuable to you? 

Predominantly, more informational value would be placed on qualitative data, for the audience of the evaluation and for me, since even much of the quantitative data, such as test scores, could be determined by qualitative assessment.  For example, evaluating the quality of a written response takes on a qualitative dimension yet the assessment can be assigned a numerical score.  When all of the students are given norm-referenced tests with closed-ended questions or items such as multiple choice or true/false selections then the result given as a numerical score would have a more quantitative or objective dimension.  However, tests given with open-ended questions or items such as sentence contructions or essay questions would have more qualitatively, or more subjectively, assessed results even though they are also assigned a numerical score.
References:  
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications
--
U6D2 response to Dar
I think your statement that a program is not a "fixed machine"" and "its nature "varies and . . . is caused by many factors"  reflects the current relativistic thinking of the universe a organic whole rather the Newtonian mechanical and clocklike model.
U6D2
What are the advantages to using mixed methods for your research project? What questions are best answered by qualitative methods? What questions are best answered by quantitative methods? 
The formative evaluation, which will use a utilization-focused design, will involve mostly participatory qualitative analysis with me as an internal evaluator.  Qualitative methods will be used on a case studies of ESL classes and anonymous case studies of the student/stakeholder participants in the evaluation process.
Most of the research methods (tests, surveys, and interviews) will involve qualitative elements such as the issues listed below:
		1  .What the participants like about the Calvert program.
		2.  What the participants do not like about the Calvert program.
		3.  How the participants think that it could be improved:  certain subjects and activities that they would like to have added, extended, or eliminated.
		4.  The objectives or philosophy of the program which is to teach the subjects offered while teaching English as a second language in an interesting way.
		5.  How the model of the program is intended to work by teaching English with a total immersion technique using useful and interesting information.
		6.  Assessing if the model of the program does accomplish its intentions.
		7.  Concerns do you have about the program.  Different subjects are presented at different levels and the material is presented from an American-centric cultural view.
		8.  What is anticipated to be learned from the evaluation which is hopefully that the program is effective and how it might be improved.
		9.  Why these issues are considered important.   To be sure this is an effective program and a worthwhile investment for the school and students.
		10. How the information provided by the evaluation can be used, such as suggestions for improvement in the program's implementation, to the American English School and the Calvert School.
		11. Criteria for judging the program.  (Are the students improving in their reading, writing, speaking, listening skills and retaining grammar and vocabulary information?)
		12. Lesson plan assessment to determine the program's effectiveness for teaching English as a second language.
		13. Program documents which are primarily the texts and study plans.
		14. Interviews with stakeholders which include students, teachers, staff, and parents.
		15. Surveys  written and telephone surveys:  
		16. External analysis including review of other summative evaluations of the Calvert School program and examination of existing records and documents.
		17. Qualitative analysis of data including test results, survey, interview, and questionnaire results
Issues that involve quantitative elements are listed below:
		1.  Collection of data
		2.  Numerical test results
		3.  Number of students enrolled in the program
		4.  Costs involved with the program 
Qualitative research: "Inquiry that is grounded in the assumption that individuals construct social reality in the form of meanings and interpretations, and that these constructions tend to be transitory and situational.  The dominant methodology is to discover these meanings and interpretations by subjecting the resulting data to analytic induction." (Gall, 2003).  Qualitative methodology involves variables, selective sampling of smaller populations and qualitative analysis and is interpretive, transitory, naturalistic, subjective, and localized.
Quantitative research:  "Inquiry that is grounded in the assumption that features of the social environment constitute an objective reality that is relatively constant across time and settings.  The dominant methodology is to describe and explain features of this reality by collecting numerical data on observable behaviors of samples and by subjecting these data to statistical analysis."  (Gall, 2003).  Quantitative methodology involves constants, random samplings of larger populations, and quantitative analysis and is statistical, stationary, analytical, objective, and universal.
How does mixing the methods add to the evaluation? How might it detract?
Mixing methods adds to the spontaneity required for such an evaluation and the qualitative assessment necessary in determining language acquisition skills, however, too many methods could suggest a lack of a plan or system so it is best to use methods that are harmonious and as simple as possible to execute.
References:  
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Gall, M.D., Borg, W., & Gall, J.P. (2003). Educational Research: An Introduction. 7th Ed. Boston: Pearson Education, Inc.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications
***
Calvert External Evaluation: from: http://www.aypf.org/rmaa/pdfs/Calvert.pdf
Raising Minority Academic Achievement
Calvert
The report evaluates the Calvert program after it
was implemented at Dr. Carter Goodwin Woodson
Elementary School (Woodson Elementary). Calvert
is a private elementary school with a long history of
providing a high-quality education to several
generations of children from many of Baltimore’s
most affluent families. Woodson Elementary is a
public school located in a predominantly African
American community, and more than 90% of its
students are eligible for free or reduced-price
lunch. Calvert’s philosophy of education includes
high expectations, time-on-task, rapid pace of
instruction, frequent evaluations, immediate
feedback and student accuracy. The students are
required to learn with attention to detail, including
correct spelling and punctuation. Each month,
parents receive report cards and representative
samples of students’ academic work. Calvert
produces a “book” of each student’s nine monthly
folders of work and presents the book to the student
at the end of each year.
[   POPULATION
During the 1996-97 school year, 90% of the
students attending Calvert were white, 6% were
African American and 4% were Asian or
Latino. One hundred percent of Woodson’s
400 students in grades K-5 were African
American. The tuition at Calvert was $9,000
per year. The percentage of Woodson
students (90%) eligible for free or reducedprice
lunch was nearly triple the 1996 Maryland
state average (31.1%) and well above the
Baltimore City average of 70.4%.]

“Implementing a Highly Specialized,
Curricular, Instructional, and Organizational
School Design in a High-Poverty, Urban
Elementary School: Three-Year Results”
(July 1998) Johns Hopkins University. By Barbara
McHugh and Sam Stringfield.
Focus
Early Childhood
>Primary School
Middle School
Secondary School
Postsecondary
Extended Learning

Evaluators used the Comprehensive Testing
Program III to assess the impact of the program.
They compared average percentile scores of first
and second graders at Woodson prior to the
implementation of the program (the “comparison
group” for this study) with scores of the three
cohorts of first graders who were taught under the
program during school years 1994-95 to 1996-97
(see table).
In first grade reading comprehension, the
average score for the comparison group was at
the 18th percentile. After one year in the
program, the first cohort of students scored on
average at the 49th percentile, the second cohort
scored at the 40th percentile and the third cohort
scored at the 49th percentile. The program
effect size was calculated in +2.8, +2.1 and
+2.9 respectively.
In terms of first graders reading at the lowest
levels, 72% of the comparison group scored in
the lowest quartile, compared to 16% of the
first cohort, 35% of the second cohort and 6%
of the third cohort.

Comparison Groups and Cohorts

School Year   Pre-Calvert (Comparison Group)*     1st cohort     2nd cohort     3rd cohort
1993-1994              1st grade (tested for baseline)
1994-1995              2nd grade (tested for baseline)                 1st grade
1995-1996                                                                             2nd grade       1st grade
1996-1997				              3rd grade        2nd grade         1st grade

* As the baseline comparison group, this cohort (of 18 students) was not exposed to the program.
See Study Methodology for further clarification.

_ In terms of first graders reading at the highest
levels, no student in the comparison group
scored in the third and highest quartiles. In the
first cohort, 47% scored in the two highest
quartiles, 24% did so in the second cohort, and
42% did so in the third.
_ Reading gains continued in the second grade,
with 44% of the first cohort scoring in the two
highest quartiles and 72% of the second cohort.
Only 6% of second graders in the comparison
group scored at the third quartile (none at the
highest).
_ For writing, the comparison group scored on
average at the 36th percentile, while the first cohort
scored on average at the 71st percentile and the
second cohort at the 67th percentile. The third
cohort did not take the test that was administered
only to second graders. The effect sizes of the
program were +2.7 and +2.4.
_ For mathematics, 89% of the comparison group
scored in the two lowest quartiles, 11% in the
third quartile and none in the highest quartile.
For the first cohort, 22% scored in the second
lowest quartile (none in the lowest) and 78% in
the two highest quartiles. For the third cohort,
24% scored in the two lowest quartiles and 76%
in the two highest.
Note: In the Maryland State tests (MSAP) done in
spring of 1997, Woodson third graders scored
significantly above the 1996 Woodson third graders
(pre-Calvert), but still below Maryland statewide
average. Seventy-percent of the group taking the
test belonged to the first Calvert cohort while 30%
were new arrivals. Results for the past two school
years show a steady improvement in test scores,
although the school has yet to reach satisfactory
status (70% of the students passing) in any of the
subjects.

“The clearest conclusion that can be drawn
from Woodson Elementary is that the Calvert
curricular and instructional program, when
implemented with determination and drive,
can make a dramatic difference in the
educational lives of young, urban children.”
-Barbara McHugh, et al.,
evaluators, Calvert program

Program Components

Woodson Elementary School has about 400 students
in grades K-5. At the time the evaluation was
conducted, the Calvert School model was integrated
into grades 1-3, with grades 4-5 to be added within the
next year. Teachers learned to use the Calvert model
through a two-week training held the summer before
implementation for teachers and other staff, who
learned about weekly homework sheets, monthly
report cards, and other Calvert approaches. K-5
students, in classes of approximately 24 students, each
had one primary teacher who used Calvert approaches
and curriculum in all classes across all subject areas.
Calvert stressed the following approaches to learning
that went across subject areas:

Raising Minority Academic Achievement

“These kindergarten through third-grade
results leave little doubt that impoverished
urban children, given appropriate curriculum
and instruction, are capable of achieving at
levels that are much higher than current urban
averages.”
-Barbara McHugh, et al.,
evaluators, Calvert program

_ Each school day began with a 30-minute
“correction period” for students to correct
previous work, complete unfinished work,
perfect folder papers, read independently or do
other instructionally related tasks.
_ Getting meaning out of reading was stressed in
early grades. Students were taught to read for a
specific purpose, and there was also time during
each school day to read for enjoyment.
_ Sight words and phonemic skills were a
formal part of the Calvert curriculum, as were
timed fact drills on basic mathematics facts.
_ Beginning in January of first grade, all
students wrote a composition each week.
_ Teachers coordinated students’ compilations
of “error-free” papers for insertion into
students’ monthly folders. The folders were
sent home at the end of each month and were
part of school-parent communications.
School-parent interactions were both formal and
informal. All parents received folders of student
work at the end of each month. Some parents
and grandparents, mainly in first grade, helped
out during the corrections period. Additional
activities such as a trip to the movies, bowling
alley or skating rink, were scheduled periodically
for students with perfect attendance. The school
also made daily announcements of which classes
had perfect attendance on the previous day.
After the Calvert School agreed to share its model
with Woodson, the Abell Foundation financed the
implementation, including funds to pay teachers
or other staff from Calvert who trained Woodson
staff. Besides paying for staff costs, Calvert did
not charge a “usage fee” for its model. After
providing the curriculum and initial training,
Calvert staff were available on an informal
consultative basis, though their formal
involvement in training ended. Woodson shared
its evaluation information and reports with
Calvert. The Abell Foundation also reviewed
evaluations and student progress reports, though
the foundation was not directly involved in
implementation of the model.

Contributing Factors

Gradual Implementation/Faithful Replication
Woodson adopted the Calvert model grade by
grade, allowing full implementation in one grade
before moving on to another. All teachers were pretrained
and a full-time facilitator (funded by the
Abell Foundation) was onsite throughout the
implementation. For the most part, Woodson
teachers seemed to faithfully replicate the Calvert
model with few exceptions.
High Expectations
The Calvert model was built on high expectations
combined with a high degree of structure. The
curriculum centered around a rapid pace of
instruction and student accuracy - including correct
spelling and punctuation - was considered
fundamental. Timed drills - particularly in math -
were used on nearly a daily basis.
Frequent Evaluations/Immediate Feedback
The Calvert program not only gave immediate
feedback to students through teacher commentary
and grading but also shared frequent evaluations
with parents and school administrators regarding
overall student performance. Parents received
monthly report cards accompanied by representative
samples of a student’s work. In addition, the fulltime
facilitator provided constant feedback to staff
during the implementation process.

Focus on Results
The Calvert model was a results-oriented one.
Student attendance, work quality and performance
on national tests were regularly monitored and
evaluated. Students were consistently required to
correct work until it was error free. Even students
in upper grades were given weekly spelling tests.
Professional Development
In addition to the two-week training and support
from the full-time facilitator, Woodson teachers also
participated in school-wide seminars in which
teachers exchanged ideas and discussed problems.
Woodson teachers also reviewed lessons on their
own time through Calvert’s home-schooling
curriculum. Teacher input was used to decide which
textbooks to purchase in order to increase
implementation success.
Communication with Families
In addition to monthly report cards, parents and
grandparents also participated in monthly parent’s
meetings. Parents and grandparents were asked to
volunteer to be on site in the classroom helping
students complete or correct work. Parents and
grandparents also helped arrange classrooms,
participated in recreational activities and listened to
students read.

STUDY METHODOLOGY
The school implemented the Calvert program
gradually, starting with kindergarten and first
grade, and adding another grade every year.
The report focuses on the third year of the
program implementation. Data is given per
cohort. The comparison group started first grade
in September 1993 before the program was
implemented (18 students). The first cohort
started first grade in September 1994, when the
program was implemented (32 students). The
second cohort started first grade in September
1995 (29 students), and the third cohort started
first grade in September 1996 (50 students).
There was no attrition of these cohorts. All
students were tested on the Comprehensive
Testing Program III, a norm-referenced test used
in private schools. Their scores, given in Normal
Curve Equivalent (NCE), were compared to those
of students who were in first grade prior to the
implementation of the program. Results of the
analyses were then converted to percentiles.
Effect sizes were calculated as cohort mean NCE
minus comparison mean NCE divided by
comparison standard deviation.
EVALUATION & PROGRAM FUNDING
The evaluation was funded by Johns Hopkins
University. Implementation of the Calvert program
at Woodson was funded by the Abell Foundation.
Before funding implementation at Woodson, The
Abell Foundation also funded implementation of
the Calvert program at another public Baltimore
school, Barclay Elementary and Middle School.
GEOGRAPHIC AREAS
Calvert and Woodson are located in Baltimore.
The program has also been implemented at
Barclay Elementary and Middle School. Some
16,000 children worldwide are home-schooled
using the Calvert program.

CONTACT INFORMATION

Research Contacts
Sam Stringfield, Professor
Center for Social Organization of Schools
Krieger School of Arts & Sciences
Johns Hopkins University
3003 N Charles St, Suite 200
Baltimore, MD 21218
Phone: 410.516.8834
Fax: 410.516.8890
sstringfield@csos.jhu.edu

Program Contacts
Merrill Hall, Headmaster
Calvert School
105 Tuscany Rd,
Baltimore, MD 21210
Phone: 410.243.6030

Johnetta Neal, Principal
Woodson Elementary School
2501 Seabury Rd.
Baltimore, MD
Phone: 410.396.1366
Fax: 410.396.3062
jneal@bcps.k12.md.us

--
from:  http://www.netc.com/~gok/faq.html

Calvert School 105 Tuscanny Road Baltimore, MD 21210 (410) 243-6030 Tuition about $400. You receive an instruction manual as well as books, workbooks and writing materials (crayons, pencils, rulers, paper). Lessons are spelled out. (ex. Say to the child...) 
---
To Create a Graph:
Do one of the following: 
•	To create a graph using the current settings, select a graph tool . 
•	To set graph options before creating the graph, double-click a graph tool. In the Graph Type dialog box, select options and click OK. (See Selecting graph options <1_19_5_0.html>.)
Create the graph: 
•	Drag diagonally from the corner where you want the graph to begin to the opposite corner. Hold down Alt (Windows) or Option (Mac OS) to drag from the center of the graph. 
•	Click the graph tool in the artwork where you want to create the graph, and then specify the dimensions for the graph in the Graph dialog box. Hold down Alt (Windows) or Option (Mac OS) to specify a point as the center of the graph. By default, the Graph dialog box displays the dimensions of the most recently drawn graph. Dimensions are displayed in the units of measure specified in the Edit > Preferences > Units & Undo dialog box (Windows and Mac OS 9) or Illustrator > Preferences > Units & Undo dialog box (Mac OS X).
To enter graph data:
Enter the graph data. (See Entering graph data <1_19_3_1.html>.)
Note the following about graph data:
•	You must select an entire graph before you can change its data. 
•	You can enter data in up to 32,767 rows by 32,767 columns, as memory permits. 
•	Graph data cannot contain nonnumerical characters other than decimal points (.) or decimal commas; if it does, the data is not plotted. For example, you would enter 245000, not 245,000. However, you can enter nonnumerical characters in labels, as described later in this section.
To enter data manually:
Create a graph using the procedure described in Working with graphs <1_19_2_0.html>. Make sure that the graph is selected. 
If the graph data dialog box is not already open, choose Object > Graph > Data.
Graph data dialog box A. Entry line B. Import data C. Transpose row/column D. Switch x/y E. Cell style F. Revert G. Apply H. Row I. Column J. Cell 
At the top of a work sheet is the entry line for entering data. You enter labels and then data that correspond to the labels. Each label or value you enter is placed into a separate cell. A cell is the intersection of a row and a column. 
Labels are words or numbers that describe two things: categories, whose data is entered in rows; and legends, whose data is entered in columns. In the final graph, legends appear as boxes that you can place above or to the right of a graph.
Enter labels and data for the particular graph type, according to the guidelines in Entering labels and data for particular graph types <1_19_3_2.html>. Also see Selecting graph options <1_19_5_0.html>. 
Usually, you enter a label in at least the first row and the first column of the work sheet. Follow these guidelines:
•	To create category labels for the horizontal (x) axis, enter the labels in the first column. (Do not enter a label in the first cell if you are also creating legend names for the vertical (y) axis.) 
•	To create legend labels for the vertical (y) axis, enter the labels in the first row. (Do not enter a legend label in the first cell of the row if you are also creating labels for the horizontal (x) axis.) 
•	To create labels consisting only of numbers, enclose the numbers in straight quotation marks (that is, not "curly" quotation marks). For example, to use the year 1996 as a label, enter "1996"; to use quotation marks around a number label, use text quotation marks within another set of straight quotation marks--for example ""1996"". If a label includes both text and numbers, you do not need to use quotation marks. 
•	To create line breaks in legends or labels, use the vertical bar key to separate lines. For example, enter Total|subscriptions|1996 to produce the following graph label:
Total subscriptions 1996
Continue entering data in the work sheet, choosing one of the following methods to move to another cell: 
•	Press Tab to enter the data and select the next cell in the same row. 
•	Press Enter or Return to enter the data and select the next cell in the same column. 
•	Use the arrow keys to move from cell to cell. 
•	Click any cell.
To correct a mistake or to change how data is entered in the graph data dialog box, do one of the following: 
•	If you accidentally enter graph data backward (that is, in rows instead of columns, or vice versa), click the Transpose button to switch the columns and rows of data. (See Adjusting graphs <1_19_4_0.html>.) 
•	To switch the x and y axes of a scatter graph, click the Switch X/Y button . 
To adjust the cell column width to display more or fewer digits, click the Cell Style button and enter a new value in the Cell Style dialog box. When you finish entering data in the work sheet, do one of the following: 
•	Click the Apply button to keep the graph data dialog box open and apply the data to the graph. 
•	Click the close box to close the graph data dialog box and apply the data to the graph. 
•	Click the Revert button to return data to its state when you last applied it to the graph.
The following graph type options let you choose how to display line graphs: 
•	Mark Data Points places square markers at each data point. 
•	Connect Data Points draws lines that connect the data points and makes it easier to see relationships between data. 
•	Edge-to-Edge Lines draws lines that extend across the graph, from left to right along the horizontal (x) axis. 
•	Draw Filled Lines becomes available when you select Connect Data Points. Draw Filled Lines creates a wider line according to the value you enter in the Line Width text box, and it fills the line with paint according to the specifications for that series of data.
----
Dr. Gerry,
Though it was inoperational for a while, I believe I have gotten my Internet connection back.  Here are a few thoughts that I wanted to mention to you.
I don't have Word but my computer has WordPerfect, PageMaker, Adode Acrobat Reader (can you use any or all of those programs?) and some graphics programs such as Illustrator, Photoshop, and DreamWeaver.  I have WordPad which I think uses some of the Word technology but I can not compose items graphically, such as columns and indentations.  If I save something on WordPad all of the lines align to the left of the screen which is why I sent my papers to you in the body of the email letter.
Thank you for your reply of the evaluation design draft that I sent to you.  From the middle of the paper beginning with "which is the intention of the school"... all the way to the bottom of the page, all of the sentences and lines of words have a line drawn through the middle of them. At first, I thought it meant that you thought these ideas should be omitted from the evaluation design but the lines were also drawn through your blue high lighted comments so I am not sure what the lines through the middle of the words is meant to convey.  From the first drafts, I would add later and remove various items.  Some questions, should I have have anonymous descriptions of the stakeholders (including students)?  Do you think it is a good idea to list the texts used in the Calvert program?

Concerning the budget discussion, "how did you arrive at the costs associated with the evaluation? Have you factored in time as a cost? What other resources will be expended, and how will you determine their cost?"  I answered the questions about costs the way I did since this evaluation is being done as a course fulfillment (as a learning technique for the learners in the course) and is not really being paid for so to my way of thinking there is not an actual budget.  The way I will answer this is "If you were to propose a cost analysis for your evaluation, how would you itemize the costs?" and I will include the budget figures in the management plan.

By the way, I appreciate your comments within the body of the paper highlighted in blue.  Before, most of the instructors just replied to papers with a few comments about an entire paper.  Thank you,  Richard Bloodworth

----
U6D1 response to Barbara, Dr. Gerry

I thought about that very same point and I wasn't quite sure where to draw the line but someone would have to pay for the supplies. If the evaluation is to assess the effectiveness of the program (as used by the students themselves) to determine if the program should be continued to be used then a set a of books and supplies would need to be ordered for each of the students in the class.  If this evaluation were done as an experimental venture commissioned by some organization to determine if the program should be initially used or how it should be furthered developed (or commissioned by the Calvert School to determine its use a method for teaching English as a second language) then the students would not be asked to pay for the set of books used for the evaluation.  If it were to be an evaluation of an already-in-progress class then the books would already be in possession of the students and the school.  In that case, their costs would not need to be included in the costs of evaluating the program so the evaluation cost would be the minimal paper and supplies cost plus the evaluator's fee.  Ordering only one set of Calvert supplies would mean that, each day, the several books, which are hardbound books printed in color, would have to be photocopied and the copies (all of those copies would cost more than the books) distributed to the class each day and that would be a copyright violation.  Also, just ordering one set for the evaluator to take home and evaluate alone would not determine the effectiveness of the program as it is actually used by the students.  If that were the case the evaluation cost would be the evaluator's fee plus the cost of one set of books and supplies.  It really all depends on who commissions the evaluation (no one has commissioned this evaluation yet) and for what purpose.   (For this particular evaluation for the ED814 course, I suggested the evaluator's fee as $4000, do you think they are going to pay me that amount as the evaluator?) 

An example of an organization buying the product to test it is the method used by Consumer Reports Magazine.  They always buy the product(s) to be tested and evaluated at a retail store or outlet so, in that case, the cost of the product would have to be included in the cost of the evaluation.                    Richard

U6D1
Barbara:  For an evaluation of this kind (educational), where the evaluator jumps in at some point during an on-going program that has already been established and is currently in use, the main cost would be the evaluator's fee which was estimated to be $4000 for this particular proposal.  For a new or under-consideration program that is not yet in place then I think the situation would require another cost estimate proposal.  Richard 

--- 

U6 Evaluation Plan

Week of February 2, 2004 --    examine some summative Calvert evaluations
Week of February 9 --              conduct telephone surveys, questionnaires, and interviews
Week of February 16 --           Qualitative analyses of above surveys, questionaires,and interviews
Week of February 23 --          Assess, graph, and chart test results 
Week of  March 1 --               Quantitative and qualitative analyses of test results
Weeks of March 8 -- 15         Compose evaluation into a final form
This will be an participant-oriented evaluation primarily with a formative, qualitative approach for a utilization-focused design using mostly qualitative analysis.  Also, attention will be paid to conducting the evaluation in a way respectful to the Taiwanese culture and customs and familiarity of the Chinese culture, its history, its philosophy and its adherence to the Confucian hierarchal structure.
The classes being evaluated are comprised of students who are between 8 and 9 years of age.  In their elementary schools in Taiwan, they would be in the third or fourth grades but the texts that they are using are intended for native English speaking first graders so that some of the subjects such as mathematics, science, history, and geography may already be somewhat familiar to them but they are being taught within the context of a total immersion English language environment.
The Calvert School
		In this formative and participatory evaluation, the effectiveness of the Calvert School program, originally a private school originating over 100 years ago in Maryland, then later also a program for home schooling, and now a curriculum used by many schools within the educational institutions, is examined.  The evaluation is being conducted using classes at the American English School, which is a privately owned high school preparatory school (with the purpose of ultimately being a college or university preparatory school)  in San Chung City (which is adjacent to Taipei), Taiwan.  The textbooks that were developed for home schooling using the Calvert System are also being used for the purposes of educating the Taiwanese students primarily with the objective of teaching them English as a second language with a total immersion technique. The Calvert system begins with pre-kindergarten and moves up through eighth grade with additional advanced enrichments. Among other subjects, mathematics, writing, literature, and science are presented through a variety of colorful books that are all integrated into the Calvert system with suggested lesson plans that, if completed, would cover all of the material in the provided texts. The program is being evaluated for its effectiveness in educating the children in the subjects offered (reading, writing, mathematics, science, geography, history, phonics, vocabulary building, language acquistion) but especially with an emphasis on its effectiveness as a method of teaching English as a second language to young students, in this particular case, from Taiwan.  The teachers like using this method because it covers not only grammar, spelling, and vocabulary but it also introduces these skills within the context of the above-mentioned subjects thus making the teaching, and hopefully the learning, of the information more enjoyable, fulfilling, useful, and effective.  The Calvert program ranges from Pre-Kindergarten up through the eighth grade but this study will concentrate on its effectiveness on mostly the first and second grade levels.

	The Calvert Program is an extensive program covering all subjects for students being educated at home or by schools using the Calvert system.  The program is comprised of a Lesson Manual accompanied by many supplementary books, all in colorful format and design to hold the interest of the young students.  The books for Grade One are listed later in this evaluation.  The program also includes in the educational package/kit sent to the home student or schools as one package for each students containing the many books (more than 20) and supplies including ruler, compass, Cousenaire rods, pens, pencils, art supplies, paint, crayons, erasers, glue, scissors, sharpeners, and even an inflatable Earth globe the size of a beach ball.  The students follow lessons, and if they spend one day on each lesson, each level lasts one school year.  
Texts in the First grade program:
Billings, H. (2000). Maps, Globes, Graphs. Austin, TX: Steck-Vaughn Co./Harcourt Brace
Calvert School.  (2003).  Activity Pages.  Baltimore: Calvert School, Inc
Calvert School.  (2003).  History Read-Alouds. Baltimore: Calvert School, Inc.
Calvert School. (2003).  Lesson Manual.  Baltimore: Calvert School, Inc.
Calvert School.  (1976).  98 Stories for Children.  Baltimore: Calvert School, Inc.
Calvert School.  (2003).  Reading Work Cards. Baltimore: Calvert School, Inc.
Calvert School. (2003). Reading Work Pages. Baltimore: Calvert School, Inc.
Calvert School.  (2001). Stories and Longer Verses. Baltimore: Calvert School, Inc.
Calvert School. (2003). Tests. Baltimore: Calvert School, Inc.
Calvert School.  (1995). Writing Fun.  Baltimore: Calvert School, Inc.
Cooper, J.D. (2001). Here we Go.  Boston, MA: Houghton Mifflin Company.
Cooper, J.D. (2001).Let's Be Friends. Boston, MA: Houghton Mifflin Company.
Cooper, J.D. (2001). Surprises. Boston, MA: Houghton Mifflin Company.
Cooper, J.D. (2001). Treasures.  Boston, MA: Houghton Mifflin Company.
Cooper, J.D. (2001). Wonders.  Boston, MA: Houghton Mifflin Company.
George, J. (1993). Red Robin, Fly Up!  Baltimore: Calvert School, Inc.
Mathematics:
	Buffington, A. (2000).  Calvert Math.  Baltimore: Calvert School, Inc.
	Buffington, A. (2000).  Calvert Math:  Practice and Enrichment Workbook. Baltimore: Calvert School, Inc.
	Calvert School.  (2000).  Math Manual.  Baltimore: Calvert School, Inc.
Moyer, R. (2000). Science. New York: National Geographic/McGraw-Hill Co., Inc.
Phonics Library. (2000). Boston, MA: Houghton Mifflin Company.
The Stakeholders:
Owner:		
		Sabrina   --  owns a preparatory school and, with the profits from that school, opened this school.  Most of the kindergarten and elementary students who graduated from that school transferred and continued in this new school, that she named the American English School which uses a text, called "Honey Bee", of a franchised Taiwanese school for the younger students while the older students are using the Calvert School program which is being evaluated for the purposes of this report.
Teachers:  	
		Richard   --      Began teaching first grade English using the Calvert program (in addition to other classes using another program) in September, 2003.  Has also taught ESL to adults and children using other programs and texts in South Korea, Japan, Czech Republic, Turkey, and China.
		Oliver    --       Began teaching second grade English in July, 2003.  He was born in France then moved to Canada when he was 20 (so he has a very French accent.)  
Staff:		
		Patricia   --    manager, and the school disciplinarian,  a Taiwanese who was once an English teacher herself being a good English speaker who lived a few years in the USA.
		Tiffany   --     staff assistant and at times teaching assistant for the Calvert classes and a sometimes translator for the students
		Other staff and teachers    ---   there are several Taiwanese staff and teachers and some other foreign teachers as well
Students:
All of the students (male and female) are Taiwanese children, between  8 and 9 years old, who speak Mandarin Chinese (though some might also speak the local Taiwanese dialect to their grandparents but this seems to be a fading local language, whereas Mandarin Chinese, mostly as a result of the large population of China, is the most spoken native language in the world with over 1.4 billion speakers which is 3 times more than the number of native English speakers) who are studying English in order to increase their future opportunities so that they may be able to work, study, travel, or live abroad.  Also, to get in to the sort of universities and schools that they want to enter, or that now their parents want them to enter, they need to have a thorough knowledge of the English language.  The names of the students listed below are not their real Chinese names.
First grade:	
		Tina 1  --  she is a well behaved and an interested student and is learning her oral English.
		Peter --  energetic, a good speller and a helpful student and his mother is an interested parent who studies with him.
		Steven --  the best speller and makes the highest scores on the tests but pouts and cries if he does not get his way.
		Jason  -- disruptive but is getting better,  I heard that his father who was a policeman was killed in the line of duty which might help explain his displinary problems, midway during this evaluation he was transferred to a more basic level class.
		Bruce --  energetic and one of the best students (he was Rudolf in the Christmas show) but sometimes hard to control like Peter who is his best friend
		Debbie  -- quiet and her mind wanders off during the class and she often talks to Tina 2 who sits next to her so she does not pay attention
		Kevin  -- just wants to have a good time but is learning some even though he leaves many items on his tests blank.
		Howard  -- is one of the best students as he is personable but has a violent streak that often surfaces and he has a friendly, helpful mother.
		Ken  -- is one of the best students and seems to be the most mature and also physically the largest of the students (he was Santa at the Christmas show)
		Joey --  he is a good student but refuses to do some of the activities and pouts a lot and his father is an interested parent.
		Mark -- is the quietest student who does not join in many activities and does not mix with the other "naughty" boys so is only tolerated by them but he has an involved mother and he does fairly well on the spelling tests.
		Cherry   a good English speaker who once had a private Australian tutor but she is not able to write or spell well at all so because of this, and because she was having difficulty in her other classes in public school,  she discontinued the English classes at this school
		Tina 2 --  she is a fairly well-haved student though she also sometimes has difficulty with comprehension so she does not respond quickly to task assignments, midway during this evaluation she was transferred to a more basic level class.
		Andy -- is the brother of Cherry (their father is a physician) and is a fairly good English speaker.
Parents of the above students.
Most of the research methods (tests, surveys, and interviews) will involve qualitative elements such as the issues listed below:
		1  .What the participants like about the Calvert program.
		2.  What the participants do not like about the Calvert program.
		3.  How the participants think that it could be improved:  certain subjects and activities that they would like to have added, extended, or eliminated.
		4.  The objectives or philosophy of the program which is to teach the subjects offered while teaching English as a second language in an interesting way.
		5.  How the model of the program is intended to work by teaching English with a total immersion technique using useful and interesting information.
		6.  Assessing if the model of the program does accomplish its intentions.
		7.  Concerns about the program.  Different subjects are presented at different levels and the material is presented from an American-centric cultural view.
		8.  What is anticipated to be learned from the evaluation which is hopefully that the program is effective and how it might be improved.
		9.  Why these issues are considered important.   To be sure this is an effective program and a worthwhile investment for the school and students.
		10. How the information provided by the evaluation can be used, such as suggestions for improvement in the program's implementation, to the American English School and the Calvert School.
		11. Criteria for judging the program.  (Are the students improving in their reading, writing, speaking, listening skills and retaining grammar and vocabulary information?)
		12. Lesson plan assessment to determine the program's effectiveness for teaching English as a second language.
		13. Program documents which are primarily the texts and study plans.
		14. Interviews with stakeholders which include students, teachers, staff, and parents.
		15. Surveys  written and telephone surveys:  
		16. External analysis including review of other summative evaluations of the Calvert School program and examination of existing records and documents.
		17. Qualitative and quantitative analysis of data including test results, survey, interview, and questionnaire results

----  Most of the following information will be consolidated during the final three weeks of the writing of the evaluation:

IV. Brief overview of the evaluation plan and procedures
V.  Presentation of evaluation results
	A. Summary of evaluation findings
	B. Interpretation of evaluation findings
VI. Conclusions and Recommendations
	A. Criteria and standards used to judge the Calvert program
	B. Strengths and weaknesses of the object of the evaluation
	C. Recommendations
VII. Minority reports or rejoiners
VIII. Appendixes
	A. Evaluation plan, instruments, data analysis and interpretation
	B. Tabulations, analysis of quantitative data, transcripts, summaries of qualitative data
	C. Bibliography	
	D. Other information

Estimated Costs for the proposed evaluation:

1.  Evaluation staff salary and benefits   Evaluator: per hour $100 x 40 = $4000
2.  Consultants   0
3.  Travel and per diem or allowance (for staff and consultants)    0
4.  Communications (postage, telephone calls, etc.)   $6
5.  Printing and duplication   $20
6.  Data processing    0
7.  Printed matter   each     0
8.  Supplies and equipment:  Paper:  $5. Calvert educational package for each of 14 students (14 x 550 = $7,700) 
9.  Subcontracts                                   0
10. Overhead (facilities, utilities, etc.)     0

Total estimated cost for a two month evaluation:   $11,731.00

References:  
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications.

---

U6 Posting response to Dar

Dar:

According to Activity 6.3 as described in the schedule:
E-mail the first draft of the Evaluation Plan and Procedures section of your course evaluation project to your instructor (or, we posted them). This section presents the management plan (with appropriate edits from the last unit's post), copies of instrumentation (protocols for interviews, or surveys, etc.), and a description of the participants. 
Richard
----
U7D1 response to Dar
Dar, I think your advice to simplify is right on target.  Too often we seem to unnecessarily complicate matters rather than to interpret or analyze a situation.  Even Einstein said that the goal of science was to find simple explanations (and using simplified language) for the complexities of the universe.
Richard
--
U7D1
What procedures will you follow to analyze the data that you have collected? How can you be sure that this analysis will be done correctly? What resources will you seek or call upon for data analysis?
The collection and analysis of data will involve observations of student performance, questionnaires, surveys, telephone and in-person interviews but most of the evaluated data will be in the form of tests.  The tests will involve spelling, vocabulary, sentence construction, and comprehension and will be norm-referenced and domain-referenced.  The tests include writing sentences and brief compositions, oral tests, and some multiple choice questions mostly matching words with their definitions. The tests supplied by the Calvert program will be evaluated according to their answer keys and the teacher created tests will be evaluated according to the teacher's assessment.   Most of the analysis for the evaluation will be qualitative in nature although some quantitative data such as numerical test scores, ages of students, amount of time they have studied English, and attendance will also be included in the evaluation.

Below is a list of some resources that can be referred to for the purposes of data analysis:

Dewey, John. (1997). How We Think. Boston, MA: Dover Publications.
Dillman, Don A. (1999). Mail and Internet Surveys: The Tailored Design Method. New York: John Wiley & Sons, Inc.
Dillman, Don & Salant, Priscilla. How to Conduct Your Own Survey.  New York: John Wiley & Sons, Inc.
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Gall, M.D., Borg, W., & Gall, J.P. (2003). Educational Research: An Introduction. 7th Ed. Boston: Pearson Education, Inc.
Holt, J.  (1983).  How Children Learn.  New York: Perseus Books.
Jameson, Fredric. (1988). The Ideologies of Theory, Volume 2. Minneapolis, MN: The University of Minnesota Press.
Ravitch, Diane and Viteritti, Joseph P. (1997). New Schools for a New Century. London: Yale University Press.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications
Weldon, W. ( 2002). Research seminar: qualitative research. Retrieved Jan 15, 2004, from http://alfa.pedf.cuni.cz/~www_kppg/documents/qrs.rtf
--
U7D1 response to Barbara 
more:  To use a sports analogy, quanititative analysis compares to a timed foot race with a finish line and qualitative analysis compares to a judged gymnastics event.
Barbara,
Sorry for the late reply (I've been catching up on my projects),
You can find a good explanation of the terms of quanititative and qualitative analysis at http://www.ling.lancs.ac.uk/monkey/ihe/linguistics/corpus3/3qual.htm
To show the difference between quanititative and qualitative analysis, here is one example that came to my mind.  When studying biology and the organs of the body, one learns from texts with labeled diagrams and colored, delineated illustrations which compares to an objective, quantitative perception.  When someone dissects an organism, what ones sees inside is a liquid, organic conglomeration of membranes that doesn't resemble the textbook diagram and it can be difficult to tell where membrane begins and another one ends and this would correspond to the subjective, qualitative perception.   One view could be considered as machinelike and the other as organic.
How could one make a quantitative analysis, other than perhaps counting the number of letters in the passage, of the following excerpt by Andre Breton?
The ground beneath my feet is nothing but an enormous unfolded newspaper. Sometimes a photograph comes by; it is a nondescript curiosity, and from the flowers there unifomly rises the smell, the good smell, of printer's ink. I heard it said in my youth that the smell of hot bread is intolerable to sick people, but I repeat that the flowers smell of printer's ink. The trees themselves are only more or less interesting minor news items: a fire here, a derailment there. As for the animals, they have long since withdrawn from the commerce of men; with these latter, women now have only episodic relations, that are like shop windows early in the morning when the head window dresser goes out into the street to see the effect of the waves of ribbon, the slots, the wheedling winks of mannequins.
Or even of this:
To be, or not to be: that is the question: 
Whether 'tis nobler in the mind to suffer
The slings and arrows of outrageous fortune,
Or to take arms against a sea of troubles, 
And by opposing end them? To die: to sleep; 
No more; and by a sleep to say we end
The heart-ache and the thousand natural shocks 
That flesh is heir to, 'tis a consummation 
Devoutly to be wish'd. To die, to sleep;
To sleep: perchance to dream: ay, there's the rub; 
For in that sleep of death what dreams may come
When we have shuffled off this mortal coil, 
Must give us pause: there's the respect
That makes calamity of so long life;
For who would bear the whips and scorns of time,
The oppressor's wrong, the proud man's contumely, 
The pangs of despised love, the law's delay,
The insolence of office and the spurns
That patient merit of the unworthy takes, 
When he himself might his quietus make
With a bare bodkin? who would fardels bear, 
To grunt and sweat under a weary life,
But that the dread of something after death, 
The undiscover'd country from whose bourn 
No traveller returns, puzzles the will
And makes us rather bear those ills we have
Than fly to others that we know not of?
Thus conscience does make cowards of us all; 
And thus the native hue of resolution
Is sicklied o'er with the pale cast of thought, 
And enterprises of great pith and moment
With this regard their currents turn awry,
And lose the name of action.-- Richard
References:
Shakespeare, W. Hamlet. Retrieved on February 20, 2004 from http://www.unibas.ch/shine/hyperhamlet/ham_ref.htm   
Wellman, Donald. (2003). Andre Breton; Surrealism: Soluble Fish. Retrieved on February 20, 2004 from http://faculty.dwc.edu/wellman/Soluble-Fish.html
---
U7D2
What problems have you encountered as you analyze your data (or, do you anticipate encountering)? Do you have any words of wisdom for other learners? What words of wisdom would be helpful to you?
The qualitative and subjective nature of assessing language development is the major problem I anticipate encountering concerning the evaluation process.  Each student can vary in levels of ability in the categories of speaking, listening, reading, writing, comprehension and the development of vocabulary as well as the understanding and use of grammatical rules.  To try to evaluate not only their levels and rates of progress but also to determine to what extent the learning of these skills and materials can be attributed to the effectiveness of the program as offered by the Calvert School curriculum are the most difficult aspects of this evaluation.
References:  
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications.
--
U7D2 response to dd

I think we are all learning as we go and this evaluation project is good practice and a useful exercise in empirical procedures.  By conducting this evaluation we are gaining experience so that we can be prepared for evaluations that we might be asked to do in the future.  Richard
---
Response to Dr. Gerry's email

Dr. Gerry,

In response to some of your questions below:

<Who is your client here? The owner of the school? For your final report, the cost estimates and time lines will be separate documents, perhaps appendicies if you think the client would be interested.  I like your descriptions of these students. It adds a nice qualitative touch and enhances the believability and interest of your evaluation. Will there by any issue of embarrassment for them if they are recognized by the reader? Clearly the wont be recognized by me. You need to examine this list of issues and winnow it down to some specific issues that will be addressed in your evaluation. The first seven appear to be related to evaluation of the program. The remainder are good issues, but they relate more to how the evaluation findings will be used that to the evaluation of the program itself. Once you have the evaluation questions specified, you can then link your methods to the questions, indicating for each evaluation question the method that will be used to address it.>
Here are my responses to the above questions and comments:
As I had mentioned previously, since this is a project for a Capella course fulfillment, I am not exactly sure who the client is.  For the purposes of the written course evaluation I will say the client is Sabrina, the school's owner, though of course she is not paying me to do an evaluation.  Probably if I asked her if she wanted to have an evaluation done, knowing them (the school management) as I do, she would probably say "No, I'd rather you not" or (through her translator) something to the effect of "Go ahead, but don't let it detract from the teaching of the language course and we can not pay you anything for it".  I think the most possible potential client is the Calvert School ( they have not commissioned or requested the evaluation) but it could possibly be presented to them after the self-initiated one site evaluation is done.  They might have some interest in converting some of their texts, or adding new ones, to accommodate ESL students, to teach them English as a foreign language, both within the United States and worldwide.  But again, in the evalutation I will say that Sabrina, the owner, is the client.

I plan on putting the time schedule and costs estimates (with only the evalutor's fee without the cost student's course packages since this will be a formative evaluation of an on-going educational program) in the appendixes.  Also, the descriptions of the participants will be included in an appendix which is for the ED814 course.  Most of the stakeholders or participants would not be able, except perhaps for Patricia, to understand most of it but, as I had mentioned, their real names will not be used and I will use more general descriptions of them for the evaluation.  Probably none of the stakeholders will see the report but the report will be separated into sections so that if, for some reason, they do see the report they will see it without the appendixes which would contain the costs, time schedule, and the stakeholder descriptions.

Thank you for your comments and suggestions and I'll get the questions more specified and the methods more definite.  Thank you,  Richard
--

Dr. Gerry,

This email is a follow-up to the email letter that I sent you yesterday concerning who the client for the evaluation is (a response to your questions: Who is your client here? The owner of the school?).   A representative (chief financial officer) from Honey Bee Publishing was visiting our school today to get feedback and information concerning the school's curriculum and educational materials.  For their franchised chain of schools called American English Schools, they print the textbooks used for the younger students and they purchase and use the Calvert program for the more advanced students.  I told him about the evaluation project and he liked the idea and wanted to see the results when I am done.  I also plan on writing and calling them for input into the program's effectiveness, practicality, and possible improvements for the program.  With this in mind, I will say that the American English School  (the owners of the franchised school system -- rather than Sabrina who is the owner of our one school) is my client for the evaluation project.  Thank you,  Richard
---

U8 Activity from schedule
 
Activity 8.1 
Read Chapter 19 of the Worthen, Sanders, and Fitzpatrick textbook. 
Activity 8.2 
Read "How to Deliver Negative Evaluation Results Constructively <http://www.chebucto.ns.ca/~LilleyS/tips.html> 
," by Susan Lilley. (T 10c, 4b)  :
How to Deliver Negative Evaluation Results Constructively
Ten Tips for Evaluators 
The following pointers are a summary of the suggestions provided by members of Evaltalk, the American Evaluation Association Discussion List in June 2002. These suggestions are based on the assumptions that the primary purpose of evaluation is to improve programs or initiatives, and that people are more open to learning and change when they are not feeling threatened. 
1. Use a participatory approach from the start.
Engage stakeholders in describing program logic, defining evaluation questions, identifying indicators of success and selecting appropriate data collection methods and tools. When these are defined by stakeholders, evaluation results are more likely to be in line with their expectations. 
2. Discuss possible negative results in the early contracting and design stages.
Encourage clients or stakeholders to articulate their concerns and expectations early on about what the evaluation will reveal, and plan with them about how best to handle these results if they do occur. 
3. Inform clients immediately and often - a 'no surprises' approach.
The worst way for people to learn about negative results is in the evaluation report or in a near-final presentation. As soon as any negative results begin to emerge, gently inform the client through a phone call or a meeting. Continue to communicate this both formally and informally as the evaluation progresses. This approach provides time for people to come to grips with negative findings, to decide how to handle them, and to question the methods or data while there is still time to make adjustments. 
4. Build in time for course correction.
Recognize from the start that negative findings may occur, and build time into the evaluation plan for clients to initiate action to address them before the evaluation is complete. The final report can then tell the positive story of how a problem was id entified and has been corrected. 
5. Question the evaluation plan.
In cases where evaluation questions, indicators or data collection tools have been imposed on the program, question whether they are appropriate. If not, develop alternative criteria and tools, and tell both stories: how the imposed methods show no progress but locally relevant methods do. 
6. Emphasize the positives.
Every initiative will have some positive results, even if they are not very relevant to the funders' priorities. Make sure that your evaluation captures all positive outcomes, and highlight these. Begin and end reports and presentations with the positives, sandwiching the negative findings in the middle. 
7. Tell the truth.
Ethically, negative findings must be fully reported. Most of the stakeholders will already be aware of the problems and will appreciate the fact that they have been brought out into the open and can now be addressed. 
8. Present results in terms of lessons learned.
Identify what is working, what might need tweaking, and what needs to go back to the drawing board. 
9. Provide suggestions for addressing deficiencies.
Provide clients with concrete suggestions for addressing the issues, drawing on your own experience and the research literature. Refer to best practices and to how others have successfully handled similar issues. When available, provide contacts who have agreed to speak with them about how they dealt with these issues. 
10. Involve stakeholders in identifying obstacles and ways to overcome them.
There are often many good reasons why work has not been carried out as planned or objectives have not been achieved. Use a participatory process such as a force field analysis to engage stakeholders in identifying what internal and external forces were working against them, and describe these in your report. Involve stakeholders in identifying ways to overcome these hindering forces and to strengthen the forces that support their work
Lilley, Susan. (2002).  The American Evaluation Association Discussion List.  Retrieved on February 26, 2004 from http://www.chebucto.ns.ca/~lilleys/tips.html
--
U8D1 from schedule

Discussion 8.1: 
U8D1 Comment to Tony:
I think I caught all of the other abbreviations but could you please remind me what CLC stands for?
U8D1 Evaluator's responsibility
Once the evaluation report has been written and submitted to the client, what, if any, responsibility does the evaluator have to ensure that his or her recommendations are followed? Please explain your answer.
The evaluator is only preforming an evaluation and whether or not recommendations are executed are up to the stakeholders and the clients.  They may base these decisions on budget requirements or limitations, cultural considerations, or on time limitations.  Also, one should consider whether the report is summative or formative and if the evaluation is done by an internal or external evaluator.  The internal evaluator may feel personally affected and responsible if his recommendations are followed or not followed.  If the evaluator is one of the stakeholders then the responsibility of the evaluator to have his recommendations followed is increased.  Also, if the nature or result of the evaluation affects public safety, such as in health issues or national defense or product safety, etc., then it could be considered the moral duty of the evaluator to press to have the recommended changes implemented.
Richard
References:  
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications.
--
U8D2 comment to dd
I think that sounds like a good idea: "I plan to suggest to the client that periodic focus groups be convened to discuss the progress toward implementing the recommendations and discover any new suggestions for program improvement".   Richard

U8D2 from schedule

Discussion 8.2: 
What other ways could you disseminate your evaluation findings in order to increase their usefulness? What obligation in this regard do you have to client? Participants? Other Stakeholders?
U8D2 Disseminating findings
What other ways could you disseminate your evaluation findings in order to increase their usefulness? 
In addition to printed matter, mailings, advertising media, television, and radio, the use of the Internet to disseminate information or evaluation findings might be one of the most effective methods.  Printed matter still has the advantage of being a medium that everyone can hold in their hands, locate passages together, and read along together (or alone) in order to discuss the various proposals and recommendations within an evaluation.
What obligation in this regard do you have to client? Participants? Other Stakeholders?
The obligations, other than the ethical obligations referred to in the next question, would mostly be determined in the contractual arrangements established at the beginning of the evaluation process.  The evaluator is just commissioned to conduct the evaluation and any publicizing of the findings would be up to the client or commissioning organization, unless otherwise specified in the initial agreement.
What if, during the evaluation, the client asks you to ignore something, to delete reference to some variable or some result, or to change the tone of the findings one way or the other? What are the ethical concerns in such a situation?
There is often a political element involved in many of the evaluations so that the commissioning organization might not want to offend a particular market segment or it might want to win over other market segments.  Sometimes the commissioning organization might require that the reporting of the results involve a pleasant way to dilvulge unpleasant information which is a form of diplomacy  However, sometimes such a process could involve covering up illegalities, etc.  Of course the evaluator should avoid such a situation and perhaps back out of a project if this seems to be the direction that the project is going from the onset.  As I mentioned in the previous posting,  if the nature or result of the evaluation affects public safety, such as in health issues or national defense or product safety, etc., then it could be considered the moral duty of the evaluator to press to have the recommended changes implemented, and the negative findings reported, and not be involved in any sort of cover up.
References:  
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications.
----

U9 Activity
Activity 9.1 
Post your draft and respond to at least one other learner's posted draft privately via e-mail with comments and/or edits. Be sure to copy your instructor on this e-mail. (T 1b) 
Activity 9.2 
Once you have read through the edits of at least one other learner and made any appropriate changes to your own work, submit a draft of your final project to the instructor. The instructor will read through and make suggestions for necessary changes or edits for your final submission of the course project.
--

U9D1 from schedule

Discussion 9.1: 
Post your first draft of your evaluation project in the CourseRoom. 
Respond to at least one other learner's posting with substantive feedback.
U9D1 comment to Ginger:
I enjoyed reading your first draft.  I had a few thoughts about it and some questions I thought I would ask for you to clarify.
Is the acronym CIPP spelled out somewhere in the paper?  I think some of the tables at the end of the paper could use some more labeling and explanation (for example, I'm not sure of the meaning of the "school to work" column in tables C and D)  Are some of the students in table C and D different from those is table A?  Also, I'm not sure of the meaning of the statement "Appendix C and D are from the first and second failure compilation list in order to compare classes needed from semester to semester".  It might be interesting and helpful to compile charts over a several year period to see what sort of, or if, progress is being made.  Also, I think perhaps it would be helpful if your charts could have dates on them to indicate the year of the records.
In the recommendations section you mention providing incentives for the teachers to teach the courses and students to complete them before their senior year so I think it would be useful to list some possible incentives for both the teachers and students.  You could even have a questionnaire for the teachers, and perhaps for the students, to ask them what would provide incentives for them.  For the students, it is always a good idea to present the advantages of their obtaining an education for their future advancement and security.
Just some thoughts for you,
Richard
P.S. to Robert and Dr. Gerry I had just finished writing these comments so I thought I would send them, so Robert you can still send a reply and, Dr. Gerry, I can reply to another posting also.  Richard
---
Barbara,
I enjoyed reading your first draft evaluation. It is very well thought out and presented.  The visuals are nice too.  I am wondering if you could tell me how you made your bar and pie graphs (what software, etc.).  That would be greatly appreciated.
Thank you,
Richard
--

U10D1 from schedule

Discussion 10.1: 
What is the clearest concept that you learned in this course? What concept is still the least clear to you?
--
U10D1 Understanding of course concepts
What is the clearest concept that you learned in this course? 
I think the concepts of summative and formative evaluations and internal and external evaluators were the clearest concepts.  Quantitative (the use of numbers and figures and assessments based on more objective criteria) and qualitative (judgements of quality based on more subjective criteria)  analysis I think were also differentiated even though the definition of the term qualitative seems to be more nebulous.
What concept is still the least clear to you?
Since the content in each can be the same or similar, the differences between formal and informal evaluations seem to be the form in which they are presented (formal or informal) and perhaps the manner in which the information is gathered.
Some of the differences and purposes of objectives or management or consumer or expertise or participant oriented evaluations seem to overlap.
Also somewhat unclear are the differences between multisite, cluster, and metaevaluations.  Richard
--
U10D1 comment to CK
I think another metaphor that can be used is that developing and composing an evaluation is like assembling the pieces of a jigsaw puzzle to create the one total picture.  Of course, there can be several possible final pictures but the pieces of the big picture need to fit together.  Richard
--

U10D2 from schedule

Discussion 10.2: 
One goal of online learning environments is to encourage a community of learners to eventually become a community of practice. What could have been done in this course to further encourage such a community?
U10D2 Community of learners
One goal of online learning environments is to encourage a community of learners to eventually become a community of practice. What could have been done in this course to further encourage such a community
Perhaps there could have been more direct communication between the learners.  One method is the dyad or triad method of communicating on projects.  Another is perhaps to have some sort of chat room environment where ideas could be discussed. After the completion of the course, one way to encourage a community of learners is for the learners to continue to correspond with the classmates, instructors, participants, and stakeholders involved in the course(s). Perhaps there could be a directory of learners, with their specialties and majors listed, who have taken the course.
Also there could be a Capella link that lists evaluation related sites and that contains information related to Capella evaluations (in fact, it would be interesting to read the results of the some of the Capella surveys that we are periodically asked to participate in). 
Additionally, the learners can continue reading the text as I believe the first reading might have involved some scanning of the material since there is so much information in the book but it can be used as a reference book on a continuous basis and can be read again at a more leisurely pace to absorb more of the information within the text.

Richard
***************
This will be an participant-oriented evaluation primarily with a formative, qualitative approach for a utilization-focused design using mostly qualitative analysis.  Also, attention will be paid to conducting the evaluation in a way respectful to the Taiwanese culture and customs and familiarity of the Chinese culture, its history, its philosophy and its adherence to the Confucian hierarchal structure.
The classes being evaluated are comprised of students who are between 8 and 9 years of age.  In their elementary schools in Taiwan, they would be in the third or fourth grades but the texts that they are using are intended for native English speaking first graders so that some of the subjects such as mathematics, science, history, and geography may already be somewhat familiar to them but they are being taught within the context of a total immersion English language environment.
The Calvert School
		In this formative and participatory evaluation, the effectiveness of the Calvert School program, originally a private school originating over 100 years ago in Maryland, then later also a program for home schooling, and now a curriculum used by many schools within the educational institutions, is examined.  The evaluation is being conducted using classes at the American English School, which is a privately owned high school preparatory school (with the purpose of ultimately being a college or university preparatory school)  in San Chung City (which is adjacent to Taipei), Taiwan.  The textbooks that were developed for home schooling using the Calvert System are also being used for the purposes of educating the Taiwanese students primarily with the objective of teaching them English as a second language with a total immersion technique. The Calvert system begins with pre-kindergarten and moves up through eighth grade with additional advanced enrichments. Among other subjects, mathematics, writing, literature, and science are presented through a variety of colorful books that are all integrated into the Calvert system with suggested lesson plans that, if completed, would cover all of the material in the provided texts. The program is being evaluated for its effectiveness in educating the children in the subjects offered (reading, writing, mathematics, science, geography, history, phonics, vocabulary building, language acquistion) but especially with an emphasis on its effectiveness as a method of teaching English as a second language to young students, in this particular case, from Taiwan.  The teachers like using this method because it covers not only grammar, spelling, and vocabulary but it also introduces these skills within the context of the above-mentioned subjects thus making the teaching, and hopefully the learning, of the information more enjoyable, fulfilling, useful, and effective.  The Calvert program ranges from Pre-Kindergarten up through the eighth grade but this study will concentrate on its effectiveness on mostly the first and second grade levels.

	The Calvert Program is an extensive program covering all subjects for students being educated at home or by schools using the Calvert system.  The program is comprised of a Lesson Manual accompanied by many supplementary books, all in colorful format and design to hold the interest of the young students.  The books for Grade One are listed later in this evaluation.  The program also includes in the educational package/kit sent to the home student or schools as one package for each students containing the many books (more than 20) and supplies including ruler, compass, Cousenaire rods, pens, pencils, art supplies, paint, crayons, erasers, glue, scissors, sharpeners, and even an inflatable Earth globe the size of a beach ball.  The students follow lessons, and if they spend one day on each lesson, each level lasts one school year.  
Most of the research methods (tests, surveys, and open-ended paper as well as telephone interviews) will involve qualitative elements such as the issues listed below:
Initially open-ended items were distribued to the students and some of the teachers asking the following question:
		1  .What the participants like or about the Calvert program.
		2.  What the participants do not like about the Calvert program.
		3.  How the participants think that it could be improved:  certain subjects and activities that they would like to have added, extended, or eliminated.

1.  In response to question # 1 with the open-ended written survey, students answered: mathematics (3), science (2), pictures (2),  and some answered: games (2),  and playing ball (2) (some even said dogs (1) and babies (1)).   During the telephone interviews the students answered that they like: Reading Work Pages (4),  science (3),  math (2), and writing English (1).

The teachers liked the presentation of mathematics, science, reading, and history and liked the fact that a diverse and full range of subjects are offered and not only language and grammar.  Also, there is much vocabulary that is not found in the traditional ESL curricula and the texts show good critical thinking.  The material is well presented and books focus appropriately more on some topics and less on others.

2.  In response to question # 2 , with the open ended written survey, the students said they do not like math (3), science (2), frogs (2), Reading Work Pages (1), reading (1), tests (1), dogs (1), cats (1), and mice (1).  During the telephone interview, students answered that they do not like math (4), science (3), Reading Work Pages (1),  playing "Teacher Says".

Objections from the teachers include: This program was developed with the native speaking English student in mind and not for ESL students.  The books contain uneven student levels, i.e. the math is easy and the history is difficult, etc.  It is difficult for the Chinese native teachers to help the students with some of the subject matter.  The material was developed for individual home schooled students and not for classes composed of groups of children.  There is a lack of white (or black) board material.  Also, the texts are USA centered, so that a child from another country reads about US history and culture. Maps, geography, history, science, poetry, literature are all presented with the USA in mind.

3.  In response to # 3,  in the open-ended survery, the changes or additions the students want are computer related studies (10), use of television (1), more science (1), drawing in class (1).  During the telephone interview the requests were more computer use (7), reading (1), drawing (1), and speaking English to Americans (1).

The teachers thought that a chronological and an alphabetical vocabulary list could be included.  Also the mathematics could be at a higher level.

		4.  The objectives or philosophy of the program:
			 	To teach the subjects offered while teaching English as a second language in an interesting way.
		5.  How the model of the program is intended to work
				To teach English with a total immersion technique using useful and interesting information.
		6.  Assessing if the model of the program does accomplish its intentions.
				This will be based on analysis of the students test scores, speaking, and comprehension progress.  The assessment refers to surveys, comments, quantitative test results, and qualitative evaluation of the students' levels relative to the beginning of the course and progressing for a six month period from October, 2003 to March, 2004.  The evaluation itself took place during a two month period from January to the first of March, 2004 but the documents and test results began in October, 2003.
		7.  Concerns about the program:  
				Different subjects are presented at different levels and 
				The material is presented from an American-centric cultural view.
				The set of books is intended for students who use English as their native language so the program seems to make linguistic and 						phonic assumptions based on native English speakers.
		8.  What is anticipated to be learned from the evaluation which is hopefully that the program is effective for students who do not use English as their native language and ideas for how the program might be improved.
		9.  Why these issues are considered important:   
				To be sure this is an effective program and a worthwhile investment for the school and students.
		10. How the information provided by the evaluation can be used:
				This information can be used by the American English School and by the Calvert School for developing an ESL program.
				These will be provided in the suggestion section
		11. Criteria for judging the effectiveness of the program:
				To determine if the students are improving in their reading, writing, speaking, listening skills and retaining grammar and vocabulary information through the analysis of test results and speaking and comprehension progress.  Tests include daily spelling tests, weekly comprehensive tests, and Unit tests, that also include oral tests, which are paced at every 20 lessons.  The cumulative averages of the above mentioned tests are indicated in the appendix. The collection and analysis of data will involve observations of student performance, questionnaires, surveys, telephone and in-person interviews but most of the evaluated data will be in the form of tests.  The tests involve spelling, vocabulary, sentence construction, and comprehension and are norm-referenced as well as domain-referenced.  The tests include writing sentences and brief compositions, oral tests, and some multiple choice questions mostly matching words with their definitions. The tests supplied by the Calvert program are evaluated according to their answer keys and the teacher created tests are evaluated according to the teacher's assessment.   Most of the analysis for the evaluation will be qualitative in nature although some quantitative data such as numerical test scores, ages of students, amount of time they have studied English, and attendance will also be included in the evaluation. The qualitative and subjective nature of assessing language development is the major problem in conducting the evaluation process.  Each student can vary in levels of ability in the categories of speaking, listening, reading, writing, comprehension and the development of vocabulary as well as the understanding and use of grammatical rules.  To try to evaluate not only their levels and rates of progress but also to determine to what extent the learning of these skills and materials can be attributed to the effectiveness of the program as offered by the Calvert School curriculum are the most difficult aspects of this evaluation.
		12. Lesson plan assessment to determine the program's effectiveness for teaching English as a second language.
This is done by actually using the lesson plans to see how the students respond to them.  Also, each plan is written in the Lesson Manual so that they can be studied and reviewed for analysis in that way.The lesson plans are described explicitly so that all aspects of the lessons are covered.  Also, questions to be asked to the students are answered in the Lesson Manual  Each daily plan is divided into Materials needed, Books for the Lesson, and Student Assignments. An introduction and objectives is given with each lesson plan. The plan progresses toward reading activities, vocabulary building, and phonics.  The daily lessons begin with mathematics, then reading, phonics and spelling. Mathematics is taught with the main mathematics book, Calvert Math, with additional material in the Practice and Enrichment Workbook.   A recess is always placed midway in the lesson.  After the break, the subject matter for the day is presented such as science, geography, Reading Work Pages, history, etc.  The subjects are covered and in science there is the reading text with experiments and activities as well as the Activity Pages for science. In addition to the literature readings, sometimes there are additional stories and verses to read.  Games, activities, and supplementary activities are also suggested.  Each daily lesson is geared around developing reading, writing, speaking, and listening skills as well as vocabulary building and foundations in grammar.  These lessons are planned so that it daily lesson can be completed within a three hour lesson and additional activities can be added to extend the lessons extracurricularly.  The pacing of the lessons is such that all of the material can be covered within the given amount of time so that at the end of the school year all of the contents of the several books will have been covered.  All of these learning activities help the child learn the English language within the context of studying the essential academic subjects.
 Analysis of the program documents which are primarily the texts and study plans.
Calvert School. (2003).  Lesson Manual.  Baltimore: Calvert School, Inc
         This is the comprehensive lesson plan guide that is used as a teacher's guide but is also included in the student's collection of texts and the analysis is described in item 12 above.
Calvert School. (2003).  Lesson Manual.  Baltimore: Calvert School, Inc.Billings, H. (2000). 
Maps, Globes, Graphs. Austin, TX: SteckVaughn Co./Harcourt Brace
           This book gives them first hand experience with reading maps and globes.
Calvert School.  (2003).  Activity Pages.  Baltimore: Calvert School, Inc
These are supplementary activities to accompany the science, geography, and history readings.
Calvert School.  (2003).  History Read-Alouds. Baltimore: Calvert School, Inc.
This is a text of accounts of historical accounts and the reading level of these is much higher than their reading books so that these are meant primarily for their informational content and as listening exercises.
Calvert School.  (1976).  98 Stories for Children.  Baltimore: Calvert School, Inc.
These are reading stories of a higher reading level than those in their reading books.
Calvert School.  (2003).  Reading Work Cards. Baltimore: Calvert School, Inc.
These are cards, printed in book form, that contain the essential vocabulary words that the students are to learn.
Calvert School. (2003). Reading Work Pages. Baltimore: Calvert School, Inc.
These are almost daily  pages, of two or three pages, for the students to interactively fill-in-the-blanks or complete the vocabulary exercises within.
Calvert School.  (2001). Stories and Longer Verses. Baltimore: Calvert School, Inc.
These are also reading of a higher level which also contain some poetry.
Calvert School. (2003). Tests. Baltimore: Calvert School, Inc.
This book contains the unit tests to be given every 20 units which cover in general the material covered in those 20 units.
Calvert School.  (1995). Writing Fun.  Baltimore: Calvert School, Inc.
This is writing exercises for spelling and constructing words but also for penmanship proficiency.
Cooper, J.D. (2001). Here we Go.  Boston, MA: Houghton Mifflin Company.
One of the beginning level readers of stories and poems.
Cooper, J.D. (2001).Let's Be Friends. Boston, MA: Houghton Mifflin Company.
One of the beginning level readers of stories and poems.
Cooper, J.D. (2001). Surprises. Boston, MA: Houghton Mifflin Company.
One of the beginning level readers of stories and poems.
Cooper, J.D. (2001). Treasures.  Boston, MA: Houghton Mifflin Company.
One of the beginning level readers of stories and poems.
Cooper, J.D. (2001). Wonders.  Boston, MA: Houghton Mifflin Company.
One of the beginning level readers of stories and poems.
George, J. (1993). Red Robin, Fly Up!  Baltimore: Calvert School, Inc.
The first book to be read concerning learning, freedom, and growth.
Mathematics:
Buffington, A. (2000).  Calvert Math.  Baltimore: Calvert School, Inc.
Covers counting, reading and writing numbers, money, geometry, fractions, place value, time, measuring, problem solving, sets, adding and subtracting two digit numbers.
Buffington, A. (2000).  Calvert Math:  Practice and Enrichment Workbook.        -- -                       Baltimore: Calvert School, Inc.
		Additional mathematics exercises to complement Calvert Math
Calvert School.  (2000).  Math Manual.  Baltimore: Calvert School, Inc.
This is the daily lesson plan for mathematics with explanations of the concepts involved.
Moyer, R. (2000). Science. New York: National Geographic/McGraw-Hill Co., Inc.
Covers biology:plants, animals, habitats; the sky: weather,planets, Moon, sun, stars, orbital paths, Moon's phases, weather, seasons, space; Matter: comparing, using senses, properties of matter, solids, liquids, gases, physical changes; movement: push, pull, changes, forces, wholes and parts; the human body: growth, skeletons, muscles, skin, weight, height, bones, teeth, health habits.
Phonics Library. (2000). Boston, MA: Houghton Mifflin Company.
These are fairly simple reading selections using the vocabulary and phonics concepts to be learned within a lesson.
14. Interviews with stakeholders which include students, teachers, staff, and parents.
15. Surveys: both written and telephone surveys:  
The results of these are mentioned in items 1, 2, and 3 above as well as in the appendixes at the end of this evaluation.
16. External analysis including review of other summative evaluations of the Calvert School program and examination of existing records and documents.  The Woodson-Calvert evaluation is included in the appendix at the end of this evaluation.
17. Qualitative and quantitative analysis of data including test results, survey, interview, and questionnaire results
The data of this analysis is in the appendix at the end of this evaluation.
Issues that involve quantitative elements are listed below:
		1.  Collection of data
		2.  Numerical test results
		3.  Number of students enrolled in the program
		4.  Costs involved with the program Issues that involve quantitative elements are listed below:

Brief overview of the evaluation plan and procedures

	The effectiveness is assessed by interviews with the students, parents, teachers, and school staff.  Most of the data analysis is a qualitative assessment of the student's performance as indicated by verbal responses and understanding, spelling and writing ability, communication and conversational skills, accumulation of vocabulary, precision and accuracy of pronunciation, and the understanding of grammatical concepts including the understand the grammatical structure of a sentence and identifying and understanding the parts of speech (nouns, verbs, adjectives, adverbs, prepositions, conjunctions, verbals, interjections), and grammatical elements such as auxilary verbs.  Also an understanding of number and tenses (since there are none in Chinese since time is indicated by grammatical context.  For example, in Chinese, the literally translated equivalents of  "I go, I went, and I will go" are "Now I go, Yesterday I go, Tomorrow I go") will be assessed.  Similarly, in Chinese there is no singular or plural as this is also determined by context so that in Chinese the literally translated English of "[one] apple -- [six] apples" is "one apple -- six apple".  Also articles (a, an, the) are not used in Chinese so that often one hears "I want apple" or "Give me banana, please" since articles are low informational content words and are therefore not considered important and a thus among the first words to be omitted or the last to be added to one's conversational speech patterns.

Presentation of evaluation results
	A. Summary of evaluation findings
	B. Interpretation of evaluation findings
VI. Conclusions and Recommendations
	A. Criteria and standards used to judge the Calvert program
	B. Strengths and weaknesses of the object of the evaluation
	C. Recommendations
Suggestions and Recommendations:
For the purposes of using this program as a method for learning English as a second language it is suggested that two vocabulary lists be developed for each a the Calvert yearly levels.  
1.  One in the order of appearance in each daily lesson, with the English word list and beside it its Chinese (or whatever language that is native to where the course is being taught) equivalent.
2.  Another an alphabetized glossary in booklet form of the vocabulary words, with English definitions, of all of the vocabulary words within each yearly level of the Calvert program.
3.  Also, for the ESL learners, there could be a set of tapes and audio supplements to accompany the readings and the texts.

VII. Minority reports or rejoiners
VIII. Appendixes
	A. Evaluation plan, instruments, data analysis and interpretation
	B. Tabulations, analysis of quantitative data, transcripts, summaries of qualitative data
	C. Bibliography	
	D. Other information

Predominantly, more informational value would be placed on qualitative data, for the audience of this evaluation, since even much of the quantitative data, such as test scores, could be determined by qualitative assessment.  For example, evaluating the quality of a written response takes on a qualitative dimension yet the assessment can be assigned a numerical score.  When all of the students are given norm-referenced tests with closed-ended questions or items such as multiple choice or true/false selections then the result given as a numerical score would have a more quantitative or objective dimension.  However, tests given with open-ended questions or items such as sentence contructions or essay questions would have more qualitatively, or more subjectively, assessed results even though they are also assigned a numerical score.
Qualitative research: "Inquiry that is grounded in the assumption that individuals construct social reality in the form of meanings and interpretations, and that these constructions tend to be transitory and situational.  The dominant methodology is to discover these meanings and interpretations by subjecting the resulting data to analytic induction." (Gall, 2003).  Qualitative methodology involves variables, selective sampling of smaller populations and qualitative analysis and is interpretive, transitory, naturalistic, subjective, and localized.
Quantitative research:  "Inquiry that is grounded in the assumption that features of the social environment constitute an objective reality that is relatively constant across time and settings.  The dominant methodology is to describe and explain features of this reality by collecting numerical data on observable behaviors of samples and by subjecting these data to statistical analysis."  (Gall, 2003).  Quantitative methodology involves constants, random samplings of larger populations, and quantitative analysis and is statistical, stationary, analytical, objective, and universal.
What are the advantages to using mixed methods for your research project? What questions are best answered by qualitative methods? What questions are best answered by quantitative methods? 
The formative evaluation, which will use a utilization-focused design, will involve mostly participatory qualitative analysis with me as an internal evaluator.  Qualitative methods will be used on a case studies of ESL classes and anonymous case studies of the student/stakeholder participants in the evaluation process.
Issues that involve quantitative elements are listed below:
		1.  Collection of data
		2.  Numerical test results
		3.  Number of students enrolled in the program
		4.  Costs involved with the program 
---- 
CALVERT EXTERNAL EVALUATION
Calvert-Woodson External Evaluation. (1998).  Retrieved February 5, 2004 from: http://www.aypf.org/rmaa/pdfs/Calvert.pdf
This is a summary of an evaluation of the Calvert program at Dr. Carter Goodwin Woodson Elementary School.  Calvert is an established private elementary school providing a high-quality education to several generations of children from Baltimore. Students learn spelling punctuation and attention to detail. . Each month, the parents receive monthly report cards and samples of students’ work. .

STUDY METHODOLOGY

The school implemented the Calvert program gradually, starting with kindergarten and first grade, and adding another grade every year.  The report focuses on the third year of the program implementation. Data is given per cohort. The comparison group started first grade in September 1993 before the program was implemented (18 students). The first cohort started first grade in September 1994, when the program was implemented (32 students). The second cohort started first grade in September 1995 (29 students), and the third cohort started first grade in September 1996 (50 students). All students were tested on the Comprehensive
Testing Program III, a norm-referenced test used in private schools. Their scores, given in Normal Curve Equivalent (NCE), were compared to those of students who were in first grade prior to the implementation of the program. Results of the analyses were then converted to percentiles.  Effect sizes were calculated as cohort mean NCE minus comparison mean NCE divided by comparison standard deviation.  Around 16,000 children worldwide are home-schooled using the Calvert program. (McHugh and Stringfield, 1998)

Evaluators used the Comprehensive Testing Program III to assess the impact of the program. They compared average percentile scores of first and second graders at Woodson prior to the implementation of the program (the “comparison group” for this study) with scores of the three cohorts of first graders who were taught under the program during school years 1994-95 to 1996-97.  In first grade reading comprehension, the average score for the comparison group was at the 18th percentile. After one year in the program, the first cohort of students scored on average at the 49th percentile, the second cohort scored at the 40th percentile and the third cohort scored at the 49th percentile. The program effect size was calculated in +2.8, +2.1 and +2.9 respectively. In terms of first graders reading at the lowest levels, 72% of the comparison group scored in the lowest quartile, compared to 16% of the first cohort, 35% of the second cohort and 6% of the third cohort.

In terms of first graders reading at the highest levels, no student in the comparison group scored in the third and highest quartiles. In the first cohort, 47% scored in the two highest quartiles, 24% did so in the second cohort, and 42% did so in the third.  Reading gains continued in the second grade, with 44% of the first cohort scoring in the two highest quartiles and 72% of the second cohort. Only 6% of second graders in the comparison group scored at the third quartile (none at the highest).
For writing, the comparison group scored on average at the 36th percentile, while the first cohort scored on average at the 71st percentile and the second cohort at the 67th percentile. The third cohort did not take the test that was administered only to second graders. The effect sizes of the program were +2.7 and +2.4 in the two highest. Seventy-percent of the group taking the test belonged to the first Calvert cohort while 30% were new arrivals. Results for the past two school years show a steady improvement in test scores, although the school has yet to reach satisfactory status (70% of the students passing) in any of the subjects.

Program Components

Woodson Elementary School has about 400 students in grades Kindergarten-5. 

Each school day began with a half hour “correction period” for students to correct previous work and complete unfinished work..
 
Comprehension was stressed in the early grades and students were taught to read for a purpose and for enjoyment.

Students used sight words and phonemic skills as a formal part of the Calvert curriculum.

Sight words and phonemic skills were a part of the Calvert curriculum, as were timed  mathematics drills Students wrote a composition each week beginning in January of the first grade.
 
Folders of students' work were sent home at the end of each month and were part of school to parent communication.
 
Some relatives, mainly in first grade, assisted during the corrections period. Additional reward activities were scheduled periodically for students with perfect attendance. After providing the curriculum and initial training, Calvert staff members were available on a consultative basis. Woodson shared its evaluation information and reports with Calvert. The Abell Foundation also reviewed evaluations and student progress reports.

*************
Open-ended questions asked to the students and teachers in a written survey:
1.   What do you like about the Calvert program?
2.   What do you not like about the Calvert program?
3.   How do you think the Calvert program could be improved?
---
Telephone survey questions to the sudents involved in the Calvert program:
1.   How are you?
2.   What are you doing?   Can you talk now?
3.   What is your favorite color?
4.   What is your hobby?
5.   What school subjects in the Calvert program do you like?
6.   What school subjects in the Calvert program do you not like?
7.   Is this English program easy or difficult?   Why?
8.   What do you most want to learn?

1.    I think English is difficult.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

2.    I like Math.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

3.    I like Science.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

4.    I like to speak English.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

5.    I like to watch movies in English.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

6.    I like Reading Work Pages
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

7.    I like to learn new English words.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

8.    I like to learn new spelling words.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

9.    I like to take spelling tests.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

10.  I like to write sentences in English.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

11.  I like History.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

12.  I like the poems we read.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

13.  I like the pictures in the Calvert books.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

14.  I like to listen to English.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

15.  I like to hear English songs.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true

16.  I can speak English better now because of this Calvert class.
No, not at all.       Not much.        O.K.              Somewhat.                   Yes, very true
---
Suggestions and Recommendations:
For the purposes of using this program as a method for learning English as a second language it is suggested that two vocabulary lists be developed for each a the Calvert yearly levels.  
1.  One in the order of appearance in each daily lesson, with the English word list and beside it its Chinese (or whatever language that is native to where the course is being taught) equivalent.
2.  Another an alphabetized glossary in booklet form of the vocabulary words, with English definitions, of all of the vocabulary words within each yearly level of the Calvert program.
3.  Also, for the ESL learners, there could be a set of tapes and audio supplements to accompany the readings and the texts.
References:  
Fitzpatrick J.L.,  Sanders, J.R., Worthen, B.R.  (2004).  Program Evaluation: Alternative Approaches and Practical Guidelines.  New York: Pearson Educational, Inc.
Sanders, J.R.  (1994).  The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications.

McHugh, Barbara and Stringfiel, Sam.  (1998). Calvert-Woodson External Evaluation.  Retrieved February 5, 2004 from:
http://www.aypf.org/rmaa/pdfs/Calvert.pdf

Contact information for the Calvert School:
Education Services
10713 Gilroy Road, Suite B
Hunt Valley, Maryland 21031
1-888-487-4652
								http://www.calvertschool.org

 The results of the students' progress are reflected in the tables of test scores in the appendix of this evaluation but the progress can also be determined by discussions with the students and listening to their oral performance as they read out loud.  Also, a change in attitude is an indicator, too.  At the beginning of the program, the students were much more unruly, presumably because they were not able to understand what was being said to them.  As time goes on, the general behavior level in the class seems to be improving, again, presumably because they are this time understanding more of what is being said.
Another indicator of development is the performance leve of individual students.  All of them are progressing in their way and the highest performer remains the highest.  But some students are showing marked improvement in their level of interest and performance. An example of this is Kevin, who at first was making zeros on all of his spelling tests but is now known to make a few 100s, accomplishments for which he is justly proud, as a result of his now studying for the tests.

The issue of the level of difficulty is an important one because sometimes the material is quite simple as in the Phonics reading book.  Other times the material can be comparatively difficult as in the history readings and some stories and readings in the Lesson Manual.  One story involved a complicated plot with a rabbit and fox trying to get over a river with an aligator in it by outwitting the aligator and each other.  Another story has a giant who demands a farmer's food and the farmer agrees to give the giant the "tops" while the farmer takes the "bottoms".  As a result of this agreement, the giant gets a bag of wilted leaves, and the farmer gets the potatoes which are the bottoms.  For the next time the giant then demands the "bottoms".  Because of this he gets a bag of roots and the farmer gets the tomatoes which are the "tops".  The giant then demands that the next time he get both "tops" and the "bottoms".  This time the giant gets a bag of leaves/husks and roots and the farmer keeps the ears, or middle, of the corn.  I tend to agree with the giant that the "tops" and "bottoms" would include the entire plant but, however one interprets this story, the meaning of this story is difficult to convey to students learning English as a second language.

Some external summative evaluations have shown that the Calvert program is effective when administered according to the lesson plans.  One such report was the Calvert-Woodson evaluation (McHugh, 1998, Appendix III) which found that the Calvert program as administered to grades 1 through 3 at the Woodson Elementary School, a school with many economically disadvantaged students, showed significant progress in students as compared to the control groups which were not given such a structured program.  After one year the tested first grade students for reading comprehension moved up from an average in the 18th percentile in the comparison group to an average in the 46th percentile.  The first graders who were scoring in the highest.  As a result of the assessment of the results of tests such as the Comprehensive Testing Program III,  reading, writing, and reading comprehension were are examined, and each of these areas showed improved performance.

Overall, it seems that the students are learning English at a rapid pace using the Calvert program.  One may argue that they would also learn at a rapid pace using other programs but the fact that the Calvert curriculum contains a wide range of academic subjects, the instructional process is presented in an interesting and useful context for children to learn English.  Some suggestions and recommendations for adapting the program for use as a method for teaching English as a second language are indicated below. <

Client:  William Hicks, Chief Financial Officer of HoneyBee Publishing, Inc., which is based in Kaoshung, Taiwan and which is the parent company of all of the American English Schools in Taiwan.  HoneyBee Publishing supplies its own printed texts for the youngest stuedents, and for the older students, American English School has the exclusive right to use the Calvert program on the island of Taiwan.
---
Letter to Dr. Gerry

Dr. Gerry,

It sounds like your new situation sounds promising.  We read the book "Transitions" in the first course.  It is a really helpful book about changes in life.

After putting all of the information together, I plan on sending you the 814 Evaluation by this weekend or perhaps by the first of the week but before the last day of the course on March 26, 2004.

Thank you for your insights, help, and feedback during this course,  Richard

Reference:

Bridges, W.  (1980).  Transitions: Making Sense of Life's Changes.  New York: Addison-Wesley Publishing Company.

---

A formative evaluation is one where the primary purpose is to provide information for program improvement.

A summative evaluation is one in which the prime concern is to provide information to form decisions or assist in making judgements about program adoption, continuation, or expansion.

---
Letter to Dr. Gerry   March 25, 2004

Dr. Gerry,

I am enclosing two versions of the evaluation report for you to read.

One version is for you for the course and is called "814 G1 Calvert Evaluation".

The other is intended for the client and is called "Calvert Evaluation".

Again, thank you for your insights and help throughout the course.

Sincerely,  Richard

P.S.  I would appreciate it if you let me know that you have received this email.