Category Archives: evaluators convention

Evaluations For Product Development and Funding

This post will focus on formative evaluations because this evaluation is important but often overlooked because of  the methodology  it takes to conduct it. The evaluation is strategic and adheres to principles, formulas, and theories.   However, formulative evaluations are useful in every industry in producing viable products and securing funds.  There are two types of evaluations: summative and formative.  A summative evaluation informs decisions.  In other words, it determines if  the project is a go or not.  A formative evaluation informs revisions.  In other words, it determines what needs to be corrected or added in order for the product ( assessment, project, training, or learning) to meet its objective and stakeholders requirements.  The big question that both evaluations ask is: Did the project, training, or instructions meet its goal? Why or why not.

Evaluations
Evaluations

Most industries including corporate, health, education, religious, and government agencies conduct formative evaluations to produce viable products and secure grants or other funding source.  Education, training and health industries conduct formative evaluations to determine if their instructions, training needs, assessments, and hypothesises are sound. The entertainment industry uses formative evaluations to ensure a project (new innovation, movie, game, concert etc..) is developed to have the impact it needs on its intended audience, as well as meet its stakeholders goals and to secure funding.  It could be costly to develop a great movie, or game to find out the intended audience had no interest in the subject matter.

checklist
checklist

Both evaluations, formative and summative, consist of  the subject matter expert phase (SME), one to one phase, and  the small group phase.   Both evaluations ask questions, collect quanitative and qualitative data using similar methods, analyze data, and report data . Both evaluations can be conducted by internal staff or external consultants. The difference  is what is being done with the data.

Ratings
Ratings

In conclusion,  a summative evaluation determines if a project is a go or not. A formative evaluation provides the benefit of a neutral eye to a project to discover errors, impact, and need during the development stage.  The big question that both evaluations ask is: Did the project, training, or instructions meet its goal? Why or why not. The formative evaluation is useful to all industries in producing viable products and in securing outside funding.  Below is an abstract this author wrote of a  formal evaluation of a custom designed set of instructions that included assessments (test). In addition,  resources are provided for more information on this topic .

                   Evaluation Report of Microenterprise Sustainability  101:  Incorporating       

                                        Strategies  for New and Repeat Evaluation Report

                                            (Evaluation Methodology and Instrumentation) 

Abstract

 This formal evaluation research was conducted to discover problems and issues during development of a custom designed set of instructions in order for the product to meet its instructional and its stakeholders’ goals: Disabled microbusiness owners should be able to increase work efficiency in the areas of time management, customer relationship management, and marketing for sustainability.   It also evaluated the outcome (benefits) for its intended audience, disabled microbusiness owners.  This evaluation is supported with mixed qualitative and quantitative data from all three stages of the formative evaluation plan: subject matter expert (SME), one to one, and the small group phase. Included is a summary of data analysis and recommendations.  Quantitative data consisted of assessment item analysis, objective analysis, item difficulty analysis, item discrimination analysis, qualitative distractor analysis, and internal consistency reliability tests (Split Half Reliability). The qualitative data consisted of construct validity, face validity, interviews, and observations. This evaluation determined that the designed instructions with corrections met stakeholders’ goals, learning, and benefit outcomes  and is ready for distribution.

Keywords:  time management, customer relationships, micro-business, sustainability

end of abstract

 Other Resources:

Formal Evaluation

How to Evaluate Instructions including eLearning

Evaluation Research: Introduction to principles, methods and practices.

AEA (American Evaluator Association) Convention in Atlanta, Ga 2012

As promised, here is my take on the AEA (American Evaluators Association) convention held at the Ravina Hotel in Atlanta, Ga – June 3 through June 6, 2012.  It was wonderful. I don’t know why I was surprised at how many professional top notch organizations were represented.  There was a a large but manageable intellectual crowd in attendance. Atlanta also has a local chapter for AEA. 

First, I want to say thank you for such a well put together convention to Susan Kistler, Executive Director and Heidi Nye, Member Services Director. Heidi put it all together according to her Director, Susan. 
Heidi  is kind and very nice. She gave me permission to take a picture of her for my blog.
Breaks and Lunch came loaded with food. Let’s face it, if the food is no good, its a downer but no disappointments here.  Plenty of nuts, muffins, yogurts, salads, fruits, cheeses, wraps, salmon, chicken, beef,  breads, fancy deserts, coffee, tea, drinks  and bottled water – all included in the price of admission. There were plenty of well dressed employees standing around if anyone needed assistance with anything. The hotel is an older hotel but elegant, clean and provides valet and free parking.  I left my cell phone, the first thing the next morning I was able to pick it up at the front desk. Whew!   Crown Plaza Ravina Hotel on Ashford and Dunnwoody Rd.. 
Here are pictures of the three self serve banquet tables. Guest ate in the dining areas where waiters and waitresses were available. Since my workshop let out 5 min early for lunch I took the picture before a lot of people came into the room.    

I must admit , I did not get there in time (8:15am) to hear the 45 min Keynote presentations due to traffic but I was there by 9 am to attend the workshops I previously chose during registration.  The workshops were three hours long. Everyone attended one in the morning and one in the evening each day except on the last day, we were out by 12 noon.. These were top notch workshops given by top notch people in the field. The workshops I attended were: Handling Data from Logic Model to Final Report, Introduction to Economic Evaluation, Effective Evaluation Reporting Techniques, Getting started: Introduction to Evaluation Consulting and last but not least, Assessment and Action Planning for Sustainability. There  were more that I wanted to take especially the one on Media Evaluating but there’s next year. The workshops were geared toward the health industry but could be applied to all industries.

   My favorite instructor was Dr. Gail V. Barrington. She is owner of The Barrington Research Group in Canada. She has over  25 years of experience.  She is a credentialed Evaluator and a certified Management Consultant as well as a certified teacher. She instructs a Master level course on The Evaluation of Health Services and Systems for the Center for Health Studies at Athabasca University.  I took two workshops from her.  The lady is down to earth, swift and a fancy dresser.

 

Mrs. Barrington is signing her book for me.  “Consulting Start-up and Management.  A guide for Evaluators and Applied Researchers 2012-Sage publishing. It is  also available on Amazon. The book is personable and amazing in details. She shares her invaluable experiences.  If you are interested in being an evaluator, it does not require any particular licenses, you will want to get this book and write your business plan. The book gives you all the information you need to get set up. I loved the “Setting fees” and “How To Get Work Sections.”  Ms. Barrington is married to a retired attorney. She covers “Managing Contracts” and “Managing People” but did not include any sample contracts in her book. She stated that she’s not an attorney but advises that samples of contracts are obtainable through the professional organization by networking with others.”   

                                                    
Dr. Gail V. Barrington signing a copy of her book for me (Angela Clark-owner of this blog): Consulting Start-Up and Management – A Guide for Evaluators and Applied Researchers.

  Book Cover: Consulting Start-Up and Management: A Guide for evaluators and Applied Researchers.


              

Let me add one more thing; even though all the workshops were excellent and on a professional level, I really enjoyed the workshop ” An Executive Summary  is Not Enough: Effective Evaluation Reporting Techniques by Kylie Hutchinson from Vancouver, B.C.  She demonstrated many different ways to report evaluation results outside of the Executive Summary. I won’t go into details, you will have to attend her workshops.  Kylie Hutchinson

 I am posting a comedian video she played that is on You Tube-” Life after Death by PowerPoint” by Millian