2006 Benchmark results
7
pages
English
Documents
Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres
7
pages
English
Documents
Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres
??2006 IAS Benchmark Results Background Stevens Group, Inc. conducted Benchmark Scope a benchmark of their Integrated Assessment System (IAS) The scope of the benchmark was: software product in May 2006. • Stage 1 Independent Review • creation of a Stage 1 Scorebook to be The benchmark was comprised of examiners who submitted to the Baldrige office as part of would be: the pre-work for training sessions in May • attending National Baldrige training sessions. 2006. • using IAS to create their Scorebook for the Note: At the request of the Baldrige office none of 2006 Case study Arroyo Fresco. the responses for Arroyo Fresco were entered in IAS. All the examiners worked off a hard copy of the application. Summary of Participants Each examiner used IAS to develop their individual • Benchmark participants had National Baldrige review of the case study which included the and State examiner experience creation of the following: • Some examiners were first time National examiners, others had 5-10 years experience • Key Factors • Average experience: • Strengths and Opportunities for Improvement National examiners - 4 years • Key Factors linked to each Item and State examiners - 5 years Comment • Total of 13 examiners from California, Iowa, • Scoring North Carolina, Florida and Illinois. • Key Themes • Some of the benchmark participants have • Consolidated Scorebook used IAS in the North Carolina Awards for Excellence (NCAfE) ...
Voir
Publié par
Langue
English
2006 IAS Benchmark Results
Stevens Group, Inc.
www.stevensgrpinc.com
Integrated Assessment System – A new way of doing assessments!
Background
Stevens Group, Inc. conducted
a benchmark of their
Integrated
Assessment
System
(IAS)
software product in May 2006.
The benchmark was comprised of examiners who
would be:
•
attending National Baldrige training sessions.
•
using IAS to create their Scorebook for the
2006 Case study Arroyo Fresco.
Summary of Participants
•
Benchmark participants had National Baldrige
and State examiner experience
•
Some examiners were first time National
examiners, others had 5-10 years experience
•
Average experience:
¾
National examiners - 4 years
¾
State examiners - 5 years
•
Total of 13 examiners from California, Iowa,
North Carolina, Florida and Illinois.
•
Some of the benchmark participants have
used IAS in the North Carolina Awards for
Excellence (NCAfE) program and provided
results beyond Stage 1 (including additional
insights from having applicant data in IAS).
Training & Support
Each examiner received a 1 hour training session
via WebEx which was designed to provide
information on how to get started with creating a
Scorebook in IAS.
Additional information on
specific features of IAS was provided through the
online help system with email support from Stevens
Group, Inc.
Benchmark Scope
The scope of the benchmark was:
•
Stage 1 Independent Review
•
creation of a Stage 1 Scorebook to be
submitted to the Baldrige office as part of
the pre-work for training sessions in May
2006.
Note: At the request of the Baldrige office none of
the responses for Arroyo Fresco were entered in
IAS. All the examiners worked off a hard copy of the
application.
Each examiner used IAS to develop their individual
review of the case study which included the
creation of the following:
•
Key Factors
•
Strengths and Opportunities for Improvement
•
Key Factors linked to each Item and
Comment
•
Scoring
•
Key Themes
•
Consolidated Scorebook
Results
At the end of the benchmark each examiner was
asked to complete a survey on their overall
experience using IAS and on specific features.
The
chart below shows: 69% of the examiners would
highly recommend (++), 23% would recommend (+)
and 8% would slightly recommend (-) IAS.
+
23%
-
8%
++
69%
2006 IAS Benchmark Results
Stevens Group, Inc.
www.stevensgrpinc.com
Integrated Assessment System – A new way of doing assessments!
Results
The chart below shows the average of all the ratings for each survey question.
The scale for the chart is:
•
1 --
double minus – Needs major improvement
•
2 -
single minus – Needs some improvement
•
3 +
single plus – Feature is a strength
•
4 ++
double plus – Feature is strong strength
Rating of IAS Features
1
2
3
4
Install
WebEx Training
Support
Copy and paste Criteria
Copy and paste Responses
Key Factors to an Item
Key Factors to comments
Scoring
Key Themes
Generation of Scorebook
Other Reporting
Comment Writing
Online Help
Related criteria questions
Import and Export of data
Automatic spell checking
Overall ease of use
Recommend
Overall Experience
2006 IAS Benchmark Results
Stevens Group, Inc.
www.stevensgrpinc.com
Integrated Assessment System – A new way of doing assessments!
Time Saved
Each examiner was asked for the percentage of time saved using IAS for Stage 1.
¾
Average time saved in completing the Stage 1 Scorebook – 24%
The results for Stage 2 & 3 are based on feedback from examiners that have used
IAS beyond Stage 1.
¾
Average time saved in Stage 2
– 40%
¾
Average time saved in Stage 3 – 23%
The results for Report Generation are based on the time saved in generating the
Scorebook, Key Themes, Key Factors worksheets, Scoring summary and Feedback
report.
¾
Average time saved generating reports using IAS – 35%
Percentage of Time Saved by Stage
Stage 1
Stage 2
Stage 3
Report
Generation
0
5
10
15
20
25
30
35
40
45
50
1
Willing to Pay
Each examiner was asked if they would pay to use IAS with either their State or
National Baldrige program and if so what amount would they pay.
¾
61% of the examiners would be willing to pay on average $100
for the
use of IAS.
2006 IAS Benchmark Results
Stevens Group, Inc.
www.stevensgrpinc.com
Integrated Assessment System – A new way of doing assessments!
Examiner Feedback
Each examiner was asked to provide feedback on the following:
•
Identify what benefits they experienced using IAS during their case study pre-work
•
The impact of using IAS at the State or National level would have on their decision to volunteer
•
Which IAS features were most helpful to them
•
Overall comments on their experience with IAS
•
Recommended improvements that would IAS more valuable
Below are the quotes from each examiner.
Benefits
•
Less administrative work, more time to work on the content of
comments.
•
Having everything in one place and being able to print out to
proof
•
Faster.
Like having the criteria right above the comments for
continual referencing.
Kept me more on track.
•
The IAS software saved time and allowed me to more fully
comprehend the material. Since this was my first time as an
examiner, the IAS benefit goes way up in subsequent trips back
to the software because of the framework learned at the
National training.
•
Shortened overall assessment time a little
•
Completed work in shorter period of time
•
Great to print out scorebook!!!!
•
Time savings on the case study; even more so when doing
consensus with others; and even more so when the applicant’s
application is in the software.
•
Time savings and not as much material sitting around because
of the Criteria being on the page.
•
Saves time in re-keying information in from the criteria or the
applicant. Know exactly where to go without having to scroll
through the scorebook, sometimes that seemed odd to leave
what you’re doing to see what you wrote elsewhere, but better
than scrolling through the scorebook.
•
The software produces a scorebook in less time with more
accuracy and provides a valuable document for the customer. It
also helps drive the process in stage 2 and 3.
The amount of
time saved in producing the consensus document is incredible.
2006 IAS Benchmark Results
Stevens Group, Inc.
www.stevensgrpinc.com
Integrated Assessment System – A new way of doing assessments!
Volunteer Impact
Examiners were asked if using IAS would have an impact on their volunteering for either
their State or the National Baldrige program.
The most impact was found from those
examiners that have used IAS past Stage 1.
•
In North Carolina I can't imagine having the process without it. Don't KNOW that it
would impact my decision since we are in the application process and still learning.
•
Significant
•
BIGGIE, esp. now with Consensus functionality
•
I would continue regardless..but this would ease the process.
•
From an examiner point of view--its is a significant time saver for the state level especially for the
consensus; don't know that it would make or break my participation--it just helps to facilitate
•
Could have a play in the decision but probably not in the final.
•
Passion and persistence in advancing Baldrige are not related to volunteering, although it definitely
makes it more enjoyable.
Most Helpful Features
•
I think the key factor report is exemplary!
It really integrates the
connection between what is important to the organization and the criteria
comments.
It also helps with the focus on key themes because you can
look over the +/- related to each comment/key factor.
I also think the
assigning of key factors, cross cutting, and ADLI is much more precise
because of the ease of working through each comment.
The scoring is
much more accurate because it gives you the band range and you can
click and see your comments or the comments in a crosscutting category
with speed and ease
•
Cut and paste from the Criteria.
Automated Key Factor linkage, ADLI,
and ++ or cross-cutting.
•
Criteria on the page.
Loved it with the real applicant to be able to cut and
paste. Liked how to link the key factors.
Having the pop up screen for key factors
•
Quick assistance via e-mail.
•
KF linkage, Key Theme draft generation, ability to keep KF as pop up while crafting responses, linkage
to questions
•
Multiple screens and views of windows allowed for ease compared to the "traditional approach".
•
Spell check, viewing of criteria while assessing
•
Criteria readily available, easy selection of Key Factors.
•
Key factors linkage, ADLI linkage, and reporting
•
Significantly cuts down the time of aligning key factors to the comments; formats for you when printing;
spell check; pulling of cross cutting and ++ or -- comments to the key themes worksheets
•
I would love to see this product through Consensus.
That's where I expect the most help!
•
Key factor linkage, Criteria visibility and connections, tabs for strength and OFI, I have not used it but
consolidation of remarks for consensus sounds awesome
•
Cut and paste was a great help!
If there was real applicant content in it, it would have been
tremendous.
2006 IAS Benchmark Results
Stevens Group, Inc.
www.stevensgrpinc.com
Integrated Assessment System – A new way of doing assessments!
Overall Comments
•
This is a tool that will assist examiners in producing a higher quality
scorebook - Key factor and Key theme generation easier, examiner time
focused on quality of evaluation not wrestling with logistics.
Will not
replace examiner gray matter, but certainly organizes the work so that I
can use it!
•
I can see no
dis
advantage to using IAS to enable professional facilitation
of the awesome responsibilities involved in the awards process at state
and national levels.
•
Very helpful with maximizing time spent evaluating.
Thank you.
•
It would be great to incorporate the software into a training module during
new examiner week..some 1/2 or full day training would have been useful
instead of the first day orientation which really did not work until you had
the benefit of the full program.
•
This product does save time in the mechanics of developing a scorebook.
I found it difficult to keep
track of all the comments I was writing, unless I generated a report to review.
•
Testimonials of those with experience in the national process.
Forward thinking for how this could
impact the applicant and the process value if done appropriately.
•
Very helpful, terrific support
•
I also appreciate the support from Stevens Group in answering any questions, comments, or concerns.
They exceeded my expectations and truly provided service that delighted the customer.
Thank you!
Recommended Improvements
Examiners recommended several improvements to IAS that would make it easier
for future users.
Below are the improvements that have ALREADY been
implemented along with those planned for future releases.
1) Implemented Improvements
•
Scoring should be on same sheet as items
•
Counts of strengths and OFIs
•
A count of comments for an entire item (to keep track no matter what area
to address you may be working on).
Scoring screen: ability to see ALL
comments for each item not just what fits on a single page.
•
An option to set up dialog boxes as reminders of the National process. For
example, if you begin to type in a comment before selecting Key Factors for an item, then dialog box
pops up to alert you to that fact.
Another example, is the ADLI dimensions, let's you leave the
comment without identifying those and in some cases I simply forgot and it would have been nice to
have a reminder. If it is made as a preference option then more seasoned examiners who may get
annoyed with it can opt to turn that option off.
•
Reminder to add the ADLI and KF
•
Bigger window to see the full text of selected key factors.
2006 IAS Benchmark Results
Stevens Group, Inc.
www.stevensgrpinc.com
Integrated Assessment System – A new way of doing assessments!
2) Planned Improvements
•
Ability to select and edit Key Factors per item to pinpoint specific KFs.
•
Not having the application on line in the Arroyo Case was definitely a negative for that process.
•
Printed reference materials regarding screen/font size, and linking key factors to individual comments.
•
Difficult to look at comments in other items during assessment, can't make comments that apply to
more than one area at a time, can't easily see other comment for the item
•
Being able to see the full comment in the right side; ability to see both strengths and OFI comments for
the same item at the same time to ensure that they are not in conflict--can only do after printing out.
•
To make the screen larger so you don't have to scroll down to read criteria, comments, and the
application.
It would be nice to be able to pull the screen down like in word.
For additional information on IAS and how it can improve your assessment process
Contact
Stevens Group, Inc. at (919) 471-9029
Email
iasinfo@stevensgrpinc.com
Visit
www.stevensgrpinc.com/iasindex.html
