What are meaningful metrics of success for e-learning?

Yes, in 2011, I still hear of people and organizations qualifying the success of their e-learning initiatives by citing the number of  “hits” a course receives.   Hits is a complete garbage metric as it only reflects the number of server calls required to load a Web page or course elements within a learner’s Web browser.  For example, if you have 50 elements on page (e.g., images, multimedia, HTML), you will generate 50 hits.  It does not equate to 50 people. Better to use unique visitors or, preferably, registered users to accurately gauge initial reach.

To determine true reach, you need to take into account the # of learners who complete your requirements of your CME workflow (For example: Registration, pre-test, course review, post-test, evaluation, post post-evaluation).  If you get 1000 people to register, but only 30% complete the review of the activity, you may want to look at issues in the quality of the content, instructional design, or the functionality of your learning technology.  The primary reasons that learners drop out of online activities are: poor instructional design technology, unintuitive navigation, or cumbersome registration processes.  If you are going to require registration prior to viewing the course, we suggest that you create a preview of the course (e.g., using YouTube, ScreenFlow) so learners can see the value of completing the registration process to access the complete course.

It is important to recognize that reach does not equate to educational impact, which can only be determined through the design and implementation of valid outcome measurement tools.  At a minimum, we recommend that you leverage either in-house or 3^rd party expertise to assist in the development of such measures that correspond to your learning objectives and have defined endpoints.  These measures should then be implemented within your learning management system assessment and reporting modules.  Please note: Technology should not limit or dictate your ability to conduct outcomes measurement…it should advance it. If your learning management system cannot accommodate your outcomes measurement requirements, see if the software can be modified.  If it cannot, find a different LMS provider.  As with our EthosCE LMS, look for an LMS that supports SCORM 1.2, ACCME and other relevant standards to automate and streamline your ability to share outcomes data with accrediting bodies and education partners.

Using unique visitors/registered users as a metric can tell you a lot your instructional design approach and technology.  However, in the absence of demonstrating any measureable change in behavior, competency, or other metrics, does the level of reach really matter?