LMS Criteria

Penn State LMS Evaluation – Key Requirements

Given that each learning management system (LMS) does things differently, often achieving the same higher level goal, a checklist of required features/functionality is not necessarily helpful.  Therefore, this is not intended to be a comprehensive listing of all the features and functions required in the next LMS but rather a list of high level categories against which future LMSs should be evaluated. These categories reflect the complex realities of Penn State’s context for students, faculty, and staff and should apply to all campuses and modes of educational delivery (residential, blended, and fully online distance).

Where appropriate, examples of the types of features/functionality that exemplify the higher level category are provided and should be viewed as illustrative, but not exhaustive, of the types of things the system or company needs to be able to do in order to meet the higher level requirement.  Actual details of features and how functions take place will likely vary based on the specific LMS being evaluated.

There are three general types of requirements: the first pertains specifically to LMS tools and functionality; the second relates to LMS accessibility and usability; and, the third relates more broadly to Penn State’s growth and administrative needs.

LMS Requirements:

The LMS should:

  1. Provide a robust environment for content authoring.
  2. Support Penn State’s teaching and learning needs.
  3. Support social presence, interaction, collaboration, and group work.
  4. Provide a wide variety of grading, assessment, and grade management tools.
  5. Possess an intuitive interface with a contemporary look and feel.
  6. Be mobile optimized, device compatible, and work on all major browsers.
  7. Be ADA compliant and meet accessibility needs.
  8. Allow for labor scalability and efficiency in administrative setup.
  9. Support learning and administrative analytics and outcomes.
  10. Be “future proof” and able to evolve with educational and technological developments.
  11. Support IMS Global standards.
  12. Be available as close to 24/7/365 in order to meet the needs of Penn State’s increasing global distance audience.
(1) The LMS should provide a robust environment for content authoring.

The LMS environment should enable faculty, design staff, and others to create high quality learning experiences. This includes the creation of learning objects like course content, assignments, and learning and evaluation tools. These objects might be created native to the LMS, uploaded from another source, or linked in from third-party tools.

The system should (list of examples illustrative not exhaustive):

  • Have sufficient native tools that allow for the creation of necessary teaching and learning tools: e.g. a mechanism for students to upload work products; a mechanism to create forums for students to interact; a way to create assessments of student learning that include quantitative and qualitative type questions
  • Be relatively easy to use and intuitive – interface and the number of clicks the user has to jump through
  • Allow integration with third-party content management systems
  • Allow integration with third-party learning tools
  • Support multiple content and media format types: e.g. audio, video, images, text
  • Allow the import/export of content from other learning management systems
  • Allow easy and intuitive mobile use on all popular devices
  • Provide enough space in each section for all course content and student work rather than the current restricted course size
(2) The LMS should support Penn State’s teaching and learning needs.

Penn State offers a variety of course formats — including fully online, blended, and residential instruction that the LMS needs to support. Additionally a primary distinction between an LMS (learning management system) and a CMS (content management system) is that a CMS houses and (possibly) displays content whereas an LMS is built with the primary purpose of enabling effective teaching and learning. Given that there is a legitimate range of learner preferences and pedagogical approaches, the LMS should have flexibility for how teaching and learning takes place. Accordingly, the LMS should be able to adapt to new educational approaches, for example, badging and competency-based learning and easily integrate various College-level CMS including Drupal and Plone.

The system should (list of examples illustrative, not exhaustive):

  • Provide scaffolding tools that help the student know where they are and what they need to do next
    • Progress indicators
    • Automated guided instruction (feedback mechanisms)
    • Syllabus
    • Calendar that integrates with third-party calendars
  • Provide student publishing tools
    • E-portfolio
    • Blog
  • Support a variety of pedagogical approaches
    • Competency-based learning
    • Self-paced learning
    • Self-directed learning
    • Group/Collaborative learning
  • Support a variety of course delivery methods
    • Face-to-face
    • Blended/Hybrid
    • Online
(3) The LMS should support social presence, interaction, and collaboration.

Penn State encourages an active learning community and faculty and students are accustomed to a wide variety of social media and collaborative web-based tools that enable authentic connection, communication, and group work with others. Given the importance of these types of exchanges to teaching and learning, the LMS needs to support a variety of high quality social presence, interaction, collaboration, and group work tools.

The LMS should (list of examples illustrative, not exhaustive):

  • Enable social presence in the online space (within and across courses) so that students and faculty have the experience of being “real” people connecting with other real people in an active learning community
    • Through the effective use of visuals to reflect who’s active in the course (facebook, twitter, yammer)
    • Through the use of an activity stream so that it feels like the course is an active community and not just a flat file structure from which students download static documents
  • Enable authentic and efficient means of interacting within and across courses
    • Through a robust mail tool that allows users to send and receive mail, including attachments, see who has read mail, etc.
  • Include robust tools and intuitive interfaces for online synchronous and asynchronous discussions
    • A synchronous chat functionality that indicates who’s online
    • Asynchronous discussion space that has an intuitive look and feel and promotes authentic educational dialogue
  • Provide or easily integrate with tools students can use to collaboratively create and share content as well as interact in virtual meeting spaces
(4) The LMS should provide a wide variety of grading, assessment, and grade management tools.

Assessment and grading needs and requirements vary significantly across individual faculty, courses, programs, and colleges. Therefore, the LMS assessment options should support the complex reality of Penn State courses — including the ability to do both formative and summative assessment — and be relatively intuitive to set up and use. The LMS should enable professors to conduct a role play simulation to learn exactly what will happen as students go through. This simulation or “preview” should be easy and 100% accurate.

The LMS should (list of examples illustrative, not exhaustive):

  • Include a variety of assessment tools that allows for multiple assignment submission formats, question formats, and multiple response and feedback options (audio, video, textual)
  • Allow for a variety of ways of grading student work products
    • By student, assignment, or team
    • Auto-graded or manually
    • Based on rubrics
  • Provide a robust and configurable gradebook that can handle a variety of functions
    • Pull in grades from assessment activities outside the LMS
    • Allow for extra credit assignments
    • Be exportable to alternative file formats
    • Enable faculty to submit grades to registrar
    • Allow for formulas and weighting
    • Allow for selection of points or percentages
(5) The LMS should possess an intuitive interface with a contemporary look and feel.

The LMS we go with needs to have an interface with a modern, contemporary look and feel. Given that design and usability standards evolve, the LMS should be skinnable such that it doesn’t look “dated” and is able to adapt to contemporary design standards and the company should invest in interface design as an important component of the LMS.

The interface should (list of examples illustrative, not exhaustive):

  • Not require significant training for the average end user to find and use most of the features and functionality they require
  • Not get in the way of the learning; for example, it is currently the case that the design of ANGEL discussion forums interferes with collaborative learning
  • Reflect contemporary design standards
(6) Be mobile optimized, device compatible, and work on all major browsers.

In order to meet the needs of Penn State faculty, staff, and students the LMS needs to be user centered regarding devices and browsers. In most cases, users should be able to work using their preferred method for accessing their course with little to no hindrance.

The LMS should (list of examples illustrative, not exhaustive):

  • Be responsive to a user’s device
  • Have an app(s) that allows for content generation and content consumption
  • Have an app(s) that takes into account necessary functionality for students, faculty, and design staff
  • Work on all current versions of the major browsers
(7) The LMS should be compliant with federal laws and regulations protecting people with disabilities.

The LMS should meet Penn State Policy AD69 which includes compliance with WCAG 2.0 Level AA standards. If the LMS includes native apps on mobile device platforms, those apps must conform to the above standards even if data exchange does not use Open Web Technologies.

The vendor will be responsive to reports of accessibility issues, including problems that may occur as a result of code changes related to new features, major and minor version updates, and patches. The vendor agrees to fix accessibility issues in a timely manner.

Penn State staff will, at their discretion, test claims of accessibility made by the vendor of the LMS.

The LMS should:

  • Conform to Web Content Accessibility Guidelines (WCAG 2.0 AA)
  • Generate web pages that comply with HTML code standards and best practices
  • Possess accessible content creation tools with functionality to
    • Be fully operable using only a keyboard
    • Be readable and operable using commonly used screen readers (JAWS, VoiceOver, NVDA)
    • Add alt text to images o add headings to text content
    • Add labels to form fields
    • Add headers to tables
    • Paste blocks of HTML code in an HTML view of content, including language extensions like MathML
  • Allow persistent, user personalization to modify presentation style (e.g., high contrast, color inversion, color modifications)
  • Include an accessible, keyboard operable media player allowing closed captioning
  • Be accessible on mobile and small screen devices
  • Alert the screen reader user of error messages and changes in dynamic content
(8) The LMS should allow for labor scalability and efficiency in administrative setup.

Penn State is a large and complex institution comprised of multiple campuses, colleges, departments, and programs sometimes operating independently of each other in ways that have implications for LMS administration regarding roles, rights and permissions. Additionally, as Penn State continues to grow (whether World Campus to 45,000 students, blended learning, residential campus enrollments, or residential online courses with multiple sections) the LMS needs to support course and program setup such that it can, when appropriate, be done at scale.

The system should (list of examples illustrative, not exhaustive):

  • Provide delegated and granular administration such that the assignment of different permission levels can be done specific to work groups, departments, colleges and campuses
    • A specific look and feel could be designated for all courses in a college, department, or program
    • Multiple roles, like faculty, ID, TA, etc., could be assigned based on access and editing requirements
    • LTI tools could be administered differently across the University based on contractual agreements
    • Appropriate units should be able to run reports as needed, for example, the Office of Student aid should be able to run activity/login reports for students
    • If collaborative tools, like Blackboard Collaborate, are used, all users should be able to create, modify, administer their own collaborative spaces
  • Enable efficiencies within and across course sections
    • Work w/multiple course sections simultaneously
    • Within a single section, utilize functionalities like drag and drop, autopopulate, and grid edit that allow for bulk setup
    • Allows for persistent content within a course across semesters that doesn’t require updating each semester (currently use ANGEL environmental variables for this)
  • Work w/other applications such as registration, grading, advising, and course selection systems
(9) The LMS should support learning and administrative analytics and outcomes.

In May 2011, every baccalaureate program at Penn State was asked to submit a list of their student learning objectives and a brief plan for collecting data relevant to one of the objectives. The University is required to assemble and maintain a complete suite of learning objectives and data collection plans, annually, as part of our accreditation review by the Middle States Commission for Higher Education. Penn State is best positioned to support faculty in meeting these requirements and to move the institution toward a culture of assessment by choosing an LMS and a suite of learning-analytics tools that can be applied in both the residential and the online learning context.

(10) The LMS should be “future proof” and able to evolve.

While no existing LMS can predict the future, there are some models that are more adaptable and able to evolve with technology and user trends. Penn State should be investing in an LMS that is able to respond to our needs “tomorrow,” not just “today.”

The LMS vendor or consortium company should (list of examples illustrative, not exhaustive):

  • Embrace and practice a philosophy involved in looking for ways to improve how we use LMSs in education (e.g. educational standards; retention center)
  • Have a plan for responding to user trends, for example, MOOCs, badges, working online and offline, etc.
  • Have a philosophy that demonstrates innovation and support of standards
  • Have a process for innovating on the edge – testing new ideas and incorporating them into existing platform
  • Have a process for R&D that allows for experimentation, innovation, and the systematic vetting of new ideas
(11) Support IMS Global standards.
  • Learning Tools Interoperability (LTI)
  • Learning Information systems (LIS)
  • Common Cartridge (CC)
  • EduPub
(12) The LMS should be available as close to 24/7/365 in order to meet the needs of Penn State’s increasing global distance audience.

Penn State is increasingly operating in a global environment such that it is critical that our LMS (and all third party tools used in conjunction with the LMS) be available at all times when the students and faculty need them. This means operating as close to 24/7/365 as possible. Even being offline a few hours can be an inconvenience for faculty, students or staff depending on time zones, work commitments, and family obligations.

The LMS should (list of examples illustrative, not exhaustive):

  • Have adequate system architecture in place to allow for backups, testing, and staging so the production (live) instance is not impacted
  • Have a dynamic and distributed infrastructure where the outage of one server does not impact the production environment
  • Be elastic so that changes and updates are easy to install with minimal service interruption
  • Be resilient such that the system reacts intelligently to spikes in usage
  • Ensure that Penn State would always be current with the latest version

Sources for this document:

This document draft was created based on the accumulation of sources and feedback.