March 2010 Archives

The room is packed with people who want to hear about the best ways to evaluate redesign projects and their impact.  Peter Ewell, the first speaker, has been with NCAT since the beginning as their official "assessment guy".   From his experience, the types of questions asked during assessment fall into two main categories: implementation questions about the redesign itself and impact questions about the way the redesign has met the goals for improvement in student learning.

Implementation Questions: The purpose of these kinds of questions is to see whether the project rolled out according to plan, if we did what we said we were going to do, and to identify any problems in the system. 

Typical areas of investigation include:

• Getting resources and people in place as planned
• Technical support issues and technical glitches
• Whether there were different practices in different sections
• Major differences in faculty (e.g. experience, teaching style)
• Major differences in students (e.g. part-time, gender, age, evenings, different majors)
  

All of this reflects the execution of the plan and by fixing or controlling those variables, we can more precisely examine impact.  

Impact Questions boil down to one main issue: Did the project achieve the outcomes that it was supposed to achieve?  To measure impact, we can look for evidence of the following:

• Equivalent or better cognitive outcomes as measured by having students complete the same exam as control sections or previous course offerings, tracking grade performance if grades are an objective measure, having third-party people grade writing samples from current and previous students using a rubric, and examining common mistakes made by students in the redesigned course
• Improved course completion rates, typically as measured by falling DWF (earning a D or F in the course or withdrawing)
• Improved performance later in the curriculum, especially for courses that directly depend on the course that has been redesigned.  For example, if you redesign a math course, you could look at student performance in the next math course in the series or in a course like physics that depends on the redesigned course.
• Enhanced self-confidence and willingness to pursue the subject, typically measured by a student attitude scale.
  

I think that's a good starting point.  Cole and I have done some talking about some additional factors that could be included in a kind of assessment framework that would reflect some of our goals beyond strict measures of single-course performance.  Some of these factors include measures of student engagement, the ability to learn from a course redesign and generalize the findings to other courses, the creation of portfolio-worthy artifacts, and increased student compliance with legal and policy issues.  For example, if students take a redesigned version of Econ 4 and perform the same on the final exam, but work collaboratively outside class on group projects (engagement) and produce videos (portfolio-worthy) that conform to copyright requirements (compliance with law and policy), then I would judge it as a success.  If anyone has ideas about additional types of factors that should be included in that kind of assessment framework, I'd be interested in hearing about them.

The other two presenters in this session went over how they were just-in-time activities in some of their courses.  Julia Johnson from Arizona State is using a geology textbook that uses a lot of diagrams and has mini objectives for each page: "before you leave this page, identify...".  Julia collects these prompts and puts them into a "What you should know" list that she gives to students.  She has some quizzes in her course to make sure that students are going through the materials and her exams are a combination of multiple choice questions and concept sketches, in which students need to create, label, and explain diagrams of things like the four types of volcanos.

Renee Fisher from the Open Learning Initiative was showing some of the robust questions embedded in her online course content.  Students are asked to do a calculation and can get a series of hints that will help remind students how the calculation should be done and walk them partially through the process.  The hint requests are all recorded and faculty can use that information to address problematic subject areas in greater depth when the class is together.

So overall, it was a pretty interesting session.

Dennis Pearl kicked off the second day of the NCAT conference with this quote:

"This is not Burger King.  You don't get it your way.  You take it our way or you don't get a damned thing." Mary Smith (owner of a very small diner)

... and Dennis made the point that this is the approach that we typically used with our course designs.

He has used the buffet model to redesign a large enrollment statistics course.  The buffet model is pretty simple.  It's a way to let students personalize their own experience in a large enrollment course.  They accomplish this by making a contract describing how they will be taking the course and what they will be doing to meet the learning objectives.  Students are guided in their creation of their contract by taking a series of learning style and study skill inventories that indicate what content presentation, lab, and homework would best fit their

Before taking on the redesign project, the faculty had students take these inventories and then talked to groups of students about the course design.  They found that students prefer:

   * Visual information instead of verbal delivery (only 8% prefer lecture content)
   * Sequential perspective (building up to the big picture) instead of a global perspective
   * Hands-on activities instead of abstracted data analysis
   * Collaborative work instead of individualized reflective work

They also realized that they were having their most expensive personnel (senior faculty) repeating lectures three times per week, which is the least preferred method of learning.

In the redesigned course, they give students choices between two different types of lecture (global/reflective or sequential/collaborative), two different types of lab activities (hands-on data generation or abstracted data analysis), and two different types of homework assignments.  This creates 8 possible combinations, but an interesting thing is that all students end up turning in the same assignments.  These assignments explicitly list the learning objective and ask the student how this objective was addressed in the presentation, lab, and homework.  Students in the whole class are also given the same exam, regardless of which options they chose to meet the learning objectives.

Overall, I think this is an interesting approach that can work in cases where faculty are already teaching multiple sections of the same course.  It takes a bit more work to set up the multiple paths to the same objectives, but not as many as I thought (binary choices in presentation, homework, and lab activities).  It isn't a free-for-all.  Also, by completing the same assignments, they compare apples to apples.

Dennis would still like students to see the field of statistics as important and valuable, but they haven't been able to achieve that with their redesign project yet.

I decided to go to this session because 1) Carol Twigg was one of the moderators and 2) it covered course that didn't fall neatly into the other course disciplines that were being focused on at the other sessions. 

Most redesign projects fall into one of six redesign models that can improve learning outcomes or reduce costs:

• Supplemental - add to the current structure and/or change the content
• Replacement - blend f-2-f and online activities, reduce class time.  For example, foreign language instruction where activities like vocabulary are moved out of the traditional meeting time to online activities.
• Emporium - Move all classes to a lab and get rid of the lecture.
• Fully Online - Conduct most or all learning activities online
• Buffet - Give students options for getting to the same learning objectives through a variety of options (lecture, small group,  software, etc...) based on student preference
• Linked Workshop - replace developmental courses with just-in-time workshops.  For example, students enroll in a college-level math course, but students who need additional help have access to linked workshop sessions to help them through topics that are going to be in the normal session.
  

Supplemental example: A biology course implements a short quiz before a lecture to make sure that students have read the assigned readings.  During class, students get together in small groups to discuss a posted question and decide the answer as a group and then respond through a clicker.  The lecture could be designed for students who have read the material instead of assuming that students didn't read.  Also, there was some peer pressure to be prepared by the student group discussions.

Supplemental accounting example: 44 sections of an accounting course at Boise State.  They needed to address issues like efficiency, consistency, active learning, and reducing the teaching load of tenure track faculty.  Through the course redesign, they went to 12 sections of 150 students, software for homework and practice quizzes, clickers used for assignments, and replaced one lab with undergraduate learning assistants to provide assistance.  Costs were reduced from $121/student to $62/student.

Benefits of using undergraduates as teaching assistants: they cost very little or nothing (if doing the work for credit), it is easier for other students to relate to them, their language skills are often better than graduate students, it's a resume builder, and it can lead the undergrads to consider different fields (e.g. science education).  I was a tutor for several years as an undergraduate (sophomore through senior year) and it was a great way to reinforce the materials that I was teaching and it was how I discovered how much I love teaching.  I know a lot of people who were undergraduate tutors who have gone on to consider grad school - I wonder about the connection between being an undergraduate learning assistant and interest in applying for graduate school.

English Composition example of the replacement model.  They had 100 sections of 30 students each  The success rate used to be less than 60%.  They wanted to create more active learning and increase consistency.  They moved the classes to computer labs, interwove writing and reading, created a menu of reading and writing activities, started using discussion boards for peer collaboration, and started using SMARTHINKING tutors.  The cost per student dropped by 40%, increased the opportunity for immediate feedback, increased time on task, created more time for writing activities, small-group conferencing, collaborative activities, critiques, and discussion.  Learning outcomes increased as well.

Emporium model: this is where lectures are removed and replaced with lab-based activities (typically in a computer lab).  It can have students from several different, but related courses in the lab at the same time with faculty who can assist with any of them, so a single lab could have students from elementary algebra, college algebra, and statistics at the same time.  They're comparing this to the one-room-schoolhouse model.  Interesting idea that could work at a smaller campus.

At Mississippi State, a 3-hour lecture course in statistics was changed to have the content placed online.  Students watch and read it ahead of time and then attend a 3-hour lab session, supported by graduate and undergraduate teaching assistants.  The room has a bunch of tables (to be used with student laptops) and many smart whiteboards for spontaneous tutorials that can be recorded.  This can reduce faculty time needed to grade assignments as well as increasing the kind of personal connection between faculty and students. In some of these emporium designs, the faculty hold their office hours in the labs, so they can be in the same location where students are going through the materials.  

In the labs, undergraduate students can answer simple questions and then involve graduate students or faculty if there is a more difficult question.  This reminds me a lot of the models used in the Open University systems in Europe, where there are distributed tutoring centers throughout a country where students can meet and get direct assistance from a subject expert under the ultimate direction of tutoring coordinators and the faculty who wrote the course content and assessments.

Northeast State Community College has taken this type of idea and applied it to a developmental reading lab using the MyReadingLab software coupled with diagnostics assessment, individual learning plans, and individualized assistance in the lab.

Anyway, very interesting session that has gotten me thinking about different ways to restructure courses to make them more active, relevant, personalized, and cost effective.

Kay McClenney started off the keynote presentation at the ncat conference by having people stand if they fit into some of the demographics of students that she studies: people who got a degree while attending college part-time, worked more than 20 hours per week, were first generation college graduates, dropped out or stopped at some point, English was their second language, they were minorities in most situations in college, etc...  For a large presentation, it's an effective attention-grabber, demonstration of relevance, and a way to get people to wake up in the morning.

With her focus groups, they ask students if they have thought about dropping out.  Typical answer: "yes, about every 20 minutes".  So the follow-up question is "What keeps you in school?" and the answer is almost always about personal and meaningful relationships with a mentor, coach, or other students.  This is one reason that I have been interested in projects like the first-year seminar experience courses and the NSSE data.  A lot of factors of student engagement involve building relationships with others.  [And it's okay if some experiences that are designed to build relationships are made mandatory - students complain, but will do them anyway.]

I hope she puts her presentation online - she had a good slide listing the characteristics of the least engaged students: male, had taken less than 30 credits, were not seeking a credential, were part-time.  There were about 8 or 9 of these low-engagement characteristics.

Her main point here is that "students don't do optional" - so we need to make relationship-building high-impact activities mandatory, make outside-class activities inescapable, and integrate student support services into courses.  Two comments on this.  Some students DO optional, but those are typically the high-achieving students who don't need those connections and services as much.  Second, I think that we need to look at these activities and services to make them relevant and engaging before we try to make them mandatory.  No one likes busy work.  Student government, Paternoville, fraternities, THON, student clubs, etc... are optional activities that are engaging and build strong relationships.  In all of these cases, they are also student organized and therefore students may feel ownership.  So maybe the relationship is to other students and the activity itself.

Chris Long, Wayne Anderson, and I had a good discussion about the keynote afterward.  Nothing too new here, but it got me thinking a bit. 

I'm at a presentation by staff from Desire2Learn, which is a commercial course management system.  I like what I'm hearing:

The tool started off to support online courses - which means that they understand what it means to recreate and deliver a complete online educational experience.

They allow fine-grain control over user access at every level: university, campus, college, department, course, and section.  So you could have an eLSS at a campus have read-only access to the content of every course at their campus (if that's what they and their faculty want).  This is a big deal for organizations like World Campus and eLearning Institutes, but it will also raise some policy issues.

At each of the institutional levels (university, campus, ..., section) you can enable the same group of tools, so you could have quizzing or communication tools that would encompass all World Campus students, all College of Business students, all members of a department, etc...

We're moving into the demo now.  They're building a site for us to look at (http://psu.desire2learn.com), which is still in the works.

After logging in, D2L puts you into a customizable dashboard/widget screen (you can't currently rearrange these unless you are an admin and want to change it for all users).  When you go into the course level, you have a similar dashboard interface with widgets related to the course and your role within it. 

As for accessibility, they seem to have taken it to heart.  Steph said that the programmers can't make anything unless it meets the Section 508 checklists.  We'll still need to test this of course.

From the initial login dashboard, you can see your courses as well as things like the number of unread messages within those course (much like ANGEL).  This feature isn't in every system and our faculty depend on it as a time saver.

Communication tools:  They have an instant messaging (paging) tool, a chat room tool, integration with Elluminate or Adobe Connect, news (like announcements), 

They gave some examples of what happens when things aren't entered correctly and it's pretty slick.  A little notice slides down within the tool itself and says something like "Title is required" with "Title" linked.  You click on the link and the Title field gets focus and another little help bubble pops up reminding you that the title is required.  It's helpful without being obnoxious.

Materials can be released based on conditional requirements such as posting a message to a certain topic or completing a quiz with a certain score.  You can add multiple conditions, which is very nice.  Steph says that you can even release materials in a course based on something at a different level - so maybe College of Liberal Arts students have to complete an Academic Integrity module at the college or university level before any of their course content is release in any course.

On to assessment now: They have very close integration with TurnItIn that will show the originality report directly in the dropbox tool.  Assignments can be set to have different assignment dates or extra time for something like a quiz (our #1 accessibility request) based on individual settings.

Materials can be conditionally released based on something like all users having submitted their assignments.  Dropbox submissions for group assignments can have the grade attributed to the entire group, even if it was only submitted by one person.  Also, multiple people from a group can contribute different parts of a group assignment.  This is the kind of stuff that we have been requesting from ANGEL since we adopted it.

Groups in D2L can be randomly assigned and while doing that, you can automatically create discussion boards, group assignment folders, and lockers (team file sharing space).

The quizzing engine looks good: quiz banks, random selection of questions, lots of question types, secure browser, detailed feedback with links back to relevant content, etc...

Wrapping up now.  Overall, I'm really impressed.  I think that Steph and I have a similar perspective on things like scorecards and feature lists - most CMS's have the same major features.  The real difference is in how the CMS does what it does and the underlying vision that shapes its direction.  Still, I think D2L is going to score very well.  I'm looking forward to getting my hands on it.