Pool Size and Sampling in the Blackboard Testing system – Testing Best Practices Part II

In my last post, I mentioned that using random blocks of questions can result in repeated questions on an attempt. The reason behind this is that Blackboard does not track which questions have already been selected from the pool for any given test attempt. Questions are always selected from the full pool. The testing system is using sampling with replacement. This means there is a chance of the same question being chosen more than once for the same attempt (student). The likelihood of a student seeing duplicate questions depends on the size of the sample relative to the pool.

So, how large a pool do you need to decrease the likelihood of repeated questions with random blocks? To get a sense of the size of the pool needed I calculated the probability of no replication of questions for a random block of 5 questions drawn from question pools of various sizes.

Probabilities of no repeated questions in a random block of 5 questions drawn from question pools of various sizes
Question Pool Size Probability of No Repeats
10 questions 30 %
25 questions 65 %
50 questions 81 %
100 questions 90 %
200 questions 95 %

OK, these results aren’t especially surprising. As you increase the pool size the you decrease the chances of having questions duplicated. But if you are pulling questions from a single pool you are going to need a lot of questions to reduce the likelihood of duplicate questions.

Is there another way?

The point of random blocks of questions is to present each student with a different set of questions on the test. If you select a single question from any given pool you won’t repeat questions on any given test. Break a larger pool of questions into a number of small pools equal to the number of questions to be drawn and draw a single question from each pool. The pools don’t need to be large, but all the question across pools must be unique. Using this method only a few questions will be needed in each to provide variation across tests.

Using many smaller pools will require a bit more work in creating the test, however there is an additional benefit. You can better control coverage of the test. If you create the questions in each pool on the same topic, selecting a random question from each pool ensures each student receives a question in each topic area. Using a random block of multiple questions from a large pool may result in uneven topical coverage in addition to the problem of question duplication.


Best Practices for Tests in Blackboard

Giving a test or quiz in Blackboard? There are a few things to keep in mind to make the experience of giving and taking tests and quizzes in Blackboard easier.

Preview your test to make sure it appears the way you think it will appear. Are the correct number of questions in a random block displayed? If you chose to have the questions presented in random order, are they randomized? Clicking the link to the test in instructor view allows you to see the general characteristics of the test. You can use StudentView to take the test as a student would and you can see how the students will see it and how it will be scored. Put a password on the test to prevent students from seeing the test while you are testing in StudentView.

When setting the test options do not use forced completion or auto-submit. Both options will cause headaches for you and your students! The forced completion option gives students a single attempt that they must complete in a single session. They can’t stop and come back to the test later. If anything happens to interrupt a test session, the student can’t complete the test. If their web browser freezes, if their computer reboots for some unknown reason, if their laptop disconnects from the Internet, then the student will not be able to complete his/her test. The only option is for the instructor to clear the attempt, deleting any saved answers and having the student start again.

Instead of using forced completion, set a time limit. Then, if there is an interruption to the test, the student is able to re-enter and resume where they left off. If you require students to notify you of any problems, you can take that into account if they go over the time limit.

Auto-Submit submits the student’s test when they reach a specified time. If students have computer problems that prevent them completing the test in the time allowed, they will be penalized, by having the system submit their test before they can complete it. Blackboard records the time the student starts the test and runs until the test is submitted. Using the recorded time taken, allows you to make a judgement – was the extra time taken reasonable or not – and take action as needed.

Don’t get tripped up by the time of day. Blackboard considers midnight as the start of the day. When setting the start and end times for a test and you want the time to be midnight – select either 11:59 PM or 12:01 AM to be sure the exam starts or ends when you think it should. Using times on either side of midnight means you don’t have to remember how Blackboard defines it.

Finally, and to be continued in the next post, if you use pools to generate random blocks of questions, how can you reduces the chance that questions are not repeated on a student’s test. How many questions is enough? That’s the topic of my next post.

Turnitin Training Video Link

On Thursday, September 25th Turnitin provided training to faculty and staff on using the new Turnitin LTI that has been installed on our Blackboard system. For those who were not able to attend the training we have a link to the session.

Session Recording: http://turnitin.adobeconnect.com/p2vk0unaf36/

Here are the slides – saved to pdf in Notes view: Part 1 (slides 1-25) and Part 2 (Slides 25-55)

Please contact LIT with any questions.

Blackboard Datacenter Maintenance

On Friday, October 3rd, 2014 Blackboard will be performing infrastructure maintenance at the data center hosting our Blackboard system. This maintenance is not expected to affect service but it is possible that users might experience intermittent connectivity for a few minutes during the maintenance window.

Network Maintenance on September 28th will impact Blackboard

On Sunday, September 28th, between 6 AM and Noon, DTS will be performing Network Maintenance. This work will impact logging into Blackboard. Users logged in before this time will not notice any problems unless they logout and try to login again. DTS anticipates all work impacting Blackboard logins will be complete by 9 AM. During the remainder of the maintenance window, there may be additional intermittent interruptions to Wentworth’s services while testing is completed.

Blackboard Slowness/outage

We have been notified by Blackboard Managed Hosting that the slowness/outages that we are experiencing are widespread and their engineers are working to resolve the problem as soon as possible. I’ll post an update as soon as we are updated by Blackboard.

As of 8 PM we have not heard from Blackboard, however, Blackboard does appear to be running normally with no slowness or outages reported since around 2 PM.

Blackboard is back up and the problems we experienced yesterday have been resolved as of about 2 PM yesterday.


Bulk Assignment File Download

Do you use assignments in Blackboard? Do you wish you could download all your students’ assignment files at once? Would you like the files named so you know who submitted it and for which assignment? Blackboard can do that!

If your assignments require that students submit files that are not compatible with inline grading, you must download and open each file individually if you use the needs grading feature of Blackboard. But there is an early feature of this tool that might be more efficient. Bulk assignment file download.

Instead of going to the Needs Grading view in Blackboard go to the Assignments view or the Full Grade Center. Find the column associated with the assignment and click on the action button for the column header (sometimes called the downward pointing arrow) to display the menu of options available. You will see Assignment File Download as one of the options:


Select Assignment File Download and on the next screen either click on the select all or the check box for selected students. Next, select whether you want all submissions or only the last attempt and click submit to create a zip file. On the next page you will find a link to download the zip file:


You now can download a single file with all the students files within. And the best part, Blackboard renames the files to include the name of the assignment,each student’s username and, if more than one attempt was downloaded, the date and time of the attempt. OK, you still have to open each file to grade it, but getting the files out of Blackboard is easy.

Datacenter Maintenance

On Friday, September 26th, 2014 Blackboard will be performing infrastructure maintenance at the data center hosting our Blackboard system. This maintenance is not expected to affect service but it is possible that users might experience intermittent connectivity for a few minutes during the maintenance window.

Group Courses by Terms

Tired of seeing a long list of courses in Blackboard and struggling to figure out which ones are for the current term? If so, the Group by Terms feature is for you.

Group by terms allows users to have their Blackboard courses display by term. You can specify whether a term should display, whether it should be expanded when displayed, and even which courses will display within the term. Terms may be academic terms or they may be other groupings. So on my account, the My Courses module displays the terms, Blackboard Online Training, Fall 2014, Testing, and No Term Assigned.

My Courses module

My Courses module showing courses grouped by terms.

Any courses not assigned a term will be in the No Term Assigned group, until they are added to a term.

To change your My Courses display, click on the settings icon at the top right corner of the module (that thing that looks like a gear).

Personalize My Courses Page

The Personalize My Courses Page where you can turn on Group by Terms.

Click on the check box next to Group by Terms to activate the feature. Select which terms to display and whether the term should display expanded to show the courses in the term by clicking the appropriate check boxes. In section 2, of the Personalize My Courses page you can further refine what you see in My Courses by selecting the appropriate check boxes for the attributes you want displayed. Once you’ve made your selections, click submit and enjoy an organized My Courses module.

What’s New in the Upgraded Blackboard?

The beginning of any new school year brings many changes. This year brings a change in our LMS, Blackboard. This past weekend we upgraded to a recent release to gain some additional features and improvements.

One of the more anticipated enhancements is the Student View building block. This add-on creates a button on the top of the content screen that instructors can use to see the course as their students will see it. Unlike toggling the edit mode button off, when in Student View instructors can see and interact with every aspect of their course in the role of a student. The tool creates an account that is enrolled in your course as a student. Since there is an actual student account any submissions are recorded. You can see, for example, how the test grades specific responses to partial credit questions.

Group management is enhanced with an “All Users” screen which lists the group membership of each individual enrolled in the course. This enhancement allows more efficient assignment of students to groups. In addition, when you create groups, you will have an option to create a Grade Center Smart View at the same time.

Date Management is a new tool that allows you to adjust dates associated with content items copied from a previous course, all at once. Once the system shifts the dates, you will see and be able to validate the new dates all from a single page.

Grading functionality is improved with the expansion of inline grading to Wikis, Blogs, Journals and Discussion Boards. Workflows for tests have been improved with the shift of “Score Attempts Using” to the Test Options page.

My Grades for students has been update with options to reorder scores by due date, last activity, or Grade Center alignment. The interface has been improved to provide more information in the same window. Instructor feedback is now inline reducing the number of clicks needed to get a complete picture of performance on individual items.

To read more about the enhancements in the upgraded Blackboard check out the Upgrade page on our website.