When you first log into the Grader Console
, you will notice that there is an assignment already created for you if you added the quickstart assignment from Step 1, named PS1
. This assignment is there to help you get started but you can add your own assignment files too. For the purposes of this getting started guide we will use the included PS1
assignment.
The grader's first task is to validate the Jupyter Notebook file itself. Click on the ps1
link in the Name
column to open a new tab so you can view the contents of the file:
Clicking on the assignment name within the Name
column will open a Jupyter Notebook within a new browser tab. To view the file, click on the problem1.ipynb
file link. To make sure you are working on a source file, validate that it is located within the {course_name}/source/ps1
path:
At this point, you may be wondering why so many browser tabs are open. We are aware that this could lead to confusion and are working hard on providing you with a more seamless workflow. Stay tuned!
Now, let's surface the user interface elements that will allow us to update the Jupyter Notebook cell meta-data. Click on the View --> Cell Toolbar --> Create Assignment
option:
Several cell types are supported, these include:
Manually graded answer: this option identifies cells (both markdown and code cells) which should be manually graded. Manually graded cells will also display a points value so that graders can add the points to the answer during the grading process.
Manually graded task: tasks are used when the instructor needs the student to add their answer with additional cells instead of adding the answer within the cell itself.
Autograded Answer: these cells are used to define the answers needed by the auto-grader service. The student's version should create the same output as the solution answer.
Autograder tests: these are used to assign the points value after running the tests with the auto-grader service. Essentially these are unit tests to validate the output of the Autograded Answer
outputs.
Read-only: these cells are locked, i.e. not editable by learners.
Manually graded tasks
and Autograder task
values display Points
and ID
:
You are free to choose any points value (as a number). Also, you can use any value of the ID field, as long as it is unique. Once you have added points to all Manually graded tasks
and Autograder task
values a total points value will update in the main cell toolbar. This represents the total points for the Notebook.
The auto-grader service needs special syntax located within the assignment cells to customize the student version of the Jupyter Notebook. Use-cases include:
Hiding answers from students when releasing assignments
Adding answers for manually graded markdown cells
Allow students to test code with hidden tests
Solutions are equivalent to answers. Solutions are delimited by the ### BEGIN SOLUTION
and ### END SOLUTION
tags. Solution sources are hidden from students. Code located between the commented ### BEGIN SOLUTION
and ### END SOLUTION
region is replaced with:
# YOUR CODE HEREraise NotImplementedError
If the student doesn't add an answer, the auto-grader service will assign 0 points for the cell that contains the auto-graded test for the solution.
Make sure you use the correct code comments for the programming language in use by your Jupyter Notebook kernel. For example ###
for Python and //
for Javascript. If you don't add comments for the solution region, then the auto-grader service will replace all cell contents with YOUR ANSWER HERE
.
You can hide some or all tests located within the Autograder tests
cells. The region between the BEGIN HIDDEN TESTS
and END HIDDEN TESTS
allow will hide tests when the assignment is released.
Partially hidden tests are useful for when the teacher would like to provide the student with some tests to validate their answer but hide other tests that may offer a clue to how to structure the solution.
The auto-grader includes a Validate
button to validate the Notebook's format to make sure that all tests pass. You can, of course, run the cells in order and obtain the same level of validation, but the Validate
button provides a one-click and only focuses on the Autograded tests
that assert results located within the Autograded answers
. This step will ensure that the auto-grader will not encounter unexpected errors and return inaccurate results when calculating grades.
The Validate
button is located in the Jupyter Notebooks cell toolbar:
The auto-grader service will hide answers located between the === BEGIN MARK SCHEME ===
and === END MARK SCHEME ===
region. This allows teachers to view answers during manual grading.
You may have noted the use of the ===
for the mark scheme region above. Unlike the auto-graded answer and test regions, the manually graded regions must include these three equal symbols before and after the mark scheme region.
The Points
text box is available for cells defined as Autograder tests
and Manually graded tests
. The sum of these integers is located at the top of the Jupyter Notebook, within the cell toolbar:
One assignment may contain multiple Jupyter Notebook files. Nevertheless, these files may reference other files, such as CSV data sets within the same folder or as publicly available URLs.
Feel free to create your own version or update the file itself. Make sure to save your work before moving on to the next step, which is to release the assignment to your learners!