Every step of Assignment 4 talks to a server that is not ours. Part 1 downloads the planning districts from ArcGIS Open Data and the street network from the OpenStreetMap Overpass API (osmnx), and Part 2 drives a browser through five pages of Craigslist with a sleep between each. Re-running a class of these notebooks would repeat every one of those requests once per student, which is exactly the load the assignment itself warns will get an IP banned.
+
+
So rubrics/hw4.yaml says execution: mode: never, and the service honours it in every execution mode, local included: no kernel, no notebook container, no browser, no data placement. See Execution. The grade comes from the file the student handed in, read two ways:
+
+
+
The code they wrote. Each code cell is matched to the template cell it was filled in from, and the template's own lines are set aside first. The driver.get(url) and pd.concat(...) inside the provided scraping loop, or a def format_price(price_string): return stub, earn nothing. Each step is a list of requirements in the rubric, one regex each, scoped to its Part, so a TA can widen one without touching Python.
+
The outputs their own run saved. The crash count, the scraping log (Processing page 3... Number of apartments = 120), the results table, the rendered histograms, the Folium map and the Altair spec, including the scraped rows the spec embeds.
+
+
+
The one thing that does run: three pure functions
+
+
format_bedrooms, format_size and format_price turn "2br" into 2, and calling them is far better evidence than reading them. So they, and only they, are cut out of the notebook (grader.function_sandbox) and called with typical Craigslist strings in the Docker sandbox, under the same isolation flags as a notebook run, above all --network none:
+
+
+
only the top-level def of each named function is extracted. No other statement of the notebook goes anywhere, and the submission itself is not mounted into the container;
+
decorators and annotations are stripped, and a function whose default argument is not a plain literal is refused, because those run at definition time;
+
only imports of re, string, math, decimal, numpy and pandas go along. selenium, requests and osmnx do not;
+
the expected answers stay on the host. The container runs grader.function_runner, which is told what to call, not what should come back.
+
+
+
+
In local mode the functions are not called
+
The machine has a network, so item 2.5 gets a provisional score from the source and goes to a TA. The sidebar says so before the run.
+
+
+
Scraping has to be shown, not loaded
+
+
A results table read back from a CSV proves nothing about a scraper. Item 2.6 wants the completed loop in the notebook and the log it printed when the student ran it, and the results table only counts when that log is there. A table loaded from a file is recorded as such and goes to a TA.
+
+
Reference values, loosely
+
+
There is no reference solution, and the street network and the listings both change. The only numbers used are the template's own: about 3,750 crashes in Center City (3,000–4,500 accepted), 120 listings a page, and a price per square foot around 1.5. Each only raises confidence. A number far outside its range sends the item to a person; it never costs a point on its own. As everywhere, anything short of full marks reaches a person.
+
+
Where the 100 points go
+
+
+
Item
Points
+
+
Notebook run and submitted with its outputs
10
+
1.1
Central planning district polygon
4
+
1.2
Drive network from the polygon
4
+
1.3
Network edges as a GeoDataFrame
3
+
1.4
PennDOT crash data loaded and combined
4
+
1.5
Crashes as a GeoDataFrame (latitude-as-x is named)
4
+
1.6
Crashes trimmed to Center City
5
+
1.7
Graph and crashes projected to EPSG:2272
5
+
1.8
Nearest edge for each crash
5
+
1.9
Crash count per street
5
+
1.10
Crash counts merged onto the edges
4
+
1.11
Crash index
5
+
1.12
Histogram of the crash index
3
+
1.13
Interactive map of the crash index
4
+
2.1
Selenium driver opens Craigslist
3
+
2.2
BeautifulSoup from the page source
2
+
2.3
Listing elements extracted
4
+
2.4
Price, bedrooms, size and title of the first listing
4
+
2.5
format_* functions, called in the sandbox
9
+
2.6
Scraping loop completed and shown run
7
+
2.7
Histograms of price and price per square foot
3
+
2.8
Interactive Altair scatter of price vs size
3
+
Total
100
+
+
+
+
Execution and output completeness is 10 points, Part 1 is 55 and Part 2 is 35.
+
+
The demo class (python examples/make_hw4_submissions.py) has four submissions: complete, one with the usual mistakes and its Craigslist results loaded from a CSV, one that crashes in Part 1, and the untouched template. Their saved outputs are written in directly; nothing is executed, there or when grading. In Docker mode they score 100, 84, 26.5 and 3. In local mode the format_* item is not called and goes to a TA instead.
+
+
+
+
Assignments 5–6
Their student templates are in assignment_template/, but they are not graded yet. Each needs a rubric YAML and a grader class. See Adding an assignment.
Each submission is copied into its own working directory. The notebook is instrumented with probe cells, and a subprocess runs it. By default that subprocess runs inside a container with no network, capped memory and CPU, a read-only root filesystem and no host environment variables. See Docker sandbox.
+
+
When a rubric says execution: mode: never
+
Assignment 4 scrapes Craigslist and downloads from OpenStreetMap and ArcGIS, so its rubric opts out of execution entirely. The service honours that in every execution mode, local included: no data is placed, no kernel or notebook container is started, and the execution stage is recorded as skipped. The grade comes from the submitted file's code and saved outputs. The only code that runs is a few named pure functions, cut out of the notebook and called in the network-less sandbox, and only in Docker mode. See Assignment 4 and grader.function_sandbox.
+
+
Errors do not stop grading. allow_errors is on, so the notebook runs to the end, the first failing cell and its traceback are recorded, and every check that can still run does. A failed notebook never becomes an automatic zero. The execution item itself awards partial credit for how far the notebook got: failing at the last cell is worth more than failing at the first.
3. Inspection without variable names
@@ -74,6 +79,8 @@
3. Inspection without variable names<
4. Hidden tests
+
Hidden tests normally run inside the student's own kernel, during the notebook run. Assignment 4's notebook is never run, so its hidden test is type: isolated_functions: the named functions are extracted from the notebook and called on their own in the sandbox. See Rubrics.
+
calculate_percent_increase(group_df) takes a DataFrame, so it can't be tested with literal arguments. Instead the probe reads the column names and date dtype of the student's own tidy frame, builds a synthetic group frame from interpolated anchor points using those names, and calls the function with it. The expected answer follows from the anchors, so it is known whatever data the student loaded.
The anchors deliberately imply a different answer for the first and last rows than for the two dates named in the assignment. That lets the grader identify the most common wrong implementation by name instead of just marking it wrong:
Assignment 1: the center_city_zips list matches the one printed in the student template, and start_date/end_date match the dates in the instructions. Nothing else needs updating when the data changes.
Assignment 2: there is no answer key to check. hw2.yaml holds policy: how much a chart that never rendered is worth, what counts as a transformation, how many words make a discussion, and the ceiling on an unreviewed aesthetics score.
Assignment 3: every expected count and median was computed from assignment_template/assignment3-data.zip. If you hand out a different zip, they all change together. Re-run the reference solution and update the numbers, or widen the tolerances.
+
Assignment 4: there is no reference solution. The only numbers are the template's own rough figures (about 3,750 crashes, 120 listings a page), each with a wide tolerance, and they only raise confidence. Keep execution: mode: never, and settings.template_notebook pointing at the template handed out.
File layout
-
A rubric has six top-level blocks. Only assignment.id and at least one rubric item are required.
+
A rubric has seven top-level blocks. Only assignment.id and at least one rubric item are required.
assignment: # who this rubric is for
id: hw1 # required; selects the grader class registered under this id
@@ -31,6 +32,7 @@
File layout
settings: {...} # grading behaviour
discovery: {...} # how to find the notebook in a submission
data: {...} # data the instructor supplies
+execution: {...} # whether submissions are run at all (optional)
probe: {...} # hints for the in-kernel probe
rubric: [...] # the graded items
@@ -80,6 +82,15 @@
data
max_placements: 12 # cap on target paths per submission (HW3: 40)
prefer_student_files: false # grade on the student's own file if they sent one
+
execution
+
+
Optional. Leave it out and every submission is run in the execution mode the TA selected. Set mode: never and no submission of this assignment is run, in any mode, local included: no data is placed and no kernel or notebook container is started. The grade then comes from the submitted file's code and saved outputs, plus any isolated_functions hidden test. Assignment 4 uses it because its notebooks scrape Craigslist and download from OpenStreetMap and ArcGIS.
+
+
execution:
+ mode: never # grade the file as handed in; enforced in grader/service.py
Hints the in-kernel probe uses to recognise columns, plus limits on how much it reports. Hints are matched against column names, case-insensitively.
@@ -112,14 +123,14 @@
Item types
execution
Did the notebook run? Handled by the shared check_notebook_execution. Partial credit is proportional to how far it got, scaled by partial_credit_ratio.
structural
Checks on the source: imports, read calls, relative paths, functions defined.
deterministic
Checks on results: probe objects, saved answers, chart specs. This is the default type.
-
hidden_test
A function is called inside the student's kernel with inputs from the rubric. Expected values never leave the host.
+
hidden_test
A function is called with inputs from the rubric: inside the student's kernel, or on its own in the sandbox for isolated_functions. Expected values never leave the host.
qualitative
Needs a person's judgement, or an LLM if enabled. Always reviewed.
Hidden-test config
-
Two call styles are supported. scalar_args passes literal arguments. group_frame asks the probe to build a DataFrame from interpolated anchor points, using the student's own column names.
+
Three call styles are supported. scalar_args passes literal arguments. group_frame asks the probe to build a DataFrame from interpolated anchor points, using the student's own column names. Both run inside the student's kernel. isolated_functions runs without the notebook.
For scalar_args, each case has args and optional kwargs and label.
+
isolated_functions
+
+
For an assignment that is never run (execution: mode: never). Only the top-level def of each named function is cut out of the notebook and called in the Docker sandbox with --network none; no other statement of the notebook goes along, and the submission is not mounted. Decorators and annotations are stripped, a function whose default argument is not a plain literal is refused, and only top-level imports of the modules in allowed_imports are sent with it. Each case's input is passed as the single argument; the expected values stay on the host.
In local mode the functions are not called, since the machine has a network. The item is graded provisionally from the source and goes to a TA, and preflight warns before the run. The code is in grader.function_sandbox and grader.function_runner.
+
Credit ratios
Most item configs end in *_credit_ratio or *_penalty keys. These are the named near-misses: each is the fraction of the item's points awarded when the check recognises that particular mistake. They are policy, so tune them freely. The assignment grader reference documents which keys each check reads.
An assignment module maps rubric ids to check methods. Every check receives the same context and returns one RubricItemResult carrying a score, a status, a confidence and the evidence behind it (design.md §24).
Only selectors and call inputs — expected values stay on the host.
Two call styles: scalar_args passes literal arguments, group_frame asks the probe to build a DataFrame from interpolated anchor points and pass that. Either way the container is told what to call the function with, never what it should return.
Call the functions a type: isolated_functions item names.
+
The functions are cut out of the notebook and run on their own in the network-less sandbox (grader/function_sandbox.py); the rest of the notebook is never run. Only the inputs go into the container — the expected values stay in the rubric, on the host.
Run every rubric item's check_<id> method and return one result per item.
A missing check returns manual_review; a check that raises returns manual_review with confidence 0 and the error in its evidence, so a broken check never loses a submission.
Assignment 4 autograder — street networks and web scraping.
+
This assignment is graded without running the notebook (execution: never in rubrics/hw4.yaml). Every step of it talks to a server that is not ours — ArcGIS Open Data for the planning districts, the OpenStreetMap Overpass API for the street network, and five pages of Craigslist through a real browser — so a re-run would repeat all of those requests once per student. The service starts no kernel, container or browser for these submissions in any execution mode.
+
What is graded instead is the file the student handed in, read two ways:
+
The code they wrote. Lines the template already provides are set aside first, cell by cell (SubmittedCode), so the scaffold — the driver.get(url) and pd.concat inside the provided scraping loop, the def format_price(price_string): stub — earns nothing by being left in place. Each step is then a list of requirements in the rubric, a regex each, so a TA can widen one without touching Python.
The outputs their own run saved. The crash count, the scraping log, the results table, the histograms, the Folium map and the Altair spec are all in the notebook as the student ran it. Reference numbers come only from the assignment text ("about 3,750 crashes", 120 listings a page) and only raise confidence; a number outside its range sends an item to a person, it never costs points on its own.
+
The single exception is 2.5: the three format_* functions are cut out of the notebook and called on their own in the network-less Docker sandbox (grader/function_sandbox.py). Nothing else from the notebook goes with them.
+
Review policy as elsewhere: full marks when the evidence is there, and anything short of full marks reaches a person (grader/scoring.py).
Confidence for a step judged from source alone. Above the review threshold: the code is the evidence the step asks for. Lower when the student's own run shows that step raising.
4 ptsdeterministic2.3 Listing elements extractedrubrics/hw4.yaml
The listing elements are selected from the soup. The template expects 120 per page; a count is corroboration only, never a requirement, since Craigslist changes its page layout.
Each code cell, with any template scaffold it was filled in from removed.
+
A cell counts as filled in from a template cell when it keeps at least half of that cell's lines. Only then are those lines removed — so a student who writes driver.get(url) in their own 2.1 cell keeps it, even though the same line is also part of the provided scraping loop.
Two modes share one code path: the notebook is instrumented with probe cells, written into an isolated working directory, and executed by a subprocess. Docker is the default because student code is untrusted; local execution exists for machines without Docker and is labelled unsafe everywhere it appears.
A first cell that points data URLs at the supplied local file.
The container has no network, by design. A student who loads the data straight from Zillow's URL is following a reasonable pattern, so a URL whose filename matches a file the instructor supplied is read from that local copy; any other URL still fails exactly as it would offline.
docker run flags every sandboxed run gets (design.md §18).
+
Shared with grader/function_sandbox.py so the few functions Assignment 4 runs in isolation are held to exactly the same limits as a whole notebook — above all --network none.
Standalone runner for isolated function tests. Copied into a fresh, otherwise empty directory and executed inside the network-less grading container by grader/function_sandbox.py. It must not import anything from the grader package.
+
It receives function definitions that were already cut out of a student notebook on the host — never the notebook itself — plus the inputs to call them with. The expected answers stay on the host; this script only reports what came back.
Run a few named functions from a notebook, and nothing else, in the sandbox.
+
Why this exists
+
Assignment 4 must never be executed: its notebooks scrape Craigslist and pull data from OpenStreetMap and ArcGIS, and running a class of them would repeat all of those requests once per student (see never_executes in grader/rubric.py). But three of its answers are small pure functions — format_bedrooms, format_size and format_price turn "2br" into 2 — and calling a function is far better evidence of what it does than reading it.
+
So those functions, and only those, are run, under four rules:
+
1. Extracted, not executed in place. The host parses the notebook and cuts
+
out the top-level ``def`` of each named function. No other statement of the
+notebook is sent anywhere: not the Selenium driver, not the scraping loop, not
+a cell that happens to sit next to the function.
+
2. Nothing runs at definition time. A decorator, a default value or an
+
annotation is evaluated when ``def`` executes, so it could smuggle any call
+in. Decorators and annotations are stripped; a default that is not a plain
+literal makes the function ineligible, and it goes to a person instead.
+
3. Only harmless imports. A function may use re or numpy imported in
+
another cell, so top-level imports of an allowlisted set of modules are sent
+along. Anything else (``selenium``, ``requests``, ``osmnx``) is not.
+
4. **Docker with --network none, or not at all.** The functions run in the
+
grading container with the same isolation flags as a full notebook run, and
+the container sees one scratch directory holding the extracted source and the
+inputs — not the submission. In local mode the machine has a network, so the
+functions are not run; the item falls back to the source and the student's
+saved outputs and goes to a TA.
+
The expected answers never leave the host: the container is told what to call, not what should come back (the same split as hidden_test_specs).
A rubric file declares the assignment, its point breakdown, and per-item config blocks. Expected values (Center City ZIPs, tolerances, the final answer) live here rather than in code so a TA can correct them against the official solution without touching Python.
Whether the sidebar should let a TA supply data for this assignment.
Assignment 3 does not require its zip — it is graded from the outputs each student saved — but supplying it lets the grader re-run every notebook and confirm them, so the option has to be there.
The notebook must not be run during grading, by any backend.
+
Assignment 4 scrapes Craigslist and downloads from OpenStreetMap and ArcGIS. Re-running a class of those notebooks would send every request again, once per student, to servers that are not ours — so the rubric opts out of execution entirely, whatever the TA's execution mode.
Raises RubricError when the file is missing or unparsable, assignment.id is absent, the rubric is empty, an item id is missing or duplicated, an item type is unknown, or the item points do not add up to total_points.
Flags are additive and explicit: a TA should be able to read why a submission reached the queue without opening the raw results.
With review_below_full_marks on (the default), every deduction goes to the queue: an automated deduction is a claim about a student's work, and a person signs off on it before it becomes a grade. Full marks need no defence, so a clean submission passes straight through.
Streamlit never calls the executor or the rubric engine directly (design.md §4): it hands a source and a config to this service and receives progress snapshots and a GradingSession back.
Execution limits, raised if the assignment declares it needs longer.
A rubric's execution_timeout_seconds is assignment knowledge, not a preference: Assignment 3 reads a 24 MB GeoJSON, joins 34,108 points and masks a ten-band raster twice, and cutting it short times out the whole class at once. So it acts as a floor on the configured timeout, never as a cap — a TA who raises the limit keeps their value.
Problems that would make a grading run misleading rather than wrong.
The big one: this assignment's students submit a notebook and nothing else, so without instructor-supplied data every notebook fails at read_csv and the whole class looks broken.
Grade every candidate and return the saved session.
Submissions are graded one at a time. progress_callback receives a GradingProgress snapshot at every stage (a callback that raises never stops grading); setting cancel_event stops after the current submission. preserve_overrides is a snapshot from GradingSession.collect_overrides to re-apply on top of the fresh automatic scores. Working directories are deleted afterwards unless GraderSettings.keep_workdirs is set.
Every step of Assignment 4 talks to a server that is not ours. Part 1 downloads the planning districts from ArcGIS Open Data and the street network from the OpenStreetMap Overpass API (osmnx), and Part 2 drives a browser through five pages of Craigslist with a sleep between each. Re-running a class of these notebooks would repeat every one of those requests once per student, which is exactly the load the assignment itself warns will get an IP banned.
+
+
So rubrics/hw4.yaml says execution: mode: never, and the service honours it in every execution mode, local included: no kernel, no notebook container, no browser, no data placement. See Execution. The grade comes from the file the student handed in, read two ways:
+
+
+
The code they wrote. Each code cell is matched to the template cell it was filled in from, and the template's own lines are set aside first. The driver.get(url) and pd.concat(...) inside the provided scraping loop, or a def format_price(price_string): return stub, earn nothing. Each step is a list of requirements in the rubric, one regex each, scoped to its Part, so a TA can widen one without touching Python.
+
The outputs their own run saved. The crash count, the scraping log (Processing page 3... Number of apartments = 120), the results table, the rendered histograms, the Folium map and the Altair spec, including the scraped rows the spec embeds.
+
+
+
The one thing that does run: three pure functions#
+
+
format_bedrooms, format_size and format_price turn "2br" into 2, and calling them is far better evidence than reading them. So they, and only they, are cut out of the notebook (grader.function_sandbox) and called with typical Craigslist strings in the Docker sandbox, under the same isolation flags as a notebook run, above all --network none:
+
+
+
only the top-level def of each named function is extracted. No other statement of the notebook goes anywhere, and the submission itself is not mounted into the container;
+
decorators and annotations are stripped, and a function whose default argument is not a plain literal is refused, because those run at definition time;
+
only imports of re, string, math, decimal, numpy and pandas go along. selenium, requests and osmnx do not;
+
the expected answers stay on the host. The container runs grader.function_runner, which is told what to call, not what should come back.
+
+
+
+
In local mode the functions are not called
+
The machine has a network, so item 2.5 gets a provisional score from the source and goes to a TA. The sidebar says so before the run.
A results table read back from a CSV proves nothing about a scraper. Item 2.6 wants the completed loop in the notebook and the log it printed when the student ran it, and the results table only counts when that log is there. A table loaded from a file is recorded as such and goes to a TA.
There is no reference solution, and the street network and the listings both change. The only numbers used are the template's own: about 3,750 crashes in Center City (3,000–4,500 accepted), 120 listings a page, and a price per square foot around 1.5. Each only raises confidence. A number far outside its range sends the item to a person; it never costs a point on its own. As everywhere, anything short of full marks reaches a person.
Crashes as a GeoDataFrame (latitude-as-x is named)
4
+
1.6
Crashes trimmed to Center City
5
+
1.7
Graph and crashes projected to EPSG:2272
5
+
1.8
Nearest edge for each crash
5
+
1.9
Crash count per street
5
+
1.10
Crash counts merged onto the edges
4
+
1.11
Crash index
5
+
1.12
Histogram of the crash index
3
+
1.13
Interactive map of the crash index
4
+
2.1
Selenium driver opens Craigslist
3
+
2.2
BeautifulSoup from the page source
2
+
2.3
Listing elements extracted
4
+
2.4
Price, bedrooms, size and title of the first listing
4
+
2.5
format_* functions, called in the sandbox
9
+
2.6
Scraping loop completed and shown run
7
+
2.7
Histograms of price and price per square foot
3
+
2.8
Interactive Altair scatter of price vs size
3
+
Total
100
+
+
+
+
Execution and output completeness is 10 points, Part 1 is 55 and Part 2 is 35.
+
+
The demo class (python examples/make_hw4_submissions.py) has four submissions: complete, one with the usual mistakes and its Craigslist results loaded from a CSV, one that crashes in Part 1, and the untouched template. Their saved outputs are written in directly; nothing is executed, there or when grading. In Docker mode they score 100, 84, 26.5 and 3. In local mode the format_* item is not called and goes to a TA instead.
Their student templates are in assignment_template/, but they are not graded yet. Each needs a rubric YAML and a grader class. See Adding an assignment.
@@ -307,7 +384,7 @@
Assignments 4–6Edit this page on GitHubMUSA Grader 0.1.0 · MUSA 5500, University of Pennsylvania
-
+
Assignment 4 scrapes Craigslist and downloads from OpenStreetMap and ArcGIS, so its rubric opts out of execution entirely. The service honours that in every execution mode, local included: no data is placed, no kernel or notebook container is started, and the execution stage is recorded as skipped. The grade comes from the submitted file's code and saved outputs. The only code that runs is a few named pure functions, cut out of the notebook and called in the network-less sandbox, and only in Docker mode. See Assignment 4 and grader.function_sandbox.
+
+
Errors do not stop grading. allow_errors is on, so the notebook runs to the end, the first failing cell and its traceback are recorded, and every check that can still run does. A failed notebook never becomes an automatic zero. The execution item itself awards partial credit for how far the notebook got: failing at the last cell is worth more than failing at the first.
Hidden tests normally run inside the student's own kernel, during the notebook run. Assignment 4's notebook is never run, so its hidden test is type: isolated_functions: the named functions are extracted from the notebook and called on their own in the sandbox. See Rubrics.
+
calculate_percent_increase(group_df) takes a DataFrame, so it can't be tested with literal arguments. Instead the probe reads the column names and date dtype of the student's own tidy frame, builds a synthetic group frame from interpolated anchor points using those names, and calls the function with it. The expected answer follows from the anchors, so it is known whatever data the student loaded.
The anchors deliberately imply a different answer for the first and last rows than for the two dates named in the assignment. That lets the grader identify the most common wrong implementation by name instead of just marking it wrong:
Optional. Leave it out and every submission is run in the execution mode the TA selected. Set mode: never and no submission of this assignment is run, in any mode, local included: no data is placed and no kernel or notebook container is started. The grade then comes from the submitted file's code and saved outputs, plus any isolated_functions hidden test. Assignment 4 uses it because its notebooks scrape Craigslist and download from OpenStreetMap and ArcGIS.
+
+
execution:
+ mode: never # grade the file as handed in; enforced in grader/service.py
Hints the in-kernel probe uses to recognise columns, plus limits on how much it reports. Hints are matched against column names, case-insensitively.
@@ -194,14 +208,14 @@
Item types
execution
Did the notebook run? Handled by the shared check_notebook_execution. Partial credit is proportional to how far it got, scaled by partial_credit_ratio.
structural
Checks on the source: imports, read calls, relative paths, functions defined.
deterministic
Checks on results: probe objects, saved answers, chart specs. This is the default type.
-
hidden_test
A function is called inside the student's kernel with inputs from the rubric. Expected values never leave the host.
+
hidden_test
A function is called with inputs from the rubric: inside the student's kernel, or on its own in the sandbox for isolated_functions. Expected values never leave the host.
qualitative
Needs a person's judgement, or an LLM if enabled. Always reviewed.
Two call styles are supported. scalar_args passes literal arguments. group_frame asks the probe to build a DataFrame from interpolated anchor points, using the student's own column names.
+
Three call styles are supported. scalar_args passes literal arguments. group_frame asks the probe to build a DataFrame from interpolated anchor points, using the student's own column names. Both run inside the student's kernel. isolated_functions runs without the notebook.
For an assignment that is never run (execution: mode: never). Only the top-level def of each named function is cut out of the notebook and called in the Docker sandbox with --network none; no other statement of the notebook goes along, and the submission is not mounted. Decorators and annotations are stripped, a function whose default argument is not a plain literal is refused, and only top-level imports of the modules in allowed_imports are sent with it. Each case's input is passed as the single argument; the expected values stay on the host.
In local mode the functions are not called, since the machine has a network. The item is graded provisionally from the source and goes to a TA, and preflight warns before the run. The code is in grader.function_sandbox and grader.function_runner.
Most item configs end in *_credit_ratio or *_penalty keys. These are the named near-misses: each is the fraction of the item's points awarded when the check recognises that particular mistake. They are policy, so tune them freely. The assignment grader reference documents which keys each check reads.
@@ -250,7 +284,7 @@
Validating a rubricEdit this page on GitHubMUSA Grader 0.1.0 · MUSA 5500, University of Pennsylvania