diff --git a/README.md b/README.md index 4470562..41071f3 100644 --- a/README.md +++ b/README.md @@ -191,6 +191,7 @@ graded by their effect: | two maps side by side | `Layout`, shape `(1, 2)`, children `[Polygons, Polygons]` | ✓ | | the same, panels never trimmed | `Layout` whose children are `HoloMap` | still carries its widget — 75% | | the yearly eviction trend | `Curve`, 14 points **summing to 149,472** | graded on the numbers plotted | +| the same trend with its points drawn on top | `Overlay` of `Curve * Scatter`, same 14 points | the same chart ✓ (an overlay with map layers is not) | The last row is the point: a chart is not checked for existing, it is checked for plotting the right numbers. @@ -211,6 +212,27 @@ is a deduction they have to re-derive: | suburbs = the envelope | its area is city + suburbs | `difference()` never applied, 50% | | tracts left in EPSG:4326 | the polygon's area is 0.037, not 369 km² | named as degrees vs metres, 40% | +### Calibrated against a real class + +The rubric was run over a full class's submissions and checked against +**per-item aggregates only** — no individual notebook was read to tune it. Every +pattern that came back marked down when the work was right was fixed in the +grader and reproduced in `tests/` with a synthetic fixture: + +| Correct work the first version marked down | What the grader now accepts | +|---|---| +| year columns relabelled `2003`..`2016` after the melt | `"e-03"`, `"2003"` and `2003` name the same year — in widget keys, tidy-frame columns and chart frame lists | +| a trend chart whose x axis is those integer years | the bokeh decoder never reads an evenly stepped integer column as the plotted totals | +| a line with its points overlaid (`Curve * Scatter`) | an `Overlay` is the trend chart, in the re-run and in the saved output; map overlays are not | +| 2.1.2 with no polygon area printed | both NDVI medians in the saved output (0.2025 and 0.3748) prove the city and suburb polygons — only the right polygons produce both | +| 2.2.3 histogram and map in one `plt.subplots(1, 2)` cell | that cell counts as both figures | +| 2.2.3 city limits drawn as a filled background layer | counts as the city outline | + +A chart with no recorded values also no longer crashes the feedback text. The +calibration scripts themselves (`hw*_aggregate.py`, `calibration/`) are in +`.gitignore`: even anonymised, they are written against real student results and +stay off the repository. + ### What still goes to a person One item, worth 3 points. The assignment marks the two street-tree figures on @@ -364,7 +386,9 @@ For a real class, download the ZHVI extract from by ZIP code* — and upload that instead. Nothing else changes. The sidebar takes the file by **upload**, so it does not have to live anywhere in -particular; uploads are kept in `.musa_grader_data/` and survive a restart. If the +particular; uploads are kept in `.musa_grader_data/` and survive a restart. Uploads — +data files and Canvas submission ZIPs alike — can be up to **10 GB** +(`server.maxUploadSize` in `.streamlit/config.toml`). If the file is already on the grading machine, the *"or use a file already on this machine"* expander takes a path instead. Either way the files currently in use are listed with their size and can be removed individually, and the app refuses to diff --git a/docs/_pages/assignments.html b/docs/_pages/assignments.html index 78c5415..2f66f9d 100644 --- a/docs/_pages/assignments.html +++ b/docs/_pages/assignments.html @@ -142,6 +142,7 @@

Plots are graded on the objects

Two maps side by sideLayout, shape (1, 2), children [Polygons, Polygons]✓ The same, panels never trimmedLayout whose children are HoloMapstill carries its widget. 75% The yearly eviction trendCurve, 14 points summing to 149,472graded on the numbers plotted + The same trend with its points drawn on topOverlay of Curve * Scatter, same 14 pointsthe same chart ✓ (an overlay with map layers is not) @@ -160,6 +161,24 @@

Named mistakes, not just wrong answers

+

Calibrated against a real class

+ +

The rubric was run over a full class's submissions and checked against per-item aggregates only. No individual notebook was read to tune it. Every pattern that came back marked down when the work was right was fixed in the grader and reproduced in tests/ with a synthetic fixture:

+ +
+ + + + + + + + + +
Correct work the first version marked downWhat the grader now accepts
Year columns relabelled 2003..2016 after the melt"e-03", "2003" and 2003 name the same year, in widget keys, tidy-frame columns and chart frame lists
A trend chart whose x axis is those integer yearsThe bokeh decoder never reads an evenly stepped integer column as the plotted totals
A line with its points overlaid (Curve * Scatter)An Overlay is the trend chart, in the re-run and in the saved output. Map overlays are not
2.1.2 with no polygon area printedBoth NDVI medians in the saved output (0.2025 and 0.3748) prove the city and suburb polygons, since only the right polygons produce both
2.2.3 histogram and map in one plt.subplots(1, 2) cellThat cell counts as both figures
2.2.3 city limits drawn as a filled background layerCounts as the city outline
+ +

A chart with no recorded values also no longer crashes the feedback text. The calibration scripts themselves (hw*_aggregate.py, calibration/) are in .gitignore: even anonymised, they are written against real student results and stay off the repository.

+

Where the 100 points go

diff --git a/docs/api/assignments.hw3.html b/docs/api/assignments.hw3.html index 267db14..e0b6fce 100644 --- a/docs/api/assignments.hw3.html +++ b/docs/api/assignments.hw3.html @@ -83,7 +83,7 @@
-

API reference · assignments/hw3.py · 2574 lines

+

API reference · assignments/hw3.py · 2645 lines

assignments.hw3

Assignment 3 autograder — evictions, code violations and the NDVI.

Graded against assignment_template/assignment-3.ipynb.

@@ -98,14 +98,14 @@

Where the evidence comes from

The matplotlib figures (the hex bin map, the two tree plots) have no such structure, only a picture, so those reuse grader/charts.py exactly as Assignment 2 does — which already merges the executed and submitted notebooks — and the one item the assignment marks on styling comes back for a person to confirm.

The two readings are combined in grade() by taking the better of them: a step counts as done if either shows the right answer. Neither is reliably better, so neither is ranked above the other; what differs is the confidence attached, and whether a TA is told which one carried the item.

Summary

-
NameKindDescription
HW3Graderclass
+
NameKindDescription
HW3Graderclass
keys_cover()functionEvery expected key is present, reading "e-03", "2003" and 2003 as one year.

Constants

-
NameValueNotes
WIDGET_CONTAINERS('HoloMap', 'GridSpace', 'NdLayout')holoviews container types that mean "one chart per value of a dimension".
LAZY_CONTAINERS('DynamicMap',)The same thing, but never evaluated: hvplot returns this when dynamic=False is left out, and it holds no frames at all.
LAYOUT_CONTAINERS('Layout', 'NdLayout')
+
NameValueNotes
WIDGET_CONTAINERS('HoloMap', 'GridSpace', 'NdLayout')holoviews container types that mean "one chart per value of a dimension".
LAZY_CONTAINERS('DynamicMap',)The same thing, but never evaluated: hvplot returns this when dynamic=False is left out, and it holds no frames at all.
LAYOUT_CONTAINERS('Layout', 'NdLayout')
OVERLAY_CONTAINERS('Overlay', 'NdOverlay')Layers drawn on one set of axes (line * scatter): one chart, not several.
MAP_ELEMENTS{'Polygons', 'Path', 'Shape', 'Image', 'QuadMesh', 'Tiles', 'WMTS'}Layers that make an overlay a map rather than a chart of yearly totals.
YEAR_KEY_REre.compile('(?:e-)?(\\d{2}|\\d{4})(?:\\.0+)?')Eviction Lab names its year columns "e-03" to "e-16", and many students relabel them as 2003..2016 after the melt. Both name the same fourteen years.

Classes

class HW3Grader(_SavedAnswerChecks, AssignmentGrader) - #[source]
+ #[source]

registered grader

No description.

@@ -115,27 +115,27 @@

Classes

Class attributes

- +
NameTypeDefaultNotes
assignment_id'hw3'
SUBMITTED_CONFIDENCE0.9Confidence for a step seen only in the student's own saved output: above the review threshold, so it does not flag a correct submission on its own, but below the 1.0 of a result this grading run produced itself. A saved output is the student's claim about their run — it can be stale, and it can be edited by hand.
NARROW_FRAME_COLUMNS12An Eviction Lab frame carries 399 numeric columns, and across 3,217 tracts one of them will sum to almost any total by coincidence. A result of a group-by or a merge is narrow, so width is the guard: it keeps the raw file out of searches that look for a frame by what its numbers add up to.
assignment_id'hw3'
SUBMITTED_CONFIDENCE0.9Confidence for a step seen only in the student's own saved output: above the review threshold, so it does not flag a correct submission on its own, but below the 1.0 of a result this grading run produced itself. A saved output is the student's claim about their run — it can be stale, and it can be edited by hand.
NARROW_FRAME_COLUMNS12An Eviction Lab frame carries 399 numeric columns, and across 3,217 tracts one of them will sum to almost any total by coincidence. A result of a group-by or a merge is narrow, so width is the guard: it keeps the raw file out of searches that look for a frame by what its numbers add up to.
PLOT_RECEIVER_REre.compile('([A-Za-z_][\\w\\.\\[\\]\'\\"]*)\\s*\\.\\s*plot\\s*\\(')Two different frames plotted onto one map: the trees, and the city limits drawn under or over them — filled as a background layer or as an outline.

Methods

answers(ctx: GradingContext) -> answerlib.NotebookAnswers - #[source]
+ #[source]

What the notebooks show, read once per submission and reused.

grade(ctx: GradingContext) -> list[RubricItemResult] - #[source]
+ #[source]

Grade on the objects this run produced and on what the file shows.

The two readings are independent and neither is reliably better, so they are combined by taking the better of them rather than by ranking them: a step counts as done if the executed run shows the right answer or the submitted notebook does. That is what stops a data path the grader could not resolve from turning a correct submission into a column of zeros.

charts(ctx: GradingContext) -> chartlib.ChartEvidence - #[source]
+ #[source]

matplotlib figures, read once per submission and reused.

Rubric checks 22

@@ -143,145 +143,152 @@

Classes

check_eviction_data_loading(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

3 ptsstructural1.1.1 Eviction data loaded with geopandasrubrics/hw3.yaml

PA-tracts.geojson is read with geopandas from a relative path.

Config keys: absolute_path_penalty expected_rows missing_read_penalty not_geodataframe_penalty required_imports row_tolerance wrong_size_penalty

check_philadelphia_tracts(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

5 ptsdeterministic1.1.2 Data trimmed to Philadelphiarubrics/hw3.yaml

The Pennsylvania tracts are trimmed to the 384 tracts of Philadelphia County. The file has no city column — the county lives in pl ("Philadelphia County, Pennsylvania") and in the 42101 prefix of GEOID.

Config keys: expected_rows geoid_prefix impure_credit_ratio incomplete_credit_ratio min_purity

check_tidy_evictions(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

6 ptsdeterministic1.1.3 Melted to tidy formatrubrics/hw3.yaml

pd.melt() over the fourteen eviction columns e-03 to e-16, keeping GEOID and geometry as id_vars — 384 tracts x 14 years = 5,376 rows.

Config keys: expected_rows expected_value_sum expected_year_labels missing_geometry_credit_ratio required_id_columns value_column_hints value_sum_rel_tol wrong_year_span_credit_ratio

check_evictions_by_year_plot(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

5 ptsdeterministic1.1.4 Yearly eviction trend plotted with hvplotrubrics/hw3.yaml

A groupby over the year column, summed, plotted with hvplot — any chart type. Graded on the numbers actually plotted.

Config keys: accepted_elements expected_max expected_points expected_total total_rel_tol wrong_library_credit_ratio wrong_values_credit_ratio

check_evictions_choropleth(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

5 ptsdeterministic1.1.5 Year-by-year eviction choropleth with a widgetrubrics/hw3.yaml

An hvplot choropleth of the tidy frame with groupby= on the year column and dynamic=False, producing a HoloMap with one frame per year.

Config keys: dynamic_map_credit_ratio expected_frames expected_keys key_dimension_hints no_widget_credit_ratio polygon_elements wrong_dimension_credit_ratio

check_violations_choropleth(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

5 ptsdeterministic1.2.7 Choropleth per violation type with a widgetrubrics/hw3.yaml

An hvplot choropleth with groupby= on the violation type and dynamic=False — a HoloMap with one frame per type.

Config keys: dynamic_map_credit_ratio expected_frames key_dimension_hints no_widget_credit_ratio polygon_elements wrong_dimension_credit_ratio

check_violations_loading(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

4 ptsdeterministic1.2.1 Violations loaded and converted to a GeoDataFramerubrics/hw3.yaml

li_violations.csv read with pandas and converted to a GeoDataFrame of 434,052 points built from the lng/lat columns, with a CRS set.

Config keys: bounds_tolerance expected_bounds expected_crs_epsg expected_geometry expected_rows missing_crs_credit_ratio not_geodataframe_credit_ratio row_tolerance swapped_coordinates_credit_ratio

check_violations_subset(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

4 ptsdeterministic1.2.2 Trimmed to the fifteen maintenance violation typesrubrics/hw3.yaml

The listed fifteen violation types, which cover 34,108 of the 434,052 rows.

Config keys: expected_rows expected_type_count partial_types_credit_ratio violation_types

check_hexbin_map(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

4 ptsdeterministic1.2.3 Hex bin map with census tracts overlaidrubrics/hw3.yaml

matplotlib's hexbin() over the violation points with the tract boundaries drawn on top, in a projected CRS rather than EPSG:4326.

Config keys: hexbin_patterns missing_overlay_credit_ratio not_rendered_credit_ratio overlay_patterns reprojection_patterns unprojected_credit_ratio

check_spatial_join(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

5 ptsdeterministic1.2.4 Violations spatially joined to census tractsrubrics/hw3.yaml

geopandas.sjoin() attaches a GEOID to each of the 34,108 violations. Every one of them falls inside a Philadelphia tract, so a correct join loses no rows and leaves no GEOID missing.

Config keys: expected_rows geoid_prefix lossy_join_credit_ratio requires_columns row_tolerance unfiltered_expected_rows unfiltered_join_credit_ratio

check_violations_by_tract(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

5 ptsdeterministic1.2.5 Violations counted per type and tractrubrics/hw3.yaml

A groupby over violation type and GEOID giving a three-column frame (violationdescription, GEOID, N) whose counts sum to 34,108.

Config keys: allowed_row_counts count_column_hints expected_count_sum expected_type_count partial_grouping_credit_ratio required_columns

check_merge_geometries(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

3 ptsdeterministic1.2.6 Counts merged back onto tract geometriesrubrics/hw3.yaml

pandas.merge() with the tract geometries first, so the result is a GeoDataFrame and not a plain DataFrame.

Config keys: allowed_row_counts not_geodataframe_credit_ratio require_geodataframe

check_side_by_side(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

4 ptsdeterministic1.3 Evictions and violations side by siderubrics/hw3.yaml

2016 evictions and a single violation type, drawn as two choropleths and composed into one row of two columns.

Config keys: eviction_total_rel_tol expected_columns expected_eviction_year_total not_composed_credit_ratio polygon_elements still_grouped_credit_ratio wrong_orientation_credit_ratio wrong_subset_credit_ratio

check_landsat_loading(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

3 ptsdeterministic2.1.1 Landsat scene opened with rasteriorubrics/hw3.yaml

The ten-band Landsat 8 scene is opened with rasterio.

Config keys: expected_array_shape expected_band_count expected_crs_epsg expected_height expected_width wrong_raster_credit_ratio

check_city_suburb_polygons(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

5 ptsdeterministic2.1.2 City and suburb polygons builtrubrics/hw3.yaml

The city limits polygon, and a suburbs polygon made by subtracting it from its own envelope. Both must be in the raster's CRS for the mask to work.

Config keys: area_rel_tol envelope_not_differenced_credit_ratio expected_city_area expected_suburb_area expected_suburb_geom_types missing_suburbs_credit_ratio unprojected_credit_ratio

check_ndvi_masking(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

6 ptsdeterministic2.1.3 Raster masked and NDVI computed for each arearubrics/hw3.yaml

rasterio.mask.mask() applied with each polygon, and NDVI computed from the red and near-infrared bands of the masked array.

Config keys: expected_city_median expected_city_pixels expected_suburb_median expected_suburb_pixels mask_patterns median_abs_tol one_area_credit_ratio pixel_rel_tol valid_range wrong_bands_credit_ratio

check_ndvi_medians(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

4 ptsdeterministic2.1.4 Median NDVI compared between city and suburbsrubrics/hw3.yaml

np.nanmedian() over each NDVI array, printed, with the suburbs coming out greener than the city.

Config keys: expect_suburbs_greater expected_city_median expected_suburb_median median_abs_tol nan_result_credit_ratio print_patterns right_direction_credit_ratio

check_tree_data_loading(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

3 ptsdeterministic2.2.1 Street tree data loadedrubrics/hw3.yaml

The 2,480 street tree points are loaded with geopandas.

Config keys: expected_geometry expected_rows raster_crs_epsg row_tolerance unprojected_credit_ratio

check_tree_ndvi(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

4 ptsdeterministic2.2.2 NDVI sampled at the tree locationsrubrics/hw3.yaml

rasterstats used to read the NDVI at each of the 2,480 points, giving one value per tree.

Config keys: alternative_patterns count_tolerance expected_count expected_mean expected_median max_nan_fraction median_abs_tol rasterstats_patterns valid_range wrong_values_credit_ratio

check_tree_plots(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

4 ptsdeterministic2.2.3 Histogram and mapped tree pointsrubrics/hw3.yaml

A matplotlib histogram of the NDVI values with a vertical line at zero, and a geopandas plot of the tree points coloured by NDVI with the city limits drawn on it.

Config keys: boundary_patterns detail_share histogram_patterns histogram_share not_rendered_credit_ratio point_plot_patterns zero_line_patterns

check_tree_plot_styling(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

3 ptsqualitative2.2.3 Figure styling (labels, legend, colour choices)rubrics/hw3.yaml

The one item on this assignment that a person has to look at.

The assignment asks for figures that are "clear and well-styled", which is a statement about a picture. The signals below say where to look; they cannot say whether the picture reads, so this always comes back as manual review with a provisional score rather than a silent mark.

Config keys: provisional_ceiling_ratio weighted_signals

check_common_violations_extra_credit(ctx: GradingContext, item: RubricItem) -> RubricItemResult - #[source]
+ #[source]

0 ptsdeterministicExtra credit: the twenty most common violation typesrubrics/hw3.yaml

The twenty most common violation types over 2012-2016 mapped as a second set of interactive choropleths, with three types named as having little spatial overlap with evictions. Worth no points in the rubric total; when it is found, the item is flagged so a TA can award the bonus by hand.

Config keys: expected_count_sum expected_frames min_discussion_words suggested_points

+

Functions

+ +
+
keys_cover(keys: Iterable[Any], expected: Iterable[Any]) -> bool + #[source]
+

Every expected key is present, reading "e-03", "2003" and 2003 as one year.

+
- +