Automated and configurable programming project checking with Chasten

Interested in learning more about this topic? Visit my teaching page for additional details: https://www.gregorykapfhammer.com/teaching/.

Transcript

1. Automated and Configurable Programming Project Checking with Chasten Daniel Bekele, Jaclyn Pham, and Gregory M. Kapfhammer May 15, 2025 PyCon Education Summit 2025
2. What Problem Are We Solving? Students may struggle to write efficient, readable code Manual project review is time-consuming and error-prone Many courses face these challenges: Data structures Algorithm analysis Software engineering Existing tools focus on style, not semantic structure Regex is brittle, and AST tools are hard to prototype Project Goal: Chasten enables scalable and structure-aware feedback that effectively supports both instructors and students PyCon Education Summit 2025
3. Avoid Time Complexity of O(n ² ) # O(n) is acceptable 1 seen = set() 2 for item in items: 3 if item in seen: 4 return True 5 seen.add(item) 6 # O(n²) is not okay 1 for i in range(len(items)): 2 for j in range(len(items)): 3 if i != j 4 and items[i] == items[j]: 5 return True 6 Goal: Automatically scan the source code that students submit to confirm that there are no inefficient looping constructs Challenge: Linters like Ruff and Pylint don’t have rules to detect nested control structures that either are or are not acceptable Build: An extensible tool allowing instructors to scan for arbitrary code patterns without detailed AST knowledge PyCon Education Summit 2025
4. Chasten to the Rescue! Uses XPath to search Python’s AST Rules written in simple YAML Structure-first, not just style Outputs to JSON, CSV, or SQLite Result: Instructors define checks once and use Chasten to easily apply them at scale across all student submissions - name: "nested-loops" 1 code: "PERF001" 2 pattern: "//For[descendant::For]" 3 description: "Detects doubly nested for-loops (e.g., O(n²))" 4 PyCon Education Summit 2025
5. Let’s Run Chasten! Install the Tool pipx install chasten # Install Chasten in venv 1 pipx list # Confirm installation 2 chasten --help # View available commands 3 Run Chasten chasten analyze time-complexity-lab \ 1 --config chasten-configuration \ 2 --search-path time-complexity-lab \ 3 --save-directory time-complexity-results \ 4 --save 5 Save results to a JSON file and produce console output Configure the return code for different detection goals PyCon Education Summit 2025
6. Results from Running Chasten Nested Loop Analysis Check ID Check Name File Matches PERF001 nested-loops analyze.py 1 PERF001 nested-loops display.py 7 PERF001 nested-loops main.py 0 Check ID → A unique short rule code (e.g., PERF001) Check Name → The rule name that matched (e.g., nested-loops) File → The Python file that the tool scanned (e.g., analyze.py) Matches → Number of times the pattern was detected in that file (e.g., 1 match) PyCon Education Summit 2025
7. Limitations and Future Directions Limitations of the Chasten Tool Doesn’t handle style, formatting, or type inference Not optimized for fast use in continuous integration Pattern matches through use of XPath on Python’s AST Empirical Study of Chasten Frequency of false positives or false negatives? How do students respond to the tool’s feedback? Differences in scores with varied feedback types? PyCon Education Summit 2025
8. Key Takeaways Write declarative rules for AST-based code checks Focus on bespoke code structure patterns in Python Automated grading aligned with learning outcomes Generate data-rich insights into student code patterns Try out Chasten and contribute to its development! GitHub: PyPI: https://github.com/AstuteSource/chasten https://pypi.org/project/chasten/ PyCon Education Summit 2025