From Prototype to Precision: Refining Our AI Policy Generator

We invited Skylar Payne, a former student of mine. He is an AI engineer with over a decade of experience, having worked at Google, LinkedIn, and on FDA-cleared AI medical devices, among others. Skylar offered feedback on our current project and the prototype website. Skylar was very complimentary towards our students and their reasoning for the project, and he provided them with great feedback. 

1. Project Overview

• The tool aims to generate AI policy drafts based on user inputs. I worked on simple questions and answers that will generate a fully editable document.
• Due to a lack of existing policy examples from schools, the team pivoted from a scraping approach to a template-based, question-answer mapping method. Yash was able to program it to pull the sections from an individual section in the template document. 

2. Skylar’s Feedback

• Skylar praised the structured input/output method and emphasized dynamic in-context prompting to avoid baked-in bias from static prompts.
• He suggested mapping user inputs to tailored policy text that will give much more intentional output.
• He encouraged building a systematic quality assurance loop: track inputs and outputs, evaluate success as “yes/no,” and grow a set of good exemplars over time.
• He warned against over-relying on rating scales, which we originally thought about creating. He recommended using binary evaluations to reduce subjectivity.

3. Next Steps for the Team

• Reverse-engineer the exemplar policy: For each section, determine what specific question(s) would generate that text.
• Align existing brainstormed categories (e.g., AI literacy, privacy) with clearly defined user-friendly buttons or scale inputs.
• Create a reliable transformation logic that maps multiple user inputs into one output designation (e.g., mild/medium/spicy).
• Evaluate the prompt performance based on how closely outputs match the intended tone/content, refining questions and context accordingly.

4. Collaboration and Planning

• Skylar is open to reviewing code and contributing ongoing technical support.
• The team will create one question per policy section before the next working session.

Moving Forward

• We were pleased with the prototype. 
• We wanted to systematize the logic that links user input → prompt design → accurate, useful policy output.
• All refinements should be aimed at creating a repeatable, user-friendly experience for school leaders that doesn't overwhelm or confuse them.