teaching machines

Rejection from NSF STEM+C

October 2, 2015 by . Filed under failures, public.

Some people share their successes. I prefer to share my failures! This gives a more realistic picture of life and this way I can share more often. I recently received a rejection for a proposal I submitted to the National Science Foundation’s STEM+C program, which aims to integrate computation into K-12 STEM education.

First is the panel summary which collects the reviews under one umbrella:

Section I: Intellectual Merit
Project description: Forty K-12 teachers (twenty each year) will participate in a fellowship program in which they will be trained in a four day session along with a cohort of pre-service teachers. They will leave with a 3D printer and the assignment to design at least two lesson plans. They will return mid-summer to lead a one-week Computing in 3D summer camp for middle and high schoolers, using lessons they designed. They will then implement similar lessons in their classes during the academic year.

Strengths:
The plan to integrate computing concepts into 3D printer-based activities is well thought out and innovative. It connects spatial reasoning with computational fluency in a novel way.

The fellowship program, which involves teachers as learners and summer camp facilitators, can be an effective way of empowering and supporting teachers as they learn ways to integrate computational thinking into their courses. It is a professional development model that has been used successfully by other programs in the past.

The proposal includes a sound review of the literature, particularly around spatial cognition. The project is inspired by Papert’s work with Logo.

Several panelists felt the proposal suggested multiple places in the regular curriculum where Madeup programming activities could be used to integrate computational thinking into the curriculum, for example, in areas of geometry, algebra, number, and proportional reasoning.

Weaknesses:
Some panelists felt the proposal did not adequately integrate technology into content that already exists in the regular math and science curriculum. The panel agreed that this goal might be better achieved if the focus was on a more narrow grade-level band, such as middle school, instead of the full K12 spectrum.

The questions listed in the Assessment and Evaluation section are essentially research questions. Regardless of whether they are considered research or evaluation questions, there is no detailed plan described for investigating them. Further, although a faculty member from another department is identified as the person who will serve as the external evaluator, nothing is included in the budget for this work. Methodology is not discussed.

It was suggested that the ITEST program might be another NSF program that would be appropriate for the project.

Prior NSF Work: No prior NSF support is listed.

Section II: Broader Impacts

Strengths:
The Madeup programming language might be a platform for teaching computation beyond current coupling with 3D printers.

Weaknesses:
The diversity of the group of teachers and students is not discussed.

Post-doctoral Mentoring Plan: Not applicable.

Data Management Plan: It was noted that the plan discusses secure data storage but not the protection of privacy. It also does not provide for making data available to other researchers.

Section III: Specific Review Criteria: Not applicable.

Section IV: Summary
The panel was divided on whether the project adequately integrates technology into content that already exists in the regular math and science curriculum, as is intended for the STEM+C program. Panelists agreed that there might be more success at integration if the project narrowed the band and focused on a few grades, perhaps middle school, rather than all grades K-12.

The panel felt that the proposal lacked a clear research plan and that the evaluation plan was not detailed. Further, evaluation was not included in the budget.

The panel has read and approved the panel summary.

Reviewer #1 rated the proposal very good:

In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.

Strengths:
This project has the potential to change K12 STEM teachers’ knowledge, dispositions and skills about computational thinking and provide the foundation for them to integrate computational aspects into their courses.

20 teachers evenly distributed over grade levels K-12 will be recruited in a fellowship program and trained in a four-day summer intensive along with a cohort or pre-service teachers. They then will lead a summer camp for students after a short interim period of self-exploration with the tools and technologies. This fellows model will serve to instruct, empower, confirm, support, and sustain the lead teachers as they travel through the program. During the school year, the teachers, now having received instruction, having developed and led activities with students in the summer, are ready to implement similar lessons in the classroom and share with their colleagues.

The content integrates computing concepts related to the syntax and language of coding to 3D printer-based activities (all teachers receive a 3D printer for their own classroom use). A specially developed coding language Madeup, already developed and trialed successfully in previous projects will be the basis of the computational aspects. There is a well thought out effort to embody both the cognitive (coding, representational objects) and physical domains (hands on manipulatives), which the proposers argue, the 3D-printer is good at connecting. The proposal clearly outlines how the language interfaces with STEM learning concepts and its extendibility to multiple subjects and levels. Curricular connections are clearly tied to CCSS. The rise in popularity and decreases in cost of 3D-printers, along with the cost-free Madeup programming language will help the program sustain and grow over time.

An external evaluation is described, conducted by a university faculty member outside the PI’s department. An evaluation plan includes a process and product evaluation that will document successes and detail how to replicate, if successful. Team seems to have the mix of expertise and experience (teacher training/cognition, technical, software) to carry out the design. There are letters of support from surrounding school districts. The budget is relatively low, with most funding going toward equipment (3D printers) and teacher support costs; there should be big bang-for-buck if the project succeeds.

Weaknesses:
There is no mention of the team having worked together (implied in section 4.5), would be good to hear of previous collaborations, particularly since they discuss building a community of practice (do they one themselves?). Nor any mention of previous NSF grants; would be good to note they have or have not received support.

While the research plan is detailed, the evaluation section is cursory, and while done by a qualified individual at the host institution (but a separate department), is not budgeted anywhere; this implies it will be done as either a favor, or an in-kind service, neither of which typically invests the time necessary to do a quality job. If considered for award, this element should be addressed.

There is no mention of how students will be recruited for summer camp, nor is there any budget reflection for the summer camp (perhaps it is an already existing camp at the campus?). Students will need transportation, support, food, materials. As this is a main component of the ongoing PD, its resources need be detailed.

In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to broader impacts.

Strengths:
Research plan is detailed and asks questions that will help the team and field understand the impact of the work, both the organizational structure, the PD, and the value of the technologies as integrated into STEM learning. A thematic breakdown of research topics (p. 13) is very helpful to see what they intend on investigating. Instruments are implied, but seem specifically developed for this project; as an exploratory effort, the need for validated instruments is not as crucial, but this might be addressed if considered for award.

The results will be significant in how teachers are recruited, trained, and then deliver computationally-based STEM lessons in a multitude of contexts; the teachers co-learn in an intensive environment, together with pre-service teachers, then lead students in a summer camp, then take these new skills into the classroom for in-school implementation. This will bring computation to many students and colleague teachers. The 3D printer is a convenient and current platform that presents multiple access points for computation and will serve the purpose of inviting students to participate in these activities. It seems that Madeup can be used with whatever hardware might exist now or in the future, so the 3D printer might be considered a placeholder for the purpose of this grant, rather than the end-all object.

Weaknesses:
There is no mention of recruiting teachers or students from diverse communities. While perhaps implied, there is the possibility that teachers and students who will show interest in these topics will come from communities and populations that are already STEM-forward. These processes described are exactly the type that might appeal to those students and teachers less STEM-oriented and so it is suggested concerted efforts be made toward inclusion of the underrepresented communities.

Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable

Data Management adequately describes and addresses multiple kinds of anticipated data, security, IRB, protection, access and confidentiality.

Summary Statement:
This exploratory effort by a team of qualified and new faculty will provide the field with a lot of information about models of integrating computation into STEM learning, developing computational skills that mix cognitive knowledge acquisition with physical, use of an innovative coding language, and how 3D printers and related technologies and products might be leveraged across multiple grade levels and topics. The proposal is clearly written; project design, research, PD all are carefully thought out.

The team is pledging to do quite a lot on a relatively low budget. This is an exciting proposal that should be funded if at all possible.

Reviewer #2 rated the proposal good:

In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.

Strengths:

Weaknesses:

In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to broader impacts.

Strengths:

Weaknesses:

Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable

None

Summary Statement

A good proposal. Investigator may want to consider other NSF programs like ITEST.

Reviewer #3 rated the proposal very good:

In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.

This project connects spatial reasoning with computational fluency in a novel and transformative way. The Madeup programming language potential can help teachers make the connection and negotiate between 2D and 3D space. As described in the proposal, Madeup appears to be different from other language because of the potential negotiation. There is a strong geometric and language connection in this project. The examples provided Figures 1 & 2 and the Common Core Standards support the connections. The PIs described a clear project plan and research design.

In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to broader impacts.

The research plan has the potential to impact mathematics teaching and learning by helping learners make connections between logic, mathematical properties and the negotiation between 2d and 3d space. The research questions are aligned with the project goals and Table 2 in the proposal suggests a clear connection between data collect and data analysis. Madeup potential can have an impact on geometry and spatial reasoning.

Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable

Summary Statement:
This project connects spatial reasoning with computational fluency in a novel and transformative way. The Madeup programming language potential can help teachers make the connection and negotiate between 2D and 3D space. The research questions are aligned with the project goals and Table 2 in the proposal suggests a clear connection between data collect and data analysis.

Reviewer #4 rated the proposal very good:

In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to intellectual merit.

Strengths:
The staff in the proposed project will collaborate with 40 teachers (20 per year) to integrate computation into their STEM courses. A primary tool will be Madeup, a programming language developed by the PI for generating printable 3D models using universal computational ideas, such as sequence, conditional logic, symbolic manipulation, and more. The proposal describes an innovative way to integrate computing into the K12 curriculum and seems to be very much in the spirit of the STEM+C program, as described in the Program Solicitation.

The Madeup programming language does not appear to have been widely tested, and therefore it might be risky to fund a project that depends on it. However, Madeup has been ‘informally validated in several areas,’ including with 100 students in an introductory programming class, so an exploratory project such as this is a very reasonable next step in its development, refinement, and dissemination.

The plan to use K12 teachers in summer camps is a strength of the proposal. Students will benefit from experienced teachers leading the classes, and it is reasonable to assume that the teachers will be more likely to implement their new lessons during the school year if they have had a chance to try them out in with project support during the summer.

There is a lot of interest in 3D printers lately, but not always a plan for activities that use such technology to support learning. In contrast, the developers of Madeup have designed activities that use a 3D printer to support the development of computational thinking
Weaknesses:
It is unclear how long teachers will meet in the intensive course. Page 1 says one week, but pages 5-6 says 4 days.

It might be unrealistic to assume teachers will spend a significant amount of time on their assigned task of designing several lessons plans during the month between the workshop and the summer camp. Is there time built in before the summer camp to review lessons and make adjustments if the lessons prepared are not of high quality?

Fellows will return home with a variety of cohort-reviewed lesson plans. It is not clear the process for this review or when it will occur.

In the context of the five review elements, please evaluate the strengths and weaknesses of the proposal with respect to broader impacts.

Strengths:
The project will impact in-service teachers, pre-service teachers, and students. It will explore specific activities that support computational thinking and will be a model to others of how computer learning can be implemented into the K12 curriculum. Middle-grade and high school students who participate in the computer summer camp might become excited about STEM and Computer Science.

Weaknesses:
The project expects that teachers will impact at least 50 students in their normal coursework. This might not be realistic for elementary school teachers, since they generally work with only one class of students, typically about 25, for the whole year.

Please evaluate the strengths and weaknesses of the proposal with respect to any additional solicitation-specific review criteria, if applicable

Summary Statement:
The proposal describes an innovative way to integrate computing into the K12 curriculum and seems to be very much in the spirit of the STEM+C program. Its primary tool is the programming language Madeup, which is developed by the PI and provides concrete ways to use 3D printers to support computational thinking. The project has the potential to benefit students, pre-service teachers, and in-service teachers.

With three very goods and one good, I’m thinking I should resubmit. Based on the reviewer’s comments, I will probably focus on middle and high school, I’ll make it clear that our external evaluator serves our campus full-time in this capacity, and I’ll need to strengthen the evaluation process.