Jennifer Rosario (right), a social worker with Partnership with Children, teaches students from the South Bronx Academy for Applied Media to make smoothies.

An ambitious pilot to stem summer learning loss for low-income New York City students wrapped up its second year this month. But officials will have to wait a little longer before they can tell for sure how much it’s working.

When the pilot, called NYC Summer Quest, began last summer, city education officials wanted to test students to measure how much the gap narrowed by after finishing the five-week program. They asked schools to give the students an extra set of tests at the end and beginning of the school year.

But it didn’t work out as planned. Principals in participating schools quickly pointed out that June and September were too hectic to make their highest-need children sit for more tests.

“It was really hard on the schools,” said Ali Tan, Summer Quest’s program director at the Department of Education.

This year, Tan said the job of measuring academic progress is being handled by the department’s central staff and will be based on state tests that students already take April.

But the program’s success will also rely heavily on other metrics, such as attendance rates, parent engagement, and survey responses from students and parents who participated in the program.

Tan said the change in philosophy is among the hard lessons from the first year of implementation of Summer Quest, which launched a three-year pilot in 2012 with 1,100 students and $2.5 million in private donations. It expanded this year with an extra $2 million in public dollars.

The goal, officials said, is to transform summer school for students from high-need communities.

Summer school in New York City has traditionally been reserved for students who failed the state exams and face repeating their previous grade over again. But Summer Quest partnered with community based organizations like Children’s Aid Society and Good Shepherd Services to offer more than just a few hours of lessons in preparation for an end-of-summer test.

“It’s a totally different 21st-century model for what summer represents for our students and making sure that there’s both fun but that there’s also learning taking place,” said Chancellor Dennis Walcott, who visited many of the 11 schools where Summer Quest programs were held this summer.

The eight-hour daily schedule is also packed with enrichment activities. At South Bronx Academy for Applied Media, that meant baking classes and dance lessons before lunch, and field trips to the Metropolitan Museum of Art in the afternoon.

People who have worked on the pilot say the reduced emphasis on test scores is also a reminder of how challenging it is to evaluate extended learning time initiatives. They say they expect the benefits to accumulate over time, rather than suddenly appear after one five-week period.

“What our scholars are learning now is more internal,” said Andrea Lawrence, assistant principal at South Bronx Academy, which enrolled about 135 middle school students in Summer Quest this year.

That’s one reason officials aren’t alarmed that there isn’t yet academic data to show whether the program is working. They say having a place to go during the summer gives students a way to keep pace with their peers from more affluent families, who often pay to enroll in many different summer programs and camps in the summer.

Eighth grader Lyanna Marrer didn’t cite improved test scores as the reason she volunteered to return to Summer Quest for a second straight year. She said her alternative was “playing on my bed and watching TV.”

With the absence of hard academic data after one year of Summer Quest, officials pointed to other measures as proof of success during the first year. Families leave satisfied, surveys show, and parents often become more engaged in school activities in the next school year, officials said. Middle school students in particular reported feeling more connected to their schools and more confident handling more difficult work.

Summer Quest was originally designed to serve students who scored high enough on state tests to be promoted, but who officials said could still benefit from enrichment and academic activities over the summer. But this year the program also included some students required to attend summer school, and officials said they wanted to know how these students performed compared to similar students enrolled in traditional summer school.  In addition to state tests, Summer Quest participants will also take the city’s summer school exam, administered the first week of August, and then compared to similar students enrolled in traditional summer school programs.

Even though the results won’t be in for some time, people who worked on the program this year say the benefits are already detectable. Jennifer Rosario, one of 27 part-time social workers from Partnership with Children assigned to South Bronx Academy this summer, said that the smoothie-making class she taught is filled with moments where students learn new things. She recalled explaining the key nutrients of their newest ingredient, bananas, and giving students a primer on where fruits fit on the food triangle.

Sitting nearby with a small glass of a strawberry smoothie, Lydia Colon, an an eighth grader, said she literally couldn’t think of anything better to do in the summer than come to school for Summer Quest each day.

“In the beginning of the summer, I have nothing to do,” she said. “Otherwise, I’d be on the street.”

This story is part of a multi-city series on expanded learning time, with funding from the Ford Foundation, which supports “more and better learning time” in high-need communities. Also participating in the series are the Notebook (Philadelphia), Catalyst ChicagoEdSource (California), and GothamSchools’ sister site EdNewsColorado.