Alumni speak of having joined Codesmith for many reasons (academic rigor, the tight-knit community, global network of engineers, to become founders, investors & CTOs) but many of these same alums would also likely tell you that transparent outcomes reporting via the Council on Integrity in Results Reporting (CIRR) was a major factor in their decision.
And while it’s difficult to reduce an engineer’s career status to salary or “days to get hired,” CIRR reports offer a set of crucial data that allow prospective applicants to look ahead to their lives after graduation.
With that in mind, CIRR was established in 2016 as a 501(c)(6) non-profit to set shared standards on reporting student outcomes, focused entirely on reporting accurate and transparent outcomes (i.e., placement, salary within a fixed time of 180 days from graduation, etc.) that are verified by an AICPA ‘peer reviewed’ audit firm.
The standards are extremely tough (full details below and on CIRR’s site), which can be demanding and time-consuming for participating programs (and thus over the years the number of programs reporting to CIRR has decreased). But given the importance of the information, we’d always prefer that they remain stringent and uncompromising.
In addition, CIRR is strict with how graduates are classified. For example, if a student decides after the program not to engage in a job search, they will count against the ‘percentage employed in field’ statistic. And while the typical standard for finding employment following traditional higher ed is a year, CIRR has chosen 180 days as a very rigid cut off.
It’s tempting for a program to develop their own, more flexible standards in order to adjust for edge cases that can lower outcome statistics (and many have), but we built Codesmith to be a space focused on engineering rigor and a tight-knit, supportive community, and part of that includes holding ourselves to these same high standards.
Every single student that has attended Codesmith is included in CIRR reports (cohort sizes have ranged from 13 to 37 since we started reporting). If a student indicates that they do not plan to search for a job or are not legally authorized to work in the country where they are living after graduation, they indicate that to the team prior to the program start date, and that percentage is included in the report. The default assumption if not indicated is that graduates do plan to seek employment within 180 days of graduation.
Who designed CIRR?
CIRR was founded by Skills Fund in 2016 (now Ascent) - Codesmith was permitted to join in 2019 (once it had 2+ years of data) and joined the board in May 2022. The latest standards were established in 2020.
How does Codesmith collect graduate’s outcomes for CIRR?
The Outcomes team at Codesmith prides itself on working hard to continually stay in touch with graduates of the program while supporting them throughout their job search. Most graduates will proactively let us know once they have signed an offer letter. When a graduate lets the team know an offer has been signed, they are sent a form to fill out that confirms their job information. This form is not affiliated with CIRR, but is used to build out the report later submitted to the auditing team.
Although most graduates report their outcomes proactively, there are always a handful of graduates that have not reported new employment to us. Typically, some of those graduates are continuing to job search, some have gone on to work in different fields, a small percentage have had family/life emergencies that have required them to step away, and typically a number of graduates have accepted full-time employment, but did not report their offers to us. As such, the team has a process to reach out to all alumni and uses this information to determine the final category as outlined within CIRR standards using Tracking Job Outcomes.
Codesmith recently reported our outcomes H2 2021 (for graduates that completed the program between June and December of 2021, and then completed a job search from January to June of 2022). During the data gathering process, we began by emailing all remaining graduates that had not informed us of an accepted offer. After the email had gone out, program coordinators that worked with the graduates reached out via text message to check in on their progress. As a final step, a senior hiring program team member does a final reach out to ensure we’ve done our due diligence to touch base multiple times with every single graduate of the program.
While the team is touching base with graduates, it is also worth noting that these conversations are part of ongoing professional development strategy check-ins with alumni and additional one-on-one hiring support sessions. Codesmith hiring support is for life, and the team takes that commitment very seriously.
How many cohorts are included in each CIRR report?
There are 3-4 cohorts included in each individual CIRR report (each campus will see 3-4 cohorts graduate in a 6 month period). Number of graduates within each report can vary, and can range anywhere from 70-140 people, depending on each cohort size.
CIRR results are reported by campus - we currently have results for our West Coast, Central, East Coast and Part-Time Remote campuses. For the full-time campuses (West Coast, Central and East Coast), CIRR reports will include 3 to 4 cohorts (with cohorts starting every seven weeks, there are 3 or 4 cohorts graduating within a 6 month period). For the Part-Time program, CIRR reports will include 1 to 2 cohorts (with cohorts starting every 12 weeks, there are 1 to 2 cohorts graduating within a 6 month period). The total number of students included in each report depends on how many students were in each cohort during that reporting period. Cohorts are typically around 34-36 students, so the full-time reports will usually range from approximately 102 - 144 students, while part-time reports will usually range from approximately 34 - 72 students. Note that in earlier reports (2021 and below) some cohort sizes were smaller, ranging from 13 to 37 students.
Why do older reports have different program names?
Codesmith updated the nomenclature for our campuses in 2022. Our CIRR reports will be titled accordingly:
LA Immersive → West Coast Remote Immersive
NY Immersive → East Coast Remote Immersive
Remote Immersive → Central Remote Immersive
Part Time Remote Immersive → Global Part-Time Remote Immersive
What happens when Codesmith cannot get in touch with a graduate?
While we contact every single graduate multiple times, there may be a small handful we do not hear back from. The default status in reporting is ‘not found a job’ even if the person was not job searching. CIRR does allow for programs to report jobs via the grad’s LinkedIn profiles if they were unable to get in touch with them. If no contact was able to be made by the team and LinkedIn shows that the graduate is employed full time at a verified employer, their employment status/title can be included in the report. If the timeline is noted by months on the LinkedIn profile, the start date can be included as the first day of the reported month. No salary information is included if reported via LinkedIn.
What happens when someone gets an offer after 180 days?
Over the years many graduates have obtained full-time offers after the 6-month mark. These graduates are not permitted to be included in the CIRR report. Even if someone’s offer is obtained at 181 days they do not count towards an offer. We saw this happen relatively often in the cohorts that graduated on/around March of 2020, when many graduates obtained offers shortly after 180 days, but could not be included in the CIRR report. While we, of course, want to celebrate these graduates' successes, we always respect the rigorous standards that CIRR has established.
Who is included in the “hired by school” statistic?
Codesmith deeply values team members who grow their careers from the ground up and have a first-hand understanding of the mission behind the work that we do. For this reason, we appreciate instructors who have completed the immersive program themselves and understand the student experience. When you see the percentage hired by school on a Codesmith CIRR report, these are a very small handful of graduates that have moved on to work full time as instructors or Software Engineers for the company. Note that fellows are not included in this portion of the report (this is a standard that Codesmith has set within our own reports, and is not a governing standard within CIRR as an organization).
How are fellows included in the report?
Codesmith fellows are graduates of the program that stay on to assist the team in core mentorship responsibilities for incoming students, similar to a TA in a traditional university. Fellows are considered to “graduate” the program once they finish their fellowship, typically after an additional 3 months (though some fellows do extend their contract and stay on longer, and a small handful grow to have long-term careers at Codesmith). Despite working on a 3-month contract after graduation, most fellows obtain roles shortly after finishing their contracted positions, and for that reason are included in the report with their original student cohort.
Can Codesmith remove graduates from the report for low test scores/academic performance?
No, Codesmith does not remove any graduates from the reports based on their academic performance.
How much of Codesmith’s results were done from fully-remote programs?
Codesmith’s programs have been running fully remote as of March 2020, and any programs completed after that time were done in a fully remote environment. It is worth noting that our part-time immersive program launched in February 2020, and currently has only a few cohorts with published outcomes for our part-time programming. We are excited and eager to share outcomes for other part-time cohorts in the coming months.
How does auditing work?
Every single CIRR report submitted by Codesmith is audited by a third party institution. Our audits are conducted by Banks Finley White & Co . CIRR specifies that auditors must be a ‘peer reviewed’ firm to be officially included in the report (https://us.aicpa.org/research/standards/peerreview/peer-review-summary). The firm follows peer-reviewed auditing standards (AICPA AU Section 150 and AU Section 350) to corroborate the outcomes and confirm information is accurately reported.
What is Codesmith’s relationship with CIRR?
All member schools are encouraged to nominate a team member to the CIRR board to contribute to its organization’s mission. Codesmith recently joined the board of CIRR (as of May 2022).
The reporting standards of CIRR were defined in 2016 and were last updated August 10, 2020.
Standards and the worksheet to develop a report are available on the CIRR website (https://cirr.org/standards). Historical standards are available for comparison.
What does Codesmith see as some of the downfalls of CIRR?
Although we see CIRR as critical to the importance of accurate reporting, any reporting organization will have its flaws. Outlined here are some of the downfalls of CIRR:
Does Codesmith track career growth for graduates after their first role?
CIRR is designed to track outcomes information for graduates only 180 days after completion of the program. With consideration that Codesmith hiring support is for life, we have recently published a report titled Where Are They Now to document (and celebrate!) the career growth of our graduates over the years. We will continue to publish these reports to track our graduates’ ongoing growth and success.
How should I interpret CIRR results?
CIRR results should be interpreted in comparison to a range of sources in education. Graduate schools increasingly provide outcomes insights (e.g. HBS, Yale Law, Stanford Engineering MS). Results should be interpreted alongside these reports, as well as other CIRR reports, other programs’ reports, and Codemith’s Where Are They Now report (report documenting graduate outcomes and career growth beyond the CIRR standard of 180 days post graduation).
It is also important to recognize variance year to year. Even in the absence of an actual trend, year-to-year comparisons in time series data will exhibit fluctuations that aren’t indicative of statistically significant changes. This is one of the reasons why statisticians prefer the use of moving averages when analyzing time series data.
Why is the number of graduates included in the report lower than the actual cohort sizes combined?
The CIRR section documenting the number of graduates in the report is only inclusive of those that intend to seek in-field employment within 180 days post graduation. The field “Graduates Included in the Report” does not include students who did not graduate the program nor students who did not intend to seek employment post graduation.
To read more about CIRR standards, visit: https://cirr.org/standards
If you have any additional questions, email us at email@example.com to let us know! We’d love to hear from you and will continue to update this article as new questions come in.
Last Updated October 27, 2022