The CSE department is committed to excellence in education in Computer Sciences and Engineering, Software Engineering, and modern application of computing. For that, our academic programs should always be aligned with the modern trends and standards of the discipline, and with the best recognized practices to produce reliable and successful professionals. Hence, we have established a continuous improvement process that is aimed at evaluating the performance of the CSE and the SWE programs to help us design and implement improvement actions needed to always stay aligned with this commitment. Moreover, in both academic programs, the continuous improvement process is perhaps more crucial because of the following reasons: 

• The Computer Science discipline and areas of application are rapidly and continuously evolving, not only in deriving  new knowledge but also in their impact in modern society. 
• We must always guarantee to our constituencies that our alumni are capable of fulfilling their professional duties in a satisfactory and reliable manner, and that they will  be able  to evolve in knowledge as the discipline evolves to be   able to continue their contribution to the growth of our society through the practice of the profession. 

In summary, the continuous improvement process is designed to guarantee that our programs enable our students to acquire a set of fundamental skills that every professional in the discipline must have.  Those skills are captured in the student outcomes of the program.

To lead this continuous improvement effort, the CSE department has created the Continuous Assessment and Improvement Committee (CAIC),  composed of : 

1. 1. Department Chairman
   2. Assessment Coordinator
   3. Administrative assistant

Since ours are relatively new programs, we started designing and implementing an assessment plan from the ground up. The plan itself has also been under continuous revision and improvement. In the following sections, we detail our continuous assessment and improvement plan to guarantee attainment of SOs and how we have used it so far to implement specific improvement actions in specific courses. We will also present some across-the-curricula improvement actions that we have recently implemented, or which are in the process of being implemented.

The curricula in our programs is the component that is responsible for the most in fulfilling the student outcomes of the program, our plan concentrates on the continuous improvement of all our courses, especially the core courses that are managed by the Department (courses with code CIIC or INSO). The plan is a two-component plan, as follows:

Table 1: Assessment Strategy Components

In the following subsections, we describe both components in detail. 

Direct Assessment, Evaluation, and Improvement of Core Courses

As previously described, each course has its own list of Course Learning Outcomes (CLOs), which represent the fundamental skills that the course will enable students to acquire, based on the particular topics and activities of the course. In addition, each CLO is mapped to one or more SOs that are directly impacted by the skills associated with the CLO. That mapping will allow us to assess not only how the course is performing in terms of its own CLOs, but also how it is performing with regard to the SOs that are being impacted. Hence, in the process of evaluating a course and determine improvement actions, the CLOs of the course are used as performance indicators. 

This approach has been attractive to our faculty as it simplifies the work required to prepare specific assessment activities that are aligned to specific CLOs, and through which the performance of the course can be measured. It also helps us maintain uniformity in a given course when multiple faculty member teach different sections, since the CLOs provide a framework to build common educational activities. The assessment data for all courses are evaluated by the CAIC to propose actions to the faculty of the CSE Department to improve the performance of the course.

Table 2 shows the case of one particular core course, CIIC 3011 – Introduction to Computer Programming 1. It shows each of the five CLOs for this course and the SOs of the program that each CLOs impacts. The expectation is that by attaining a particular CLO in the course, the student will be acquiring particular skills that contribute to one or more SOs as specified by the table.

Table 2: CLOs for Course CIIC 3011

When a course is under assessment, it is expected that the professors in charge define how the CLOs of the course will be measured and evaluated. That effort will be done in conjunction with the CAIC. If there are several professors in charge of multiple sections of the course, it is expected that they coordinate this effort, so that all sections use the same assessment instruments. Those instruments are activities that are to be graded and are part of the work used in the course to determine the final grade of the students. For example, they could be: 

1. 1. 1. 1. Quizzes
         2. Specific exercises in exams
         3. Home works
         4. Projects
         5. Laboratory work
         6. Written reports
         7. Oral presentations

Each of these instruments is targeted to measure a particular CLO, as well as the SOs that the CLO impacts. This is a measure of how the students that passed the course performed on each CLO based on the selected graded work. This assessment process will include at least one activity for each CLO. All the scores for each student are normalized to 100 as the maximum possible value on each such activity. Then, if there are m activities intended to measure a given CLO x, the maximum possible score that a student can get for that CLO would be 100m. The percentage corresponding to the total score reached by the student on those graded activities measuring CLO x is used to measure the level reached by the particular student on CLO x. Notice that this value is equal to the average of those m values that the student gets. The following illustrates this using some notation and formulas. 

For a student to be considered “in good standing for CLO x”, that final value  needs to be ≥ 70. 

The previous approach is also applied to SOs in a course to measure performance of the course in terms of the SOs that it contributes to. This will be described in more detail in a coming subsection. 

To obtain a final measure of the performance of a CLO x in the course, we use the following metric: the percentage of passing students that end in good standing with respect to CLO x.

Course maps are those for which the course is expected to contribute with the particular set of skills that the students would have acquired by attaining its CLOs. Table 3 details the actual mapping of the CLOs of core CSE courses to the student outcomes. The numbers in each cell are specific CLOs of the particular course that represent skills aligned to (or that impact) the particular SO that the column represents. A cell in blank means that the given course does not impact the particular student outcome. 

Table 3: Mapping the CLOs to SOs of core courses that are common in CSE and SWE programs.

Table 3.a: Mapping the CLOs to SOs of core courses that are unique in the CSE program.

Table 3.b: Mapping the CLOs to SOs of core courses that are unique in the SWE program.

Based on the tables, the activities used to measure the performance of CLOs are also used to measure performance of SOs, and that is done following the same approach. For instance, for the course CIIC 3011, the assessment activities used to assess CLOs 1 and 2 are combined to assess SO 4 on that course, and those used to assess CLOs 3, 4, and 5 are combined to assess SO 6. A student passing the course will be in good standing with respect to a particular SO jif the student score at least 70% in the corresponding graded activities. To finally measure the performance of a particular SO j in the course, we use the following metric: the percentage of passing students that are in good standing with respect to SO j.

In this subsection, we discuss the details of the cyclical process that has been established for the continuous assessment and improvement process for core courses. We also discuss the data collection mechanism and the assessment of the capstone course.

Fundamental Details of the Assessment/Actions Cycle

In order to guarantee timely and frequent assessment and continuous improvement of all the core courses, while minimizing the additional workload imposed on our faculty, we have divided our courses into four groups according to how each course is being planned to be offered in the program’s curricula. Each semester we focus our improvement efforts on only one of these groups, following a round-robin approach. Thus, all courses must have been assessed every two years. And during each such 2-year cycle, we will have the chance to implement proposed improvement actions  for every core course on each offering of the course as part of the continuous improvement process. Courses that are offered every semester will have four semesters to implement those actions before the next time that it is under direct assessment as per our assessment plan. Those that are offered every other semester, will have at least two semesters to implement those actions.

Based on our current programs’ curricula, each of the core courses needs to be offered at least as illustrated in the last four columns of Table 4. In practice, demand might be high, hence, the first two columns show if, historically,  the course has been offered both semesters of an academic year, or just one. The last four rows on the table show three courses that are not core for the CSE program, but are core to the Software Engineering Program (SWE), our other undergraduate program. These courses are INSO 4115, INSO 4116, INSO 4117, and INSO 4151. We choose to show them here for the sake of completion,  because our assessment process is integrated across both programs.

Table 4: Schedule for Core Courses

Legend: a cell in blank indicates that the course is not scheduled to be offered on the particular semester of the particular year corresponding to the column of the cell; a checkmark indicates that the course is planned to be offered on the particular year/semester and that it is a core course of both programs; the s means that the course is core only of the SWE program; the c indicates that it is a core course of the CSE program only.

Based on the table above, we divided the core courses into four groups. Each group will correspond to a set of four courses that are schedule to be offered on the same semester. The course groups, as well as the semesters in which they are planned to be under assessment are shown in Table 5.

Table 5: Schedule for Assessment of Each Course Group

In addition to those four groups, the capstone courses on each program managed by the CSE department (CSE and SWE) are going to be under assessment on every semester that they are offered (We say more about the assessment of the capstone course later on.).

Because we initiated our assessment of courses using a different scheme than the one being described here (as per changes approved by CSE Department on March 2019), we needed to realign our efforts in order to fall into the planned assessment cycle. Table 6 shows the courses as they have been assessed or will be assessed in 8 consecutive semesters from 2018 to 2021. The groups in the table are based on the groups previously described and as approved by the CSE Department in the Spring of 2019. In our original groups, CIIC 3075 was not part of Group 1, but CIIC 3081 and CIIC 4050 were. That is why those additional courses listed in the particular semester are shown with a “+” or a “–” as a prefix. The first case (+) indicates that the course was assessed in addition to those courses now in Group 1. The other (-) indicates that the course was not assessed during that semester. In Spring 2019, the table shows “+c3075” in addition to Group 2. That is because on that semester we were doing direct assessment of courses in Group 2, but we needed to add the course CIIC 3075 in order to synchronize with the new assessment cycle. Then, beginning on Fall 2019, all the assessment activities will be completely synchronized with the plan; that is, only courses in a particular group will be on direct assessment in a particular semester as per our assessment plan. The table also highlights that the Capstone course will be under direct assessment on every occasion that it is offered.

Table 6: Assessment schedule from 2018 through 2021

On a particular semester, our effort will concentrate in the group of courses that corresponds to that semester as per the plan. Assessment data and results will be collected by professors offering the particular courses. That data will be evaluated by the CAIC, in conjunction with the professors teaching the course. From that analysis, decisions are made for necessary improvement actions in each particular course. Those actions will be implemented on the successive semesters that the course is offered. The whole faculty will be informed of results and consulted on those actions that may require departmental approval.

Figure 1: CSE Assessment Plan

Figure 1 summarizes the general assessment and improvement cycle of courses per semester. The assessment process on each course of the group of courses being under assessment on a particular semester is as follows.

In essence, individual course assessment is fundamentally achieved by measuring the performance of the students passing the course in all of the CLOs of the course. Using that data, and the pre-established mapping from CLOs of the course to SOs, we also get the results measure their performance in terms of the SOs that the course is expected to contribute to. The CAIC, in conjunction with the faculty teaching the course, will analyze possible reasons for those one or two CLOs, as well as SOs, that exhibit lowest performance values. Then, an improvement plan, which may include modifying existing course activities or incorporating new ones, is developed to improve student performance on those weakest outcomes.

Departmental Retreats for Assessment and Continuous Improvement

The CSE department is committed to organize a faculty retreat at least once per year. The retreat will include in the agenda the following important tasks regarding the continuous improvement process:

1. 1. Report status on faculty participation in the assessment process for the two groups of courses that were under direct assessment in the two preceding semesters.
   2. Report assessment data that has been collected during the previous two semesters.
   3. Report results of the preliminary evaluation based on collected assessment data that must have been done by CAIC.
   4. Discuss and approve continuous improvement actions recommended by CAIC and faculty.
   5. Other new issues.

Some urgent issues may be presented and discussed in regular departmental meetings too. For that, every ordinary departmental meeting will have in the agenda an open item to discuss relevant issues regarding the continuous improvement process.

Data Collection Mechanism

The CAIC is aware that the participation of the faculty is crucial in this assessment process. And also, that the more the additional work the process imposes on that faculty the less encouraged will they be to effectively provide their assessment part.  With that in mind, the CAIC has prepared Google Sheet forms that the faculty can use to facilitate the submission of data for the courses under assessment. From that form, once the data is submitted, it automatically generates the results for each of the CLOs of the course based on what was previously discussed. Those forms will also show how each SO being impacted by the course is performing based on the data that is provided.

We have created a specific form for each course to fill in students data from the course activities. When the course is under assessment, that form is shared with the faculty teaching the course, so that they can enter the data once the semester is finished. We will see concrete cases of the results generated using these forms when we discuss specific results in a later section of this report. The complete form will be available for review, including those with specific data for courses that have already gone through direct assessment, during the days of the site visit or as requested by the program evaluators.

The CSE department is committed to excellence in education in Computer Science and Engineering, and its fundamental applications. For that, our programs should always be in alignment to the modern trends and standards of the discipline and with the best recognized practices to produce successful professionals in the discipline. To achieve that goal to our best, we have established a continuous assessment and improvement process that periodically measures the level of achievement attained by individual courses as well as by the curricula as a whole along a set of Student Learning Outcomes.

Here we present the assessment plan for the two undergraduate programs that are hosted by the CSE department. These are the BS in Computer Science and Engineering, and the BS in Software Engineering. This continuous assessment and improvement process is fundamental for the following reasons:

1. The Computer Science discipline is a rapidly evolving discipline, not only in deriving new knowledge but also in the impact that the discipline has in modern society.
2. We must always guarantee our constituents that our alumni are capable of fulfilling their professional duties in a satisfactory and reliable manner. And in addition, that they will be able to evolve in knowledge as the discipline evolves to be able to continue their contribution to the growth of our society through the practice of the profession.

Our continuous assessment and improvement process is designed to guarantee that our programs enable our students to acquire a set of fundamental skills that every professional in the discipline must possess.  Those skills are captured in the student learning outcomes (SLO) shared by the two academic programs. For both programs, the CSE department has decided those SLO to be the following:

(a) An ability to apply knowledge of computing and mathematics (in the solutions of problems…) appropriate to the program’s student outcomes and to the discipline

(b) An ability to analyze a problem, and identify and define the computing requirements appropriate to its solution, including designing and conducting experiments to this end

(c) An ability to design, implement, and evaluate a computer-based system, process, component, or program to meet desired needs, including:  economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability

(d) An ability to function effectively on multidisciplinary teams to accomplish a common goal

(e) An understanding of professional, ethical, legal, security and social issues and responsibilities

(f)   An ability to communicate effectively with a range of audiences

(g) An ability to analyze the local and global impact of computing on individuals, organizations, and society

(h) Recognition of the need for and an ability to engage in continuing professional development

(i) An ability to use current techniques, skills, and tools necessary for computing practice.

(j) An ability to apply mathematical foundations, algorithmic principles, and computer science theory in the modeling and design of computer-based systems in a way that demonstrates comprehension of the tradeoffs involved in design choices.

(k) An ability to apply design and development principles in the construction of software systems of varying complexity.

(l) A knowledge of contemporary issues.

Since the curricula in our programs is the component that is responsible the most in fulfilling the above student learning outcomes, our plan concentrates on the continuous improvement of all our courses. That improvement will be based on results of the periodic assessment of courses as established by the assessment plan. Each course includes specific student learning outcomes referred to as Course Learning Outcomes (CLO), which are often specific to the topics that the course covers. All CLOs for our courses have been incorporated into the corresponding course syllabi which are publicly available on our website to all our students and constituents.  To guarantee that all student learning outcomes are addressed by at least one course, each CLO is mapped to the list of SLOs that it impacts. The following table displays part of that mapping; the full version is found here.

The table maps each course to the program’s SLOs. In the second column, we have included the number of CLOs that the course in the particular row has. Also, for each programs SLOs, the table includes one column. The number on each cell corresponds to the number of CLOs in the particular course that impact the corresponding program’s SLO. White cells imply that the particular course does not impact the particular program’s SLO.

Finally, we expect that our students are able to show attainment of the program’s SLOs at least when they work in their corresponding capstone course. Therefore, the capstone course will play a crucial role in the assessment of the whole program. From the results obtained on that course, we will be able to decide further improvement actions that might be required in the corresponding curricula.  

In order to guarantee timely and frequent assessment of all courses, while minimizing the additional workload imposed on our faculty, we have divided our courses into six groups according to Computing subdisciplines. Every year  we focus our improvement efforts on only one of this groups, thus the faculty involved in the assessment and improvement efforts will mostly consists of faculty within one disciplinary area. Course groups are evaluated in round-robin fashion. Thus all courses must have been assessed and improved every five years. We have identified the following five course groups that partition our courses in subdiciplinary areas as follows:

The direct assessment of the courses was initiated on the academic year 2017-2018. From that year on, we will concentrate in doing direct assessment on those courses belonging to Group 1. And the process will be repeated in a Round-Robin fashion every five years. The capstone course will be under assessment on every occasion that it is offered. The next figure summarizes the cyclical process that we have implemented to assess the courses and the whole programs. After those first five years, the process is repeated starting again with the set of courses in Group1, each five years after 2017.

On a particular academic year, our effort will concentrate in the group of courses that corresponds to that academic year. Assessment data and results will be collected by professors offering the particular courses. That data will be evaluated by the CAI Committee, in conjunction with the professors teaching the course. From that analysis, decisions are made for necessary improvement actions in each particular course. Those actions will be implemented on the next semester that the course is offered. The whole faculty will be informed of results, and consulted on those actions that may require departmental approval.

The following figure summarizes the general assessment and improvement cycle per academic year.

The following boxes describe the one year long course assessment and improvement process in detail, including the time frame for each activity:

Departmental Retreat for Assessment and Continuous Improvement:

Inform results to all faculty and discuss general issues regarding the assessment and continuous improvement process. All faculty members will have the opportunity to propose and approve new general improvement actions to both courses as well as the improvement process itself.

From the above process, the part that is specific to the assessment and improvement activities during the two semesters is summarized in the figure that follows.

In essence, individual course improvement is achieved fundamentally by measuring the performance in all of the course learning outcomes by those students approving the course. A subset of course learning outcomes achieving lowest performance levels are selected as candidates for improvement.  Then, an improvement plan, which may include modifying existing or incorporating new course activities, is developed to improve student performance on these candidate outcomes.

2. Assessment Course Learning Outcomes using Graded Work – This is a measure of how the students performed on each CLO based on graded work. For each CLO, the report for the course will include how it was measured and what was the level reached on each. The “how it is measured” part is fundamentally decided by the professors teaching the course. It may include homeworks, quizzes, exam parts, lab activities, course projects, oral presentations, etc. The “maximum score” is the sum of the maximum scores on each particular activity used to measure the particular CLO. Finally, the most important part is the “level reached.” This corresponds to the percentage of passing students who scored at least 70% of the maximum score possible. This percentage of students will be the criteria used to determine how the particular CLO was met in the particular course.  This information is captured in form of a table, such as the following.

3. Proposed Actions for Improvement – This are the actions that are preliminarily proposed for improvement of the performance of the course on each particular actions. It is initially proposed by the professor(s) themselves, but it is expected to be analyzed in conjunction the the CAI committee to eventually determine the final actions. Those final actions will be the same proposed actions proposed in this table, perhaps with some additional actions which might be decided in conjunction with CAI.

We expect that there will be proposed actions for at least those CLOs that exhibit low performance, even if that is above the established threshold.

We measure the performance achieved on a particular CLO using the results of the graded activities that were selected by professors teaching the course as the activities to target that CLO. Those results are tabulated at the end of the semester using only the results of students passing the course.

We have established the threshold value of 70% as the minimum score that the student has obtained after counting his/her performance in all those graded activities. Therefore, any score representing less than 70% of the total possible score in the selected activities implies that the CLO was not reached by the particular student.

Using the data generated for each student on each CLO, we then determine what has been the performance of the particular course in regards to its CLOs. For this, we have also established the threshold of 70%. That is, a CLO is considered to be in good standing if at least 70% of those passing students have achieved a score of 70% or more in the graded activities targeting that CLO.

The following table summarizes what is described above.

Those CLOs that do not reach the good standing status will be considered as failing.

Courses in our programs are under constant assessment in order to measure how each course needs to be improved for successfully fulfilling the role that particular courses have in the whole academic program. However, the final determination as to whether our graduating students are reaching the needed skills that the PLOs imply, is based on how they perform in their capstone courses. Hence the capstone courses in both programs will be under assessment on each occasion that they are offered. In them, we measure how students perform in every program student outcome.