You shouldn't need to write a scraper just to list the available electives

Thursday, July 24, 2025

My future alma mater is one of the best engineering schools in Canada (and in the world). But, you definitely wouldn’t think that by looking at the course selection webpage, course search page or student information system[1].

Waterloo's course selection list A list of every course offered by the school, but not a particularly helpful list if you’re trying to choose an elective as a first year student

One of my classes was missing from my schedule, so I logged into the course enrolment portal in search of answers.

Waterloo Quest class enrolment page

After clicking around at the menus for a few minutes, I found out that my chosen elective’s enrolment was “unsuccessful” due to a scheduling conflict, so I needed to find a new elective, fast.

The school publishes a “List of courses that you might be able to choose” for incoming first year students… but for some reason, they only list course codes, and expect students to visit the Academic Calendar and then individually search up each course code just to find out the course titles. This would take hours by hand!!

Waterloo's first year elective course codes How am I supposed to know which of these course codes interest me!?

Moreover if I want to see if a class conflicts with my schedule or is already full, I need to use the “Schedule of Classes” tool. It allows users to search a subject and see the classes, time and enrolment numbers, (among other information). Unfortunately, my brain simply can’t parse the information they present.

Apparently, I’m not alone. Because Waterloo publishes this 1,800 word long guide explaining how to read the table and understand its terminology.

I’m sure there are great legacy reasons to keep around this tool, but honestly, it’s overwhelming for new students like me. I’m not sure why, but I decided to write a web app which parses the official “Schedule of Classes” and presents it in a marginally more readable format for my purposes (and the idea was that, hopefully, along the way, I’d learn what all of the numbers mean)

I got relatively far with this…

… before learning that there’s an existing unofficial web service called ‘UW Flow’ that does exactly this, just way better, and with way more data (they’ve apparently been working on it since 2012).

UW Flow is great, but like the course search tool, it doesn’t solve my problems with browsing the first year appropriate electives (namely that I don’t know the names of the courses, just their course codes), so I began working on a second (and much less redundant) tool.

First, I grabbed the list of the course codes from the “List of courses that you might be able to choose”

Array.from(document.querySelectorAll('tbody td:nth-child(2)')) // fall 2025
  .flatMap(td => td.innerText.split(','))
  .map(code => code.trim().replaceAll(' ', '')) // SCI 206 --> SCI206
  .filter(code => code) // removes empty strings

// Array(173) [ "AFM101", "AFM123", "AFM131", "ASL101R", "ANTH100", "ARABIC101R", "ARBUS101", "BIOL130", "BIOL130L", "BIOL225", … ]

Then I scraped the data from UW Flow (sorry UW Flow maintainers!)

const courseCodes = [
  "AFM101",
  // ...
]

async function fetchCourseData(code) {
  const query = {
    operationName: "explore",
    variables: { query: `${code.toLowerCase()}:*`, code_only: false },
    query: `
      query explore($query: String, $code_only: Boolean) {
        search_courses(args: { query: $query, code_only: $code_only }) {
          course_id
          name
          code
          useful
          terms
          terms_with_seats
          ratings
          prof_ids
          liked
          easy
          has_prereqs
          __typename
        }
        search_profs(args: { query: $query, code_only: $code_only }) {
          prof_id
          name
          code
          clear
          course_ids
          course_codes
          engaging
          liked
          ratings
          __typename
        }
      }`
  };

  const response = await fetch("https://uwflow.com/graphql", {
    credentials: "include",
    method: "POST",
    mode: "cors",
    headers: {
      "Content-Type": "application/json",
    },
    referrer: `https://uwflow.com/explore?q=${code}`,
    body: JSON.stringify(query)
  });

  const json = await response.json();
  console.log(`${code}: ${json?.data?.search_courses.length}`);
  
  return json?.data?.search_courses?.[0] || null;
}

async function fetchAllCourses() {
  const courseDetails = [];
  for (const code of courseCodes) {
    try {
      const data = await fetchCourseData(code);
      if (data) {
        courseDetails.push(data);
      } else {
        console.warn(`No course found for: ${code}`);
      }
    } catch (err) {
      console.error(`Error fetching ${code}:`, err);
    }
  }
  return courseDetails;
}

fetchAllCourses().then(result => {
  console.log(result);
});

Once I had the course information, I threw together a webpage[2] with a sortable table listing the course names, codes, IDs, and their ratings from UW Flow… and just for fun, I added a dice button to randomly highlight a potential elective.

uw-firstyear-electives.varunbiniwale.com

My course listing page

This whole thing took under half an hour, and should save me (and anyone else) much more time than that.

To be honest, I’m not sure how other students were supposed to pick electives without either coding a tool or slowly losing their will to live in the Schedule of Classes. Maybe they visited each elective’s page by hand and put together spreadsheets. Maybe they picked the first thing that didn’t sound terrible. But for me, a bit of scraping, parsing, and duct-taped UI was easier than decoding the legacy interface that looks like it was designed for robots by robots with a deep hatred of undergrads.

Also, I still haven’t picked an elective. But at least now I can scroll past 172 of them faster.