As third block came to a close, students staggered to finish last minute presentations, put the final touches on their final papers and race through pages of their textbooks to study for their exams. But come dusk of Thursday, Oct. 18, they found themselves sans access to their e-mail accounts.
Confusion and bewilderment ensued.
“I mainly just couldn’t view replies about classwork from my professor and I couldn’t see e-mails to figure out my schedule for the upcoming weekend,” said junior Erin Slay. “I just waited it out, hoping for the best.”
The outage was initially caused by a hardware failure that broke communication to the backup systems, said Joseph Sharman, CC Systems Administrator. When the e-mail servers were unable to run backups, the hard drives eventually filled up and that caused problems for students accessing their accounts.
CC runs a full backup of all e-mail every night.
The problem was not quickly fixed and, throughout the weekend, student e-mail systems continued to crash.
“Once we learned of the problem Friday morning Oct. 19, we resolved it within 55 minutes,” Sharman said. “But, unfortunately we didn’t hear about it until after students had been affected much of Thursday night, Oct. 18,and early morning, Oct. 19.”
When administrators finally realized the problem had returned, they quickly corrected the system error.
“We heard a report Saturday morning, Oct. 20…,” Sharman said. “We resolved that by Saturday [afternoon] by freeing up space on a hard drive that had filled up again.”
The matters were finally resolved completely on Monday morning. Sharman called the sudden hardware failure “unexpected,” and David White, manager of the Help Desk, echoed this sentiment.
“Once we found out that there was an issue from the students, we checked in with our networking team and they were able to eventually resolve the issue,” White said.
On Friday morning, CC’s network vendor replaced the faulty hardware, which allowed backups to run again. Until the backups completed, however, the system continued to have problems with the hard drives filling up.
“One e-mail database’s backup had been corrupted by the hardware failure, and we worked a good part of the weekend, including working through the night Saturday, to complete a successful backup of that database,” Sharman said.
The issue of database space was resolved when the backups completed.
“As far as we know, no e-mail sent to student accounts was lost during the outage,” Sharman said.
No other parts of the network were affected, however, through the course of the failure, systems were not being backed up.
“We will purchase additional network hardware over the next few months to create more redundancy so that we should be able to withstand a hardware failure like this in the future,” Sharman said.
The second block third week e-mail failure was a lesson for all.
“We are implementing a new monitoring system that should give us early warning about potential problems and outages,” Sharman said, “Especially when they occur after hours when no one is working at the Help Desk to receive calls.”
This monitoring system is expected to be in place by the end of December.
Slay shares some advice: “[Students] should forward their CC e-mail to another email account so they [can] access their e-mail [in the event of another outage].”
“There have occasionally been other e-mail outages, but they were mostly caused by human error,” Sharman said.
Such human errors chiefly involve students or employees falling prey to Internet scams. Scammers in turn utilize accounts to send spam and take down the entire e-mail system.
In the instance the CC email system was to malfunction again, Sharman and White both recommend students first contact the Help Desk. Sharman suggests then contacting Tutt Library’s Reference Desk or Campus Safety to report the matter.
He places the utmost emphasis on calling only if the problem affects multiple people and system outages, not individual computer problems.
On a similar note, in the 2011-2012 school year, CCSGA instigated a new policy that allows all alumni to keep their Colorado College e-mail accounts indefinitely, as long as alumni log into their accounts at least once every six months. This in turn will give students 25 GB of mailbox space in Office 365, compared to the 1 GB space currently allotted for each student in Office Outlook.
Colleen Leong
Staff Writer