[Infowarrior] - Collegiate Cyber Defense Competition

[Richard Forno]

A Student-Hacker Showdown at the Collegiate Cyber Defense Competition
Date: Mar 31, 2006 By Seth Fogie.
http://www.informit.com/articles/printerfriendly.asp?p=462526

Students faced off against experienced hackers at the Mid-Atlantic Regional
Collegiate Cyber Defense Competition. The students' goal: lock down
unfamiliar systems and secure their networks. The hackers' goal: to own the
students' networks and steal important data. Seth Fogie witnessed this
real-world competition and reports on its unexpected twists, turns, and even
drama.

Imagine if you just graduated with an IS degree and landed a job at a small
business as their only IT staffer. You know your way around an operating
system and understand some of the protocols and programs that keep data
flowing, but for the most part your skills are untested in the real world.
Regardless, you are the only thing separating the company's users and data
from downtime. Sound like a tough situation? Oh, I forgot to mention there
are four of the best hackers in the world trying to get into your digital
domain and steal anything of value, including a database of 10,000 credit
card numbers. This isn't something seasoned administrators would want to
face, much less fresh graduates.

Well, this is exactly what several groups of college students faced recently
at the Regional Collegiate Cyber Defense Competition, which was held at
several locations around the US. I was able to attend this three day event,
and this is my story. Get ready for some fun, shocks, and dohs! as you
follow along with me, the Red Team, and the students.
Collegiate Cyber Defense Competition

Before we get into the actual details of the event, it is important to
highlight the reason behind the competition. As per their website at
http://utsa.edu/cias/CCDC, "Unlike traditional 'hack and defend' or 'capture
the flag' contests, this competition tests each team¹s ability to operate,
secure, manage, and maintain a corporate network. This competition is the
first to create, as closely as possible, a realistic corporate
administration and security experience ‹ giving the competitors a chance to
compare their education and training against their peers and the real world
challenges that await them."

In other words, the competition is about creating a practical experience
from the classroom knowledge. Students simply take what they have learned
and apply it in a simulated environment for educational purposes. However,
the competition aspect helps schools know where they stand in relation to
other schools that are teaching related content. It is important to note
that the program is funded 'in part' by the National Science Foundation
(Award ID 0501828 
http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0501828) and has
paid out a little over $1,000,000 so far to create "problem-based and
case-based learning methodologies in order to provide students with
activities that simulate real-life work experiences <with a focus on
security>." I suggest you check out the above link to see the details of
this award.

The contests are first held regionally, and the winner of each regional
exercise competes against each other at the national finals in San Antonio,
Texas. I attended the Mid-Atlantic Regional Collegiate Cyber Defense
Competition, which was hosted in Lancaster, PA by White Wolf Security. The
five schools where from the PA/VA/MD area and were a mix of two/four year
degrees that range from networking to programming. The participating schools
were:

    * Anne Arundel Community College
    * Community College of Baltimore County
    * George Mason University
    * Millersville University
    * Towson University

White Wolf Security

The host and operator of the event (White Wolf Security) operates a training
facility that serves as a working-training environment for hands on computer
lab-based education. Everyone, from local colleges to the US Secret Service,
use White Wolf's equipment as a hands-on lab for training events and other
related activities. The owner, Tim Rosenberg, is well known in
educational/government circles for both his lab (which is mobile) and for
his training events.
The Details

There are several different groups involved with the games. Each has a
function, and a tag. The white cell is there to keep the games running
smoothly. Gold members are the judges and professors who basically monitored
the games from afar. The red cell was there to attack, and finally, the law
enforcement was there to arrest the hackers.
The Scoring

At the beginning of the game, everyone starts with zero points. If the team
can keep all the services open, and a selection of 'target' files available,
they keep that zero. However, if a team suffers from a loss of
confidentiality, integrity, or availability in any other way, they start
collecting points ‹ sometimes quite rapidly. Figure 1 is a shot of the
scorebot system, and backend components of the network.
Figure 1

Figure 1: Scorebot system
The Feds

One particular aspect of this game that added significant value was the
inclusion of a reporting process to the authorities. On hand was a real US
Secret Service agent who deals with computer-related crimes on a regular
basis. His job in the game was to show up on scene when a student team
detected an attack. If the incident report was filled out correctly (see a
real incident report from USSS web site:
http://www.secretservice.gov/forms/form_ssf4017.pdf), the team would get
points taken away from their growing score. This aspect to the game was
actually one of the most valuable as one who has had to deal with the
authorities before. Not only will this experience give each of the students
something to look back on if they ever have to deal with the government for
real, but they also now have someone they can talk to if something does
arise.
The Network Layout

The network was separated into seven different subnets. Five were split up
between the teams, one was used for scoring systems, and the final was for
the Red Team. At the edge of each of the student's networks was a router and
firewall, which were off limits until the second day and third days,
respectively. Finally, each school had four servers in a DMZ that were
connected to the firewall, each with a 'public' IP and a specific purpose.
In addition to this, they had two workstations that were in the protected
area of the network that were used for syslogging and other functions. The
following breaks down the initial system setup ‹ try not to grimace.

    * Alpha1 ­ Windows 2003 Server running IIS 6.0 (HTTP/HTTPS), MYSQL and
OSCommerce (with PHP support).
    * Alpha2 ­ Fedora Core 4.0 with VSFTPD and DNS (BIND)
    * Alpha3 ­ Fedora Core 3.0 with SSH
    * Alpha4 ­ Windows 2000 Server with IIS5.0, MYSQL, Telnet, SMTP/POP, and
DNS (Secondary) running an HR database.

Each system and program was of an unknown patch state/version. In addition,
there was a network IP camera thrown in for grins and giggles. Welcome to
most small IT shops where money is tight and time is valuable. Figure 2
provides a look at an unmanned pod. Note the four monitors that are
connected to the stack via KVM.
Figure 2

Figure 2: Unmanned pod
Business Objectives

To make the games a bit more realistic, each team would receive various
business objectives that would have to be completed in due course, or they
would lose points. This could be something as simple as add an email account
or even install PGP. The details of the objective were up to those running
the games.
The Rules

The rules were fairly simple ‹ at least at first glance. Basically, the Red
Team could do anything but hurt someone or perform a denial of service
attack (network flood). The student teams were a bit restricted, with regard
to changing IP addresses and messing with the infrastructure.

Communication was allowed between team members, but only the team leader
could talk to the white cell members about problems, etc. The feds could be
called over for an investigation and the Red Team was allowed to try to talk
to the teams to put a social engineering twist on the games. Finally, all
business objectives and administrative requests are sent to the CEO via
email.

There are some other rules and regulations that are laid out in some detail
(http://student.ccbcmd.edu/~cobrie12/ccdc/docs/
Mid_Atlantic_Regional_Team_Packet.pdf). However, for the most part, the
rules were fairly loose to allow some dynamics in the game.
The Competition

The event kicked off around 1PM on a Friday afternoon. All the students were
sitting at the 'pod,' waiting for the green light. The Red Team was all set
to go in a separate room with their equipment. After some general
announcements, Tim Rosenberg introduced the students to their mission. "Your
job is to keep the services up, the router routing and keep the store open ‹
as well as everything else". After some brief descriptions as to what and
who was involved, he gave the green light. At this point, the students had
three hours to figure out what they had just inherited from 'the previous IT
person' and fix it. Meanwhile, the Red Team was set loose to discover just
who was out there and figure out what they were running. They were not to
attack anything until after the three hour limit was up. However, the term
'attack' is very grey and seemed to include rooting routers and firewalls.

When the teams were set loose I positioned myself in the red room to see how
the initial information discovery process would go. It was at this point one
of the Red Team members stood up, kicked everyone out, and locked the door.
Fortunately, I was labeled as trustworthy and was able to stay inside. He
next reached inside his bag and pulled out a complete description of the
student's setup, including all operating systems, services, web
applications, and IP addresses he had obtained from an anonymous source.
Everyone in the room immediately got a slightly evil grin on their face as
they realized the results of this social engineering reward. Oh yes ‹ things
were about to get very bad for the students. Figure 3 gives you a shot of
the Red Team in action.
Figure 3

Figure 3: Red team in action

After the disclosure of this damning piece of information, I stepped outside
the room to see how the students were managing. Ironically, the students at
this point knew less then the Red Team about what was running on their
systems. Once again, fortune shined down on me because I happened to know
one of the school¹s teams leaders. After a short catch-up (I knew him from
high school), I started to ask what he knew about the competition and what
his students were dealing with. As it turned out, his team was all
programmers who jumped into the event at late notice. That said, they seemed
to be very busy figuring out what they had to fix and seemed to be fairly
astute as to what they needed to do. I saw kernel recompiling, service packs
being downloaded and installed, account permissions being locked down, and
much more. In fact, as I looked around the room, all of the teams seemed to
be in a frantic rush as they tried secure their VERY insecure systems.

I walked around from team to team and quickly realized that no one trusted
me, which is a good thing as social engineering was allowed. After an
introduction ("I am press") and assurance I was not going to tell the Red
Team anything, they allowed me to be near, but still kept one eye on me and
the other on what I was looking at. Paranoia had set in. Since the first
three hours were critical to their success, I decided to keep my distance
from the teams and watched from the sidelines. Considering the feat they
were trying to accomplish, they did not need me interfering.

During this time, I was able to talk to several of the team leaders, who
were not allowed to interact with their teams. Most of them are college
professors (PhD types) who wanted to expose their teams to some real world
experience. Since the bill was paid for by the grant, there was little to
lose and much to be gained by joining the competition. In fact, I am pretty
sure there will be at least one school that will be including a class on
router configuration.

Back in the red room, the Red Team was working hard at 'information
gathering.' This involved scanning the systems with nmap and popular GUI
applications from Windows. After looking at the results, it was pretty
obvious that the students had some serious issues to address. However, it
was equally as obvious that the much of the content of the Red Team's
information packet was going to be learned by the schools in the first half
hour ‹ except for the default passwords.

Since the Red Team knew the default passwords for most of the accounts and
services of the running servers, they had logged into each of the teams
routers and changed the default password to something a bit more hard to
guess. They were also logging into the Linux servers via SSH and changing
account passwords, plus doing a little system level recon to see what kind
of vulnerabilities they could use to raise their newly acquired accounts to
root level access. Some might call this active hacking, but the lines were
not that clearly drawn, which leaves much to 'interpretation.'

This type of network and system recon continued for roughly three hours,
during which time I bounced between the red room and student pods. There
were several hiccups in the process, such as an overload on a power circuit
that led to a complete loss of power for two teams, but that just made the
event that more realistic IMHO. As the teams started to get things under
control, they acknowledged my presence and started to talk a bit. I learned
that most of the students were expecting to be slaughtered when the Red Team
was set loose. I personally agreed. Even if everyone on the team was an
experienced veteran, there was no way they could lock down everything in
three hours.
Don't Let Your Momma Dress You

When I first entered the White Wolf Security lab, the first thing that
caught my eye was a team that was wearing all blue. Everyone else was in
typical student attire that mostly consisted of jeans and a t-shirt.
Ironically, this attempt at professionalism turned out to be a bad idea
because the Red Team also noticed the blue shirts. The result: 'the blue
team' became target number 1.
Let the Games Begin ‹ Day One

When the three hour grace period was over, the Red Team slowly worked their
way into attack mode. One member started to sort through the information
they gleamed from their scans and investigated each possible exploit.
Another member fired up a MySQL database client and started to poke around
the students databases looking for sensitive data. The two others were
adding/changing accounts to routers, firewalls, and systems. However, for
the most part, the students were not being pelted with attacks. And this
continued for the next several hours.

One interesting event occurred during this first stretch that warrants a
mention. As it turns out, a team detected that their router¹s default
password did not work. They corrected this problem by uploading new configs
to the router, which gave them control again. However, a Red Team member
realized what happened and decided to find another way into the device. It
took a few minutes, but they quickly learned that the router had SNMP
enabled and allowed read/write access for public and private. The result was
that the attacker used 'private' community access to add a new account to
router. Once again, this activity was detected by the students, at which
point they attempted to completely secure their router. Unfortunately for
them, they messed up this process and inadvertently took themselves out of
the game. Since the router is the doorway to their servers, the scoring bot
had no way to tell if their servers were running. The point to this is,
killing your device might keep an attacker out, but it also keeps valid
communications from occurring.
Day Two

Saturday started out slowly, but by the end of the day things would not be
looking good for most of the teams. With one team pretty much out of the
picture thanks to a router issue, the Red Team really focused on the 'blue
shirts'. Their first target was an OSCommerce application that was running
on one of their Windows machines. Unfortunately, the blue shirts forgot to
change the permissions for the admin directory on the application. As a
result, the Red Team had complete access to the configuration manager
portion of the application. This not only gave them access to all the order
information that included 10,000 credit cards, but also gave them access to
a file manager application that allowed them to upload/download/edit files
on the system, which they did.

One of the members on the Red Team decided to make the ownering of this
application obvious, and renamed the Title of the OSCommerce site to
something like 'Welcome to Tim Rosenberg¹s School of UDP.' Unfortunately for
the Red Team, this was quickly spotted by the students who then started to
look at how and why this happened. Meanwhile, the Red Team member had also
defaced their home page, which the students again spotted. Access to the
admin folder was soon disabled, but the damage was done ‹ integrity was
lost, services were denied, and confidentiality was gone. The blue shirts
were able to detect and report the web server defacement to the authorities,
but they missed the customer information download. Since web defacement is
minor on the scale of attacks, the Red Team was only given community service
instead of the felony charge they could have been hit with. The end result
is that the color blue was a good pick for this team as depression sunk in.

The Red Team did spread the love around a bit after pummeling the blue
shirts for a few hours. They discovered an unprotected HR program that was
loaded with SQL injection vulnerabilities. This was then used to
download/alter employee data, which represented a major loss in
confidentiality and integrity. However, it was what the Red Team did after
this that was quite clever.

Using some custom code, the Red Team created a SQL injection query that then
connected to another team's web server in an attempt to create a denial of
service attack. This odd attack forced one of the teams to approach the
other team with an apology that went something like this: "Hi. Uh, I am
sorry if we are attacking you, but we aren't really doing it." The DoS was
stopped soon afterwards upon request of the judges.

At about 2PM the second day, attention was shifted to one team in particular
because they had managed to stay out of the limelight. It was soon
discovered that this team had failed to change their postmaster email
password, which gave the Red Team full control over the emails coming and
going to the server. Various methods of abuse were discussed, but it was
concluded that the best thing to do was to change the password on the
administrator accounts, create a new account, and forward all email from the
CEO email account to the Red Team¹s account. Once this was set up, the Red
Team was told to take a break because the student teams were getting
overwhelmed. Thus the attacks stopped, which gave the students time to focus
on reporting incidents and securing their systems from the various attacks
that had been occurring during the day.
Confusion Techniques

One of the interesting tricks that the Red Team did to keep the students
guessing was to run continuous scans from programs like Nessus. They did
this for one reason ‹ overload the students. In addition to indirect
misinformation, the Red Teams also employed tools like mucus, which have no
other purpose but to trigger IDS alerts. They also noted the use of ethereal
and injected malicious packets into the network that would crash the sniffer
and cause general havoc. It is important to note these techniques because in
a real attack, it is not only possible for this to occur, but even probable
‹ especially if the attacker knows you are watching.
The Hacker Mentality

As I sat in with the Red Team, I got to watch how each worked and what
tools/methods were used. The four members really represented a wide range of
skills and personalities. On the one side was the professional information
assurance who really knew the technology and was able to assist with various
tasks that led to loss of confidentiality. Another member was very prepared
with a rack server that had six CPUs. His system was loaded with programs
like Canvas, metasploit, and other automated penetration testing tools. Next
was a member who blended the casual professional with rule bending hacker.
He proved to be a valuable asset and was the one who coded up the script
that performed the SQL denial of service attack. The final member was the
pony tail/ear-ring type that really stretched the rules and thought nothing
of it. As a career penetration tester, his skills were valuable for the team
as well. The point is, each member took a different approach and used
different tools to get the job done. They used everything from OS X to Linux
and even Windows, not to mention intimidation, ladders, and glow sticks
(more on that in the next section).
Day Three

I arrived early on day three with an understanding that there would be one
more hour of active Red Team hacking. The rest of the day was set aside for
some competition between the students to allow them some Red Team action.
However, to my surprise, I arrived to find all the students in a panic.
Apparently, someone had messed with the computer systems during the night
and no one would fess up as to who had done it or what was done.

While we all waited on the Red Team, it was discovered that during the night
the forwarded CEO email account had intercepted two emails from one of the
student teams. Unfortunately for them, it contained all the user/passes for
every member of the team. As a result, the present Red Team member was able
to log into the OSCommerce site and download the customer database and
access the accounts on the SSH server, not to mention anything else that
required an account. Of interest, the Red Team was not able to use the file
manager in OSCommerce to upload/download files because the students had only
allowed read/execute access to the admin directory. It was at this time that
the Red Team arrived and explained what had happened.

To keep the games interesting, and provide a bit of a educational anomaly,
the Red Team had done what any criminal hacker would consider ‹ they broke
into the teams¹ pods and installed backdoors. Using only the light from a
glow stick (the hotel they were staying at didn't have any flashlights),
they found a ladder, climbed up the outside of the room (12 foot ceilings),
pulled back a drop ceiling tile, and climbed down a wooden rod they
collected from nearby. With physical access granted, the Red Team went to
town.

Rootkits, backdoors, password changes, system configuration changes and more
were fair game with no one around to stop them. One team had locked down the
KVM device with a password, but this was quickly bypassed by plugging the
monitor into the actual computer. Another team used BIOS password
protection, but again, a quick short of the CMOS and the BIOS flash was
reset back to default. Windows administrator accounts fell quickly to boot
disk based password reset attacks. Root account was gained by 'single user'
mode hacks on the Linux machines. From there, log files were deleted, PHP
scripts were embedded in programs, backdoors were installed, accounts were
created with root level access, and much more. Simply put, the Red Team
owned the students through and through. The only way anyone could
realistically recover is if they took everything offline and started from
scratch, which is exactly what a business would have to do. In fact, in a
real case, the feds would probably ask to take the systems as evidence.
The Summary

After a brief break for lunch and some major discussions about the games and
the physical break in, the Red Team gave a short talk to the students about
what they did. Not including the physical attacks, 90% of the issues were
related to default passwords. The remaining problems were related to bad
code. They also brought up the blue shirt affect and that avoiding attention
is a great technique to staying out of harm¹s way. It was also discovered
that most of the teams were expecting serious 0-day attacks that they would
have to find and stop, when in reality telnet, SSH, and a web browser were
the primary weapons. The winning team was actually the one that kept their
router running (two teams hosed themselves on this issue), changed most of
the default passwords, locked down their permissions, and didn't attract
attention to themselves. Oh, and of interest, it was the same team that had
only a week to prepare and were all programmers (as seen in figure 4).
Figure 4

Figure 4: The Winners!

The end result was that a group of students got a first-hand experience of
just how bad it can be in the real world, and what they would need to do if
they ever had to deal with a similar scenario. From setting up a secure
shopping cart to understanding how the chain of evidence and how to deal
with authorities, the experience was valuable for everyone there, including
me. I for one will be back again next year to watch the games!
Winning Team: Millersville University

Todd E Echterling: System administrator for the computer science department
Chad A Billman
Edward J Schwartz
Thomas J Miller
Cory W Adams
Michael A Vicinsky
Mark A Olszewski
Bradley J Chronister
Red Team:

Joe Harwell: Joe is a Security Specialist for Nortel Government Solutions.
He currently is responsible for design, integration and testing of many of
the "three letter agencies" security systems, and has over 15 years of
experience in the field. He was CERT penetration tester for the US Army in a
previous life.

Ryan Trost: Ryan is a Senior Security Engineer for Criterion Systems,
currently working on a DHS contract. When not overseeing the security
architecture of his team, he spends his free time developing a Network
Security Snap-on Application that involves IDS Geocoding (patent pending).
Ryan will be graduating from George Washington University this May with a
Masters in Computer Science.

Adam Meyers, CCE, IAM, IEM: As an information security professional and
consultant, Adam Meyers provides clients with complete security expertise,
ranging from assessments, forensics, incident response, penetration testing,
and security architecture. Additionally he provides physical security
assessments and threat analysis. Mr. Meyers is a Certified Computer Examiner
(CCE). Prior to joining SRA, he worked with the George Washington University
Security Team, as the Network Manager for the 2000 National Democratic
Convention, and as a private security consultant, all while pursuing a
degree in political science with specific attention to inter-state
information warfare.

Tom Parker: Tom is a computer security analyst who, alongside his work
providing integral security services for some of the world's largest
organizations, is widely known for his vulnerability research on a wide
range of platforms and commercial products. Tom regularly presents at
closed-door and public security conferences, including the Blackhat
briefings, and is often referenced by the world's media on matters relating
to computer security.