How Will Investigators Sift Through All the Imagery of the Boston Marathon Bombings?
April 18, 2013
Investigators sifting through the flood of cellphone, surveillance camera, and TV footage of Monday’s bombings at the Boston Marathon are being aided by technology similar to the software that the military has used to collect intelligence about IED attacks in Iraq and Afghanistan.
"There’s a different twist to it this time. The different twist is the increased degree of crowd-sourcing if you will, in terms of providing information. You have many, many more sensors in the context of people with video devices in their smartphones," said retired Lt. Gen. David Deptula, who was in charge of the Air Force’s intelligence efforts from 2006 to 2010. "You had many, many more collectors than we had in the past."
The amount of video and photo documentation of the marathon attacks may be unprecedented, so how do you sift through all that data quickly to find clues? Software, naturally.
As ABC News reported, investigators from the FBI’s Operational Technology Division are likely using a computer program that can do things like recognize faces in a crowd if they match those listed in a criminal database. This is similar to the software that the military has been developing for years in an effort to quickly glean information from UAV videos.
As the U.S. military flocked to the skies of Iraq and Afghanistan with all manner of camera-equipped spy-planes, intelligence officials soon realized they were collecting far more footage — thousands of hours a day — than human beings could sort through in time to use the information it contained. The military turned to tech companies to produce software capable of quickly identifying certain things analysts were looking for — say, a red Toyota pickup truck that had been seen at a bombing site.
"There are software programs that are out there that allow one to rapidly search through that information and key in on what the investigators may find of interest," said Deptula. "Exponential growth is not hyperbole when it comes to motion imagery, much less still imagery, because we’ve had an explosion in that kind of information. As the information [available] has grown, people have moved from human analytic teams to more automated means to sift through all that data."
"Let’s say somebody reported that they saw somebody that was Caucasian, with a yellow sweatshirt, with powder burns on their hands running away before the explosion — that’s a hypothetical — you could tell the software to look for a yellow sweatshirt, Caucasian running before a certain period of time," said Brian Cunningham, a former White House security official and now a senior advisor to the Chertoff Group who works with firms that develop this kind of software. New York City and London both have massive video surveillance systems that use similar software.
Still, another homeland security consultant who wished to remain anonymous tells Killer Apps that it might not be that easy. First of all, Boston doesn’t have a massive, centralized video camera system the way New York does. Many of the images will come from people’s phones and other private cameras, meaning that investigators will probably have to receive and review each photo and film clip individually.
"There are some automated tools that exist for this type of thing, but for the most part it’s just a very labor-intensive process to go through things and try to correlate and sequence things in time and look for suspicious activity and then try to build a profile for how somebody’s moving around," said the former DHS official. "There are capabilities like in London and lower New York where they can follow a person who is of concern as they walk from camera to camera. When you’re dealing with public-source information it’s just a different process."
Cunningham agrees that while the Boston Police Department or the FBI has the software capable of identifying a particular person or bag as they appear in the mountains of video, investigators still face the challenge of uploading all that footage so the software can analyze it. "The biggest challenge will be: how do you upload that volume of video onto a single server or a couple of servers that can be searched against?" he said.
Investigators have identified two people they want to talk to in connection with suspects (see the video above) in the Boston bombing. But, Cunningham said, "It’s not clear yet whether it was good old-fashioned shoe leather as much as analytic software."
He explained how the process could work: "You’d figure out where the devices were, and while you had street cops out interviewing people and collecting video of cellphones and you would go to fixed cameras in department stores or ATMs and pole cameras that are right around the area of the devices" and then upload the footage into the software, said Cunningham. "They also may have just had officers sitting there watching the footage. Let’s say there were 15 cameras that were fixed, that had a good line of site of where the device was, then you could throw 100 officers at it; you probably wouldn’t need software."
Cunningham also points out that investigators are working with cellphone companies to find cellphone records of the calls that were made close to the site of the explosions. Cellphones might allow them to find calls that were used to detonate the explosives. It’s not clear if the explosives were triggered by timing devices or cellphones. Initial reports suggest that at least one of the suspects sought by investigators was actually talking with someone on the phone rather than triggering a bomb.
"Once they know what cellphone was his, that’s the jackpot because they can find out where he was right before, and they can find out where he is today if he’s dumb enough to be carrying that same cellphone," added Cunningham. Even if the phone the suspect used was a cheap, pay as you go phone, investigators would immediately begin to look for the store where that phone was sold.
