One of the hallmarks of the Dodd-Frank bill was the creation of the Office of Financial Research (OFR), an agency dedicated to collecting and sifting data from financial firms to help predict the next crisis.

More than a year after the law's passage, however, it's clear regulators are still struggling to implement the agency's mission. At a recent conference hosted by the Financial Stability Oversight Council dedicated to helping to set up the OFR, Treasury Secretary Tim Geithner equated the challenge with trying to predict the weather.

"Policy makers are always looking for the financial system equivalent of the MRI, over-the-horizon radar, the kind we put on aircraft carriers; or, to borrow Andrew Lo's analogy, a National Weather Service," Geithner said at the opening of the conference, which was held Dec. 1 and Dec. 2. "That goal will always elude us. But we will keep pursuing it, and we can do a lot better than we have done to date."

The conference featured academics, former regulators, industry representatives and others who weighed in on the best way to collect data that can help spot emerging systemic risks.

While there appears to be some initial headway on the subject, those invited painted a stark picture of the amount of work still left to be done by regulators.

Challenge #1: Defining Systemic Risk

The chief obstacle to heading off systemic risk turns out to be agreeing on a definition for it, including regulators both here and abroad and financial firms.

David Newman, vice president and strategic planning manager of enterprise architecture of Wells Fargo Bank, said what's needed is a public-private partnership between regulators and industry to aid the process of defining specific standards.

"Unless we find a way to standardize the data in order to effectively understand it so that we have consistent data, so that the data can be defined precisely with clarity, so that we have trust in the data," said Newman. "Without having that we certainly cannot draw the conclusion of risk that we will need to draw going forward."

One of the lessons learned from the crisis is the necessity to identify aggregate exposures both by individual entities and instruments, said Linda Powell, chief of economic data management and analysis section in the division of research and statistics at the Board of Governors of the Federal Reserve System. That means regulators must be able to readily identify the institution and the instruments it holds, not an easy task because of the number of different asset classes that institutions hold which are constantly changing.

"We need to aggregate data not just to the industry level, but a lot of times we need to know the total aggregate exposure for a firm," said Powell, citing the example of how Wall Street firms weren't aware of their aggregate exposure to Lehman Brothers at the height of the crisis.

Work is already underway to develop a legal entity identifier that can be universally agreed upon, but nothing has been signed off on yet. International regulators including the Bank for International Settlements, the International Organization of Securities Commissioners and the Financial Stability Board have been making headway on the issue.

Use of the LEI, as it is referred to, is not a new concept, or a new necessity. What's different is the idea of standardizing it globally by users and stakeholders, said Karla McKenna, who is the chair of International Organization for Standardization Technical Committee 68 and also director of market practice and standards for global transaction services for Citigroup.

Without a high-quality industry-wide identification system, it makes it difficult for financial firms to assess their risk exposure in the case of default by one or more of their counterparties. It also creates an inability for regulators to quickly assess risk in order to respond to the crisis.

Challenge #2: The Past is Not Prologue

As a whole, regulators and the industry tend to naturally rely on past data as an indicator of future performance. The problem, however, is that the financial system is evolving so quickly it is difficult to capture risk effectively by relying on older models.

Lewis Alexander, now chief economist at Nomura and former counselor to Geithner who led initial efforts to establish the OFR, stressed the importance of a measurement system that constantly evolves, or else the system will inevitably face risk.

Every crisis dating back to the 1800s is essentially different, Alexander said.

"The financial system that has evolved over these periods is very different now then when it was during these crises, and when you think about how you measure things, you have to take into account the fact that the system we deal with is going to evolve at a very rapid pace relative to our ability to observe these things called systemic risks — and that has important implications of how you think about systemic risk," said Alexander.

Any system used cannot be inflexible as institutions and market conditions change over time, participants agreed.

Lo, who is the Harris & Harris Group professor of finance at the MIT Sloan School of Management, presented a clear example of this in a survey of systemic risk measurements he conducted with colleagues.

In the study, Lo and his researchers looked at 30 different measures of risk including co-risk, which was first proposed by the International Monetary Fund in 2009 as a means measuring the relationship of credit default swap spreads of one set of firms with others to determine the probability of default.

What researchers found looking at a topographical map of midtown Manhattan is that if you surveyed each firm, and then conducted the same survey six months later — the results were dramatically different.

In an even more dramatic example, Lo showed a network diagram of connections between hedge funds, broker dealers, banks, and insurance companies, which looks like a yarn ball. The diagram, which is designed to be automated, shows how those connections change rapidly month-to-month, he explained.

"It seems pretty clear that the number of systemic risk measures that we have really tell us different and important things about the kind of exposures we have," said Lo. "The more measures, I think, the better for identifying potential shocks."

That's why new methods will need to be developed in order not to get stuck using relics of the past to measure risk, panelists urged.

"We need to start thinking of new and novel solutions," Newman said. "This is an existential problem and we need to be very careful. We need to make sure that we are using the newest and the best tools in order to build that bridge … we are going to travel on. If that bridge collapses, you will know the consequences of that."

Mark Carey, senior advisor of division of international finance at the Federal Reserve Board and also co-director of the National Bureau of Economic Research, reiterated that point.

Carey urged regulators and industry to look beyond the usual lampposts that the policymaking community has focused on in the past. Pre-crisis, he said, there was a reliance on indicators as tools to determine crises, like imbalances in credit to gross domestic product ratio, for example.

"Obviously, that is an incomplete way of looking at things," said Carey. "Here in the Washington policy community, as recently as a year or nine months ago, there is still a kind of focus on that familiar lamppost. It's easy. There's work out there already about all of those things. But what has become eminently clear to anybody who is thinking about giving a financial stability briefing to a policymaker is that those kinds of indicators and those kinds of tools are not very relevant today."

Challenge #3: Creating a New, Flexible System

If past models have proven ineffective and too inflexible, policymakers are still struggling with a system that could work much better. Regulators must determine what new information is needed to measure risk.

One of the suggestions made by panelists was for OFR to use its authority to encourage institutions to disclose more information than is already been required of them under new regulations. Institutions should disclose data on short-term debt, or even collateral, where true liquidity risk resides, some argue.

"Try to focus more on what financial firms are not going to be asked to disclose under the Dodd-Frank Act explicitly. Maybe the OFR could make an effort to fill in the gaps," said Viral Acharya, C.V. Starr Professor of Economics at New York University.

While the concept of systemic risk has been around for at least the last century, regulators must also adjust to a new reality: much of that risk no longer comes from just banks.

"The financial system has gotten a lot more interesting, a lot more complex, and so one new aspect of systemic risk … has made it a far broader affair," Lo said. "It turns out that systemic risk comes out in all quarters of the financial industry and that requires a more complex set of analytics."

Subscribe Now

Access to a full range of industry content, analysis and expert commentary.

30-Day Free Trial

No credit card required. Access coverage of the securitization marketplace, including breaking news updated throughout the day.