How Weather Forecasting Can Democratize Big Data

When you consider it, the National Oceanic and Atmospheric Administration is actually an ideal federal participant in Big Data research. The Administration reportedly aggregates more than 20 terabytes per day through its satellite systems alone, and its supercomputer used to develop climate and weather models similarly provides massive doves of information.

While the NOAA’s main purpose is to provide public information regarding severe weather, to protect life and property, and to analyze weather events, the agency’s ability to aggregate, store and assess Big Data is a model that could help businesses large and small do the same. And, because the NOAA is part of the Department of Commerce, the group is a public supporter of enterprise and the market’s use of Big Data for economic growth.

Less than one month ago, the NOAA and the Chamber of Commerce announced a new partnership with a slew of some of the world’s largest technology conglomerates, including Google Cloud Platform, Amazon Web Services, Microsoft, IBM and the Open Cloud Consortium. In forging these ties, the groups will aim to explore and develop ways to more efficiently analyze, compute, and store Big Data.

In a blog post published earlier this month by the White House, officials explained that the collaboration stemmed from the NOAA’s 2014 Request for Information, an initiative that asked the public how data could be made more accessible and useful to the country. The NOAA took the most commonly referenced Infrastructure-as-a-Service providers from the survey – the businesses named above – to launch further “data alliances” with other companies of various sectors, including SMEs.

While the plan aims to make the NOAA’s weather data available to any enterprise that wants it, the group said these partnerships “will each serve as a prototype for the larger market, ensuring that while NOAA and its collaborators research and develop technology to efficiently distribute the data, they can also test out the hypotheses of self-sustainability within a fully representative market ecosystem.”

In other words: it’s not just weather buffs that will benefit from innovation in Big Data analytics and distribution.

In terms of its business model, the NOAA said it will cover all shipping and handling costs of data distribution, but predicts that the profits earned through pubic use of NOAA data will subsidize these costs.

According to the Office of Science and Technology Policy, “if NOAA’s endeavor proves successful, it could lead to far greater availability of open data across the entire government, providing taxpayers and private industry with information vital to economic growth, innovation and discovery.”

Federal Big Data Goals

The NOAA’s efforts are just the latest in recent efforts by the White House to explore how Big Data can be reigned in and used to boost the economy. In early 2014, President Barack Obama requested the formation of a Big Data working group, led by counselor to the president John Podesta, while the President’s Committee of Advisors on Science and Technology similarly probed the topic.

Months later, the group published the results of its 90-day study, “Big Data: Seizing Opportunities, Preserving Value.” A generous portion highlighted the need for robust data security and consumer privacy initiatives. But the working group also found major opportunities for enterprise, from technology companies to banks to health care providers. “The benefits of big data,” the report concluded, “can be felt across a range of sectors, in both large ad small firms, as access to data and the tools for processing it are further democratized.”

That working group released an interim progress report last February revealing falling costs to data collection, storage and analysis. Coupled with the increased connectivity in the Internet of Things, the working group found that “we live in a world where data collection is nearly ubiquitous, where data retention can be functionally permanent, and where data analysis is increasingly conducted in speeds approaching real time.”

Again, the focus of the report largely landed on data security and protection. But the NOAA’s latest participation in the White House’s exploration of Big Data reveals a new, enterprise- and economic-focused angle at which the White House is exploring the uses of data storage and analytics, and further promotes this democratization of Big Data tools for businesses.