Posted October 28th, 2010 by Michael Byrne - Geographic Information Officer
Thanks to a combination of the advancement of computer science, demand for data, and ability for businesses and large institutions to make use of large datasets, Big Data is here.
Last week, I was honored to be part of the Booz Allen Series for Fed News Radio "Expert Voices," speaking on the topic of Big Data. From a government perspective, we at the FCC are not dealing with big data (billions of records) like some partners are, but we oversee and work with many industries that do. But to do the FCC's work, we are constantly collecting and tracking records in in the tens and hundreds of millions of records to help make sense of the communications industry.
As you might be able to tell from our data innovation initative, we clearly are in the business of better understanding our data assets and knowing where and how to make our own data better. We feel strongly that one of the most important aspects to a more efficient and effective FCC is strong data assets -- and in particular, ensuring our data assets are fully geospatially-enabled. When we have geospatially enabled data assets our ability to analyze trends over space and time will provide benefit to the policy discussion and provide a more informed and empowered arena for consumers.
Listen to the whole big data discussion or watch my small trailer on big data.
We're curious: How are other government organizations and large industry partners looking out at the horizon of Big Data? Leave us thoughts in the comments below.
Posted in DataPosted October 13th, 2010 by Michael Byrne - Geographic Information Officer
Census geography changes, while small or obscure, can be significant. A tiny change in a census boundary can mean that a rate based calculation includes a completely different denominator for population or demographic value. These changes, if not watched for carefully can be significant to the results of querying large federal databases.
To assist the community of developers building off FCC tools, we'll try and point out these small but significant changes when we see them.
The biggest change between 2000 census boundaries and 2009 census boundaries was the addition of a sub identifier to the smallest unit of boundary, the block. This addition allows for finer resolution to the map base. However, because of other changes like population growth, demographic switches, and land use changes, the external boundaries of the block boundaries have also changed.
In order to keep up with these issues, we are supplementing our FCC Census Block API with the ability to query for the current year as well as previous years. From now on, the current (e.g. 2009) year search will be the normal REST query on the documentation page like this;
http://data.fcc.gov/api//block/find?latitude=37.21&longitude=-120.4356
To gain access to a previous year, all you need to do is insert the year in the url like this;
http://data.fcc.gov/api//block/2000/find?latitude=37.21&longitude=-120.4356
The stylistic change we made removes the @ symbol from the return, moving to a natural JSON notation to allow for better integration with some client libraries.
These changes are live now. Here's a preview of /Developer updates coming down the pipeline:
See below for a more complete description of the most recent changes. Happy Coding, and don’t forget to let us know what kind of applications you are building with FCC Developer tools.
-----------
1)Changed JSON output to follow the natural convention for Block and Speed Test APIs. This changes the output from:
{"@executionTime":"0.071","@status":"OK","Block":{"@FIPS":"060470009012065"},
"County":{"@name":"Merced","@FIPS":"06047"},
"State":{"@name":"California","@code":"CA","@FIPS":"06"}}
To
{"Block":{"FIPS":"060470009012065B"},
"County":{"FIPS":"06047","name":"Merced"},
"State":{"FIPS":"06","code":"CA","name":"California"},
"status":"OK","executionTime":"0.445"}
Please note that the @ symbol has been removed from the attributes within a JSON object. This should make it easier for these data structures to be consumed from some libraries such as jQuery.
2) Added version year for census block geospatial search
There is an additional option to request Census Block data for Census 2000. The original call will return the most recent Census Block information (2009)
Original query for Census information (current information)
http://data.fcc.gov/api//block/find?latitude=37.21&longitude=-120.4356
<Response executionTime="0.054" status="OK">
<Block FIPS="060470009012065B"/>
<County name="Merced" FIPS="06047"/>
<State name="California" code="CA" FIPS="06"/>
</Response>
Census 2000 block information
http://data.fcc.gov/api//block/2000/find?latitude=37.21&longitude=-120.4356
<Response executionTime="0.052" status="OK">
<Block FIPS="060470009012065"/>
<County name="Merced" FIPS="06047"/>
<State name="California" code="CA" FIPS="06"/>
</Response>
3) Fixed the following bug
When passing wrong coordinate information (i.e. outside of the U.S. and territories) the response returned a badly formed XML if no format for the response type was specified.
Posted September 14th, 2010 by Steven VanRoekel - Managing Director, Federal Communications Commission
FCC License View is a tool designed to make FCC license management information more transparent and accessible to a broad range of users.
This release follows last Tuesday's launch of a suite of developer tools and APIs, including the FCC License View API.
FCC License View is an initial release of functionality from the FCC's ongoing Consolidated Licensing System (CLS) project. Thanks to efforts stemming from our the new Data Innovation Initiative, our team was able to expedite the release of FCC License View for speedy release to the public.
FCC License View is available now at http://fcc.gov/licenseview.
Last week at the Gov 2.0 Summit here in Washington, D.C., FCC leadership reaffirmed our commitment to providing powerful, innovative tools into our robust community of developers. Today's release marks our ongoing progress towards those goals -- and the first in a regular release schedule of tools and tweaks.
With this new tool, users from across private and public sectors can digest complex licensing info through a simple and easy-to-use dashboard. FCC License View lets users digest snapshots of FCC license management data that are at the core of the agency’s mission. At launch, FCC License View lets users explore over 3 million total licenses, 2 million of which are active.
This consolidated portal allows users of FCC License View to access information on the number of different licenses across services, the number of licenses owned by particular entities, and which licenses are up for renewal in the future.
The underlying dataset is composed of data across the FCC's five licensing databases, and gives users three options -- HTML pages, raw data files, and Application Programming Interaces (APIs) -- through which to access and reuse agency data.
Get started using FCC License View now, then make sure to leave us your feedback and sign up for our FCC Developer community.
Posted in Reform - Redesign , Open Government , Consumers , DataPosted September 9th, 2010 by Learon Dalby
The challenge- finding the niche and sustaining the system. It is critical that FCC quickly determine what API’s developers are finding useful and which ones need additional work. In other words; the success of www.FCC.gov/developer will hinge on the FCC’s ability to work with the developer community not just provide another way to access information. After all, isn’t bi-directional communication what Gov 2.0 is all about?
Learon Dalby currently serves as the GIS Program Manager in the Arkansas Geographic Information Office. He is responsible for managing a number of statewide incentives focused at providing open access to GIS data.
Posted in Data , DeveloperPosted September 7th, 2010 by Michael Byrne - Geographic Information Officer
When we publish data through open standards like these APIs and smart people make use of trusted government data in innovative ways, we realize the ideal of the Gov 2.0 movement of government and private sector innovating together to solve our great policy challenges.
Several aspects of this release deserve highlighting. First, APIs are central to our efforts for data transparency and open government. Second, all of these API’s are RESTful in nature and return open structured data as a service. RESTFul APIs are popular on the web because they involve less programming effort to incorporate dynamically in mash-ups. We want many developers to view and use our data assets.
Finally, and perhaps most importantly, two of these API’s (Consumer Speed Test and Census Block search) are location-based services returning data based on a latitude and longitude. We imagine use of these APIs to unlock a myriad of government data based any user location. As many notice the sea change in the use of geography by the public, the government has to respond with strong leadership. In our case, leadership means publishing via open standards which fosters consumer and citizen connecting, communicating and collaborating to solve policy issues.
The release of these APIs marks an important day for us at the FCC. The FCC has long published many data sets. Now we are allowing developers direct access to our data via live queries. Your feedback on these APIs—what you think, how you are using them, what needs to be improved—helps us continue in this direction.
Below is a quick digest of each service.
Consumer Broadband Speed Test API
This API returns the Consumer Broadband Test speed test statistics for a US County given the passed Latitude and Longitude. The statistics are grouped into wireline and wireless and are the number of tests, average download speed, average upload speed, maximum download speed and maximum upload speed. This data is calculated nightly and includes all tests to date performed through the consumer broadband test.
Census Block Search API
This API returns the US Census Bureau Census Block number (aka the 15 character FIPS Code) given a passed Latitude and Longitude. The API also returns the US State and County name associated with the Block. Just about every major dataset in the US Federal Government uses some form of Census geography. The lowest level of that geography is the Census Block. This geography is highly nested, such that removing coded values from the right hand site aggregates to the next highest geography (e.g. from block to tract, and from tract to County). Providing this search allows developers to build applications which foster connectivity from their individual location to the full array of federal databases based on census geography.
FCC Registration Number (FRN) Conversion
Filers with the FCC must get a registration number (aka FRN) whose company names, parent names and subsidiary names often change from state to state and region to region. We are providing this API to increase the value of the transparency of broadband providers providing service in each state. Providing a single place for understanding the complex naming of these providers will benefit consumers. This API has 2 method calls: getList and getInfo. GetList call takes the parameters of a state's abbreviation or state FIPS code, and a yes-or-no for multiple-state indicator, and returns all the broadband providers in that state. GetInfo call takes FRN number of a broadband provider, and return information about the provider.
FCC License View
The FCC issues licenses for use of the nation's airwaves and other purposes. License View API provides snapshots such as the number of licenses across different services, how many licenses different entities have, and how many licenses are up for renewal in the near future. The following APIs represent a step toward reform of our licensing systems and improvement in how the FCC makes licensing information available to the public.
Posted August 9th, 2010 by Greg Elin - Chief Data Officer
Resources: reboot.fcc.gov/data/review
If you've given any thought about data at the FCC and filing comments on the opening round of the FCC's Data Innovation Initiative – the Public Notices of Data Reviews released by the Media, Wireline Competition and Wireless Telecommunications Bureaus - this blog post is for you. Initial comments are due this Friday, August 13.
Though the scope of the Public Notices is significant and welcomes comments on any or all of 340 data sets across three bureaus, filing comments does not have to be a major project. You can make a difference with as little as 15 minutes of effort. The following Top Ten Things You Should Know explains why.
1. You don't have to make your filing a multi-day effort. The conversation is just starting. If you only have 15 minutes, use that time to layout the big concerns. This stage doesn't have to be a research project. Simply share with us what you already know needs attention and rethinking. Think of your comments as writing a quick email to a colleague about your long standing concerns about FCC data. And if you are working on more substantive analysis and comments of how to improve FCC data, please keep at it. This filing is perfect timing.
2. Don't feel you need to comment on every data collection. Comment on the ones you know and are important to you. Treat the data collection list as a helpful guide and don't think you need to comment on all collections. Ignore the list if you want. (Identifying data sets by OMB Paperwork Reduction Act approvals is just one lens onto Commission data; use it to the extent it is helpful.) We offer the spreadsheets of the data collections in each Bureau for transparency sake and as a handy reference. We expect people to file comments about the data and data filings with which they are already familiar. But we also want to make it easier for others reading such comments to learn about the data set being discussed and to provide thoughtful replies. The data collection list exists as a guide and reference, not a threshold or checklist for participation.
3. Share high level thoughts on FCC current data and future data practices. The Public Notices are clear that we are seeking comments on all aspects of how we collect, manage, analyze, and share data. High level comments about the forest may be even more important at this stage than specific comments about individual trees. What data collection techniques are standard now that might have been cutting edge even a few years ago? What issues, or solutions, cut across multiple data sets?
4. Repeat yourself. File anew data-related comments you filed before. It's OK, even encouraged, to re-file comments or link to comments on data issues you've previously filed. Why? Because we are doing a "zero-based review" of all agency data, we are putting everything on the table for review, even data sets that have been reviewed recently, as if we were starting from scratch. The Paperwork Reduction Act requires agencies to review and seek re-approval of a data set at least every three years to keep data current to changes in technology and the market. In addition to those periodic reviews, we are trying a larger, agency wide review to pursue macro-level changes across multiple data collections. Another reason to be comfortable re-filing data-related comments are the new resources the FCC has committed to tuning our data for the digital 21st century including Chief Data Officers in the three Bureaus posting the Public Notices, a Chief Data Officer for the agency and our first-ever Geographic Information Officer.
5. File more than one comment. Don't let the perfect be the enemy of the good (or even the just in case). Start your bullet list of points right now as you are reading this blog post and file some quick comments and make yourself a participant in this conversation. Feel free to file comments today and later this week as you think about them.
6. File comments that help us prioritize. We are reviewing, for purposes of improving, all data collected and used by the Media Bureau, the Wireless Telecommunications Bureau, and the Wireline Competition Bureau. Particularly useful are comments helping prioritize what data and data practices to revise first.
7. Apart from the underlying data sets, the Public Notices are identical in the three Bureaus. File on the Notice that seems most relevant. Our Electronic Comment Filing System also supports filing one comment across multiple proceedings.
8. Share links to existing articles, papers, blog posts. A good filing would be one that lists existing articles, research papers, and blog posts discussing ways the FCC could collect, use, and disseminate data. Coordinate a list with a few colleagues. Keep the list manageable for us, between 10 and 20 links. You can also attach whole documents with your filing. Provide a paragraph or two of context for the list describing what themes in the linked material are most applicable.
9. We already know where we are, so let's talk about where we need to go and how to get there. Pointing out the obviously wrong is always welcomed. Just please recognize there are real reasons the FCC might not be collecting data that to you seems a no brainer and possible statutory reasons we are collecting data that may no longer seem relevant. Like any organization, at any given moment the FCC is juggling new ideas, legacy systems, and resources. Accept there are real challenges associated with modifying data practices and help us overcome those challenges. Don't simply tell us we are doing X when we should be doing Y. Instead, share with us workable road maps to get from X to Y. We know our forms could be easier; share examples of easier to use forms and screens. We know duplication exists in our data; tell us which duplications are easiest and best to address first. We know we want more transparency in our data; offer recommendations about which data sets are most valuable to publish first what formats and techniques make the data most useful. If we should be using RDF and taxonomies, what three steps do you recommend to get started? Do we do one sector at a time, or focus on just a few attributes across the breadth of FCC data? If we should be collecting data in XML, what standards or other trends should we be aligning with? If we need to share collected data better, who are the experts and what are the techniques that can help us enable data sharing while addressing legitimate privacy and proprietary concerns? If a data collection requires significant effort to gather, what is a better way to gather the information?
10. Finally, tell us what things we doing well with data at the FCC that we can build further upon. More than 40 specific databases searches are available on FCC.gov. We offer dozens of data sets for bulk download. Improvements recently made our Electronic Comment Filing System easier to search and use. We know there’s more work to do. That's why we are doing a zero-based data review. But building upon what we are already doing well speeds change. What existing assets can we extend to provide a strong foundation for further improvements?
There you have it. Ten Things You Should Know that should making filing comments on the Data Review Public Notices by the end of this week a snap. So stop fretting at the seeming enormity of improving data at the FCC or worrying you do not have time to file comments on the Public Notices this week. Detailed recommendations on our data collections, high level points, and even quick listings of things to change are all encouraged. In the time it takes to watch a couple Seinfeld reruns you could help improve data at the FCC.
Never filed before? You can file using the ECFS Standard form. Just enter the appropriate proceeding number (10-103, 10-131, or 10-132) and complete the form. To see already filed comments, follow these links: MB Docket No. 10-103 comments; WC Docket No. 10-132 comments; WT Docket No 10-131 comments.
Posted in Reform - Data , Office Of Managing Director , Data
