It's been a long time since this thread started however a recent meeting brought this question to my attention. I have a use case that might just help answer this question. A federated organisation had problems integrating 200 systems across 43 locations. The problem was they could not agree data standards. It transpired that they needed just 308 code lists for the integration to work but they had over 12000 to consolidate. The cost to update all the local systems was totally prohibitive and the fore the project failed after 6 years of trying. At that point they decided to approach the problem from the bottom up and created tools to publish and map source system code lists into the target system (analysis tools). This mapping layer was exported in XML and used to translate all the historic and new data. The outcome was interesting - firstly they achieved a single view of intelligence across the enterprise. The second outcome was less expected. Rather than new systems and data standards being enforced from the top down, they are adopting the central data standards over time and as they upgrade local systems. Therefore investment has not been wasted, natural upgrades adopt standards over time and ultimately they will have a single standard. In the interim they use the mapping translation layer to get what they needed in the first place but at a significantly lower cost. My conclusion is that top down data standard projects continue to fail and are clearly not the answer to most data integration challenges. In order to achieve common data standards, provide the means to link competing standards together, embrace diverse applications and allow for the adoption over time.

I am currently working in the UK with the educational attainment data provided by http://data.gov.uk/ to create a standardised reading test system - http://www.readingtest.co.uk/ - and they have taken a top down approach but have also allowed input from data users in the form of comments and it has been nice to see that the UK Gov have acted on some of this feedback for datasets. However, I do think that countries are very compartmentalised with their data and systems and maybe their should be a total top approach to standards so that world data is in useable formats but as yet I do not think a worldwide organisation exists ?

I would think adopting standards would be the best solution unless you had specific issues in mind.  Even then I would consult business and university data experts.  I'm sure google would be happy to provide an engineer for you to bounce ideas off of.

-tom

I would think that adopting standards would be the path to go down.

What problems specifficly do you have with current data formats?, or better yet, what feedback have you been getting?

You seem to cover alot of bases, I can see people used to using excel (finance/economist) would use the cvs format and the web developers will create mashups with the xml feeds.  I'm not as familar with the other formats you provide.  But even as xml goes, I thought it was pretty extendable as long as you provide the schema.

BTW, the data you provide is an amazingly wonderful resource and this is a great site, very well done.

-tom

I am also prefer Bottom up approach ,because by using these appraoch ,you understand any type of problem ,from starts to end,nice post thanks for share this post with us...

I think bottom up approach is better at the beginning. The top down approach may risk to fail if no examples are provided and could result too abstract. So first bottom up to give real examples and then combine it with top down approach.

I agree that the development needs to come from both approaches. Each has different but important outlooks on the data. And each outlook addresses different aspects of the standard.

Having a catalog of standards here at Data.gov makes sense to me because I view the site not only as a data store but also a resource of guidelines and BMPs.

Great discussion at the IOGDC on how the choice of Data.gov to use standards such as RDF, allows others to create easy apps, mashups, and visualizations.  Kendall Clark shared a story of a young developer taking only 90 minutes to create a faceted browser that shows ~250,000 Data.gov datasets, parses those to the Library of Congress categories, and connects to WorldCat.  See it at http://datagov.clarkparsia.com/ This was possible only because there are standards that of all us work to, even though the efforts are completely uncoordinated with each other.

The question came up whether Data.gov should point to or somehow help people understand the standards that are in place (without necessarily endorsing a specific standard).  Would it be helpful for us to do this?  What are some of the ways you would recommend or find useful?

There are many defacto-standards that were established in a seemingly bottom-up way, but if they are not open standards then they encourage vendor lock-in and limit platform agility and the full openness of the data. Many top-down approaches end up being too complex and limit agility and adoption as well. The organic "small pieces loosely joined" agile iterative approach to stabilization and harmonization is the most viable way to get broad adoption. Give people time to develop their own normalization schemes and XSLT transformations between many different formats to help inform a normalized standard.

The Civic Commons wiki has started cataloging a "civic stack" of open systems and standards. The mediawiki infrastructure might not be the ideal platform to manage this, but it's being used to germinate and aggregate some of these while we implement a more structured and easy to use system.

See http://wiki.civiccommons.com/Civic_Stack and http://wiki.civiccommons.com/#Data_Standards

We need both. From the "top down" approach, we need a basic set of standards, and direction on how to extend/expand them. From the "bottom up" approach, we need to get and use real world examples of ontologies that have been developed to solve real world problems.