Have you shared this with the Open Data community Data.gov, too? I think it would be very interested to make it easier data.gov all types of developers. Of course, the ideas you have expressed here are not limited to BLS data, or even health-related materials only.

Jeanne,

If you brought all the agency data inside data.gov, you might run out of disk space.  A redirect to the data is acceptable for applications that will reuse the data.  For example, http://www.data.gov/raw/324/csv  might redirect to the BLS link for the CSV doc on their server.   Something to distinguish 'live' versus 'dead' documents might be helpful too.  The BLS link is live since it is updated by BLS with new employment projections every 2 years.

Here is an example of a recruiting site for the Army that uses an RSS feed from usajobs.gov as input:

http://wafo.cpol.army.mil/issue/employment.svg

Note:  To see this you need the latest version of Internet Explorer (IE9), Firefox, Safari, Chrome, or Opera.  IE8/7 will not work (they need the Adobe SVG plugin).

This application would fail if OPM were to change the URL for the Army jobs, or change the format of the RSS feed.  The RSS feed typically changes hourly and the application polls the URL hourly. 

Regards,

Andrew

Andrew--

Great suggestion and thanks.  We'll look into fixing that specific issue, and work with BLS on it.

There are a few approaches we take in connecting agency data into Data.gov. The first, as you've gathered, is to upload information about an agency's data, and then link back to the agency-hosted dataset.  A second is to upload information and the dataset itself on Data.gov.  A third, to be available in May, in Data.gov Next Generation, which will bring some of the datasets directly into Data.gov and rendered on screen, allow you to search within the dataset itself, and create visualizations.  Each agency makes choices about the approach for each of their datasets, but the ability to download the dataset in one click, at a minimum, should be there for all choices.  I'll work to fix this one.

Thanks for the suggestion.

--Jeanne Holm, Evangelist, Data.gov (@JeanneHolm/jholm@jpl.nasa.gov)

Cristian,

Please feel free to disseminate the ideas presented as you see fit.  Most if not all the ideas I presented are in various published papers and I cannot lay claim to them.  I work on behalf of a client with limited resources and must stay focused on specifics. 

Regards,

Andrew

Andrew,

Have you shared this with the Open Data community of Data.gov also?  I think they'd be very interested in making data.gov more accessible to all sorts of developers.  Certainly the ideas that you're expressing here are not limited to just the BLS datasets or even health related datasets only.

Cristian

Just like to add that HTTPS, while not necessary, is preferred over HTTP since it confirms the identity of the server.  Ie my application can be assured that the data is really from BLS and not some computer posing as BLS.  Granted in this case security is not strictly necessary for my application, however, it is much preferred, and depending on what the application is doing with the data may often be a requirement.

So I would say that for all links on data.gov, HTTPS should be used and not HTTP. 

Cristian,

A permanent HTTP link to a CSV document (which is updated every 2 years with new 10 year projections) would satisfy my present needs.  Historical documents could have permanent links such as

http://www.data.gov/raw/324/2008/csv

However, for the application, I only need the most recent 10 year projections, so I don't care about the historical documents.

A web service based on the 2007 WSDL 2 standard (not the earlier non-standard WSDL predecessor)  as described in http://www.w3.org/TR/wsdl20/ or http://www.w3schools.com/wsdl/wsdl_intro.asp would work too.  Ie, if the data were returned as a well structured XML document, that in my mind would be superior to CSV since more semantic information (such as the column headers, footnotes etc) could be put in and insertion of new information can be done via creation of new XML tags without breaking existing processing (ie the XML format is extensible whereas CSV is not).  Perhaps http://www.data.gov/raw/324/2008/xml

I can however, readily process either, so either way is good. 

I was expecting data.gov to follow these e-government guidelines:

http://www.w3.org/TR/2009/WD-gov-data-20090908/

which specify permanent links, open formats, and machine readability (in addition to human readability).

Regards,

Andrew

Andrew,

Point well taken.  I've forwarded your comments over to the folks at BLS also. In the case of incorporating the data for use in your application, would you find it more beneficial if the data were put online download permalinked as a .csv file, or would you prefer the data to be encapsulated by a web service that your application could just call when it needs the data?

Cheers,
Cristian

Christian,

I understand the purpose of data.gov is to provide permanent and fixed URLs to all government data to support the idea of application 'mashups'.  In contrast, there is no gaurantee that the link at BLS is stable.  Perhaps next month it will change to something else.  If I develop an application that uses the BLS data as one of its inputs, the change of the URL by BLS would be disastrous.  It would break the application.  Hence the preference for the permanent links of data.gov.  The data I want is actually in this link (after one follows http://www.data.gov/raw/324/csv):

1.2 Numerical and percent change, by detailed occupation [PDF] [XLS]

Unfortunately, a second easily correctable issue arrises:  The data is only available in a choice of 2 proprietary formats (PDF and XLS) which are far less amenable for automated processing than the standard CSV format (which was what was promised).

The real value add of data.gov is to present the data in a way that it can be used by automated software.  Without permanent URLs and data in a format amenable to automated processing, data.gov's potential is compromised.

Regards,

Andrew

Andrew,

You make a really good point here.  Have you found a way to download the data as a csv file directly from the BLS site rather than data.gov?  What's the reason for having the link point to data.gov versus the BLS site (not that it's bad, just trying to get an understanding).

Cheers,
Cristian