MED 2011 Data Use FAQ
FAQ #1: Can we use the knowledge from the test event kit and clips?
For example, we want to built some high-related semantic concepts for these test events. Of course, these concepts are selected base on the event kits. Is this OK?
Answer: Yes, this is OK.
Per Section 1 of the Evaluation Plan, "the metadata store may be optimized with knowledge of the events to be evaluated". Your semantic concepts form knowledge in the metadata store. Further, Section 3.1 of the Evaluation Plan states "All resources in the event kits can be used for development and testing."
FAQ #2: Are there any positive Event006-015 clips in the DEVT Collection?
Answer: The only positive clips identified for E006-E015 are in the event kits. There may be unidentified positive clips in the DEVT collection. Researchers should post newly found positives in the DEVT collection to the HAVIC corpus issue tracker website via your team's Data Contact.
FAQ #3: Are we allowed to use the DEVT clips to prepare for the Dry Run?
Answer: Yes. As described under the Dry Run section on the MED '11 website, developers can use the Event Kits and/or the DEVT collection to prepare their Dry Run submission -- even clips in the dry run database files. The Dry Run is a test of the evaluation infrastructure, not system performance.
FAQ #4: Can we get negative examples from the Test Set?
Answer: No. Per Section 3.3 of the Evaluation Plan, no knowledge should be gained from procession the Test Set. Only the CDR should be generated. The only exemption is fully automatic feature adaptation is allowed during CDR generation per the first bulleted item in Section 3.3 of the Evaluation Plan.
FAQ #5: Can videos be used in presentations?
Answer: Yes. The MED '11 license agreement states "User and User's Research Group may include limited excerpts from the Data in articles, reports and other documents describing the results of work performed in the MED11 evaluation and of research toward improved MED performance."
ALADDIN contractors must obscure faces when displaying HAVIC images in public forums. Non-ALADDIN contractors are encouraged to do the same to protect people's.
FAQ #6: "Near_miss" and "related" clips are similar, can you explain the difference?
Answer: The INSTANCE_TYPEs are defined in Table 2 of the Evaluation Plan. "Near_miss" clips are closer to the event including many aspects of the event. "Related" clips is a much broader class and logically subsumes "Near_miss". The distinction is subjective.
Not all clips were judged against all instance types. Where clips were judged against multiple instance types, the most restrictive membership was used.
FAQ #7: Will the "near_miss" and "related" categories be used in the evaluation?
Answer: The "Primary Measures" summarizing system performance will not use these annotations. However, NIST will perform conditioned analysis to help identify strengths and weaknesses of systems.
FAQ #8: On page 13 of the Evaluation Plan, parallel processing steps contribute to TPT as a single step, was this intended?
Answer: Yes. This is different than previous evaluations. The processing time for parallelized operations is not multiplied by the number of processor units. The computing hardware in your system description (Appendix B.1 - Sections 3 and 4 of the system descriptions) should document the composition of the cluster.
FAQ #9: The banned clip file (located in the LDC12011E42 release) contains 44 banned files not in the LDC2011E06 and LDC2011E41 releases. Why are they listed?
Answer: The clip location file covers all MED data including LDC2010XXX. The 44 banned files were released in 2010, but not again in the subsequent releases.
FAQ #10: Can we optimize parameters for the Training and Testing events using the DEVT Collection?
Answer: Yes. Per Section 5.0 of the MED-11 Documentation and Metadata README, there is unrestricted use of the event kits in conjunction with the DEVT Collection.
FAQ #11: Can we use external detectors, like person detectors, trained on other images.
Answer: Yes, as long as the code is run in-house.
FAQ #12: Can we use sounds downloaded from FreeSound.org and data from other sources like YouTube?
Answer: The rules of the MED evaluation do not prevent the use of non-HAVIC data, however the site should make sure they have the legal right to use the data. ALADDIN contractors must request changes in research data through the process established by the program via your team's PI.
Revision History
- July 11, 2011 - Initial page created for FAQ #1-#12.
