文献类型： 会议论文

第一作者： Daniel Garijo

作者： Daniel Garijo ¹ ; Varun Ratnakar ¹ ; Rajiv Mayani ¹ ; Thomas Yu ² ; Raghu Machiraju ³ ; Yolanda Gil ¹ ; Parag Mallick ⁴ ; Arunima Srivastava ³ ; Ravali Adusumilli ⁴ ; Hunter Boyce ⁴ ;

作者机构： 1.Information Sciences Institute, University of Southern California, Marina del Rey, Los Angeles, CA 90292

2.Sage Bionetworks, 2901 Third Ave., Suite 330, Seattle WA 98121

3.Computer Science and Engineering, The Ohio State University, 2015 Neil Ave Columbus, OH 43210

4.Canary Center for Cancer Early Detection, Stanford University, 3155 Porter Dr., Palo Alto, CA, 94305

关键词： Workflows;Semantic Workflows;DREAM Challenges;Proteogenomics;Benchmarking;Big Data

会议名称： Pacific Symposium on Biocomputing

主办单位：

页码： 208-219

摘要： Benchmark challenges, such as the Critical Assessment of Structure Prediction (CASP) and Dialogue for Reverse Engineering Assessments and Methods (DREAM) have been instrumental in driving the development of bioinformatics methods. Typically, challenges are posted, and then competitors perform a prediction based upon blinded test data. Challengers then submit their answers to a central server where they are scored. Recent efforts to automate these challenges have been enabled by systems in which challengers submit Docker containers, a unit of software that packages up code and all of its dependencies, to be run on the cloud. Despite their incredible value for providing an unbiased test-bed for the bioinformatics community, there remain opportunities to further enhance the potential impact of benchmark challenges. Specifically, current approaches only evaluate end-to-end performance; it is nearly impossible to directly compare methodologies or parameters. Furthermore, the scientific community cannot easily reuse challengers' approaches, due to lack of specifics, ambiguity in tools and parameters as well as problems in sharing and maintenance. Lastly, the intuition behind why particular steps are used is not captured, as the proposed workflows are not explicitly defined, making it cumbersome to understand the flow and utilization of data. Here we introduce an approach to overcome these limitations based upon the WINGS semantic workflow system. Specifically, WINGS enables researchers to submit complete semantic workflows as challenge submissions. By submitting entries as workflows, it then becomes possible to compare not just the results and performance of a challenger, but also the methodology employed. This is particularly important when dozens of challenge entries may use nearly identical tools, but with only subtle changes in parameters (and radical differences in results). WINGS uses a component driven workflow design and offers intelligent parameter and data selec

分类号： q811