GECCO 2015 logo

1st Combinatorial Black-Box Optimization Competition (CBBOC)

July 11-15, 2015 @ GECCO 2015 in Madrid, Spain

Rankings

No Training Track
RankSubmission NameTeam Programming Language
1CMA-VNS Fan "Frank" Xue, The Hong Kong Polytechnic University, China C++
   Geoffrey Q. P. Shen, The Hong Kong Polytechnic University, China  
2 P3 Brian Goldman, Michigan State University, USA C++
3Grafo Abraham Duarte, Universidad Rey Juan Carlos, Spain Java
   Manuel Laguna, Leeds School of Business, University of Colorado at Boulder, USA 
   Rafael Martí, Universidad de Valencia, Spain  
   Jesús Sánchez-Oro, Universidad Rey Juan Carlos, Spain  
4 ABC Carlos García Martínez, Department of Computer Science and Numerical Analysis, University of Córdoba, SpainC++
   Alberto Cano, Department of Computer Science and Numerical Analysis, University of Córdoba, Spain 
   Sebastián Ventura, Department of Computer Science and Numerical Analysis, University of Córdoba, Spain 

Short Training Track
RankSubmission NameTeam Programming Language
1CMA-VNS Fan "Frank" Xue, The Hong Kong Polytechnic University, China C++
   Geoffrey Q. P. Shen, The Hong Kong Polytechnic University, China  
2LTGA Brian Goldman, Michigan State University, USA C++
3SAHH Jerry Swan, University of York, UK C++

Long Training Track
RankSubmission NameTeam Programming Language
1CMA-VNS Fan "Frank" Xue, The Hong Kong Polytechnic University, China C++
   Geoffrey Q. P. Shen, The Hong Kong Polytechnic University, China  
2LaguerreHH Jerry Swan, University of York, UK Java
3Hydra Matthew Martin, Missouri University of Science and Technology, USA Python

Click here for a concise description of the CMA-VNS.

Competition Data

The randomly generated competition problem instances can be downloaded here.

Detailed Result Data
Track Detailed Rankings Log Files
No Training NoneFinalResults.txt NoneFinalResults.tar.gz
Short TrainingShortFinalResults.txtShortFinalResults.tar.gz
Long Training LongFinalResults.txt LongFinalResults.tar.gz

Awards Ceremony

Date & Time: Monday, 13 July 2015, 11:10-12:50
Location: Tapices room, GECCO 2015 conference hotel

Description

This is CBBOC's inaugural year. It's designed to provide the GECCO community with detailed performance comparisons of a wide variety of meta-heuristics and hyper-heuristics on combinatorial problems, where the real-world problems which induce combinatorial problems have been categorized into those with no training time (good fit for out-of-the-box algorithms such as the Parameter-less Population Pyramid), those with short training time (good fit for typical evolutionary algorithms), and those with long training time (good fit for hyper-heuristics). Training and testing time is measured in terms of number of fitness evaluations, although wall time will be used to time-out algorithms taking infeasibly long to complete. Competitors choose which category or categories they want to submit to. While trained differently, all three categories will be compared employing instances drawn from the same test set. This can create a Pareto set of winners, maximizing solution quality while minimizing training time, with at most three nondominated points. The competition problems will be randomly generated by a meta-class based on NK-Landscapes, including - but not limited to - varying number of genes/bits (N), epistasis size (K), epistasis model, fitness table, and maximum number of evaluations per instance. A light-weight API is now available for C++, Java, and Python; instructions for using the API are available here. Competitors who require other languages are encouraged to contact the competition organizers. Competitors are encouraged, though not required, to allow the source code of their competing algorithms to be made available on the competition website. Competitors are not required to attend GECCO 2015.

Competition Details

After all submissions have been made, the competition organizers will use MetaNK.py to generate 20 brand new problem classes. Each submission will then be run in the provided VM against each instance. During the training phase, competitors are provided training instances and can perform a predetermined total number of evaluations, divided how they see fit. The difference between the training categories is only how many evaluations are allowed at this phase. After training is complete, the submission is applied to each testing instance serially, with a specific number of evaluations per instance. This is designed to mimic real-world applications in which initial offline tuning can be flexibly performed, while actual optimization must be done in a specified time window.

We will use the following criteria to compare algorithms. For each testing instance, all submissions will be ranked based on the quality of the best solution found. Ties are broken by the total number of evaluations used before that best solution was found. The overall score of a submission is equal to its average ranking across all test instances of all problem classes. This is designed to favor algorithms which do well across all problems.

Submission Instructions

The submission deadline is June 1, 2015. Submissions can be from a single competitor or a team of competitors. No competitor may be part of more than three submissions, regardless of whether those are single competitor or team submissions. Until the submission deadline, competitors can replace their submissions with updated ones or entirely withdraw a submission. By default, submitted code will be made available from the competition website after the submission deadline, unless the submission E-mail clearly states that the code may not be distributed. Submissions should include the necessary source code as well as instructions about how to build and run it, and be E-mailed to cbboc.organizers@gmail.com. The submission E-mail should furthermore state the full name of each competitor associated with that submission along with their E-mail address and affiliation. It should also specify which category or categories are being submitted to (no training time, short training time, and long training time).

Contact Info

For all matters related to CBBOC, please contact the organizers at cbboc.organizers@gmail.com.

Organizers

[Picture]

E-mail: brianwgoldman@acm.org

Brian W. Goldman is a Ph. D. student in the Department of Computer Science and Engineering at the Michigan State University, and a member of The BEACON Center for the Study of Evolution in Action. He has been a member of the GECCO program committee since 2012, and last year was awarded Best Paper in the Genetic Algorithms track. His research interests focus on how to perform efficient optimization without expert input.
[Picture]

E-mail: jerry.swan@york.ac.uk

Jerry Swan is a Research Fellow at the University of York. Before entering academia, Jerry spent 20 years in industry as a systems architect and software company owner. His research includes meta- and hyper-heuristics, symbolic computation and machine learning. He has published more than 50 papers in international journals and conferences. Jerry has lectured and presented his research worldwide, and has been running international workshops and tutorials on the automated design of metaheuristics since 2011.
[Picture]

E-mail: dtauritz@acm.org

Daniel R. Tauritz is an Associate Professor in the Department of Computer Science at the Missouri University of Science and Technology (S&T), on sabbatical at Sandia National Laboratories for the 2014-2015 academic year, a former Guest Scientist at Los Alamos National Laboratory (LANL), the founding director of S&T's Natural Computation Laboratory, and founding academic director of the LANL/S&T Cyber Security Sciences Institute. He received his Ph.D. in 2002 from Leiden University for Adaptive Information Filtering employing a novel type of evolutionary algorithm. He served previously as GECCO 2010 Late Breaking Papers Chair, COMPSAC 2011 Doctoral Symposium Chair, GECCO 2012 GA Track Co-Chair, and GECCO 2013 GA Track Co-Chair. For several years he has served on the GECCO GA track program committee, the Congress on Evolutionary Computation program committee, and a variety of other international conference program committees. His research interests include the design of hyper-heuristics and self-configuring evolutionary algorithms and the application of computational intelligence techniques in cyber security, critical infrastructure protection, and search-based software engineering. He was granted a US patent for an artificially intelligent rule-based system to assist teams in becoming more effective by improving the communication process between team members.