Approaching the ARC challenge is different than every other coding problem existing today. In this competition, you create an AI that can solve reasoning tasks it has never seen before. Each ARC task consists of three to five pairs of training inputs and outputs, and a test input for which you must predict the corresponding output based on the pattern identified from the training examples.
Your goal is to create a program capable of solving the 100 private ARC tasks that are not available to the public. To train this program, you will find 400 public training tasks in the ARC data below. You will also find 400 public evaluation tasks to test how your program performs. Of course, this is just a suggestion of how you can use these 800 public tasks - you can split these datasets as you see fit.
If successful, you’ll help bring computers closer to human cognition and you'll open the door to completely new AI applications! Since state-of-the-art deep learning models including large language models are incapable of solving ARC, you should try to think outside the box and find new ways to program human-level abstracting and reasoning.
The task files are stored in two directories:
- Training set: contains the 400 task files for training. You can use these tasks to prototype your algorithm or train it to learn cognitive priors that are critical to solving ARC.
- Evaluation set: contains the 400 task files for your own evaluation.
The tasks are stored in JSON format. Each task JSON file contains a dictionary with two fields:
- "train": These are the example input/output-pairs. It is a list of typically three pairs.
- "test": These are the test input/output-pairs. It is a list of typically one pair.
A pair is a dictionary with two fields:
- "input": This is the input-grid for the corresponding pair.
- "output": This is the output grid for the corresponding pair.
The general format for submissions is Docker. Dockers have the advantage that they are portable and can be run on any virtual machine. It uses virtualization at the operating system level to deploy software in packages called containers. All code submitted to ARCathon must conform to the format outlined in the Submission Guide and be submitted as a Docker image!
Beginning December 19th, you will be able to submit two Docker containers per calendar week until December 31, 2022, which are evaluated by Cubera Solutions, a trusted and esteemed Swiss software and design agency based at Lake Zurich.
Please submit your Docker image as a direct download link through OneDrive, Google Drive, Dropbox, or your preferred platform. From November 22, 2022, solutions must be submitted via the submission form. All previous submissions via email will be evaluated and do not need to be resubmitted via the form.
We've created a short tutorial that explains how your code can be integrated in Docker if you use Python as your programming language. But please note that the very reason for using Docker is that you can use any programming language you want!