ARCathon

Competition Guide

Competition Guide

Approaching the ARC challenge is different than every other coding problem existing today. In this competition, you create an AI that can solve reasoning tasks it has never seen before. Each ARC task consists of three to five pairs of training inputs and outputs, and a test input for which you must predict the corresponding output based on the pattern identified from the training examples.

 

Your goal is to create a program capable of solving the 100 private ARC tasks that are not available to the public. To train this program, you will find 400 public training tasks in the ARC data below. You will also find 400 public evaluation tasks to test how your program performs. Of course, this is just a suggestion of how you can use these 800 public tasks - you can split these datasets as you see fit.

 

If successful, you’ll help bring computers closer to human cognition and you'll open the door to completely new AI applications! Since state-of-the-art deep learning models including large language models are incapable of solving ARC, you should try to think outside the box and find new ways to program human-level abstracting and reasoning.

Where to start

To get familiar with ARC it is advisable to read Chollet's paper On the Measure of Intelligence - especially pp. 46-58 - and read through our ARC Page. Furthermore, Lab42 has created an ARC interface where you can solve the original ARC tasks and even create new ones: ARC Playground.

From there, you can start brainstorming about your own approach. If you need a stepping stone for creativity, see the document Possible First Steps to a Solution.

ARC Data

The task files are stored in two directories:

  • Training set: contains the 400 task files for training. You can use these tasks to prototype your algorithm or train it to learn cognitive priors that are critical to solving ARC.
  • Evaluation set:  contains the 400 task files for your own evaluation.

The tasks are stored in JSON format. Each task JSON file contains a dictionary with two fields:

  • "train": These are the example input/output-pairs. It is a list of typically three pairs.
  • "test": These are the test input/output-pairs. It is a list of typically one pair.

A pair is a dictionary with two fields:

  • "input": This is the input-grid for the corresponding pair.
  • "output": This is the output grid for the corresponding pair.

Download ARC

Submission

The general format for submissions is Docker. Dockers have the advantage that they are portable and can be run on any virtual machine. It uses virtualization at the operating system level to deploy software in packages called containers. All code submitted to ARCathon must conform to the format outlined in the Submission Guide and be submitted as a Docker image!

Beginning December 19th, you will be able to submit two Docker containers per calendar week until December 31, 2022, which are evaluated by Cubera Solutions, a trusted and esteemed Swiss software and design agency based at Lake Zurich.

 

Please submit your Docker image as a direct download link through OneDrive, Google Drive, Dropbox, or your preferred platform. From November 22, 2022, solutions must be submitted via the submission form. All previous submissions via email will be evaluated and do not need to be resubmitted via the form.

Submission Guide

Docker Tutorial

We've created a short tutorial that explains how your code can be integrated in Docker if you use Python as your programming language. But please note that the very reason for using Docker is that you can use any programming language you want!

Docker Python Tutorial

Essential Documents

General Terms & Conditions

Submission Rules & Instructions

Possible First Steps
to a Solution

Lab42

Powered by 

For Humankind

Villa Fontana
Obere Straße 22B
CH-7270 Davos

For Humankind

Lab42
Villa Fontana
Obere Straße 22B
7270 Davos, Switzerland

Lab42

Powered by