ARCathon 2023

Competition Guide

ARC Challenge

Outline

The ARC challenge is a unique coding problem that requires you to build an AI that can tackle reasoning tasks it has never encountered before. You will be presented with three to five pairs of training inputs and outputs, and a test input, where you must use the pattern identified from the training examples to predict the output.

 

Your objective is to develop a program that can solve 100 secret ARC tasks (the private test set), which are not accessible to the public. To train your program, you have access to 400 public training tasks and 400 public evaluation tasks. These 800 public tasks you can use in any way you see fit.

 

By successfully completing the ARC challenge, you will advance AI towards human-like cognition and unlock new possibilities for AI applications. Given that current deep learning models including large language models cannot solve ARC, it is necessary to think creatively and find new ways to program abstract reasoning at a human level.

Join the Lab42 Community

The Lab42 Discord server is a place for all AI enthusiasts, challenge participants and experts from different fields around the globe. Collaborate, discuss and exchange insights across disciplines!

Where to start

To get familiar with ARC it is advisable to read Chollet's paper On the Measure of Intelligence - especially pp. 46-58 - and read through our ARC Page. Furthermore, Lab42 has created an ARC interface where you can solve the original ARC tasks and even create new ones: ARC Playground.

From there, you can start brainstorming about your own approach. If you need a stepping stone for creativity, see the document Possible First Steps to a Solution.

Guide & Documents

The ultimate Guide to ARCathon

The Ultimate Guide to ARCathon contains all the necessary information about ARC data, submission rules, and Docker, which is the means to submit in the ARCathon. It serves as a kind of encyclopedia, which is constantly expanded according to questions and hints from the community.

 

If you have worked with Docker before and just want to submit your code, the Submission Guide (short) below should be sufficient for you.

ARC Data

The task files are stored in two directories:

  • Training set: contains the 400 task files for training. You can use these tasks to prototype your algorithm or train it to learn cognitive priors that are critical to solving ARC.
  • Evaluation set:  contains the 400 task files for your own evaluation.

The tasks are stored in JSON format. Each task JSON file contains a dictionary with two fields:

  • "train": These are the example input/output-pairs. It is a list of typically three pairs.
  • "test": These are the test input/output-pairs. It is a list of typically one pair.

A pair is a dictionary with two fields:

  • "input": This is the input-grid for the corresponding pair.
  • "output": This is the output grid for the corresponding pair.

Download ARC Data

Submission

The general format for submissions is Docker. Dockers have the advantage that they are portable and can be run on any virtual machine. It uses virtualization at the operating system level to deploy software in packages called containers. All code submitted to ARCathon must conform to the format outlined in the Submission Guide and be submitted as a Docker image!

Starting February 13, you can submit three Docker containers per calendar week until December 1, 2023, which will be evaluated by our team within a maximum of 72 hours. Please submit your Docker image as a link to a registry (Docker Hub, etc.) or as a direct download link via OneDrive, Google Drive, Dropbox, or your preferred platform. Solutions must be submitted via the submission form.

Submission Guide (short)

Essential Documents

General Terms & Conditions

The ultimate Guide to ARCathon

Possible First Steps
to a Solution

Possible First Steps to a Solution

For Humankind

Villa Fontana
Obere Strasse 22B
7270 Davos, Switzerland

Lab42

Powered by 

For Humankind

Lab42
Villa Fontana
Obere Strasse 22B
7270 Davos, Switzerland

Lab42

Powered by

For Humankind

Lab42
Villa Fontana
Obere Strasse 22B
7270 Davos, Switzerland

Lab42

Powered by