Lab42's first challenge
Lab42calls AI-enthusiasts around the globe to tackle ARC 2
ARC 2 – The Abstraction and Reasoning Corpus 2 Challenge
hosted by François Chollet, holder of the Swiss AI Award
The Abstraction and Reasoning Corpus 2 (ARC 2), challenges everyone to develop a program that not only displays skills but has also the intelligence to create new skills. ARC 2 will feature thousands of new tasks, all unique and created in a crowdsourced approach by people from all over the world. This gives the ARC 2 challenge an unprecedented and unique variety of tasks, unsolvable to current algorithms.
Background
Is a clock intelligent? It is highly skilled at one task: accurately telling time. Does high skill at one task relate to intelligence? What about high skill at a large set of tasks? Is a multifunction electronic watch intelligent?
The seminal publication from 2019, «On the Measure of Intelligence”, by François Chollet, argues that intelligence is the ability of an agent to adapt to an ever-changing environment and produce appropriate behavior in never-seen-before situations. It is the ability to efficiently acquire new skills to tackle novel tasks. Therefore skill, rather than being the same thing as intelligence, is a product of intelligence: intelligence is a process, and skill is its output artifact. Equating skill and intelligence is a category error, much like confusing a basket of bread and a bakery.
Whatever task one picks – Chess or Go or Starcraft – once the task is fixed and known in advance, it becomes possible to prepare specifically for the test, to design or train a system that can leverage arbitrary amounts of human-crafted rules or arbitrary amounts of training data to perform well on the task. The only intelligence on display in such a system is the intelligence of the engineers who came up with the system.
Why feature ARC 2
The Abstraction and Reasoning Corpus (ARC) is a benchmark for human level intelligence. It is meant to be a game that cannot be practiced for: one will be scored on one’s ability to solve problems never seen before, that only share abstract similarities with problems one is already familiar with. If an algorithm or any other scalable method can adapt to the unique tasks of the test set and successfully solve them, it will have exhibit AI that comes close to human level reasoning.
ARC 2 is an ambitious extension of ARC designed to address certain weak points of ARC 1.
ARC 1 was originally released in November 2019 and was the subject of a public Kaggle competition with a $20,000 prize in early 2020 and that ended with 20.6% of test tasks solved by the best entrant. ARC 2 is intended to be larger and more diverse than ARC 1, with no specific task type used more than once. While ARC 1 was the work of a single person, ARC 2 will be the work of many: to ensure scale and task diversity, this is why we are opening the task-creation process to the crowd.
Everyone can create new tasks using the public ARC editor, which will be available soon, and submit tasks for inclusion in ARC 2.
How the competition works
Starting in late 2022, we will run a competition on ARC 2 and make the crowdsourced tasks accessible.
The goal of the competition is to develop a program that is able to solve at least 90.0% of the tasks in the private test set, without advance knowledge of the tasks. Lab42 plans to run the competition until the goal is achieved.
This time not only will there be prize money, but successful participants will also have their names engraved in stone for eternity in the Swiss mountains and the winners will be invited to a special trip to Davos, Switzerland. Depending on progress, Lab42 can award intermediate results, eg half-yearly so as to communicate the current state.
AI-enthusiasts are invited to sign-up to receive the latest news and be one of the first to generate unique tests that will be included in the ARC 2 Challenge, as well as participate in the ARC 2 Challenge itself. Be successful and get a chance to win prizes and literally carve your name in stone for eternity by tackling one of the greatest endeavors of humankind.