Video2Plan: Learning Domain and Problem Representation

In this task you are given the DROID dataset and you are asked for the block object type to create the PDDL domain and multiple (at least 3) problem files.

You will use a Visual Language Model (VLM) that can process images and text to help you extract the necessary information from the DROID dataset or finetune it to the DROID dataset. You can use pretrained models such as CLIP, BLIP, or any other suitable VLM for this task.

PDDL Domain

Write a PDDL domain file that includes the following:

• A set of types that represent the entities in the domain.

• A set of predicates that represent the relationships between the entities.

• A set of actions that represent the operations that can be performed in the domain. Each action should have preconditions and effects that are relevant to the domain.

PDDL Problem

Write a PDDL problem file that includes the following:

• A set of objects that represent the specific instances of the types defined in the domain file.

• A set of init predicates that represent the initial state.

• A set of goal predicates that represent the desired state.

The generation of the PDDL specs should be automated. You can use abstractions of the Unified Planning library if you need to and use its ability to read and write PDDL files to verify the correctness of your generated PDDL files.

Note

You are free to use relevant VSCode plugins to help you with syntax highlighting of the PDDL domain and problem specs.