A modular embodied agent architecture and platform for building embodied agents

This repository, corresponding tutorials and docs are still being refined (and not ready yet).

droidlet helps you rapidly build agents (real or virtual) that perform a wide variety of tasks specified by humans. The agents can use natural language, memory and humans in the loop.

droidlet is an early research project for AI researchers to explore ideas around grounded dialogue, interactive learning and human-computer interfaces.

droidlet is in active development and is fairly unstable in design, API, performance and correctness. It is not meant for any production use.

Reach out to us at droidlet@fb.com, to discuss your use case or just share your thoughts!

Getting Started

You want to do one of three things:

1. Robots: Reproduce and extend the PyRobot based agent on real robots such as LocoBot or photo-realistic simulators such as AIHabitat.
2. Minecraft: Reproduced and extend the minecraft based game agent
3. New Agent: write your own agent from scratch, starting from our base_agent abstraction

git clone --recursive https://github.com/facebookresearch/droidlet.git
cd droidlet
            

- Linux
- Python 3 (Anaconda recommended)
- NVIDIA GPU (8GB+)
- PyRobot-compatible robot or sim
  - Habitat-sim instructions below
        

        
- Linux
- Python 3 (Anaconda recommended)
- NVIDIA GPU (4GB+)
- Minecraft
  - more instructions below
        

        
- Linux
- Python 3 (Anaconda recommended)
        

conda create -n droidlet_env python=3.7 \
   pytorch==1.7.1 torchvision==0.8.2 \
   cudatoolkit=11.0 -c pytorch
conda activate droidlet_env
pip install -r \
    agents/locobot/requirements.txt
python setup.py develop
        

        
pip install -r \
    agents/craftassist/requirements.txt
        

        
pip install -r requirements.txt
        

Documentation, Tutorials and Papers

Two papers cover the design of droidlet:

1. droidlet: modular, heterogenous, multi-modal agents covers the overall design of droidlet as an embodied AI platform that is extensible to physical robots and simulators.
2. CraftAssist: A Framework for Dialogue-enabled Interactive Agents covers the design of the dialogue parser and the task system of an earlier version of droidlet that is specific to the game Minecraft

Citation

If you use droidlet in your work, please cite our arXiv paper:

@misc{pratik2021droidlet,
      title={droidlet: modular, heterogenous, multi-modal agents}, 
      author={Anurag Pratik and Soumith Chintala and Kavya Srinet and Dhiraj Gandhi and Rebecca Qian and Yuxuan Sun and Ryan Drew and Sara Elkafrawy and Anoushka Tiwari and Tucker Hart and Mary Williamson and Abhinav Gupta and Arthur Szlam},
      year={2021},
      eprint={2101.10384},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}

License

droidlet is MIT licensed.

Other Links

Datasets

Download links to the datasets described in section 6 of Technical Whitepaper are provided here:

• The house dataset: https://craftassist.s3-us-west-2.amazonaws.com/pubr/house_data.tar.gz
• The segmentation dataset: https://craftassist.s3-us-west-2.amazonaws.com/pubr/instance_segmentation_data.tar.gz
• The dialogue dataset: https://craftassist.s3-us-west-2.amazonaws.com/pubr/dialogue_data.tar.gz

In the root of each tarball is a README that details the file structure contained within.