How to set up a Databend development environment

Sharp tools make good work. Before starting the journey of Databend contribution, let's configure our own development environment. express setup

express setup

To facilitate developers to quickly establish a development environment, Databend maintains a complex shell script located in scripts/setup/ Only one instruction is needed to complete the development environment configuration:

$ make setup -d

Note: this process will assist in the installation of some python environments, which may affect the local original development environment. It is recommended to execute the following commands in advance to create and enable a dedicated virtual environment.

$ python -m venv .databend
$ source .databend/bin/activate

If you encounter the problem of missing dependencies, you can refer to the "step-by-step installation - Test prerequisites" section for installation. Step by step installation

Step by step installation

Here, take Fedora 36 as an example. Considering the differences between different systems and distributions, you may need to install gcc, python and openssl yourself.

1. Install Rust toolchain:
It is recommended to use rustup to manage Rust toolchain. Refer to Install. about MacOS and Linux users, performing:

$ curl --proto '=https' --tlsv1.2 -sSf | sh

Databend usually uses the newly released nightly tool chain for development, and relevant information is recorded in trust toolchain Toml. Rustup will automatically reload the tool chain when it is used, and only the default configuration is required during installation.

$ cargo build
info: syncing channel updates for 'nightly-2022-05-19-x86_64-unknown-linux-gnu'
info: latest update on 2022-05-19, rust version 1.63.0-nightly (cd282d7f7 2022-05-18)

2. Installation prerequisites:
The following lists some key steps for installing, building and testing the necessary dependencies, and the descriptions and error messages are presented in the form of comments.
Build prerequisites:

#Common hive meta store needs, thrift not found
$ sudo dnf install thrift
#OpenSSL sys requires Can't locate, File/ in @INC
$ sudo dnf install perl-FindBin perl-File-Compare
#is' cmake 'not installed?
# The CMAKE_CXX_COMPILER: c++ is not a full PATH and was not found in the PATH., gcc-c++ and llvm will also be installed when installing clang
$ sudo dnf install cmake clang

Test prerequisites:

#Functional test and subsequent experience needs
$ sudo dnf install mysql
#It includes all Python dependencies required by current functional testing and lint
$ cd tests
$ pip install poetry
$ poetry install
$ poetry shell
#sqllogic test needs (included in the above steps, as needed)
(tests) $ cd logictest
$ pip install -r requirements.txt
#Fuzzy test needs
(tests) $ cd fuzz
$ pip install -r requirements.txt

Lint must have:

#taplo fmt needs
$ cargo install taplo-cli

Editor - Visual Studio Code

  • Access Visual Studio Code.

1. Plug in Recommendation


  • Author: The Rust Programming Language
  • Provide Rust language support for Visual Studio Code.


  • Author: Seray Uzgur
  • Help Rust developers manage cargo Dependencies in toml. Only crites IO dependency.


  • Author: Vadim Chugunov
  • A native debugging tool driven by LLDB. Support debugging C++, Rust and other compiled languages.

Remote - Containers

  • Author: Microsoft
  • Open any folder or Repo in the Docker container and take advantage of the full functionality of Visual Studio Code.

2. Develop with Dev Containers (For Linux)

Install the "Remote - Containers" plug-in. After opening the Databend, you will see a pop-up window in the lower right corner and prompt "Reopen in Container".

Install Docker

Install and start the docker corresponding to your distribution according to Docker Docs - Install. Take Fedora 36 as an example, the steps are as follows:

#Remove old version docker
$ sudo dnf remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-selinux \
                  docker-engine-selinux \
#Set up repository
$ sudo dnf -y install dnf-plugins-core
$ sudo dnf config-manager \
    --add-repo \
#Install Docker Engine
$ sudo dnf install docker-ce docker-ce-cli docker-compose-plugin

Add the current User to docker group reference Docker Docs - PostInstall In the manage docker as a non root user section, you may need to restart.
The steps are as follows

#Add docker user group
$ sudo groupadd docker
#Add users to the docker group
$ sudo usermod -aG docker $USER
#Activate changes
$ newgrp docker
#Change permissions to fix permission denied
$ sudo chown "$USER":"$USER" /home/"$USER"/.docker -R
$ sudo chmod g+rwx "$HOME/.docker" -R

Other steps

Enable Docker:

$ sudo systemctl start docker

Click "open remote window" in the lower left corner and select "Reopen in Container" to experience.

Other utility recommendations

Here are some practical tools that may be helpful to the development of Databend, which can be selected according to the actual situation.


Lightweight, responsive, and infinitely customizable high-value terminal!

 Refer to starship - installation for installation.


curl -sS | sh


Command line benchmarking tool.

Reference hyperfine - installation Install.

cargo install hyperfine

About Databend

Databend is an open-source, flexible, low-cost data warehouse that can do real-time analysis based on object storage. We look forward to your attention and explore cloud native data warehouse solutions to create a new generation of open source Data Cloud.

The article was first published on official account: Databend


Tags: Back-end Big Data programming language

Posted by hanwei on Thu, 04 Aug 2022 01:43:35 +0930