OpenDAL

Open Data Access Layer: Access data freely, painlessly, and efficiently

Documentation: stable | main
Release notes

Services

azblob: Azure Storage Blob services.
azdfs: Azure Data Lake Storage Gen2 services. (As known as abfs)
dashmap: dashmap backend support.
fs: POSIX alike file system.
ftp: FTP and FTPS support.
gcs: Google Cloud Storage Service.
ghac: Github Action Cache Service.
hdfs: Hadoop Distributed File System(HDFS).
http: HTTP read-only services.
ipfs: InterPlanetary File System HTTP Gateway support.
ipmfs: InterPlanetary File System MFS API support.
memcached: Memcached service support.
memory: In memory backend.
moka: moka backend support.
obs: Huawei Cloud Object Storage Service (OBS).
oss: Aliyun Object Storage Service (OSS).
redis: Redis services support.
rocksdb: RocksDB services support.
s3: AWS S3 alike services.
sled: sled services support.
webdav: WebDAV Service Support.
webhdfs: WebHDFS Service Support.

Features

Access data freely

Access different storage services in the same way
Behavior tests for all services
Cross language/project bindings (working on)

Access data painlessly

100% documents covered
Powerful Layers
Automatic retry support
Full observability: logging, tracing, metrics.
Native chaos testing

Access data efficiently

Zero cost: Maps to API calls directly
Best effort: Automatically selects best read/seek/next based on services
Avoid extra calls: Reuses metadata when possible

Quickstart

use opendal::Result;
use opendal::layers::LoggingLayer;
use opendal::services;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    // Pick a builder and configure it.
    let mut builder = services::S3::default();
    builder.bucket("test");

    // Init an operator
    let op = Operator::create(builder)?
        // Init with logging layer enabled.
        .layer(LoggingLayer::default())
        .finish();

    // Write data
    op.write("hello.txt", "Hello, World!").await?;

    // Read data
    let bs = op.read("hello.txt").await?;

    // Fetch metadata
    let meta = op.stat("hello.txt").await?;
    let mode = meta.mode();
    let length = meta.content_length();

    // Delete
    op.delete("hello.txt").await?;

    Ok(())
}

More examples could be found at Documentation.

Projects

Databend: A modern Elasticity and Performance cloud data warehouse.
GreptimeDB: An open-source, cloud-native, distributed time-series database.
deepeth/mars: The powerful analysis platform to explore and visualize data from blockchain.
mozilla/sccache: sccache is ccache with cloud storage
risingwave: A Distributed SQL Database for Stream Processing
Vector: A high-performance observability data pipeline.

Contributing

Check out the CONTRIBUTING guide for more details on getting started with contributing to this project.

Getting help

Submit issues for bug report or asking questions in the Discussions forum.

Talk to develops at discord.

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0