GitHub - deepseek-ai/3FS: A high-performance distributed file system designed to address the challenges of AI training and inference workloads.

A high-performance distributed file system designed to address the challenges of AI training and inference workloads. - GitHub - deepseek-ai/3FS: A high-performance distributed file system designed to address the challenges of AI training and inference workloads.

Visit Website
GitHub - deepseek-ai/3FS: A high-performance distributed file system designed to address the challenges of AI training and inference workloads.

Introduction

Fire-Flyer File System (3FS)

Build License

Overview

The Fire-Flyer File System (3FS) is a high-performance distributed file system designed to tackle the challenges of AI training and inference workloads. By leveraging modern SSDs and RDMA networks, 3FS provides a shared storage layer that simplifies distributed application development.

Key Features

  • High Performance & Scalability

    • Disaggregated Architecture: Aggregates thousands of SSDs and hundreds of storage nodes, enabling applications to access storage in a locality-oblivious manner.
    • Strong Consistency: Uses Chain Replication with Apportioned Queries (CRAQ) to ensure strong consistency, simplifying application logic.
    • Familiar File Interfaces: Built on FoundationDB with transactional key-value storage, ensuring compatibility with existing storage APIs.

  • Optimized for AI Workloads

    • Data Preparation: Organizes large-scale data pipelines efficiently with hierarchical directory structures.
    • Dataloaders: Eliminates the need for prefetching or shuffling datasets by enabling random access to training samples across compute nodes.
    • Checkpointing: Supports high-throughput, parallel checkpointing for large-scale AI training.
    • KVCache for Inference: Provides a cost-effective alternative to DRAM-based caching, with high throughput and large capacity.

Performance Benchmarks

1. Peak Throughput

A 180-node 3FS cluster achieved an aggregate read throughput of 6.6 TiB/s under stress testing. Each storage node was equipped with 16×14TiB NVMe SSDs and 2×200Gbps InfiniBand NICs, while 500+ client nodes were used for testing.

2. GraySort Benchmark

3FS was evaluated using the GraySort benchmark, sorting 110.5 TiB of data across 8,192 partitions in 30 minutes and 14 seconds, achieving an average throughput of 3.66 TiB/min.

3. KVCache Performance

For LLM inference optimization, 3FS’s KVCache reached a peak read throughput of 40 GiB/s, providing an efficient caching mechanism for AI applications.

Installation & Setup

Clone the Repository

git clone https://github.com/deepseek-ai/3fs
cd 3fs
git submodule update --init --recursive
./patches/apply.sh

Install Dependencies

For Ubuntu 20.04:

apt install cmake libuv1-dev liblz4-dev liblzma-dev libdouble-conversion-dev \
  libprocps-dev libdwarf-dev libunwind-dev libaio-dev libgflags-dev \
  libgoogle-glog-dev libgtest-dev libgmock-dev clang-format-14 clang-14 \
  clang-tidy-14 lld-14 libgoogle-perftools-dev google-perftools libssl-dev \
  ccache libclang-rt-14-dev gcc-10 g++-10 libboost1.71-all-dev

For Ubuntu 22.04:

apt install cmake libuv1-dev liblz4-dev liblzma-dev libdouble-conversion-dev \
  libprocps-dev libdwarf-dev libunwind-dev libaio-dev libgflags-dev \
  libgoogle-glog-dev libgtest-dev libgmock-dev clang-format-14 clang-14 \
  clang-tidy-14 lld-14 libgoogle-perftools-dev google-perftools libssl-dev \
  ccache gcc-12 g++-12 libboost-all-dev

Additional prerequisites:

Build 3FS

cmake -S . -B build -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14 \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
cmake --build build -j 32

Run a Test Cluster

Follow the setup guide to deploy a test cluster.

Reporting Issues

For bug reports and feature requests, visit the 3FS GitHub Issues page.