Module tokio::fs

Asynchronous file utilities.

This module contains utility methods for working with the file system asynchronously. This includes reading/writing to files, and working with directories.

Be aware that most operating systems do not provide asynchronous file system APIs. Because of that, Tokio will use ordinary blocking file operations behind the scenes. This is done using the spawn_blocking threadpool to run them in the background.

The tokio::fs module should only be used for ordinary files. Trying to use it with e.g., a named pipe on Linux can result in surprising behavior, such as hangs during runtime shutdown. For special files, you should use a dedicated type such as tokio::net::unix::pipe or AsyncFd instead.

Currently, Tokio will always use spawn_blocking on all platforms, but it may be changed to use asynchronous file system APIs such as io_uring in the future.

Usage

The easiest way to use this module is to use the utility functions that operate on entire files:

• tokio::fs::read
• tokio::fs::read_to_string
• tokio::fs::write

The two read functions reads the entire file and returns its contents. The write function takes the contents of the file and writes those contents to the file. It overwrites the existing file, if any.

For example, to read the file:

let contents = tokio::fs::read_to_string("my_file.txt").await?;

println!("File has {} lines.", contents.lines().count());

To overwrite the file:

let contents = "First line.\nSecond line.\nThird line.\n";

tokio::fs::write("my_file.txt", contents.as_bytes()).await?;

Using File

The main type for interacting with files is File. It can be used to read from and write to a given file. This is done using the AsyncRead and AsyncWrite traits. This type is generally used when you want to do something more complex than just reading or writing the entire contents in one go.

Note: It is important to use flush when writing to a Tokio File. This is because calls to write will return before the write has finished, and flush will wait for the write to finish. (The write will happen even if you don’t flush; it will just happen later.) This is different from std::fs::File, and is due to the fact that File uses spawn_blocking behind the scenes.

For example, to count the number of lines in a file without loading the entire file into memory:

use tokio::fs::File;
use tokio::io::AsyncReadExt;

let mut file = File::open("my_file.txt").await?;

let mut chunk = vec![0; 4096];
let mut number_of_lines = 0;
loop {
    let len = file.read(&mut chunk).await?;
    if len == 0 {
        // Length of zero means end of file.
        break;
    }
    for &b in &chunk[..len] {
        if b == b'\n' {
            number_of_lines += 1;
        }
    }
}

println!("File has {} lines.", number_of_lines);

For example, to write a file line-by-line:

use tokio::fs::File;
use tokio::io::AsyncWriteExt;

let mut file = File::create("my_file.txt").await?;

file.write_all(b"First line.\n").await?;
file.write_all(b"Second line.\n").await?;
file.write_all(b"Third line.\n").await?;

// Remember to call `flush` after writing!
file.flush().await?;

Tuning your file IO

Tokio’s file uses spawn_blocking behind the scenes, and this has serious performance consequences. To get good performance with file IO on Tokio, it is recommended to batch your operations into as few spawn_blocking calls as possible.

One example of this difference can be seen by comparing the two reading examples above. The first example uses tokio::fs::read, which reads the entire file in a single spawn_blocking call, and then returns it. The second example will read the file in chunks using many spawn_blocking calls. This means that the second example will most likely be more expensive for large files. (Of course, using chunks may be necessary for very large files that don’t fit in memory.)

The following examples will show some strategies for this:

When creating a file, write the data to a String or Vec<u8> and then write the entire file in a single spawn_blocking call with tokio::fs::write.

let mut contents = String::new();

contents.push_str("First line.\n");
contents.push_str("Second line.\n");
contents.push_str("Third line.\n");

tokio::fs::write("my_file.txt", contents.as_bytes()).await?;

Use BufReader and BufWriter to buffer many small reads or writes into a few large ones. This example will most likely only perform one spawn_blocking call.

use tokio::fs::File;
use tokio::io::{AsyncWriteExt, BufWriter};

let mut file = BufWriter::new(File::create("my_file.txt").await?);

file.write_all(b"First line.\n").await?;
file.write_all(b"Second line.\n").await?;
file.write_all(b"Third line.\n").await?;

// Due to the BufWriter, the actual write and spawn_blocking
// call happens when you flush.
file.flush().await?;

Manually use std::fs inside spawn_blocking.

use std::fs::File;
use std::io::{self, Write};
use tokio::task::spawn_blocking;

spawn_blocking(move || {
    let mut file = File::create("my_file.txt")?;

    file.write_all(b"First line.\n")?;
    file.write_all(b"Second line.\n")?;
    file.write_all(b"Third line.\n")?;

    // Unlike Tokio's file, the std::fs file does
    // not need flush.

    io::Result::Ok(())
}).await.unwrap()?;

It’s also good to be aware of File::set_max_buf_size, which controls the maximum amount of bytes that Tokio’s File will read or write in a single spawn_blocking call. The default is two megabytes, but this is subject to change.

Modules

• canonicalize 🔒
• copy 🔒
• create_dir 🔒
• create_dir_all 🔒
• dir_builder 🔒
• file 🔒
  Types for working with File.
• hard_link 🔒
• metadata 🔒
• open_options 🔒
• read 🔒
• read_dir 🔒
• read_link 🔒
• read_to_string 🔒
• remove_dir 🔒
• remove_dir_all 🔒
• remove_file 🔒
• rename 🔒
• set_permissions 🔒
• symlink 🔒
• symlink_metadata 🔒
• try_exists 🔒
• write 🔒

Structs

• DirBuilder
  A builder for creating directories in various manners.
• DirEntry
  Entries returned by the ReadDir stream.
• File
  A reference to an open file on the filesystem.
• OpenOptions
  Options and flags which can be used to configure how a file is opened.
• ReadDir
  Reads the entries in a directory.

Functions

• asyncify 🔒
• canonicalize
  Returns the canonical, absolute form of a path with all intermediate components normalized and symbolic links resolved.
• copy
  Copies the contents of one file to another. This function will also copy the permission bits of the original file to the destination file. This function will overwrite the contents of to.
• create_dir
  Creates a new, empty directory at the provided path.
• create_dir_all
  Recursively creates a directory and all of its parent components if they are missing.
• hard_link
  Creates a new hard link on the filesystem.
• metadata
  Given a path, queries the file system to get information about a file, directory, etc.
• read
  Reads the entire contents of a file into a bytes vector.
• read_dir
  Returns a stream over the entries within a directory.
• read_link
  Reads a symbolic link, returning the file that the link points to.
• read_to_string
  Creates a future which will open a file for reading and read the entire contents into a string and return said string.
• remove_dir
  Removes an existing, empty directory.
• remove_dir_all
  Removes a directory at this path, after removing all its contents. Use carefully!
• remove_file
  Removes a file from the filesystem.
• rename
  Renames a file or directory to a new name, replacing the original file if to already exists.
• set_permissions
  Changes the permissions found on a file or a directory.
• symlink
  Creates a new symbolic link on the filesystem.
• symlink_metadata
  Queries the file system metadata for a path.
• try_exists
  Returns Ok(true) if the path points at an existing entity.
• write
  Creates a future that will open a file for writing and write the entire contents of contents to it.