# libaio

**Repository Path**: damon_SJTU/libaio

## Basic Information

- **Project Name**: libaio
- **Description**: libaio，异步读取文件与多线程
- **Primary Language**: C++
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2023-08-24
- **Last Updated**: 2023-09-13

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# libaio

## 1. 相关链接：

（1） 参考的一个写的很清楚的： [littedan/linux-aio](https://github.com/littledan/linux-aio)

（2）官网：[libaio](https://pagure.io/libaio)

## 2. 安装

在 libaio_v1.cpp 里，使用了 [Google glog logging library](https://github.com/google/glog) 和 [Google gflags command-line flags library](http://gflags.github.io/gflags/).
```
sudo apt-get install libgoogle-glog-dev
sudo apt-get install libgflags-dev
```

安装 libaio 库：
```
sudo apt-get install libaio-dev
```

安装 BS_thread_pool 库：

只需要将 BS_thread_pool.hpp 放到编译器能找到的位置，比如当前项目下，或者 /usr/include 下，然后在项目文件里，直接 #include "BS_thread_pool.hpp" 即可。这是因为 BS::thread_pool 项目里的所有有效的代码，都被囊括在 #include "BS_thread_pool.hpp" 中了。这是一种被叫做单头文件库的形式。 


## 3. 运行

### (1) libaio_v1.cpp

编译命令： `g++ libaio_v1.cpp -o v1 -laio  -lgflags -lglog`

运行命令： `./v1  --path=log.txt  --file_size=6  --concurrent_requests=2  --min_nr=2  --max_nr=4`

## 4. libaio 使用总结

### AIO 是什么？
libaio 是一个异步读写文件（asynchronous io）的库。相比于最朴素的文件读写方式，比如 std::fstream 而言，fstream 是同步的，aio 是异步的。这也就意味着在进行文件读写的过程中，调用 aio 的主进程可以做其他事情或者被休眠而不使用 CPU 资源，但是同步读写的主进程却会一直阻塞等待，消耗 CPU 资源。不过在文件读写里，aio 似乎不会特别适用，因为我们的读写速度仍然受到带宽的限制。

libaio 是直接使用 linux system call？在更新版本的操作系统等，可以使用一个新的异步io的库，叫做io_uring。这个库的效率比 libaio 的效率更高。

### AIO 使用：

#### I/O context

`io_context_t` is a pointer-sized opaque datatype that represents an “AIO context”. It can be safely passed around by value. Requests in the form of a `struct iocb` are submitted to an `io_context_t` and completions are read from the `io_context_t`. Internally, this structure contains a queue of completed requests. The length of the queue forms an upper bound on the number of concurrent requests which may be submitted to the `io_context_t`.

To create a new io_context_t, use the function

```
int io_setup(int maxevents, io_context_t *ctxp);
```
Here, `ctxp` is the output and `maxevents` is the input. The function creates an `io_context_t` with an internal queue of length maxevents. To deallocate an `io_context_t`, use

```
int io_destroy(io_context_t ctx);
```

There is a system-wide maximum number of allocated `io_context_t`` objects, set at 65536.

An `io_context_t` object can be shared between threads, both for submission and completion. No guarantees are provided about ordering of submission and completion with respect to interaction from multiple threads. There may be performance implications from sharing `io_context_t` objects between threads.

注：使用 libaio 需要一个 io_context_t，在读取前使用 `io_setup` 对其进行初始化，在读取完成后使用 `io_destroy` 进行资源的回收。初始化主要是在内部创建一个 queue，这个 queue 的长度决定了可以同时存在的异步任务的数量的上限（同时，操作系统有一个上限，一般为65536）。

#### Submitting requests
`struct iocb` represents a single request for a read or write operation. The following struct shows a simplification on the struct definition; a full definition is found in `<libaio.h>` within the libaio source code.

```
struct iocb {
    void *data;
    short aio_lio_opcode;
    int aio_fildes;

    union {
        struct {
            void *buf;
            unsigned long nbytes;
            long long offset;
        } c;
    } u;
};
```
The meaning of the fields is as follows: `data`` is a pointer to a user-defined object used to represent the operation

- `aio_lio_opcode` is a flag indicate whether the operation is a read `(IO_CMD_PREAD)` or a write `(IO_CMD_PWRITE)` or one of the other supported operations;
- `aio_fildes` is the fd (file descriptor) of the file that the iocb reads or writes;
- `buf` is the pointer to memory that is read or written;
- `nbytes` is the length of the request;
- `offset` is the initial offset of the read or write within the file;

The convenience functions `io_prep_pread` and `io_prep_pwrite` can be used to initialize a `struct iocb`. New operations are sent to the device with `io_submit`.

注：使用 `io_prep_pread` 和 `io_prep_pwrite` 可以作分别为读和写操作准备好 iocb 对象。
```
void io_prep_pread(struct iocb *iocb, int fd, void *buf, size_t count, off_t offset);
```
需要提供的参数包括，iocb对象指针，文件描述符 fd，读写位置的指针 buf，读写字节数量 count，读写起始位置的偏移量 offset。

```
int io_submit(io_context_t ctx, long nr, struct iocb *ios[]);
```

`io_submit` allows an array of pointers to struct iocbs to be submitted all at once. In this function call, `nr` is the length of the ios array. If multiple operations are sent in one array, then no ordering guarantees are given between the `iocb`s. Submitting in larger batches sometimes results in a performance improvement due to a reduction in CPU usage. A performance improvement also sometimes results from keeping many I/Os ‘in flight’ simultaneously.

If the submission includes too many iocbs such that the internal queue of the `io_context_t` would overfill on completion, then io_submit will return a non-zero number and set errno to `EAGAIN`.

When used under the right conditions, `io_submit` should not block. However, when used in certain ways, it may block, undermining the purpose of asynchronous I/O. If this is a problem for your application, be sure to use the O_DIRECT flag when opening a file, and operate on a raw block device. Work is ongoing to fix the problem.

注：通过 `io_submit` 提交异步读写的任务，第一个参数是 `io_context_t ctx`，第二个参数是任务的数量 `nr`，第三个参数是指向 iocb* 数组的指针，即 iocb** 类型的指针。
需要注意的是，这里的 iocb** 必须是指向 iocb*（或其智能指针也可以） 数组的指针，内部应该会每次移动一个指针的字节数，来寻找下一个iocb* 的位置。所以不能直接定义一个 iocb 对象的数组，并提供这个数组的首位置。

#### Processing results
Completions read from an `io_context_` are of the type `struct io_event`, which contains the following relevant fields.

```
struct io_event {
    void *data;
    struct iocb *obj;
    long long res;
};
```

Here, `data` is the same data pointer that was passed in with the `struct iocb`, and `obj` is the original struct iocb. res is the return value of the read or write.

Completions are reaped with `io_getevents`.
```
int io_getevents(io_context_t ctx_id, long min_nr, long nr, struct io_event *events, struct timespec *timeout);
```

This function has a good number of parameters, so an explanation is in order:

- `ctx_id` is the `io_context_t` that is being reaped from.
- `min_nr`` is the minimum number of io_events to return. io_gevents will block until there are `min_nr` completions to report, if this is not already the case when the function call is made.
- `nr` is the maximum number of completions to return. It is expected to be the length of the events array.
- `events` is an array of io_events into which the information about completions is written.
- `timeout` is the maximum time that a call to io_getevents may block until it will return. If NULL is passed, then io_getevents will block until min_nr completions are available.

The return value represents how many completions were reported, i.e. how much of events was written. The return value will be between 0 and `nr`. The return value may be lower than min_nr if the timeout expires; if the timeout is NULL, then the return value will be between min_nr and nr.

The parameters give a broad range of flexibility in how AIO can be used.

- `min_nr = 0` (or, equivalently, timeout = 0). This option forms a non-blocking polling technique: it will always return immediately, regardless of whether any completions are available. It makes sense to use min_nr = 0 when calling io_getevents as part of a main run-loop of an application, on each iteration.

- `min_nr = 1`. This option blocks until a single completion is available. This parameter is the minimum value which will produce a blocking call, and therefore may be the best value for low latency operations for some users. When an application notices that an eventfd corresponding to an iocb is triggered (see the next section about epoll), then the application can call io_getevents on the corresponding io_context_t with a guarantee that no blocking will occur.

- `min_nr > 1`. This option waits for multiple completions to return, unless the timeout expires. Waiting for multiple completions may improve throughput due to reduced CPU usage, both due to fewer io_getevents calls and because if there is more space in the completion queue due to the removed completions, then a later io_submit call may have a larger granularity, as well as a reduced number of context switches back to the calling thread when the event is available. This option runs the risk of increasing the latency of operations, especially when the operation rate is lower.

Even if min_nr = 0 or 1, it is useful to make nr a bit bigger for performance reasons: more than one event may be already complete, and it could be processed without multiple calls to io_getevents. The only cost of a larger nr value library is that the user must allocate a larger array of events and be prepared to accept them.

注：很多时候，在使用 libaio 的某个结束阶段，我们必须要使用 `io_getevents` 对异步操作进行同步（synchronizing）。只有调用 io_getevents 并得到结果之后，我们才能确定读写操作已经真的完成了。同时，也只有调用 io_getevents 操作后，已完成的事件才会从 io_context_t 的队列 queue 中被取走。
`io_getevents` 函数的两个参数， min_nr, nr 分别指明了至少和至多有多少异步任务被完成之后，返回一次。

#### BS::thread_pool 使用总结

BS::thread_pool 是一种线程池，其主要目的是减少线程创建过程可能带来的严重的 overhead。其好处在于使用很方便，单头文件库的组织形式，使得要使用这个线程池，只需要 #include "BS_thread_pool.hpp" 即可。

#include "BS_thread_pool.hpp" // 引入头文件

BS::thread_pool thread_pool(2); // 创建一个具有2个线程的线程池

thread_pool.submit(...) // 向线程池提交任务
 
res.get(); // 等待任务完成并获得结果，wait() 也会等到任务完成，但不会拿任务的返回值。这里的 res 是上一步 submit() 的返回值，是一个 std::future 类型的对象。