Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BJ266-2 第二组:第二题开发文档翻译 #93

Closed
wants to merge 6 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions blog-cn/PouchContainer 与 Disk quota.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# PouchContainer 与 Disk quota

## 什么是 Disk quota

Disk quota 是一种限制文件系统磁盘用量的技术。PouchContainer 使用 disk quota 来限制文件系统磁盘空间。众所周知,基于块设备的方法能够通过设定块设备的大小,轻松直接地帮助限制磁盘空间的使用,然而基于文件系统的方式很难达成这一点。Disk quota旨在限制文件系统磁盘使用。 目前 PouchContainer 支持以graph driver为基础的OverlayFS。

目前的底层文件系统中仅有 ext4 和 xfs 支持 disk quota。此外,有三种实现方法:**用户配额**,**组配额**和**项目配额**。

限制磁盘用量的两个维度:

* usage quota(block quota):为文件系统设置磁盘用量(并非 inode number);
* file quota (inode quota):限制文件或 inode 的分配。

PouchContainer 目前仅支持 block quota,暂时还不支持 inode。

## PouchContainer 中的 Disk quota

PouchContainer 中的 disk quota 依赖于 PouchContainer 运行的内核版本。下表描述了每个文件系统对 disk quota 的支持:

|| user/group quota | project quota|
|:---:| :----:| :---:|
|ext4| >= 2.6|>= 4.5|
|xfs|>= 2.6|>= 3.10|

虽然相关内核版本中的每个文件系统都支持disk quota,用户还是需要安装 [quota-tools-4.04](https://nchc.dl.sourceforge.net/project/linuxquota/quota-tools/4.04/quota-4.04.tar.gz)。
此配额工具尚未打包到 PouchContainer 的 rpm 中。我们会在将来完成该工具。

## 入门

PouchContainer中有两种方法可以让容器参与底层文件系统。一个是 container rootfs,另一个是从主机(容器外部)绑定的 container volume 。PouchContainer 同时涵盖了这两个维度。

### Container Rootfs Disk quota

用户可以为已创建的 container rootfs 设置标志 `--disk-quota` 以限制磁盘空间使用情况,例:`--disk-quota 10g`。设置成功后,我们可以通过命令 `df -h` 看到rootfs的大小是10GB。这表明disk quota 已经生效了。

```bash
$ pouch run -ti --disk-quota 10g registry.hub.docker.com/library/busybox:latest df -h
Filesystem Size Used Available Use% Mounted on
overlay 10.0G 24.0K 10.0G 0% /
tmpfs 64.0M 0 64.0M 0% /dev
shm 64.0M 0 64.0M 0% /dev/shm
tmpfs 64.0M 0 64.0M 0% /run
tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs 1.9G 0 1.9G 0% /sys/firmware
tmpfs 1.9G 0 1.9G 0% /proc/scsi
```

### Volume Disk quota

用户还可以在创建 volume 时设置其磁盘配额。可以通过添加 `--option` 或 `-o` 标志轻松指定所需的磁盘空间限制数字,例:`-o size = 10g`。


创建 disk quota limited volume 后,用户可以将其绑定到运行容器。以下示例执行了命令:
`pouch run -ti -v volume-quota-test:/ mnt registry.hub.docker.com/library/busybox:latest df -h`。在运行容器中,目录 `/ mnt` 的大小被限制为10GB。

```bash
$ pouch volume create -n volume-quota-test -d local -o mount=/data/volume -o size=10g
Name: volume-quota-test
Scope:
Status: map[mount:/data/volume sifter:Default size:10g]
CreatedAt: 2018-3-24 13:35:08
Driver: local
Labels: map[]
Mountpoint: /data/volume/volume-quota-test

$ pouch run -ti -v volume-quota-test:/mnt registry.hub.docker.com/library/busybox:latest df -h
Filesystem Size Used Available Use% Mounted on
overlay 20.9G 212.9M 19.6G 1% /
tmpfs 64.0M 0 64.0M 0% /dev
shm 64.0M 0 64.0M 0% /dev/shm
tmpfs 64.0M 0 64.0M 0% /run
/dev/sdb2 10.0G 4.0K 10.0G 0% /mnt
tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs 1.9G 0 1.9G 0% /sys/firmware
tmpfs 1.9G 0 1.9G 0% /proc/scsi
```
<!--对于科技文章,它包含大量的英文专有名词,很多时候这些全新的科技术语还没有对应的中文翻译,或者其中文翻译并不是开发者最常用的表达方式。比如文章中的“quota”,并没有太多的开发者在其文档或论坛中使用“配额”一词来代替“quota”。 所以在翻译文档的时候我们还是要注意各国开发者习惯的词汇用法。-->
60 changes: 60 additions & 0 deletions blog-cn/PouchContainer 与 Dragonfly.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# PouchContainer 与 Dragonfly


容器技术有助于IT操作和维护,但是同时,镜像分发带来了巨大挑战。镜像可能有几个 Gibs的大小,拉镜像会非常慢,更不用说同时发送多个拉取请求或网络带宽低的情形。 Dragonfly此时可以发挥重要作用,作为一种P2P技术,它提供了非常高效的分发,避免了镜像分发成为容器技术的瓶颈。

## 什么是 Dragonfly

[Dragonfly](https://github.com/alibaba/Dragonfly#installation) 是一个基于P2P的文件分发系统。它解决了大规模文件分发场景下分发耗时、成功率低、带宽浪费等难题。Dragonfly 显著提升了诸如数据预热、大规模容器镜像分发等业务能力。
### 构成

#### 服务器

服务器由集群管理器组成,它从源代码下载文件并构建P2P网络。

#### 客户端

客户端分为两部分: dfget 和 proxy。dfget是一个用于下载文件的主机端工具,proxy用于截获 http 请求并将它们路由到 dfget。

## 安装Dragonfly


1.安装服务器

服务器可以分别以在物理机上或在容器上这两种方式部署, [安装Dragonfly服务器](https://github.com/alibaba/Dragonfly/blob/master/docs/install_server.md) 的步骤很简单。

2.安装客户端
- 下载[客户端软件包](https://github.com/alibaba/Dragonfly/blob/master/package/df-client.linux-amd64.tar.gz)。
- 解压缩包`tar xzvf df-client.linux-amd64.tar.gz -C /usr/local/bin`,或者你可以解压缩到你喜欢的任何目录,但记得将客户端路径添加到PATH环境中,`export PATH=$PATH:/usr/local/bin/df-client`。
- 将服务器 ip 添加到客户端配置文件中,`/etc/dragonfly.conf`,nodeIp 是你部署服务器的主机 ip。


```
[node]
address=nodeIp1,nodeIp2,...
```

关于详细安装信息,您可以在 [安装Dragonfly客户端](https://github.com/alibaba/Dragonfly/blob/master/docs/install_client.md)中找到。
## 运行 PouchContainer with dragonfly

1.启动Dragonfly代理

启动Dragonfly代理, `/usr/local/bin/df-client/df-daemon --registry https://reg.docker.alibaba-inc.com`, 你可以添加 `--verbose` 标志来获取调试日志。Dragonfly日志可以在目录 `~/.small-dragonfly/logs`中找到。

您可以在[Dragonfly使用](https://github.com/alibaba/Dragonfly/blob/master/docs/usage.md)中找到更多的Dragonfly使用信息。

2.在PouchContainer配置文件中添加以下配置`/etc/pouch/config.json`

```
{
"image-proxy": "http://127.0.0.1:65001"
}
```

3.拉镜像 `reg.docker.alibaba-inc.com/base/busybox:latest`, 您可以在 `~/.small-dragonfly/logs/dfdaemon.log`中找到以下输出,这意味着Dragonfly安装成功。

```
time="2018-03-06 20:08:00" level=debug msg="pre access:http://storage.docker.aliyun-inc.com/docker/registry/v2/blobs/sha256/1b/1b5110ff48b0aa7112544e1666cc7199f812243ded4128f0a1b2be027c7 38bec/data?Expires=1520339335&OSSAccessKeyId=LTAIfYaNrksx0ktL&Signature=fVEYIQzIaXyqIcAhypbmzaUx5x8%3D"
time="2018-03-06 20:08:00" level=debug msg="post access:http://storage.docker.aliyun-inc.com"
time="2018-03-06 20:08:00" level=debug msg="pre access:http://storage.docker.aliyun-inc.com/docker/registry/v2/blobs/sha256/98/9869810a78644038c8d37a5a3344de0217cb37bcc2caa2313036c6948b0 ee8da/data?Expires=1520339335&OSSAccessKeyId=LTAIfYaNrksx0ktL&Signature=Tx7JYU07Gap8RfasvCe0JGAUCo4%3D"
```
61 changes: 61 additions & 0 deletions blog-en/PouchContainer with Dragonfly.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# PouchContainer with Dragonfly

Container technology is helpful to facilitate IT operation and maintenance, but the distribution of images has brought a huge challenge at the same time. The images can be as large as several Gibs, pulling images will be very slow, not to mention the situation lots of pulling requests simultaneously or low network bandwidth. Dragonfly can play an important role here, as a P2P technology, it provides very efficient distribution, avoiding images distribution as a bottleneck in container technology.

## What is Dragonfly

[Dragonfly](https://github.com/alibaba/Dragonfly#installation) is a P2P file distribution system. It solves the problems of time-consuming distribution, low success rate, and wasteful bandwidth in large-scale file distribution scenarios. Dragonfly significantly improves service capabilities such as data preheating, and large-scale container image distribution.

### Composition

#### Server

Server consists of cluster managers, it downloads files from source and constructs a P2P network.

#### Client

Client concludes two part: dfget and proxy. Dfget is a host-side tool for downloading files, and proxy is used for intercepting the http requests and routing them to dfget.

## Install dragonfly

1.Install server

Server can be deployed in two ways, on a physical machine or on a container.The steps are simple in the [install dragonfly server](https://github.com/alibaba/Dragonfly/blob/master/docs/install_server.md)

2.Install client

- Downloading [client package](https://github.com/alibaba/Dragonfly/blob/master/package/df-client.linux-amd64.tar.gz).
- Untar the package `tar xzvf df-client.linux-amd64.tar.gz -C /usr/local/bin`, or you can untar it to any directory you like, but remember to add client path into PATH environment, `export PATH=$PATH:/usr/local/bin/df-client`.
- Add server ip into client config file, `/etc/dragonfly.conf`, nodeIp is the host ip of the sever you deployed.

```
[node]
address=nodeIp1,nodeIp2,...
```

About detail install information, you can find it in [install dragonfly client](https://github.com/alibaba/Dragonfly/blob/master/docs/install_client.md).

## Run PouchContainer with dragonfly

1.Start dragonfly proxy

Start dragonfly proxy, `/usr/local/bin/df-client/df-daemon --registry https://reg.docker.alibaba-inc.com`, you can add flag `--verbose` to get debug log. Dragonfly logs can be found in
directory `~/.small-dragonfly/logs`.

You can find more dragonfly usage information in [dragonfly usage](https://github.com/alibaba/Dragonfly/blob/master/docs/usage.md)

2.Add the following configuration in PouchContainer config file `/etc/pouch/config.json`

```
{
"image-proxy": "http://127.0.0.1:65001"
}
```

3.Pull a image `reg.docker.alibaba-inc.com/base/busybox:latest`, you can find the following output in `~/.small-dragonfly/logs/dfdaemon.log`, which means dragonfly works.

```
time="2018-03-06 20:08:00" level=debug msg="pre access:http://storage.docker.aliyun-inc.com/docker/registry/v2/blobs/sha256/1b/1b5110ff48b0aa7112544e1666cc7199f812243ded4128f0a1b2be027c7 38bec/data?Expires=1520339335&OSSAccessKeyId=LTAIfYaNrksx0ktL&Signature=fVEYIQzIaXyqIcAhypbmzaUx5x8%3D"
time="2018-03-06 20:08:00" level=debug msg="post access:http://storage.docker.aliyun-inc.com"
time="2018-03-06 20:08:00" level=debug msg="pre access:http://storage.docker.aliyun-inc.com/docker/registry/v2/blobs/sha256/98/9869810a78644038c8d37a5a3344de0217cb37bcc2caa2313036c6948b0 ee8da/data?Expires=1520339335&OSSAccessKeyId=LTAIfYaNrksx0ktL&Signature=Tx7JYU07Gap8RfasvCe0JGAUCo4%3D"
```
98 changes: 98 additions & 0 deletions blog-en/pouch_with_diskquota.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# PouchContainer with Disk quota

## What is Disk quota

Disk quota is one kind of technology which is used to restrict filesystem's disk
usage. PouchContainer uses disk quota to limit filesystem disk space. We all know that
the way based on block devices could directly help limit disk space usage
easily via setting size of block device. While the way based on filesystem can
hardly do this. Disk quota is designed for limitting filesystem disk usage.
Currently PouchContainer supports disk quota which is based on graphdriver overlayFS.

Currently in underlying filesystems only ext4 and xfs support disk quota. In
addition, there are three ways to make it: **user quota**, **group quota** and
**project quota**.

There are two dimensions to limit disk usage:

* usage quota(block quota): setting disk usage limit for a filesystem directory(not for inode number);
* file quota(inode quota): restrict file or inode allocation.

PouchContainer only supports block quota now with no inode support temporarily.

## Disk quota in PouchContainer

Disk quota in PouchContainer relies on kernel version PouchContainer runs on. Here is a table
describing when each filesystem supports disk quota.

|| user/group quota | project quota|
|:---:| :----:| :---:|
|ext4| >= 2.6|>= 4.5|
|xfs|>= 2.6|>= 3.10|

Although each filesystem in related kernel version supports disk quota, user
still needs to install [quota-tools-4.04](https://nchc.dl.sourceforge.net/project/linuxquota/quota-tools/4.04/quota-4.04.tar.gz).
This quota tool has not packaged into PouchContainer rpm yet. We will do this in the
future.

## Get Started

There are two ways in PouchContainer for a container to get involved in underlying
filesystems. One is container rootfs, the other is container volume bind from
host(outside of container) to inside. Both two dimensions are covered in PouchContainer.

### Container Rootfs Disk quota

Users can set flag `--disk-quota` for a created container's rootfs to limit
disk space usage, for example `--disk-quota 10g`. After setting this
successfully, we can see rootfs size is 10GB via command `df -h`. And it shows
that disk quota has taken effects.

```bash
$ pouch run -ti --disk-quota 10g registry.hub.docker.com/library/busybox:latest df -h
Filesystem Size Used Available Use% Mounted on
overlay 10.0G 24.0K 10.0G 0% /
tmpfs 64.0M 0 64.0M 0% /dev
shm 64.0M 0 64.0M 0% /dev/shm
tmpfs 64.0M 0 64.0M 0% /run
tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs 1.9G 0 1.9G 0% /sys/firmware
tmpfs 1.9G 0 1.9G 0% /proc/scsi
```

### Volume Disk quota

Users can also setting volume's disk quota when creating one. It is quite easy
to add a `--option` or `-o` flag to specify disk space limit to be desired
number, for example `-o size=10g`.

After creating disk quota limited volume, users can bind this volume to a
running container. In the following example, it executes command
`pouch run -ti -v volume-quota-test:/mnt registry.hub.docker.com/library/busybox:latest df -h`.
And in running container, directory `/mnt` is restricted to be size 10GB.

```bash
$ pouch volume create -n volume-quota-test -d local -o mount=/data/volume -o size=10g
Name: volume-quota-test
Scope:
Status: map[mount:/data/volume sifter:Default size:10g]
CreatedAt: 2018-3-24 13:35:08
Driver: local
Labels: map[]
Mountpoint: /data/volume/volume-quota-test

$ pouch run -ti -v volume-quota-test:/mnt registry.hub.docker.com/library/busybox:latest df -h
Filesystem Size Used Available Use% Mounted on
overlay 20.9G 212.9M 19.6G 1% /
tmpfs 64.0M 0 64.0M 0% /dev
shm 64.0M 0 64.0M 0% /dev/shm
tmpfs 64.0M 0 64.0M 0% /run
/dev/sdb2 10.0G 4.0K 10.0G 0% /mnt
tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs 1.9G 0 1.9G 0% /sys/firmware
tmpfs 1.9G 0 1.9G 0% /proc/scsi
```