Huntersafe508

A couple of years ago, Will Pleasant-Ryan wrote Docker for Mac: Overcoming Slow Mounted Volumes, describing his desire to use Docker for local development. He talked about some of the filesystem performance problems that can arise when using a shared volume from the host machine on Mac OS X, along with some potential workarounds and word of some upcoming performance improvements.

• Windows Partition Manager Ext4
• Docker For Mac Slow Ext4 Partition Usb
• Docker For Mac Slow Ext4 Partition Recovery Software
• Docker For Mac Slow Ext4 Partition Recovery Software

In the time since that post was written, those improvements have been released. But I’ve still found it to be unacceptably slow to have the source code mounted from the host machine, at least for something like a Ruby on Rails application.

Docker Desktop Version: docker version Client: Docker Engine - Community Cloud integration 0.1.18 Version: 19.03.13 API version: 1.40 Go version: go1.13.15 Git commit: 4484c46d9d Built: Wed Sep 16 17: OS/Arch: windows/amd64 Experimental: false.

However, earlier this summer, a Visual Studio Code update introduced the ability to “attach to a running container.” Once attached, the experience when editing files that reside on a container using VS Code is nearly indistinguishable from that of editing files on the local filesystem with VS Code!

1. Docker Compose

I’m using Docker Compose to manage the containers that make up my development environment. There’s plenty of existing documentation on how to use Docker Compose, but I want to point out a couple of things that I’ve done because I’m using a container to host the source code I’m editing.

A named volume for source code

If you just checked out a repository into a normal directory inside the container, you’d have to be careful not to lose any changes that had not been committed and pushed whenever making changes to container (updating something in the Dockerfile, etc.). But if the source code is checked out into a named volume, the container can be re-built over and over again, with your source code being mounted exactly as it was before.

• The way you are trying to install docker on another partition is not correct. You need a partition (sdb7 in your case).Suppose its mounted at /mnt/sdb7. Docker by default store all its data (containers, images and so on) in /var/lib/docker you need to change this in your case to something like /mnt/sdb7/docker using data-root option of docker. While starting docker deamon use -data-root option.
• Jul 07, 2019 If you work with Linux, you probably have a hard drive or two formatted with Ext4 or a related filesystem. Assuming you only work with Linux, that isn’t a problem. When you need to access data from that Ext4 filesystem on another operating system, you start to run into trouble. Macs, for example, don’t support Ext4 filesystems.
• I've found that creating a CoreOS VM under Parallels, then using the Docker that is inside CoreOS is far faster than Docker for Mac (currently running Version 17.12.0-ce-mac49 (21995)). I'm doing Linux code builds using CMAKE/Ninja/GCC and it's almost twice as fast as the exact same build from Docker for Mac.

A long-running container

Similar to a database container, your “dev” container should stay running when you do a docker-compose up so you can attach to it from VS Code. With Docker Compose, you just need to set the tty option to true in your docker-compose.yml.

Here’s an example docker-compose.yml file that has a couple of named volumes, one for the source code directory and the other for the contents of the database. I’m using this for an old Ruby on Rails project, hence the old version of MySQL.

2. Working in the Dev Container

The idea is that any developer could check out the repository on their laptop, run docker-compose up, and have a working development environment. But you’ll still need a little customization of the container once it’s been built.

Check out the source code

Once you have your container up and running (docker-compose up), you’ll need to check out your source code inside the container. You can’t do this as part of the Dockerfile because we want the repository to live in the mounted volume.

If you rely on your SSH keys to authenticate with your Git server, then you’ll want to either mount your ~/.ssh directory in the container or copy your private key into the container manually. I needed to do the latter because my ~/.ssh/config has some things that aren’t compatible with the Ubuntu version running in the container.

These examples assume the container will be run as a user called “dev.”

I also like to have my normal Git configuration available as well:

Now you just check out your source code, but inside the container instead of on your host machine.

At this point, you can open a shell in the container and run tests, start the development server, etc.

3. Editing in VS Code

With the “dev” container running, inside of Visual Studio Code, run the “Remote-Containers: Attach to Running Container” command and select your “dev” container. VS Code will attach and install everything it needs to run on the container.

If you select the File Explorer, you’ll see an “Open Folder” button that, when clicked, will display folders on the container. Select the location of the source code that was checked out previously, and click OK. You’ll now be able to edit files as if they were hosted on the local filesystem.

You can even open an integrated terminal in VS Code, and it will automatically open a shell in the attached container!

Conclusion

I’ve only recently started playing with hosting my entire development environment, including the source code, inside Docker containers, but the experience using VS Code as an editor has been fantastic.

With the Mac OS X filesystem integration performance improvements that have been made, many projects will be just fine using shared volumes mounted from the host system. But in cases where that just won’t cut it, hosting the source in the container and using VS Code to attach to the container is a really, really nice solution.

Estimated reading time: 11 minutes

Btrfs is a next generation copy-on-write filesystem that supports many advancedstorage technologies that make it a good fit for Docker. Btrfs is included inthe mainline Linux kernel.

Docker’s btrfs storage driver leverages many Btrfs features for image andcontainer management. Among these features are block-level operations, thinprovisioning, copy-on-write snapshots, and ease of administration. You caneasily combine multiple physical block devices into a single Btrfs filesystem.

This article refers to Docker’s Btrfs storage driver as btrfs and the overallBtrfs Filesystem as Btrfs.

Note: The btrfs storage driver is only supported on Docker Engine - Community on Ubuntu or Debian.

Prerequisites

btrfs is supported if you meet the following prerequisites:

• Docker Engine - Community: For Docker Engine - Community, btrfs is only recommended on Ubuntu or Debian.

• Changing the storage driver makes any containers you have alreadycreated inaccessible on the local system. Use docker save to save containers,and push existing images to Docker Hub or a private repository, so that younot need to re-create them later.

  To resolve this, go to File option, click on Save As, and rename your file. Check if the issue is settled. If the issue isn’t solved, proceed to the next fix. Option 2: Save your file to some other location: As mentioned above, Excel won’t save your file due to an insufficient amount of space in your disk, antivirus conflict with the excel. Won%27t save file. May 04, 2019 If the file you opened is on a read-only medium like a CD-ROM or DVD-ROM, you cannot make changes and save them to the original file. Solution Open the file and Save as to a new location on a writable medium such as your hard drive.

• btrfs requires a dedicated block storage device such as a physical disk. Thisblock device must be formatted for Btrfs and mounted into /var/lib/docker/.The configuration instructions below walk you through this procedure. Bydefault, the SLES / filesystem is formatted with BTRFS, so for SLES, you donot need to use a separate block device, but you can choose to do so forperformance reasons.

• btrfs support must exist in your kernel. To check this, run the followingcommand:

• To manage BTRFS filesystems at the level of the operating system, you need thebtrfs command. If you do not have this command, install the btrfsprogspackage (SLES) or btrfs-tools package (Ubuntu).

Configure Docker to use the btrfs storage driver

This procedure is essentially identical on SLES and Ubuntu.

1. Stop Docker.

2. Copy the contents of /var/lib/docker/ to a backup location, then emptythe contents of /var/lib/docker/:

3. Format your dedicated block device or devices as a Btrfs filesystem. Thisexample assumes that you are using two block devices called /dev/xvdf and/dev/xvdg. Double-check the block device names because this is adestructive operation.

   There are many more options for Btrfs, including striping and RAID. See theBtrfs documentation.

4. Mount the new Btrfs filesystem on the /var/lib/docker/ mount point. Youcan specify any of the block devices used to create the Btrfs filesystem.

   Don’t forget to make the change permanent across reboots by adding anentry to /etc/fstab.

5. Copy the contents of /var/lib/docker.bk to /var/lib/docker/.

6. Configure Docker to use the btrfs storage driver. This is required eventhough /var/lib/docker/ is now using a Btrfs filesystem.Edit or create the file /etc/docker/daemon.json. If it is a new file, addthe following contents. If it is an existing file, add the key and valueonly, being careful to end the line with a comma if it is not the finalline before an ending curly bracket (}).

   See all storage options for each storage driver in thedaemon reference documentation

7. Start Docker. After it is running, verify that btrfs is being used as thestorage driver.

8. When you are ready, remove the /var/lib/docker.bk directory.

Manage a Btrfs volume

One of the benefits of Btrfs is the ease of managing Btrfs filesystems withoutthe need to unmount the filesystem or restart Docker.

When space gets low, Btrfs automatically expands the volume in chunks ofroughly 1 GB.

To add a block device to a Btrfs volume, use the btrfs device add andbtrfs filesystem balance commands.

Note: While you can do these operations with Docker running, performancesuffers. It might be best to plan an outage window to balance the Btrfsfilesystem.

How the btrfs storage driver works

The btrfs storage driver works differently from devicemapper or otherstorage drivers in that your entire /var/lib/docker/ directory is stored on aBtrfs volume.

Image and container layers on-disk

Information about image layers and writable container layers is stored in/var/lib/docker/btrfs/subvolumes/. This subdirectory contains one directoryper image or container layer, with the unified filesystem built from a layerplus all its parent layers. Subvolumes are natively copy-on-write and have spaceallocated to them on-demand from an underlying storage pool. They can also benested and snapshotted. The diagram below shows 4 subvolumes. ‘Subvolume 2’ and‘Subvolume 3’ are nested, whereas ‘Subvolume 4’ shows its own internal directorytree.

Only the base layer of an image is stored as a true subvolume. All the otherlayers are stored as snapshots, which only contain the differences introducedin that layer. You can create snapshots of snapshots as shown in the diagrambelow.

On disk, snapshots look and feel just like subvolumes, but in reality they aremuch smaller and more space-efficient. Copy-on-write is used to maximize storageefficiency and minimize layer size, and writes in the container’s writable layerare managed at the block level. The following image shows a subvolume and itssnapshot sharing data.

Windows Partition Manager Ext4

For maximum efficiency, when a container needs more space, it is allocated inchunks of roughly 1 GB in size.

Docker For Mac Slow Ext4 Partition Usb

Docker’s btrfs storage driver stores every image layer and container in itsown Btrfs subvolume or snapshot. The base layer of an image is stored as asubvolume whereas child image layers and containers are stored as snapshots.This is shown in the diagram below.

Docker For Mac Slow Ext4 Partition Recovery Software

The high level process for creating images and containers on Docker hostsrunning the btrfs driver is as follows:

1. The image’s base layer is stored in a Btrfs subvolume under/var/lib/docker/btrfs/subvolumes.

2. Subsequent image layers are stored as a Btrfs snapshot of the parentlayer’s subvolume or snapshot, but with the changes introduced by thislayer. These differences are stored at the block level.

3. The container’s writable layer is a Btrfs snapshot of the final image layer,with the differences introduced by the running container. These differencesare stored at the block level.

How container reads and writes work with btrfs

Docker For Mac Slow Ext4 Partition Recovery Software

Reading files

A container is a space-efficient snapshot of an image. Metadata in the snapshotpoints to the actual data blocks in the storage pool. This is the same as witha subvolume. Therefore, reads performed against a snapshot are essentially thesame as reads performed against a subvolume.

Writing files

• Writing new files: Writing a new file to a container invokes an allocate-on-demandoperation to allocate new data block to the container’s snapshot. The file isthen written to this new space. The allocate-on-demand operation is native toall writes with Btrfs and is the same as writing new data to a subvolume. As aresult, writing new files to a container’s snapshot operates at native Btrfsspeeds.

• Modifying existing files: Updating an existing file in a container is a copy-on-writeoperation (redirect-on-write is the Btrfs terminology). The original data isread from the layer where the file currently exists, and only the modifiedblocks are written into the container’s writable layer. Next, the Btrfs driverupdates the filesystem metadata in the snapshot to point to this new data.This behavior incurs very little overhead.

• Deleting files or directories: If a container deletes a file or directorythat exists in a lower layer, Btrfs masks the existence of the file ordirectory in the lower layer. If a container creates a file and then deletesit, this operation is performed in the Btrfs filesystem itself and the spaceis reclaimed.

With Btrfs, writing and updating lots of small files can result in slowperformance.

Btrfs and Docker performance

There are several factors that influence Docker’s performance under the btrfsstorage driver.

Note: Many of these factors are mitigated by using Docker volumes forwrite-heavy workloads, rather than relying on storing data in the container’swritable layer. However, in the case of Btrfs, Docker volumes still sufferfrom these draw-backs unless /var/lib/docker/volumes/ is not backed byBtrfs.

• Page caching. Btrfs does not support page cache sharing. This means thateach process accessing the same file copies the file into the Docker hosts’smemory. As a result, the btrfs driver may not be the best choicehigh-density use cases such as PaaS.

• Small writes. Containers performing lots of small writes (this usagepattern matches what happens when you start and stop many containers in a shortperiod of time, as well) can lead to poor use of Btrfs chunks. This canprematurely fill the Btrfs filesystem and lead to out-of-space conditions onyour Docker host. Use btrfs filesys show to closely monitor the amount offree space on your Btrfs device.

• Sequential writes. Btrfs uses a journaling technique when writing to disk.This can impact the performance of sequential writes, reducing performance byup to 50%.

• Fragmentation. Fragmentation is a natural byproduct of copy-on-writefilesystems like Btrfs. Many small random writes can compound this issue.Fragmentation can manifest as CPU spikes when using SSDs or head thrashingwhen using spinning disks. Either of these issues can harm performance.

  If your Linux kernel version is 3.9 or higher, you can enable the autodefragfeature when mounting a Btrfs volume. Test this feature on your own workloadsbefore deploying it into production, as some tests have shown a negativeimpact on performance.

• SSD performance: Btrfs includes native optimizations for SSD media.To enable these features, mount the Btrfs filesystem with the -o ssd mountoption. These optimizations include enhanced SSD write performance by avoidingoptimization such as seek optimizations which do not apply to solid-statemedia.

• Balance Btrfs filesystems often: Use operating system utilities such as acron job to balance the Btrfs filesystem regularly, during non-peak hours.This reclaims unallocated blocks and helps to prevent the filesystem fromfilling up unnecessarily. You cannot rebalance a totally full Btrfsfilesystem unless you add additional physical block devices to the filesystem.See theBTRFS Wiki.

• Use fast storage: Solid-state drives (SSDs) provide faster reads andwrites than spinning disks.

• Use volumes for write-heavy workloads: Volumes provide the best and mostpredictable performance for write-heavy workloads. This is because they bypassthe storage driver and do not incur any of the potential overheads introducedby thin provisioning and copy-on-write. Volumes have other benefits, such asallowing you to share data among containers and persisting even when norunning container is using them.

Related Information