Merge pull request #4499 from xulike666/function-name-modification

function-name-modification for tests/*

.dockerignore

@@ -1,4 +1,9 @@
+*.egg-info
+.coverage
 .git
+.tox
 build
-dist
+coverage-html
+docs/_site
 venv
+.tox

.gitignore

@@ -1,8 +1,11 @@
 *.egg-info
 *.pyc
-.tox
+.coverage*
+/.tox
 /build
+/coverage-html
 /dist
 /docs/_site
 /venv
-docker-compose.spec
+README.rst
+compose/GITSHA

.pre-commit-config.yaml

@@ -0,0 +1,25 @@
+- repo: git://github.com/pre-commit/pre-commit-hooks
+  sha: 'v0.4.2'
+  hooks:
+    - id: check-added-large-files
+    - id: check-docstring-first
+    - id: check-merge-conflict
+    - id: check-yaml
+    - id: check-json
+    - id: debug-statements
+    - id: end-of-file-fixer
+    - id: flake8
+    - id: name-tests-test
+      exclude: 'tests/(integration/testcases\.py|helpers\.py)'
+    - id: requirements-txt-fixer
+    - id: trailing-whitespace
+- repo: git://github.com/asottile/reorder_python_imports
+  sha: v0.1.0
+  hooks:
+    - id: reorder-python-imports
+      language_version: 'python2.7'
+      args:
+        - --add-import
+        - from __future__ import absolute_import
+        - --add-import
+        - from __future__ import unicode_literals

.travis.yml

@@ -0,0 +1,29 @@
+sudo: required
+
+language: python
+
+matrix:
+  include:
+    - os: linux
+      services:
+      - docker
+    - os: osx
+      language: generic
+
+install: ./script/travis/install
+
+script:
+  - ./script/travis/ci
+  - ./script/travis/build-binary
+
+before_deploy:
+  - "./script/travis/render-bintray-config.py < ./script/travis/bintray.json.tmpl > ./bintray.json"
+
+deploy:
+  provider: bintray
+  user: docker-compose-roleuser
+  key: '$BINTRAY_API_KEY'
+  file: ./bintray.json
+  skip_cleanup: true
+  on:
+    all_branches: true

CHANGES.md

@@ -1,437 +0,0 @@
-Change log
-==========
-
-1.4.2 (2015-09-22)
-------------------
-
-Fixes a regression in the 1.4.1 release that would cause `docker-compose up`
-without the `-d` option to exit immediately.
-
-
-1.4.1 (2015-09-10)
-------------------
-
-The following bugs have been fixed:
-
--   Some configuration changes (notably changes to `links`, `volumes_from`, and
-    `net`) were not properly triggering a container recreate as part of
-    `docker-compose up`.
--   `docker-compose up <service>` was showing logs for all services instead of
-    just the specified services.
--   Containers with custom container names were showing up in logs as
-    `service_number` instead of their custom container name.
--   When scaling a service sometimes containers would be recreated even when
-    the configuration had not changed.
-
-
-1.4.0 (2015-08-04)
-------------------
-
--   By default, `docker-compose up` now only recreates containers for services whose configuration has changed since they were created. This should result in a dramatic speed-up for many applications.
-
-    The experimental `--x-smart-recreate` flag which introduced this feature in Compose 1.3.0 has been removed, and a `--force-recreate` flag has been added for when you want to recreate everything.
-
--   Several of Compose's commands - `scale`, `stop`, `kill` and `rm` - now perform actions on multiple containers in parallel, rather than in sequence, which will run much faster on larger applications.
-
--   You can now specify a custom name for a service's container with `container_name`. Because Docker container names must be unique, this means you can't scale the service beyond one container.
-
--   You no longer have to specify a `file` option when using `extends` - it will default to the current file.
-
--   Service names can now contain dots, dashes and underscores.
-
--   Compose can now read YAML configuration from standard input, rather than from a file, by specifying `-` as the filename. This makes it easier to generate configuration dynamically:
-
-        $ echo 'redis: {"image": "redis"}' | docker-compose --file - up
-
--   There's a new `docker-compose version` command which prints extended information about Compose's bundled dependencies.
-
--   `docker-compose.yml` now supports `log_opt` as well as `log_driver`, allowing you to pass extra configuration to a service's logging driver.
-
--   `docker-compose.yml` now supports `memswap_limit`, similar to `docker run --memory-swap`.
-
--   When mounting volumes with the `volumes` option, you can now pass in any mode supported by the daemon, not just `:ro` or `:rw`. For example, SELinux users can pass `:z` or `:Z`.
-
--   You can now specify a custom volume driver with the `volume_driver` option in `docker-compose.yml`, much like `docker run --volume-driver`.
-
--   A bug has been fixed where Compose would fail to pull images from private registries serving plain (unsecured) HTTP. The `--allow-insecure-ssl` flag, which was previously used to work around this issue, has been deprecated and now has no effect.
-
--   A bug has been fixed where `docker-compose build` would fail if the build depended on a private Hub image or an image from a private registry.
-
--   A bug has been fixed where Compose would crash if there were containers which the Docker daemon had not finished removing.
-
--   Two bugs have been fixed where Compose would sometimes fail with a "Duplicate bind mount" error, or fail to attach volumes to a container, if there was a volume path specified in `docker-compose.yml` with a trailing slash.
-
-Thanks @mnowster, @dnephin, @ekristen, @funkyfuture, @jeffk and @lukemarsden!
-
-1.3.3 (2015-07-15)
-------------------
-
-Two regressions have been fixed:
-
-- When stopping containers gracefully, Compose was setting the timeout to 0, effectively forcing a SIGKILL every time.
-- Compose would sometimes crash depending on the formatting of container data returned from the Docker API.
-
-1.3.2 (2015-07-14)
-------------------
-
-The following bugs have been fixed:
-
-- When there were one-off containers created by running `docker-compose run` on an older version of Compose, `docker-compose run` would fail with a name collision. Compose now shows an error if you have leftover containers of this type lying around, and tells you how to remove them.
-- Compose was not reading Docker authentication config files created in the new location, `~/docker/config.json`, and authentication against private registries would therefore fail.
-- When a container had a pseudo-TTY attached, its output in `docker-compose up` would be truncated.
-- `docker-compose up --x-smart-recreate` would sometimes fail when an image tag was updated.
-- `docker-compose up` would sometimes create two containers with the same numeric suffix.
-- `docker-compose rm` and `docker-compose ps` would sometimes list services that aren't part of the current project (though no containers were erroneously removed).
-- Some `docker-compose` commands would not show an error if invalid service names were passed in.
-
-Thanks @dano, @josephpage, @kevinsimper, @lieryan, @phemmer, @soulrebel and @sschepens!
-
-1.3.1 (2015-06-21)
-------------------
-
-The following bugs have been fixed:
-
-- `docker-compose build` would always attempt to pull the base image before building.
-- `docker-compose help migrate-to-labels` failed with an error.
-- If no network mode was specified, Compose would set it to "bridge", rather than allowing the Docker daemon to use its configured default network mode.
-
-1.3.0 (2015-06-18)
-------------------
-
-Firstly, two important notes:
-
-- **This release contains breaking changes, and you will need to either remove or migrate your existing containers before running your app** - see the [upgrading section of the install docs](https://github.com/docker/compose/blob/1.3.0rc1/docs/install.md#upgrading) for details.
-
-- Compose now requires Docker 1.6.0 or later.
-
-We've done a lot of work in this release to remove hacks and make Compose more stable:
-
-- Compose now uses container labels, rather than names, to keep track of containers. This makes Compose both faster and easier to integrate with your own tools.
-
-- Compose no longer uses "intermediate containers" when recreating containers for a service. This makes `docker-compose up` less complex and more resilient to failure.
-
-There are some new features:
-
-- `docker-compose up` has an **experimental** new behaviour: it will only recreate containers for services whose configuration has changed in `docker-compose.yml`. This will eventually become the default, but for now you can take it for a spin:
-
-        $ docker-compose up --x-smart-recreate
-
-- When invoked in a subdirectory of a project, `docker-compose` will now climb up through parent directories until it finds a `docker-compose.yml`.
-
-Several new configuration keys have been added to `docker-compose.yml`:
-
-- `dockerfile`, like `docker build --file`, lets you specify an alternate Dockerfile to use with `build`.
-- `labels`, like `docker run --labels`, lets you add custom metadata to containers.
-- `extra_hosts`, like `docker run --add-host`, lets you add entries to a container's `/etc/hosts` file.
-- `pid: host`, like `docker run --pid=host`, lets you reuse the same PID namespace as the host machine.
-- `cpuset`, like `docker run --cpuset-cpus`, lets you specify which CPUs to allow execution in.
-- `read_only`, like `docker run --read-only`, lets you mount a container's filesystem as read-only.
-- `security_opt`, like `docker run --security-opt`, lets you specify [security options](https://docs.docker.com/reference/run/#security-configuration).
-- `log_driver`, like `docker run --log-driver`, lets you specify a [log driver](https://docs.docker.com/reference/run/#logging-drivers-log-driver).
-
-Many bugs have been fixed, including the following:
-
-- The output of `docker-compose run` was sometimes truncated, especially when running under Jenkins.
-- A service's volumes would sometimes not update after volume configuration was changed in `docker-compose.yml`.
-- Authenticating against third-party registries would sometimes fail.
-- `docker-compose run --rm` would fail to remove the container if the service had a `restart` policy in place.
-- `docker-compose scale` would refuse to scale a service beyond 1 container if it exposed a specific port number on the host.
-- Compose would refuse to create multiple volume entries with the same host path.
-
-Thanks @ahromis, @albers, @aleksandr-vin, @antoineco, @ccverak, @chernjie, @dnephin, @edmorley, @fordhurley, @josephpage, @KyleJamesWalker, @lsowen, @mchasal, @noironetworks, @sdake, @sdurrheimer, @sherter, @stephenlawrence, @thaJeztah, @thieman, @turtlemonvh, @twhiteman, @vdemeester, @xuxinkun and @zwily!
-
-1.2.0 (2015-04-16)
-------------------
-
-- `docker-compose.yml` now supports an `extends` option, which enables a service to inherit configuration from another service in another configuration file. This is really good for sharing common configuration between apps, or for configuring the same app for different environments. Here's the [documentation](https://github.com/docker/compose/blob/master/docs/yml.md#extends).
-
-- When using Compose with a Swarm cluster, containers that depend on one another will be co-scheduled on the same node. This means that most Compose apps will now work out of the box, as long as they don't use `build`.
-
-- Repeated invocations of `docker-compose up` when using Compose with a Swarm cluster now work reliably.
-
-- Directories passed to `build`, filenames passed to `env_file` and volume host paths passed to `volumes` are now treated as relative to the *directory of the configuration file*, not the directory that `docker-compose` is being run in. In the majority of cases, those are the same, but if you use the `-f|--file` argument to specify a configuration file in another directory, **this is a breaking change**.
-
-- A service can now share another service's network namespace with `net: container:<service>`.
-
-- `volumes_from` and `net: container:<service>` entries are taken into account when resolving dependencies, so `docker-compose up <service>` will correctly start all dependencies of `<service>`.
-
-- `docker-compose run` now accepts a `--user` argument to specify a user to run the command as, just like `docker run`.
-
-- The `up`, `stop` and `restart` commands now accept a `--timeout` (or `-t`) argument to specify how long to wait when attempting to gracefully stop containers, just like `docker stop`.
-
-- `docker-compose rm` now accepts `-f` as a shorthand for `--force`, just like `docker rm`.
-
-Thanks, @abesto, @albers, @alunduil, @dnephin, @funkyfuture, @gilclark, @IanVS, @KingsleyKelly, @knutwalker, @thaJeztah and @vmalloc!
-
-1.1.0 (2015-02-25)
-------------------
-
-Fig has been renamed to Docker Compose, or just Compose for short. This has several implications for you:
-
-- The command you type is now `docker-compose`, not `fig`.
-- You should rename your fig.yml to docker-compose.yml.
-- If you’re installing via PyPi, the package is now `docker-compose`, so install it with `pip install docker-compose`.
-
-Besides that, there’s a lot of new stuff in this release:
-
-- We’ve made a few small changes to ensure that Compose will work with Swarm, Docker’s new clustering tool (https://github.com/docker/swarm). Eventually you'll be able to point Compose at a Swarm cluster instead of a standalone Docker host and it’ll run your containers on the cluster with no extra work from you. As Swarm is still developing, integration is rough and lots of Compose features don't work yet.
-
-- `docker-compose run` now has a `--service-ports` flag for exposing ports on the given service. This is useful for e.g. running your webapp with an interactive debugger.
-
-- You can now link to containers outside your app with the `external_links` option in docker-compose.yml.
-
-- You can now prevent `docker-compose up` from automatically building images with the `--no-build` option. This will make fewer API calls and run faster.
-
-- If you don’t specify a tag when using the `image` key, Compose will default to the `latest` tag, rather than pulling all tags.
-
-- `docker-compose kill` now supports the `-s` flag, allowing you to specify the exact signal you want to send to a service’s containers.
-
-- docker-compose.yml now has an `env_file` key, analogous to `docker run --env-file`, letting you specify multiple environment variables in a separate file. This is great if you have a lot of them, or if you want to keep sensitive information out of version control.
-
-- docker-compose.yml now supports the `dns_search`, `cap_add`, `cap_drop`, `cpu_shares` and `restart` options, analogous to `docker run`’s `--dns-search`, `--cap-add`, `--cap-drop`, `--cpu-shares` and `--restart` options.
-
-- Compose now ships with Bash tab completion - see the installation and usage docs at https://github.com/docker/compose/blob/1.1.0/docs/completion.md
-
-- A number of bugs have been fixed - see the milestone for details: https://github.com/docker/compose/issues?q=milestone%3A1.1.0+
-
-Thanks @dnephin, @squebe, @jbalonso, @raulcd, @benlangfield, @albers, @ggtools, @bersace, @dtenenba, @petercv, @drewkett, @TFenby, @paulRbr, @Aigeruth and @salehe!
-
-1.0.1 (2014-11-04)
-------------------
-
- - Added an `--allow-insecure-ssl` option to allow `fig up`, `fig run` and `fig pull` to pull from insecure registries.
- - Fixed `fig run` not showing output in Jenkins.
- - Fixed a bug where Fig couldn't build Dockerfiles with ADD statements pointing at URLs.
-
-1.0.0 (2014-10-16)
-------------------
-
-The highlights:
-
- - [Fig has joined Docker.](https://www.orchardup.com/blog/orchard-is-joining-docker) Fig will continue to be maintained, but we'll also be incorporating the best bits of Fig into Docker itself.
-
-   This means the GitHub repository has moved to [https://github.com/docker/fig](https://github.com/docker/fig) and our IRC channel is now #docker-fig on Freenode.
-
- - Fig can be used with the [official Docker OS X installer](https://docs.docker.com/installation/mac/). Boot2Docker will mount the home directory from your host machine so volumes work as expected.
-
- - Fig supports Docker 1.3.
-
- - It is now possible to connect to the Docker daemon using TLS by using the `DOCKER_CERT_PATH` and `DOCKER_TLS_VERIFY` environment variables.
-
- - There is a new `fig port` command which outputs the host port binding of a service, in a similar way to `docker port`.
-
- - There is a new `fig pull` command which pulls the latest images for a service.
-
- - There is a new `fig restart` command which restarts a service's containers.
-
- - Fig creates multiple containers in service by appending a number to the service name (e.g. `db_1`, `db_2`, etc). As a convenience, Fig will now give the first container an alias of the service name (e.g. `db`).
-   
-   This link alias is also a valid hostname and added to `/etc/hosts` so you can connect to linked services using their hostname. For example, instead of resolving the environment variables `DB_PORT_5432_TCP_ADDR` and `DB_PORT_5432_TCP_PORT`, you could just use the hostname `db` and port `5432` directly.
-
- - Volume definitions now support `ro` mode, expanding `~` and expanding environment variables.
-
- - `.dockerignore` is supported when building.
-
- - The project name can be set with the `FIG_PROJECT_NAME` environment variable.
-
- - The `--env` and `--entrypoint` options have been added to `fig run`.
-
- - The Fig binary for Linux is now linked against an older version of glibc so it works on CentOS 6 and Debian Wheezy.
-
-Other things:
-
- - `fig ps` now works on Jenkins and makes fewer API calls to the Docker daemon.
- - `--verbose` displays more useful debugging output.
- - When starting a service where `volumes_from` points to a service without any containers running, that service will now be started.
- - Lots of docs improvements. Notably, environment variables are documented and official repositories are used throughout.
-
-Thanks @dnephin, @d11wtq, @marksteve, @rubbish, @jbalonso, @timfreund, @alunduil, @mieciu, @shuron, @moss, @suzaku and @chmouel! Whew.
-
-0.5.2 (2014-07-28)
-------------------
-
- - Added a `--no-cache` option to `fig build`, which bypasses the cache just like `docker build --no-cache`.
- - Fixed the `dns:` fig.yml option, which was causing fig to error out.
- - Fixed a bug where fig couldn't start under Python 2.6.
- - Fixed a log-streaming bug that occasionally caused fig to exit.
-
-Thanks @dnephin and @marksteve!
-
-
-0.5.1 (2014-07-11)
-------------------
-
- - If a service has a command defined, `fig run [service]` with no further arguments will run it.
- - The project name now defaults to the directory containing fig.yml, not the current working directory (if they're different)
- - `volumes_from` now works properly with containers as well as services
- - Fixed a race condition when recreating containers in `fig up`
-
-Thanks @ryanbrainard and @d11wtq!
-
-
-0.5.0 (2014-07-11)
-------------------
-
- - Fig now starts links when you run `fig run` or `fig up`.
-   
-   For example, if you have a `web` service which depends on a `db` service, `fig run web ...` will start the `db` service.
-
- - Environment variables can now be resolved from the environment that Fig is running in. Just specify it as a blank variable in your `fig.yml` and, if set, it'll be resolved:
-   ```
-   environment:
-     RACK_ENV: development
-     SESSION_SECRET:
-   ```
-
- - `volumes_from` is now supported in `fig.yml`. All of the volumes from the specified services and containers will be mounted:
-
-   ```
-   volumes_from:
-    - service_name
-    - container_name
-   ```
-
- - A host address can now be specified in `ports`:
-
-   ```
-   ports:
-    - "0.0.0.0:8000:8000"
-    - "127.0.0.1:8001:8001"
-   ```
-
- - The `net` and `workdir` options are now supported in `fig.yml`.
- - The `hostname` option now works in the same way as the Docker CLI, splitting out into a `domainname` option.
- - TTY behaviour is far more robust, and resizes are supported correctly.
- - Load YAML files safely.
-
-Thanks to @d11wtq, @ryanbrainard, @rail44, @j0hnsmith, @binarin, @Elemecca, @mozz100 and @marksteve for their help with this release!
-
-
-0.4.2 (2014-06-18)
-------------------
-
- - Fix various encoding errors when using `fig run`, `fig up` and `fig build`.
-
-0.4.1 (2014-05-08)
-------------------
-
- - Add support for Docker 0.11.0. (Thanks @marksteve!)
- - Make project name configurable. (Thanks @jefmathiot!)
- - Return correct exit code from `fig run`.
-
-0.4.0 (2014-04-29)
-------------------
-
- - Support Docker 0.9 and 0.10
- - Display progress bars correctly when pulling images (no more ski slopes)
- - `fig up` now stops all services when any container exits
- - Added support for the `privileged` config option in fig.yml (thanks @kvz!)
- - Shortened and aligned log prefixes in `fig up` output
- - Only containers started with `fig run` link back to their own service
- - Handle UTF-8 correctly when streaming `fig build/run/up` output (thanks @mauvm and @shanejonas!)
- - Error message improvements
-
-0.3.2 (2014-03-05)
-------------------
-
- - Added an `--rm` option to `fig run`. (Thanks @marksteve!)
- - Added an `expose` option to `fig.yml`.
-
-0.3.1 (2014-03-04)
-------------------
-
- - Added contribution instructions. (Thanks @kvz!)
- - Fixed `fig rm` throwing an error.
- - Fixed a bug in `fig ps` on Docker 0.8.1 when there is a container with no command.
-
-0.3.0 (2014-03-03)
-------------------
-
- - We now ship binaries for OS X and Linux. No more having to install with Pip!
- - Add `-f` flag to specify alternate `fig.yml` files
- - Add support for custom link names
- - Fix a bug where recreating would sometimes hang
- - Update docker-py to support Docker 0.8.0.
- - Various documentation improvements
- - Various error message improvements
-
-Thanks @marksteve, @Gazler and @teozkr!
-
-0.2.2 (2014-02-17)
-------------------
-
- - Resolve dependencies using Cormen/Tarjan topological sort
- - Fix `fig up` not printing log output
- - Stop containers in reverse order to starting
- - Fix scale command not binding ports
-
-Thanks to @barnybug and @dustinlacewell for their work on this release.
-
-0.2.1 (2014-02-04)
-------------------
-
- - General improvements to error reporting (#77, #79)
-
-0.2.0 (2014-01-31)
-------------------
-
- - Link services to themselves so run commands can access the running service. (#67)
- - Much better documentation.
- - Make service dependency resolution more reliable. (#48)
- - Load Fig configurations with a `.yaml` extension. (#58)
-
-Big thanks to @cameronmaske, @mrchrisadams and @damianmoore for their help with this release.
-
-0.1.4 (2014-01-27)
-------------------
-
- - Add a link alias without the project name. This makes the environment variables a little shorter: `REDIS_1_PORT_6379_TCP_ADDR`. (#54)
-
-0.1.3 (2014-01-23)
-------------------
-
- - Fix ports sometimes being configured incorrectly. (#46)
- - Fix log output sometimes not displaying. (#47)
-
-0.1.2 (2014-01-22)
-------------------
-
- - Add `-T` option to `fig run` to disable pseudo-TTY. (#34)
- - Fix `fig up` requiring the ubuntu image to be pulled to recreate containers. (#33) Thanks @cameronmaske!
- - Improve reliability, fix arrow keys and fix a race condition in `fig run`. (#34, #39, #40)
-
-0.1.1 (2014-01-17)
-------------------
-
- - Fix bug where ports were not exposed correctly (#29). Thanks @dustinlacewell!
-
-0.1.0 (2014-01-16)
-------------------
-
- - Containers are recreated on each `fig up`, ensuring config is up-to-date with `fig.yml` (#2)
- - Add `fig scale` command (#9)
- - Use `DOCKER_HOST` environment variable to find Docker daemon, for consistency with the official Docker client (was previously `DOCKER_URL`) (#19)
- - Truncate long commands in `fig ps` (#18)
- - Fill out CLI help banners for commands (#15, #16)
- - Show a friendlier error when `fig.yml` is missing (#4)
- - Fix bug with `fig build` logging (#3)
- - Fix bug where builds would time out if a step took a long time without generating output (#6)
- - Fix bug where streaming container output over the Unix socket raised an error (#7)
-
-Big thanks to @tomstuart, @EnTeQuAk, @schickling, @aronasorman and @GeoffreyPlitt.
-
-0.0.2 (2014-01-02)
-------------------
-
- - Improve documentation
- - Try to connect to Docker on `tcp://localdocker:4243` and a UNIX socket in addition to `localhost`.
- - Improve `fig up` behaviour
- - Add confirmation prompt to `fig rm`
- - Add `fig build` command
-
-0.0.1 (2013-12-20)
-------------------
-
-Initial release.
-
-

CHANGES.md

@@ -0,0 +1 @@
+CHANGELOG.md

CONTRIBUTING.md

@@ -30,34 +30,45 @@ that should get you started.
    `docker-compose` from anywhere on your machine, it will run your development
    version of Compose.
 
+## Install pre-commit hooks
+
+This step is optional, but recommended. Pre-commit hooks will run style checks
+and in some cases fix style issues for you, when you commit code.
+
+Install the git pre-commit hooks using [tox](https://tox.readthedocs.io) by
+running `tox -e pre-commit` or by following the
+[pre-commit install guide](http://pre-commit.com/#install).
+
+To run the style checks at any time run `tox -e pre-commit`.
+
 ## Submitting a pull request
 
-See Docker's [basic contribution workflow](https://docs.docker.com/project/make-a-contribution/#the-basic-contribution-workflow) for a guide on how to submit a pull request for code or documentation.
+See Docker's [basic contribution workflow](https://docs.docker.com/opensource/workflow/make-a-contribution/#the-basic-contribution-workflow) for a guide on how to submit a pull request for code or documentation.
 
 ## Running the test suite
 
 Use the test script to run linting checks and then the full test suite against
 different Python interpreters:
 
-    $ script/test
+    $ script/test/default
 
 Tests are run against a Docker daemon inside a container, so that we can test
 against multiple Docker versions. By default they'll run against only the latest
 Docker version - set the `DOCKER_VERSIONS` environment variable to "all" to run
 against all supported versions:
 
-    $ DOCKER_VERSIONS=all script/test
+    $ DOCKER_VERSIONS=all script/test/default
 
-Arguments to `script/test` are passed through to the `nosetests` executable, so
+Arguments to `script/test/default` are passed through to the `tox` executable, so
 you can specify a test directory, file, module, class or method:
 
-    $ script/test tests/unit
-    $ script/test tests/unit/cli_test.py
-    $ script/test tests.integration.service_test
-    $ script/test tests.integration.service_test:ServiceTest.test_containers
+    $ script/test/default tests/unit
+    $ script/test/default tests/unit/cli_test.py
+    $ script/test/default tests/unit/config_test.py::ConfigTest
+    $ script/test/default tests/unit/config_test.py::ConfigTest::test_load
 
 ## Finding things to work on
 
-We use a [Waffle.io board](https://waffle.io/docker/compose) to keep track of specific things we are working on and planning to work on. If you're looking for things to work on, stuff in the backlog is a great place to start.
+We use a [ZenHub board](https://www.zenhub.io/) to keep track of specific things we are working on and planning to work on. If you're looking for things to work on, stuff in the backlog is a great place to start.
 
 For more information about our project planning, take a look at our [GitHub wiki](https://github.com/docker/compose/wiki).

Dockerfile

@@ -3,74 +3,69 @@ FROM debian:wheezy
 RUN set -ex; \
     apt-get update -qq; \
     apt-get install -y \
+        locales \
         gcc \
         make \
         zlib1g \
         zlib1g-dev \
         libssl-dev \
         git \
-        apt-transport-https \
         ca-certificates \
         curl \
-        lxc \
-        iptables \
+        libsqlite3-dev \
+        libbz2-dev \
     ; \
     rm -rf /var/lib/apt/lists/*
 
-# Build Python 2.7.9 from source
+RUN curl https://get.docker.com/builds/Linux/x86_64/docker-1.8.3 \
+        -o /usr/local/bin/docker && \
+    chmod +x /usr/local/bin/docker
+
+# Build Python 2.7.13 from source
 RUN set -ex; \
-    curl -LO https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tgz; \
-    tar -xzf Python-2.7.9.tgz; \
-    cd Python-2.7.9; \
+    curl -L https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tgz | tar -xz; \
+    cd Python-2.7.13; \
     ./configure --enable-shared; \
     make; \
     make install; \
     cd ..; \
-    rm -rf /Python-2.7.9; \
-    rm Python-2.7.9.tgz
+    rm -rf /Python-2.7.13
+
+# Build python 3.4 from source
+RUN set -ex; \
+    curl -L https://www.python.org/ftp/python/3.4.6/Python-3.4.6.tgz | tar -xz; \
+    cd Python-3.4.6; \
+    ./configure --enable-shared; \
+    make; \
+    make install; \
+    cd ..; \
+    rm -rf /Python-3.4.6
 
 # Make libpython findable
 ENV LD_LIBRARY_PATH /usr/local/lib
 
-# Install setuptools
-RUN set -ex; \
-    curl -LO https://bootstrap.pypa.io/ez_setup.py; \
-    python ez_setup.py; \
-    rm ez_setup.py
-
 # Install pip
 RUN set -ex; \
-    curl -LO https://pypi.python.org/packages/source/p/pip/pip-7.0.1.tar.gz; \
-    tar -xzf pip-7.0.1.tar.gz; \
-    cd pip-7.0.1; \
-    python setup.py install; \
-    cd ..; \
-    rm -rf pip-7.0.1; \
-    rm pip-7.0.1.tar.gz
+    curl -L https://bootstrap.pypa.io/get-pip.py | python
 
-ENV ALL_DOCKER_VERSIONS 1.7.1 1.8.0-rc3
-
-RUN set -ex; \
-    curl https://get.docker.com/builds/Linux/x86_64/docker-1.7.1 -o /usr/local/bin/docker-1.7.1; \
-    chmod +x /usr/local/bin/docker-1.7.1; \
-    curl https://test.docker.com/builds/Linux/x86_64/docker-1.8.0-rc3 -o /usr/local/bin/docker-1.8.0-rc3; \
-    chmod +x /usr/local/bin/docker-1.8.0-rc3
-
-# Set the default Docker to be run
-RUN ln -s /usr/local/bin/docker-1.7.1 /usr/local/bin/docker
+# Python3 requires a valid locale
+RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && locale-gen
+ENV LANG en_US.UTF-8
 
 RUN useradd -d /home/user -m -s /bin/bash user
 WORKDIR /code/
 
-ADD requirements.txt /code/
-RUN pip install -r requirements.txt
+RUN pip install tox==2.1.1
 
+ADD requirements.txt /code/
 ADD requirements-dev.txt /code/
-RUN pip install -r requirements-dev.txt
+ADD .pre-commit-config.yaml /code/
+ADD setup.py /code/
+ADD tox.ini /code/
+ADD compose /code/compose/
+RUN tox --notest
 
 ADD . /code/
-RUN python setup.py install
-
 RUN chown -R user /code/
 
-ENTRYPOINT ["/usr/local/bin/docker-compose"]
+ENTRYPOINT ["/code/.tox/py27/bin/docker-compose"]

Dockerfile.run

@@ -0,0 +1,14 @@
+
+FROM    alpine:3.4
+ARG     version
+RUN     apk -U add \
+            python \
+            py-pip
+
+COPY    requirements.txt /code/requirements.txt
+RUN     pip install -r /code/requirements.txt
+
+COPY     dist/docker_compose-${version}-py2.py3-none-any.whl /code/
+RUN     pip install --no-deps /code/docker_compose-${version}-py2.py3-none-any.whl
+
+ENTRYPOINT ["/usr/bin/docker-compose"]

Jenkinsfile

@@ -0,0 +1,64 @@
+#!groovy
+
+def image
+
+def buildImage = { ->
+  wrappedNode(label: "ubuntu && !zfs", cleanWorkspace: true) {
+    stage("build image") {
+      checkout(scm)
+      def imageName = "dockerbuildbot/compose:${gitCommit()}"
+      image = docker.image(imageName)
+      try {
+        image.pull()
+      } catch (Exception exc) {
+        image = docker.build(imageName, ".")
+        image.push()
+      }
+    }
+  }
+}
+
+def runTests = { Map settings ->
+  def dockerVersions = settings.get("dockerVersions", null)
+  def pythonVersions = settings.get("pythonVersions", null)
+
+  if (!pythonVersions) {
+    throw new Exception("Need Python versions to test. e.g.: `runTests(pythonVersions: 'py27,py34')`")
+  }
+  if (!dockerVersions) {
+    throw new Exception("Need Docker versions to test. e.g.: `runTests(dockerVersions: 'all')`")
+  }
+
+  { ->
+    wrappedNode(label: "ubuntu && !zfs", cleanWorkspace: true) {
+      stage("test python=${pythonVersions} / docker=${dockerVersions}") {
+        checkout(scm)
+        def storageDriver = sh(script: 'docker info | awk -F \': \' \'$1 == "Storage Driver" { print $2; exit }\'', returnStdout: true).trim()
+        echo "Using local system's storage driver: ${storageDriver}"
+        sh """docker run \\
+          -t \\
+          --rm \\
+          --privileged \\
+          --volume="\$(pwd)/.git:/code/.git" \\
+          --volume="/var/run/docker.sock:/var/run/docker.sock" \\
+          -e "TAG=${image.id}" \\
+          -e "STORAGE_DRIVER=${storageDriver}" \\
+          -e "DOCKER_VERSIONS=${dockerVersions}" \\
+          -e "BUILD_NUMBER=\$BUILD_TAG" \\
+          -e "PY_TEST_VERSIONS=${pythonVersions}" \\
+          --entrypoint="script/ci" \\
+          ${image.id} \\
+          --verbose
+        """
+      }
+    }
+  }
+}
+
+buildImage()
+// TODO: break this out into meaningful "DOCKER_VERSIONS" values instead of all
+parallel(
+  failFast: true,
+  all_py27: runTests(pythonVersions: "py27", dockerVersions: "all"),
+  all_py34: runTests(pythonVersions: "py34", dockerVersions: "all"),
+)

MAINTAINERS

@@ -1,4 +1,46 @@
-Aanand Prasad <aanand.prasad@gmail.com> (@aanand)
-Ben Firshman <ben@firshman.co.uk> (@bfirsh)
-Daniel Nephin <dnephin@gmail.com> (@dnephin)
-Mazz Mosley <mazz@houseofmnowster.com> (@mnowster)
+# Compose maintainers file
+#
+# This file describes who runs the docker/compose project and how.
+# This is a living document - if you see something out of date or missing, speak up!
+#
+# It is structured to be consumable by both humans and programs.
+# To extract its contents programmatically, use any TOML-compliant parser.
+#
+# This file is compiled into the MAINTAINERS file in docker/opensource.
+#
+[Org]
+	[Org."Core maintainers"]
+		people = [
+			"aanand",
+			"bfirsh",
+			"dnephin",
+			"mnowster",
+		]
+
+[people]
+
+# A reference list of all people associated with the project.
+# All other sections should refer to people by their canonical key
+# in the people section.
+
+	# ADD YOURSELF HERE IN ALPHABETICAL ORDER
+
+	[people.aanand]
+	Name = "Aanand Prasad"
+	Email = "aanand.prasad@gmail.com"
+	GitHub = "aanand"
+
+	[people.bfirsh]
+	Name = "Ben Firshman"
+	Email = "ben@firshman.co.uk"
+	GitHub = "bfirsh"
+
+	[people.dnephin]
+	Name = "Daniel Nephin"
+	Email = "dnephin@gmail.com"
+	GitHub = "dnephin"
+
+	[people.mnowster]
+	Name = "Mazz Mosley"
+	Email = "mazz@houseofmnowster.com"
+	GitHub = "mnowster"

MANIFEST.in

@@ -4,6 +4,10 @@ include requirements.txt
 include requirements-dev.txt
 include tox.ini
 include *.md
+exclude README.md
+include README.rst
+include compose/config/*.json
+include compose/GITSHA
 recursive-include contrib/completion *
 recursive-include tests *
 global-exclude *.pyc

README.md

@@ -1,14 +1,16 @@
 Docker Compose
 ==============
-*(Previously known as Fig)*
+![Docker Compose](logo.png?raw=true "Docker Compose Logo")
 
-Compose is a tool for defining and running multi-container applications with
-Docker. With Compose, you define a multi-container application in a single
-file, then spin your application up in a single command which does everything
-that needs to be done to get it running.
+Compose is a tool for defining and running multi-container Docker applications.
+With Compose, you use a Compose file to configure your application's services.
+Then, using a single command, you create and start all the services
+from your configuration. To learn more about all the features of Compose
+see [the list of features](https://github.com/docker/docker.github.io/blob/master/compose/overview.md#features).
 
-Compose is great for development environments, staging servers, and CI. We don't
-recommend that you use it in production yet.
+Compose is great for development, testing, and staging environments, as well as
+CI workflows. You can learn more about each case in
+[Common Use Cases](https://github.com/docker/docker.github.io/blob/master/compose/overview.md#common-use-cases).
 
 Using Compose is basically a three-step process.
 
@@ -20,16 +22,20 @@ they can be run together in an isolated environment:
 
 A `docker-compose.yml` looks like this:
 
-    web:
-      build: .
-      ports:
-       - "5000:5000"
-      volumes:
-       - .:/code
-      links:
-       - redis
-    redis:
-      image: redis
+    version: '2'
+
+    services:
+      web:
+        build: .
+        ports:
+         - "5000:5000"
+        volumes:
+         - .:/code
+      redis:
+        image: redis
+
+For more information about the Compose file, see the
+[Compose file reference](https://github.com/docker/docker.github.io/blob/master/compose/compose-file/compose-versioning.md)
 
 Compose has commands for managing the whole lifecycle of your application:
 
@@ -41,8 +47,10 @@ Compose has commands for managing the whole lifecycle of your application:
 Installation and documentation
 ------------------------------
 
-- Full documentation is available on [Docker's website](http://docs.docker.com/compose/).
+- Full documentation is available on [Docker's website](https://docs.docker.com/compose/).
 - If you have any questions, you can talk in real-time with other developers in the #docker-compose IRC channel on Freenode. [Click here to join using IRCCloud.](https://www.irccloud.com/invite?hostname=irc.freenode.net&channel=%23docker-compose)
+- Code repository for Compose is on [Github](https://github.com/docker/compose)
+- If you find any problems please fill out an [issue](https://github.com/docker/compose/issues/new)
 
 Contributing
 ------------
@@ -54,4 +62,4 @@ Want to help build Compose? Check out our [contributing documentation](https://g
 Releasing
 ---------
 
-Releases are built by maintainers, following an outline of the [release process](https://github.com/docker/compose/blob/master/RELEASE_PROCESS.md).
+Releases are built by maintainers, following an outline of the [release process](https://github.com/docker/compose/blob/master/project/RELEASE-PROCESS.md).

RELEASE_PROCESS.md

@@ -1,36 +0,0 @@
-# Building a Compose release
-
-## Building binaries
-
-`script/build-linux` builds the Linux binary inside a Docker container:
-
-    $ script/build-linux
-
-`script/build-osx` builds the Mac OS X binary inside a virtualenv:
-
-    $ script/build-osx
-
-For official releases, you should build inside a Mountain Lion VM for proper
-compatibility. Run the this script first to prepare the environment before
-building - it will use Homebrew to make sure Python is installed and
-up-to-date.
-
-    $ script/prepare-osx
-
-## Release process
-
-1. Open pull request that:
- - Updates the version in `compose/__init__.py`
- - Updates the binary URL in `docs/install.md`
- - Adds release notes to `CHANGES.md`
-2. Create unpublished GitHub release with release notes
-3. Build Linux version on any Docker host with `script/build-linux` and attach
-   to release
-4. Build OS X version on Mountain Lion with `script/build-osx` and attach to
-   release as `docker-compose-Darwin-x86_64` and `docker-compose-Linux-x86_64`.
-5. Publish GitHub release, creating tag
-6. Update website with `script/deploy-docs`
-7. Upload PyPi package
-
-        $ git checkout $VERSION
-        $ python setup.py sdist upload

ROADMAP.md

@@ -1,13 +1,21 @@
 # Roadmap
 
+## An even better tool for development environments
+
+Compose is a great tool for development environments, but it could be even better. For example:
+
+- It should be possible to define hostnames for containers which work from the host machine, e.g. “mywebcontainer.local”. This is needed by apps comprising multiple web services which generate links to one another (e.g. a frontend website and a separate admin webapp)
+
 ## More than just development environments
 
-Over time we will extend Compose's remit to cover test, staging and production environments. This is not a simple task, and will take many incremental improvements such as:
+Compose currently works really well in development, but we want to make the Compose file format better for test, staging, and production environments. To support these use cases, there will need to be improvements to the file format, improvements to the command-line tool, integrations with other tools, and perhaps new tools altogether.
+
+Some specific things we are considering:
 
 - Compose currently will attempt to get your application into the correct state when running `up`, but it has a number of shortcomings:
   - It should roll back to a known good state if it fails.
   - It should allow a user to check the actions it is about to perform before running them.
-- It should be possible to partially modify the config file for different environments (dev/test/staging/prod), passing in e.g. custom ports or volume mount paths. ([#1377](https://github.com/docker/compose/issues/1377))
+- It should be possible to partially modify the config file for different environments (dev/test/staging/prod), passing in e.g. custom ports, volume mount paths, or volume drivers. ([#1377](https://github.com/docker/compose/issues/1377))
 - Compose should recommend a technique for zero-downtime deploys.
 - It should be possible to continuously attempt to keep an application in the correct state, instead of just performing `up` a single time.
 
@@ -22,10 +30,3 @@ The current state of integration is documented in [SWARM.md](SWARM.md).
 Compose works well for applications that are in a single repository and depend on services that are hosted on Docker Hub. If your application depends on another application within your organisation, Compose doesn't work as well.
 
 There are several ideas about how this could work, such as [including external files](https://github.com/docker/fig/issues/318).
-
-## An even better tool for development environments
-
-Compose is a great tool for development environments, but it could be even better. For example:
-
-- [Compose could watch your code and automatically kick off builds when something changes.](https://github.com/docker/fig/issues/184)
-- It should be possible to define hostnames for containers which work from the host machine, e.g. “mywebcontainer.local”. This is needed by apps comprising multiple web services which generate links to one another (e.g. a frontend website and a separate admin webapp)

SWARM.md

@@ -1,39 +1 @@
-Docker Compose/Swarm integration
-================================
-
-Eventually, Compose and Swarm aim to have full integration, meaning you can point a Compose app at a Swarm cluster and have it all just work as if you were using a single Docker host.
-
-However, integration is currently incomplete: Compose can create containers on a Swarm cluster, but the majority of Compose apps won’t work out of the box unless all containers are scheduled on one host, because links between containers do not work across hosts.
-
-Docker networking is [getting overhauled](https://github.com/docker/libnetwork) in such a way that it’ll fit the multi-host model much better. For now, linked containers are automatically scheduled on the same host.
-
-Building
---------
-
-Swarm can build an image from a Dockerfile just like a single-host Docker instance can, but the resulting image will only live on a single node and won't be distributed to other nodes.
-
-If you want to use Compose to scale the service in question to multiple nodes, you'll have to build it yourself, push it to a registry (e.g. the Docker Hub) and reference it from `docker-compose.yml`:
-
-    $ docker build -t myusername/web .
-    $ docker push myusername/web
-
-    $ cat docker-compose.yml
-    web:
-      image: myusername/web
-
-    $ docker-compose up -d
-    $ docker-compose scale web=3
-
-Scheduling
-----------
-
-Swarm offers a rich set of scheduling and affinity hints, enabling you to control where containers are located. They are specified via container environment variables, so you can use Compose's `environment` option to set them.
-
-    environment:
-      # Schedule containers on a node that has the 'storage' label set to 'ssd'
-      - "constraint:storage==ssd"
-
-      # Schedule containers where the 'redis' image is already pulled
-      - "affinity:image==redis"
-
-For the full set of available filters and expressions, see the [Swarm documentation](https://docs.docker.com/swarm/scheduler/filter/).
+This file has moved to: https://docs.docker.com/compose/swarm/

appveyor.yml

@@ -0,0 +1,24 @@
+
+version: '{branch}-{build}'
+
+install:
+  - "SET PATH=C:\\Python27-x64;C:\\Python27-x64\\Scripts;%PATH%"
+  - "python --version"
+  - "pip install tox==2.1.1 virtualenv==13.1.2"
+
+# Build the binary after tests
+build: false
+
+test_script:
+  - "tox -e py27,py34 -- tests/unit"
+  - ps: ".\\script\\build\\windows.ps1"
+
+artifacts:
+  - path: .\dist\docker-compose-Windows-x86_64.exe
+    name: "Compose Windows binary"
+
+deploy:
+  - provider: Environment
+    name: master-builds
+    on:
+      branch: master

compose/__init__.py

@@ -1,3 +1,4 @@
+from __future__ import absolute_import
 from __future__ import unicode_literals
 
-__version__ = '1.4.2'
+__version__ = '1.12.0dev'

compose/__main__.py

@@ -0,0 +1,6 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+from compose.cli.main import main
+
+main()

compose/bundle.py

@@ -0,0 +1,258 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import json
+import logging
+
+import six
+from docker.utils import split_command
+from docker.utils.ports import split_port
+
+from .cli.errors import UserError
+from .config.serialize import denormalize_config
+from .network import get_network_defs_for_service
+from .service import format_environment
+from .service import NoSuchImageError
+from .service import parse_repository_tag
+
+
+log = logging.getLogger(__name__)
+
+
+SERVICE_KEYS = {
+    'working_dir': 'WorkingDir',
+    'user': 'User',
+    'labels': 'Labels',
+}
+
+IGNORED_KEYS = {'build'}
+
+SUPPORTED_KEYS = {
+    'image',
+    'ports',
+    'expose',
+    'networks',
+    'command',
+    'environment',
+    'entrypoint',
+} | set(SERVICE_KEYS)
+
+VERSION = '0.1'
+
+
+class NeedsPush(Exception):
+    def __init__(self, image_name):
+        self.image_name = image_name
+
+
+class NeedsPull(Exception):
+    def __init__(self, image_name, service_name):
+        self.image_name = image_name
+        self.service_name = service_name
+
+
+class MissingDigests(Exception):
+    def __init__(self, needs_push, needs_pull):
+        self.needs_push = needs_push
+        self.needs_pull = needs_pull
+
+
+def serialize_bundle(config, image_digests):
+    return json.dumps(to_bundle(config, image_digests), indent=2, sort_keys=True)
+
+
+def get_image_digests(project, allow_push=False):
+    digests = {}
+    needs_push = set()
+    needs_pull = set()
+
+    for service in project.services:
+        try:
+            digests[service.name] = get_image_digest(
+                service,
+                allow_push=allow_push,
+            )
+        except NeedsPush as e:
+            needs_push.add(e.image_name)
+        except NeedsPull as e:
+            needs_pull.add(e.service_name)
+
+    if needs_push or needs_pull:
+        raise MissingDigests(needs_push, needs_pull)
+
+    return digests
+
+
+def get_image_digest(service, allow_push=False):
+    if 'image' not in service.options:
+        raise UserError(
+            "Service '{s.name}' doesn't define an image tag. An image name is "
+            "required to generate a proper image digest for the bundle. Specify "
+            "an image repo and tag with the 'image' option.".format(s=service))
+
+    _, _, separator = parse_repository_tag(service.options['image'])
+    # Compose file already uses a digest, no lookup required
+    if separator == '@':
+        return service.options['image']
+
+    try:
+        image = service.image()
+    except NoSuchImageError:
+        action = 'build' if 'build' in service.options else 'pull'
+        raise UserError(
+            "Image not found for service '{service}'. "
+            "You might need to run `docker-compose {action} {service}`."
+            .format(service=service.name, action=action))
+
+    if image['RepoDigests']:
+        # TODO: pick a digest based on the image tag if there are multiple
+        # digests
+        return image['RepoDigests'][0]
+
+    if 'build' not in service.options:
+        raise NeedsPull(service.image_name, service.name)
+
+    if not allow_push:
+        raise NeedsPush(service.image_name)
+
+    return push_image(service)
+
+
+def push_image(service):
+    try:
+        digest = service.push()
+    except:
+        log.error(
+            "Failed to push image for service '{s.name}'. Please use an "
+            "image tag that can be pushed to a Docker "
+            "registry.".format(s=service))
+        raise
+
+    if not digest:
+        raise ValueError("Failed to get digest for %s" % service.name)
+
+    repo, _, _ = parse_repository_tag(service.options['image'])
+    identifier = '{repo}@{digest}'.format(repo=repo, digest=digest)
+
+    # only do this if RepoDigests isn't already populated
+    image = service.image()
+    if not image['RepoDigests']:
+        # Pull by digest so that image['RepoDigests'] is populated for next time
+        # and we don't have to pull/push again
+        service.client.pull(identifier)
+        log.info("Stored digest for {}".format(service.image_name))
+
+    return identifier
+
+
+def to_bundle(config, image_digests):
+    if config.networks:
+        log.warn("Unsupported top level key 'networks' - ignoring")
+
+    if config.volumes:
+        log.warn("Unsupported top level key 'volumes' - ignoring")
+
+    config = denormalize_config(config)
+
+    return {
+        'Version': VERSION,
+        'Services': {
+            name: convert_service_to_bundle(
+                name,
+                service_dict,
+                image_digests[name],
+            )
+            for name, service_dict in config['services'].items()
+        },
+    }
+
+
+def convert_service_to_bundle(name, service_dict, image_digest):
+    container_config = {'Image': image_digest}
+
+    for key, value in service_dict.items():
+        if key in IGNORED_KEYS:
+            continue
+
+        if key not in SUPPORTED_KEYS:
+            log.warn("Unsupported key '{}' in services.{} - ignoring".format(key, name))
+            continue
+
+        if key == 'environment':
+            container_config['Env'] = format_environment({
+                envkey: envvalue for envkey, envvalue in value.items()
+                if envvalue
+            })
+            continue
+
+        if key in SERVICE_KEYS:
+            container_config[SERVICE_KEYS[key]] = value
+            continue
+
+    set_command_and_args(
+        container_config,
+        service_dict.get('entrypoint', []),
+        service_dict.get('command', []))
+    container_config['Networks'] = make_service_networks(name, service_dict)
+
+    ports = make_port_specs(service_dict)
+    if ports:
+        container_config['Ports'] = ports
+
+    return container_config
+
+
+# See https://github.com/docker/swarmkit/blob/agent/exec/container/container.go#L95
+def set_command_and_args(config, entrypoint, command):
+    if isinstance(entrypoint, six.string_types):
+        entrypoint = split_command(entrypoint)
+    if isinstance(command, six.string_types):
+        command = split_command(command)
+
+    if entrypoint:
+        config['Command'] = entrypoint + command
+        return
+
+    if command:
+        config['Args'] = command
+
+
+def make_service_networks(name, service_dict):
+    networks = []
+
+    for network_name, network_def in get_network_defs_for_service(service_dict).items():
+        for key in network_def.keys():
+            log.warn(
+                "Unsupported key '{}' in services.{}.networks.{} - ignoring"
+                .format(key, name, network_name))
+
+        networks.append(network_name)
+
+    return networks
+
+
+def make_port_specs(service_dict):
+    ports = []
+
+    internal_ports = [
+        internal_port
+        for port_def in service_dict.get('ports', [])
+        for internal_port in split_port(port_def)[0]
+    ]
+
+    internal_ports += service_dict.get('expose', [])
+
+    for internal_port in internal_ports:
+        spec = make_port_spec(internal_port)
+        if spec not in ports:
+            ports.append(spec)
+
+    return ports
+
+
+def make_port_spec(value):
+    components = six.text_type(value).partition('/')
+    return {
+        'Protocol': components[2] or 'tcp',
+        'Port': int(components[0]),
+    }

compose/cli/__init__.py

@@ -0,0 +1,37 @@
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+import subprocess
+import sys
+
+# Attempt to detect https://github.com/docker/compose/issues/4344
+try:
+    # We don't try importing pip because it messes with package imports
+    # on some Linux distros (Ubuntu, Fedora)
+    # https://github.com/docker/compose/issues/4425
+    # https://github.com/docker/compose/issues/4481
+    # https://github.com/pypa/pip/blob/master/pip/_vendor/__init__.py
+    s_cmd = subprocess.Popen(
+        ['pip', 'freeze'], stderr=subprocess.PIPE, stdout=subprocess.PIPE
+    )
+    packages = s_cmd.communicate()[0].splitlines()
+    dockerpy_installed = len(
+        list(filter(lambda p: p.startswith(b'docker-py=='), packages))
+    ) > 0
+    if dockerpy_installed:
+        from .colors import red
+        print(
+            red('ERROR:'),
+            "Dependency conflict: an older version of the 'docker-py' package "
+            "is polluting the namespace. "
+            "Run the following command to remedy the issue:\n"
+            "pip uninstall docker docker-py; pip install docker",
+            file=sys.stderr
+        )
+        sys.exit(1)
+
+except OSError:
+    # pip command is not available, which indicates it's probably the binary
+    # distribution of Compose which is not affected
+    pass

compose/cli/colors.py

@@ -1,4 +1,8 @@
+from __future__ import absolute_import
 from __future__ import unicode_literals
+
+import colorama
+
 NAMES = [
     'grey',
     'red',
@@ -29,6 +33,7 @@ def make_color_fn(code):
     return lambda s: ansi_color(code, s)
 
 
+colorama.init(strip=False)
 for (name, code) in get_pairs():
     globals()[name] = make_color_fn(code)
 

compose/cli/command.py

@@ -1,102 +1,132 @@
-from __future__ import unicode_literals
 from __future__ import absolute_import
-from requests.exceptions import ConnectionError, SSLError
+from __future__ import unicode_literals
+
 import logging
 import os
 import re
+import ssl
+
 import six
 
-from .. import config
-from ..project import Project
-from ..service import ConfigError
-from .docopt_command import DocoptCommand
-from .utils import call_silently, is_mac, is_ubuntu
-from .docker_client import docker_client
-from . import verbose_proxy
 from . import errors
-from .. import __version__
+from . import verbose_proxy
+from .. import config
+from ..config.environment import Environment
+from ..const import API_VERSIONS
+from ..project import Project
+from .docker_client import docker_client
+from .docker_client import tls_config_from_options
+from .utils import get_version_info
 
 log = logging.getLogger(__name__)
 
 
-class Command(DocoptCommand):
-    base_dir = '.'
+def project_from_options(project_dir, options):
+    environment = Environment.from_env_file(project_dir)
+    host = options.get('--host')
+    if host is not None:
+        host = host.lstrip('=')
+    return get_project(
+        project_dir,
+        get_config_path_from_options(project_dir, options, environment),
+        project_name=options.get('--project-name'),
+        verbose=options.get('--verbose'),
+        host=host,
+        tls_config=tls_config_from_options(options),
+        environment=environment
+    )
 
-    def dispatch(self, *args, **kwargs):
-        try:
-            super(Command, self).dispatch(*args, **kwargs)
-        except SSLError as e:
-            raise errors.UserError('SSL error: %s' % e)
-        except ConnectionError:
-            if call_silently(['which', 'docker']) != 0:
-                if is_mac():
-                    raise errors.DockerNotFoundMac()
-                elif is_ubuntu():
-                    raise errors.DockerNotFoundUbuntu()
-                else:
-                    raise errors.DockerNotFoundGeneric()
-            elif call_silently(['which', 'boot2docker']) == 0:
-                raise errors.ConnectionErrorBoot2Docker()
-            else:
-                raise errors.ConnectionErrorGeneric(self.get_client().base_url)
 
-    def perform_command(self, options, handler, command_options):
-        if options['COMMAND'] in ('help', 'version'):
-            # Skip looking up the compose file.
-            handler(None, command_options)
-            return
+def get_config_from_options(base_dir, options):
+    environment = Environment.from_env_file(base_dir)
+    config_path = get_config_path_from_options(
+        base_dir, options, environment
+    )
+    return config.load(
+        config.find(base_dir, config_path, environment)
+    )
 
-        if 'FIG_FILE' in os.environ:
-            log.warn('The FIG_FILE environment variable is deprecated.')
-            log.warn('Please use COMPOSE_FILE instead.')
 
-        explicit_config_path = options.get('--file') or os.environ.get('COMPOSE_FILE') or os.environ.get('FIG_FILE')
-        project = self.get_project(
-            explicit_config_path,
-            project_name=options.get('--project-name'),
-            verbose=options.get('--verbose'))
+def get_config_path_from_options(base_dir, options, environment):
+    file_option = options.get('--file')
+    if file_option:
+        return file_option
 
-        handler(project, command_options)
+    config_files = environment.get('COMPOSE_FILE')
+    if config_files:
+        return config_files.split(os.pathsep)
+    return None
 
-    def get_client(self, verbose=False):
-        client = docker_client()
-        if verbose:
-            version_info = six.iteritems(client.version())
-            log.info("Compose version %s", __version__)
-            log.info("Docker base_url: %s", client.base_url)
-            log.info("Docker version: %s",
-                     ", ".join("%s=%s" % item for item in version_info))
-            return verbose_proxy.VerboseProxy('docker', client)
-        return client
 
-    def get_project(self, config_path=None, project_name=None, verbose=False):
-        config_details = config.find(self.base_dir, config_path)
+def get_tls_version(environment):
+    compose_tls_version = environment.get('COMPOSE_TLS_VERSION', None)
+    if not compose_tls_version:
+        return None
 
-        try:
-            return Project.from_dicts(
-                self.get_project_name(config_details.working_dir, project_name),
-                config.load(config_details),
-                self.get_client(verbose=verbose))
-        except ConfigError as e:
-            raise errors.UserError(six.text_type(e))
+    tls_attr_name = "PROTOCOL_{}".format(compose_tls_version)
+    if not hasattr(ssl, tls_attr_name):
+        log.warn(
+            'The "{}" protocol is unavailable. You may need to update your '
+            'version of Python or OpenSSL. Falling back to TLSv1 (default).'
+            .format(compose_tls_version)
+        )
+        return None
 
-    def get_project_name(self, working_dir, project_name=None):
-        def normalize_name(name):
-            return re.sub(r'[^a-z0-9]', '', name.lower())
+    return getattr(ssl, tls_attr_name)
 
-        if 'FIG_PROJECT_NAME' in os.environ:
-            log.warn('The FIG_PROJECT_NAME environment variable is deprecated.')
-            log.warn('Please use COMPOSE_PROJECT_NAME instead.')
 
-        project_name = (
-            project_name or
-            os.environ.get('COMPOSE_PROJECT_NAME') or
-            os.environ.get('FIG_PROJECT_NAME'))
-        if project_name is not None:
-            return normalize_name(project_name)
+def get_client(environment, verbose=False, version=None, tls_config=None, host=None,
+               tls_version=None):
 
-        project = os.path.basename(os.path.abspath(working_dir))
-        if project:
-            return normalize_name(project)
+    client = docker_client(
+        version=version, tls_config=tls_config, host=host,
+        environment=environment, tls_version=get_tls_version(environment)
+    )
+    if verbose:
+        version_info = six.iteritems(client.version())
+        log.info(get_version_info('full'))
+        log.info("Docker base_url: %s", client.base_url)
+        log.info("Docker version: %s",
+                 ", ".join("%s=%s" % item for item in version_info))
+        return verbose_proxy.VerboseProxy('docker', client)
+    return client
 
-        return 'default'
+
+def get_project(project_dir, config_path=None, project_name=None, verbose=False,
+                host=None, tls_config=None, environment=None):
+    if not environment:
+        environment = Environment.from_env_file(project_dir)
+    config_details = config.find(project_dir, config_path, environment)
+    project_name = get_project_name(
+        config_details.working_dir, project_name, environment
+    )
+    config_data = config.load(config_details)
+
+    api_version = environment.get(
+        'COMPOSE_API_VERSION',
+        API_VERSIONS[config_data.version])
+
+    client = get_client(
+        verbose=verbose, version=api_version, tls_config=tls_config,
+        host=host, environment=environment
+    )
+
+    with errors.handle_connection_errors(client):
+        return Project.from_config(project_name, config_data, client)
+
+
+def get_project_name(working_dir, project_name=None, environment=None):
+    def normalize_name(name):
+        return re.sub(r'[^a-z0-9]', '', name.lower())
+
+    if not environment:
+        environment = Environment.from_env_file(working_dir)
+    project_name = project_name or environment.get('COMPOSE_PROJECT_NAME')
+    if project_name:
+        return normalize_name(project_name)
+
+    project = os.path.basename(os.path.abspath(working_dir))
+    if project:
+        return normalize_name(project)
+
+    return 'default'

compose/cli/docker_client.py

@@ -1,37 +1,74 @@
-from docker import Client
-from docker import tls
-import ssl
-import os
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import logging
+
+from docker import APIClient
+from docker.errors import TLSParameterError
+from docker.tls import TLSConfig
+from docker.utils import kwargs_from_env
+
+from ..const import HTTP_TIMEOUT
+from .errors import UserError
+from .utils import generate_user_agent
+from .utils import unquote_path
+
+log = logging.getLogger(__name__)
 
 
-def docker_client():
+def tls_config_from_options(options):
+    tls = options.get('--tls', False)
+    ca_cert = unquote_path(options.get('--tlscacert'))
+    cert = unquote_path(options.get('--tlscert'))
+    key = unquote_path(options.get('--tlskey'))
+    verify = options.get('--tlsverify')
+    skip_hostname_check = options.get('--skip-hostname-check', False)
+
+    advanced_opts = any([ca_cert, cert, key, verify])
+
+    if tls is True and not advanced_opts:
+        return True
+    elif advanced_opts:  # --tls is a noop
+        client_cert = None
+        if cert or key:
+            client_cert = (cert, key)
+
+        return TLSConfig(
+            client_cert=client_cert, verify=verify, ca_cert=ca_cert,
+            assert_hostname=False if skip_hostname_check else None
+        )
+
+    return None
+
+
+def docker_client(environment, version=None, tls_config=None, host=None,
+                  tls_version=None):
     """
     Returns a docker-py client configured using environment variables
     according to the same logic as the official Docker client.
     """
-    cert_path = os.environ.get('DOCKER_CERT_PATH', '')
-    if cert_path == '':
-        cert_path = os.path.join(os.environ.get('HOME', ''), '.docker')
+    try:
+        kwargs = kwargs_from_env(environment=environment, ssl_version=tls_version)
+    except TLSParameterError:
+        raise UserError(
+            "TLS configuration is invalid - make sure your DOCKER_TLS_VERIFY "
+            "and DOCKER_CERT_PATH are set correctly.\n"
+            "You might need to run `eval \"$(docker-machine env default)\"`")
 
-    base_url = os.environ.get('DOCKER_HOST')
-    api_version = os.environ.get('COMPOSE_API_VERSION', '1.19')
+    if host:
+        kwargs['base_url'] = host
+    if tls_config:
+        kwargs['tls'] = tls_config
 
-    tls_config = None
+    if version:
+        kwargs['version'] = version
 
-    if os.environ.get('DOCKER_TLS_VERIFY', '') != '':
-        parts = base_url.split('://', 1)
-        base_url = '%s://%s' % ('https', parts[1])
+    timeout = environment.get('COMPOSE_HTTP_TIMEOUT')
+    if timeout:
+        kwargs['timeout'] = int(timeout)
+    else:
+        kwargs['timeout'] = HTTP_TIMEOUT
 
-        client_cert = (os.path.join(cert_path, 'cert.pem'), os.path.join(cert_path, 'key.pem'))
-        ca_cert = os.path.join(cert_path, 'ca.pem')
+    kwargs['user_agent'] = generate_user_agent()
 
-        tls_config = tls.TLSConfig(
-            ssl_version=ssl.PROTOCOL_TLSv1,
-            verify=True,
-            assert_hostname=False,
-            client_cert=client_cert,
-            ca_cert=ca_cert,
-        )
-
-    timeout = int(os.environ.get('DOCKER_CLIENT_TIMEOUT', 60))
-    return Client(base_url=base_url, tls=tls_config, version=api_version, timeout=timeout)
+    return APIClient(**kwargs)

compose/cli/docopt_command.py

@@ -1,9 +1,10 @@
-from __future__ import unicode_literals
 from __future__ import absolute_import
-import sys
+from __future__ import unicode_literals
 
 from inspect import getdoc
-from docopt import docopt, DocoptExit
+
+from docopt import docopt
+from docopt import DocoptExit
 
 
 def docopt_full_help(docstring, *args, **kwargs):
@@ -13,27 +14,21 @@ def docopt_full_help(docstring, *args, **kwargs):
         raise SystemExit(docstring)
 
 
-class DocoptCommand(object):
-    def docopt_options(self):
-        return {'options_first': True}
+class DocoptDispatcher(object):
 
-    def sys_dispatch(self):
-        self.dispatch(sys.argv[1:], None)
+    def __init__(self, command_class, options):
+        self.command_class = command_class
+        self.options = options
 
-    def dispatch(self, argv, global_options):
-        self.perform_command(*self.parse(argv, global_options))
-
-    def perform_command(self, options, handler, command_options):
-        handler(command_options)
-
-    def parse(self, argv, global_options):
-        options = docopt_full_help(getdoc(self), argv, **self.docopt_options())
+    def parse(self, argv):
+        command_help = getdoc(self.command_class)
+        options = docopt_full_help(command_help, argv, **self.options)
         command = options['COMMAND']
 
         if command is None:
-            raise SystemExit(getdoc(self))
+            raise SystemExit(command_help)
 
-        handler = self.get_handler(command)
+        handler = get_handler(self.command_class, command)
         docstring = getdoc(handler)
 
         if docstring is None:
@@ -42,13 +37,18 @@ class DocoptCommand(object):
         command_options = docopt_full_help(docstring, options['ARGS'], options_first=True)
         return options, handler, command_options
 
-    def get_handler(self, command):
-        command = command.replace('-', '_')
 
-        if not hasattr(self, command):
-            raise NoSuchCommand(command, self)
+def get_handler(command_class, command):
+    command = command.replace('-', '_')
+    # we certainly want to have "exec" command, since that's what docker client has
+    # but in python exec is a keyword
+    if command == "exec":
+        command = "exec_command"
 
-        return getattr(self, command)
+    if not hasattr(command_class, command):
+        raise NoSuchCommand(command, command_class)
+
+    return getattr(command_class, command)
 
 
 class NoSuchCommand(Exception):

compose/cli/errors.py

@@ -1,8 +1,30 @@
 from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import contextlib
+import logging
+import socket
+from distutils.spawn import find_executable
 from textwrap import dedent
 
+from docker.errors import APIError
+from requests.exceptions import ConnectionError as RequestsConnectionError
+from requests.exceptions import ReadTimeout
+from requests.exceptions import SSLError
+from requests.packages.urllib3.exceptions import ReadTimeoutError
+
+from ..const import API_VERSION_TO_ENGINE_VERSION
+from .utils import is_docker_for_mac_installed
+from .utils import is_mac
+from .utils import is_ubuntu
+from .utils import is_windows
+
+
+log = logging.getLogger(__name__)
+
 
 class UserError(Exception):
+
     def __init__(self, msg):
         self.msg = dedent(msg).strip()
 
@@ -12,44 +34,104 @@ class UserError(Exception):
     __str__ = __unicode__
 
 
-class DockerNotFoundMac(UserError):
-    def __init__(self):
-        super(DockerNotFoundMac, self).__init__("""
-        Couldn't connect to Docker daemon. You might need to install docker-osx:
-
-        https://github.com/noplay/docker-osx
-        """)
+class ConnectionError(Exception):
+    pass
 
 
-class DockerNotFoundUbuntu(UserError):
-    def __init__(self):
-        super(DockerNotFoundUbuntu, self).__init__("""
-        Couldn't connect to Docker daemon. You might need to install Docker:
-
-        http://docs.docker.io/en/latest/installation/ubuntulinux/
-        """)
+@contextlib.contextmanager
+def handle_connection_errors(client):
+    try:
+        yield
+    except SSLError as e:
+        log.error('SSL error: %s' % e)
+        raise ConnectionError()
+    except RequestsConnectionError as e:
+        if e.args and isinstance(e.args[0], ReadTimeoutError):
+            log_timeout_error(client.timeout)
+            raise ConnectionError()
+        exit_with_error(get_conn_error_message(client.base_url))
+    except APIError as e:
+        log_api_error(e, client.api_version)
+        raise ConnectionError()
+    except (ReadTimeout, socket.timeout) as e:
+        log_timeout_error(client.timeout)
+        raise ConnectionError()
 
 
-class DockerNotFoundGeneric(UserError):
-    def __init__(self):
-        super(DockerNotFoundGeneric, self).__init__("""
-        Couldn't connect to Docker daemon. You might need to install Docker:
-
-        http://docs.docker.io/en/latest/installation/
-        """)
+def log_timeout_error(timeout):
+    log.error(
+        "An HTTP request took too long to complete. Retry with --verbose to "
+        "obtain debug information.\n"
+        "If you encounter this issue regularly because of slow network "
+        "conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher "
+        "value (current value: %s)." % timeout)
 
 
-class ConnectionErrorBoot2Docker(UserError):
-    def __init__(self):
-        super(ConnectionErrorBoot2Docker, self).__init__("""
-        Couldn't connect to Docker daemon - you might need to run `boot2docker up`.
-        """)
+def log_api_error(e, client_version):
+    if b'client is newer than server' not in e.explanation:
+        log.error(e.explanation)
+        return
+
+    version = API_VERSION_TO_ENGINE_VERSION.get(client_version)
+    if not version:
+        # They've set a custom API version
+        log.error(e.explanation)
+        return
+
+    log.error(
+        "The Docker Engine version is less than the minimum required by "
+        "Compose. Your current project requires a Docker Engine of "
+        "version {version} or greater.".format(version=version))
 
 
-class ConnectionErrorGeneric(UserError):
-    def __init__(self, url):
-        super(ConnectionErrorGeneric, self).__init__("""
-        Couldn't connect to Docker daemon at %s - is it running?
+def exit_with_error(msg):
+    log.error(dedent(msg).strip())
+    raise ConnectionError()
 
-        If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
-        """ % url)
+
+def get_conn_error_message(url):
+    if find_executable('docker') is None:
+        return docker_not_found_msg("Couldn't connect to Docker daemon.")
+    if is_docker_for_mac_installed():
+        return conn_error_docker_for_mac
+    if find_executable('docker-machine') is not None:
+        return conn_error_docker_machine
+    return conn_error_generic.format(url=url)
+
+
+def docker_not_found_msg(problem):
+    return "{} You might need to install Docker:\n\n{}".format(
+        problem, docker_install_url())
+
+
+def docker_install_url():
+    if is_mac():
+        return docker_install_url_mac
+    elif is_ubuntu():
+        return docker_install_url_ubuntu
+    elif is_windows():
+        return docker_install_url_windows
+    else:
+        return docker_install_url_generic
+
+
+docker_install_url_mac = "https://docs.docker.com/engine/installation/mac/"
+docker_install_url_ubuntu = "https://docs.docker.com/engine/installation/ubuntulinux/"
+docker_install_url_windows = "https://docs.docker.com/engine/installation/windows/"
+docker_install_url_generic = "https://docs.docker.com/engine/installation/"
+
+
+conn_error_docker_machine = """
+    Couldn't connect to Docker daemon - you might need to run `docker-machine start default`.
+"""
+
+conn_error_docker_for_mac = """
+    Couldn't connect to Docker daemon. You might need to start Docker for Mac.
+"""
+
+
+conn_error_generic = """
+    Couldn't connect to Docker daemon at {url} - is it running?
+
+    If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
+"""

compose/cli/formatter.py

@@ -1,8 +1,14 @@
-from __future__ import unicode_literals
 from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import logging
 import os
+
+import six
 import texttable
 
+from compose.cli import colors
+
 
 def get_tty_width():
     tty_size = os.popen('stty size', 'r').read().split()
@@ -13,6 +19,7 @@ def get_tty_width():
 
 
 class Formatter(object):
+    """Format tabular data for printing."""
     def table(self, headers, rows):
         table = texttable.Texttable(max_width=get_tty_width())
         table.set_cols_dtype(['t' for h in headers])
@@ -21,3 +28,24 @@ class Formatter(object):
         table.set_chars(['-', '|', '+', '-'])
 
         return table.draw()
+
+
+class ConsoleWarningFormatter(logging.Formatter):
+    """A logging.Formatter which prints WARNING and ERROR messages with
+    a prefix of the log level colored appropriate for the log level.
+    """
+
+    def get_level_message(self, record):
+        separator = ': '
+        if record.levelno == logging.WARNING:
+            return colors.yellow(record.levelname) + separator
+        if record.levelno == logging.ERROR:
+            return colors.red(record.levelname) + separator
+
+        return ''
+
+    def format(self, record):
+        if isinstance(record.msg, six.binary_type):
+            record.msg = record.msg.decode('utf-8')
+        message = super(ConsoleWarningFormatter, self).format(record)
+        return '{0}{1}'.format(self.get_level_message(record), message)

compose/cli/log_printer.py

@@ -1,82 +1,237 @@
-from __future__ import unicode_literals
 from __future__ import absolute_import
+from __future__ import unicode_literals
+
 import sys
-
+from collections import namedtuple
 from itertools import cycle
+from threading import Thread
+
+from docker.errors import APIError
+from six.moves import _thread as thread
+from six.moves.queue import Empty
+from six.moves.queue import Queue
 
-from .multiplexer import Multiplexer, STOP
 from . import colors
-from .utils import split_buffer
+from compose import utils
+from compose.cli.signals import ShutdownException
+from compose.utils import split_buffer
+
+
+class LogPresenter(object):
+
+    def __init__(self, prefix_width, color_func):
+        self.prefix_width = prefix_width
+        self.color_func = color_func
+
+    def present(self, container, line):
+        prefix = container.name_without_project.ljust(self.prefix_width)
+        return '{prefix} {line}'.format(
+            prefix=self.color_func(prefix + ' |'),
+            line=line)
+
+
+def build_log_presenters(service_names, monochrome):
+    """Return an iterable of functions.
+
+    Each function can be used to format the logs output of a container.
+    """
+    prefix_width = max_name_width(service_names)
+
+    def no_color(text):
+        return text
+
+    for color_func in cycle([no_color] if monochrome else colors.rainbow()):
+        yield LogPresenter(prefix_width, color_func)
+
+
+def max_name_width(service_names, max_index_width=3):
+    """Calculate the maximum width of container names so we can make the log
+    prefixes line up like so:
+
+    db_1  | Listening
+    web_1 | Listening
+    """
+    return max(len(name) for name in service_names) + max_index_width
 
 
 class LogPrinter(object):
-    def __init__(self, containers, attach_params=None, output=sys.stdout, monochrome=False):
+    """Print logs from many containers to a single output stream."""
+
+    def __init__(self,
+                 containers,
+                 presenters,
+                 event_stream,
+                 output=sys.stdout,
+                 cascade_stop=False,
+                 log_args=None):
         self.containers = containers
-        self.attach_params = attach_params or {}
-        self.prefix_width = self._calculate_prefix_width(containers)
-        self.generators = self._make_log_generators(monochrome)
-        self.output = output
+        self.presenters = presenters
+        self.event_stream = event_stream
+        self.output = utils.get_output_stream(output)
+        self.cascade_stop = cascade_stop
+        self.log_args = log_args or {}
 
     def run(self):
-        mux = Multiplexer(self.generators)
-        for line in mux.loop():
+        if not self.containers:
+            return
+
+        queue = Queue()
+        thread_args = queue, self.log_args
+        thread_map = build_thread_map(self.containers, self.presenters, thread_args)
+        start_producer_thread((
+            thread_map,
+            self.event_stream,
+            self.presenters,
+            thread_args))
+
+        for line in consume_queue(queue, self.cascade_stop):
+            remove_stopped_threads(thread_map)
+
+            if not line:
+                if not thread_map:
+                    # There are no running containers left to tail, so exit
+                    return
+                # We got an empty line because of a timeout, but there are still
+                # active containers to tail, so continue
+                continue
+
             self.output.write(line)
+            self.output.flush()
 
-    def _calculate_prefix_width(self, containers):
-        """
-        Calculate the maximum width of container names so we can make the log
-        prefixes line up like so:
 
-        db_1  | Listening
-        web_1 | Listening
-        """
-        prefix_width = 0
-        for container in containers:
-            prefix_width = max(prefix_width, len(container.name_without_project))
-        return prefix_width
+def remove_stopped_threads(thread_map):
+    for container_id, tailer_thread in list(thread_map.items()):
+        if not tailer_thread.is_alive():
+            thread_map.pop(container_id, None)
 
-    def _make_log_generators(self, monochrome):
-        color_fns = cycle(colors.rainbow())
-        generators = []
 
-        def no_color(text):
-            return text
+def build_thread(container, presenter, queue, log_args):
+    tailer = Thread(
+        target=tail_container_logs,
+        args=(container, presenter, queue, log_args))
+    tailer.daemon = True
+    tailer.start()
+    return tailer
 
-        for container in self.containers:
-            if monochrome:
-                color_fn = no_color
-            else:
-                color_fn = next(color_fns)
-            generators.append(self._make_log_generator(container, color_fn))
 
-        return generators
+def build_thread_map(initial_containers, presenters, thread_args):
+    return {
+        container.id: build_thread(container, next(presenters), *thread_args)
+        for container in initial_containers
+    }
 
-    def _make_log_generator(self, container, color_fn):
-        prefix = color_fn(self._generate_prefix(container)).encode('utf-8')
-        # Attach to container before log printer starts running
-        line_generator = split_buffer(self._attach(container), '\n')
 
-        for line in line_generator:
-            yield prefix + line
+class QueueItem(namedtuple('_QueueItem', 'item is_stop exc')):
 
+    @classmethod
+    def new(cls, item):
+        return cls(item, None, None)
+
+    @classmethod
+    def exception(cls, exc):
+        return cls(None, None, exc)
+
+    @classmethod
+    def stop(cls):
+        return cls(None, True, None)
+
+
+def tail_container_logs(container, presenter, queue, log_args):
+    generator = get_log_generator(container)
+
+    try:
+        for item in generator(container, log_args):
+            queue.put(QueueItem.new(presenter.present(container, item)))
+    except Exception as e:
+        queue.put(QueueItem.exception(e))
+        return
+
+    if log_args.get('follow'):
+        queue.put(QueueItem.new(presenter.color_func(wait_on_exit(container))))
+    queue.put(QueueItem.stop())
+
+
+def get_log_generator(container):
+    if container.has_api_logs:
+        return build_log_generator
+    return build_no_log_generator
+
+
+def build_no_log_generator(container, log_args):
+    """Return a generator that prints a warning about logs and waits for
+    container to exit.
+    """
+    yield "WARNING: no logs are available with the '{}' log driver\n".format(
+        container.log_driver)
+
+
+def build_log_generator(container, log_args):
+    # if the container doesn't have a log_stream we need to attach to container
+    # before log printer starts running
+    if container.log_stream is None:
+        stream = container.logs(stdout=True, stderr=True, stream=True, **log_args)
+    else:
+        stream = container.log_stream
+
+    return split_buffer(stream)
+
+
+def wait_on_exit(container):
+    try:
         exit_code = container.wait()
-        yield color_fn("%s exited with code %s\n" % (container.name, exit_code))
-        yield STOP
+        return "%s exited with code %s\n" % (container.name, exit_code)
+    except APIError as e:
+        return "Unexpected API error for %s (HTTP code %s)\nResponse body:\n%s\n" % (
+            container.name, e.response.status_code,
+            e.response.text or '[empty]'
+        )
 
-    def _generate_prefix(self, container):
-        """
-        Generate the prefix for a log line without colour
-        """
-        name = container.name_without_project
-        padding = ' ' * (self.prefix_width - len(name))
-        return ''.join([name, padding, ' | '])
 
-    def _attach(self, container):
-        params = {
-            'stdout': True,
-            'stderr': True,
-            'stream': True,
-        }
-        params.update(self.attach_params)
-        params = dict((name, 1 if value else 0) for (name, value) in list(params.items()))
-        return container.attach(**params)
+def start_producer_thread(thread_args):
+    producer = Thread(target=watch_events, args=thread_args)
+    producer.daemon = True
+    producer.start()
+
+
+def watch_events(thread_map, event_stream, presenters, thread_args):
+    for event in event_stream:
+        if event['action'] == 'stop':
+            thread_map.pop(event['id'], None)
+
+        if event['action'] != 'start':
+            continue
+
+        if event['id'] in thread_map:
+            if thread_map[event['id']].is_alive():
+                continue
+            # Container was stopped and started, we need a new thread
+            thread_map.pop(event['id'], None)
+
+        thread_map[event['id']] = build_thread(
+            event['container'],
+            next(presenters),
+            *thread_args)
+
+
+def consume_queue(queue, cascade_stop):
+    """Consume the queue by reading lines off of it and yielding them."""
+    while True:
+        try:
+            item = queue.get(timeout=0.1)
+        except Empty:
+            yield None
+            continue
+        # See https://github.com/docker/compose/issues/189
+        except thread.error:
+            raise ShutdownException()
+
+        if item.exc:
+            raise item.exc
+
+        if item.is_stop:
+            if cascade_stop:
+                raise StopIteration
+            else:
+                continue
+
+        yield item.item

compose/cli/multiplexer.py

@@ -1,42 +0,0 @@
-from __future__ import absolute_import
-from threading import Thread
-
-try:
-    from Queue import Queue, Empty
-except ImportError:
-    from queue import Queue, Empty  # Python 3.x
-
-
-# Yield STOP from an input generator to stop the
-# top-level loop without processing any more input.
-STOP = object()
-
-
-class Multiplexer(object):
-    def __init__(self, generators):
-        self.generators = generators
-        self.queue = Queue()
-
-    def loop(self):
-        self._init_readers()
-
-        while True:
-            try:
-                item = self.queue.get(timeout=0.1)
-                if item is STOP:
-                    break
-                else:
-                    yield item
-            except Empty:
-                pass
-
-    def _init_readers(self):
-        for generator in self.generators:
-            t = Thread(target=_enqueue_output, args=(generator, self.queue))
-            t.daemon = True
-            t.start()
-
-
-def _enqueue_output(generator, queue):
-    for item in generator:
-        queue.put(item)

compose/cli/signals.py

@@ -0,0 +1,21 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import signal
+
+
+class ShutdownException(Exception):
+    pass
+
+
+def shutdown(signal, frame):
+    raise ShutdownException()
+
+
+def set_signal_handler(handler):
+    signal.signal(signal.SIGINT, handler)
+    signal.signal(signal.SIGTERM, handler)
+
+
+def set_signal_handler_to_shutdown():
+    set_signal_handler(shutdown)

compose/cli/utils.py

@@ -1,14 +1,24 @@
-from __future__ import unicode_literals
 from __future__ import absolute_import
 from __future__ import division
+from __future__ import unicode_literals
 
-from .. import __version__
-import datetime
-from docker import version as docker_py_version
 import os
 import platform
-import subprocess
 import ssl
+import subprocess
+import sys
+
+import docker
+
+import compose
+from ..const import IS_WINDOWS_PLATFORM
+
+# WindowsError is not defined on non-win32 platforms. Avoid runtime errors by
+# defining it as OSError (its parent class) if missing.
+try:
+    WindowsError
+except NameError:
+    WindowsError = OSError
 
 
 def yesno(prompt, default=None):
@@ -21,7 +31,7 @@ def yesno(prompt, default=None):
     Unrecognised input (anything other than "y", "n", "yes",
     "no" or "") will return None.
     """
-    answer = raw_input(prompt).strip().lower()
+    answer = input(prompt).strip().lower()
 
     if answer == "y" or answer == "yes":
         return True
@@ -33,81 +43,14 @@ def yesno(prompt, default=None):
         return None
 
 
-# http://stackoverflow.com/a/5164027
-def prettydate(d):
-    diff = datetime.datetime.utcnow() - d
-    s = diff.seconds
-    if diff.days > 7 or diff.days < 0:
-        return d.strftime('%d %b %y')
-    elif diff.days == 1:
-        return '1 day ago'
-    elif diff.days > 1:
-        return '{0} days ago'.format(diff.days)
-    elif s <= 1:
-        return 'just now'
-    elif s < 60:
-        return '{0} seconds ago'.format(s)
-    elif s < 120:
-        return '1 minute ago'
-    elif s < 3600:
-        return '{0} minutes ago'.format(s / 60)
-    elif s < 7200:
-        return '1 hour ago'
-    else:
-        return '{0} hours ago'.format(s / 3600)
-
-
-def mkdir(path, permissions=0o700):
-    if not os.path.exists(path):
-        os.mkdir(path)
-
-    os.chmod(path, permissions)
-
-    return path
-
-
-def find_candidates_in_parent_dirs(filenames, path):
+def input(prompt):
     """
-    Given a directory path to start, looks for filenames in the
-    directory, and then each parent directory successively,
-    until found.
-
-    Returns tuple (candidates, path).
+    Version of input (raw_input in Python 2) which forces a flush of sys.stdout
+    to avoid problems where the prompt fails to appear due to line buffering
     """
-    candidates = [filename for filename in filenames
-                  if os.path.exists(os.path.join(path, filename))]
-
-    if len(candidates) == 0:
-        parent_dir = os.path.join(path, '..')
-        if os.path.abspath(parent_dir) != os.path.abspath(path):
-            return find_candidates_in_parent_dirs(filenames, parent_dir)
-
-    return (candidates, path)
-
-
-def split_buffer(reader, separator):
-    """
-    Given a generator which yields strings and a separator string,
-    joins all input, splits on the separator and yields each chunk.
-
-    Unlike string.split(), each chunk includes the trailing
-    separator, except for the last one if none was found on the end
-    of the input.
-    """
-    buffered = str('')
-    separator = str(separator)
-
-    for data in reader:
-        buffered += data
-        while True:
-            index = buffered.find(separator)
-            if index == -1:
-                break
-            yield buffered[:index + 1]
-            buffered = buffered[index + 1:]
-
-    if len(buffered) > 0:
-        yield buffered
+    sys.stdout.write(prompt)
+    sys.stdout.flush()
+    return sys.stdin.readline().rstrip('\n')
 
 
 def call_silently(*args, **kwargs):
@@ -115,7 +58,12 @@ def call_silently(*args, **kwargs):
     Like subprocess.call(), but redirects stdout and stderr to /dev/null.
     """
     with open(os.devnull, 'w') as shutup:
-        return subprocess.call(*args, stdout=shutup, stderr=shutup, **kwargs)
+        try:
+            return subprocess.call(*args, stdout=shutup, stderr=shutup, **kwargs)
+        except WindowsError:
+            # On Windows, subprocess.call() can still raise exceptions. Normalize
+            # to POSIXy behaviour by returning a nonzero exit code.
+            return 1
 
 
 def is_mac():
@@ -126,14 +74,64 @@ def is_ubuntu():
     return platform.system() == 'Linux' and platform.linux_distribution()[0] == 'Ubuntu'
 
 
+def is_windows():
+    return IS_WINDOWS_PLATFORM
+
+
 def get_version_info(scope):
-    versioninfo = 'docker-compose version: %s' % __version__
+    versioninfo = 'docker-compose version {}, build {}'.format(
+        compose.__version__,
+        get_build_version())
+
     if scope == 'compose':
         return versioninfo
-    elif scope == 'full':
-        return versioninfo + '\n' \
-            + "docker-py version: %s\n" % docker_py_version \
-            + "%s version: %s\n" % (platform.python_implementation(), platform.python_version()) \
-            + "OpenSSL version: %s" % ssl.OPENSSL_VERSION
+    if scope == 'full':
+        return (
+            "{}\n"
+            "docker-py version: {}\n"
+            "{} version: {}\n"
+            "OpenSSL version: {}"
+        ).format(
+            versioninfo,
+            docker.version,
+            platform.python_implementation(),
+            platform.python_version(),
+            ssl.OPENSSL_VERSION)
+
+    raise ValueError("{} is not a valid version scope".format(scope))
+
+
+def get_build_version():
+    filename = os.path.join(os.path.dirname(compose.__file__), 'GITSHA')
+    if not os.path.exists(filename):
+        return 'unknown'
+
+    with open(filename) as fh:
+        return fh.read().strip()
+
+
+def is_docker_for_mac_installed():
+    return is_mac() and os.path.isdir('/Applications/Docker.app')
+
+
+def generate_user_agent():
+    parts = [
+        "docker-compose/{}".format(compose.__version__),
+        "docker-py/{}".format(docker.__version__),
+    ]
+    try:
+        p_system = platform.system()
+        p_release = platform.release()
+    except IOError:
+        pass
     else:
-        raise RuntimeError('passed unallowed value to `cli.utils.get_version_info`')
+        parts.append("{}/{}".format(p_system, p_release))
+    return " ".join(parts)
+
+
+def unquote_path(s):
+    if not s:
+        return s
+    if s[0] == '"' and s[-1] == '"':
+        return s[1:-1]
+    return s

compose/cli/verbose_proxy.py

@@ -1,8 +1,10 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
 
 import functools
-from itertools import chain
 import logging
 import pprint
+from itertools import chain
 
 import six
 

compose/config.py

@@ -1,590 +0,0 @@
-import logging
-import os
-import sys
-import yaml
-from collections import namedtuple
-
-import six
-
-from compose.cli.utils import find_candidates_in_parent_dirs
-
-
-DOCKER_CONFIG_KEYS = [
-    'cap_add',
-    'cap_drop',
-    'command',
-    'cpu_shares',
-    'cpuset',
-    'detach',
-    'devices',
-    'dns',
-    'dns_search',
-    'domainname',
-    'entrypoint',
-    'env_file',
-    'environment',
-    'extra_hosts',
-    'hostname',
-    'image',
-    'labels',
-    'links',
-    'log_driver',
-    'log_opt',
-    'mac_address',
-    'mem_limit',
-    'memswap_limit',
-    'net',
-    'pid',
-    'ports',
-    'privileged',
-    'read_only',
-    'restart',
-    'security_opt',
-    'stdin_open',
-    'tty',
-    'user',
-    'volume_driver',
-    'volumes',
-    'volumes_from',
-    'working_dir',
-]
-
-ALLOWED_KEYS = DOCKER_CONFIG_KEYS + [
-    'build',
-    'container_name',
-    'dockerfile',
-    'expose',
-    'external_links',
-    'name',
-]
-
-DOCKER_CONFIG_HINTS = {
-    'cpu_share': 'cpu_shares',
-    'add_host': 'extra_hosts',
-    'hosts': 'extra_hosts',
-    'extra_host': 'extra_hosts',
-    'device': 'devices',
-    'link': 'links',
-    'memory_swap': 'memswap_limit',
-    'port': 'ports',
-    'privilege': 'privileged',
-    'priviliged': 'privileged',
-    'privilige': 'privileged',
-    'volume': 'volumes',
-    'workdir': 'working_dir',
-}
-
-
-SUPPORTED_FILENAMES = [
-    'docker-compose.yml',
-    'docker-compose.yaml',
-    'fig.yml',
-    'fig.yaml',
-]
-
-
-PATH_START_CHARS = [
-    '/',
-    '.',
-    '~',
-]
-
-
-log = logging.getLogger(__name__)
-
-
-ConfigDetails = namedtuple('ConfigDetails', 'config working_dir filename')
-
-
-def find(base_dir, filename):
-    if filename == '-':
-        return ConfigDetails(yaml.safe_load(sys.stdin), os.getcwd(), None)
-
-    if filename:
-        filename = os.path.join(base_dir, filename)
-    else:
-        filename = get_config_path(base_dir)
-    return ConfigDetails(load_yaml(filename), os.path.dirname(filename), filename)
-
-
-def get_config_path(base_dir):
-    (candidates, path) = find_candidates_in_parent_dirs(SUPPORTED_FILENAMES, base_dir)
-
-    if len(candidates) == 0:
-        raise ComposeFileNotFound(SUPPORTED_FILENAMES)
-
-    winner = candidates[0]
-
-    if len(candidates) > 1:
-        log.warn("Found multiple config files with supported names: %s", ", ".join(candidates))
-        log.warn("Using %s\n", winner)
-
-    if winner == 'docker-compose.yaml':
-        log.warn("Please be aware that .yml is the expected extension "
-                 "in most cases, and using .yaml can cause compatibility "
-                 "issues in future.\n")
-
-    if winner.startswith("fig."):
-        log.warn("%s is deprecated and will not be supported in future. "
-                 "Please rename your config file to docker-compose.yml\n" % winner)
-
-    return os.path.join(path, winner)
-
-
-def load(config_details):
-    dictionary, working_dir, filename = config_details
-    service_dicts = []
-
-    for service_name, service_dict in list(dictionary.items()):
-        if not isinstance(service_dict, dict):
-            raise ConfigurationError('Service "%s" doesn\'t have any configuration options. All top level keys in your docker-compose.yml must map to a dictionary of configuration options.' % service_name)
-        loader = ServiceLoader(working_dir=working_dir, filename=filename)
-        service_dict = loader.make_service_dict(service_name, service_dict)
-        validate_paths(service_dict)
-        service_dicts.append(service_dict)
-
-    return service_dicts
-
-
-class ServiceLoader(object):
-    def __init__(self, working_dir, filename=None, already_seen=None):
-        self.working_dir = os.path.abspath(working_dir)
-        if filename:
-            self.filename = os.path.abspath(filename)
-        else:
-            self.filename = filename
-        self.already_seen = already_seen or []
-
-    def detect_cycle(self, name):
-        if self.signature(name) in self.already_seen:
-            raise CircularReference(self.already_seen + [self.signature(name)])
-
-    def make_service_dict(self, name, service_dict):
-        service_dict = service_dict.copy()
-        service_dict['name'] = name
-        service_dict = resolve_environment(service_dict, working_dir=self.working_dir)
-        service_dict = self.resolve_extends(service_dict)
-        return process_container_options(service_dict, working_dir=self.working_dir)
-
-    def resolve_extends(self, service_dict):
-        if 'extends' not in service_dict:
-            return service_dict
-
-        extends_options = self.validate_extends_options(service_dict['name'], service_dict['extends'])
-
-        if self.working_dir is None:
-            raise Exception("No working_dir passed to ServiceLoader()")
-
-        if 'file' in extends_options:
-            extends_from_filename = extends_options['file']
-            other_config_path = expand_path(self.working_dir, extends_from_filename)
-        else:
-            other_config_path = self.filename
-
-        other_working_dir = os.path.dirname(other_config_path)
-        other_already_seen = self.already_seen + [self.signature(service_dict['name'])]
-        other_loader = ServiceLoader(
-            working_dir=other_working_dir,
-            filename=other_config_path,
-            already_seen=other_already_seen,
-        )
-
-        other_config = load_yaml(other_config_path)
-        other_service_dict = other_config[extends_options['service']]
-        other_loader.detect_cycle(extends_options['service'])
-        other_service_dict = other_loader.make_service_dict(
-            service_dict['name'],
-            other_service_dict,
-        )
-        validate_extended_service_dict(
-            other_service_dict,
-            filename=other_config_path,
-            service=extends_options['service'],
-        )
-
-        return merge_service_dicts(other_service_dict, service_dict)
-
-    def signature(self, name):
-        return (self.filename, name)
-
-    def validate_extends_options(self, service_name, extends_options):
-        error_prefix = "Invalid 'extends' configuration for %s:" % service_name
-
-        if not isinstance(extends_options, dict):
-            raise ConfigurationError("%s must be a dictionary" % error_prefix)
-
-        if 'service' not in extends_options:
-            raise ConfigurationError(
-                "%s you need to specify a service, e.g. 'service: web'" % error_prefix
-            )
-
-        if 'file' not in extends_options and self.filename is None:
-            raise ConfigurationError(
-                "%s you need to specify a 'file', e.g. 'file: something.yml'" % error_prefix
-            )
-
-        for k, _ in extends_options.items():
-            if k not in ['file', 'service']:
-                raise ConfigurationError(
-                    "%s unsupported configuration option '%s'" % (error_prefix, k)
-                )
-
-        return extends_options
-
-
-def validate_extended_service_dict(service_dict, filename, service):
-    error_prefix = "Cannot extend service '%s' in %s:" % (service, filename)
-
-    if 'links' in service_dict:
-        raise ConfigurationError("%s services with 'links' cannot be extended" % error_prefix)
-
-    if 'volumes_from' in service_dict:
-        raise ConfigurationError("%s services with 'volumes_from' cannot be extended" % error_prefix)
-
-    if 'net' in service_dict:
-        if get_service_name_from_net(service_dict['net']) is not None:
-            raise ConfigurationError("%s services with 'net: container' cannot be extended" % error_prefix)
-
-
-def process_container_options(service_dict, working_dir=None):
-    for k in service_dict:
-        if k not in ALLOWED_KEYS:
-            msg = "Unsupported config option for %s service: '%s'" % (service_dict['name'], k)
-            if k in DOCKER_CONFIG_HINTS:
-                msg += " (did you mean '%s'?)" % DOCKER_CONFIG_HINTS[k]
-            raise ConfigurationError(msg)
-
-    service_dict = service_dict.copy()
-
-    if 'memswap_limit' in service_dict and 'mem_limit' not in service_dict:
-        raise ConfigurationError("Invalid 'memswap_limit' configuration for %s service: when defining 'memswap_limit' you must set 'mem_limit' as well" % service_dict['name'])
-
-    if 'volumes' in service_dict and service_dict.get('volume_driver') is None:
-        service_dict['volumes'] = resolve_volume_paths(service_dict, working_dir=working_dir)
-
-    if 'build' in service_dict:
-        service_dict['build'] = resolve_build_path(service_dict['build'], working_dir=working_dir)
-
-    if 'labels' in service_dict:
-        service_dict['labels'] = parse_labels(service_dict['labels'])
-
-    return service_dict
-
-
-def merge_service_dicts(base, override):
-    d = base.copy()
-
-    if 'environment' in base or 'environment' in override:
-        d['environment'] = merge_environment(
-            base.get('environment'),
-            override.get('environment'),
-        )
-
-    path_mapping_keys = ['volumes', 'devices']
-
-    for key in path_mapping_keys:
-        if key in base or key in override:
-            d[key] = merge_path_mappings(
-                base.get(key),
-                override.get(key),
-            )
-
-    if 'labels' in base or 'labels' in override:
-        d['labels'] = merge_labels(
-            base.get('labels'),
-            override.get('labels'),
-        )
-
-    if 'image' in override and 'build' in d:
-        del d['build']
-
-    if 'build' in override and 'image' in d:
-        del d['image']
-
-    list_keys = ['ports', 'expose', 'external_links']
-
-    for key in list_keys:
-        if key in base or key in override:
-            d[key] = base.get(key, []) + override.get(key, [])
-
-    list_or_string_keys = ['dns', 'dns_search']
-
-    for key in list_or_string_keys:
-        if key in base or key in override:
-            d[key] = to_list(base.get(key)) + to_list(override.get(key))
-
-    already_merged_keys = ['environment', 'labels'] + path_mapping_keys + list_keys + list_or_string_keys
-
-    for k in set(ALLOWED_KEYS) - set(already_merged_keys):
-        if k in override:
-            d[k] = override[k]
-
-    return d
-
-
-def merge_environment(base, override):
-    env = parse_environment(base)
-    env.update(parse_environment(override))
-    return env
-
-
-def parse_links(links):
-    return dict(parse_link(l) for l in links)
-
-
-def parse_link(link):
-    if ':' in link:
-        source, alias = link.split(':', 1)
-        return (alias, source)
-    else:
-        return (link, link)
-
-
-def get_env_files(options, working_dir=None):
-    if 'env_file' not in options:
-        return {}
-
-    if working_dir is None:
-        raise Exception("No working_dir passed to get_env_files()")
-
-    env_files = options.get('env_file', [])
-    if not isinstance(env_files, list):
-        env_files = [env_files]
-
-    return [expand_path(working_dir, path) for path in env_files]
-
-
-def resolve_environment(service_dict, working_dir=None):
-    service_dict = service_dict.copy()
-
-    if 'environment' not in service_dict and 'env_file' not in service_dict:
-        return service_dict
-
-    env = {}
-
-    if 'env_file' in service_dict:
-        for f in get_env_files(service_dict, working_dir=working_dir):
-            env.update(env_vars_from_file(f))
-        del service_dict['env_file']
-
-    env.update(parse_environment(service_dict.get('environment')))
-    env = dict(resolve_env_var(k, v) for k, v in six.iteritems(env))
-
-    service_dict['environment'] = env
-    return service_dict
-
-
-def parse_environment(environment):
-    if not environment:
-        return {}
-
-    if isinstance(environment, list):
-        return dict(split_env(e) for e in environment)
-
-    if isinstance(environment, dict):
-        return dict(environment)
-
-    raise ConfigurationError(
-        "environment \"%s\" must be a list or mapping," %
-        environment
-    )
-
-
-def split_env(env):
-    if '=' in env:
-        return env.split('=', 1)
-    else:
-        return env, None
-
-
-def resolve_env_var(key, val):
-    if val is not None:
-        return key, val
-    elif key in os.environ:
-        return key, os.environ[key]
-    else:
-        return key, ''
-
-
-def env_vars_from_file(filename):
-    """
-    Read in a line delimited file of environment variables.
-    """
-    if not os.path.exists(filename):
-        raise ConfigurationError("Couldn't find env file: %s" % filename)
-    env = {}
-    for line in open(filename, 'r'):
-        line = line.strip()
-        if line and not line.startswith('#'):
-            k, v = split_env(line)
-            env[k] = v
-    return env
-
-
-def resolve_volume_paths(service_dict, working_dir=None):
-    if working_dir is None:
-        raise Exception("No working_dir passed to resolve_volume_paths()")
-
-    return [
-        resolve_volume_path(v, working_dir, service_dict['name'])
-        for v in service_dict['volumes']
-    ]
-
-
-def resolve_volume_path(volume, working_dir, service_name):
-    container_path, host_path = split_path_mapping(volume)
-    container_path = os.path.expanduser(os.path.expandvars(container_path))
-
-    if host_path is not None:
-        host_path = os.path.expanduser(os.path.expandvars(host_path))
-
-        if not any(host_path.startswith(c) for c in PATH_START_CHARS):
-            log.warn(
-                'Warning: the mapping "{0}:{1}" in the volumes config for '
-                'service "{2}" is ambiguous. In a future version of Docker, '
-                'it will designate a "named" volume '
-                '(see https://github.com/docker/docker/pull/14242). '
-                'To prevent unexpected behaviour, change it to "./{0}:{1}"'
-                .format(host_path, container_path, service_name)
-            )
-
-        return "%s:%s" % (expand_path(working_dir, host_path), container_path)
-    else:
-        return container_path
-
-
-def resolve_build_path(build_path, working_dir=None):
-    if working_dir is None:
-        raise Exception("No working_dir passed to resolve_build_path")
-    return expand_path(working_dir, build_path)
-
-
-def validate_paths(service_dict):
-    if 'build' in service_dict:
-        build_path = service_dict['build']
-        if not os.path.exists(build_path) or not os.access(build_path, os.R_OK):
-            raise ConfigurationError("build path %s either does not exist or is not accessible." % build_path)
-
-
-def merge_path_mappings(base, override):
-    d = dict_from_path_mappings(base)
-    d.update(dict_from_path_mappings(override))
-    return path_mappings_from_dict(d)
-
-
-def dict_from_path_mappings(path_mappings):
-    if path_mappings:
-        return dict(split_path_mapping(v) for v in path_mappings)
-    else:
-        return {}
-
-
-def path_mappings_from_dict(d):
-    return [join_path_mapping(v) for v in d.items()]
-
-
-def split_path_mapping(string):
-    if ':' in string:
-        (host, container) = string.split(':', 1)
-        return (container, host)
-    else:
-        return (string, None)
-
-
-def join_path_mapping(pair):
-    (container, host) = pair
-    if host is None:
-        return container
-    else:
-        return ":".join((host, container))
-
-
-def merge_labels(base, override):
-    labels = parse_labels(base)
-    labels.update(parse_labels(override))
-    return labels
-
-
-def parse_labels(labels):
-    if not labels:
-        return {}
-
-    if isinstance(labels, list):
-        return dict(split_label(e) for e in labels)
-
-    if isinstance(labels, dict):
-        return labels
-
-    raise ConfigurationError(
-        "labels \"%s\" must be a list or mapping" %
-        labels
-    )
-
-
-def split_label(label):
-    if '=' in label:
-        return label.split('=', 1)
-    else:
-        return label, ''
-
-
-def expand_path(working_dir, path):
-    return os.path.abspath(os.path.join(working_dir, path))
-
-
-def to_list(value):
-    if value is None:
-        return []
-    elif isinstance(value, six.string_types):
-        return [value]
-    else:
-        return value
-
-
-def get_service_name_from_net(net_config):
-    if not net_config:
-        return
-
-    if not net_config.startswith('container:'):
-        return
-
-    _, net_name = net_config.split(':', 1)
-    return net_name
-
-
-def load_yaml(filename):
-    try:
-        with open(filename, 'r') as fh:
-            return yaml.safe_load(fh)
-    except IOError as e:
-        raise ConfigurationError(six.text_type(e))
-
-
-class ConfigurationError(Exception):
-    def __init__(self, msg):
-        self.msg = msg
-
-    def __str__(self):
-        return self.msg
-
-
-class CircularReference(ConfigurationError):
-    def __init__(self, trail):
-        self.trail = trail
-
-    @property
-    def msg(self):
-        lines = [
-            "{} in {}".format(service_name, filename)
-            for (filename, service_name) in self.trail
-        ]
-        return "Circular reference:\n  {}".format("\n  extends ".join(lines))
-
-
-class ComposeFileNotFound(ConfigurationError):
-    def __init__(self, supported_filenames):
-        super(ComposeFileNotFound, self).__init__("""
-        Can't find a suitable configuration file in this directory or any parent. Are you in the right directory?
-
-        Supported filenames: %s
-        """ % ", ".join(supported_filenames))

compose/config/__init__.py

@@ -0,0 +1,11 @@
+# flake8: noqa
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+from . import environment
+from .config import ConfigurationError
+from .config import DOCKER_CONFIG_KEYS
+from .config import find
+from .config import load
+from .config import merge_environment
+from .config import parse_environment

compose/config/config_schema_v1.json

@@ -0,0 +1,188 @@
+{
+  "$schema": "http://json-schema.org/draft-04/schema#",
+  "id": "config_schema_v1.json",
+
+  "type": "object",
+
+  "patternProperties": {
+    "^[a-zA-Z0-9._-]+$": {
+      "$ref": "#/definitions/service"
+    }
+  },
+
+  "additionalProperties": false,
+
+  "definitions": {
+    "service": {
+      "id": "#/definitions/service",
+      "type": "object",
+
+      "properties": {
+        "build": {"type": "string"},
+        "cap_add": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "cap_drop": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "cgroup_parent": {"type": "string"},
+        "command": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "container_name": {"type": "string"},
+        "cpu_shares": {"type": ["number", "string"]},
+        "cpu_quota": {"type": ["number", "string"]},
+        "cpuset": {"type": "string"},
+        "devices": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "dns": {"$ref": "#/definitions/string_or_list"},
+        "dns_search": {"$ref": "#/definitions/string_or_list"},
+        "dockerfile": {"type": "string"},
+        "domainname": {"type": "string"},
+        "entrypoint": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "env_file": {"$ref": "#/definitions/string_or_list"},
+        "environment": {"$ref": "#/definitions/list_or_dict"},
+
+        "expose": {
+          "type": "array",
+          "items": {
+            "type": ["string", "number"],
+            "format": "expose"
+          },
+          "uniqueItems": true
+        },
+
+        "extends": {
+          "oneOf": [
+            {
+              "type": "string"
+            },
+            {
+              "type": "object",
+
+              "properties": {
+                "service": {"type": "string"},
+                "file": {"type": "string"}
+              },
+              "required": ["service"],
+              "additionalProperties": false
+            }
+          ]
+        },
+
+        "extra_hosts": {"$ref": "#/definitions/list_or_dict"},
+        "external_links": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "hostname": {"type": "string"},
+        "image": {"type": "string"},
+        "ipc": {"type": "string"},
+        "labels": {"$ref": "#/definitions/list_or_dict"},
+        "links": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "log_driver": {"type": "string"},
+        "log_opt": {"type": "object"},
+        "mac_address": {"type": "string"},
+        "mem_limit": {"type": ["number", "string"]},
+        "memswap_limit": {"type": ["number", "string"]},
+        "mem_swappiness": {"type": "integer"},
+        "net": {"type": "string"},
+        "pid": {"type": ["string", "null"]},
+
+        "ports": {
+          "type": "array",
+          "items": {
+            "type": ["string", "number"],
+            "format": "ports"
+          },
+          "uniqueItems": true
+        },
+
+        "privileged": {"type": "boolean"},
+        "read_only": {"type": "boolean"},
+        "restart": {"type": "string"},
+        "security_opt": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "shm_size": {"type": ["number", "string"]},
+        "stdin_open": {"type": "boolean"},
+        "stop_signal": {"type": "string"},
+        "tty": {"type": "boolean"},
+        "ulimits": {
+          "type": "object",
+          "patternProperties": {
+            "^[a-z]+$": {
+              "oneOf": [
+                {"type": "integer"},
+                {
+                  "type":"object",
+                  "properties": {
+                    "hard": {"type": "integer"},
+                    "soft": {"type": "integer"}
+                  },
+                  "required": ["soft", "hard"],
+                  "additionalProperties": false
+                }
+              ]
+            }
+          }
+        },
+        "user": {"type": "string"},
+        "volumes": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "volume_driver": {"type": "string"},
+        "volumes_from": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "working_dir": {"type": "string"}
+      },
+
+      "dependencies": {
+        "memswap_limit": ["mem_limit"]
+      },
+      "additionalProperties": false
+    },
+
+    "string_or_list": {
+      "oneOf": [
+        {"type": "string"},
+        {"$ref": "#/definitions/list_of_strings"}
+      ]
+    },
+
+    "list_of_strings": {
+      "type": "array",
+      "items": {"type": "string"},
+      "uniqueItems": true
+    },
+
+    "list_or_dict": {
+      "oneOf": [
+        {
+          "type": "object",
+          "patternProperties": {
+            ".+": {
+              "type": ["string", "number", "null"]
+            }
+          },
+          "additionalProperties": false
+        },
+        {"type": "array", "items": {"type": "string"}, "uniqueItems": true}
+      ]
+    },
+
+    "constraints": {
+      "service": {
+        "id": "#/definitions/constraints/service",
+        "anyOf": [
+          {
+            "required": ["build"],
+            "not": {"required": ["image"]}
+          },
+          {
+            "required": ["image"],
+            "not": {"anyOf": [
+              {"required": ["build"]},
+              {"required": ["dockerfile"]}
+            ]}
+          }
+        ]
+      }
+    }
+  }
+}

compose/config/config_schema_v2.0.json

@@ -0,0 +1,329 @@
+{
+  "$schema": "http://json-schema.org/draft-04/schema#",
+  "id": "config_schema_v2.0.json",
+  "type": "object",
+
+  "properties": {
+    "version": {
+      "type": "string"
+    },
+
+    "services": {
+      "id": "#/properties/services",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/service"
+        }
+      },
+      "additionalProperties": false
+    },
+
+    "networks": {
+      "id": "#/properties/networks",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/network"
+        }
+      }
+    },
+
+    "volumes": {
+      "id": "#/properties/volumes",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/volume"
+        }
+      },
+      "additionalProperties": false
+    }
+  },
+
+  "additionalProperties": false,
+
+  "definitions": {
+
+    "service": {
+      "id": "#/definitions/service",
+      "type": "object",
+
+      "properties": {
+        "build": {
+          "oneOf": [
+            {"type": "string"},
+            {
+              "type": "object",
+              "properties": {
+                "context": {"type": "string"},
+                "dockerfile": {"type": "string"},
+                "args": {"$ref": "#/definitions/list_or_dict"}
+              },
+              "additionalProperties": false
+            }
+          ]
+        },
+        "cap_add": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "cap_drop": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "cgroup_parent": {"type": "string"},
+        "command": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "container_name": {"type": "string"},
+        "cpu_shares": {"type": ["number", "string"]},
+        "cpu_quota": {"type": ["number", "string"]},
+        "cpuset": {"type": "string"},
+        "depends_on": {"$ref": "#/definitions/list_of_strings"},
+        "devices": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "dns": {"$ref": "#/definitions/string_or_list"},
+        "dns_search": {"$ref": "#/definitions/string_or_list"},
+        "domainname": {"type": "string"},
+        "entrypoint": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "env_file": {"$ref": "#/definitions/string_or_list"},
+        "environment": {"$ref": "#/definitions/list_or_dict"},
+
+        "expose": {
+          "type": "array",
+          "items": {
+            "type": ["string", "number"],
+            "format": "expose"
+          },
+          "uniqueItems": true
+        },
+
+        "extends": {
+          "oneOf": [
+            {
+              "type": "string"
+            },
+            {
+              "type": "object",
+
+              "properties": {
+                "service": {"type": "string"},
+                "file": {"type": "string"}
+              },
+              "required": ["service"],
+              "additionalProperties": false
+            }
+          ]
+        },
+
+        "external_links": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "extra_hosts": {"$ref": "#/definitions/list_or_dict"},
+        "hostname": {"type": "string"},
+        "image": {"type": "string"},
+        "ipc": {"type": "string"},
+        "labels": {"$ref": "#/definitions/list_or_dict"},
+        "links": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+
+        "logging": {
+            "type": "object",
+
+            "properties": {
+                "driver": {"type": "string"},
+                "options": {"type": "object"}
+            },
+            "additionalProperties": false
+        },
+
+        "mac_address": {"type": "string"},
+        "mem_limit": {"type": ["number", "string"]},
+        "memswap_limit": {"type": ["number", "string"]},
+        "mem_swappiness": {"type": "integer"},
+        "network_mode": {"type": "string"},
+
+        "networks": {
+          "oneOf": [
+            {"$ref": "#/definitions/list_of_strings"},
+            {
+              "type": "object",
+              "patternProperties": {
+                "^[a-zA-Z0-9._-]+$": {
+                  "oneOf": [
+                    {
+                      "type": "object",
+                      "properties": {
+                        "aliases": {"$ref": "#/definitions/list_of_strings"},
+                        "ipv4_address": {"type": "string"},
+                        "ipv6_address": {"type": "string"}
+                      },
+                      "additionalProperties": false
+                    },
+                    {"type": "null"}
+                  ]
+                }
+              },
+              "additionalProperties": false
+            }
+          ]
+        },
+        "oom_score_adj": {"type": "integer", "minimum": -1000, "maximum": 1000},
+        "group_add": {
+            "type": "array",
+            "items": {
+                "type": ["string", "number"]
+            },
+            "uniqueItems": true
+        },
+        "pid": {"type": ["string", "null"]},
+
+        "ports": {
+          "type": "array",
+          "items": {
+            "type": ["string", "number"],
+            "format": "ports"
+          },
+          "uniqueItems": true
+        },
+
+        "privileged": {"type": "boolean"},
+        "read_only": {"type": "boolean"},
+        "restart": {"type": "string"},
+        "security_opt": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "shm_size": {"type": ["number", "string"]},
+        "stdin_open": {"type": "boolean"},
+        "stop_grace_period": {"type": "string", "format": "duration"},
+        "stop_signal": {"type": "string"},
+        "tmpfs": {"$ref": "#/definitions/string_or_list"},
+        "tty": {"type": "boolean"},
+        "ulimits": {
+          "type": "object",
+          "patternProperties": {
+            "^[a-z]+$": {
+              "oneOf": [
+                {"type": "integer"},
+                {
+                  "type":"object",
+                  "properties": {
+                    "hard": {"type": "integer"},
+                    "soft": {"type": "integer"}
+                  },
+                  "required": ["soft", "hard"],
+                  "additionalProperties": false
+                }
+              ]
+            }
+          }
+        },
+        "user": {"type": "string"},
+        "volumes": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "volume_driver": {"type": "string"},
+        "volumes_from": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "working_dir": {"type": "string"}
+      },
+
+      "dependencies": {
+        "memswap_limit": ["mem_limit"]
+      },
+      "additionalProperties": false
+    },
+
+    "network": {
+      "id": "#/definitions/network",
+      "type": "object",
+      "properties": {
+        "driver": {"type": "string"},
+        "driver_opts": {
+          "type": "object",
+          "patternProperties": {
+            "^.+$": {"type": ["string", "number"]}
+          }
+        },
+        "ipam": {
+            "type": "object",
+            "properties": {
+                "driver": {"type": "string"},
+                "config": {
+                    "type": "array"
+                }
+            },
+            "additionalProperties": false
+        },
+        "external": {
+          "type": ["boolean", "object"],
+          "properties": {
+            "name": {"type": "string"}
+          },
+          "additionalProperties": false
+        },
+        "internal": {"type": "boolean"}
+      },
+      "additionalProperties": false
+    },
+
+    "volume": {
+      "id": "#/definitions/volume",
+      "type": ["object", "null"],
+      "properties": {
+        "driver": {"type": "string"},
+        "driver_opts": {
+          "type": "object",
+          "patternProperties": {
+            "^.+$": {"type": ["string", "number"]}
+          }
+        },
+        "external": {
+          "type": ["boolean", "object"],
+          "properties": {
+            "name": {"type": "string"}
+          },
+          "additionalProperties": false
+        }
+      },
+      "additionalProperties": false
+    },
+
+    "string_or_list": {
+      "oneOf": [
+        {"type": "string"},
+        {"$ref": "#/definitions/list_of_strings"}
+      ]
+    },
+
+    "list_of_strings": {
+      "type": "array",
+      "items": {"type": "string"},
+      "uniqueItems": true
+    },
+
+    "list_or_dict": {
+      "oneOf": [
+        {
+          "type": "object",
+          "patternProperties": {
+            ".+": {
+              "type": ["string", "number", "null"]
+            }
+          },
+          "additionalProperties": false
+        },
+        {"type": "array", "items": {"type": "string"}, "uniqueItems": true}
+      ]
+    },
+
+    "constraints": {
+      "service": {
+        "id": "#/definitions/constraints/service",
+        "anyOf": [
+          {"required": ["build"]},
+          {"required": ["image"]}
+        ],
+        "properties": {
+          "build": {
+            "required": ["context"]
+          }
+        }
+      }
+    }
+  }
+}

compose/config/config_schema_v2.1.json

@@ -0,0 +1,376 @@
+{
+  "$schema": "http://json-schema.org/draft-04/schema#",
+  "id": "config_schema_v2.1.json",
+  "type": "object",
+
+  "properties": {
+    "version": {
+      "type": "string"
+    },
+
+    "services": {
+      "id": "#/properties/services",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/service"
+        }
+      },
+      "additionalProperties": false
+    },
+
+    "networks": {
+      "id": "#/properties/networks",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/network"
+        }
+      }
+    },
+
+    "volumes": {
+      "id": "#/properties/volumes",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/volume"
+        }
+      },
+      "additionalProperties": false
+    }
+  },
+
+  "additionalProperties": false,
+
+  "definitions": {
+
+    "service": {
+      "id": "#/definitions/service",
+      "type": "object",
+
+      "properties": {
+        "build": {
+          "oneOf": [
+            {"type": "string"},
+            {
+              "type": "object",
+              "properties": {
+                "context": {"type": "string"},
+                "dockerfile": {"type": "string"},
+                "args": {"$ref": "#/definitions/list_or_dict"}
+              },
+              "additionalProperties": false
+            }
+          ]
+        },
+        "cap_add": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "cap_drop": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "cgroup_parent": {"type": "string"},
+        "command": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "container_name": {"type": "string"},
+        "cpu_shares": {"type": ["number", "string"]},
+        "cpu_quota": {"type": ["number", "string"]},
+        "cpuset": {"type": "string"},
+        "depends_on": {
+          "oneOf": [
+            {"$ref": "#/definitions/list_of_strings"},
+            {
+              "type": "object",
+              "additionalProperties": false,
+              "patternProperties": {
+                "^[a-zA-Z0-9._-]+$": {
+                  "type": "object",
+                  "additionalProperties": false,
+                  "properties": {
+                    "condition": {
+                      "type": "string",
+                      "enum": ["service_started", "service_healthy"]
+                    }
+                  },
+                  "required": ["condition"]
+                }
+              }
+            }
+          ]
+        },
+        "devices": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "dns": {"$ref": "#/definitions/string_or_list"},
+        "dns_search": {"$ref": "#/definitions/string_or_list"},
+        "domainname": {"type": "string"},
+        "entrypoint": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "env_file": {"$ref": "#/definitions/string_or_list"},
+        "environment": {"$ref": "#/definitions/list_or_dict"},
+
+        "expose": {
+          "type": "array",
+          "items": {
+            "type": ["string", "number"],
+            "format": "expose"
+          },
+          "uniqueItems": true
+        },
+
+        "extends": {
+          "oneOf": [
+            {
+              "type": "string"
+            },
+            {
+              "type": "object",
+
+              "properties": {
+                "service": {"type": "string"},
+                "file": {"type": "string"}
+              },
+              "required": ["service"],
+              "additionalProperties": false
+            }
+          ]
+        },
+
+        "external_links": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "extra_hosts": {"$ref": "#/definitions/list_or_dict"},
+        "healthcheck": {"$ref": "#/definitions/healthcheck"},
+        "hostname": {"type": "string"},
+        "image": {"type": "string"},
+        "ipc": {"type": "string"},
+        "isolation": {"type": "string"},
+        "labels": {"$ref": "#/definitions/list_or_dict"},
+        "links": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+
+        "logging": {
+            "type": "object",
+
+            "properties": {
+                "driver": {"type": "string"},
+                "options": {"type": "object"}
+            },
+            "additionalProperties": false
+        },
+
+        "mac_address": {"type": "string"},
+        "mem_limit": {"type": ["number", "string"]},
+        "memswap_limit": {"type": ["number", "string"]},
+        "mem_swappiness": {"type": "integer"},
+        "network_mode": {"type": "string"},
+
+        "networks": {
+          "oneOf": [
+            {"$ref": "#/definitions/list_of_strings"},
+            {
+              "type": "object",
+              "patternProperties": {
+                "^[a-zA-Z0-9._-]+$": {
+                  "oneOf": [
+                    {
+                      "type": "object",
+                      "properties": {
+                        "aliases": {"$ref": "#/definitions/list_of_strings"},
+                        "ipv4_address": {"type": "string"},
+                        "ipv6_address": {"type": "string"},
+                        "link_local_ips": {"$ref": "#/definitions/list_of_strings"}
+                      },
+                      "additionalProperties": false
+                    },
+                    {"type": "null"}
+                  ]
+                }
+              },
+              "additionalProperties": false
+            }
+          ]
+        },
+        "oom_score_adj": {"type": "integer", "minimum": -1000, "maximum": 1000},
+        "group_add": {
+            "type": "array",
+            "items": {
+                "type": ["string", "number"]
+            },
+            "uniqueItems": true
+        },
+        "pid": {"type": ["string", "null"]},
+
+        "ports": {
+          "type": "array",
+          "items": {
+            "type": ["string", "number"],
+            "format": "ports"
+          },
+          "uniqueItems": true
+        },
+
+        "privileged": {"type": "boolean"},
+        "read_only": {"type": "boolean"},
+        "restart": {"type": "string"},
+        "security_opt": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "shm_size": {"type": ["number", "string"]},
+        "sysctls": {"$ref": "#/definitions/list_or_dict"},
+        "stdin_open": {"type": "boolean"},
+        "stop_grace_period": {"type": "string", "format": "duration"},
+        "stop_signal": {"type": "string"},
+        "tmpfs": {"$ref": "#/definitions/string_or_list"},
+        "tty": {"type": "boolean"},
+        "ulimits": {
+          "type": "object",
+          "patternProperties": {
+            "^[a-z]+$": {
+              "oneOf": [
+                {"type": "integer"},
+                {
+                  "type":"object",
+                  "properties": {
+                    "hard": {"type": "integer"},
+                    "soft": {"type": "integer"}
+                  },
+                  "required": ["soft", "hard"],
+                  "additionalProperties": false
+                }
+              ]
+            }
+          }
+        },
+        "user": {"type": "string"},
+        "userns_mode": {"type": "string"},
+        "volumes": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "volume_driver": {"type": "string"},
+        "volumes_from": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "working_dir": {"type": "string"}
+      },
+
+      "dependencies": {
+        "memswap_limit": ["mem_limit"]
+      },
+      "additionalProperties": false
+    },
+
+    "healthcheck": {
+      "id": "#/definitions/healthcheck",
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "disable": {"type": "boolean"},
+        "interval": {"type": "string"},
+        "retries": {"type": "number"},
+        "test": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "timeout": {"type": "string"}
+      }
+    },
+
+    "network": {
+      "id": "#/definitions/network",
+      "type": "object",
+      "properties": {
+        "driver": {"type": "string"},
+        "driver_opts": {
+          "type": "object",
+          "patternProperties": {
+            "^.+$": {"type": ["string", "number"]}
+          }
+        },
+        "ipam": {
+            "type": "object",
+            "properties": {
+                "driver": {"type": "string"},
+                "config": {
+                    "type": "array"
+                }
+            },
+            "additionalProperties": false
+        },
+        "external": {
+          "type": ["boolean", "object"],
+          "properties": {
+            "name": {"type": "string"}
+          },
+          "additionalProperties": false
+        },
+        "internal": {"type": "boolean"},
+        "enable_ipv6": {"type": "boolean"},
+        "labels": {"$ref": "#/definitions/list_or_dict"}
+      },
+      "additionalProperties": false
+    },
+
+    "volume": {
+      "id": "#/definitions/volume",
+      "type": ["object", "null"],
+      "properties": {
+        "driver": {"type": "string"},
+        "driver_opts": {
+          "type": "object",
+          "patternProperties": {
+            "^.+$": {"type": ["string", "number"]}
+          }
+        },
+        "external": {
+          "type": ["boolean", "object"],
+          "properties": {
+            "name": {"type": "string"}
+          },
+          "additionalProperties": false
+        },
+        "labels": {"$ref": "#/definitions/list_or_dict"}
+      },
+      "additionalProperties": false
+    },
+
+    "string_or_list": {
+      "oneOf": [
+        {"type": "string"},
+        {"$ref": "#/definitions/list_of_strings"}
+      ]
+    },
+
+    "list_of_strings": {
+      "type": "array",
+      "items": {"type": "string"},
+      "uniqueItems": true
+    },
+
+    "list_or_dict": {
+      "oneOf": [
+        {
+          "type": "object",
+          "patternProperties": {
+            ".+": {
+              "type": ["string", "number", "null"]
+            }
+          },
+          "additionalProperties": false
+        },
+        {"type": "array", "items": {"type": "string"}, "uniqueItems": true}
+      ]
+    },
+
+    "constraints": {
+      "service": {
+        "id": "#/definitions/constraints/service",
+        "anyOf": [
+          {"required": ["build"]},
+          {"required": ["image"]}
+        ],
+        "properties": {
+          "build": {
+            "required": ["context"]
+          }
+        }
+      }
+    }
+  }
+}

compose/config/config_schema_v3.0.json

@@ -0,0 +1,383 @@
+{
+  "$schema": "http://json-schema.org/draft-04/schema#",
+  "id": "config_schema_v3.0.json",
+  "type": "object",
+  "required": ["version"],
+
+  "properties": {
+    "version": {
+      "type": "string"
+    },
+
+    "services": {
+      "id": "#/properties/services",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/service"
+        }
+      },
+      "additionalProperties": false
+    },
+
+    "networks": {
+      "id": "#/properties/networks",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/network"
+        }
+      }
+    },
+
+    "volumes": {
+      "id": "#/properties/volumes",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/volume"
+        }
+      },
+      "additionalProperties": false
+    }
+  },
+
+  "additionalProperties": false,
+
+  "definitions": {
+
+    "service": {
+      "id": "#/definitions/service",
+      "type": "object",
+
+      "properties": {
+        "deploy": {"$ref": "#/definitions/deployment"},
+        "build": {
+          "oneOf": [
+            {"type": "string"},
+            {
+              "type": "object",
+              "properties": {
+                "context": {"type": "string"},
+                "dockerfile": {"type": "string"},
+                "args": {"$ref": "#/definitions/list_or_dict"}
+              },
+              "additionalProperties": false
+            }
+          ]
+        },
+        "cap_add": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "cap_drop": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "cgroup_parent": {"type": "string"},
+        "command": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "container_name": {"type": "string"},
+        "depends_on": {"$ref": "#/definitions/list_of_strings"},
+        "devices": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "dns": {"$ref": "#/definitions/string_or_list"},
+        "dns_search": {"$ref": "#/definitions/string_or_list"},
+        "domainname": {"type": "string"},
+        "entrypoint": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "env_file": {"$ref": "#/definitions/string_or_list"},
+        "environment": {"$ref": "#/definitions/list_or_dict"},
+
+        "expose": {
+          "type": "array",
+          "items": {
+            "type": ["string", "number"],
+            "format": "expose"
+          },
+          "uniqueItems": true
+        },
+
+        "external_links": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "extra_hosts": {"$ref": "#/definitions/list_or_dict"},
+        "healthcheck": {"$ref": "#/definitions/healthcheck"},
+        "hostname": {"type": "string"},
+        "image": {"type": "string"},
+        "ipc": {"type": "string"},
+        "labels": {"$ref": "#/definitions/list_or_dict"},
+        "links": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+
+        "logging": {
+            "type": "object",
+
+            "properties": {
+                "driver": {"type": "string"},
+                "options": {
+                  "type": "object",
+                  "patternProperties": {
+                    "^.+$": {"type": ["string", "number", "null"]}
+                  }
+                }
+            },
+            "additionalProperties": false
+        },
+
+        "mac_address": {"type": "string"},
+        "network_mode": {"type": "string"},
+
+        "networks": {
+          "oneOf": [
+            {"$ref": "#/definitions/list_of_strings"},
+            {
+              "type": "object",
+              "patternProperties": {
+                "^[a-zA-Z0-9._-]+$": {
+                  "oneOf": [
+                    {
+                      "type": "object",
+                      "properties": {
+                        "aliases": {"$ref": "#/definitions/list_of_strings"},
+                        "ipv4_address": {"type": "string"},
+                        "ipv6_address": {"type": "string"}
+                      },
+                      "additionalProperties": false
+                    },
+                    {"type": "null"}
+                  ]
+                }
+              },
+              "additionalProperties": false
+            }
+          ]
+        },
+        "pid": {"type": ["string", "null"]},
+
+        "ports": {
+          "type": "array",
+          "items": {
+            "type": ["string", "number"],
+            "format": "ports"
+          },
+          "uniqueItems": true
+        },
+
+        "privileged": {"type": "boolean"},
+        "read_only": {"type": "boolean"},
+        "restart": {"type": "string"},
+        "security_opt": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "shm_size": {"type": ["number", "string"]},
+        "sysctls": {"$ref": "#/definitions/list_or_dict"},
+        "stdin_open": {"type": "boolean"},
+        "stop_grace_period": {"type": "string", "format": "duration"},
+        "stop_signal": {"type": "string"},
+        "tmpfs": {"$ref": "#/definitions/string_or_list"},
+        "tty": {"type": "boolean"},
+        "ulimits": {
+          "type": "object",
+          "patternProperties": {
+            "^[a-z]+$": {
+              "oneOf": [
+                {"type": "integer"},
+                {
+                  "type":"object",
+                  "properties": {
+                    "hard": {"type": "integer"},
+                    "soft": {"type": "integer"}
+                  },
+                  "required": ["soft", "hard"],
+                  "additionalProperties": false
+                }
+              ]
+            }
+          }
+        },
+        "user": {"type": "string"},
+        "userns_mode": {"type": "string"},
+        "volumes": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "working_dir": {"type": "string"}
+      },
+      "additionalProperties": false
+    },
+
+    "healthcheck": {
+      "id": "#/definitions/healthcheck",
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "disable": {"type": "boolean"},
+        "interval": {"type": "string"},
+        "retries": {"type": "number"},
+        "test": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "timeout": {"type": "string"}
+      }
+    },
+    "deployment": {
+      "id": "#/definitions/deployment",
+      "type": ["object", "null"],
+      "properties": {
+        "mode": {"type": "string"},
+        "replicas": {"type": "integer"},
+        "labels": {"$ref": "#/definitions/list_or_dict"},
+        "update_config": {
+          "type": "object",
+          "properties": {
+            "parallelism": {"type": "integer"},
+            "delay": {"type": "string", "format": "duration"},
+            "failure_action": {"type": "string"},
+            "monitor": {"type": "string", "format": "duration"},
+            "max_failure_ratio": {"type": "number"}
+          },
+          "additionalProperties": false
+        },
+        "resources": {
+          "type": "object",
+          "properties": {
+            "limits": {"$ref": "#/definitions/resource"},
+            "reservations": {"$ref": "#/definitions/resource"}
+          }
+        },
+        "restart_policy": {
+          "type": "object",
+          "properties": {
+            "condition": {"type": "string"},
+            "delay": {"type": "string", "format": "duration"},
+            "max_attempts": {"type": "integer"},
+            "window": {"type": "string", "format": "duration"}
+          },
+          "additionalProperties": false
+        },
+        "placement": {
+          "type": "object",
+          "properties": {
+            "constraints": {"type": "array", "items": {"type": "string"}}
+          },
+          "additionalProperties": false
+        }
+      },
+      "additionalProperties": false
+    },
+
+    "resource": {
+      "id": "#/definitions/resource",
+      "type": "object",
+      "properties": {
+        "cpus": {"type": "string"},
+        "memory": {"type": "string"}
+      },
+      "additionalProperties": false
+    },
+
+    "network": {
+      "id": "#/definitions/network",
+      "type": ["object", "null"],
+      "properties": {
+        "driver": {"type": "string"},
+        "driver_opts": {
+          "type": "object",
+          "patternProperties": {
+            "^.+$": {"type": ["string", "number"]}
+          }
+        },
+        "ipam": {
+          "type": "object",
+          "properties": {
+            "driver": {"type": "string"},
+            "config": {
+              "type": "array",
+              "items": {
+                "type": "object",
+                "properties": {
+                  "subnet": {"type": "string"}
+                },
+                "additionalProperties": false
+              }
+            }
+          },
+          "additionalProperties": false
+        },
+        "external": {
+          "type": ["boolean", "object"],
+          "properties": {
+            "name": {"type": "string"}
+          },
+          "additionalProperties": false
+        },
+        "internal": {"type": "boolean"},
+        "labels": {"$ref": "#/definitions/list_or_dict"}
+      },
+      "additionalProperties": false
+    },
+
+    "volume": {
+      "id": "#/definitions/volume",
+      "type": ["object", "null"],
+      "properties": {
+        "driver": {"type": "string"},
+        "driver_opts": {
+          "type": "object",
+          "patternProperties": {
+            "^.+$": {"type": ["string", "number"]}
+          }
+        },
+        "external": {
+          "type": ["boolean", "object"],
+          "properties": {
+            "name": {"type": "string"}
+          },
+          "additionalProperties": false
+        },
+        "labels": {"$ref": "#/definitions/list_or_dict"}
+      },
+      "additionalProperties": false
+    },
+
+    "string_or_list": {
+      "oneOf": [
+        {"type": "string"},
+        {"$ref": "#/definitions/list_of_strings"}
+      ]
+    },
+
+    "list_of_strings": {
+      "type": "array",
+      "items": {"type": "string"},
+      "uniqueItems": true
+    },
+
+    "list_or_dict": {
+      "oneOf": [
+        {
+          "type": "object",
+          "patternProperties": {
+            ".+": {
+              "type": ["string", "number", "null"]
+            }
+          },
+          "additionalProperties": false
+        },
+        {"type": "array", "items": {"type": "string"}, "uniqueItems": true}
+      ]
+    },
+
+    "constraints": {
+      "service": {
+        "id": "#/definitions/constraints/service",
+        "anyOf": [
+          {"required": ["build"]},
+          {"required": ["image"]}
+        ],
+        "properties": {
+          "build": {
+            "required": ["context"]
+          }
+        }
+      }
+    }
+  }
+}

compose/config/config_schema_v3.1.json

@@ -0,0 +1,428 @@
+{
+  "$schema": "http://json-schema.org/draft-04/schema#",
+  "id": "config_schema_v3.1.json",
+  "type": "object",
+  "required": ["version"],
+
+  "properties": {
+    "version": {
+      "type": "string"
+    },
+
+    "services": {
+      "id": "#/properties/services",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/service"
+        }
+      },
+      "additionalProperties": false
+    },
+
+    "networks": {
+      "id": "#/properties/networks",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/network"
+        }
+      }
+    },
+
+    "volumes": {
+      "id": "#/properties/volumes",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/volume"
+        }
+      },
+      "additionalProperties": false
+    },
+
+    "secrets": {
+      "id": "#/properties/secrets",
+      "type": "object",
+      "patternProperties": {
+        "^[a-zA-Z0-9._-]+$": {
+          "$ref": "#/definitions/secret"
+        }
+      },
+      "additionalProperties": false
+    }
+  },
+
+  "additionalProperties": false,
+
+  "definitions": {
+
+    "service": {
+      "id": "#/definitions/service",
+      "type": "object",
+
+      "properties": {
+        "deploy": {"$ref": "#/definitions/deployment"},
+        "build": {
+          "oneOf": [
+            {"type": "string"},
+            {
+              "type": "object",
+              "properties": {
+                "context": {"type": "string"},
+                "dockerfile": {"type": "string"},
+                "args": {"$ref": "#/definitions/list_or_dict"}
+              },
+              "additionalProperties": false
+            }
+          ]
+        },
+        "cap_add": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "cap_drop": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "cgroup_parent": {"type": "string"},
+        "command": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "container_name": {"type": "string"},
+        "depends_on": {"$ref": "#/definitions/list_of_strings"},
+        "devices": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "dns": {"$ref": "#/definitions/string_or_list"},
+        "dns_search": {"$ref": "#/definitions/string_or_list"},
+        "domainname": {"type": "string"},
+        "entrypoint": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "env_file": {"$ref": "#/definitions/string_or_list"},
+        "environment": {"$ref": "#/definitions/list_or_dict"},
+
+        "expose": {
+          "type": "array",
+          "items": {
+            "type": ["string", "number"],
+            "format": "expose"
+          },
+          "uniqueItems": true
+        },
+
+        "external_links": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "extra_hosts": {"$ref": "#/definitions/list_or_dict"},
+        "healthcheck": {"$ref": "#/definitions/healthcheck"},
+        "hostname": {"type": "string"},
+        "image": {"type": "string"},
+        "ipc": {"type": "string"},
+        "labels": {"$ref": "#/definitions/list_or_dict"},
+        "links": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+
+        "logging": {
+            "type": "object",
+
+            "properties": {
+                "driver": {"type": "string"},
+                "options": {
+                  "type": "object",
+                  "patternProperties": {
+                    "^.+$": {"type": ["string", "number", "null"]}
+                  }
+                }
+            },
+            "additionalProperties": false
+        },
+
+        "mac_address": {"type": "string"},
+        "network_mode": {"type": "string"},
+
+        "networks": {
+          "oneOf": [
+            {"$ref": "#/definitions/list_of_strings"},
+            {
+              "type": "object",
+              "patternProperties": {
+                "^[a-zA-Z0-9._-]+$": {
+                  "oneOf": [
+                    {
+                      "type": "object",
+                      "properties": {
+                        "aliases": {"$ref": "#/definitions/list_of_strings"},
+                        "ipv4_address": {"type": "string"},
+                        "ipv6_address": {"type": "string"}
+                      },
+                      "additionalProperties": false
+                    },
+                    {"type": "null"}
+                  ]
+                }
+              },
+              "additionalProperties": false
+            }
+          ]
+        },
+        "pid": {"type": ["string", "null"]},
+
+        "ports": {
+          "type": "array",
+          "items": {
+            "type": ["string", "number"],
+            "format": "ports"
+          },
+          "uniqueItems": true
+        },
+
+        "privileged": {"type": "boolean"},
+        "read_only": {"type": "boolean"},
+        "restart": {"type": "string"},
+        "security_opt": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "shm_size": {"type": ["number", "string"]},
+        "secrets": {
+          "type": "array",
+          "items": {
+            "oneOf": [
+              {"type": "string"},
+              {
+                "type": "object",
+                "properties": {
+                  "source": {"type": "string"},
+                  "target": {"type": "string"},
+                  "uid": {"type": "string"},
+                  "gid": {"type": "string"},
+                  "mode": {"type": "number"}
+                }
+              }
+            ]
+          }
+        },
+        "sysctls": {"$ref": "#/definitions/list_or_dict"},
+        "stdin_open": {"type": "boolean"},
+        "stop_grace_period": {"type": "string", "format": "duration"},
+        "stop_signal": {"type": "string"},
+        "tmpfs": {"$ref": "#/definitions/string_or_list"},
+        "tty": {"type": "boolean"},
+        "ulimits": {
+          "type": "object",
+          "patternProperties": {
+            "^[a-z]+$": {
+              "oneOf": [
+                {"type": "integer"},
+                {
+                  "type":"object",
+                  "properties": {
+                    "hard": {"type": "integer"},
+                    "soft": {"type": "integer"}
+                  },
+                  "required": ["soft", "hard"],
+                  "additionalProperties": false
+                }
+              ]
+            }
+          }
+        },
+        "user": {"type": "string"},
+        "userns_mode": {"type": "string"},
+        "volumes": {"type": "array", "items": {"type": "string"}, "uniqueItems": true},
+        "working_dir": {"type": "string"}
+      },
+      "additionalProperties": false
+    },
+
+    "healthcheck": {
+      "id": "#/definitions/healthcheck",
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "disable": {"type": "boolean"},
+        "interval": {"type": "string"},
+        "retries": {"type": "number"},
+        "test": {
+          "oneOf": [
+            {"type": "string"},
+            {"type": "array", "items": {"type": "string"}}
+          ]
+        },
+        "timeout": {"type": "string"}
+      }
+    },
+    "deployment": {
+      "id": "#/definitions/deployment",
+      "type": ["object", "null"],
+      "properties": {
+        "mode": {"type": "string"},
+        "replicas": {"type": "integer"},
+        "labels": {"$ref": "#/definitions/list_or_dict"},
+        "update_config": {
+          "type": "object",
+          "properties": {
+            "parallelism": {"type": "integer"},
+            "delay": {"type": "string", "format": "duration"},
+            "failure_action": {"type": "string"},
+            "monitor": {"type": "string", "format": "duration"},
+            "max_failure_ratio": {"type": "number"}
+          },
+          "additionalProperties": false
+        },
+        "resources": {
+          "type": "object",
+          "properties": {
+            "limits": {"$ref": "#/definitions/resource"},
+            "reservations": {"$ref": "#/definitions/resource"}
+          }
+        },
+        "restart_policy": {
+          "type": "object",
+          "properties": {
+            "condition": {"type": "string"},
+            "delay": {"type": "string", "format": "duration"},
+            "max_attempts": {"type": "integer"},
+            "window": {"type": "string", "format": "duration"}
+          },
+          "additionalProperties": false
+        },
+        "placement": {
+          "type": "object",
+          "properties": {
+            "constraints": {"type": "array", "items": {"type": "string"}}
+          },
+          "additionalProperties": false
+        }
+      },
+      "additionalProperties": false
+    },
+
+    "resource": {
+      "id": "#/definitions/resource",
+      "type": "object",
+      "properties": {
+        "cpus": {"type": "string"},
+        "memory": {"type": "string"}
+      },
+      "additionalProperties": false
+    },
+
+    "network": {
+      "id": "#/definitions/network",
+      "type": ["object", "null"],
+      "properties": {
+        "driver": {"type": "string"},
+        "driver_opts": {
+          "type": "object",
+          "patternProperties": {
+            "^.+$": {"type": ["string", "number"]}
+          }
+        },
+        "ipam": {
+          "type": "object",
+          "properties": {
+            "driver": {"type": "string"},
+            "config": {
+              "type": "array",
+              "items": {
+                "type": "object",
+                "properties": {
+                  "subnet": {"type": "string"}
+                },
+                "additionalProperties": false
+              }
+            }
+          },
+          "additionalProperties": false
+        },
+        "external": {
+          "type": ["boolean", "object"],
+          "properties": {
+            "name": {"type": "string"}
+          },
+          "additionalProperties": false
+        },
+        "internal": {"type": "boolean"},
+        "labels": {"$ref": "#/definitions/list_or_dict"}
+      },
+      "additionalProperties": false
+    },
+
+    "volume": {
+      "id": "#/definitions/volume",
+      "type": ["object", "null"],
+      "properties": {
+        "driver": {"type": "string"},
+        "driver_opts": {
+          "type": "object",
+          "patternProperties": {
+            "^.+$": {"type": ["string", "number"]}
+          }
+        },
+        "external": {
+          "type": ["boolean", "object"],
+          "properties": {
+            "name": {"type": "string"}
+          },
+          "additionalProperties": false
+        },
+        "labels": {"$ref": "#/definitions/list_or_dict"}
+      },
+      "additionalProperties": false
+    },
+
+    "secret": {
+      "id": "#/definitions/secret",
+      "type": "object",
+      "properties": {
+        "file": {"type": "string"},
+        "external": {
+          "type": ["boolean", "object"],
+          "properties": {
+            "name": {"type": "string"}
+          }
+        },
+        "labels": {"$ref": "#/definitions/list_or_dict"}
+      },
+      "additionalProperties": false
+    },
+
+    "string_or_list": {
+      "oneOf": [
+        {"type": "string"},
+        {"$ref": "#/definitions/list_of_strings"}
+      ]
+    },
+
+    "list_of_strings": {
+      "type": "array",
+      "items": {"type": "string"},
+      "uniqueItems": true
+    },
+
+    "list_or_dict": {
+      "oneOf": [
+        {
+          "type": "object",
+          "patternProperties": {
+            ".+": {
+              "type": ["string", "number", "null"]
+            }
+          },
+          "additionalProperties": false
+        },
+        {"type": "array", "items": {"type": "string"}, "uniqueItems": true}
+      ]
+    },
+
+    "constraints": {
+      "service": {
+        "id": "#/definitions/constraints/service",
+        "anyOf": [
+          {"required": ["build"]},
+          {"required": ["image"]}
+        ],
+        "properties": {
+          "build": {
+            "required": ["context"]
+          }
+        }
+      }
+    }
+  }
+}

compose/config/environment.py

@@ -0,0 +1,120 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import codecs
+import contextlib
+import logging
+import os
+
+import six
+
+from ..const import IS_WINDOWS_PLATFORM
+from .errors import ConfigurationError
+
+log = logging.getLogger(__name__)
+
+
+def split_env(env):
+    if isinstance(env, six.binary_type):
+        env = env.decode('utf-8', 'replace')
+    if '=' in env:
+        return env.split('=', 1)
+    else:
+        return env, None
+
+
+def env_vars_from_file(filename):
+    """
+    Read in a line delimited file of environment variables.
+    """
+    if not os.path.exists(filename):
+        raise ConfigurationError("Couldn't find env file: %s" % filename)
+    elif not os.path.isfile(filename):
+        raise ConfigurationError("%s is not a file." % (filename))
+    env = {}
+    with contextlib.closing(codecs.open(filename, 'r', 'utf-8')) as fileobj:
+        for line in fileobj:
+            line = line.strip()
+            if line and not line.startswith('#'):
+                k, v = split_env(line)
+                env[k] = v
+    return env
+
+
+class Environment(dict):
+    def __init__(self, *args, **kwargs):
+        super(Environment, self).__init__(*args, **kwargs)
+        self.missing_keys = []
+
+    @classmethod
+    def from_env_file(cls, base_dir):
+        def _initialize():
+            result = cls()
+            if base_dir is None:
+                return result
+            env_file_path = os.path.join(base_dir, '.env')
+            try:
+                return cls(env_vars_from_file(env_file_path))
+            except ConfigurationError:
+                pass
+            return result
+        instance = _initialize()
+        instance.update(os.environ)
+        return instance
+
+    @classmethod
+    def from_command_line(cls, parsed_env_opts):
+        result = cls()
+        for k, v in parsed_env_opts.items():
+            # Values from the command line take priority, unless they're unset
+            # in which case they take the value from the system's environment
+            if v is None and k in os.environ:
+                result[k] = os.environ[k]
+            else:
+                result[k] = v
+        return result
+
+    def __getitem__(self, key):
+        try:
+            return super(Environment, self).__getitem__(key)
+        except KeyError:
+            if IS_WINDOWS_PLATFORM:
+                try:
+                    return super(Environment, self).__getitem__(key.upper())
+                except KeyError:
+                    pass
+            if key not in self.missing_keys:
+                log.warn(
+                    "The {} variable is not set. Defaulting to a blank string."
+                    .format(key)
+                )
+                self.missing_keys.append(key)
+
+            return ""
+
+    def __contains__(self, key):
+        result = super(Environment, self).__contains__(key)
+        if IS_WINDOWS_PLATFORM:
+            return (
+                result or super(Environment, self).__contains__(key.upper())
+            )
+        return result
+
+    def get(self, key, *args, **kwargs):
+        if IS_WINDOWS_PLATFORM:
+            return super(Environment, self).get(
+                key,
+                super(Environment, self).get(key.upper(), *args, **kwargs)
+            )
+        return super(Environment, self).get(key, *args, **kwargs)
+
+    def get_boolean(self, key):
+        # Convert a value to a boolean using "common sense" rules.
+        # Unset, empty, "0" and "false" (i-case) yield False.
+        # All other values yield True.
+        value = self.get(key)
+        if not value:
+            return False
+        if value.lower() in ['0', 'false']:
+            return False
+        return True

compose/config/errors.py

@@ -0,0 +1,46 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+
+VERSION_EXPLANATION = (
+    'You might be seeing this error because you\'re using the wrong Compose file version. '
+    'Either specify a supported version ("2.0", "2.1", "3.0") and place your '
+    'service definitions under the `services` key, or omit the `version` key '
+    'and place your service definitions at the root of the file to use '
+    'version 1.\nFor more on the Compose file format versions, see '
+    'https://docs.docker.com/compose/compose-file/')
+
+
+class ConfigurationError(Exception):
+    def __init__(self, msg):
+        self.msg = msg
+
+    def __str__(self):
+        return self.msg
+
+
+class DependencyError(ConfigurationError):
+    pass
+
+
+class CircularReference(ConfigurationError):
+    def __init__(self, trail):
+        self.trail = trail
+
+    @property
+    def msg(self):
+        lines = [
+            "{} in {}".format(service_name, filename)
+            for (filename, service_name) in self.trail
+        ]
+        return "Circular reference:\n  {}".format("\n  extends ".join(lines))
+
+
+class ComposeFileNotFound(ConfigurationError):
+    def __init__(self, supported_filenames):
+        super(ComposeFileNotFound, self).__init__("""
+        Can't find a suitable configuration file in this directory or any
+        parent. Are you in the right directory?
+
+        Supported filenames: %s
+        """ % ", ".join(supported_filenames))

compose/config/interpolation.py

@@ -0,0 +1,103 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import logging
+from string import Template
+
+import six
+
+from .errors import ConfigurationError
+from compose.const import COMPOSEFILE_V1 as V1
+from compose.const import COMPOSEFILE_V2_0 as V2_0
+
+
+log = logging.getLogger(__name__)
+
+
+class Interpolator(object):
+
+    def __init__(self, templater, mapping):
+        self.templater = templater
+        self.mapping = mapping
+
+    def interpolate(self, string):
+        try:
+            return self.templater(string).substitute(self.mapping)
+        except ValueError:
+            raise InvalidInterpolation(string)
+
+
+def interpolate_environment_variables(version, config, section, environment):
+    if version in (V2_0, V1):
+        interpolator = Interpolator(Template, environment)
+    else:
+        interpolator = Interpolator(TemplateWithDefaults, environment)
+
+    def process_item(name, config_dict):
+        return dict(
+            (key, interpolate_value(name, key, val, section, interpolator))
+            for key, val in (config_dict or {}).items()
+        )
+
+    return dict(
+        (name, process_item(name, config_dict or {}))
+        for name, config_dict in config.items()
+    )
+
+
+def interpolate_value(name, config_key, value, section, interpolator):
+    try:
+        return recursive_interpolate(value, interpolator)
+    except InvalidInterpolation as e:
+        raise ConfigurationError(
+            'Invalid interpolation format for "{config_key}" option '
+            'in {section} "{name}": "{string}"'.format(
+                config_key=config_key,
+                name=name,
+                section=section,
+                string=e.string))
+
+
+def recursive_interpolate(obj, interpolator):
+    if isinstance(obj, six.string_types):
+        return interpolator.interpolate(obj)
+    if isinstance(obj, dict):
+        return dict(
+            (key, recursive_interpolate(val, interpolator))
+            for (key, val) in obj.items()
+        )
+    if isinstance(obj, list):
+        return [recursive_interpolate(val, interpolator) for val in obj]
+    return obj
+
+
+class TemplateWithDefaults(Template):
+    idpattern = r'[_a-z][_a-z0-9]*(?::?-[^}]+)?'
+
+    # Modified from python2.7/string.py
+    def substitute(self, mapping):
+        # Helper function for .sub()
+        def convert(mo):
+            # Check the most common path first.
+            named = mo.group('named') or mo.group('braced')
+            if named is not None:
+                if ':-' in named:
+                    var, _, default = named.partition(':-')
+                    return mapping.get(var) or default
+                if '-' in named:
+                    var, _, default = named.partition('-')
+                    return mapping.get(var, default)
+                val = mapping[named]
+                return '%s' % (val,)
+            if mo.group('escaped') is not None:
+                return self.delimiter
+            if mo.group('invalid') is not None:
+                self._invalid(mo)
+            raise ValueError('Unrecognized named group in pattern',
+                             self.pattern)
+        return self.pattern.sub(convert, self.template)
+
+
+class InvalidInterpolation(Exception):
+    def __init__(self, string):
+        self.string = string

compose/config/serialize.py

@@ -0,0 +1,108 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import six
+import yaml
+
+from compose.config import types
+from compose.config.config import V1
+from compose.config.config import V2_1
+
+
+def serialize_config_type(dumper, data):
+    representer = dumper.represent_str if six.PY3 else dumper.represent_unicode
+    return representer(data.repr())
+
+
+yaml.SafeDumper.add_representer(types.VolumeFromSpec, serialize_config_type)
+yaml.SafeDumper.add_representer(types.VolumeSpec, serialize_config_type)
+
+
+def denormalize_config(config):
+    denormalized_services = [
+        denormalize_service_dict(service_dict, config.version)
+        for service_dict in config.services
+    ]
+    services = {
+        service_dict.pop('name'): service_dict
+        for service_dict in denormalized_services
+    }
+    networks = config.networks.copy()
+    for net_name, net_conf in networks.items():
+        if 'external_name' in net_conf:
+            del net_conf['external_name']
+
+    volumes = config.volumes.copy()
+    for vol_name, vol_conf in volumes.items():
+        if 'external_name' in vol_conf:
+            del vol_conf['external_name']
+
+    version = config.version
+    if version == V1:
+        version = V2_1
+
+    return {
+        'version': version,
+        'services': services,
+        'networks': networks,
+        'volumes': volumes,
+    }
+
+
+def serialize_config(config):
+    return yaml.safe_dump(
+        denormalize_config(config),
+        default_flow_style=False,
+        indent=2,
+        width=80)
+
+
+def serialize_ns_time_value(value):
+    result = (value, 'ns')
+    table = [
+        (1000., 'us'),
+        (1000., 'ms'),
+        (1000., 's'),
+        (60., 'm'),
+        (60., 'h')
+    ]
+    for stage in table:
+        tmp = value / stage[0]
+        if tmp == int(value / stage[0]):
+            value = tmp
+            result = (int(value), stage[1])
+        else:
+            break
+    return '{0}{1}'.format(*result)
+
+
+def denormalize_service_dict(service_dict, version):
+    service_dict = service_dict.copy()
+
+    if 'restart' in service_dict:
+        service_dict['restart'] = types.serialize_restart_spec(
+            service_dict['restart']
+        )
+
+    if version == V1 and 'network_mode' not in service_dict:
+        service_dict['network_mode'] = 'bridge'
+
+    if 'depends_on' in service_dict and version != V2_1:
+        service_dict['depends_on'] = sorted([
+            svc for svc in service_dict['depends_on'].keys()
+        ])
+
+    if 'healthcheck' in service_dict:
+        if 'interval' in service_dict['healthcheck']:
+            service_dict['healthcheck']['interval'] = serialize_ns_time_value(
+                service_dict['healthcheck']['interval']
+            )
+        if 'timeout' in service_dict['healthcheck']:
+            service_dict['healthcheck']['timeout'] = serialize_ns_time_value(
+                service_dict['healthcheck']['timeout']
+            )
+
+    if 'secrets' in service_dict:
+        service_dict['secrets'] = map(lambda s: s.repr(), service_dict['secrets'])
+
+    return service_dict

compose/config/sort_services.py

@@ -0,0 +1,72 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+from compose.config.errors import DependencyError
+
+
+def get_service_name_from_network_mode(network_mode):
+    return get_source_name_from_network_mode(network_mode, 'service')
+
+
+def get_container_name_from_network_mode(network_mode):
+    return get_source_name_from_network_mode(network_mode, 'container')
+
+
+def get_source_name_from_network_mode(network_mode, source_type):
+    if not network_mode:
+        return
+
+    if not network_mode.startswith(source_type+':'):
+        return
+
+    _, net_name = network_mode.split(':', 1)
+    return net_name
+
+
+def get_service_names(links):
+    return [link.split(':')[0] for link in links]
+
+
+def get_service_names_from_volumes_from(volumes_from):
+    return [volume_from.source for volume_from in volumes_from]
+
+
+def get_service_dependents(service_dict, services):
+    name = service_dict['name']
+    return [
+        service for service in services
+        if (name in get_service_names(service.get('links', [])) or
+            name in get_service_names_from_volumes_from(service.get('volumes_from', [])) or
+            name == get_service_name_from_network_mode(service.get('network_mode')) or
+            name in service.get('depends_on', []))
+    ]
+
+
+def sort_service_dicts(services):
+    # Topological sort (Cormen/Tarjan algorithm).
+    unmarked = services[:]
+    temporary_marked = set()
+    sorted_services = []
+
+    def visit(n):
+        if n['name'] in temporary_marked:
+            if n['name'] in get_service_names(n.get('links', [])):
+                raise DependencyError('A service can not link to itself: %s' % n['name'])
+            if n['name'] in n.get('volumes_from', []):
+                raise DependencyError('A service can not mount itself as volume: %s' % n['name'])
+            if n['name'] in n.get('depends_on', []):
+                raise DependencyError('A service can not depend on itself: %s' % n['name'])
+            raise DependencyError('Circular dependency between %s' % ' and '.join(temporary_marked))
+
+        if n in unmarked:
+            temporary_marked.add(n['name'])
+            for m in get_service_dependents(n, services):
+                visit(m)
+            temporary_marked.remove(n['name'])
+            unmarked.remove(n)
+            sorted_services.insert(0, n)
+
+    while unmarked:
+        visit(unmarked[-1])
+
+    return sorted_services

compose/config/types.py

@@ -0,0 +1,260 @@
+"""
+Types for objects parsed from the configuration.
+"""
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import os
+import re
+from collections import namedtuple
+
+import six
+
+from ..const import COMPOSEFILE_V1 as V1
+from .errors import ConfigurationError
+from compose.const import IS_WINDOWS_PLATFORM
+from compose.utils import splitdrive
+
+win32_root_path_pattern = re.compile(r'^[A-Za-z]\:\\.*')
+
+
+class VolumeFromSpec(namedtuple('_VolumeFromSpec', 'source mode type')):
+
+    # TODO: drop service_names arg when v1 is removed
+    @classmethod
+    def parse(cls, volume_from_config, service_names, version):
+        func = cls.parse_v1 if version == V1 else cls.parse_v2
+        return func(service_names, volume_from_config)
+
+    @classmethod
+    def parse_v1(cls, service_names, volume_from_config):
+        parts = volume_from_config.split(':')
+        if len(parts) > 2:
+            raise ConfigurationError(
+                "volume_from {} has incorrect format, should be "
+                "service[:mode]".format(volume_from_config))
+
+        if len(parts) == 1:
+            source = parts[0]
+            mode = 'rw'
+        else:
+            source, mode = parts
+
+        type = 'service' if source in service_names else 'container'
+        return cls(source, mode, type)
+
+    @classmethod
+    def parse_v2(cls, service_names, volume_from_config):
+        parts = volume_from_config.split(':')
+        if len(parts) > 3:
+            raise ConfigurationError(
+                "volume_from {} has incorrect format, should be one of "
+                "'<service name>[:<mode>]' or "
+                "'container:<container name>[:<mode>]'".format(volume_from_config))
+
+        if len(parts) == 1:
+            source = parts[0]
+            return cls(source, 'rw', 'service')
+
+        if len(parts) == 2:
+            if parts[0] == 'container':
+                type, source = parts
+                return cls(source, 'rw', type)
+
+            source, mode = parts
+            return cls(source, mode, 'service')
+
+        if len(parts) == 3:
+            type, source, mode = parts
+            if type not in ('service', 'container'):
+                raise ConfigurationError(
+                    "Unknown volumes_from type '{}' in '{}'".format(
+                        type,
+                        volume_from_config))
+
+        return cls(source, mode, type)
+
+    def repr(self):
+        return '{v.type}:{v.source}:{v.mode}'.format(v=self)
+
+
+def parse_restart_spec(restart_config):
+    if not restart_config:
+        return None
+    parts = restart_config.split(':')
+    if len(parts) > 2:
+        raise ConfigurationError(
+            "Restart %s has incorrect format, should be "
+            "mode[:max_retry]" % restart_config)
+    if len(parts) == 2:
+        name, max_retry_count = parts
+    else:
+        name, = parts
+        max_retry_count = 0
+
+    return {'Name': name, 'MaximumRetryCount': int(max_retry_count)}
+
+
+def serialize_restart_spec(restart_spec):
+    if not restart_spec:
+        return ''
+    parts = [restart_spec['Name']]
+    if restart_spec['MaximumRetryCount']:
+        parts.append(six.text_type(restart_spec['MaximumRetryCount']))
+    return ':'.join(parts)
+
+
+def parse_extra_hosts(extra_hosts_config):
+    if not extra_hosts_config:
+        return {}
+
+    if isinstance(extra_hosts_config, dict):
+        return dict(extra_hosts_config)
+
+    if isinstance(extra_hosts_config, list):
+        extra_hosts_dict = {}
+        for extra_hosts_line in extra_hosts_config:
+            # TODO: validate string contains ':' ?
+            host, ip = extra_hosts_line.split(':', 1)
+            extra_hosts_dict[host.strip()] = ip.strip()
+        return extra_hosts_dict
+
+
+def normalize_path_for_engine(path):
+    """Windows paths, c:\my\path\shiny, need to be changed to be compatible with
+    the Engine. Volume paths are expected to be linux style /c/my/path/shiny/
+    """
+    drive, tail = splitdrive(path)
+
+    if drive:
+        path = '/' + drive.lower().rstrip(':') + tail
+
+    return path.replace('\\', '/')
+
+
+class VolumeSpec(namedtuple('_VolumeSpec', 'external internal mode')):
+
+    @classmethod
+    def _parse_unix(cls, volume_config):
+        parts = volume_config.split(':')
+
+        if len(parts) > 3:
+            raise ConfigurationError(
+                "Volume %s has incorrect format, should be "
+                "external:internal[:mode]" % volume_config)
+
+        if len(parts) == 1:
+            external = None
+            internal = os.path.normpath(parts[0])
+        else:
+            external = os.path.normpath(parts[0])
+            internal = os.path.normpath(parts[1])
+
+        mode = 'rw'
+        if len(parts) == 3:
+            mode = parts[2]
+
+        return cls(external, internal, mode)
+
+    @classmethod
+    def _parse_win32(cls, volume_config, normalize):
+        # relative paths in windows expand to include the drive, eg C:\
+        # so we join the first 2 parts back together to count as one
+        mode = 'rw'
+
+        def separate_next_section(volume_config):
+            drive, tail = splitdrive(volume_config)
+            parts = tail.split(':', 1)
+            if drive:
+                parts[0] = drive + parts[0]
+            return parts
+
+        parts = separate_next_section(volume_config)
+        if len(parts) == 1:
+            internal = parts[0]
+            external = None
+        else:
+            external = parts[0]
+            parts = separate_next_section(parts[1])
+            external = os.path.normpath(external)
+            internal = parts[0]
+            if len(parts) > 1:
+                if ':' in parts[1]:
+                    raise ConfigurationError(
+                        "Volume %s has incorrect format, should be "
+                        "external:internal[:mode]" % volume_config
+                    )
+                mode = parts[1]
+
+        if normalize:
+            external = normalize_path_for_engine(external) if external else None
+
+        return cls(external, internal, mode)
+
+    @classmethod
+    def parse(cls, volume_config, normalize=False):
+        """Parse a volume_config path and split it into external:internal[:mode]
+        parts to be returned as a valid VolumeSpec.
+        """
+        if IS_WINDOWS_PLATFORM:
+            return cls._parse_win32(volume_config, normalize)
+        else:
+            return cls._parse_unix(volume_config)
+
+    def repr(self):
+        external = self.external + ':' if self.external else ''
+        return '{ext}{v.internal}:{v.mode}'.format(ext=external, v=self)
+
+    @property
+    def is_named_volume(self):
+        res = self.external and not self.external.startswith(('.', '/', '~'))
+        if not IS_WINDOWS_PLATFORM:
+            return res
+
+        return (
+            res and not self.external.startswith('\\') and
+            not win32_root_path_pattern.match(self.external)
+        )
+
+
+class ServiceLink(namedtuple('_ServiceLink', 'target alias')):
+
+    @classmethod
+    def parse(cls, link_spec):
+        target, _, alias = link_spec.partition(':')
+        if not alias:
+            alias = target
+        return cls(target, alias)
+
+    def repr(self):
+        if self.target == self.alias:
+            return self.target
+        return '{s.target}:{s.alias}'.format(s=self)
+
+    @property
+    def merge_field(self):
+        return self.alias
+
+
+class ServiceSecret(namedtuple('_ServiceSecret', 'source target uid gid mode')):
+
+    @classmethod
+    def parse(cls, spec):
+        if isinstance(spec, six.string_types):
+            return cls(spec, None, None, None, None)
+        return cls(
+            spec.get('source'),
+            spec.get('target'),
+            spec.get('uid'),
+            spec.get('gid'),
+            spec.get('mode'),
+        )
+
+    @property
+    def merge_field(self):
+        return self.source
+
+    def repr(self):
+        return dict(
+            [(k, v) for k, v in self._asdict().items() if v is not None]
+        )

compose/config/validation.py

@@ -0,0 +1,423 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import json
+import logging
+import os
+import re
+import sys
+
+import six
+from docker.utils.ports import split_port
+from jsonschema import Draft4Validator
+from jsonschema import FormatChecker
+from jsonschema import RefResolver
+from jsonschema import ValidationError
+
+from ..const import COMPOSEFILE_V1 as V1
+from .errors import ConfigurationError
+from .errors import VERSION_EXPLANATION
+from .sort_services import get_service_name_from_network_mode
+
+
+log = logging.getLogger(__name__)
+
+
+DOCKER_CONFIG_HINTS = {
+    'cpu_share': 'cpu_shares',
+    'add_host': 'extra_hosts',
+    'hosts': 'extra_hosts',
+    'extra_host': 'extra_hosts',
+    'device': 'devices',
+    'link': 'links',
+    'memory_swap': 'memswap_limit',
+    'port': 'ports',
+    'privilege': 'privileged',
+    'priviliged': 'privileged',
+    'privilige': 'privileged',
+    'volume': 'volumes',
+    'workdir': 'working_dir',
+}
+
+
+VALID_NAME_CHARS = '[a-zA-Z0-9\._\-]'
+VALID_EXPOSE_FORMAT = r'^\d+(\-\d+)?(\/[a-zA-Z]+)?$'
+
+
+@FormatChecker.cls_checks(format="ports", raises=ValidationError)
+def format_ports(instance):
+    try:
+        split_port(instance)
+    except ValueError as e:
+        raise ValidationError(six.text_type(e))
+    return True
+
+
+@FormatChecker.cls_checks(format="expose", raises=ValidationError)
+def format_expose(instance):
+    if isinstance(instance, six.string_types):
+        if not re.match(VALID_EXPOSE_FORMAT, instance):
+            raise ValidationError(
+                "should be of the format 'PORT[/PROTOCOL]'")
+
+    return True
+
+
+def match_named_volumes(service_dict, project_volumes):
+    service_volumes = service_dict.get('volumes', [])
+    for volume_spec in service_volumes:
+        if volume_spec.is_named_volume and volume_spec.external not in project_volumes:
+            raise ConfigurationError(
+                'Named volume "{0}" is used in service "{1}" but no'
+                ' declaration was found in the volumes section.'.format(
+                    volume_spec.repr(), service_dict.get('name')
+                )
+            )
+
+
+def python_type_to_yaml_type(type_):
+    type_name = type(type_).__name__
+    return {
+        'dict': 'mapping',
+        'list': 'array',
+        'int': 'number',
+        'float': 'number',
+        'bool': 'boolean',
+        'unicode': 'string',
+        'str': 'string',
+        'bytes': 'string',
+    }.get(type_name, type_name)
+
+
+def validate_config_section(filename, config, section):
+    """Validate the structure of a configuration section. This must be done
+    before interpolation so it's separate from schema validation.
+    """
+    if not isinstance(config, dict):
+        raise ConfigurationError(
+            "In file '{filename}', {section} must be a mapping, not "
+            "{type}.".format(
+                filename=filename,
+                section=section,
+                type=anglicize_json_type(python_type_to_yaml_type(config))))
+
+    for key, value in config.items():
+        if not isinstance(key, six.string_types):
+            raise ConfigurationError(
+                "In file '{filename}', the {section} name {name} must be a "
+                "quoted string, i.e. '{name}'.".format(
+                    filename=filename,
+                    section=section,
+                    name=key))
+
+        if not isinstance(value, (dict, type(None))):
+            raise ConfigurationError(
+                "In file '{filename}', {section} '{name}' must be a mapping not "
+                "{type}.".format(
+                    filename=filename,
+                    section=section,
+                    name=key,
+                    type=anglicize_json_type(python_type_to_yaml_type(value))))
+
+
+def validate_top_level_object(config_file):
+    if not isinstance(config_file.config, dict):
+        raise ConfigurationError(
+            "Top level object in '{}' needs to be an object not '{}'.".format(
+                config_file.filename,
+                type(config_file.config)))
+
+
+def validate_ulimits(service_config):
+    ulimit_config = service_config.config.get('ulimits', {})
+    for limit_name, soft_hard_values in six.iteritems(ulimit_config):
+        if isinstance(soft_hard_values, dict):
+            if not soft_hard_values['soft'] <= soft_hard_values['hard']:
+                raise ConfigurationError(
+                    "Service '{s.name}' has invalid ulimit '{ulimit}'. "
+                    "'soft' value can not be greater than 'hard' value ".format(
+                        s=service_config,
+                        ulimit=ulimit_config))
+
+
+def validate_extends_file_path(service_name, extends_options, filename):
+    """
+    The service to be extended must either be defined in the config key 'file',
+    or within 'filename'.
+    """
+    error_prefix = "Invalid 'extends' configuration for %s:" % service_name
+
+    if 'file' not in extends_options and filename is None:
+        raise ConfigurationError(
+            "%s you need to specify a 'file', e.g. 'file: something.yml'" % error_prefix
+        )
+
+
+def validate_network_mode(service_config, service_names):
+    network_mode = service_config.config.get('network_mode')
+    if not network_mode:
+        return
+
+    if 'networks' in service_config.config:
+        raise ConfigurationError("'network_mode' and 'networks' cannot be combined")
+
+    dependency = get_service_name_from_network_mode(network_mode)
+    if not dependency:
+        return
+
+    if dependency not in service_names:
+        raise ConfigurationError(
+            "Service '{s.name}' uses the network stack of service '{dep}' which "
+            "is undefined.".format(s=service_config, dep=dependency))
+
+
+def validate_links(service_config, service_names):
+    for link in service_config.config.get('links', []):
+        if link.split(':')[0] not in service_names:
+            raise ConfigurationError(
+                "Service '{s.name}' has a link to service '{link}' which is "
+                "undefined.".format(s=service_config, link=link))
+
+
+def validate_depends_on(service_config, service_names):
+    deps = service_config.config.get('depends_on', {})
+    for dependency in deps.keys():
+        if dependency not in service_names:
+            raise ConfigurationError(
+                "Service '{s.name}' depends on service '{dep}' which is "
+                "undefined.".format(s=service_config, dep=dependency)
+            )
+
+
+def get_unsupported_config_msg(path, error_key):
+    msg = "Unsupported config option for {}: '{}'".format(path_string(path), error_key)
+    if error_key in DOCKER_CONFIG_HINTS:
+        msg += " (did you mean '{}'?)".format(DOCKER_CONFIG_HINTS[error_key])
+    return msg
+
+
+def anglicize_json_type(json_type):
+    if json_type.startswith(('a', 'e', 'i', 'o', 'u')):
+        return 'an ' + json_type
+    return 'a ' + json_type
+
+
+def is_service_dict_schema(schema_id):
+    return schema_id in ('config_schema_v1.json', '#/properties/services')
+
+
+def handle_error_for_schema_with_id(error, path):
+    schema_id = error.schema['id']
+
+    if is_service_dict_schema(schema_id) and error.validator == 'additionalProperties':
+        return "Invalid service name '{}' - only {} characters are allowed".format(
+            # The service_name is the key to the json object
+            list(error.instance)[0],
+            VALID_NAME_CHARS)
+
+    if error.validator == 'additionalProperties':
+        if schema_id == '#/definitions/service':
+            invalid_config_key = parse_key_from_error_msg(error)
+            return get_unsupported_config_msg(path, invalid_config_key)
+
+        if not error.path:
+            return '{}\n\n{}'.format(error.message, VERSION_EXPLANATION)
+
+
+def handle_generic_error(error, path):
+    msg_format = None
+    error_msg = error.message
+
+    if error.validator == 'oneOf':
+        msg_format = "{path} {msg}"
+        config_key, error_msg = _parse_oneof_validator(error)
+        if config_key:
+            path.append(config_key)
+
+    elif error.validator == 'type':
+        msg_format = "{path} contains an invalid type, it should be {msg}"
+        error_msg = _parse_valid_types_from_validator(error.validator_value)
+
+    elif error.validator == 'required':
+        error_msg = ", ".join(error.validator_value)
+        msg_format = "{path} is invalid, {msg} is required."
+
+    elif error.validator == 'dependencies':
+        config_key = list(error.validator_value.keys())[0]
+        required_keys = ",".join(error.validator_value[config_key])
+
+        msg_format = "{path} is invalid: {msg}"
+        path.append(config_key)
+        error_msg = "when defining '{}' you must set '{}' as well".format(
+            config_key,
+            required_keys)
+
+    elif error.cause:
+        error_msg = six.text_type(error.cause)
+        msg_format = "{path} is invalid: {msg}"
+
+    elif error.path:
+        msg_format = "{path} value {msg}"
+
+    if msg_format:
+        return msg_format.format(path=path_string(path), msg=error_msg)
+
+    return error.message
+
+
+def parse_key_from_error_msg(error):
+    return error.message.split("'")[1]
+
+
+def path_string(path):
+    return ".".join(c for c in path if isinstance(c, six.string_types))
+
+
+def _parse_valid_types_from_validator(validator):
+    """A validator value can be either an array of valid types or a string of
+    a valid type. Parse the valid types and prefix with the correct article.
+    """
+    if not isinstance(validator, list):
+        return anglicize_json_type(validator)
+
+    if len(validator) == 1:
+        return anglicize_json_type(validator[0])
+
+    return "{}, or {}".format(
+        ", ".join([anglicize_json_type(validator[0])] + validator[1:-1]),
+        anglicize_json_type(validator[-1]))
+
+
+def _parse_oneof_validator(error):
+    """oneOf has multiple schemas, so we need to reason about which schema, sub
+    schema or constraint the validation is failing on.
+    Inspecting the context value of a ValidationError gives us information about
+    which sub schema failed and which kind of error it is.
+    """
+    types = []
+    for context in error.context:
+
+        if context.validator == 'oneOf':
+            _, error_msg = _parse_oneof_validator(context)
+            return path_string(context.path), error_msg
+
+        if context.validator == 'required':
+            return (None, context.message)
+
+        if context.validator == 'additionalProperties':
+            invalid_config_key = parse_key_from_error_msg(context)
+            return (None, "contains unsupported option: '{}'".format(invalid_config_key))
+
+        if context.path:
+            return (
+                path_string(context.path),
+                "contains {}, which is an invalid type, it should be {}".format(
+                    json.dumps(context.instance),
+                    _parse_valid_types_from_validator(context.validator_value)),
+            )
+
+        if context.validator == 'uniqueItems':
+            return (
+                None,
+                "contains non unique items, please remove duplicates from {}".format(
+                    context.instance),
+            )
+
+        if context.validator == 'type':
+            types.append(context.validator_value)
+
+    valid_types = _parse_valid_types_from_validator(types)
+    return (None, "contains an invalid type, it should be {}".format(valid_types))
+
+
+def process_service_constraint_errors(error, service_name, version):
+    if version == V1:
+        if 'image' in error.instance and 'build' in error.instance:
+            return (
+                "Service {} has both an image and build path specified. "
+                "A service can either be built to image or use an existing "
+                "image, not both.".format(service_name))
+
+        if 'image' in error.instance and 'dockerfile' in error.instance:
+            return (
+                "Service {} has both an image and alternate Dockerfile. "
+                "A service can either be built to image or use an existing "
+                "image, not both.".format(service_name))
+
+    if 'image' not in error.instance and 'build' not in error.instance:
+        return (
+            "Service {} has neither an image nor a build context specified. "
+            "At least one must be provided.".format(service_name))
+
+
+def process_config_schema_errors(error):
+    path = list(error.path)
+
+    if 'id' in error.schema:
+        error_msg = handle_error_for_schema_with_id(error, path)
+        if error_msg:
+            return error_msg
+
+    return handle_generic_error(error, path)
+
+
+def validate_against_config_schema(config_file):
+    schema = load_jsonschema(config_file.version)
+    format_checker = FormatChecker(["ports", "expose"])
+    validator = Draft4Validator(
+        schema,
+        resolver=RefResolver(get_resolver_path(), schema),
+        format_checker=format_checker)
+    handle_errors(
+        validator.iter_errors(config_file.config),
+        process_config_schema_errors,
+        config_file.filename)
+
+
+def validate_service_constraints(config, service_name, version):
+    def handler(errors):
+        return process_service_constraint_errors(errors, service_name, version)
+
+    schema = load_jsonschema(version)
+    validator = Draft4Validator(schema['definitions']['constraints']['service'])
+    handle_errors(validator.iter_errors(config), handler, None)
+
+
+def get_schema_path():
+    return os.path.dirname(os.path.abspath(__file__))
+
+
+def load_jsonschema(version):
+    filename = os.path.join(
+        get_schema_path(),
+        "config_schema_v{0}.json".format(version))
+
+    with open(filename, "r") as fh:
+        return json.load(fh)
+
+
+def get_resolver_path():
+    schema_path = get_schema_path()
+    if sys.platform == "win32":
+        scheme = "///"
+        # TODO: why is this necessary?
+        schema_path = schema_path.replace('\\', '/')
+    else:
+        scheme = "//"
+    return "file:{}{}/".format(scheme, schema_path)
+
+
+def handle_errors(errors, format_error_func, filename):
+    """jsonschema returns an error tree full of information to explain what has
+    gone wrong. Process each error and pull out relevant information and re-write
+    helpful error messages that are relevant.
+    """
+    errors = list(sorted(errors, key=str))
+    if not errors:
+        return
+
+    error_msg = '\n'.join(format_error_func(error) for error in errors)
+    raise ConfigurationError(
+        "The Compose file{file_msg} is invalid because:\n{error_msg}".format(
+            file_msg=" '{}'".format(filename) if filename else "",
+            error_msg=error_msg))

compose/const.py

@@ -1,8 +1,41 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import sys
 
 DEFAULT_TIMEOUT = 10
+HTTP_TIMEOUT = 60
+IMAGE_EVENTS = ['delete', 'import', 'load', 'pull', 'push', 'save', 'tag', 'untag']
+IS_WINDOWS_PLATFORM = (sys.platform == "win32")
 LABEL_CONTAINER_NUMBER = 'com.docker.compose.container-number'
 LABEL_ONE_OFF = 'com.docker.compose.oneoff'
 LABEL_PROJECT = 'com.docker.compose.project'
 LABEL_SERVICE = 'com.docker.compose.service'
+LABEL_NETWORK = 'com.docker.compose.network'
 LABEL_VERSION = 'com.docker.compose.version'
+LABEL_VOLUME = 'com.docker.compose.volume'
 LABEL_CONFIG_HASH = 'com.docker.compose.config-hash'
+
+SECRETS_PATH = '/run/secrets'
+
+COMPOSEFILE_V1 = '1'
+COMPOSEFILE_V2_0 = '2.0'
+COMPOSEFILE_V2_1 = '2.1'
+COMPOSEFILE_V3_0 = '3.0'
+COMPOSEFILE_V3_1 = '3.1'
+
+API_VERSIONS = {
+    COMPOSEFILE_V1: '1.21',
+    COMPOSEFILE_V2_0: '1.22',
+    COMPOSEFILE_V2_1: '1.24',
+    COMPOSEFILE_V3_0: '1.25',
+    COMPOSEFILE_V3_1: '1.25',
+}
+
+API_VERSION_TO_ENGINE_VERSION = {
+    API_VERSIONS[COMPOSEFILE_V1]: '1.9.0',
+    API_VERSIONS[COMPOSEFILE_V2_0]: '1.10.0',
+    API_VERSIONS[COMPOSEFILE_V2_1]: '1.12.0',
+    API_VERSIONS[COMPOSEFILE_V3_0]: '1.13.0',
+    API_VERSIONS[COMPOSEFILE_V3_1]: '1.13.0',
+}

compose/container.py

@@ -1,10 +1,13 @@
-from __future__ import unicode_literals
 from __future__ import absolute_import
+from __future__ import unicode_literals
 
-import six
 from functools import reduce
 
-from .const import LABEL_CONTAINER_NUMBER, LABEL_SERVICE
+import six
+
+from .const import LABEL_CONTAINER_NUMBER
+from .const import LABEL_PROJECT
+from .const import LABEL_SERVICE
 
 
 class Container(object):
@@ -16,6 +19,7 @@ class Container(object):
         self.client = client
         self.dictionary = dictionary
         self.has_been_inspected = has_been_inspected
+        self.log_stream = None
 
     @classmethod
     def from_ps(cls, client, dictionary, **kwargs):
@@ -35,7 +39,7 @@ class Container(object):
 
     @classmethod
     def from_id(cls, client, id):
-        return cls(client, client.inspect_container(id))
+        return cls(client, client.inspect_container(id), has_been_inspected=True)
 
     @classmethod
     def create(cls, client, **options):
@@ -56,7 +60,7 @@ class Container(object):
 
     @property
     def short_id(self):
-        return self.id[:10]
+        return self.id[:12]
 
     @property
     def name(self):
@@ -68,7 +72,12 @@ class Container(object):
 
     @property
     def name_without_project(self):
-        return '{0}_{1}'.format(self.service, self.number)
+        project = self.labels.get(LABEL_PROJECT)
+
+        if self.name.startswith('{0}_{1}'.format(project, self.service)):
+            return '{0}_{1}'.format(self.service, self.number)
+        else:
+            return self.name
 
     @property
     def number(self):
@@ -98,12 +107,20 @@ class Container(object):
     def labels(self):
         return self.get('Config.Labels') or {}
 
+    @property
+    def stop_signal(self):
+        return self.get('Config.StopSignal')
+
     @property
     def log_config(self):
         return self.get('HostConfig.LogConfig') or None
 
     @property
     def human_readable_state(self):
+        if self.is_paused:
+            return 'Paused'
+        if self.is_restarting:
+            return 'Restarting'
         if self.is_running:
             return 'Ghost' if self.get('State.Ghost') else 'Up'
         else:
@@ -117,12 +134,44 @@ class Container(object):
 
     @property
     def environment(self):
-        return dict(var.split("=", 1) for var in self.get('Config.Env') or [])
+        def parse_env(var):
+            if '=' in var:
+                return var.split("=", 1)
+            return var, None
+        return dict(parse_env(var) for var in self.get('Config.Env') or [])
+
+    @property
+    def exit_code(self):
+        return self.get('State.ExitCode')
 
     @property
     def is_running(self):
         return self.get('State.Running')
 
+    @property
+    def is_restarting(self):
+        return self.get('State.Restarting')
+
+    @property
+    def is_paused(self):
+        return self.get('State.Paused')
+
+    @property
+    def log_driver(self):
+        return self.get('HostConfig.LogConfig.Type')
+
+    @property
+    def has_api_logs(self):
+        log_type = self.log_driver
+        return not log_type or log_type in ('json-file', 'journald')
+
+    def attach_log_stream(self):
+        """A log stream can only be attached if the container uses a json-file
+        log driver.
+        """
+        if self.has_api_logs:
+            self.log_stream = self.attach(stdout=True, stderr=True, stream=True)
+
     def get(self, key):
         """Return a value from the container or None if the value is not set.
 
@@ -140,12 +189,24 @@ class Container(object):
         port = self.ports.get("%s/%s" % (port, protocol))
         return "{HostIp}:{HostPort}".format(**port[0]) if port else None
 
+    def get_mount(self, mount_dest):
+        for mount in self.get('Mounts'):
+            if mount['Destination'] == mount_dest:
+                return mount
+        return None
+
     def start(self, **options):
         return self.client.start(self.id, **options)
 
     def stop(self, **options):
         return self.client.stop(self.id, **options)
 
+    def pause(self, **options):
+        return self.client.pause(self.id, **options)
+
+    def unpause(self, **options):
+        return self.client.unpause(self.id, **options)
+
     def kill(self, **options):
         return self.client.kill(self.id, **options)
 
@@ -155,6 +216,21 @@ class Container(object):
     def remove(self, **options):
         return self.client.remove_container(self.id, **options)
 
+    def create_exec(self, command, **options):
+        return self.client.exec_create(self.id, command, **options)
+
+    def start_exec(self, exec_id, **options):
+        return self.client.exec_start(exec_id, **options)
+
+    def rename_to_tmp_name(self):
+        """Rename the container to a hopefully unique temporary container name
+        by prepending the short id.
+        """
+        self.client.rename(
+            self.id,
+            '%s_%s' % (self.short_id, self.name)
+        )
+
     def inspect_if_not_inspected(self):
         if not self.has_been_inspected:
             self.inspect()
@@ -170,22 +246,9 @@ class Container(object):
         self.has_been_inspected = True
         return self.dictionary
 
-    # TODO: only used by tests, move to test module
-    def links(self):
-        links = []
-        for container in self.client.containers():
-            for name in container['Names']:
-                bits = name.split('/')
-                if len(bits) > 2 and bits[1] == self.name:
-                    links.append(bits[2])
-        return links
-
     def attach(self, *args, **kwargs):
         return self.client.attach(self.id, *args, **kwargs)
 
-    def attach_socket(self, **kwargs):
-        return self.client.attach_socket(self.id, **kwargs)
-
     def __repr__(self):
         return '<Container: %s (%s)>' % (self.name, self.id[:6])
 

compose/errors.py

@@ -0,0 +1,33 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+
+class OperationFailedError(Exception):
+    def __init__(self, reason):
+        self.msg = reason
+
+
+class StreamParseError(RuntimeError):
+    def __init__(self, reason):
+        self.msg = reason
+
+
+class HealthCheckException(Exception):
+    def __init__(self, reason):
+        self.msg = reason
+
+
+class HealthCheckFailed(HealthCheckException):
+    def __init__(self, container_id):
+        super(HealthCheckFailed, self).__init__(
+            'Container "{}" is unhealthy.'.format(container_id)
+        )
+
+
+class NoHealthCheckConfigured(HealthCheckException):
+    def __init__(self, service_name):
+        super(NoHealthCheckConfigured, self).__init__(
+            'Service "{}" is missing a healthcheck configuration'.format(
+                service_name
+            )
+        )

compose/legacy.py

@@ -1,180 +0,0 @@
-import logging
-import re
-
-from .const import LABEL_VERSION
-from .container import get_container_name, Container
-
-
-log = logging.getLogger(__name__)
-
-
-# TODO: remove this section when migrate_project_to_labels is removed
-NAME_RE = re.compile(r'^([^_]+)_([^_]+)_(run_)?(\d+)$')
-
-ERROR_MESSAGE_FORMAT = """
-Compose found the following containers without labels:
-
-{names_list}
-
-As of Compose 1.3.0, containers are identified with labels instead of naming convention. If you want to continue using these containers, run:
-
-    $ docker-compose migrate-to-labels
-
-Alternatively, remove them:
-
-    $ docker rm -f {rm_args}
-"""
-
-ONE_OFF_ADDENDUM_FORMAT = """
-You should also remove your one-off containers:
-
-    $ docker rm -f {rm_args}
-"""
-
-ONE_OFF_ERROR_MESSAGE_FORMAT = """
-Compose found the following containers without labels:
-
-{names_list}
-
-As of Compose 1.3.0, containers are identified with labels instead of naming convention.
-
-Remove them before continuing:
-
-    $ docker rm -f {rm_args}
-"""
-
-
-def check_for_legacy_containers(
-        client,
-        project,
-        services,
-        allow_one_off=True):
-    """Check if there are containers named using the old naming convention
-    and warn the user that those containers may need to be migrated to
-    using labels, so that compose can find them.
-    """
-    containers = get_legacy_containers(client, project, services, one_off=False)
-
-    if containers:
-        one_off_containers = get_legacy_containers(client, project, services, one_off=True)
-
-        raise LegacyContainersError(
-            [c.name for c in containers],
-            [c.name for c in one_off_containers],
-        )
-
-    if not allow_one_off:
-        one_off_containers = get_legacy_containers(client, project, services, one_off=True)
-
-        if one_off_containers:
-            raise LegacyOneOffContainersError(
-                [c.name for c in one_off_containers],
-            )
-
-
-class LegacyError(Exception):
-    def __unicode__(self):
-        return self.msg
-
-    __str__ = __unicode__
-
-
-class LegacyContainersError(LegacyError):
-    def __init__(self, names, one_off_names):
-        self.names = names
-        self.one_off_names = one_off_names
-
-        self.msg = ERROR_MESSAGE_FORMAT.format(
-            names_list="\n".join("    {}".format(name) for name in names),
-            rm_args=" ".join(names),
-        )
-
-        if one_off_names:
-            self.msg += ONE_OFF_ADDENDUM_FORMAT.format(rm_args=" ".join(one_off_names))
-
-
-class LegacyOneOffContainersError(LegacyError):
-    def __init__(self, one_off_names):
-        self.one_off_names = one_off_names
-
-        self.msg = ONE_OFF_ERROR_MESSAGE_FORMAT.format(
-            names_list="\n".join("    {}".format(name) for name in one_off_names),
-            rm_args=" ".join(one_off_names),
-        )
-
-
-def add_labels(project, container):
-    project_name, service_name, one_off, number = NAME_RE.match(container.name).groups()
-    if project_name != project.name or service_name not in project.service_names:
-        return
-    service = project.get_service(service_name)
-    service.recreate_container(container)
-
-
-def migrate_project_to_labels(project):
-    log.info("Running migration to labels for project %s", project.name)
-
-    containers = get_legacy_containers(
-        project.client,
-        project.name,
-        project.service_names,
-        one_off=False,
-    )
-
-    for container in containers:
-        add_labels(project, container)
-
-
-def get_legacy_containers(
-        client,
-        project,
-        services,
-        one_off=False):
-
-    return list(_get_legacy_containers_iter(
-        client,
-        project,
-        services,
-        one_off=one_off,
-    ))
-
-
-def _get_legacy_containers_iter(
-        client,
-        project,
-        services,
-        one_off=False):
-
-    containers = client.containers(all=True)
-
-    for service in services:
-        for container in containers:
-            if LABEL_VERSION in (container.get('Labels') or {}):
-                continue
-
-            name = get_container_name(container)
-            if has_container(project, service, name, one_off=one_off):
-                yield Container.from_ps(client, container)
-
-
-def has_container(project, service, name, one_off=False):
-    if not name or not is_valid_name(name, one_off):
-        return False
-    container_project, container_service, _container_number = parse_name(name)
-    return container_project == project and container_service == service
-
-
-def is_valid_name(name, one_off=False):
-    match = NAME_RE.match(name)
-    if match is None:
-        return False
-    if one_off:
-        return match.group(3) == 'run_'
-    else:
-        return match.group(3) is None
-
-
-def parse_name(name):
-    match = NAME_RE.match(name)
-    (project, service_name, _, suffix) = match.groups()
-    return (project, service_name, int(suffix))

compose/network.py

@@ -0,0 +1,231 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import logging
+
+from docker.errors import NotFound
+from docker.types import IPAMConfig
+from docker.types import IPAMPool
+from docker.utils import version_gte
+from docker.utils import version_lt
+
+from .config import ConfigurationError
+from .const import LABEL_NETWORK
+from .const import LABEL_PROJECT
+
+
+log = logging.getLogger(__name__)
+
+OPTS_EXCEPTIONS = [
+    'com.docker.network.driver.overlay.vxlanid_list',
+]
+
+
+class Network(object):
+    def __init__(self, client, project, name, driver=None, driver_opts=None,
+                 ipam=None, external_name=None, internal=False, enable_ipv6=False,
+                 labels=None):
+        self.client = client
+        self.project = project
+        self.name = name
+        self.driver = driver
+        self.driver_opts = driver_opts
+        self.ipam = create_ipam_config_from_dict(ipam)
+        self.external_name = external_name
+        self.internal = internal
+        self.enable_ipv6 = enable_ipv6
+        self.labels = labels
+
+    def ensure(self):
+        if self.external_name:
+            try:
+                self.inspect()
+                log.debug(
+                    'Network {0} declared as external. No new '
+                    'network will be created.'.format(self.name)
+                )
+            except NotFound:
+                raise ConfigurationError(
+                    'Network {name} declared as external, but could'
+                    ' not be found. Please create the network manually'
+                    ' using `{command} {name}` and try again.'.format(
+                        name=self.external_name,
+                        command='docker network create'
+                    )
+                )
+            return
+
+        try:
+            data = self.inspect()
+            check_remote_network_config(data, self)
+        except NotFound:
+            driver_name = 'the default driver'
+            if self.driver:
+                driver_name = 'driver "{}"'.format(self.driver)
+
+            log.info(
+                'Creating network "{}" with {}'
+                .format(self.full_name, driver_name)
+            )
+
+            self.client.create_network(
+                name=self.full_name,
+                driver=self.driver,
+                options=self.driver_opts,
+                ipam=self.ipam,
+                internal=self.internal,
+                enable_ipv6=self.enable_ipv6,
+                labels=self._labels,
+                attachable=version_gte(self.client._version, '1.24') or None,
+            )
+
+    def remove(self):
+        if self.external_name:
+            log.info("Network %s is external, skipping", self.full_name)
+            return
+
+        log.info("Removing network {}".format(self.full_name))
+        self.client.remove_network(self.full_name)
+
+    def inspect(self):
+        return self.client.inspect_network(self.full_name)
+
+    @property
+    def full_name(self):
+        if self.external_name:
+            return self.external_name
+        return '{0}_{1}'.format(self.project, self.name)
+
+    @property
+    def _labels(self):
+        if version_lt(self.client._version, '1.23'):
+            return None
+        labels = self.labels.copy() if self.labels else {}
+        labels.update({
+            LABEL_PROJECT: self.project,
+            LABEL_NETWORK: self.name,
+        })
+        return labels
+
+
+def create_ipam_config_from_dict(ipam_dict):
+    if not ipam_dict:
+        return None
+
+    return IPAMConfig(
+        driver=ipam_dict.get('driver'),
+        pool_configs=[
+            IPAMPool(
+                subnet=config.get('subnet'),
+                iprange=config.get('ip_range'),
+                gateway=config.get('gateway'),
+                aux_addresses=config.get('aux_addresses'),
+            )
+            for config in ipam_dict.get('config', [])
+        ],
+    )
+
+
+def check_remote_network_config(remote, local):
+    if local.driver and remote.get('Driver') != local.driver:
+        raise ConfigurationError(
+            'Network "{}" needs to be recreated - driver has changed'
+            .format(local.full_name)
+        )
+    local_opts = local.driver_opts or {}
+    remote_opts = remote.get('Options') or {}
+    for k in set.union(set(remote_opts.keys()), set(local_opts.keys())):
+        if k in OPTS_EXCEPTIONS:
+            continue
+        if remote_opts.get(k) != local_opts.get(k):
+            raise ConfigurationError(
+                'Network "{}" needs to be recreated - options have changed'
+                .format(local.full_name)
+            )
+
+
+def build_networks(name, config_data, client):
+    network_config = config_data.networks or {}
+    networks = {
+        network_name: Network(
+            client=client, project=name, name=network_name,
+            driver=data.get('driver'),
+            driver_opts=data.get('driver_opts'),
+            ipam=data.get('ipam'),
+            external_name=data.get('external_name'),
+            internal=data.get('internal'),
+            enable_ipv6=data.get('enable_ipv6'),
+            labels=data.get('labels'),
+        )
+        for network_name, data in network_config.items()
+    }
+
+    if 'default' not in networks:
+        networks['default'] = Network(client, name, 'default')
+
+    return networks
+
+
+class ProjectNetworks(object):
+
+    def __init__(self, networks, use_networking):
+        self.networks = networks or {}
+        self.use_networking = use_networking
+
+    @classmethod
+    def from_services(cls, services, networks, use_networking):
+        service_networks = {
+            network: networks.get(network)
+            for service in services
+            for network in get_network_names_for_service(service)
+        }
+        unused = set(networks) - set(service_networks) - {'default'}
+        if unused:
+            log.warn(
+                "Some networks were defined but are not used by any service: "
+                "{}".format(", ".join(unused)))
+        return cls(service_networks, use_networking)
+
+    def remove(self):
+        if not self.use_networking:
+            return
+        for network in self.networks.values():
+            try:
+                network.remove()
+            except NotFound:
+                log.warn("Network %s not found.", network.full_name)
+
+    def initialize(self):
+        if not self.use_networking:
+            return
+
+        for network in self.networks.values():
+            network.ensure()
+
+
+def get_network_defs_for_service(service_dict):
+    if 'network_mode' in service_dict:
+        return {}
+    networks = service_dict.get('networks', {'default': None})
+    return dict(
+        (net, (config or {}))
+        for net, config in networks.items()
+    )
+
+
+def get_network_names_for_service(service_dict):
+    return get_network_defs_for_service(service_dict).keys()
+
+
+def get_networks(service_dict, network_definitions):
+    networks = {}
+    for name, netdef in get_network_defs_for_service(service_dict).items():
+        network = network_definitions.get(name)
+        if network:
+            networks[network.full_name] = netdef
+        else:
+            raise ConfigurationError(
+                'Service "{}" uses an undefined network "{}"'
+                .format(service_dict['name'], name))
+
+    return networks

compose/parallel.py

@@ -0,0 +1,259 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import logging
+import operator
+import sys
+from threading import Thread
+
+from docker.errors import APIError
+from six.moves import _thread as thread
+from six.moves.queue import Empty
+from six.moves.queue import Queue
+
+from compose.cli.signals import ShutdownException
+from compose.errors import HealthCheckFailed
+from compose.errors import NoHealthCheckConfigured
+from compose.errors import OperationFailedError
+from compose.utils import get_output_stream
+
+
+log = logging.getLogger(__name__)
+
+STOP = object()
+
+
+def parallel_execute(objects, func, get_name, msg, get_deps=None):
+    """Runs func on objects in parallel while ensuring that func is
+    ran on object only after it is ran on all its dependencies.
+
+    get_deps called on object must return a collection with its dependencies.
+    get_name called on object must return its name.
+    """
+    objects = list(objects)
+    stream = get_output_stream(sys.stderr)
+
+    writer = ParallelStreamWriter(stream, msg)
+    for obj in objects:
+        writer.initialize(get_name(obj))
+
+    events = parallel_execute_iter(objects, func, get_deps)
+
+    errors = {}
+    results = []
+    error_to_reraise = None
+
+    for obj, result, exception in events:
+        if exception is None:
+            writer.write(get_name(obj), 'done')
+            results.append(result)
+        elif isinstance(exception, APIError):
+            errors[get_name(obj)] = exception.explanation
+            writer.write(get_name(obj), 'error')
+        elif isinstance(exception, (OperationFailedError, HealthCheckFailed, NoHealthCheckConfigured)):
+            errors[get_name(obj)] = exception.msg
+            writer.write(get_name(obj), 'error')
+        elif isinstance(exception, UpstreamError):
+            writer.write(get_name(obj), 'error')
+        else:
+            errors[get_name(obj)] = exception
+            error_to_reraise = exception
+
+    for obj_name, error in errors.items():
+        stream.write("\nERROR: for {}  {}\n".format(obj_name, error))
+
+    if error_to_reraise:
+        raise error_to_reraise
+
+    return results, errors
+
+
+def _no_deps(x):
+    return []
+
+
+class State(object):
+    """
+    Holds the state of a partially-complete parallel operation.
+
+    state.started:   objects being processed
+    state.finished:  objects which have been processed
+    state.failed:    objects which either failed or whose dependencies failed
+    """
+    def __init__(self, objects):
+        self.objects = objects
+
+        self.started = set()
+        self.finished = set()
+        self.failed = set()
+
+    def is_done(self):
+        return len(self.finished) + len(self.failed) >= len(self.objects)
+
+    def pending(self):
+        return set(self.objects) - self.started - self.finished - self.failed
+
+
+def parallel_execute_iter(objects, func, get_deps):
+    """
+    Runs func on objects in parallel while ensuring that func is
+    ran on object only after it is ran on all its dependencies.
+
+    Returns an iterator of tuples which look like:
+
+    # if func returned normally when run on object
+    (object, result, None)
+
+    # if func raised an exception when run on object
+    (object, None, exception)
+
+    # if func raised an exception when run on one of object's dependencies
+    (object, None, UpstreamError())
+    """
+    if get_deps is None:
+        get_deps = _no_deps
+
+    results = Queue()
+    state = State(objects)
+
+    while True:
+        feed_queue(objects, func, get_deps, results, state)
+
+        try:
+            event = results.get(timeout=0.1)
+        except Empty:
+            continue
+        # See https://github.com/docker/compose/issues/189
+        except thread.error:
+            raise ShutdownException()
+
+        if event is STOP:
+            break
+
+        obj, _, exception = event
+        if exception is None:
+            log.debug('Finished processing: {}'.format(obj))
+            state.finished.add(obj)
+        else:
+            log.debug('Failed: {}'.format(obj))
+            state.failed.add(obj)
+
+        yield event
+
+
+def producer(obj, func, results):
+    """
+    The entry point for a producer thread which runs func on a single object.
+    Places a tuple on the results queue once func has either returned or raised.
+    """
+    try:
+        result = func(obj)
+        results.put((obj, result, None))
+    except Exception as e:
+        results.put((obj, None, e))
+
+
+def feed_queue(objects, func, get_deps, results, state):
+    """
+    Starts producer threads for any objects which are ready to be processed
+    (i.e. they have no dependencies which haven't been successfully processed).
+
+    Shortcuts any objects whose dependencies have failed and places an
+    (object, None, UpstreamError()) tuple on the results queue.
+    """
+    pending = state.pending()
+    log.debug('Pending: {}'.format(pending))
+
+    for obj in pending:
+        deps = get_deps(obj)
+        try:
+            if any(dep[0] in state.failed for dep in deps):
+                log.debug('{} has upstream errors - not processing'.format(obj))
+                results.put((obj, None, UpstreamError()))
+                state.failed.add(obj)
+            elif all(
+                dep not in objects or (
+                    dep in state.finished and (not ready_check or ready_check(dep))
+                ) for dep, ready_check in deps
+            ):
+                log.debug('Starting producer thread for {}'.format(obj))
+                t = Thread(target=producer, args=(obj, func, results))
+                t.daemon = True
+                t.start()
+                state.started.add(obj)
+        except (HealthCheckFailed, NoHealthCheckConfigured) as e:
+            log.debug(
+                'Healthcheck for service(s) upstream of {} failed - '
+                'not processing'.format(obj)
+            )
+            results.put((obj, None, e))
+
+    if state.is_done():
+        results.put(STOP)
+
+
+class UpstreamError(Exception):
+    pass
+
+
+class ParallelStreamWriter(object):
+    """Write out messages for operations happening in parallel.
+
+    Each operation has it's own line, and ANSI code characters are used
+    to jump to the correct line, and write over the line.
+    """
+
+    def __init__(self, stream, msg):
+        self.stream = stream
+        self.msg = msg
+        self.lines = []
+
+    def initialize(self, obj_index):
+        if self.msg is None:
+            return
+        self.lines.append(obj_index)
+        self.stream.write("{} {} ... \r\n".format(self.msg, obj_index))
+        self.stream.flush()
+
+    def write(self, obj_index, status):
+        if self.msg is None:
+            return
+        position = self.lines.index(obj_index)
+        diff = len(self.lines) - position
+        # move up
+        self.stream.write("%c[%dA" % (27, diff))
+        # erase
+        self.stream.write("%c[2K\r" % 27)
+        self.stream.write("{} {} ... {}\r".format(self.msg, obj_index, status))
+        # move back down
+        self.stream.write("%c[%dB" % (27, diff))
+        self.stream.flush()
+
+
+def parallel_operation(containers, operation, options, message):
+    parallel_execute(
+        containers,
+        operator.methodcaller(operation, **options),
+        operator.attrgetter('name'),
+        message)
+
+
+def parallel_remove(containers, options):
+    stopped_containers = [c for c in containers if not c.is_running]
+    parallel_operation(stopped_containers, 'remove', options, 'Removing')
+
+
+def parallel_start(containers, options):
+    parallel_operation(containers, 'start', options, 'Starting')
+
+
+def parallel_pause(containers, options):
+    parallel_operation(containers, 'pause', options, 'Pausing')
+
+
+def parallel_unpause(containers, options):
+    parallel_operation(containers, 'unpause', options, 'Unpausing')
+
+
+def parallel_kill(containers, options):
+    parallel_operation(containers, 'kill', options, 'Killing')

compose/progress_stream.py

@@ -1,6 +1,7 @@
-import json
-import os
-import codecs
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+from compose import utils
 
 
 class StreamOutputError(Exception):
@@ -8,35 +9,41 @@ class StreamOutputError(Exception):
 
 
 def stream_output(output, stream):
-    is_terminal = hasattr(stream, 'fileno') and os.isatty(stream.fileno())
-    stream = codecs.getwriter('utf-8')(stream)
+    is_terminal = hasattr(stream, 'isatty') and stream.isatty()
+    stream = utils.get_output_stream(stream)
     all_events = []
     lines = {}
     diff = 0
 
-    for chunk in output:
-        event = json.loads(chunk)
+    for event in utils.json_stream(output):
         all_events.append(event)
+        is_progress_event = 'progress' in event or 'progressDetail' in event
 
-        if 'progress' in event or 'progressDetail' in event:
-            image_id = event.get('id')
-            if not image_id:
-                continue
+        if not is_progress_event:
+            print_output_event(event, stream, is_terminal)
+            stream.flush()
+            continue
 
-            if image_id in lines:
-                diff = len(lines) - lines[image_id]
-            else:
-                lines[image_id] = len(lines)
-                stream.write("\n")
-                diff = 0
+        if not is_terminal:
+            continue
 
-            if is_terminal:
-                # move cursor up `diff` rows
-                stream.write("%c[%dA" % (27, diff))
+        # if it's a progress event and we have a terminal, then display the progress bars
+        image_id = event.get('id')
+        if not image_id:
+            continue
+
+        if image_id not in lines:
+            lines[image_id] = len(lines)
+            stream.write("\n")
+
+        diff = len(lines) - lines[image_id]
+
+        # move cursor up `diff` rows
+        stream.write("%c[%dA" % (27, diff))
 
         print_output_event(event, stream, is_terminal)
 
-        if 'id' in event and is_terminal:
+        if 'id' in event:
             # move cursor back down
             stream.write("%c[%dB" % (27, diff))
 
@@ -55,7 +62,6 @@ def print_output_event(event, stream, is_terminal):
         # erase current line
         stream.write("%c[2K\r" % 27)
         terminator = "\r"
-        pass
     elif 'progressDetail' in event:
         return
 
@@ -84,3 +90,22 @@ def print_output_event(event, stream, is_terminal):
         stream.write("%s%s" % (event['stream'], terminator))
     else:
         stream.write("%s%s\n" % (status, terminator))
+
+
+def get_digest_from_pull(events):
+    for event in events:
+        status = event.get('status')
+        if not status or 'Digest' not in status:
+            continue
+
+        _, digest = status.split(':', 1)
+        return digest.strip()
+    return None
+
+
+def get_digest_from_push(events):
+    for event in events:
+        digest = event.get('aux', {}).get('Digest')
+        if digest:
+            return digest
+    return None

compose/project.py

@@ -1,97 +1,128 @@
-from __future__ import unicode_literals
 from __future__ import absolute_import
-from functools import reduce
-import logging
+from __future__ import unicode_literals
 
+import datetime
+import logging
+import operator
+from functools import reduce
+
+import enum
 from docker.errors import APIError
 
-from .config import get_service_name_from_net, ConfigurationError
-from .const import DEFAULT_TIMEOUT, LABEL_PROJECT, LABEL_SERVICE, LABEL_ONE_OFF
+from . import parallel
+from .config import ConfigurationError
+from .config.config import V1
+from .config.sort_services import get_container_name_from_network_mode
+from .config.sort_services import get_service_name_from_network_mode
+from .const import IMAGE_EVENTS
+from .const import LABEL_ONE_OFF
+from .const import LABEL_PROJECT
+from .const import LABEL_SERVICE
 from .container import Container
-from .legacy import check_for_legacy_containers
-from .service import ContainerNet
-from .service import Net
+from .network import build_networks
+from .network import get_networks
+from .network import ProjectNetworks
+from .service import BuildAction
+from .service import ContainerNetworkMode
+from .service import ConvergenceStrategy
+from .service import NetworkMode
 from .service import Service
-from .service import ServiceNet
-from .utils import parallel_execute
+from .service import ServiceNetworkMode
+from .utils import microseconds_from_time_nano
+from .volume import ProjectVolumes
+
 
 log = logging.getLogger(__name__)
 
 
-def sort_service_dicts(services):
-    # Topological sort (Cormen/Tarjan algorithm).
-    unmarked = services[:]
-    temporary_marked = set()
-    sorted_services = []
+@enum.unique
+class OneOffFilter(enum.Enum):
+    include = 0
+    exclude = 1
+    only = 2
 
-    def get_service_names(links):
-        return [link.split(':')[0] for link in links]
-
-    def get_service_dependents(service_dict, services):
-        name = service_dict['name']
-        return [
-            service for service in services
-            if (name in get_service_names(service.get('links', [])) or
-                name in service.get('volumes_from', []) or
-                name == get_service_name_from_net(service.get('net')))
-        ]
-
-    def visit(n):
-        if n['name'] in temporary_marked:
-            if n['name'] in get_service_names(n.get('links', [])):
-                raise DependencyError('A service can not link to itself: %s' % n['name'])
-            if n['name'] in n.get('volumes_from', []):
-                raise DependencyError('A service can not mount itself as volume: %s' % n['name'])
-            else:
-                raise DependencyError('Circular import between %s' % ' and '.join(temporary_marked))
-        if n in unmarked:
-            temporary_marked.add(n['name'])
-            for m in get_service_dependents(n, services):
-                visit(m)
-            temporary_marked.remove(n['name'])
-            unmarked.remove(n)
-            sorted_services.insert(0, n)
-
-    while unmarked:
-        visit(unmarked[-1])
-
-    return sorted_services
+    @classmethod
+    def update_labels(cls, value, labels):
+        if value == cls.only:
+            labels.append('{0}={1}'.format(LABEL_ONE_OFF, "True"))
+        elif value == cls.exclude:
+            labels.append('{0}={1}'.format(LABEL_ONE_OFF, "False"))
+        elif value == cls.include:
+            pass
+        else:
+            raise ValueError("Invalid value for one_off: {}".format(repr(value)))
 
 
 class Project(object):
     """
     A collection of services.
     """
-    def __init__(self, name, services, client):
+    def __init__(self, name, services, client, networks=None, volumes=None):
         self.name = name
         self.services = services
         self.client = client
+        self.volumes = volumes or ProjectVolumes({})
+        self.networks = networks or ProjectNetworks({}, False)
 
-    def labels(self, one_off=False):
-        return [
-            '{0}={1}'.format(LABEL_PROJECT, self.name),
-            '{0}={1}'.format(LABEL_ONE_OFF, "True" if one_off else "False"),
-        ]
+    def labels(self, one_off=OneOffFilter.exclude):
+        labels = ['{0}={1}'.format(LABEL_PROJECT, self.name)]
+
+        OneOffFilter.update_labels(one_off, labels)
+        return labels
 
     @classmethod
-    def from_dicts(cls, name, service_dicts, client):
+    def from_config(cls, name, config_data, client):
         """
-        Construct a ServiceCollection from a list of dicts representing services.
+        Construct a Project from a config.Config object.
         """
-        project = cls(name, [], client)
-        for service_dict in sort_service_dicts(service_dicts):
+        use_networking = (config_data.version and config_data.version != V1)
+        networks = build_networks(name, config_data, client)
+        project_networks = ProjectNetworks.from_services(
+            config_data.services,
+            networks,
+            use_networking)
+        volumes = ProjectVolumes.from_config(name, config_data, client)
+        project = cls(name, [], client, project_networks, volumes)
+
+        for service_dict in config_data.services:
+            service_dict = dict(service_dict)
+            if use_networking:
+                service_networks = get_networks(service_dict, networks)
+            else:
+                service_networks = {}
+
+            service_dict.pop('networks', None)
             links = project.get_links(service_dict)
-            volumes_from = project.get_volumes_from(service_dict)
-            net = project.get_net(service_dict)
+            network_mode = project.get_network_mode(
+                service_dict, list(service_networks.keys())
+            )
+            volumes_from = get_volumes_from(project, service_dict)
+
+            if config_data.version != V1:
+                service_dict['volumes'] = [
+                    volumes.namespace_spec(volume_spec)
+                    for volume_spec in service_dict.get('volumes', [])
+                ]
+
+            secrets = get_secrets(
+                service_dict['name'],
+                service_dict.pop('secrets', None) or [],
+                config_data.secrets)
 
             project.services.append(
                 Service(
+                    service_dict.pop('name'),
                     client=client,
                     project=name,
+                    use_networking=use_networking,
+                    networks=service_networks,
                     links=links,
-                    net=net,
+                    network_mode=network_mode,
                     volumes_from=volumes_from,
-                    **service_dict))
+                    secrets=secrets,
+                    **service_dict)
+            )
+
         return project
 
     @property
@@ -134,20 +165,24 @@ class Project(object):
         Raises NoSuchService if any of the named services do not exist.
         """
         if service_names is None or len(service_names) == 0:
-            return self.get_services(
-                service_names=self.service_names,
-                include_deps=include_deps
-            )
-        else:
-            unsorted = [self.get_service(name) for name in service_names]
-            services = [s for s in self.services if s in unsorted]
+            service_names = self.service_names
 
-            if include_deps:
-                services = reduce(self._inject_deps, services, [])
+        unsorted = [self.get_service(name) for name in service_names]
+        services = [s for s in self.services if s in unsorted]
 
-            uniques = []
-            [uniques.append(s) for s in services if s not in uniques]
-            return uniques
+        if include_deps:
+            services = reduce(self._inject_deps, services, [])
+
+        uniques = []
+        [uniques.append(s) for s in services if s not in uniques]
+
+        return uniques
+
+    def get_services_without_duplicate(self, service_names=None, include_deps=False):
+        services = self.get_services(service_names, include_deps)
+        for service in services:
+            service.remove_duplicate_containers()
+        return services
 
     def get_links(self, service_dict):
         links = []
@@ -160,182 +195,318 @@ class Project(object):
                 try:
                     links.append((self.get_service(service_name), link_name))
                 except NoSuchService:
-                    raise ConfigurationError('Service "%s" has a link to service "%s" which does not exist.' % (service_dict['name'], service_name))
+                    raise ConfigurationError(
+                        'Service "%s" has a link to service "%s" which does not '
+                        'exist.' % (service_dict['name'], service_name))
             del service_dict['links']
         return links
 
-    def get_volumes_from(self, service_dict):
-        volumes_from = []
-        if 'volumes_from' in service_dict:
-            for volume_name in service_dict.get('volumes_from', []):
-                try:
-                    service = self.get_service(volume_name)
-                    volumes_from.append(service)
-                except NoSuchService:
-                    try:
-                        container = Container.from_id(self.client, volume_name)
-                        volumes_from.append(container)
-                    except APIError:
-                        raise ConfigurationError('Service "%s" mounts volumes from "%s", which is not the name of a service or container.' % (service_dict['name'], volume_name))
-            del service_dict['volumes_from']
-        return volumes_from
+    def get_network_mode(self, service_dict, networks):
+        network_mode = service_dict.pop('network_mode', None)
+        if not network_mode:
+            if self.networks.use_networking:
+                return NetworkMode(networks[0]) if networks else NetworkMode('none')
+            return NetworkMode(None)
 
-    def get_net(self, service_dict):
-        net = service_dict.pop('net', None)
-        if not net:
-            return Net(None)
+        service_name = get_service_name_from_network_mode(network_mode)
+        if service_name:
+            return ServiceNetworkMode(self.get_service(service_name))
 
-        net_name = get_service_name_from_net(net)
-        if not net_name:
-            return Net(net)
+        container_name = get_container_name_from_network_mode(network_mode)
+        if container_name:
+            try:
+                return ContainerNetworkMode(Container.from_id(self.client, container_name))
+            except APIError:
+                raise ConfigurationError(
+                    "Service '{name}' uses the network stack of container '{dep}' which "
+                    "does not exist.".format(name=service_dict['name'], dep=container_name))
 
-        try:
-            return ServiceNet(self.get_service(net_name))
-        except NoSuchService:
-            pass
-        try:
-            return ContainerNet(Container.from_id(self.client, net_name))
-        except APIError:
-            raise ConfigurationError(
-                'Service "%s" is trying to use the network of "%s", '
-                'which is not the name of a service or container.' % (
-                    service_dict['name'],
-                    net_name))
+        return NetworkMode(network_mode)
 
     def start(self, service_names=None, **options):
-        for service in self.get_services(service_names):
-            service.start(**options)
+        containers = []
 
-    def stop(self, service_names=None, **options):
-        parallel_execute(
-            objects=self.containers(service_names),
-            obj_callable=lambda c: c.stop(**options),
-            msg_index=lambda c: c.name,
-            msg="Stopping"
-        )
+        def start_service(service):
+            service_containers = service.start(quiet=True, **options)
+            containers.extend(service_containers)
+
+        services = self.get_services(service_names)
+
+        def get_deps(service):
+            return {
+                (self.get_service(dep), config)
+                for dep, config in service.get_dependency_configs().items()
+            }
+
+        parallel.parallel_execute(
+            services,
+            start_service,
+            operator.attrgetter('name'),
+            'Starting',
+            get_deps)
+
+        return containers
+
+    def stop(self, service_names=None, one_off=OneOffFilter.exclude, **options):
+        containers = self.containers(service_names, one_off=one_off)
+
+        def get_deps(container):
+            # actually returning inversed dependencies
+            return {(other, None) for other in containers
+                    if container.service in
+                    self.get_service(other.service).get_dependency_names()}
+
+        parallel.parallel_execute(
+            containers,
+            self.build_container_operation_with_timeout_func('stop', options),
+            operator.attrgetter('name'),
+            'Stopping',
+            get_deps)
+
+    def pause(self, service_names=None, **options):
+        containers = self.containers(service_names)
+        parallel.parallel_pause(reversed(containers), options)
+        return containers
+
+    def unpause(self, service_names=None, **options):
+        containers = self.containers(service_names)
+        parallel.parallel_unpause(containers, options)
+        return containers
 
     def kill(self, service_names=None, **options):
-        parallel_execute(
-            objects=self.containers(service_names),
-            obj_callable=lambda c: c.kill(**options),
-            msg_index=lambda c: c.name,
-            msg="Killing"
-        )
+        parallel.parallel_kill(self.containers(service_names), options)
 
-    def remove_stopped(self, service_names=None, **options):
-        all_containers = self.containers(service_names, stopped=True)
-        stopped_containers = [c for c in all_containers if not c.is_running]
-        parallel_execute(
-            objects=stopped_containers,
-            obj_callable=lambda c: c.remove(**options),
-            msg_index=lambda c: c.name,
-            msg="Removing"
-        )
+    def remove_stopped(self, service_names=None, one_off=OneOffFilter.exclude, **options):
+        parallel.parallel_remove(self.containers(
+            service_names, stopped=True, one_off=one_off
+        ), options)
+
+    def down(self, remove_image_type, include_volumes, remove_orphans=False):
+        self.stop(one_off=OneOffFilter.include)
+        self.find_orphan_containers(remove_orphans)
+        self.remove_stopped(v=include_volumes, one_off=OneOffFilter.include)
+
+        self.networks.remove()
+
+        if include_volumes:
+            self.volumes.remove()
+
+        self.remove_images(remove_image_type)
+
+    def remove_images(self, remove_image_type):
+        for service in self.get_services():
+            service.remove_image(remove_image_type)
 
     def restart(self, service_names=None, **options):
-        for service in self.get_services(service_names):
-            service.restart(**options)
+        containers = self.containers(service_names, stopped=True)
 
-    def build(self, service_names=None, no_cache=False):
+        parallel.parallel_execute(
+            containers,
+            self.build_container_operation_with_timeout_func('restart', options),
+            operator.attrgetter('name'),
+            'Restarting')
+        return containers
+
+    def build(self, service_names=None, no_cache=False, pull=False, force_rm=False):
         for service in self.get_services(service_names):
             if service.can_be_built():
-                service.build(no_cache)
+                service.build(no_cache, pull, force_rm)
             else:
                 log.info('%s uses an image, skipping' % service.name)
 
+    def create(
+        self,
+        service_names=None,
+        strategy=ConvergenceStrategy.changed,
+        do_build=BuildAction.none,
+    ):
+        services = self.get_services_without_duplicate(service_names, include_deps=True)
+
+        for svc in services:
+            svc.ensure_image_exists(do_build=do_build)
+        plans = self._get_convergence_plans(services, strategy)
+
+        for service in services:
+            service.execute_convergence_plan(
+                plans[service.name],
+                detached=True,
+                start=False)
+
+    def events(self, service_names=None):
+        def build_container_event(event, container):
+            time = datetime.datetime.fromtimestamp(event['time'])
+            time = time.replace(
+                microsecond=microseconds_from_time_nano(event['timeNano']))
+            return {
+                'time': time,
+                'type': 'container',
+                'action': event['status'],
+                'id': container.id,
+                'service': container.service,
+                'attributes': {
+                    'name': container.name,
+                    'image': event['from'],
+                },
+                'container': container,
+            }
+
+        service_names = set(service_names or self.service_names)
+        for event in self.client.events(
+            filters={'label': self.labels()},
+            decode=True
+        ):
+            # The first part of this condition is a guard against some events
+            # broadcasted by swarm that don't have a status field.
+            # See https://github.com/docker/compose/issues/3316
+            if 'status' not in event or event['status'] in IMAGE_EVENTS:
+                # We don't receive any image events because labels aren't applied
+                # to images
+                continue
+
+            # TODO: get labels from the API v1.22 , see github issue 2618
+            try:
+                # this can fail if the container has been removed
+                container = Container.from_id(self.client, event['id'])
+            except APIError:
+                continue
+            if container.service not in service_names:
+                continue
+            yield build_container_event(event, container)
+
     def up(self,
            service_names=None,
            start_deps=True,
-           allow_recreate=True,
-           force_recreate=False,
-           do_build=True,
-           timeout=DEFAULT_TIMEOUT):
+           strategy=ConvergenceStrategy.changed,
+           do_build=BuildAction.none,
+           timeout=None,
+           detached=False,
+           remove_orphans=False):
 
-        if force_recreate and not allow_recreate:
-            raise ValueError("force_recreate and allow_recreate are in conflict")
+        warn_for_swarm_mode(self.client)
 
-        services = self.get_services(service_names, include_deps=start_deps)
+        self.initialize()
+        self.find_orphan_containers(remove_orphans)
 
-        for service in services:
-            service.remove_duplicate_containers()
+        services = self.get_services_without_duplicate(
+            service_names,
+            include_deps=start_deps)
 
-        plans = self._get_convergence_plans(
+        for svc in services:
+            svc.ensure_image_exists(do_build=do_build)
+        plans = self._get_convergence_plans(services, strategy)
+
+        def do(service):
+            return service.execute_convergence_plan(
+                plans[service.name],
+                timeout=timeout,
+                detached=detached
+            )
+
+        def get_deps(service):
+            return {
+                (self.get_service(dep), config)
+                for dep, config in service.get_dependency_configs().items()
+            }
+
+        results, errors = parallel.parallel_execute(
             services,
-            allow_recreate=allow_recreate,
-            force_recreate=force_recreate,
+            do,
+            operator.attrgetter('name'),
+            None,
+            get_deps
         )
+        if errors:
+            raise ProjectError(
+                'Encountered errors while bringing up the project.'
+            )
 
         return [
             container
-            for service in services
-            for container in service.execute_convergence_plan(
-                plans[service.name],
-                do_build=do_build,
-                timeout=timeout
-            )
+            for svc_containers in results
+            if svc_containers is not None
+            for container in svc_containers
         ]
 
-    def _get_convergence_plans(self,
-                               services,
-                               allow_recreate=True,
-                               force_recreate=False):
+    def initialize(self):
+        self.networks.initialize()
+        self.volumes.initialize()
 
+    def _get_convergence_plans(self, services, strategy):
         plans = {}
 
         for service in services:
             updated_dependencies = [
                 name
                 for name in service.get_dependency_names()
-                if name in plans
-                and plans[name].action == 'recreate'
+                if name in plans and
+                plans[name].action in ('recreate', 'create')
             ]
 
-            if updated_dependencies and allow_recreate:
-                log.debug(
-                    '%s has upstream changes (%s)',
-                    service.name, ", ".join(updated_dependencies),
-                )
-                plan = service.convergence_plan(
-                    allow_recreate=allow_recreate,
-                    force_recreate=True,
-                )
+            if updated_dependencies and strategy.allows_recreate:
+                log.debug('%s has upstream changes (%s)',
+                          service.name,
+                          ", ".join(updated_dependencies))
+                plan = service.convergence_plan(ConvergenceStrategy.always)
             else:
-                plan = service.convergence_plan(
-                    allow_recreate=allow_recreate,
-                    force_recreate=force_recreate,
-                )
+                plan = service.convergence_plan(strategy)
 
             plans[service.name] = plan
 
         return plans
 
-    def pull(self, service_names=None):
-        for service in self.get_services(service_names, include_deps=True):
-            service.pull()
+    def pull(self, service_names=None, ignore_pull_failures=False):
+        for service in self.get_services(service_names, include_deps=False):
+            service.pull(ignore_pull_failures)
 
-    def containers(self, service_names=None, stopped=False, one_off=False):
+    def push(self, service_names=None, ignore_push_failures=False):
+        for service in self.get_services(service_names, include_deps=False):
+            service.push(ignore_push_failures)
+
+    def _labeled_containers(self, stopped=False, one_off=OneOffFilter.exclude):
+        return list(filter(None, [
+            Container.from_ps(self.client, container)
+            for container in self.client.containers(
+                all=stopped,
+                filters={'label': self.labels(one_off=one_off)})])
+        )
+
+    def containers(self, service_names=None, stopped=False, one_off=OneOffFilter.exclude):
         if service_names:
             self.validate_service_names(service_names)
         else:
             service_names = self.service_names
 
-        containers = filter(None, [
-            Container.from_ps(self.client, container)
-            for container in self.client.containers(
-                all=stopped,
-                filters={'label': self.labels(one_off=one_off)})])
+        containers = self._labeled_containers(stopped, one_off)
 
         def matches_service_names(container):
             return container.labels.get(LABEL_SERVICE) in service_names
 
-        if not containers:
-            check_for_legacy_containers(
-                self.client,
-                self.name,
-                self.service_names,
-            )
+        return [c for c in containers if matches_service_names(c)]
 
-        return filter(matches_service_names, containers)
+    def find_orphan_containers(self, remove_orphans):
+        def _find():
+            containers = self._labeled_containers()
+            for ctnr in containers:
+                service_name = ctnr.labels.get(LABEL_SERVICE)
+                if service_name not in self.service_names:
+                    yield ctnr
+        orphans = list(_find())
+        if not orphans:
+            return
+        if remove_orphans:
+            for ctnr in orphans:
+                log.info('Removing orphan container "{0}"'.format(ctnr.name))
+                ctnr.kill()
+                ctnr.remove(force=True)
+        else:
+            log.warning(
+                'Found orphan containers ({0}) for this project. If '
+                'you removed or renamed this service in your compose '
+                'file, you can run this command with the '
+                '--remove-orphans flag to clean it up.'.format(
+                    ', '.join(["{}".format(ctnr.name) for ctnr in orphans])
+                )
+            )
 
     def _inject_deps(self, acc, service):
         dep_names = service.get_dependency_names()
@@ -351,6 +522,85 @@ class Project(object):
         dep_services.append(service)
         return acc + dep_services
 
+    def build_container_operation_with_timeout_func(self, operation, options):
+        def container_operation_with_timeout(container):
+            if options.get('timeout') is None:
+                service = self.get_service(container.service)
+                options['timeout'] = service.stop_timeout(None)
+            return getattr(container, operation)(**options)
+        return container_operation_with_timeout
+
+
+def get_volumes_from(project, service_dict):
+    volumes_from = service_dict.pop('volumes_from', None)
+    if not volumes_from:
+        return []
+
+    def build_volume_from(spec):
+        if spec.type == 'service':
+            try:
+                return spec._replace(source=project.get_service(spec.source))
+            except NoSuchService:
+                pass
+
+        if spec.type == 'container':
+            try:
+                container = Container.from_id(project.client, spec.source)
+                return spec._replace(source=container)
+            except APIError:
+                pass
+
+        raise ConfigurationError(
+            "Service \"{}\" mounts volumes from \"{}\", which is not the name "
+            "of a service or container.".format(
+                service_dict['name'],
+                spec.source))
+
+    return [build_volume_from(vf) for vf in volumes_from]
+
+
+def get_secrets(service, service_secrets, secret_defs):
+    secrets = []
+
+    for secret in service_secrets:
+        secret_def = secret_defs.get(secret.source)
+        if not secret_def:
+            raise ConfigurationError(
+                "Service \"{service}\" uses an undefined secret \"{secret}\" "
+                .format(service=service, secret=secret.source))
+
+        if secret_def.get('external_name'):
+            log.warn("Service \"{service}\" uses secret \"{secret}\" which is external. "
+                     "External secrets are not available to containers created by "
+                     "docker-compose.".format(service=service, secret=secret.source))
+            continue
+
+        if secret.uid or secret.gid or secret.mode:
+            log.warn("Service \"{service}\" uses secret \"{secret}\" with uid, "
+                     "gid, or mode. These fields are not supported by this "
+                     "implementation of the Compose file".format(
+                        service=service, secret=secret.source))
+
+        secrets.append({'secret': secret, 'file': secret_def.get('file')})
+
+    return secrets
+
+
+def warn_for_swarm_mode(client):
+    info = client.info()
+    if info.get('Swarm', {}).get('LocalNodeState') == 'active':
+        if info.get('ServerVersion', '').startswith('ucp'):
+            # UCP does multi-node scheduling with traditional Compose files.
+            return
+
+        log.warn(
+            "The Docker Engine you're using is running in swarm mode.\n\n"
+            "Compose does not use swarm mode to deploy services to multiple nodes in a swarm. "
+            "All containers will be scheduled on the current node.\n\n"
+            "To deploy your application across the swarm, "
+            "use `docker stack deploy`.\n"
+        )
+
 
 class NoSuchService(Exception):
     def __init__(self, name):
@@ -361,5 +611,6 @@ class NoSuchService(Exception):
         return self.msg
 
 
-class DependencyError(ConfigurationError):
-    pass
+class ProjectError(Exception):
+    def __init__(self, msg):
+        self.msg = msg

compose/timeparse.py

@@ -0,0 +1,96 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+'''
+timeparse.py
+(c) Will Roberts <wildwilhelm@gmail.com>  1 February, 2014
+
+This is a vendored and modified copy of:
+github.com/wroberts/pytimeparse @ cc0550d
+
+It has been modified to mimic the behaviour of
+https://golang.org/pkg/time/#ParseDuration
+'''
+# MIT LICENSE
+#
+# Permission is hereby granted, free of charge, to any person
+# obtaining a copy of this software and associated documentation files
+# (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge,
+# publish, distribute, sublicense, and/or sell copies of the Software,
+# and to permit persons to whom the Software is furnished to do so,
+# subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+# BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import re
+
+HOURS = r'(?P<hours>[\d.]+)h'
+MINS = r'(?P<mins>[\d.]+)m'
+SECS = r'(?P<secs>[\d.]+)s'
+MILLI = r'(?P<milli>[\d.]+)ms'
+MICRO = r'(?P<micro>[\d.]+)(?:us|µs)'
+NANO = r'(?P<nano>[\d.]+)ns'
+
+
+def opt(x):
+    return r'(?:{x})?'.format(x=x)
+
+
+TIMEFORMAT = r'{HOURS}{MINS}{SECS}{MILLI}{MICRO}{NANO}'.format(
+    HOURS=opt(HOURS),
+    MINS=opt(MINS),
+    SECS=opt(SECS),
+    MILLI=opt(MILLI),
+    MICRO=opt(MICRO),
+    NANO=opt(NANO),
+)
+
+MULTIPLIERS = dict([
+    ('hours',   60 * 60),
+    ('mins',    60),
+    ('secs',    1),
+    ('milli',   1.0 / 1000),
+    ('micro',   1.0 / 1000.0 / 1000),
+    ('nano',    1.0 / 1000.0 / 1000.0 / 1000.0),
+])
+
+
+def timeparse(sval):
+    """Parse a time expression, returning it as a number of seconds.  If
+    possible, the return value will be an `int`; if this is not
+    possible, the return will be a `float`.  Returns `None` if a time
+    expression cannot be parsed from the given string.
+
+    Arguments:
+    - `sval`: the string value to parse
+
+    >>> timeparse('1m24s')
+    84
+    >>> timeparse('1.2 minutes')
+    72
+    >>> timeparse('1.2 seconds')
+    1.2
+    """
+    match = re.match(r'\s*' + TIMEFORMAT + r'\s*$', sval, re.I)
+    if not match or not match.group(0).strip():
+        return
+
+    mdict = match.groupdict()
+    return sum(
+        MULTIPLIERS[k] * cast(v) for (k, v) in mdict.items() if v is not None)
+
+
+def cast(value):
+    return int(value, 10) if value.isdigit() else float(value)

compose/utils.py

@@ -1,100 +1,135 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
 import codecs
 import hashlib
 import json
+import json.decoder
 import logging
-import sys
+import ntpath
 
-from docker.errors import APIError
-from Queue import Queue, Empty
-from threading import Thread
+import six
+
+from .errors import StreamParseError
+from .timeparse import timeparse
 
 
+json_decoder = json.JSONDecoder()
 log = logging.getLogger(__name__)
 
 
-def parallel_execute(objects, obj_callable, msg_index, msg):
+def get_output_stream(stream):
+    if six.PY3:
+        return stream
+    return codecs.getwriter('utf-8')(stream)
+
+
+def stream_as_text(stream):
+    """Given a stream of bytes or text, if any of the items in the stream
+    are bytes convert them to text.
+
+    This function can be removed once docker-py returns text streams instead
+    of byte streams.
     """
-    For a given list of objects, call the callable passing in the first
-    object we give it.
+    for data in stream:
+        if not isinstance(data, six.text_type):
+            data = data.decode('utf-8', 'replace')
+        yield data
+
+
+def line_splitter(buffer, separator=u'\n'):
+    index = buffer.find(six.text_type(separator))
+    if index == -1:
+        return None
+    return buffer[:index + 1], buffer[index + 1:]
+
+
+def split_buffer(stream, splitter=None, decoder=lambda a: a):
+    """Given a generator which yields strings and a splitter function,
+    joins all input, splits on the separator and yields each chunk.
+
+    Unlike string.split(), each chunk includes the trailing
+    separator, except for the last one if none was found on the end
+    of the input.
     """
-    stream = codecs.getwriter('utf-8')(sys.stdout)
-    lines = []
-    errors = {}
+    splitter = splitter or line_splitter
+    buffered = six.text_type('')
 
-    for obj in objects:
-        write_out_msg(stream, lines, msg_index(obj), msg)
+    for data in stream_as_text(stream):
+        buffered += data
+        while True:
+            buffer_split = splitter(buffered)
+            if buffer_split is None:
+                break
 
-    q = Queue()
+            item, buffered = buffer_split
+            yield item
 
-    def inner_execute_function(an_callable, parameter, msg_index):
+    if buffered:
         try:
-            result = an_callable(parameter)
-        except APIError as e:
-            errors[msg_index] = e.explanation
-            result = "error"
+            yield decoder(buffered)
         except Exception as e:
-            errors[msg_index] = e
-            result = 'unexpected_exception'
-
-        q.put((msg_index, result))
-
-    for an_object in objects:
-        t = Thread(
-            target=inner_execute_function,
-            args=(obj_callable, an_object, msg_index(an_object)),
-        )
-        t.daemon = True
-        t.start()
-
-    done = 0
-    total_to_execute = len(objects)
-
-    while done < total_to_execute:
-        try:
-            msg_index, result = q.get(timeout=1)
-
-            if result == 'unexpected_exception':
-                raise errors[msg_index]
-            if result == 'error':
-                write_out_msg(stream, lines, msg_index, msg, status='error')
-            else:
-                write_out_msg(stream, lines, msg_index, msg)
-            done += 1
-        except Empty:
-            pass
-
-    if errors:
-        stream.write("\n")
-        for error in errors:
-            stream.write("ERROR: for {}  {} \n".format(error, errors[error]))
+            log.error(
+                'Compose tried decoding the following data chunk, but failed:'
+                '\n%s' % repr(buffered)
+            )
+            raise StreamParseError(e)
 
 
-def write_out_msg(stream, lines, msg_index, msg, status="done"):
+def json_splitter(buffer):
+    """Attempt to parse a json object from a buffer. If there is at least one
+    object, return it and the rest of the buffer, otherwise return None.
     """
-    Using special ANSI code characters we can write out the msg over the top of
-    a previous status message, if it exists.
-    """
-    obj_index = msg_index
-    if msg_index in lines:
-        position = lines.index(obj_index)
-        diff = len(lines) - position
-        # move up
-        stream.write("%c[%dA" % (27, diff))
-        # erase
-        stream.write("%c[2K\r" % 27)
-        stream.write("{} {}... {}\n".format(msg, obj_index, status))
-        # move back down
-        stream.write("%c[%dB" % (27, diff))
-    else:
-        diff = 0
-        lines.append(obj_index)
-        stream.write("{} {}... \r\n".format(msg, obj_index))
+    buffer = buffer.strip()
+    try:
+        obj, index = json_decoder.raw_decode(buffer)
+        rest = buffer[json.decoder.WHITESPACE.match(buffer, index).end():]
+        return obj, rest
+    except ValueError:
+        return None
 
-    stream.flush()
+
+def json_stream(stream):
+    """Given a stream of text, return a stream of json objects.
+    This handles streams which are inconsistently buffered (some entries may
+    be newline delimited, and others are not).
+    """
+    return split_buffer(stream, json_splitter, json_decoder.decode)
 
 
 def json_hash(obj):
     dump = json.dumps(obj, sort_keys=True, separators=(',', ':'))
     h = hashlib.sha256()
-    h.update(dump)
+    h.update(dump.encode('utf8'))
     return h.hexdigest()
+
+
+def microseconds_from_time_nano(time_nano):
+    return int(time_nano % 1000000000 / 1000)
+
+
+def nanoseconds_from_time_seconds(time_seconds):
+    return time_seconds * 1000000000
+
+
+def parse_seconds_float(value):
+    return timeparse(value or '')
+
+
+def parse_nanoseconds_int(value):
+    parsed = timeparse(value or '')
+    if parsed is None:
+        return None
+    return int(parsed * 1000000000)
+
+
+def build_string_dict(source_dict):
+    return dict((k, str(v if v is not None else '')) for k, v in source_dict.items())
+
+
+def splitdrive(path):
+    if len(path) == 0:
+        return ('', '')
+    if path[0] in ['.', '\\', '/', '~']:
+        return ('', path)
+    return ntpath.splitdrive(path)

compose/volume.py

@@ -0,0 +1,151 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import logging
+
+from docker.errors import NotFound
+from docker.utils import version_lt
+
+from .config import ConfigurationError
+from .const import LABEL_PROJECT
+from .const import LABEL_VOLUME
+
+log = logging.getLogger(__name__)
+
+
+class Volume(object):
+    def __init__(self, client, project, name, driver=None, driver_opts=None,
+                 external_name=None, labels=None):
+        self.client = client
+        self.project = project
+        self.name = name
+        self.driver = driver
+        self.driver_opts = driver_opts
+        self.external_name = external_name
+        self.labels = labels
+
+    def create(self):
+        return self.client.create_volume(
+            self.full_name, self.driver, self.driver_opts, labels=self._labels
+        )
+
+    def remove(self):
+        if self.external:
+            log.info("Volume %s is external, skipping", self.full_name)
+            return
+        log.info("Removing volume %s", self.full_name)
+        return self.client.remove_volume(self.full_name)
+
+    def inspect(self):
+        return self.client.inspect_volume(self.full_name)
+
+    def exists(self):
+        try:
+            self.inspect()
+        except NotFound:
+            return False
+        return True
+
+    @property
+    def external(self):
+        return bool(self.external_name)
+
+    @property
+    def full_name(self):
+        if self.external_name:
+            return self.external_name
+        return '{0}_{1}'.format(self.project, self.name)
+
+    @property
+    def _labels(self):
+        if version_lt(self.client._version, '1.23'):
+            return None
+        labels = self.labels.copy() if self.labels else {}
+        labels.update({
+            LABEL_PROJECT: self.project,
+            LABEL_VOLUME: self.name,
+        })
+        return labels
+
+
+class ProjectVolumes(object):
+
+    def __init__(self, volumes):
+        self.volumes = volumes
+
+    @classmethod
+    def from_config(cls, name, config_data, client):
+        config_volumes = config_data.volumes or {}
+        volumes = {
+            vol_name: Volume(
+                client=client,
+                project=name,
+                name=vol_name,
+                driver=data.get('driver'),
+                driver_opts=data.get('driver_opts'),
+                external_name=data.get('external_name'),
+                labels=data.get('labels')
+            )
+            for vol_name, data in config_volumes.items()
+        }
+        return cls(volumes)
+
+    def remove(self):
+        for volume in self.volumes.values():
+            try:
+                volume.remove()
+            except NotFound:
+                log.warn("Volume %s not found.", volume.full_name)
+
+    def initialize(self):
+        try:
+            for volume in self.volumes.values():
+                volume_exists = volume.exists()
+                if volume.external:
+                    log.debug(
+                        'Volume {0} declared as external. No new '
+                        'volume will be created.'.format(volume.name)
+                    )
+                    if not volume_exists:
+                        raise ConfigurationError(
+                            'Volume {name} declared as external, but could'
+                            ' not be found. Please create the volume manually'
+                            ' using `{command}{name}` and try again.'.format(
+                                name=volume.full_name,
+                                command='docker volume create --name='
+                            )
+                        )
+                    continue
+
+                if not volume_exists:
+                    log.info(
+                        'Creating volume "{0}" with {1} driver'.format(
+                            volume.full_name, volume.driver or 'default'
+                        )
+                    )
+                    volume.create()
+                else:
+                    driver = volume.inspect()['Driver']
+                    if volume.driver is not None and driver != volume.driver:
+                        raise ConfigurationError(
+                            'Configuration for volume {0} specifies driver '
+                            '{1}, but a volume with the same name uses a '
+                            'different driver ({3}). If you wish to use the '
+                            'new configuration, please remove the existing '
+                            'volume "{2}" first:\n'
+                            '$ docker volume rm {2}'.format(
+                                volume.name, volume.driver, volume.full_name,
+                                volume.inspect()['Driver']
+                            )
+                        )
+        except NotFound:
+            raise ConfigurationError(
+                'Volume %s specifies nonexistent driver %s' % (volume.name, volume.driver)
+            )
+
+    def namespace_spec(self, volume_spec):
+        if not volume_spec.is_named_volume:
+            return volume_spec
+
+        volume = self.volumes[volume_spec.external]
+        return volume_spec._replace(external=volume.full_name)

contrib/completion/bash/docker-compose

@@ -17,94 +17,145 @@
 #    . ~/.docker-compose-completion.sh
 
 
-# For compatibility reasons, Compose and therefore its completion supports several
-# stack compositon files as listed here, in descending priority.
-# Support for these filenames might be dropped in some future version.
-__docker-compose_compose_file() {
-	local file
-	for file in docker-compose.y{,a}ml fig.y{,a}ml ; do
-		[ -e $file ] && {
-			echo $file
-			return
-		}
-	done
-	echo docker-compose.yml
+__docker_compose_q() {
+	docker-compose 2>/dev/null $daemon_options "$@"
+}
+
+# Transforms a multiline list of strings into a single line string
+# with the words separated by "|".
+__docker_compose_to_alternatives() {
+	local parts=( $1 )
+	local IFS='|'
+	echo "${parts[*]}"
+}
+
+# Transforms a multiline list of options into an extglob pattern
+# suitable for use in case statements.
+__docker_compose_to_extglob() {
+	local extglob=$( __docker_compose_to_alternatives "$1" )
+	echo "@($extglob)"
+}
+
+# suppress trailing whitespace
+__docker_compose_nospace() {
+	# compopt is not available in ancient bash versions
+	type compopt &>/dev/null && compopt -o nospace
 }
 
 # Extracts all service names from the compose file.
-___docker-compose_all_services_in_compose_file() {
-	awk -F: '/^[a-zA-Z0-9]/{print $1}' "${compose_file:-$(__docker-compose_compose_file)}" 2>/dev/null
+___docker_compose_all_services_in_compose_file() {
+	__docker_compose_q config --services
 }
 
 # All services, even those without an existing container
-__docker-compose_services_all() {
-	COMPREPLY=( $(compgen -W "$(___docker-compose_all_services_in_compose_file)" -- "$cur") )
+__docker_compose_services_all() {
+	COMPREPLY=( $(compgen -W "$(___docker_compose_all_services_in_compose_file)" -- "$cur") )
 }
 
 # All services that have an entry with the given key in their compose_file section
-___docker-compose_services_with_key() {
-	# flatten sections to one line, then filter lines containing the key and return section name.
-	awk '/^[a-zA-Z0-9]/{printf "\n"};{printf $0;next;}' "${compose_file:-$(__docker-compose_compose_file)}" | awk -F: -v key=": +$1:" '$0 ~ key {print $1}'
+___docker_compose_services_with_key() {
+	# flatten sections under "services" to one line, then filter lines containing the key and return section name
+	__docker_compose_q config \
+		| sed -n -e '/^services:/,/^[^ ]/p' \
+		| sed -n 's/^  //p' \
+		| awk '/^[a-zA-Z0-9]/{printf "\n"};{printf $0;next;}' \
+		| awk -F: -v key=": +$1:" '$0 ~ key {print $1}'
 }
 
 # All services that are defined by a Dockerfile reference
-__docker-compose_services_from_build() {
-	COMPREPLY=( $(compgen -W "$(___docker-compose_services_with_key build)" -- "$cur") )
+__docker_compose_services_from_build() {
+	COMPREPLY=( $(compgen -W "$(___docker_compose_services_with_key build)" -- "$cur") )
 }
 
 # All services that are defined by an image
-__docker-compose_services_from_image() {
-	COMPREPLY=( $(compgen -W "$(___docker-compose_services_with_key image)" -- "$cur") )
+__docker_compose_services_from_image() {
+	COMPREPLY=( $(compgen -W "$(___docker_compose_services_with_key image)" -- "$cur") )
 }
 
 # The services for which containers have been created, optionally filtered
 # by a boolean expression passed in as argument.
-__docker-compose_services_with() {
+__docker_compose_services_with() {
 	local containers names
-	containers="$(docker-compose 2>/dev/null ${compose_file:+-f $compose_file} ${compose_project:+-p $compose_project} ps -q)"
-	names=( $(docker 2>/dev/null inspect --format "{{if ${1:-true}}} {{ .Name }} {{end}}" $containers) )
-	names=( ${names[@]%_*} )  # strip trailing numbers
-	names=( ${names[@]#*_} )  # strip project name
-	COMPREPLY=( $(compgen -W "${names[*]}" -- "$cur") )
+	containers="$(__docker_compose_q ps -q)"
+	names=$(docker 2>/dev/null inspect -f "{{if ${1:-true}}}{{range \$k, \$v := .Config.Labels}}{{if eq \$k \"com.docker.compose.service\"}}{{\$v}}{{end}}{{end}}{{end}}" $containers)
+	COMPREPLY=( $(compgen -W "$names" -- "$cur") )
+}
+
+# The services for which at least one paused container exists
+__docker_compose_services_paused() {
+	__docker_compose_services_with '.State.Paused'
 }
 
 # The services for which at least one running container exists
-__docker-compose_services_running() {
-	__docker-compose_services_with '.State.Running'
+__docker_compose_services_running() {
+	__docker_compose_services_with '.State.Running'
 }
 
 # The services for which at least one stopped container exists
-__docker-compose_services_stopped() {
-	__docker-compose_services_with 'not .State.Running'
+__docker_compose_services_stopped() {
+	__docker_compose_services_with 'not .State.Running'
 }
 
 
-_docker-compose_build() {
+_docker_compose_build() {
 	case "$cur" in
 		-*)
-			COMPREPLY=( $( compgen -W "--help --no-cache" -- "$cur" ) )
+			COMPREPLY=( $( compgen -W "--force-rm --help --no-cache --pull" -- "$cur" ) )
 			;;
 		*)
-			__docker-compose_services_from_build
+			__docker_compose_services_from_build
 			;;
 	esac
 }
 
 
-_docker-compose_docker-compose() {
+_docker_compose_bundle() {
 	case "$prev" in
+		--output|-o)
+			_filedir
+			return
+			;;
+	esac
+
+	COMPREPLY=( $( compgen -W "--push-images --help --output -o" -- "$cur" ) )
+}
+
+
+_docker_compose_config() {
+	COMPREPLY=( $( compgen -W "--help --quiet -q --services" -- "$cur" ) )
+}
+
+
+_docker_compose_create() {
+	case "$cur" in
+		-*)
+			COMPREPLY=( $( compgen -W "--force-recreate --help --no-build --no-recreate" -- "$cur" ) )
+			;;
+		*)
+			__docker_compose_services_all
+			;;
+	esac
+}
+
+
+_docker_compose_docker_compose() {
+	case "$prev" in
+		--tlscacert|--tlscert|--tlskey)
+			_filedir
+			return
+			;;
 		--file|-f)
 			_filedir "y?(a)ml"
 			return
 			;;
-		--project-name|-p)
+		$(__docker_compose_to_extglob "$daemon_options_with_args") )
 			return
 			;;
 	esac
 
 	case "$cur" in
 		-*)
-			COMPREPLY=( $( compgen -W "--help -h --verbose --version -v --file -f --project-name -p" -- "$cur" ) )
+			COMPREPLY=( $( compgen -W "$daemon_boolean_options  $daemon_options_with_args --help -h --verbose --version -v" -- "$cur" ) )
 			;;
 		*)
 			COMPREPLY=( $( compgen -W "${commands[*]}" -- "$cur" ) )
@@ -113,12 +164,64 @@ _docker-compose_docker-compose() {
 }
 
 
-_docker-compose_help() {
+_docker_compose_down() {
+	case "$prev" in
+		--rmi)
+			COMPREPLY=( $( compgen -W "all local" -- "$cur" ) )
+			return
+			;;
+	esac
+
+	case "$cur" in
+		-*)
+			COMPREPLY=( $( compgen -W "--help --rmi --volumes -v --remove-orphans" -- "$cur" ) )
+			;;
+	esac
+}
+
+
+_docker_compose_events() {
+	case "$prev" in
+		--json)
+			return
+			;;
+	esac
+
+	case "$cur" in
+		-*)
+			COMPREPLY=( $( compgen -W "--help --json" -- "$cur" ) )
+			;;
+		*)
+			__docker_compose_services_all
+			;;
+	esac
+}
+
+
+_docker_compose_exec() {
+	case "$prev" in
+		--index|--user)
+			return
+			;;
+	esac
+
+	case "$cur" in
+		-*)
+			COMPREPLY=( $( compgen -W "-d --help --index --privileged -T --user" -- "$cur" ) )
+			;;
+		*)
+			__docker_compose_services_running
+			;;
+	esac
+}
+
+
+_docker_compose_help() {
 	COMPREPLY=( $( compgen -W "${commands[*]}" -- "$cur" ) )
 }
 
 
-_docker-compose_kill() {
+_docker_compose_kill() {
 	case "$prev" in
 		-s)
 			COMPREPLY=( $( compgen -W "SIGHUP SIGINT SIGKILL SIGUSR1 SIGUSR2" -- "$(echo $cur | tr '[:lower:]' '[:upper:]')" ) )
@@ -131,34 +234,43 @@ _docker-compose_kill() {
 			COMPREPLY=( $( compgen -W "--help -s" -- "$cur" ) )
 			;;
 		*)
-			__docker-compose_services_running
+			__docker_compose_services_running
 			;;
 	esac
 }
 
 
-_docker-compose_logs() {
+_docker_compose_logs() {
+	case "$prev" in
+		--tail)
+			return
+			;;
+	esac
+
 	case "$cur" in
 		-*)
-			COMPREPLY=( $( compgen -W "--help --no-color" -- "$cur" ) )
+			COMPREPLY=( $( compgen -W "--follow -f --help --no-color --tail --timestamps -t" -- "$cur" ) )
 			;;
 		*)
-			__docker-compose_services_all
+			__docker_compose_services_all
 			;;
 	esac
 }
 
 
-_docker-compose_migrate-to-labels() {
+_docker_compose_pause() {
 	case "$cur" in
 		-*)
 			COMPREPLY=( $( compgen -W "--help" -- "$cur" ) )
 			;;
+		*)
+			__docker_compose_services_running
+			;;
 	esac
 }
 
 
-_docker-compose_port() {
+_docker_compose_port() {
 	case "$prev" in
 		--protocol)
 			COMPREPLY=( $( compgen -W "tcp udp" -- "$cur" ) )
@@ -174,39 +286,51 @@ _docker-compose_port() {
 			COMPREPLY=( $( compgen -W "--help --index --protocol" -- "$cur" ) )
 			;;
 		*)
-			__docker-compose_services_all
+			__docker_compose_services_all
 			;;
 	esac
 }
 
 
-_docker-compose_ps() {
+_docker_compose_ps() {
 	case "$cur" in
 		-*)
 			COMPREPLY=( $( compgen -W "--help -q" -- "$cur" ) )
 			;;
 		*)
-			__docker-compose_services_all
+			__docker_compose_services_all
 			;;
 	esac
 }
 
 
-_docker-compose_pull() {
+_docker_compose_pull() {
 	case "$cur" in
 		-*)
-			COMPREPLY=( $( compgen -W "--help" -- "$cur" ) )
+			COMPREPLY=( $( compgen -W "--help --ignore-pull-failures" -- "$cur" ) )
 			;;
 		*)
-			__docker-compose_services_from_image
+			__docker_compose_services_from_image
 			;;
 	esac
 }
 
 
-_docker-compose_restart() {
+_docker_compose_push() {
+	case "$cur" in
+		-*)
+			COMPREPLY=( $( compgen -W "--help --ignore-push-failures" -- "$cur" ) )
+			;;
+		*)
+			__docker_compose_services_all
+			;;
+	esac
+}
+
+
+_docker_compose_restart() {
 	case "$prev" in
-		-t | --timeout)
+		--timeout|-t)
 			return
 			;;
 	esac
@@ -216,81 +340,54 @@ _docker-compose_restart() {
 			COMPREPLY=( $( compgen -W "--help --timeout -t" -- "$cur" ) )
 			;;
 		*)
-			__docker-compose_services_running
+			__docker_compose_services_running
 			;;
 	esac
 }
 
 
-_docker-compose_rm() {
+_docker_compose_rm() {
 	case "$cur" in
 		-*)
 			COMPREPLY=( $( compgen -W "--force -f --help -v" -- "$cur" ) )
 			;;
 		*)
-			__docker-compose_services_stopped
+			__docker_compose_services_stopped
 			;;
 	esac
 }
 
 
-_docker-compose_run() {
+_docker_compose_run() {
 	case "$prev" in
 		-e)
 			COMPREPLY=( $( compgen -e -- "$cur" ) )
-			compopt -o nospace
+			__docker_compose_nospace
 			return
 			;;
-		--entrypoint|--user|-u)
+		--entrypoint|--name|--user|-u|--workdir|-w)
 			return
 			;;
 	esac
 
 	case "$cur" in
 		-*)
-			COMPREPLY=( $( compgen -W "-d --entrypoint -e --help --no-deps --rm --service-ports -T --user -u" -- "$cur" ) )
+			COMPREPLY=( $( compgen -W "-d --entrypoint -e --help --name --no-deps --publish -p --rm --service-ports -T --user -u --workdir -w" -- "$cur" ) )
 			;;
 		*)
-			__docker-compose_services_all
+			__docker_compose_services_all
 			;;
 	esac
 }
 
 
-_docker-compose_scale() {
+_docker_compose_scale() {
 	case "$prev" in
 		=)
 			COMPREPLY=("$cur")
+			return
 			;;
-		*)
-			COMPREPLY=( $(compgen -S "=" -W "$(___docker-compose_all_services_in_compose_file)" -- "$cur") )
-			compopt -o nospace
-			;;
-	esac
-
-	case "$cur" in
-		-*)
-			COMPREPLY=( $( compgen -W "--help" -- "$cur" ) )
-			;;
-	esac
-}
-
-
-_docker-compose_start() {
-	case "$cur" in
-		-*)
-			COMPREPLY=( $( compgen -W "--help" -- "$cur" ) )
-			;;
-		*)
-			__docker-compose_services_stopped
-			;;
-	esac
-}
-
-
-_docker-compose_stop() {
-	case "$prev" in
-		-t | --timeout)
+		--timeout|-t)
 			return
 			;;
 	esac
@@ -300,31 +397,86 @@ _docker-compose_stop() {
 			COMPREPLY=( $( compgen -W "--help --timeout -t" -- "$cur" ) )
 			;;
 		*)
-			__docker-compose_services_running
+			COMPREPLY=( $(compgen -S "=" -W "$(___docker_compose_all_services_in_compose_file)" -- "$cur") )
+			__docker_compose_nospace
 			;;
 	esac
 }
 
 
-_docker-compose_up() {
+_docker_compose_start() {
+	case "$cur" in
+		-*)
+			COMPREPLY=( $( compgen -W "--help" -- "$cur" ) )
+			;;
+		*)
+			__docker_compose_services_stopped
+			;;
+	esac
+}
+
+
+_docker_compose_stop() {
 	case "$prev" in
-		-t | --timeout)
+		--timeout|-t)
 			return
 			;;
 	esac
 
 	case "$cur" in
 		-*)
-			COMPREPLY=( $( compgen -W "-d --help --no-build --no-color --no-deps --no-recreate --force-recreate --timeout -t" -- "$cur" ) )
+			COMPREPLY=( $( compgen -W "--help --timeout -t" -- "$cur" ) )
 			;;
 		*)
-			__docker-compose_services_all
+			__docker_compose_services_running
 			;;
 	esac
 }
 
 
-_docker-compose_version() {
+_docker_compose_top() {
+	case "$cur" in
+		-*)
+			COMPREPLY=( $( compgen -W "--help" -- "$cur" ) )
+			;;
+		*)
+			__docker_compose_services_running
+			;;
+	esac
+}
+
+
+_docker_compose_unpause() {
+	case "$cur" in
+		-*)
+			COMPREPLY=( $( compgen -W "--help" -- "$cur" ) )
+			;;
+		*)
+			__docker_compose_services_paused
+			;;
+	esac
+}
+
+
+_docker_compose_up() {
+	case "$prev" in
+		--timeout|-t)
+			return
+			;;
+	esac
+
+	case "$cur" in
+		-*)
+			COMPREPLY=( $( compgen -W "--abort-on-container-exit --build -d --force-recreate --help --no-build --no-color --no-deps --no-recreate --timeout -t --remove-orphans" -- "$cur" ) )
+			;;
+		*)
+			__docker_compose_services_all
+			;;
+	esac
+}
+
+
+_docker_compose_version() {
 	case "$cur" in
 		-*)
 			COMPREPLY=( $( compgen -W "--short" -- "$cur" ) )
@@ -333,47 +485,74 @@ _docker-compose_version() {
 }
 
 
-_docker-compose() {
+_docker_compose() {
 	local previous_extglob_setting=$(shopt -p extglob)
 	shopt -s extglob
 
 	local commands=(
 		build
+		bundle
+		config
+		create
+		down
+		events
+		exec
 		help
 		kill
 		logs
-		migrate-to-labels
+		pause
 		port
 		ps
 		pull
+		push
 		restart
 		rm
 		run
 		scale
 		start
 		stop
+		top
+		unpause
 		up
 		version
 	)
 
+	# options for the docker daemon that have to be passed to secondary calls to
+	# docker-compose executed by this script
+	local daemon_boolean_options="
+		--skip-hostname-check
+		--tls
+		--tlsverify
+	"
+	local daemon_options_with_args="
+		--file -f
+		--host -H
+		--project-name -p
+		--tlscacert
+		--tlscert
+		--tlskey
+	"
+
 	COMPREPLY=()
 	local cur prev words cword
 	_get_comp_words_by_ref -n : cur prev words cword
 
 	# search subcommand and invoke its handler.
 	# special treatment of some top-level options
-	local command='docker-compose'
+	local command='docker_compose'
+	local daemon_options=()
 	local counter=1
-	local compose_file compose_project
+
 	while [ $counter -lt $cword ]; do
 		case "${words[$counter]}" in
-			-f|--file)
-				(( counter++ ))
-				compose_file="${words[$counter]}"
+			$(__docker_compose_to_extglob "$daemon_boolean_options") )
+				local opt=${words[counter]}
+				daemon_options+=($opt)
 				;;
-			-p|--project-name)
-				(( counter++ ))
-				compose_project="${words[$counter]}"
+			$(__docker_compose_to_extglob "$daemon_options_with_args") )
+				local opt=${words[counter]}
+				local arg=${words[++counter]}
+				daemon_options+=($opt $arg)
 				;;
 			-*)
 				;;
@@ -385,11 +564,11 @@ _docker-compose() {
 		(( counter++ ))
 	done
 
-	local completions_func=_docker-compose_${command}
+	local completions_func=_docker_compose_${command//-/_}
 	declare -F $completions_func >/dev/null && $completions_func
 
 	eval "$previous_extglob_setting"
 	return 0
 }
 
-complete -F _docker-compose docker-compose
+complete -F _docker_compose docker-compose

contrib/completion/zsh/_docker-compose

@@ -7,7 +7,7 @@
 # -------------------------------------------------------------------------
 # Version
 # -------
-#  0.1.0
+#  1.5.0
 # -------------------------------------------------------------------------
 # Authors
 # -------
@@ -19,58 +19,69 @@
 #  * @felixr docker zsh completion script : https://github.com/felixr/docker-zsh-completion
 # -------------------------------------------------------------------------
 
-# For compatibility reasons, Compose and therefore its completion supports several
-# stack compositon files as listed here, in descending priority.
-# Support for these filenames might be dropped in some future version.
-__docker-compose_compose_file() {
-    local file
-    for file in docker-compose.y{,a}ml fig.y{,a}ml ; do
-        [ -e $file ] && {
-            echo $file
-            return
-        }
-    done
-    echo docker-compose.yml
+__docker-compose_q() {
+    docker-compose 2>/dev/null $compose_options "$@"
 }
 
-# Extracts all service names from docker-compose.yml.
-___docker-compose_all_services_in_compose_file() {
+# All services defined in docker-compose.yml
+__docker-compose_all_services_in_compose_file() {
     local already_selected
     local -a services
-    already_selected=$(echo ${words[@]} | tr " " "|")
-    awk -F: '/^[a-zA-Z0-9]/{print $1}' "${compose_file:-$(__docker-compose_compose_file)}" 2>/dev/null | grep -Ev "$already_selected"
+    already_selected=$(echo $words | tr " " "|")
+    __docker-compose_q config --services \
+        | grep -Ev "^(${already_selected})$"
 }
 
 # All services, even those without an existing container
 __docker-compose_services_all() {
-    services=$(___docker-compose_all_services_in_compose_file)
-    _alternative "args:services:($services)"
+    [[ $PREFIX = -* ]] && return 1
+    integer ret=1
+    services=$(__docker-compose_all_services_in_compose_file)
+    _alternative "args:services:($services)" && ret=0
+
+    return ret
 }
 
 # All services that have an entry with the given key in their docker-compose.yml section
-___docker-compose_services_with_key() {
+__docker-compose_services_with_key() {
     local already_selected
     local -a buildable
-    already_selected=$(echo ${words[@]} | tr " " "|")
+    already_selected=$(echo $words | tr " " "|")
     # flatten sections to one line, then filter lines containing the key and return section name.
-    awk '/^[a-zA-Z0-9]/{printf "\n"};{printf $0;next;}' "${compose_file:-$(__docker-compose_compose_file)}" 2>/dev/null | awk -F: -v key=": +$1:" '$0 ~ key {print $1}' 2>/dev/null | grep -Ev "$already_selected"
+    __docker-compose_q config \
+        | sed -n -e '/^services:/,/^[^ ]/p' \
+        | sed -n 's/^  //p' \
+        | awk '/^[a-zA-Z0-9]/{printf "\n"};{printf $0;next;}' \
+        | grep " \+$1:" \
+        | cut -d: -f1 \
+        | grep -Ev "^(${already_selected})$"
 }
 
 # All services that are defined by a Dockerfile reference
 __docker-compose_services_from_build() {
-    buildable=$(___docker-compose_services_with_key build)
-    _alternative "args:buildable services:($buildable)"
+    [[ $PREFIX = -* ]] && return 1
+    integer ret=1
+    buildable=$(__docker-compose_services_with_key build)
+    _alternative "args:buildable services:($buildable)" && ret=0
+
+   return ret
 }
 
 # All services that are defined by an image
 __docker-compose_services_from_image() {
-    pullable=$(___docker-compose_services_with_key image)
-    _alternative "args:pullable services:($pullable)"
+    [[ $PREFIX = -* ]] && return 1
+    integer ret=1
+    pullable=$(__docker-compose_services_with_key image)
+    _alternative "args:pullable services:($pullable)" && ret=0
+
+    return ret
 }
 
 __docker-compose_get_services() {
-    local kind expl
-    declare -a running stopped lines args services
+    [[ $PREFIX = -* ]] && return 1
+    integer ret=1
+    local kind
+    declare -a running paused stopped lines args services
 
     docker_status=$(docker ps > /dev/null 2>&1)
     if [ $? -ne 0 ]; then
@@ -80,64 +91,78 @@ __docker-compose_get_services() {
 
     kind=$1
     shift
-    [[ $kind = (stopped|all) ]] && args=($args -a)
+    [[ $kind =~ (stopped|all) ]] && args=($args -a)
 
-    lines=(${(f)"$(_call_program commands docker ps ${args})"})
-    services=(${(f)"$(_call_program commands docker-compose 2>/dev/null ${compose_file:+-f $compose_file} ${compose_project:+-p $compose_project} ps -q)"})
+    lines=(${(f)"$(_call_program commands docker $docker_options ps $args)"})
+    services=(${(f)"$(_call_program commands docker-compose 2>/dev/null $compose_options ps -q)"})
 
     # Parse header line to find columns
     local i=1 j=1 k header=${lines[1]}
     declare -A begin end
-    while (( $j < ${#header} - 1 )) {
-        i=$(( $j + ${${header[$j,-1]}[(i)[^ ]]} - 1))
-        j=$(( $i + ${${header[$i,-1]}[(i)  ]} - 1))
-        k=$(( $j + ${${header[$j,-1]}[(i)[^ ]]} - 2))
-        begin[${header[$i,$(($j-1))]}]=$i
-        end[${header[$i,$(($j-1))]}]=$k
-    }
+    while (( j < ${#header} - 1 )); do
+        i=$(( j + ${${header[$j,-1]}[(i)[^ ]]} - 1 ))
+        j=$(( i + ${${header[$i,-1]}[(i)  ]} - 1 ))
+        k=$(( j + ${${header[$j,-1]}[(i)[^ ]]} - 2 ))
+        begin[${header[$i,$((j-1))]}]=$i
+        end[${header[$i,$((j-1))]}]=$k
+    done
     lines=(${lines[2,-1]})
 
     # Container ID
     local line s name
     local -a names
     for line in $lines; do
-        if [[ $services == *"${line[${begin[CONTAINER ID]},${end[CONTAINER ID]}]%% ##}"* ]]; then
+        if [[ ${services[@]} == *"${line[${begin[CONTAINER ID]},${end[CONTAINER ID]}]%% ##}"* ]]; then
             names=(${(ps:,:)${${line[${begin[NAMES]},-1]}%% *}})
             for name in $names; do
                 s="${${name%_*}#*_}:${(l:15:: :::)${${line[${begin[CREATED]},${end[CREATED]}]/ ago/}%% ##}}"
                 s="$s, ${line[${begin[CONTAINER ID]},${end[CONTAINER ID]}]%% ##}"
-                s="$s, ${${${line[$begin[IMAGE],$end[IMAGE]]}/:/\\:}%% ##}"
+                s="$s, ${${${line[${begin[IMAGE]},${end[IMAGE]}]}/:/\\:}%% ##}"
                 if [[ ${line[${begin[STATUS]},${end[STATUS]}]} = Exit* ]]; then
                     stopped=($stopped $s)
                 else
+                    if [[  ${line[${begin[STATUS]},${end[STATUS]}]} = *\(Paused\)* ]]; then
+                        paused=($paused $s)
+                    fi
                     running=($running $s)
                 fi
             done
         fi
     done
 
-    [[ $kind = (running|all) ]] && _describe -t services-running "running services" running
-    [[ $kind = (stopped|all) ]] && _describe -t services-stopped "stopped services" stopped
+    [[ $kind =~ (running|all) ]] && _describe -t services-running "running services" running "$@" && ret=0
+    [[ $kind =~ (paused|all) ]] && _describe -t services-paused "paused services" paused "$@" && ret=0
+    [[ $kind =~ (stopped|all) ]] && _describe -t services-stopped "stopped services" stopped "$@" && ret=0
+
+    return ret
+}
+
+__docker-compose_pausedservices() {
+    [[ $PREFIX = -* ]] && return 1
+    __docker-compose_get_services paused "$@"
 }
 
 __docker-compose_stoppedservices() {
+    [[ $PREFIX = -* ]] && return 1
     __docker-compose_get_services stopped "$@"
 }
 
 __docker-compose_runningservices() {
+    [[ $PREFIX = -* ]] && return 1
     __docker-compose_get_services running "$@"
 }
 
-__docker-compose_services () {
+__docker-compose_services() {
+    [[ $PREFIX = -* ]] && return 1
     __docker-compose_get_services all "$@"
 }
 
 __docker-compose_caching_policy() {
-    oldp=( "$1"(Nmh+1) )     # 1 hour
+    oldp=( "$1"(Nmh+1) )            # 1 hour
     (( $#oldp ))
 }
 
-__docker-compose_commands () {
+__docker-compose_commands() {
     local cache_policy
 
     zstyle -s ":completion:${curcontext}:" cache-policy cache_policy
@@ -151,123 +176,210 @@ __docker-compose_commands () {
         local -a lines
         lines=(${(f)"$(_call_program commands docker-compose 2>&1)"})
         _docker_compose_subcommands=(${${${lines[$((${lines[(i)Commands:]} + 1)),${lines[(I)  *]}]}## #}/ ##/:})
-        _store_cache docker_compose_subcommands _docker_compose_subcommands
+        (( $#_docker_compose_subcommands > 0 )) && _store_cache docker_compose_subcommands _docker_compose_subcommands
     fi
     _describe -t docker-compose-commands "docker-compose command" _docker_compose_subcommands
 }
 
-__docker-compose_subcommand () {
-    local -a _command_args
+__docker-compose_subcommand() {
+    local opts_help opts_force_recreate opts_no_recreate opts_no_build opts_remove_orphans opts_timeout opts_no_color opts_no_deps
+
+    opts_help='(: -)--help[Print usage]'
+    opts_force_recreate="(--no-recreate)--force-recreate[Recreate containers even if their configuration and image haven't changed. Incompatible with --no-recreate.]"
+    opts_no_recreate="(--force-recreate)--no-recreate[If containers already exist, don't recreate them. Incompatible with --force-recreate.]"
+    opts_no_build="(--build)--no-build[Don't build an image, even if it's missing.]"
+    opts_remove_orphans="--remove-orphans[Remove containers for services not defined in the Compose file]"
+    opts_timeout=('(-t --timeout)'{-t,--timeout}"[Specify a shutdown timeout in seconds. (default: 10)]:seconds: ")
+    opts_no_color='--no-color[Produce monochrome output.]'
+    opts_no_deps="--no-deps[Don't start linked services.]"
+
     integer ret=1
+
     case "$words[1]" in
         (build)
             _arguments \
-                '--help[Print usage]' \
-                '--no-cache[Do not use cache when building the image]' \
+                $opts_help \
+                '--force-rm[Always remove intermediate containers.]' \
+                '--no-cache[Do not use cache when building the image.]' \
+                '--pull[Always attempt to pull a newer version of the image.]' \
                 '*:services:__docker-compose_services_from_build' && ret=0
             ;;
+        (bundle)
+            _arguments \
+                $opts_help \
+                '--push-images[Automatically push images for any services which have a `build` option specified.]' \
+                '(--output -o)'{--output,-o}'[Path to write the bundle file to. Defaults to "<project name>.dab".]:file:_files' && ret=0
+            ;;
+        (config)
+            _arguments \
+                $opts_help \
+                '(--quiet -q)'{--quiet,-q}"[Only validate the configuration, don't print anything.]" \
+                '--services[Print the service names, one per line.]' && ret=0
+            ;;
+        (create)
+            _arguments \
+                $opts_help \
+                $opts_force_recreate \
+                $opts_no_recreate \
+                $opts_no_build \
+                "(--no-build)--build[Build images before creating containers.]" \
+                '*:services:__docker-compose_services_all' && ret=0
+            ;;
+        (down)
+            _arguments \
+                $opts_help \
+                "--rmi[Remove images. Type must be one of: 'all': Remove all images used by any service. 'local': Remove only images that don't have a custom tag set by the \`image\` field.]:type:(all local)" \
+                '(-v --volumes)'{-v,--volumes}"[Remove named volumes declared in the \`volumes\` section of the Compose file and anonymous volumes attached to containers.]" \
+                $opts_remove_orphans && ret=0
+            ;;
+        (events)
+            _arguments \
+                $opts_help \
+                '--json[Output events as a stream of json objects]' \
+                '*:services:__docker-compose_services_all' && ret=0
+            ;;
+        (exec)
+            _arguments \
+                $opts_help \
+                '-d[Detached mode: Run command in the background.]' \
+                '--privileged[Give extended privileges to the process.]' \
+                '--user=[Run the command as this user.]:username:_users' \
+                '-T[Disable pseudo-tty allocation. By default `docker-compose exec` allocates a TTY.]' \
+                '--index=[Index of the container if there are multiple instances of a service \[default: 1\]]:index: ' \
+                '(-):running services:__docker-compose_runningservices' \
+                '(-):command: _command_names -e' \
+                '*::arguments: _normal' && ret=0
+            ;;
         (help)
             _arguments ':subcommand:__docker-compose_commands' && ret=0
             ;;
         (kill)
             _arguments \
-                '--help[Print usage]' \
+                $opts_help \
                 '-s[SIGNAL to send to the container. Default signal is SIGKILL.]:signal:_signals' \
                 '*:running services:__docker-compose_runningservices' && ret=0
             ;;
         (logs)
             _arguments \
-                '--help[Print usage]' \
-                '--no-color[Produce monochrome output.]' \
+                $opts_help \
+                '(-f --follow)'{-f,--follow}'[Follow log output]' \
+                $opts_no_color \
+                '--tail=[Number of lines to show from the end of the logs for each container.]:number of lines: ' \
+                '(-t --timestamps)'{-t,--timestamps}'[Show timestamps]' \
                 '*:services:__docker-compose_services_all' && ret=0
             ;;
-        (migrate-to-labels)
-            _arguments -A '-*' \
-                '--help[Print usage]' \
-                '(-):Recreate containers to add labels' && ret=0
+        (pause)
+            _arguments \
+                $opts_help \
+                '*:running services:__docker-compose_runningservices' && ret=0
             ;;
         (port)
             _arguments \
-                '--help[Print usage]' \
-                '--protocol=-[tcp or udap (defaults to tcp)]:protocol:(tcp udp)' \
-                '--index=-[index of the container if there are mutiple instances of a service (defaults to 1)]:index: ' \
+                $opts_help \
+                '--protocol=[tcp or udp \[default: tcp\]]:protocol:(tcp udp)' \
+                '--index=[index of the container if there are multiple instances of a service \[default: 1\]]:index: ' \
                 '1:running services:__docker-compose_runningservices' \
                 '2:port:_ports' && ret=0
             ;;
         (ps)
             _arguments \
-                '--help[Print usage]' \
+                $opts_help \
                 '-q[Only display IDs]' \
                 '*:services:__docker-compose_services_all' && ret=0
             ;;
         (pull)
             _arguments \
-                '--help[Print usage]' \
+                $opts_help \
+                '--ignore-pull-failures[Pull what it can and ignores images with pull failures.]' \
                 '*:services:__docker-compose_services_from_image' && ret=0
             ;;
+        (push)
+            _arguments \
+                $opts_help \
+                '--ignore-push-failures[Push what it can and ignores images with push failures.]' \
+                '*:services:__docker-compose_services' && ret=0
+            ;;
         (rm)
             _arguments \
+                $opts_help \
                 '(-f --force)'{-f,--force}"[Don't ask to confirm removal]" \
-                '--help[Print usage]' \
-                '-v[Remove volumes associated with containers]' \
+                '-v[Remove any anonymous volumes attached to containers]' \
                 '*:stopped services:__docker-compose_stoppedservices' && ret=0
             ;;
         (run)
             _arguments \
+                $opts_help \
                 '-d[Detached mode: Run container in the background, print new container name.]' \
-                '--entrypoint[Overwrite the entrypoint of the image.]:entry point: ' \
                 '*-e[KEY=VAL Set an environment variable (can be used multiple times)]:environment variable KEY=VAL: ' \
-                '--help[Print usage]' \
-                '(-u --user)'{-u,--user=-}'[Run as specified username or uid]:username or uid:_users' \
-                "--no-deps[Don't start linked services.]" \
+                '--entrypoint[Overwrite the entrypoint of the image.]:entry point: ' \
+                '--name=[Assign a name to the container]:name: ' \
+                $opts_no_deps \
+                '(-p --publish)'{-p,--publish=}"[Publish a container's port(s) to the host]" \
                 '--rm[Remove container after run. Ignored in detached mode.]' \
                 "--service-ports[Run command with the service's ports enabled and mapped to the host.]" \
                 '-T[Disable pseudo-tty allocation. By default `docker-compose run` allocates a TTY.]' \
+                '(-u --user)'{-u,--user=}'[Run as specified username or uid]:username or uid:_users' \
+                '(-w --workdir)'{-w,--workdir=}'[Working directory inside the container]:workdir: ' \
                 '(-):services:__docker-compose_services' \
                 '(-):command: _command_names -e' \
                 '*::arguments: _normal' && ret=0
             ;;
         (scale)
             _arguments \
-                '--help[Print usage]' \
+                $opts_help \
+                $opts_timeout \
                 '*:running services:__docker-compose_runningservices' && ret=0
             ;;
         (start)
             _arguments \
-                '--help[Print usage]' \
+                $opts_help \
                 '*:stopped services:__docker-compose_stoppedservices' && ret=0
             ;;
         (stop|restart)
             _arguments \
-                '--help[Print usage]' \
-                '(-t --timeout)'{-t,--timeout}"[Specify a shutdown timeout in seconds. (default: 10)]:seconds: " \
+                $opts_help \
+                $opts_timeout \
                 '*:running services:__docker-compose_runningservices' && ret=0
             ;;
+        (top)
+            _arguments \
+                $opts_help \
+                '*:running services:__docker-compose_runningservices' && ret=0
+            ;;
+        (unpause)
+            _arguments \
+                $opts_help \
+                '*:paused services:__docker-compose_pausedservices' && ret=0
+            ;;
         (up)
             _arguments \
-                '-d[Detached mode: Run containers in the background, print new container names.]' \
-                '--help[Print usage]' \
-                '--no-color[Produce monochrome output.]' \
-                "--no-deps[Don't start linked services.]" \
-                "--no-recreate[If containers already exist, don't recreate them.]" \
-                "--force-recreate[Recreate containers even if their configuration and image haven't changed]" \
-                "--no-build[Don't build an image, even if it's missing]" \
-                '(-t --timeout)'{-t,--timeout}"[Specify a shutdown timeout in seconds. (default: 10)]:seconds: " \
+                $opts_help \
+                '(--abort-on-container-exit)-d[Detached mode: Run containers in the background, print new container names. Incompatible with --abort-on-container-exit.]' \
+                $opts_no_color \
+                $opts_no_deps \
+                $opts_force_recreate \
+                $opts_no_recreate \
+                $opts_no_build \
+                "(--no-build)--build[Build images before starting containers.]" \
+                "(-d)--abort-on-container-exit[Stops all containers if any container was stopped. Incompatible with -d.]" \
+                '(-t --timeout)'{-t,--timeout}"[Use this timeout in seconds for container shutdown when attached or when containers are already running. (default: 10)]:seconds: " \
+                $opts_remove_orphans \
                 '*:services:__docker-compose_services_all' && ret=0
             ;;
         (version)
             _arguments \
-                '--help[Print usage]' \
+                $opts_help \
                 "--short[Shows only Compose's version number.]" && ret=0
             ;;
         (*)
-            _message 'Unknown sub command'
+            _message 'Unknown sub command' && ret=1
+            ;;
     esac
 
     return ret
 }
 
-_docker-compose () {
+_docker-compose() {
     # Support for subservices, which allows for `compdef _docker docker-shell=_docker_containers`.
     # Based on /usr/share/zsh/functions/Completion/Unix/_git without support for `ret`.
     if [[ $service != docker-compose ]]; then
@@ -275,34 +387,70 @@ _docker-compose () {
         return
     fi
 
-    local curcontext="$curcontext" state line ret=1
+    local curcontext="$curcontext" state line
+    integer ret=1
     typeset -A opt_args
 
+    local file_description
+
+    if [[ -n ${words[(r)-f]} || -n ${words[(r)--file]} ]] ; then
+        file_description="Specify an override docker-compose file (default: docker-compose.override.yml)"
+    else
+        file_description="Specify an alternate docker-compose file (default: docker-compose.yml)"
+    fi
+
     _arguments -C \
         '(- :)'{-h,--help}'[Get help]' \
+        '*'{-f,--file}"[${file_description}]:file:_files -g '*.yml'" \
+        '(-p --project-name)'{-p,--project-name}'[Specify an alternate project name (default: directory name)]:project name:' \
         '--verbose[Show more output]' \
         '(- :)'{-v,--version}'[Print version and exit]' \
-        '(-f --file)'{-f,--file}'[Specify an alternate docker-compose file (default: docker-compose.yml)]:file:_files -g "*.yml"' \
-        '(-p --project-name)'{-p,--project-name}'[Specify an alternate project name (default: directory name)]:project name:' \
+        '(-H --host)'{-H,--host}'[Daemon socket to connect to]:host:' \
+        '--tls[Use TLS; implied by --tlsverify]' \
+        '--tlscacert=[Trust certs signed only by this CA]:ca path:' \
+        '--tlscert=[Path to TLS certificate file]:client cert path:' \
+        '--tlskey=[Path to TLS key file]:tls key path:' \
+        '--tlsverify[Use TLS and verify the remote]' \
+        "--skip-hostname-check[Don't check the daemon's hostname against the name specified in the client certificate (for example if your docker host is an IP address)]" \
         '(-): :->command' \
         '(-)*:: :->option-or-argument' && ret=0
 
-    local counter=1
-    #local compose_file compose_project
-    while [ $counter -lt ${#words[@]} ]; do
-        case "${words[$counter]}" in
-            -f|--file)
-                (( counter++ ))
-                compose_file="${words[$counter]}"
-                ;;
-            -p|--project-name)
-                (( counter++ ))
-                compose_project="${words[$counter]}"
-                ;;
-            *)
-                ;;
-        esac
-        (( counter++ ))
+    local -a relevant_compose_flags relevant_docker_flags compose_options docker_options
+
+    relevant_compose_flags=(
+        "--file" "-f"
+        "--host" "-H"
+        "--project-name" "-p"
+        "--tls"
+        "--tlscacert"
+        "--tlscert"
+        "--tlskey"
+        "--tlsverify"
+        "--skip-hostname-check"
+    )
+
+    relevant_docker_flags=(
+        "--host" "-H"
+        "--tls"
+        "--tlscacert"
+        "--tlscert"
+        "--tlskey"
+        "--tlsverify"
+    )
+
+    for k in "${(@k)opt_args}"; do
+        if [[ -n "${relevant_docker_flags[(r)$k]}" ]]; then
+            docker_options+=$k
+            if [[ -n "$opt_args[$k]" ]]; then
+                docker_options+=$opt_args[$k]
+            fi
+        fi
+        if [[ -n "${relevant_compose_flags[(r)$k]}" ]]; then
+            compose_options+=$k
+            if [[ -n "$opt_args[$k]" ]]; then
+                compose_options+=$opt_args[$k]
+            fi
+        fi
     done
 
     case $state in

contrib/migration/migrate-compose-file-v1-to-v2.py

@@ -0,0 +1,173 @@
+#!/usr/bin/env python
+"""
+Migrate a Compose file from the V1 format in Compose 1.5 to the V2 format
+supported by Compose 1.6+
+"""
+from __future__ import absolute_import
+from __future__ import unicode_literals
+
+import argparse
+import logging
+import sys
+
+import ruamel.yaml
+
+from compose.config.types import VolumeSpec
+
+
+log = logging.getLogger('migrate')
+
+
+def migrate(content):
+    data = ruamel.yaml.load(content, ruamel.yaml.RoundTripLoader)
+
+    service_names = data.keys()
+
+    for name, service in data.items():
+        warn_for_links(name, service)
+        warn_for_external_links(name, service)
+        rewrite_net(service, service_names)
+        rewrite_build(service)
+        rewrite_logging(service)
+        rewrite_volumes_from(service, service_names)
+
+    services = {name: data.pop(name) for name in data.keys()}
+
+    data['version'] = "2"
+    data['services'] = services
+    create_volumes_section(data)
+
+    return data
+
+
+def warn_for_links(name, service):
+    links = service.get('links')
+    if links:
+        example_service = links[0].partition(':')[0]
+        log.warn(
+            "Service {name} has links, which no longer create environment "
+            "variables such as {example_service_upper}_PORT. "
+            "If you are using those in your application code, you should "
+            "instead connect directly to the hostname, e.g. "
+            "'{example_service}'."
+            .format(name=name, example_service=example_service,
+                    example_service_upper=example_service.upper()))
+
+
+def warn_for_external_links(name, service):
+    external_links = service.get('external_links')
+    if external_links:
+        log.warn(
+            "Service {name} has external_links: {ext}, which now work "
+            "slightly differently. In particular, two containers must be "
+            "connected to at least one network in common in order to "
+            "communicate, even if explicitly linked together.\n\n"
+            "Either connect the external container to your app's default "
+            "network, or connect both the external container and your "
+            "service's containers to a pre-existing network. See "
+            "https://docs.docker.com/compose/networking/ "
+            "for more on how to do this."
+            .format(name=name, ext=external_links))
+
+
+def rewrite_net(service, service_names):
+    if 'net' in service:
+        network_mode = service.pop('net')
+
+        # "container:<service name>" is now "service:<service name>"
+        if network_mode.startswith('container:'):
+            name = network_mode.partition(':')[2]
+            if name in service_names:
+                network_mode = 'service:{}'.format(name)
+
+        service['network_mode'] = network_mode
+
+
+def rewrite_build(service):
+    if 'dockerfile' in service:
+        service['build'] = {
+            'context': service.pop('build'),
+            'dockerfile': service.pop('dockerfile'),
+        }
+
+
+def rewrite_logging(service):
+    if 'log_driver' in service:
+        service['logging'] = {'driver': service.pop('log_driver')}
+        if 'log_opt' in service:
+            service['logging']['options'] = service.pop('log_opt')
+
+
+def rewrite_volumes_from(service, service_names):
+    for idx, volume_from in enumerate(service.get('volumes_from', [])):
+        if volume_from.split(':', 1)[0] not in service_names:
+            service['volumes_from'][idx] = 'container:%s' % volume_from
+
+
+def create_volumes_section(data):
+    named_volumes = get_named_volumes(data['services'])
+    if named_volumes:
+        log.warn(
+            "Named volumes ({names}) must be explicitly declared. Creating a "
+            "'volumes' section with declarations.\n\n"
+            "For backwards-compatibility, they've been declared as external. "
+            "If you don't mind the volume names being prefixed with the "
+            "project name, you can remove the 'external' option from each one."
+            .format(names=', '.join(list(named_volumes))))
+
+        data['volumes'] = named_volumes
+
+
+def get_named_volumes(services):
+    volume_specs = [
+        VolumeSpec.parse(volume)
+        for service in services.values()
+        for volume in service.get('volumes', [])
+    ]
+    names = {
+        spec.external
+        for spec in volume_specs
+        if spec.is_named_volume
+    }
+    return {name: {'external': True} for name in names}
+
+
+def write(stream, new_format, indent, width):
+    ruamel.yaml.dump(
+        new_format,
+        stream,
+        Dumper=ruamel.yaml.RoundTripDumper,
+        indent=indent,
+        width=width)
+
+
+def parse_opts(args):
+    parser = argparse.ArgumentParser()
+    parser.add_argument("filename", help="Compose file filename.")
+    parser.add_argument("-i", "--in-place", action='store_true')
+    parser.add_argument(
+        "--indent", type=int, default=2,
+        help="Number of spaces used to indent the output yaml.")
+    parser.add_argument(
+        "--width", type=int, default=80,
+        help="Number of spaces used as the output width.")
+    return parser.parse_args()
+
+
+def main(args):
+    logging.basicConfig(format='\033[33m%(levelname)s:\033[37m %(message)s\033[0m\n')
+
+    opts = parse_opts(args)
+
+    with open(opts.filename, 'r') as fh:
+        new_format = migrate(fh.read())
+
+    if opts.in_place:
+        output = open(opts.filename, 'w')
+    else:
+        output = sys.stdout
+    write(output, new_format, opts.indent, opts.width)
+
+
+if __name__ == "__main__":
+    main(sys.argv)

docker-compose.spec

@@ -0,0 +1,56 @@
+# -*- mode: python -*-
+
+block_cipher = None
+
+a = Analysis(['bin/docker-compose'],
+             pathex=['.'],
+             hiddenimports=[],
+             hookspath=None,
+             runtime_hooks=None,
+             cipher=block_cipher)
+
+pyz = PYZ(a.pure, cipher=block_cipher)
+
+exe = EXE(pyz,
+          a.scripts,
+          a.binaries,
+          a.zipfiles,
+          a.datas,
+          [
+            (
+                'compose/config/config_schema_v1.json',
+                'compose/config/config_schema_v1.json',
+                'DATA'
+            ),
+            (
+                'compose/config/config_schema_v2.0.json',
+                'compose/config/config_schema_v2.0.json',
+                'DATA'
+            ),
+            (
+                'compose/config/config_schema_v2.1.json',
+                'compose/config/config_schema_v2.1.json',
+                'DATA'
+            ),
+            (
+                'compose/config/config_schema_v3.0.json',
+                'compose/config/config_schema_v3.0.json',
+                'DATA'
+            ),
+            (
+                'compose/config/config_schema_v3.1.json',
+                'compose/config/config_schema_v3.1.json',
+                'DATA'
+            ),
+            (
+                'compose/GITSHA',
+                'compose/GITSHA',
+                'DATA'
+            )
+          ],
+
+          name='docker-compose',
+          debug=False,
+          strip=None,
+          upx=True,
+          console=True)

docs/Dockerfile

@@ -1,26 +0,0 @@
-FROM docs/base:latest
-MAINTAINER Mary Anthony <mary@docker.com> (@moxiegirl)
-
-# To get the git info for this repo
-COPY . /src
-
-COPY . /docs/content/compose/
-
-RUN svn checkout https://github.com/docker/docker/trunk/docs /docs/content/docker
-RUN svn checkout https://github.com/docker/swarm/trunk/docs /docs/content/swarm
-RUN svn checkout https://github.com/docker/machine/trunk/docs /docs/content/machine
-RUN svn checkout https://github.com/docker/distribution/trunk/docs /docs/content/registry
-RUN svn checkout https://github.com/docker/tutorials/trunk/docs /docs/content/tutorials
-RUN svn checkout https://github.com/docker/opensource/trunk/docs /docs/content
-
-
-# Sed to process GitHub Markdown
-# 1-2 Remove comment code from metadata block
-# 3 Change ](/word to ](/project/ in links
-# 4 Change ](word.md) to ](/project/word)
-# 5 Remove .md extension from link text
-# 6 Change ](../ to ](/project/word) 
-# 7 Change ](../../ to ](/project/ --> not implemented
-# 
-# 
-RUN /src/pre-process.sh /docs

docs/Makefile

@@ -1,55 +0,0 @@
-.PHONY: all binary build cross default docs docs-build docs-shell shell test test-unit test-integration test-integration-cli test-docker-py validate
-
-# env vars passed through directly to Docker's build scripts
-# to allow things like `make DOCKER_CLIENTONLY=1 binary` easily
-# `docs/sources/contributing/devenvironment.md ` and `project/PACKAGERS.md` have some limited documentation of some of these
-DOCKER_ENVS := \
-	-e BUILDFLAGS \
-	-e DOCKER_CLIENTONLY \
-	-e DOCKER_EXECDRIVER \
-	-e DOCKER_GRAPHDRIVER \
-	-e TESTDIRS \
-	-e TESTFLAGS \
-	-e TIMEOUT
-# note: we _cannot_ add "-e DOCKER_BUILDTAGS" here because even if it's unset in the shell, that would shadow the "ENV DOCKER_BUILDTAGS" set in our Dockerfile, which is very important for our official builds
-
-# to allow `make DOCSDIR=docs docs-shell` (to create a bind mount in docs)
-DOCS_MOUNT := $(if $(DOCSDIR),-v $(CURDIR)/$(DOCSDIR):/$(DOCSDIR))
-
-# to allow `make DOCSPORT=9000 docs`
-DOCSPORT := 8000
-
-# Get the IP ADDRESS
-DOCKER_IP=$(shell python -c "import urlparse ; print urlparse.urlparse('$(DOCKER_HOST)').hostname or ''")
-HUGO_BASE_URL=$(shell test -z "$(DOCKER_IP)" && echo localhost || echo "$(DOCKER_IP)")
-HUGO_BIND_IP=0.0.0.0
-
-GIT_BRANCH := $(shell git rev-parse --abbrev-ref HEAD 2>/dev/null)
-DOCKER_IMAGE := docker$(if $(GIT_BRANCH),:$(GIT_BRANCH))
-DOCKER_DOCS_IMAGE := docs-base$(if $(GIT_BRANCH),:$(GIT_BRANCH))
-
-
-DOCKER_RUN_DOCS := docker run --rm -it $(DOCS_MOUNT) -e AWS_S3_BUCKET -e NOCACHE
-
-# for some docs workarounds (see below in "docs-build" target)
-GITCOMMIT := $(shell git rev-parse --short HEAD 2>/dev/null)
-
-default: docs
-
-docs: docs-build
-	$(DOCKER_RUN_DOCS) -p $(if $(DOCSPORT),$(DOCSPORT):)8000 -e DOCKERHOST "$(DOCKER_DOCS_IMAGE)" hugo server --port=$(DOCSPORT) --baseUrl=$(HUGO_BASE_URL) --bind=$(HUGO_BIND_IP)
-
-docs-draft: docs-build
-	$(DOCKER_RUN_DOCS) -p $(if $(DOCSPORT),$(DOCSPORT):)8000 -e DOCKERHOST "$(DOCKER_DOCS_IMAGE)" hugo server --buildDrafts="true" --port=$(DOCSPORT) --baseUrl=$(HUGO_BASE_URL) --bind=$(HUGO_BIND_IP)
-
-
-docs-shell: docs-build
-	$(DOCKER_RUN_DOCS) -p $(if $(DOCSPORT),$(DOCSPORT):)8000 "$(DOCKER_DOCS_IMAGE)" bash
-
-
-docs-build:
-#	( git remote | grep -v upstream ) || git diff --name-status upstream/release..upstream/docs ./ > ./changed-files
-#	echo "$(GIT_BRANCH)" > GIT_BRANCH
-#	echo "$(AWS_S3_BUCKET)" > AWS_S3_BUCKET
-#	echo "$(GITCOMMIT)" > GITCOMMIT
-	docker build -t "$(DOCKER_DOCS_IMAGE)" .

docs/README.md

@@ -1,77 +1,16 @@
-# Contributing to the Docker Compose documentation
+# The docs have been moved!
 
-The documentation in this directory is part of the [https://docs.docker.com](https://docs.docker.com) website.  Docker uses [the Hugo static generator](http://gohugo.io/overview/introduction/) to convert project Markdown files to a static HTML site. 
+The documentation for Compose has been merged into
+[the general documentation repo](https://github.com/docker/docker.github.io).
 
-You don't need to be a Hugo expert to contribute to the compose documentation. If you are familiar with Markdown, you can modify the content in the `docs` files.  
+The docs for Compose are now here:
+https://github.com/docker/docker.github.io/tree/master/compose
 
-If you want to add a new file or change the location of the document in the menu, you do need to know a little more.
+Please submit pull requests for unpublished features on the `vnext-compose` branch (https://github.com/docker/docker.github.io/tree/vnext-compose).
 
-## Documentation contributing workflow
+If you submit a PR to this codebase that has a docs impact, create a second docs PR on `docker.github.io`. Use the docs PR template provided (coming soon - watch this space).
 
-1. Edit a Markdown file in the tree.
+PRs for typos, additional information, etc. for already-published features should be labeled as `okay-to-publish` (we are still settling on a naming convention, will provide a label soon). You can submit these PRs either to `vnext-compose` or directly to `master` on `docker.github.io`
 
-2. Save your changes.
-
-3. Make sure you are in the `docs` subdirectory.
-
-4. Build the documentation.
-
-        $ make docs
-         ---> ffcf3f6c4e97
-        Removing intermediate container a676414185e8
-        Successfully built ffcf3f6c4e97
-        docker run --rm -it  -e AWS_S3_BUCKET -e NOCACHE -p 8000:8000 -e DOCKERHOST "docs-base:test-tooling" hugo server --port=8000 --baseUrl=192.168.59.103 --bind=0.0.0.0
-        ERROR: 2015/06/13 MenuEntry's .Url is deprecated and will be removed in Hugo 0.15. Use .URL instead.
-        0 of 4 drafts rendered
-        0 future content 
-        12 pages created
-        0 paginator pages created
-        0 tags created
-        0 categories created
-        in 55 ms
-        Serving pages from /docs/public
-        Web Server is available at http://0.0.0.0:8000/
-        Press Ctrl+C to stop
-
-5. Open the available server in your browser.
-
-    The documentation server has the complete menu but only the Docker Compose
-    documentation resolves.  You can't access the other project docs from this
-    localized build.
-
-## Tips on Hugo metadata and menu positioning
-
-The top of each Docker Compose documentation file contains TOML metadata. The metadata is commented out to prevent it from appearing in GitHub.
-
-    <!--[metadata]>
-    +++
-    title = "Extending services in Compose"
-    description = "How to use Docker Compose's extends keyword to share configuration between files and projects"
-    keywords = ["fig, composition, compose, docker, orchestration, documentation, docs"]
-    [menu.main]
-    parent="smn_workw_compose"
-    weight=2
-    +++
-    <![end-metadata]-->  
-
-The metadata alone has this structure:
-
-    +++
-    title = "Extending services in Compose"
-    description = "How to use Docker Compose's extends keyword to share configuration between files and projects"
-    keywords = ["fig, composition, compose, docker, orchestration, documentation, docs"]
-    [menu.main]
-    parent="smn_workw_compose"
-    weight=2
-    +++
-    
-The `[menu.main]` section refers to navigation defined [in the main Docker menu](https://github.com/docker/docs-base/blob/hugo/config.toml). This metadata says *add a menu item called* Extending services in Compose *to the menu with the* `smn_workdw_compose` *identifier*.  If you locate the menu in the configuration, you'll find *Create multi-container applications* is the menu title.
-
-You can move an article in the tree by specifying a new parent. You can shift the location of the item by changing its weight.  Higher numbers are heavier and shift the item to the bottom of menu. Low or no numbers shift it up.
-
-
-## Other key documentation repositories
-
-The `docker/docs-base` repository contains [the Hugo theme and menu configuration](https://github.com/docker/docs-base). If you open the `Dockerfile` you'll see the `make docs` relies on this as a base image for building the Compose documentation.
-    
-The `docker/docs.docker.com` repository contains [build system for building the Docker documentation site](https://github.com/docker/docs.docker.com). Fork this repository to build the entire documentation site.
+As always, the docs remain open-source and we appreciate your feedback and
+pull requests!

docs/completion.md

@@ -1,69 +0,0 @@
-<!--[metadata]>
-+++
-title = "Command Completion"
-description = "Compose CLI reference"
-keywords = ["fig, composition, compose, docker, orchestration, cli,  reference"]
-[menu.main]
-parent="smn_workw_compose"
-weight=3
-+++
-<![end-metadata]-->
-
-# Command Completion
-
-Compose comes with [command completion](http://en.wikipedia.org/wiki/Command-line_completion)
-for the bash and zsh shell.
-
-## Installing Command Completion
-
-### Bash
-
-Make sure bash completion is installed. If you use a current Linux in a non-minimal installation, bash completion should be available.
-On a Mac, install with `brew install bash-completion`
-
-Place the completion script in `/etc/bash_completion.d/` (`/usr/local/etc/bash_completion.d/` on a Mac), using e.g.
-
-     curl -L https://raw.githubusercontent.com/docker/compose/$(docker-compose --version | awk 'NR==1{print $NF}')/contrib/completion/bash/docker-compose > /etc/bash_completion.d/docker-compose
-
-Completion will be available upon next login.
-
-### Zsh
-
-Place the completion script in your `/path/to/zsh/completion`, using e.g. `~/.zsh/completion/`
-
-    mkdir -p ~/.zsh/completion
-    curl -L https://raw.githubusercontent.com/docker/compose/$(docker-compose --version | awk 'NR==1{print $NF}')/contrib/completion/zsh/_docker-compose > ~/.zsh/completion/_docker-compose
-
-Include the directory in your `$fpath`, e.g. by adding in `~/.zshrc`
-
-    fpath=(~/.zsh/completion $fpath)
-
-Make sure `compinit` is loaded or do it by adding in `~/.zshrc`
-
-    autoload -Uz compinit && compinit -i
-
-Then reload your shell
-
-    exec $SHELL -l
-
-## Available completions
-
-Depending on what you typed on the command line so far, it will complete
-
- - available docker-compose commands
- - options that are available for a particular command
- - service names that make sense in a given context (e.g. services with running or stopped instances or services based on images vs. services based on Dockerfiles). For `docker-compose scale`, completed service names will automatically have "=" appended.
- - arguments for selected options, e.g. `docker-compose kill -s` will complete some signals like SIGHUP and SIGUSR1.
-
-Enjoy working with Compose faster and with less typos!
-
-## Compose documentation
-
-- [User guide](/)
-- [Installing Compose](install.md)
-- [Get started with Django](django.md)
-- [Get started with Rails](rails.md)
-- [Get started with Wordpress](wordpress.md)
-- [Command line reference](cli.md)
-- [Yaml file reference](yml.md)
-- [Compose environment variables](env.md)

docs/django.md

@@ -1,135 +0,0 @@
-<!--[metadata]>
-+++
-title = "Quickstart Guide: Compose and Django"
-description = "Getting started with Docker Compose and Django"
-keywords = ["documentation, docs,  docker, compose, orchestration, containers"]
-[menu.main]
-parent="smn_workw_compose"
-weight=4
-+++
-<![end-metadata]-->
-
-
-## Quickstart Guide: Compose and Django
-
-
-This Quick-start Guide will demonstrate how to use Compose to set up and run a
-simple Django/PostgreSQL app. Before starting, you'll need to have
-[Compose installed](install.md).
-
-### Define the project
-
-Start by setting up the three files you'll need to build the app. First, since
-your app is going to run inside a Docker container containing all of its
-dependencies, you'll need to define exactly what needs to be included in the
-container. This is done using a file called `Dockerfile`. To begin with, the
-Dockerfile consists of:
-
-    FROM python:2.7
-    ENV PYTHONUNBUFFERED 1
-    RUN mkdir /code
-    WORKDIR /code
-    ADD requirements.txt /code/
-    RUN pip install -r requirements.txt
-    ADD . /code/
-
-This Dockerfile will define an image that is used to build a container that
-includes your application and has Python installed alongside all of your Python
-dependencies. For more information on how to write Dockerfiles, see the
-[Docker user guide](https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile) and the [Dockerfile reference](http://docs.docker.com/reference/builder/).
-
-Second, you'll define your Python dependencies in a file called
-`requirements.txt`:
-
-    Django
-    psycopg2
-
-Finally, this is all tied together with a file called `docker-compose.yml`. It
-describes the services that comprise your app (here, a web server and database),
-which Docker images they use, how they link together, what volumes will be
-mounted inside the containers, and what ports they expose.
-
-    db:
-      image: postgres
-    web:
-      build: .
-      command: python manage.py runserver 0.0.0.0:8000
-      volumes:
-        - .:/code
-      ports:
-        - "8000:8000"
-      links:
-        - db
-
-See the [`docker-compose.yml` reference](yml.md) for more information on how
-this file works.
-
-### Build the project
-
-You can now start a Django project with `docker-compose run`:
-
-    $ docker-compose run web django-admin.py startproject composeexample .
-
-First, Compose will build an image for the `web` service using the `Dockerfile`.
-It will then run `django-admin.py startproject composeexample .` inside a
-container built using that image.
-
-This will generate a Django app inside the current directory:
-
-    $ ls
-    Dockerfile       docker-compose.yml          composeexample       manage.py        requirements.txt
-
-### Connect the database
-
-Now you need to set up the database connection. Replace the `DATABASES = ...`
-definition in `composeexample/settings.py` to read:
-
-    DATABASES = {
-        'default': {
-            'ENGINE': 'django.db.backends.postgresql_psycopg2',
-            'NAME': 'postgres',
-            'USER': 'postgres',
-            'HOST': 'db',
-            'PORT': 5432,
-        }
-    }
-
-These settings are determined by the
-[postgres](https://registry.hub.docker.com/_/postgres/) Docker image specified
-in the Dockerfile.
-
-Then, run `docker-compose up`:
-
-    Recreating myapp_db_1...
-    Recreating myapp_web_1...
-    Attaching to myapp_db_1, myapp_web_1
-    myapp_db_1 |
-    myapp_db_1 | PostgreSQL stand-alone backend 9.1.11
-    myapp_db_1 | 2014-01-27 12:17:03 UTC LOG:  database system is ready to accept connections
-    myapp_db_1 | 2014-01-27 12:17:03 UTC LOG:  autovacuum launcher started
-    myapp_web_1 | Validating models...
-    myapp_web_1 |
-    myapp_web_1 | 0 errors found
-    myapp_web_1 | January 27, 2014 - 12:12:40
-    myapp_web_1 | Django version 1.6.1, using settings 'composeexample.settings'
-    myapp_web_1 | Starting development server at http://0.0.0.0:8000/
-    myapp_web_1 | Quit the server with CONTROL-C.
-
-Your Django app should nw be running at port 8000 on your Docker daemon. If you are using a Docker Machine VM, you can use the `docker-machine ip MACHINE_NAME` to get the IP address.
-
-You can also run management commands with Docker. To set up your database, for
-example, run `docker-compose up` and in another terminal run:
-
-    $ docker-compose run web python manage.py syncdb
-
-## More Compose documentation
-
-- [User guide](/)
-- [Installing Compose](install.md)
-- [Get started with Django](django.md)
-- [Get started with Rails](rails.md)
-- [Get started with Wordpress](wordpress.md)
-- [Command line reference](cli.md)
-- [Yaml file reference](yml.md)
-- [Compose environment variables](env.md)
-- [Compose command line completion](completion.md)

docs/env.md

@@ -1,49 +0,0 @@
-<!--[metadata]>
-+++
-title = "Compose environment variables reference"
-description = "Compose CLI reference"
-keywords = ["fig, composition, compose, docker, orchestration, cli,  reference"]
-[menu.main]
-parent="smn_compose_ref"
-weight=3
-+++
-<![end-metadata]-->
-
-# Compose environment variables reference
-
-**Note:** Environment variables are no longer the recommended method for connecting to linked services. Instead, you should use the link name (by default, the name of the linked service) as the hostname to connect to. See the [docker-compose.yml documentation](yml.md#links) for details.
-
-Compose uses [Docker links] to expose services' containers to one another. Each linked container injects a set of environment variables, each of which begins with the uppercase name of the container.
-
-To see what environment variables are available to a service, run `docker-compose run SERVICE env`.
-
-<b><i>name</i>\_PORT</b><br>
-Full URL, e.g. `DB_PORT=tcp://172.17.0.5:5432`
-
-<b><i>name</i>\_PORT\_<i>num</i>\_<i>protocol</i></b><br>
-Full URL, e.g. `DB_PORT_5432_TCP=tcp://172.17.0.5:5432`
-
-<b><i>name</i>\_PORT\_<i>num</i>\_<i>protocol</i>\_ADDR</b><br>
-Container's IP address, e.g. `DB_PORT_5432_TCP_ADDR=172.17.0.5`
-
-<b><i>name</i>\_PORT\_<i>num</i>\_<i>protocol</i>\_PORT</b><br>
-Exposed port number, e.g. `DB_PORT_5432_TCP_PORT=5432`
-
-<b><i>name</i>\_PORT\_<i>num</i>\_<i>protocol</i>\_PROTO</b><br>
-Protocol (tcp or udp), e.g. `DB_PORT_5432_TCP_PROTO=tcp`
-
-<b><i>name</i>\_NAME</b><br>
-Fully qualified container name, e.g. `DB_1_NAME=/myapp_web_1/myapp_db_1`
-
-[Docker links]: http://docs.docker.com/userguide/dockerlinks/
-
-## Compose documentation
-
-- [User guide](/)
-- [Installing Compose](install.md)
-- [Get started with Django](django.md)
-- [Get started with Rails](rails.md)
-- [Get started with Wordpress](wordpress.md)
-- [Command line reference](cli.md)
-- [Yaml file reference](yml.md)
-- [Compose command line completion](completion.md)

docs/extends.md

@@ -1,363 +0,0 @@
-<!--[metadata]>
-+++
-title = "Extending services in Compose"
-description = "How to use Docker Compose's extends keyword to share configuration between files and projects"
-keywords = ["fig, composition, compose, docker, orchestration, documentation, docs"]
-[menu.main]
-parent="smn_workw_compose"
-weight=2
-+++
-<![end-metadata]-->
-
-
-## Extending services in Compose
-
-Docker Compose's `extends` keyword enables sharing of common configurations
-among different files, or even different projects entirely. Extending services
-is useful if you have several applications that reuse commonly-defined services.
-Using `extends` you can define a service in one place and refer to it from
-anywhere.
-
-Alternatively, you can deploy the same application to multiple environments with
-a slightly different set of services in each case (or with changes to the
-configuration of some services). Moreover, you can do so without copy-pasting
-the configuration around.
-
-### Understand the extends configuration
-
-When defining any service in `docker-compose.yml`, you can declare that you are
-extending another service like this:
-
-    web:
-      extends:
-        file: common-services.yml
-        service: webapp
-
-This instructs Compose to re-use the configuration for the `webapp` service
-defined in the `common-services.yml` file. Suppose that `common-services.yml`
-looks like this:
-
-    webapp:
-      build: .
-      ports:
-        - "8000:8000"
-      volumes:
-        - "/data"
-
-In this case, you'll get exactly the same result as if you wrote
-`docker-compose.yml` with that `build`, `ports` and `volumes` configuration
-defined directly under `web`.
-
-You can go further and define (or re-define) configuration locally in
-`docker-compose.yml`:
-
-    web:
-      extends:
-        file: common-services.yml
-        service: webapp
-      environment:
-        - DEBUG=1
-      cpu_shares: 5
-
-You can also write other services and link your `web` service to them:
-
-    web:
-      extends:
-        file: common-services.yml
-        service: webapp
-      environment:
-        - DEBUG=1
-      cpu_shares: 5
-      links:
-        - db
-    db:
-      image: postgres
-
-For full details on how to use `extends`, refer to the [reference](#reference).
-
-### Example use case
-
-In this example, you’ll repurpose the example app from the [quick start
-guide](index.md). (If you're not familiar with Compose, it's recommended that
-you go through the quick start first.) This example assumes you want to use
-Compose both to develop an application locally and then deploy it to a
-production environment.
-
-The local and production environments are similar, but there are some
-differences. In development, you mount the application code as a volume so that
-it can pick up changes; in production, the code should be immutable from the
-outside. This ensures it’s not accidentally changed. The development environment
-uses a local Redis container, but in production another team manages the Redis
-service, which is listening at `redis-production.example.com`.
-
-To configure with `extends` for this sample, you must:
-
-1.  Define the web application as a Docker image in `Dockerfile` and a Compose
-    service in `common.yml`.
-
-2.  Define the development environment in the standard Compose file,
-    `docker-compose.yml`.
-
-    - Use `extends` to pull in the web service.
-    - Configure a volume to enable code reloading.
-    - Create an additional Redis service for the application to use locally.
-
-3.  Define the production environment in a third Compose file, `production.yml`.
-
-    - Use `extends` to pull in the web service.
-    - Configure the web service to talk to the external, production Redis service.
-
-#### Define the web app
-
-Defining the web application requires the following:
-
-1.  Create an `app.py` file.
-
-    This file contains a simple Python application that uses Flask to serve HTTP
-    and increments a counter in Redis:
-
-        from flask import Flask
-        from redis import Redis
-        import os
-
-        app = Flask(__name__)
-        redis = Redis(host=os.environ['REDIS_HOST'], port=6379)
-
-        @app.route('/')
-        def hello():
-           redis.incr('hits')
-           return 'Hello World! I have been seen %s times.\n' % redis.get('hits')
-
-        if __name__ == "__main__":
-           app.run(host="0.0.0.0", debug=True)
-
-    This code uses a `REDIS_HOST` environment variable to determine where to
-    find Redis.
-
-2.  Define the Python dependencies in a `requirements.txt` file:
-
-        flask
-        redis
-
-3.  Create a `Dockerfile` to build an image containing the app:
-
-        FROM python:2.7
-        ADD . /code
-        WORKDIR /code
-        RUN pip install -r requirements.txt
-        CMD python app.py
-
-4.  Create a Compose configuration file called `common.yml`:
-
-    This configuration defines how to run the app.
-
-        web:
-          build: .
-          ports:
-            - "5000:5000"
-
-    Typically, you would have dropped this configuration into
-    `docker-compose.yml` file, but in order to pull it into multiple files with
-    `extends`, it needs to be in a separate file.
-
-#### Define the development environment
-
-1.  Create a `docker-compose.yml` file.
-
-    The `extends` option pulls in the `web` service from the `common.yml` file
-    you created in the previous section.
-
-        web:
-          extends:
-            file: common.yml
-            service: web
-          volumes:
-            - .:/code
-          links:
-            - redis
-          environment:
-            - REDIS_HOST=redis
-        redis:
-          image: redis
-
-    The new addition defines a `web` service that:
-
-    - Fetches the base configuration for `web` out of `common.yml`.
-    - Adds `volumes` and `links` configuration to the base (`common.yml`)
-    configuration.
-    - Sets the `REDIS_HOST` environment variable to point to the linked redis
-    container. This environment uses a stock `redis` image from the Docker Hub.
-
-2.  Run `docker-compose up`.
-
-    Compose creates, links, and starts a web and redis container linked together.
-    It mounts your application code inside the web container.
-
-3.  Verify that the code is mounted by changing the message in
-    `app.py`&mdash;say, from `Hello world!` to `Hello from Compose!`.
-
-    Don't forget to refresh your browser to see the change!
-
-#### Define the production environment
-
-You are almost done. Now, define your production environment:
-
-1.  Create a `production.yml` file.
-
-    As with `docker-compose.yml`, the `extends` option pulls in the `web` service
-    from `common.yml`.
-
-        web:
-          extends:
-            file: common.yml
-            service: web
-          environment:
-            - REDIS_HOST=redis-production.example.com
-
-2.  Run `docker-compose -f production.yml up`.
-
-    Compose creates *just* a web container and configures the Redis connection via
-    the `REDIS_HOST` environment variable. This variable points to the production
-    Redis instance.
-
-    > **Note**: If you try to load up the webapp in your browser you'll get an
-    > error&mdash;`redis-production.example.com` isn't actually a Redis server.
-
-You've now done a basic `extends` configuration. As your application develops,
-you can make any necessary changes to the web service in `common.yml`. Compose
-picks up both the development and production environments when you next run
-`docker-compose`. You don't have to do any copy-and-paste, and you don't have to
-manually keep both environments in sync.
-
-
-### Reference
-
-You can use `extends` on any service together with other configuration keys. It
-always expects a dictionary that should always contain the key: `service` and optionally the `file` key.
-
-The `file` key specifies the location of a Compose configuration file defining
-the extension. The `file` value can be an absolute or relative path. If you
-specify a relative path, Docker Compose treats it as relative to the location
-of the current file. If you don't specify a `file`, Compose looks in the
-current configuration file.
-
-The `service` key specifies the name of the service to extend, for example `web`
-or `database`.
-
-You can extend a service that itself extends another. You can extend
-indefinitely. Compose does not support circular references and `docker-compose`
-returns an error if it encounters them.
-
-#### Adding and overriding configuration
-
-Compose copies configurations from the original service over to the local one,
-**except** for `links` and `volumes_from`. These exceptions exist to avoid
-implicit dependencies&mdash;you always define `links` and `volumes_from`
-locally. This ensures dependencies between services are clearly visible when
-reading the current file. Defining these locally also ensures changes to the
-referenced file don't result in breakage.
-
-If a configuration option is defined in both the original service and the local
-service, the local value either *override*s or *extend*s the definition of the
-original service. This works differently for other configuration options.
-
-For single-value options like `image`, `command` or `mem_limit`, the new value
-replaces the old value. **This is the default behaviour - all exceptions are
-listed below.**
-
-    # original service
-    command: python app.py
-
-    # local service
-    command: python otherapp.py
-
-    # result
-    command: python otherapp.py
-
-In the case of `build` and `image`, using one in the local service causes
-Compose to discard the other, if it was defined in the original service.
-
-    # original service
-    build: .
-
-    # local service
-    image: redis
-
-    # result
-    image: redis
-
-    # original service
-    image: redis
-
-    # local service
-    build: .
-
-    # result
-    build: .
-
-For the **multi-value options** `ports`, `expose`, `external_links`, `dns` and
-`dns_search`, Compose concatenates both sets of values:
-
-    # original service
-    expose:
-      - "3000"
-
-    # local service
-    expose:
-      - "4000"
-      - "5000"
-
-    # result
-    expose:
-      - "3000"
-      - "4000"
-      - "5000"
-
-In the case of `environment` and `labels`, Compose "merges" entries together
-with locally-defined values taking precedence:
-
-    # original service
-    environment:
-      - FOO=original
-      - BAR=original
-
-    # local service
-    environment:
-      - BAR=local
-      - BAZ=local
-
-    # result
-    environment:
-      - FOO=original
-      - BAR=local
-      - BAZ=local
-
-Finally, for `volumes` and `devices`, Compose "merges" entries together with
-locally-defined bindings taking precedence:
-
-    # original service
-    volumes:
-      - /original-dir/foo:/foo
-      - /original-dir/bar:/bar
-
-    # local service
-    volumes:
-      - /local-dir/bar:/bar
-      - /local-dir/baz/:baz
-
-    # result
-    volumes:
-      - /original-dir/foo:/foo
-      - /local-dir/bar:/bar
-      - /local-dir/baz/:baz
-
-## Compose documentation
-
-- [User guide](/)
-- [Installing Compose](install.md)
-- [Get started with Django](django.md)
-- [Get started with Rails](rails.md)
-- [Get started with Wordpress](wordpress.md)
-- [Command line reference](cli.md)
-- [Yaml file reference](yml.md)
-- [Compose command line completion](completion.md)

docs/index.md

@@ -1,233 +0,0 @@
-<!--[metadata]>
-+++
-title = "Overview of Docker Compose"
-description = "Introduction and Overview of Compose"
-keywords = ["documentation, docs,  docker, compose, orchestration,  containers"]
-[menu.main]
-parent="smn_workw_compose"
-+++
-<![end-metadata]-->
-
-
-# Overview of Docker Compose
-
-Compose is a tool for defining and running multi-container applications with
-Docker. With Compose, you define a multi-container application in a single
-file, then spin your application up in a single command which does everything
-that needs to be done to get it running.
-
-Compose is great for development environments, staging servers, and CI. We don't
-recommend that you use it in production yet.
-
-Using Compose is basically a three-step process.
-
-1. Define your app's environment with a `Dockerfile` so it can be
-reproduced anywhere.
-2. Define the services that make up your app in `docker-compose.yml` so
-they can be run together in an isolated environment:
-3. Lastly, run `docker-compose up` and Compose will start and run your entire app.
-
-A `docker-compose.yml` looks like this:
-
-    web:
-      build: .
-      ports:
-       - "5000:5000"
-      volumes:
-       - .:/code
-      links:
-       - redis
-    redis:
-      image: redis
-
-Compose has commands for managing the whole lifecycle of your application:
-
- * Start, stop and rebuild services
- * View the status of running services
- * Stream the log output of running services
- * Run a one-off command on a service
-
-## Compose documentation
-
-- [Installing Compose](install.md)
-- [Get started with Django](django.md)
-- [Get started with Rails](rails.md)
-- [Get started with Wordpress](wordpress.md)
-- [Command line reference](cli.md)
-- [Yaml file reference](yml.md)
-- [Compose environment variables](env.md)
-- [Compose command line completion](completion.md)
-
-## Quick start
-
-Let's get started with a walkthrough of getting a simple Python web app running
-on Compose. It assumes a little knowledge of Python, but the concepts
-demonstrated here should be understandable even if you're not familiar with
-Python.
-
-### Installation and set-up
-
-First, [install Docker and Compose](install.md).
-
-Next, you'll want to make a directory for the project:
-
-    $ mkdir composetest
-    $ cd composetest
-
-Inside this directory, create `app.py`, a simple web app that uses the Flask
-framework and increments a value in Redis. Don't worry if you don't have Redis installed, docker is going to take care of that for you when we [define services](#define-services):
-
-    from flask import Flask
-    from redis import Redis
-
-    app = Flask(__name__)
-    redis = Redis(host='redis', port=6379)
-
-    @app.route('/')
-    def hello():
-        redis.incr('hits')
-        return 'Hello World! I have been seen %s times.' % redis.get('hits')
-
-    if __name__ == "__main__":
-        app.run(host="0.0.0.0", debug=True)
-
-Next, define the Python dependencies in a file called `requirements.txt`:
-
-    flask
-    redis
-
-### Create a Docker image
-
-Now, create a Docker image containing all of your app's dependencies. You
-specify how to build the image using a file called
-[`Dockerfile`](http://docs.docker.com/reference/builder/):
-
-    FROM python:2.7
-    ADD . /code
-    WORKDIR /code
-    RUN pip install -r requirements.txt
-    CMD python app.py
-
-This tells Docker to:
-
-* Build an image starting with the Python 2.7 image.
-* Add the current directory `.` into the path `/code` in the image.
-* Set the working directory to `/code`.
-* Install your Python dependencies.
-* Set the default command for the container to `python app.py`
-
-For more information on how to write Dockerfiles, see the [Docker user guide](https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile) and the [Dockerfile reference](http://docs.docker.com/reference/builder/).
-
-You can test that this builds by running `docker build -t web .`.
-
-### Define services
-
-Next, define a set of services using `docker-compose.yml`:
-
-    web:
-      build: .
-      ports:
-       - "5000:5000"
-      volumes:
-       - .:/code
-      links:
-       - redis
-    redis:
-      image: redis
-
-This defines two services:
-
-#### web
-
-* Builds from the `Dockerfile` in the current directory.
-* Forwards the exposed port 5000 on the container to port 5000 on the host machine.
-* Connects the web container to the Redis service via a link.
-* Mounts the current directory on the host to `/code` inside the container allowing you to modify the code without having to rebuild the image.
-
-#### redis
-
-* Uses the public [Redis](https://registry.hub.docker.com/_/redis/) image which gets pulled from the Docker Hub registry.
-
-### Build and run your app with Compose
-
-Now, when you run `docker-compose up`, Compose will pull a Redis image, build an image for your code, and start everything up:
-
-    $ docker-compose up
-    Pulling image redis...
-    Building web...
-    Starting composetest_redis_1...
-    Starting composetest_web_1...
-    redis_1 | [8] 02 Jan 18:43:35.576 # Server started, Redis version 2.8.3
-    web_1   |  * Running on http://0.0.0.0:5000/
-    web_1   |  * Restarting with stat
-
-If you're using [Docker Machine](https://docs.docker.com/machine), then `docker-machine ip MACHINE_VM` will tell you its address and you can open `http://MACHINE_VM_IP:5000` in a browser. 
-
-If you're not using Boot2docker and are on linux, then the web app should now be listening on port 5000 on your Docker daemon host. If http://0.0.0.0:5000 doesn't resolve, you can also try localhost:5000.
-
-You should get a message in your browser saying:
-
-`Hello World! I have been seen 1 times.`
-
-Refreshing the page will increment the number.
-
-If you want to run your services in the background, you can pass the `-d` flag
-(for "detached" mode) to `docker-compose up` and use `docker-compose ps` to
-see what is currently running:
-
-    $ docker-compose up -d
-    Starting composetest_redis_1...
-    Starting composetest_web_1...
-    $ docker-compose ps
-	    Name                 Command            State       Ports
-    -------------------------------------------------------------------
-    composetest_redis_1   /usr/local/bin/run         Up
-    composetest_web_1     /bin/sh -c python app.py   Up      5000->5000/tcp
-
-The `docker-compose run` command allows you to run one-off commands for your
-services. For example, to see what environment variables are available to the
-`web` service:
-
-    $ docker-compose run web env
-
-See `docker-compose --help` to see other available commands. You can also install [command completion](completion.md) for the bash and zsh shell, which will also show you available commands.
-
-If you started Compose with `docker-compose up -d`, you'll probably want to stop
-your services once you've finished with them:
-
-    $ docker-compose stop
-
-At this point, you have seen the basics of how Compose works.
-
-- Next, try the quick start guide for [Django](django.md),
-  [Rails](rails.md), or [Wordpress](wordpress.md).
-- See the reference guides for complete details on the [commands](cli.md), the
-  [configuration file](yml.md) and [environment variables](env.md).
-
-## Release Notes
-
-### Version 1.2.0 (April 7, 2015)
-
-For complete information on this release, see the [1.2.0 Milestone project page](https://github.com/docker/compose/wiki/1.2.0-Milestone-Project-Page).
-In addition to bug fixes and refinements, this release adds the following:
-
-* The `extends` keyword, which adds the ability to extend services by sharing  common configurations. For details, see
-[PR #1088](https://github.com/docker/compose/pull/1088).
-
-* Better integration with Swarm. Swarm will now schedule inter-dependent
-containers on the same host. For details, see
-[PR #972](https://github.com/docker/compose/pull/972).
-
-## Getting help
-
-Docker Compose is still in its infancy and under active development. If you need
-help, would like to contribute, or simply want to talk about the project with
-like-minded individuals, we have a number of open channels for communication.
-
-* To report bugs or file feature requests: please use the [issue tracker on Github](https://github.com/docker/compose/issues).
-
-* To talk about the project with people in real time: please join the `#docker-compose` channel on IRC.
-
-* To contribute code or documentation changes: please submit a [pull request on Github](https://github.com/docker/compose/pulls).
-
-For more information and resources, please visit the [Getting Help project page](https://docs.docker.com/project/get-help/).

docs/install.md

@@ -1,104 +0,0 @@
-<!--[metadata]>
-+++
-title = "Docker Compose"
-description = "How to install Docker Compose"
-keywords = ["compose, orchestration, install, installation, docker, documentation"]
-[menu.main]
-parent="mn_install"
-weight=4
-+++
-<![end-metadata]-->
-
-
-# Install Docker Compose
-
-You can run Compose on OS X and 64-bit Linux.  It is currently not supported on
-the Windows operating system. To install Compose, you'll need to install Docker
-first. 
-
-Depending on how your system is configured, you may require `sudo` access to
-install Compose. If your system requires `sudo`, you will receive "Permission
-denied" errors when installing Compose. If this is the case for you, preface the
-install commands with `sudo` to install.
-
-To install Compose, do the following:
-
-1. Install Docker Engine version 1.7.1 or greater:
-
-     * <a href="https://docs.docker.com/installation/mac/" target="_blank">Mac OS X installation</a> (installs both Engine and Compose)
-     
-     * <a href="https://docs.docker.com/installation/ubuntulinux/" target="_blank">Ubuntu installation</a>
-     
-     * <a href="https://docs.docker.com/installation/" target="_blank">other system installations</a>
- 
-2. Mac OS X users are done installing. Others should continue to the next step.
-   
-3. Go to the <a href="https://github.com/docker/compose/releases" target="_blank">repository release page</a>.
-
-4. Enter the `curl` command in your termial.
-
-     The command has the following format:
-
-        curl -L https://github.com/docker/compose/releases/download/VERSION_NUM/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
-   
-     If you have problems installing with `curl`, you can use `pip` instead: `pip install -U docker-compose`
-      
-4. Apply executable permissions to the binary:
-
-        $ chmod +x /usr/local/bin/docker-compose
-
-5.  Optionally, install [command completion](completion.md) for the
-`bash` and `zsh` shell.
-
-6. Test the installation.
-
-        $ docker-compose --version
-        docker-compose version: 1.4.2
-
-## Upgrading
-
-If you're upgrading from Compose 1.2 or earlier, you'll need to remove or migrate
-your existing containers after upgrading Compose. This is because, as of version
-1.3, Compose uses Docker labels to keep track of containers, and so they need to
-be recreated with labels added.
-
-If Compose detects containers that were created without labels, it will refuse
-to run so that you don't end up with two sets of them. If you want to keep using
-your existing containers (for example, because they have data volumes you want
-to preserve) you can migrate them with the following command:
-
-    $ docker-compose migrate-to-labels
-
-Alternatively, if you're not worried about keeping them, you can remove them &endash;
-Compose will just create new ones.
-
-    $ docker rm -f -v myapp_web_1 myapp_db_1 ...
-
-
-## Uninstallation
-
-To uninstall Docker Compose if you installed using `curl`:
-
-    $ rm /usr/local/bin/docker-compose
-
-
-To uninstall Docker Compose if you installed using `pip`:
-
-    $ pip uninstall docker-compose
-    
->**Note**: If you get a "Permission denied" error using either of the above
->methods, you probably do not have the proper permissions to remove
->`docker-compose`.  To force the removal, prepend `sudo` to either of the above
->commands and run again.
-
-
-## Where to go next
-
-- [User guide](/)
-- [Get started with Django](django.md)
-- [Get started with Rails](rails.md)
-- [Get started with Wordpress](wordpress.md)
-- [Command line reference](cli.md)
-- [Yaml file reference](yml.md)
-- [Compose environment variables](env.md)
-- [Compose command line completion](completion.md)

docs/pre-process.sh

@@ -1,61 +0,0 @@
-#!/bin/bash -e
-
-# Populate an array with just docker dirs and one with content dirs
-docker_dir=(`ls -d /docs/content/docker/*`)
-content_dir=(`ls -d /docs/content/*`)
-
-# Loop content not of docker/
-#
-# Sed to process GitHub Markdown
-# 1-2 Remove comment code from metadata block
-# 3 Remove .md extension from link text
-# 4 Change ](/ to ](/project/ in links
-# 5 Change ](word) to ](/project/word)
-# 6 Change ](../../ to ](/project/
-# 7 Change ](../ to ](/project/word)
-# 
-for i in "${content_dir[@]}"
-do
-   :
-   case $i in
-      "/docs/content/windows")
-      ;;
-      "/docs/content/mac")
-      ;;
-      "/docs/content/linux")
-      ;;
-      "/docs/content/docker")
-         y=${i##*/}
-         find $i -type f -name "*.md" -exec sed -i.old \
-         -e '/^<!.*metadata]>/g' \
-         -e '/^<!.*end-metadata.*>/g' {} \;
-      ;;
-      *)
-        y=${i##*/}
-        find $i -type f -name "*.md" -exec sed -i.old \
-        -e '/^<!.*metadata]>/g' \
-        -e '/^<!.*end-metadata.*>/g' \
-        -e 's/\(\]\)\([(]\)\(\/\)/\1\2\/'$y'\//g' \
-        -e 's/\(\][(]\)\([A-z].*\)\(\.md\)/\1\/'$y'\/\2/g' \
-        -e 's/\([(]\)\(.*\)\(\.md\)/\1\2/g'  \
-        -e 's/\(\][(]\)\(\.\/\)/\1\/'$y'\//g' \
-        -e 's/\(\][(]\)\(\.\.\/\.\.\/\)/\1\/'$y'\//g' \
-        -e 's/\(\][(]\)\(\.\.\/\)/\1\/'$y'\//g' {} \;
-      ;;
-      esac
-done
-
-#
-#  Move docker directories to content
-#
-for i in "${docker_dir[@]}"
-do
-   :
-    if [ -d $i ]    
-      then
-        mv $i /docs/content/ 
-      fi
-done
-
-rm -rf /docs/content/docker
-

docs/production.md

@@ -1,96 +0,0 @@
-<!--[metadata]>
-+++
-title = "Using Compose in production"
-description = "Guide to using Docker Compose in production"
-keywords = ["documentation, docs,  docker, compose, orchestration, containers,  production"]
-[menu.main]
-parent="smn_workw_compose"
-weight=1
-+++
-<![end-metadata]-->
-
-
-## Using Compose in production
-
-While **Compose is not yet considered production-ready**, if you'd like to experiment and learn more about using it in production deployments, this guide
-can help.
-The project is actively working towards becoming
-production-ready; to learn more about the progress being made, check out the
-[roadmap](https://github.com/docker/compose/blob/master/ROADMAP.md) for details
-on how it's coming along and what still needs to be done.
-
-When deploying to production, you'll almost certainly want to make changes to
-your app configuration that are more appropriate to a live environment. These
-changes may include:
-
-- Removing any volume bindings for application code, so that code stays inside
-  the container and can't be changed from outside
-- Binding to different ports on the host
-- Setting environment variables differently (e.g., to decrease the verbosity of
-  logging, or to enable email sending)
-- Specifying a restart policy (e.g., `restart: always`) to avoid downtime
-- Adding extra services (e.g., a log aggregator)
-
-For this reason, you'll probably want to define a separate Compose file, say
-`production.yml`, which specifies production-appropriate configuration.
-
-> **Note:** The [extends](extends.md) keyword is useful for maintaining multiple
-> Compose files which re-use common services without having to manually copy and
-> paste.
-
-Once you've got an alternate configuration file, make Compose use it
-by setting the `COMPOSE_FILE` environment variable:
-
-    $ COMPOSE_FILE=production.yml
-    $ docker-compose up -d
-
-> **Note:** You can also use the file for a one-off command without setting
-> an environment variable. You do this by passing the `-f` flag, e.g.,
-> `docker-compose -f production.yml up -d`.
-
-### Deploying changes
-
-When you make changes to your app code, you'll need to rebuild your image and
-recreate your app's containers. To redeploy a service called
-`web`, you would use:
-
-    $ docker-compose build web
-    $ docker-compose up --no-deps -d web
-
-This will first rebuild the image for `web` and then stop, destroy, and recreate
-*just* the `web` service. The `--no-deps` flag prevents Compose from also
-recreating any services which `web` depends on.
-
-### Running Compose on a single server
-
-You can use Compose to deploy an app to a remote Docker host by setting the
-`DOCKER_HOST`, `DOCKER_TLS_VERIFY`, and `DOCKER_CERT_PATH` environment variables
-appropriately. For tasks like this,
-[Docker Machine](https://docs.docker.com/machine) makes managing local and
-remote Docker hosts very easy, and is recommended even if you're not deploying
-remotely.
-
-Once you've set up your environment variables, all the normal `docker-compose`
-commands will work with no further configuration.
-
-### Running Compose on a Swarm cluster
-
-[Docker Swarm](https://docs.docker.com/swarm), a Docker-native clustering
-system, exposes the same API as a single Docker host, which means you can use
-Compose against a Swarm instance and run your apps across multiple hosts.
-
-Compose/Swarm integration is still in the experimental stage, and Swarm is still
-in beta, but if you'd like to explore and experiment, check out the
-[integration guide](https://github.com/docker/compose/blob/master/SWARM.md).
-
-## Compose documentation
-
-- [Installing Compose](install.md)
-- [Get started with Django](django.md)
-- [Get started with Rails](rails.md)
-- [Get started with Wordpress](wordpress.md)
-- [Command line reference](cli.md)
-- [Yaml file reference](yml.md)
-- [Compose environment variables](env.md)
-- [Compose command line completion](completion.md)
-

docs/rails.md

@@ -1,133 +0,0 @@
-<!--[metadata]>
-+++
-title = "Quickstart Guide: Compose and Rails"
-description = "Getting started with Docker Compose and Rails"
-keywords = ["documentation, docs,  docker, compose, orchestration, containers"]
-[menu.main]
-parent="smn_workw_compose"
-weight=5
-+++
-<![end-metadata]-->
-
-## Quickstart Guide: Compose and Rails
-
-This Quickstart guide will show you how to use Compose to set up and run a Rails/PostgreSQL app. Before starting, you'll need to have [Compose installed](install.md).
-
-### Define the project
-
-Start by setting up the three files you'll need to build the app. First, since
-your app is going to run inside a Docker container containing all of its
-dependencies, you'll need to define exactly what needs to be included in the
-container. This is done using a file called `Dockerfile`. To begin with, the
-Dockerfile consists of:
-
-    FROM ruby:2.2.0
-    RUN apt-get update -qq && apt-get install -y build-essential libpq-dev
-    RUN mkdir /myapp
-    WORKDIR /myapp
-    ADD Gemfile /myapp/Gemfile
-    RUN bundle install
-    ADD . /myapp
-
-That'll put your application code inside an image that will build a container with Ruby, Bundler and all your dependencies inside it. For more information on how to write Dockerfiles, see the [Docker user guide](https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile) and the [Dockerfile reference](http://docs.docker.com/reference/builder/).
-
-Next, create a bootstrap `Gemfile` which just loads Rails. It'll be overwritten in a moment by `rails new`.
-
-    source 'https://rubygems.org'
-    gem 'rails', '4.2.0'
-
-Finally, `docker-compose.yml` is where the magic happens. This file describes the services that comprise your app (a database and a web app), how to get each one's Docker image (the database just runs on a pre-made PostgreSQL image, and the web app is built from the current directory), and the configuration needed to link them together and expose the web app's port.
-
-    db:
-      image: postgres
-    web:
-      build: .
-      command: bundle exec rails s -p 3000 -b '0.0.0.0'
-      volumes:
-        - .:/myapp
-      ports:
-        - "3000:3000"
-      links:
-        - db
-
-### Build the project
-
-With those three files in place, you can now generate the Rails skeleton app
-using `docker-compose run`:
-
-    $ docker-compose run web rails new . --force --database=postgresql --skip-bundle
-
-First, Compose will build the image for the `web` service using the
-`Dockerfile`. Then it'll run `rails new` inside a new container, using that
-image. Once it's done, you should have generated a fresh app:
-
-    $ ls
-    Dockerfile   app          docker-compose.yml      tmp
-    Gemfile      bin          lib          vendor
-    Gemfile.lock config       log
-    README.rdoc  config.ru    public
-    Rakefile     db           test
-
-Uncomment the line in your new `Gemfile` which loads `therubyracer`, so you've
-got a Javascript runtime:
-
-    gem 'therubyracer', platforms: :ruby
-
-Now that you've got a new `Gemfile`, you need to build the image again. (This,
-and changes to the Dockerfile itself, should be the only times you'll need to
-rebuild.)
-
-    $ docker-compose build
-
-### Connect the database
-
-The app is now bootable, but you're not quite there yet. By default, Rails
-expects a database to be running on `localhost` - so you need to point it at the
-`db` container instead. You also need to change the database and username to
-align with the defaults set by the `postgres` image.
-
-Open up your newly-generated `database.yml` file. Replace its contents with the
-following:
-
-    development: &default
-      adapter: postgresql
-      encoding: unicode
-      database: postgres
-      pool: 5
-      username: postgres
-      password:
-      host: db
-
-    test:
-      <<: *default
-      database: myapp_test
-
-You can now boot the app with:
-
-    $ docker-compose up
-
-If all's well, you should see some PostgreSQL output, and then—after a few
-seconds—the familiar refrain:
-
-    myapp_web_1 | [2014-01-17 17:16:29] INFO  WEBrick 1.3.1
-    myapp_web_1 | [2014-01-17 17:16:29] INFO  ruby 2.2.0 (2014-12-25) [x86_64-linux-gnu]
-    myapp_web_1 | [2014-01-17 17:16:29] INFO  WEBrick::HTTPServer#start: pid=1 port=3000
-
-Finally, you need to create the database. In another terminal, run:
-
-    $ docker-compose run web rake db:create
-
-That's it. Your app should now be running on port 3000 on your Docker daemon. If you're using [Docker Machine](https://docs.docker.com/machine), then `docker-machine ip MACHINE_VM` returns the Docker host IP address. 
-
-
-## More Compose documentation
-
-- [User guide](/)
-- [Installing Compose](install.md)
-- [Get started with Django](django.md)
-- [Get started with Rails](rails.md)
-- [Get started with Wordpress](wordpress.md)
-- [Command line reference](cli.md)
-- [Yaml file reference](yml.md)
-- [Compose environment variables](env.md)
-- [Compose command line completion](completion.md)

docs/reference/build.md

@@ -1,23 +0,0 @@
-<!--[metadata]>
-+++
-title = "build"
-description = "build"
-keywords = ["fig, composition, compose, docker, orchestration, cli,  build"]
-[menu.main]
-identifier="build.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# build
-
-```
-Usage: build [options] [SERVICE...]
-
-Options:
---no-cache  Do not use cache when building the image.
-```
-
-Services are built once and then tagged as `project_service`, e.g.,
-`composetest_db`. If you change a service's Dockerfile or the contents of its
-build directory, run `docker-compose build` to rebuild it.

docs/reference/docker-compose.md

@@ -1,55 +0,0 @@
-<!--[metadata]>
-+++
-title = "docker-compose"
-description = "docker-compose Command Binary"
-keywords = ["fig, composition, compose, docker, orchestration, cli,  docker-compose"]
-[menu.main]
-parent = "smn_compose_cli"
-weight=-2	
-+++
-<![end-metadata]-->
-
-
-# docker-compose Command
-
-```
-Usage:
-  docker-compose [options] [COMMAND] [ARGS...]
-  docker-compose -h|--help
-
-Options:
-  -f, --file FILE           Specify an alternate compose file (default: docker-compose.yml)
-  -p, --project-name NAME   Specify an alternate project name (default: directory name)
-  --verbose                 Show more output
-  -v, --version             Print version and exit
-
-Commands:
-  build              Build or rebuild services
-  help               Get help on a command
-  kill               Kill containers
-  logs               View output from containers
-  port               Print the public port for a port binding
-  ps                 List containers
-  pull               Pulls service images
-  restart            Restart services
-  rm                 Remove stopped containers
-  run                Run a one-off command
-  scale              Set number of containers for a service
-  start              Start services
-  stop               Stop services
-  up                 Create and start containers
-  migrate-to-labels  Recreate containers to add labels
-```
-
-The Docker Compose binary. You use this command to build and manage multiple services in Docker containers.
-
-Use the `-f` flag to specify the location of a Compose configuration file. This
-flag is optional. If you don't provide this flag. Compose looks for a file named
-`docker-compose.yml` in the  working directory. If the file is not found,
-Compose looks in each parent directory successively, until it finds the file.
-
-Use a `-` as the filename to read configuration file from stdin. When stdin is
-used all paths in the configuration are relative to the current working
-directory.
-
-Each configuration can has a project name. If you supply a `-p` flag, you can specify a project name. If you don't specify the flag, Compose uses the current directory name.

docs/reference/help.md

@@ -1,18 +0,0 @@
-<!--[metadata]>
-+++
-title = "help"
-description = "help"
-keywords = ["fig, composition, compose, docker, orchestration, cli,  help"]
-[menu.main]
-identifier="help.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# help
-
-```
-Usage: help COMMAND
-```
-
-Displays help and usage instructions for a command.

docs/reference/index.md

@@ -1,29 +0,0 @@
-<!--[metadata]>
-+++
-title = "Compose CLI reference"
-description = "Compose CLI reference"
-keywords = ["fig, composition, compose, docker, orchestration, cli,  reference"]
-[menu.main]
-identifier = "smn_compose_cli"
-parent = "smn_compose_ref"	
-+++
-<![end-metadata]-->
-
-## Compose CLI reference
-
-The following pages describe the usage information for the [docker-compose](/reference/docker-compose.md) subcommands. You can also see this information by running `docker-compose [SUBCOMMAND] --help` from the command line.
-
-* [build](/reference/reference/build.md)
-* [help](/reference/help.md)
-* [kill](/reference/kill.md)          
-* [ps](/reference/ps.md)
-* [restart](/reference/restart.md)
-* [run](/reference/run.md)
-* [start](/reference/start.md)
-* [up](/reference/up.md)
-* [logs](/reference/logs.md)
-* [port](/reference/port.md)
-* [pull](/reference/pull.md) 
-* [rm](/reference/rm.md)
-* [scale](/reference/scale.md)
-* [stop](/reference/stop.md)

docs/reference/kill.md

@@ -1,24 +0,0 @@
-<!--[metadata]>
-+++
-title = "kill"
-description = "Forces running containers to stop."
-keywords = ["fig, composition, compose, docker, orchestration, cli,  kill"]
-[menu.main]
-identifier="kill.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# kill
-
-```
-Usage: kill [options] [SERVICE...]
-
-Options:
--s SIGNAL         SIGNAL to send to the container. Default signal is SIGKILL.
-```
-
-Forces running containers to stop by sending a `SIGKILL` signal. Optionally the
-signal can be passed, for example:
-
-    $ docker-compose kill -s SIGINT

docs/reference/logs.md

@@ -1,21 +0,0 @@
-<!--[metadata]>
-+++
-title = "logs"
-description = "Displays log output from services."
-keywords = ["fig, composition, compose, docker, orchestration, cli,  logs"]
-[menu.main]
-identifier="logs.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# logs
-
-```
-Usage: logs [options] [SERVICE...]
-
-Options:
---no-color  Produce monochrome output.
-```
-
-Displays log output from services.

docs/reference/overview.md

@@ -1,62 +0,0 @@
-<!--[metadata]>
-+++
-title = "Introduction to the CLI"
-description = "Introduction to the CLI"
-keywords = ["fig, composition, compose, docker, orchestration, cli,  reference"]
-[menu.main]
-parent = "smn_compose_cli"
-weight=-2	
-+++
-<![end-metadata]-->
-
-
-# Introduction to the CLI
-
-This section describes the subcommands you can use with the `docker-compose` command.  You can run subcommand against one or more services. To run against a specific service, you supply the service name from your compose configuration. If you do not specify the service name, the command runs against all the services in your configuration.
-
-## Environment Variables
-
-Several environment variables are available for you to configure the Docker Compose command-line behaviour.
-
-Variables starting with `DOCKER_` are the same as those used to configure the
-Docker command-line client. If you're using `docker-machine`, then the `eval "$(docker-machine env my-docker-vm)"` command should set them to their correct values. (In this example, `my-docker-vm` is the name of a machine you created.)
-
-### COMPOSE\_PROJECT\_NAME
-
-Sets the project name. This value is prepended along with the service name to the container container on start up. For example, if you project name is `myapp` and it includes two services `db` and `web` then compose starts containers named  `myapp_db_1` and `myapp_web_1` respectively.
-
-Setting this is optional. If you do not set this, the `COMPOSE_PROJECT_NAME` defaults to the `basename` of the current working directory.
-
-### COMPOSE\_FILE
-
-Specify the file containing the compose configuration. If not provided, Compose looks for a file named  `docker-compose.yml` in the current directory and then each parent directory in succession until a file by that name is found.
-
-### DOCKER\_HOST
-
-Sets the URL of the `docker` daemon. As with the Docker client, defaults to `unix:///var/run/docker.sock`.
-
-### DOCKER\_TLS\_VERIFY
-
-When set to anything other than an empty string, enables TLS communication with
-the `docker` daemon.
-
-### DOCKER\_CERT\_PATH
-
-Configures the path to the `ca.pem`, `cert.pem`, and `key.pem` files used for TLS verification. Defaults to `~/.docker`.
-
-
-
-
-
-
-
-## Compose documentation
-
-- [User guide](/)
-- [Installing Compose](install.md)
-- [Get started with Django](django.md)
-- [Get started with Rails](rails.md)
-- [Get started with Wordpress](wordpress.md)
-- [Yaml file reference](yml.md)
-- [Compose environment variables](env.md)
-- [Compose command line completion](completion.md)

docs/reference/port.md

@@ -1,23 +0,0 @@
-<!--[metadata]>
-+++
-title = "port"
-description = "Prints the public port for a port binding.s"
-keywords = ["fig, composition, compose, docker, orchestration, cli,  port"]
-[menu.main]
-identifier="port.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# port
-
-```
-Usage: port [options] SERVICE PRIVATE_PORT
-
-Options:
---protocol=proto  tcp or udp [default: tcp]
---index=index     index of the container if there are multiple
-                  instances of a service [default: 1]
-```
-
-Prints the public port for a port binding.

docs/reference/ps.md

@@ -1,21 +0,0 @@
-<!--[metadata]>
-+++
-title = "ps"
-description = "Lists containers."
-keywords = ["fig, composition, compose, docker, orchestration, cli,  ps"]
-[menu.main]
-identifier="ps.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# ps
-
-```
-Usage: ps [options] [SERVICE...]
-
-Options:
--q    Only display IDs
-```
-
-Lists containers.

docs/reference/pull.md

@@ -1,18 +0,0 @@
-<!--[metadata]>
-+++
-title = "pull"
-description = "Pulls service images."
-keywords = ["fig, composition, compose, docker, orchestration, cli,  pull"]
-[menu.main]
-identifier="pull.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# pull
-
-```
-Usage: pull [options] [SERVICE...]
-```
-
-Pulls service images.

docs/reference/restart.md

@@ -1,21 +0,0 @@
-<!--[metadata]>
-+++
-title = "restart"
-description = "Restarts Docker Compose services."
-keywords = ["fig, composition, compose, docker, orchestration, cli,  restart"]
-[menu.main]
-identifier="restart.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# restart
-
-```
-Usage: restart [options] [SERVICE...]
-
-Options:
--t, --timeout TIMEOUT      Specify a shutdown timeout in seconds. (default: 10)
-```
-
-Restarts services.

docs/reference/rm.md

@@ -1,22 +0,0 @@
-<!--[metadata]>
-+++
-title = "rm"
-description = "Removes stopped service containers."
-keywords = ["fig, composition, compose, docker, orchestration, cli,  rm"]
-[menu.main]
-identifier="rm.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# rm
-
-```
-Usage: rm [options] [SERVICE...]
-
-Options:
--f, --force   Don't ask to confirm removal
--v            Remove volumes associated with containers
-```
-
-Removes stopped service containers.

docs/reference/run.md

@@ -1,53 +0,0 @@
-<!--[metadata]>
-+++
-title = "run"
-description = "Runs a one-off command on a service."
-keywords = ["fig, composition, compose, docker, orchestration, cli,  run"]
-[menu.main]
-identifier="run.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# run
-
-```
-Usage: run [options] [-e KEY=VAL...] SERVICE [COMMAND] [ARGS...]
-
-Options:
--d                    Detached mode: Run container in the background, print
-                          new container name.
---entrypoint CMD      Override the entrypoint of the image.
--e KEY=VAL            Set an environment variable (can be used multiple times)
--u, --user=""         Run as specified username or uid
---no-deps             Don't start linked services.
---rm                  Remove container after run. Ignored in detached mode.
---service-ports       Run command with the service's ports enabled and mapped to the host.
--T                    Disable pseudo-tty allocation. By default `docker-compose run` allocates a TTY.
-```
-
-Runs a one-time command against a service. For example, the following command starts the `web` service and runs `bash` as its command. 
-
-    $ docker-compose run web bash
-
-Commands you use with `run` start in new containers with the same configuration as defined by the service' configuration. This means the container has the same volumes, links, as defined in the configuration file. There two differences though.
-
-First, the command passed by `run` overrides the command defined in the service configuration. For example, if the  `web` service configuration is started with `bash`, then `docker-compose run web python app.py` overrides it with `python app.py`.
-
-The second difference is the `docker-compose run` command does not create any of the ports specified in the service configuration. This prevents the port collisions with already open ports. If you *do want* the service's ports created and mapped to the host, specify the `--service-ports` flag:
-
-    $ docker-compose run --service-ports web python manage.py shell
-
-If you start a service configured with links, the `run` command first checks to see if the linked service is running and starts the service if it is stopped.  Once all the linked services are running, the `run` executes the command you passed it.  So, for example, you could run:
-
-    $ docker-compose run db psql -h db -U docker
-
-This would open up an interactive PostgreSQL shell for the linked `db` container.
-
-If you do not want the `run` command to start linked containers, specify the `--no-deps` flag:
-
-    $ docker-compose run --no-deps web python manage.py shell
-
-
-
-

docs/reference/scale.md

@@ -1,21 +0,0 @@
-<!--[metadata]>
-+++
-title = "scale"
-description = "Sets the number of containers to run for a service."
-keywords = ["fig, composition, compose, docker, orchestration, cli,  scale"]
-[menu.main]
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# scale
-
-```
-Usage: scale [SERVICE=NUM...]
-```
-
-Sets the number of containers to run for a service.
-
-Numbers are specified as arguments in the form `service=num`. For example:
-
-    $ docker-compose scale web=2 worker=3

docs/reference/start.md

@@ -1,18 +0,0 @@
-<!--[metadata]>
-+++
-title = "start"
-description = "Starts existing containers for a service."
-keywords = ["fig, composition, compose, docker, orchestration, cli,  start"]
-[menu.main]
-identifier="start.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# start
-
-```
-Usage: start [SERVICE...]
-```
-
-Starts existing containers for a service.

docs/reference/stop.md

@@ -1,22 +0,0 @@
-<!--[metadata]>
-+++
-title = "stop"
-description = "Stops running containers without removing them. "
-keywords = ["fig, composition, compose, docker, orchestration, cli, stop"]
-[menu.main]
-identifier="stop.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# stop
-
-```
-Usage: stop [options] [SERVICE...]
-
-Options:
--t, --timeout TIMEOUT      Specify a shutdown timeout in seconds (default: 10).
-```
-
-Stops running containers without removing them. They can be started again with
-`docker-compose start`.

docs/reference/up.md

@@ -1,47 +0,0 @@
-<!--[metadata]>
-+++
-title = "up"
-description = "Builds, (re)creates, starts, and attaches to containers for a service."
-keywords = ["fig, composition, compose, docker, orchestration, cli,  up"]
-[menu.main]
-identifier="up.compose"
-parent = "smn_compose_cli"
-+++
-<![end-metadata]-->
-
-# up
-
-```
-Usage: up [options] [SERVICE...]
-
-Options:
--d                     Detached mode: Run containers in the background,
-                       print new container names.
---no-color             Produce monochrome output.
---no-deps              Don't start linked services.
---force-recreate       Recreate containers even if their configuration and
-                       image haven't changed. Incompatible with --no-recreate.
---no-recreate          If containers already exist, don't recreate them.
-                       Incompatible with --force-recreate.
---no-build             Don't build an image, even if it's missing
--t, --timeout TIMEOUT  Use this timeout in seconds for container shutdown
-                       when attached or when containers are already
-                       running. (default: 10)
-```
-
-Builds, (re)creates, starts, and attaches to containers for a service.
-
-Unless they are already running, this command also starts any linked services.
-
-The `docker-compose up` command aggregates the output of each container. When
-the command exits, all containers are stopped. Running `docker-compose up -d`
-starts the containers in the background and leaves them running.
-
-If there are existing containers for a service, and the service's configuration
-or image was changed after the container's creation, `docker-compose up` picks
-up the changes by stopping and recreating the containers (preserving mounted
-volumes). To prevent Compose from picking up changes, use the `--no-recreate`
-flag.
-
-If you want to force Compose to stop and recreate all containers, use the
-`--force-recreate` flag.

docs/wordpress.md

@@ -1,123 +0,0 @@
-<!--[metadata]>
-+++
-title = "Quickstart Guide: Compose and Wordpress"
-description = "Getting started with Compose and Wordpress"
-keywords = ["documentation, docs,  docker, compose, orchestration, containers"]
-[menu.main]
-parent="smn_workw_compose"
-weight=6
-+++
-<![end-metadata]-->
-
-
-# Quickstart Guide: Compose and Wordpress
-
-You can use Compose to easily run Wordpress in an isolated environment built
-with Docker containers. 
-
-## Define the project
-
-First, [Install Compose](install.md) and then download Wordpress into the
-current directory:
-
-    $ curl https://wordpress.org/latest.tar.gz | tar -xvzf -
-
-This will create a directory called `wordpress`. If you wish, you can rename it
-to the name of your project.
-
-Next, inside that directory, create a `Dockerfile`, a file that defines what
-environment your app is going to run in. For more information on how to write
-Dockerfiles, see the
-[Docker user guide](https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile) and the
-[Dockerfile reference](http://docs.docker.com/reference/builder/). In this case,
-your Dockerfile should be:
-
-    FROM orchardup/php5
-    ADD . /code
-
-This tells Docker how to build an image defining a container that contains PHP
-and Wordpress. 
-
-Next you'll create a `docker-compose.yml` file that will start your web service
-and a separate MySQL instance:
-
-    web:
-      build: .
-      command: php -S 0.0.0.0:8000 -t /code
-      ports:
-        - "8000:8000"
-      links:
-        - db
-      volumes:
-        - .:/code
-    db:
-      image: orchardup/mysql
-      environment:
-        MYSQL_DATABASE: wordpress
-
-Two supporting files are needed to get this working - first, `wp-config.php` is
-the standard Wordpress config file with a single change to point the database
-configuration at the `db` container:
-
-    <?php
-    define('DB_NAME', 'wordpress');
-    define('DB_USER', 'root');
-    define('DB_PASSWORD', '');
-    define('DB_HOST', "db:3306");
-    define('DB_CHARSET', 'utf8');
-    define('DB_COLLATE', '');
-
-    define('AUTH_KEY',         'put your unique phrase here');
-    define('SECURE_AUTH_KEY',  'put your unique phrase here');
-    define('LOGGED_IN_KEY',    'put your unique phrase here');
-    define('NONCE_KEY',        'put your unique phrase here');
-    define('AUTH_SALT',        'put your unique phrase here');
-    define('SECURE_AUTH_SALT', 'put your unique phrase here');
-    define('LOGGED_IN_SALT',   'put your unique phrase here');
-    define('NONCE_SALT',       'put your unique phrase here');
-
-    $table_prefix  = 'wp_';
-    define('WPLANG', '');
-    define('WP_DEBUG', false);
-
-    if ( !defined('ABSPATH') )
-        define('ABSPATH', dirname(__FILE__) . '/');
-
-    require_once(ABSPATH . 'wp-settings.php');
-
-Second, `router.php` tells PHP's built-in web server how to run Wordpress:
-
-    <?php
-
-    $root = $_SERVER['DOCUMENT_ROOT'];
-    chdir($root);
-    $path = '/'.ltrim(parse_url($_SERVER['REQUEST_URI'])['path'],'/');
-    set_include_path(get_include_path().':'.__DIR__);
-    if(file_exists($root.$path))
-    {
-        if(is_dir($root.$path) && substr($path,strlen($path) - 1, 1) !== '/')
-            $path = rtrim($path,'/').'/index.php';
-        if(strpos($path,'.php') === false) return false;
-        else {
-            chdir(dirname($root.$path));
-            require_once $root.$path;
-        }
-    }else include_once 'index.php';
-
-### Build the project
-
-With those four files in place, run `docker-compose up` inside your Wordpress
-directory and it'll pull and build the needed images, and then start the web and
-database containers. If you're using [Docker Machine](https://docs.docker.com/machine), then `docker-machine ip MACHINE_VM` gives you the machine address and you can open `http://MACHINE_VM_IP:8000` in a browser. 
-
-## More Compose documentation
-
-- [User guide](/)
-- [Installing Compose](install.md)
-- [Get started with Django](django.md)
-- [Get started with Rails](rails.md)
-- [Get started with Wordpress](wordpress.md)
-- [Command line reference](cli.md)
-- [Yaml file reference](yml.md)
-- [Compose environment variables](env.md)
-- [Compose command line completion](completion.md)

docs/yml.md

@@ -1,382 +0,0 @@
-<!--[metadata]>
-+++
-title = "docker-compose.yml reference"
-description = "docker-compose.yml reference"
-keywords = ["fig, composition, compose,  docker"]
-[menu.main]
-parent="smn_compose_ref"
-+++
-<![end-metadata]-->
-
-
-# docker-compose.yml reference
-
-Each service defined in `docker-compose.yml` must specify exactly one of
-`image` or `build`. Other keys are optional, and are analogous to their
-`docker run` command-line counterparts.
-
-As with `docker run`, options specified in the Dockerfile (e.g., `CMD`,
-`EXPOSE`, `VOLUME`, `ENV`) are respected by default - you don't need to
-specify them again in `docker-compose.yml`.
-
-### image
-
-Tag or partial image ID. Can be local or remote - Compose will attempt to
-pull if it doesn't exist locally.
-
-    image: ubuntu
-    image: orchardup/postgresql
-    image: a4bc65fd
-
-### build
-
-Path to a directory containing a Dockerfile. When the value supplied is a
-relative path, it is interpreted as relative to the location of the yml file
-itself. This directory is also the build context that is sent to the Docker daemon.
-
-Compose will build and tag it with a generated name, and use that image thereafter.
-
-    build: /path/to/build/dir
-
-### dockerfile
-
-Alternate Dockerfile.
-
-Compose will use an alternate file to build with.
-
-    dockerfile: Dockerfile-alternate
-
-### command
-
-Override the default command.
-
-    command: bundle exec thin -p 3000
-
-<a name="links"></a>
-### links
-
-Link to containers in another service. Either specify both the service name and
-the link alias (`SERVICE:ALIAS`), or just the service name (which will also be
-used for the alias).
-
-    links:
-     - db
-     - db:database
-     - redis
-
-An entry with the alias' name will be created in `/etc/hosts` inside containers
-for this service, e.g:
-
-    172.17.2.186  db
-    172.17.2.186  database
-    172.17.2.187  redis
-
-Environment variables will also be created - see the [environment variable
-reference](env.md) for details.
-
-### external_links
-
-Link to containers started outside this `docker-compose.yml` or even outside
-of Compose, especially for containers that provide shared or common services.
-`external_links` follow semantics similar to `links` when specifying both the
-container name and the link alias (`CONTAINER:ALIAS`).
-
-    external_links:
-     - redis_1
-     - project_db_1:mysql
-     - project_db_1:postgresql
-
-### extra_hosts
-
-Add hostname mappings. Use the same values as the docker client `--add-host` parameter.
-
-    extra_hosts:
-     - "somehost:162.242.195.82"
-     - "otherhost:50.31.209.229"
-
-An entry with the ip address and hostname will be created in `/etc/hosts` inside containers for this service, e.g:
-
-    162.242.195.82  somehost
-    50.31.209.229   otherhost
-
-### ports
-
-Expose ports. Either specify both ports (`HOST:CONTAINER`), or just the container
-port (a random host port will be chosen).
-
-> **Note:** When mapping ports in the `HOST:CONTAINER` format, you may experience
-> erroneous results when using a container port lower than 60, because YAML will
-> parse numbers in the format `xx:yy` as sexagesimal (base 60). For this reason,
-> we recommend always explicitly specifying your port mappings as strings.
-
-    ports:
-     - "3000"
-     - "8000:8000"
-     - "49100:22"
-     - "127.0.0.1:8001:8001"
-
-### expose
-
-Expose ports without publishing them to the host machine - they'll only be
-accessible to linked services. Only the internal port can be specified.
-
-    expose:
-     - "3000"
-     - "8000"
-
-### volumes
-
-Mount paths as volumes, optionally specifying a path on the host machine
-(`HOST:CONTAINER`), or an access mode (`HOST:CONTAINER:ro`).
-
-    volumes:
-     - /var/lib/mysql
-     - ./cache:/tmp/cache
-     - ~/configs:/etc/configs/:ro
-
-You can mount a relative path on the host, which will expand relative to
-the directory of the Compose configuration file being used. Relative paths
-should always begin with `.` or `..`.
-
-> Note: No path expansion will be done if you have also specified a
-> `volume_driver`.
-
-### volumes_from
-
-Mount all of the volumes from another service or container.
-
-    volumes_from:
-     - service_name
-     - container_name
-
-### environment
-
-Add environment variables. You can use either an array or a dictionary.
-
-Environment variables with only a key are resolved to their values on the
-machine Compose is running on, which can be helpful for secret or host-specific values.
-
-    environment:
-      RACK_ENV: development
-      SESSION_SECRET:
-
-    environment:
-      - RACK_ENV=development
-      - SESSION_SECRET
-
-### env_file
-
-Add environment variables from a file. Can be a single value or a list.
-
-If you have specified a Compose file with `docker-compose -f FILE`, paths in
-`env_file` are relative to the directory that file is in.
-
-Environment variables specified in `environment` override these values.
-
-    env_file: .env
-
-    env_file:
-      - ./common.env
-      - ./apps/web.env
-      - /opt/secrets.env
-
-Compose expects each line in an env file to be in `VAR=VAL` format. Lines
-beginning with `#` (i.e. comments) are ignored, as are blank lines.
-
-    # Set Rails/Rack environment
-    RACK_ENV=development
-
-### extends
-
-Extend another service, in the current file or another, optionally overriding
-configuration.
-
-Here's a simple example. Suppose we have 2 files - **common.yml** and
-**development.yml**. We can use `extends` to define a service in
-**development.yml** which uses configuration defined in **common.yml**:
-
-**common.yml**
-
-    webapp:
-      build: ./webapp
-      environment:
-        - DEBUG=false
-        - SEND_EMAILS=false
-
-**development.yml**
-
-    web:
-      extends:
-        file: common.yml
-        service: webapp
-      ports:
-        - "8000:8000"
-      links:
-        - db
-      environment:
-        - DEBUG=true
-    db:
-      image: postgres
-
-Here, the `web` service in **development.yml** inherits the configuration of
-the `webapp` service in **common.yml** - the `build` and `environment` keys -
-and adds `ports` and `links` configuration. It overrides one of the defined
-environment variables (DEBUG) with a new value, and the other one
-(SEND_EMAILS) is left untouched.
-
-The `file` key is optional, if it is not set then Compose will look for the
-service within the current file.
-
-For more on `extends`, see the [tutorial](extends.md#example) and
-[reference](extends.md#reference).
-
-### labels
-
-Add metadata to containers using [Docker labels](http://docs.docker.com/userguide/labels-custom-metadata/). You can use either an array or a dictionary.
-
-It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software.
-
-    labels:
-      com.example.description: "Accounting webapp"
-      com.example.department: "Finance"
-      com.example.label-with-empty-value: ""
-
-    labels:
-      - "com.example.description=Accounting webapp"
-      - "com.example.department=Finance"
-      - "com.example.label-with-empty-value"
-
-### container_name
-
-Specify a custom container name, rather than a generated default name.
-
-    container_name: my-web-container
-
-Because Docker container names must be unique, you cannot scale a service
-beyond 1 container if you have specified a custom name. Attempting to do so
-results in an error.
-
-### log driver
-
-Specify a logging driver for the service's containers, as with the ``--log-driver`` option for docker run ([documented here](http://docs.docker.com/reference/run/#logging-drivers-log-driver)).
-
-Allowed values are currently ``json-file``, ``syslog`` and ``none``. The list will change over time as more drivers are added to the Docker engine.
-
-The default value is json-file.
-
-    log_driver: "json-file"
-    log_driver: "syslog"
-    log_driver: "none"
-
-Specify logging options with `log_opt` for the logging driver, as with the ``--log-opt`` option for `docker run`.
-
-Logging options are key value pairs. An example of `syslog` options:
-
-    log_driver: "syslog"
-    log_opt:
-      address: "tcp://192.168.0.42:123"
-
-### net
-
-Networking mode. Use the same values as the docker client `--net` parameter.
-
-    net: "bridge"
-    net: "none"
-    net: "container:[name or id]"
-    net: "host"
-
-### pid
-
-    pid: "host"
-
-Sets the PID mode to the host PID mode.  This turns on sharing between
-container and the host operating system the PID address space.  Containers
-launched with this flag will be able to access and manipulate other
-containers in the bare-metal machine's namespace and vise-versa.
-
-### dns
-
-Custom DNS servers. Can be a single value or a list.
-
-    dns: 8.8.8.8
-    dns:
-      - 8.8.8.8
-      - 9.9.9.9
-
-### cap_add, cap_drop
-
-Add or drop container capabilities.
-See `man 7 capabilities` for a full list.
-
-    cap_add:
-      - ALL
-
-    cap_drop:
-      - NET_ADMIN
-      - SYS_ADMIN
-
-### dns_search
-
-Custom DNS search domains. Can be a single value or a list.
-
-    dns_search: example.com
-    dns_search:
-      - dc1.example.com
-      - dc2.example.com
-
-### devices
-
-List of device mappings.  Uses the same format as the `--device` docker 
-client create option.
-
-    devices:
-      - "/dev/ttyUSB0:/dev/ttyUSB0"
-
-### security_opt
-
-Override the default labeling scheme for each container.
-
-      security_opt:
-        - label:user:USER
-        - label:role:ROLE
-
-### working\_dir, entrypoint, user, hostname, domainname, mac\_address, mem\_limit, memswap\_limit, privileged, restart, stdin\_open, tty, cpu\_shares, cpuset, read\_only, volume\_driver
-
-Each of these is a single value, analogous to its
-[docker run](https://docs.docker.com/reference/run/) counterpart.
-
-    cpu_shares: 73
-    cpuset: 0,1
-
-    working_dir: /code
-    entrypoint: /code/entrypoint.sh
-    user: postgresql
-
-    hostname: foo
-    domainname: foo.com
-
-    mac_address: 02:42:ac:11:65:43
-
-    mem_limit: 1000000000
-    memswap_limit: 2000000000
-    privileged: true
-
-    restart: always
-
-    stdin_open: true
-    tty: true
-    read_only: true
-
-    volume_driver: mydriver
-```
-
-## Compose documentation
-
-- [User guide](/)
-- [Installing Compose](install.md)
-- [Get started with Django](django.md)
-- [Get started with Rails](rails.md)
-- [Get started with Wordpress](wordpress.md)
-- [Command line reference](cli.md)
-- [Compose environment variables](env.md)
-- [Compose command line completion](completion.md)

experimental/compose_swarm_networking.md

@@ -1,183 +1,5 @@
 # Experimental: Compose, Swarm and Multi-Host Networking
 
-The [experimental build of Docker](https://github.com/docker/docker/tree/master/experimental) has an entirely new networking system, which enables secure communication between containers on multiple hosts. In combination with Docker Swarm and Docker Compose, you can now run multi-container apps on multi-host clusters with the same tooling and configuration format you use to develop them locally.
+Compose now supports multi-host networking as standard. Read more here:
 
-> Note: This functionality is in the experimental stage, and contains some hacks and workarounds which will be removed as it matures.
-
-## Prerequisites
-
-Before you start, you’ll need to install the experimental build of Docker, and the latest versions of Machine and Compose.
-
--   To install the experimental Docker build on a Linux machine, follow the instructions [here](https://github.com/docker/docker/tree/master/experimental#install-docker-experimental).
-
--   To install the experimental Docker build on a Mac, run these commands:
-
-        $ curl -L https://experimental.docker.com/builds/Darwin/x86_64/docker-latest > /usr/local/bin/docker
-        $ chmod +x /usr/local/bin/docker
-
--   To install Machine, follow the instructions [here](http://docs.docker.com/machine/).
-
--   To install Compose, follow the instructions [here](http://docs.docker.com/compose/install/).
-
-You’ll also need a [Docker Hub](https://hub.docker.com/account/signup/) account and a [Digital Ocean](https://www.digitalocean.com/) account.
-
-## Set up a swarm with multi-host networking
-
-Set the `DIGITALOCEAN_ACCESS_TOKEN` environment variable to a valid Digital Ocean API token, which you can generate in the [API panel](https://cloud.digitalocean.com/settings/applications).
-
-    DIGITALOCEAN_ACCESS_TOKEN=abc12345
-
-Start a consul server:
-
-    docker-machine create -d digitalocean --engine-install-url https://experimental.docker.com consul
-    docker $(docker-machine config consul) run -d -p 8500:8500 -h consul progrium/consul -server -bootstrap
-
-(In a real world setting you’d set up a distributed consul, but that’s beyond the scope of this guide!)
-
-Create a Swarm token:
-
-    SWARM_TOKEN=$(docker run swarm create)
-
-Create a Swarm master:
-
-    docker-machine create -d digitalocean --swarm --swarm-master --swarm-discovery=token://$SWARM_TOKEN --engine-install-url="https://experimental.docker.com" --digitalocean-image "ubuntu-14-10-x64" --engine-opt=default-network=overlay:multihost --engine-label=com.docker.network.driver.overlay.bind_interface=eth0 --engine-opt=kv-store=consul:$(docker-machine ip consul):8500 swarm-0
-
-Create a Swarm node:
-
-    docker-machine create -d digitalocean --swarm --swarm-discovery=token://$SWARM_TOKEN --engine-install-url="https://experimental.docker.com" --digitalocean-image "ubuntu-14-10-x64" --engine-opt=default-network=overlay:multihost --engine-label=com.docker.network.driver.overlay.bind_interface=eth0 --engine-opt=kv-store=consul:$(docker-machine ip consul):8500 --engine-label com.docker.network.driver.overlay.neighbor_ip=$(docker-machine ip swarm-0) swarm-1
-
-You can create more Swarm nodes if you want - it’s best to give them sensible names (swarm-2, swarm-3, etc).
-
-Finally, point Docker at your swarm:
-
-    eval "$(docker-machine env --swarm swarm-0)"
-
-## Run containers and get them communicating
-
-Now that you’ve got a swarm up and running, you can create containers on it just like a single Docker instance:
-
-    $ docker run busybox echo hello world
-    hello world
-
-If you run `docker ps -a`, you can see what node that container was started on by looking at its name (here it’s swarm-3):
-
-    $ docker ps -a
-    CONTAINER ID        IMAGE                      COMMAND                CREATED              STATUS                      PORTS                                   NAMES
-    41f59749737b        busybox                    "echo hello world"     15 seconds ago       Exited (0) 13 seconds ago                                           swarm-3/trusting_leakey
-
-As you start more containers, they’ll be placed on different nodes across the cluster, thanks to Swarm’s default “spread” scheduling strategy.
-
-Every container started on this swarm will use the “overlay:multihost” network by default, meaning they can all intercommunicate. Each container gets an IP address on that network, and an `/etc/hosts` file which will be updated on-the-fly with every other container’s IP address and name. That means that if you have a running container named ‘foo’, other containers can access it at the hostname ‘foo’.
-
-Let’s verify that multi-host networking is functioning. Start a long-running container:
-
-    $ docker run -d --name long-running busybox top
-    <container id>
-
-If you start a new container and inspect its /etc/hosts file, you’ll see the long-running container in there:
-
-    $ docker run busybox cat /etc/hosts
-    ...
-    172.21.0.6  long-running
-
-Verify that connectivity works between containers:
-
-    $ docker run busybox ping long-running
-    PING long-running (172.21.0.6): 56 data bytes
-    64 bytes from 172.21.0.6: seq=0 ttl=64 time=7.975 ms
-    64 bytes from 172.21.0.6: seq=1 ttl=64 time=1.378 ms
-    64 bytes from 172.21.0.6: seq=2 ttl=64 time=1.348 ms
-    ^C
-    --- long-running ping statistics ---
-    3 packets transmitted, 3 packets received, 0% packet loss
-    round-trip min/avg/max = 1.140/2.099/7.975 ms
-
-## Run a Compose application
-
-Here’s an example of a simple Python + Redis app using multi-host networking on a swarm.
-
-Create a directory for the app:
-
-    $ mkdir composetest
-    $ cd composetest
-
-Inside this directory, create 2 files.
-
-First, create `app.py` - a simple web app that uses the Flask framework and increments a value in Redis:
-
-    from flask import Flask
-    from redis import Redis
-    import os
-    app = Flask(__name__)
-    redis = Redis(host='composetest_redis_1', port=6379)
-
-    @app.route('/')
-    def hello():
-        redis.incr('hits')
-        return 'Hello World! I have been seen %s times.' % redis.get('hits')
-
-    if __name__ == "__main__":
-        app.run(host="0.0.0.0", debug=True)
-
-Note that we’re connecting to a host called `composetest_redis_1` - this is the name of the Redis container that Compose will start.
-
-Second, create a Dockerfile for the app container:
-
-    FROM python:2.7
-    RUN pip install flask redis
-    ADD . /code
-    WORKDIR /code
-    CMD ["python", "app.py"]
-
-Build the Docker image and push it to the Hub (you’ll need a Hub account). Replace `<username>` with your Docker Hub username:
-
-    $ docker build -t <username>/counter .
-    $ docker push <username>/counter
-
-Next, create a `docker-compose.yml`, which defines the configuration for the web and redis containers. Once again, replace `<username>` with your Hub username:
-
-    web:
-      image: <username>/counter
-      ports:
-       - "80:5000"
-    redis:
-      image: redis
-
-Now start the app:
-
-    $ docker-compose up -d
-    Pulling web (username/counter:latest)...
-    swarm-0: Pulling username/counter:latest... : downloaded
-    swarm-2: Pulling username/counter:latest... : downloaded
-    swarm-1: Pulling username/counter:latest... : downloaded
-    swarm-3: Pulling username/counter:latest... : downloaded
-    swarm-4: Pulling username/counter:latest... : downloaded
-    Creating composetest_web_1...
-    Pulling redis (redis:latest)...
-    swarm-2: Pulling redis:latest... : downloaded
-    swarm-1: Pulling redis:latest... : downloaded
-    swarm-3: Pulling redis:latest... : downloaded
-    swarm-4: Pulling redis:latest... : downloaded
-    swarm-0: Pulling redis:latest... : downloaded
-    Creating composetest_redis_1...
-
-Swarm has created containers for both web and redis, and placed them on different nodes, which you can check with `docker ps`:
-
-    $ docker ps
-    CONTAINER ID        IMAGE                      COMMAND                CREATED             STATUS              PORTS                                  NAMES
-    92faad2135c9        redis                      "/entrypoint.sh redi   43 seconds ago      Up 42 seconds                                              swarm-2/composetest_redis_1
-    adb809e5cdac        username/counter           "/bin/sh -c 'python    55 seconds ago      Up 54 seconds       45.67.8.9:80->5000/tcp                 swarm-1/composetest_web_1
-
-You can also see that the web container has exposed port 80 on its swarm node. If you curl that IP, you’ll get a response from the container:
-
-    $ curl http://45.67.8.9
-    Hello World! I have been seen 1 times.
-
-If you hit it repeatedly, the counter will increment, demonstrating that the web and redis container are communicating:
-
-    $ curl http://45.67.8.9
-    Hello World! I have been seen 2 times.
-    $ curl http://45.67.8.9
-    Hello World! I have been seen 3 times.
-    $ curl http://45.67.8.9
-    Hello World! I have been seen 4 times.
+https://docs.docker.com/compose/networking

project/ISSUE-TRIAGE.md

@@ -0,0 +1,35 @@
+Triaging of issues
+------------------
+
+The docker-compose issue triage process follows
+https://github.com/docker/docker/blob/master/project/ISSUE-TRIAGE.md
+with the following additions or exceptions.
+
+
+### Classify the Issue
+
+The following labels are provided in additional to the standard labels:
+
+| Kind         | Description                                                       |
+|--------------|-------------------------------------------------------------------|
+| kind/cleanup | A refactor or improvement that is related to quality not function |
+| kind/parity  | A request for feature parity with docker cli                      |
+
+
+### Functional areas
+
+Most issues should fit into one of the following functional areas:
+
+| Area            |
+|-----------------|
+| area/build      |
+| area/cli        |
+| area/config     |
+| area/logs       |
+| area/networking |
+| area/packaging  |
+| area/run        |
+| area/scale      |
+| area/tests      |
+| area/up         |
+| area/volumes    |