This commit allows `docker-compose` to access `top` for containers
much like running `docker top` directly on a given container.
This commit includes:
* `docker-compose` CLI changes to expose `top`
* Completions for `bash` and `zsh`
* Required testing for the new `top` command
Signed-off-by: Peter Urda <peter.urda@gmail.com>
Fixes#3349
This allows the key `userns_mode` to be used in service definitions.
Since `userns_mode` requires API version > 1.23, this is only available
in 2.1 and 3.0 versions of compose file
Signed-off-by: Yong Wen Chua <me@yongwen.xyz>
The ANSI escape codes \e[0A (cursor up 0 lines) and \e[0B (cursor down 0 lines)
are not well defined and are treated differently by different terminals. In
particular xterm treats 0 as a missing parameter and therefore defaults to 1,
whereas rxvt-unicode treats these escapes as a request to move 0 lines.
However the use of these codes is unnecessary and were really just hiding the
fact that we were not correctly computing diff when adding a new line. Having
added the new line to the ids map and output the corresponding \n the correct
diff would be 1 and not 0 (which xterm interprets as 1) as currently.
Rather than changing the hardcoded 0 to a 1 pull the diff calculation out and
always do it since it produces the correct answer in both cases.
This fixes similar corruption when compose is pulling an image to that seen
with `docker pull` and rxvt-unicode (and likely other terminals in that family)
seen in docker/docker#28111.
This is the same as the fix made to Docker's pkg/jsonmessage in
https://github.com/docker/docker/pull/28238 (and I have shamelessly ripped off
most of this commit message from there).
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Before this change:
```
$ docker-compose --file docker-compose.yml -<TAB>
-- option --
--help -h -- Get help
--host -H -- Daemon socket to connect to
--project-name -p -- Specify an alternate project name (default: directory name)
--skip-hostname-check -- Don't check the daemon's hostname against the name specified in the client certificate (for example if your docker host is an IP address)
--tls -- Use TLS; implied by --tlsverify
--tlscacert -- Trust certs signed only by this CA
--tlscert -- Path to TLS certificate file
--tlskey -- Path to TLS key file
--tlsverify -- Use TLS and verify the remote
--verbose -- Show more output
--version -v -- Print version and exit
```
(Note the `--file` argument is no longer available to complete.)
After this change:
```
docker-compose --file docker-compose.yml -<TAB>
-- option --
--file -f -- Specify an alternate docker-compose file (default: docker-compose.yml)
--help -h -- Get help
--host -H -- Daemon socket to connect to
--project-name -p -- Specify an alternate project name (default: directory name)
--skip-hostname-check -- Don't check the daemon's hostname against the name specified in the client certificate (for example if your docker host is an IP address)
--tls -- Use TLS; implied by --tlsverify
--tlscacert -- Trust certs signed only by this CA
--tlscert -- Path to TLS certificate file
--tlskey -- Path to TLS key file
--tlsverify -- Use TLS and verify the remote
--verbose -- Show more output
--version -v -- Print version and exit
```
Signed-off-by: Matt Bray <mattjbray@gmail.com>
Receiving this message when using the -a flag : `--all flag is obsolete. This is now the default behavior of `docker-compose rm`, I proposed to mark it in the docs but I don't know which way is the best
Signed-off-by: jgsqware <garciagonzalez.julien@gmail.com>
Currently it doesn't actually seem to make any practical difference that this
is missing, but it seems like good practice to do so anyway, to be robust
against future test case changes which might require cleanup done in the super
class.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Each volume in the array holds a reference to a docker.client.Client object and
therefore a connection pool which leaves fds open once the test has completed.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
These hold a reference to a docker.client.Client object and therefore a
connection pool which leaves fds open once the test has completed.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
This is a docker.client.Client and therefore contains a connection pool, so
each subclass of DockerClientTestCase can end up holding on to up to 10 Unix
socket file descriptors after the tests contained in the sub-class are
complete.
Before this by the end of a test run I was seeing ~100 open file descriptors,
~80 of which were Unix domain sockets. By cleaning these up only 15 Unix
sockets remain at the end (out of ~25 fds, the rest of which are the Python
interpretter, opened libraries etc).
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
self._project.client is a docker.client.Client, so creating a new self._project
leaks (via the embedded connection pool) a bunch of Unix socket file
descriptors for each test which overrides self.project using this mechanism.
In my tests I observed the test harness using 800-900 file descriptor, which is
OK on Linux with the default limit of 1024 but breaks on OSX (e.g. with
Docker4Mac) where the default limit is only 256. The failure can be provoked on
Linux too with `ulimit -n 256`.
With this fix I have observed the process ending with ~100 file descriptors
open, including 83 Unix sockets, so I think there is likely at least one more
leak lurking.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Add note to say that environment variables will not be automatically
made available at build time, and point to the `args` documentation.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
- When no command is passed but `--entrypoint` is, set Cmd to `[]`
- When command is a single empty string, set Cmd to `[""]`
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Unset env vars passed to `run` via command line options take the
value of the system's var with the same name.
Signed-off-by: Joffrey F <joffrey@docker.com>
The postgres image expects a specific volume path. The docs had a typo in that path. This can cause a lot of agony. This is not hypothetical agony, but very real agony that was very recently experienced :)
Previously, the filtering on already selected services would break when
one service was a substring of another.
This commit fixes that.
Signed-off-by: Andre Eriksson <aepubemail@gmail.com>
Previously, the autocomplete for the build/pull commands would only add
services for which build/image were the _first_ keys, respectively, in
the docker-compose file.
This commit fixes this, so the appropriate services are listed
regardless of the order in which they appear
Signed-off-by: Andre Eriksson <aepubemail@gmail.com>
For autocomplete to work properly, we need to pass along some flags when
calling docker (--host, --tls, …) and docker-compose (--file, --tls, …).
Previously flags would only be passed to docker-compose, and the only
flags passed were --file and --project-name.
This commit makes sure that all relevant flags are passed to both
docker-compose and docker.
Signed-off-by: Andre Eriksson <aepubemail@gmail.com>
This has the added benefit of making autocompletion work when the
docker-compose config file is in a parent directory.
Signed-off-by: Andre Eriksson <aepubemail@gmail.com>
As per [their blog post of the 27th April](https://blog.readthedocs.com/securing-subdomains/) ‘Securing subdomains’:
> Starting today, Read the Docs will start hosting projects from subdomains on the domain readthedocs.io, instead of on readthedocs.org. This change addresses some security concerns around site cookies while hosting user generated data on the same domain as our dashboard.
Test Plan: Manually visited all the links I’ve modified.
Signed-off-by: Adam Chainz <adam@adamj.eu>
Fix the issue when build arg is set to None instead of empty string. Usecase:
cat docker-compose.yml
.... args:
- http_proxy
- https_proxy
- no_proxy
If http_proxy, https_proxy, no_proxy environment variables are not defined then http_proxy,
https_proxy, no_proxy build args will be set to string None which breaks all downloads
With this change undefined build args will be set to empty string instead of string None
Signed-off-by: Andrey Devyatkin <andrey.a.devyatkin@gmail.com>
Fix the issue when build arg is set to None instead of empty string. Usecase:
cat docker-compose.yml
.... args:
- http_proxy
- https_proxy
- no_proxy
If http_proxy, https_proxy, no_proxy environment variables are not defined then http_proxy,
https_proxy, no_proxy build args will be set to string None which breaks all downloads
With this change undefined build args will be set to empty string instead of string None
Signed-off-by: Andrey Devyatkin <andrey.a.devyatkin@gmail.com>
CLITestCase.test_events_human_readable fails due to
wrong assumption that host where tests were launched
will have the same date time as Docker daemon.
This fix introduces internal method for validating
timestamp in Docker logs
Signed-off-by: Denys Makogon <lildee1991@gmail.com>
Fix the issue when build arg is set to None instead of empty string.
Usecase:
cat docker-compose.yml
....
args:
- http_proxy
- https_proxy
- no_proxy
If http_proxy, https_proxy, no_proxy environment variables are not defined
then http_proxy, https_proxy, no_proxy build args will be set to string None which breaks all downloads
With this change build args will not passed to docker engine if they are equal to string None
Signed-off-by: Andrey Devyatkin <andrey.a.devyatkin@gmail.com>
The 'env_file' directive and feature precludes the use of the name
'.env' in the path shared with 'docker-config.yml', regardless of
whether or not it is enabled.
This change adds an additional validation to allow the use of this
path provided it is not a file.
Signed-off-by: Thom Linton <thom.linton@gmail.com>
This explanation looked like it was part of the error. Added an
extra new line and a bit of copy to explain the explanation.
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
"ERROR: Validation failed in file './docker-compose.yml', reason(s):"
is now:
"ERROR: The Compose file './docker-compose.yml' is invalid because:"
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
When jumping straight to this bit of the docs, it's not clear
that these are options under a service rather than the top-level
`networks` key. Added a service to make this super clear.
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
If processing of all objects finishes before the queue is drained,
parallel_execute_iter() returns prematurely.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
- Stop sharing set objects across threads
- Use a second queue to signal when producer threads are done
- Use a single consumer thread to check dependencies and kick off new
producers
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Bring it inline with current plans:
- Use in production is not necessarily about the command-line
tool, but also improving file format, integrations, new tools,
etc.
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
When using the latest version of Rails, the tutorial currently errors when running `docker-compose up` with the following error:
```
/usr/local/lib/ruby/gems/2.3.0/gems/bundler-1.11.2/lib/bundler/runtime.rb:80:in `rescue in block (2 levels) in require': There was an error while trying to load the gem 'uglifier'. (Bundler::GemRequireError)
```
Installing nodejs in the build fixes the issue.
Signed-off-by: Jon Lemmon <jon.lemmon@enspiral.com>
Up to now there were two special top-level options that required special
treatment: --project-name and --file (and their short forms).
For 1.7.0, several TLS related options were added that have to be passed
to secondary docker-compose invocations as well.
This commit introduces a scalable treatment of those options.
Signed-off-by: Harald Albers <github@albersweb.de>
Select tls version based of COMPOSE_TLS_VERSION
Changed from SSL to TLS
Also did docs - missing default value
Using getattr and raises AttributeError in case of unsupported version
Signed-off-by: Kalle Møller <github.com@k-moeller.dk>
Traceback (most recent call last):
File "/tmp/tmp.02tgGaAGtW/docker-compose/bin/docker-compose", line 11, in <module>
sys.exit(main())
File "/tmp/tmp.02tgGaAGtW/docker-compose/lib/python3.4/site-packages/compose/cli/main.py", line 68, in main
log_api_error(e)
File "/tmp/tmp.02tgGaAGtW/docker-compose/lib/python3.4/site-packages/compose/cli/main.py", line 89, in log_api_error
if 'client is newer than server' in e.explanation:
TypeError: 'str' does not support the buffer interface
Signed-off-by: Thomas Grainger <tagrain@gmail.com>
Add --remove-orphans to CLI reference docs
Add --remove-orphans to bash completion file
Test orphan warning and remove_orphan option in up
Signed-off-by: Joffrey F <joffrey@docker.com>
The Compose/Swarm integration has been working really well for
users, so it seems pretty safe to remove the scary warnings about
it not being ready.
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
- Added ipv4_network and ipv6_network to the networks section in the
service section for each configured network
- Added feature documentation
- Added unit tests
Signed-off-by: Matt Daue <mattdaue@gmail.com>
Also adds a warning when up builds an image without the --build flag
so that users know it wont happen on the next up.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Since 1.6.0, Compose supports multiple compose files specified with `-f`.
These need to be passed to the docker invocations done by the
completion.
Signed-off-by: Harald Albers <github@albersweb.de>
Raise a ShutdownException instead of a KeyboardInterupt when a thread.error
is caught. This thread.error is only raised when run from a pyinstaller
binary (for reasons unknown).
Signed-off-by: Daniel Nephin <dnephin@docker.com>
This requires a change to the make-branch script, to have it push the bump branch
to the docker remote instead of the user remote. Pushing to the docker remote
triggers the travis build, which builds the binary.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Display in the log output the name of those containers created using the scale command and
change the test_scale_with_api_error test to support the containers name when scale
Signed-off-by: Jesus Rodriguez Tinoco <jesus.rodriguez.tinoco@gmail.com>
- It happens on recreate, not just stop
- We now support `stop_signal`, which can help in some cases
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Reworked the two schema validation functions to read from the same schema but
use different parts of it. Error handling is now split as well by the
schema that is being used to validate.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
This reverts commit 677c50650c.
Revert "Modify service_test.py::ServiceTest::test_resolve_env to reflect new behavior"
This reverts commit 0019037712.
Revert "Mangle the tests. They pass for better or worse!"
This reverts commit 7ab9509ce6.
Revert "If an env var is passthrough but not defined on the host don't set it."
This reverts commit 6540efb3d3.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
After conversion a file would immediately not load in docker-compose 1.6.0 with the message:
ERROR: Version in "./converted.yml" is invalid - it should be a string.
Signed-off-by: Anthon van der Neut anthon@mnt.org
Signed-off-by: Anthon van der Neut <a.van.der.neut@ruamel.eu>
updated descriptions of changing file ownership and images per Seb's comments
fixed line wraps
fixed line breaks per Joffrey's comments
Signed-off-by: Victoria Bialas <victoria.bialas@docker.com>
If a warning is shown, and you happen to have no color setting in your (bash) prompt, the \033[37m setting, stays active. With the message hardly readable (light grey on my default light yellow background), that means the prompt is barely visible and you need to do `tput reset`.
Would probably be better if the background color was set as well in case you have dark on light theme by default in your terminal.
Signed-off-by: Anthon van der Neut <a.van.der.neut@ruamel.eu>
shm_size controls the size of /dev/shm in the container and requires
Docker 1.10 or newer (API version 1.22). This requires docker-py 1.8.0
(docker/docker-py#923).
Similar to fields like `mem_limit`, `shm_size` may be specified as
either an integer or a string (e.g., `64M`).
Updating docker-py to the master branch in order to get the unreleased
dependency on `shm_size` there in place.
Signed-off-by: Spencer Rinehart <anubis@overthemonkey.com>
Markdown was acting against expectations here by merging the example indented YAML into the previous list item instead of treating it as a code block.
I decided that a better way of handling this would be to add a "Simple example:" line that is also used elsewhere in this file. It resets the markdown indentation in a way that works.
Signed-off-by: Spencer Rinehart <anubis@overthemonkey.com>
This ensures and already existing `depends_on` is not deleted when the service on which it is defined also employs `extends`.
Signed-off-by: Ryan Taylor Long <ryan@rtlong.com>
- Instead of "Service 'web' configuration key 'image'", just say
"web.image"
- Fix the "Service 'services'" bug in the v2 file format
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Containers connected to the bridge network can't have aliases, so it's
simpler to rule that they can *either* be connected to the bridge
network (via `network_mode: bridge`) *or* be connected to user-defined
networks (via `networks` or the default network).
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
The previous approach assumed that the service list could be extracted
from a single file. It did not follow extends and overrides.
Signed-off-by: Harald Albers <github@albersweb.de>
- Explain each version in its own section
- Explain how to upgrade from version 1 to 2
- Note which keys are restricted to particular versions
- A few corrections to the docs for version-specific keys
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Make sure we connect the container to all required networks *after*
starting the container and *before* hijacking the terminal.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
This relies on an Engine behaviour which is a bug, not an intentional
feature - we have to connect to networks one at a time
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
- Don't run them against Engine < 1.10
- Set the API version appropriately for the Engine version, so all tests
use API version 1.22 against Engine 1.10
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
External volumes are created and managed by the user.
They are not namespaced.
They are expected to exist at the beginning of the up phase.
Signed-off-by: Joffrey F <joffrey@docker.com>
During the initialize_volumes phase, if a volume using the
non-namespaced name already exists, don't create the namespaced
equivalent.
Signed-off-by: Joffrey F <joffrey@docker.com>
- Remove --x-networking and --x-network-driver
- There's now no way to set a network driver - this will be added back
with the 'networks' key
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Assume version=1 if file is empty in get_config_version
Empty files are invalid anyway, so this simplifies the algorithm
somewhat.
https://github.com/docker/compose/pull/2421#discussion_r47223144
Don't leak version considerations in interpolation/service validation
Signed-off-by: Joffrey F <joffrey@docker.com>
When created through the compose file, volumes are prefixed
with the name of the project they belong to + underscore,
similarly to how containers are currently handled.
Signed-off-by: Joffrey F <joffrey@docker.com>
* Bump default API version to 1.21 (required for named volume management)
* Introduce new, versioned compose file format while maintaining support
for current (legacy) format
* Test updates to reflect changes made to the internal API
Signed-off-by: Joffrey F <joffrey@docker.com>
[Proof of read-write](cfb1b37da2/compose/config/types.py (L26)).
I found myself wondering what the default was a couple of times, and finally decided to change it :)
Signed-off-by: Michael Gilliland <gillilandmj@upmc.edu>
These files are created because we run acceptance tests in a subprocess.
They have the process id in their name, so they wont be removed by the
normal coverage cleanup on each run.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
this updates the MAINTAINERS file to the new format,
so that it can be parsed and collected in the docker/opensource
repository.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- The `raises` kwarg to the `cls_check` decorator was being used
incorrectly (it should be an exception class, not an object).
- We need to check for `error.cause` and get the message out of the
exception object.
NB: The particular case where validation fails in the case of `ports` is
only when ranges don't match in length - no further validation is
currently performed client-side.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
We were essentially only testing that *at least one* of the invalid
values fails the validation check, rather than that *all* of them fail.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Split env resolving into two phases. The first phase is to expand the paths
of env_files, which is done before merging extends. Once all files are merged
together, the final phase is to read the env_files and use them as the base
for environment variables.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Previously these empty objects would hit a bug in splitting objects causing it crash.
With this fix the empty objects are returned properly.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
See http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html
Environment variable names used by the utilities in the Shell and
Utilities volume of POSIX.1-2008 consist solely of uppercase letters,
digits, and the <underscore> ( '_' ) from the characters defined in
Portable Character Set and do not begin with a digit. Other characters may
be permitted by an implementation; applications shall tolerate the
presence of such names.
Signed-off-by: Joffrey F <joffrey@docker.com>
Refactors config validation of a service to use a ServiceConfig data object.
Instead of passing around a bunch of related scalars, we can use the
ServiceConfig object as a parameter to most of the service validation functions.
This allows for a fix to the config schema, where the name is a field in the
schema, but not actually in the configuration. My passing the name around as
part of the ServiceConfig object, we don't need to add it to the config options.
Fixes#2299
validate_against_service_schema() is moved from a conditional branch in
ServiceExtendsResolver() to happen as one of the last steps after all
configuration is merged. This schema only contains constraints which only need
to be true at the very end of merging.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
This bug can be seen by the change to the test case. When the extended service
uses a different name, the error was reported incorrectly.
By fixing this bug we can simplify self.signature and self.detect_cycles to
always use self.service_name.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
ServiceLoader has evolved to be not really all that related to "loading" a
service. It's responsibility is more to do with handling the `extends`
field, which is only part of loading. The class and its primary method
(make_service_dict()) were renamed to better reflect their responsibility.
As part of that change process_container_options() was removed from
make_service_dict() and renamed to process_service(). It contains logic for
handling the non-extends options.
This change allows us to remove the hacks from testcase.py and only call
the functions we need to format a service dict correctly for integration tests.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Consolidates all the top level config handling into `process_config_file` which
is now used for both files and merge sources.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
It's a flag passed to docker build that removes the intermediate
containers left behind on fail builds.
Signed-off-by: Adrian Budau <budau.adi@gmail.com>
Ensure config_hash is updated when volumes_from mode is changed, and
service is recreated on next up as a result.
Signed-off-by: Joffrey F <joffrey@docker.com>
We were outputting an extra line, which in *some* cases, on *some*
terminals, was causing the output of parallel actions to get messed up.
In particular, it would happen when the terminal had just been cleared
or hadn't yet filled up with a screen's worth of text.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
- make it a positional arg, since it's required
- make it the first argument for all functions that require it
- remove an unnecessary one-line function that was only called in one place
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Includes some refactoring of log_printer_test to support checking for flush(), and so that each test calls the unit-under-test directly, instead of through a helper function.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Start restructuring extends docs in preparation for adding documentation about using multiple compose files.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
It has been an unnecessary wrapper around container.start() for a little while now, so we can call it directly.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
These cli tests are now a different kind of that that run the compose binary. They are not the same as integration tests that test some internal interface.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Code cleanup. We no longer need this as the api returns a 304 for any
stopped containers, which doesn't raise an error.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
So we're not displaying output of all previous logs for a container, we attach,
if possible, to a container before the container is started.
LogPrinter checks if a container has a log_stream already attached and
print from that rather than always attempting to attach one itself.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
I've included Python 2/3 as they are not parent classifiers but
sibling classifiers. They denote that this project will work with
*some* versions of python and by having them, they'll show up for
people searching for python 2 or 3 projects.
According to the internet :)
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Mutable objects are harder to debug and harder to reason about. ServiceLoader was almost immutable. There was just a single function which set fields for a second function. Instead of mutating the object, we can pass those values as parameters to the next function.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
- fixed a docstring to make it PEP257 compliant
- wrapped some long lines
- used a more specific error
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Remove some unnecessary newlines.
Remove a unittest that was attempting to test behaviour that was removed a while ago, so isn't testing anything.
Updated some unit tests to use mocks instead of a custom fake.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
If there is a branch with the same name as a tag it fails without the
--tags. This was only a problem when we're branching from a git tag.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Environment keys that contain no value, get populated with values taken
from the environment not from the build phase but from running the command `up`.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This script assumes the typical environment that Windows users operate,
namely, VirtualBox running the boot2docker ISO, managed by
docker-machine. I wrote this script for my Windows using colleagues and
first placed it in the public domain as a gist:
https://gist.github.com/au-phiware/25213e72c80040f398ba
In short, that script works for me.
I have adapted that script to use the (yet to be) official image
(docker/compose:latest) and also added an extra environment variable to
provide additional options to docker run.
Signed-off-by: Corin Lawson <corin@phiware.com.au>
People are likely to run into their env vars being set to empty strings,
if they're not aware that they need to escape them for Compose to not
interpolate them.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
I'm not sure if it should be there at all, but at least it should hardly be where it currently is located.
Signed-off-by: Per Persson <per.persson@exertisztorm.com>
This change adds cgroup-parent support to compose project. It allows
each service to specify a 'cgroup_parent' option.
Signed-off-by: Mohit Soni <mosoni@paypal.com>
Also incorporated the structural changes by @moxiegirl in PR #1994 as well as subsequent issues reported by @aanand.
Signed-off-by: Charles Chan <charleswhchan@users.noreply.github.com>
splitdrive doesn't handle relative paths, so if volume_path contains
a relative path, we handle that differently and manually set drive to ''.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Removed 'Compose command line completion' and 'Compose environment variables' from the list. command line completion is linked to from install docs, and environment variables are deprecated.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Remove Set-PSDebug -trace to prevent the 9000+ lines of debug output
from spamming the logs on appveyor.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
The Dockerfile should use the same Gemfile.lock as the app, to make sure the container gets the expected versions of gems installed. Aside from wanting that in principle, without it you can get mysterious gem dependency errors. Here's the scenario:
1. Suppose `Gemfile` includes `gem "some-active-gem", "~> 1.0"
2. When developing the app, you run `bundle install`, which installs the latest version--let's say, 1.0.1-and records it in `Gemfile.lock`
3. Suppose the developers of `some-active-gem` then release v1.0.2
4. Now build the container: docker runs `bundle install`, which installs v1.0.2 and records it in `Gemfile.lock` and then "ADD"s the app worktree, which replaces the `Gemfile.lock` with the one from the worktree that lists v1.0.1.
5. Immediately run your app and it fails with the error `Could not find some-active-gem-1.0.1 in any of the sources` which is a bit befuddling since you just saw it run bundle install so you expect all gem dependencies to be resolved properly.
Signed-off-by: ronen barzel <ronen@barzel.org>
Including examples of more boolean types, eg yes/N as it's not
always immediately clear that they are treated as booleans.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Rather than inefficiently looping through all the containers that a
service has and overriding each volumes_from value, pick the first
one and return that.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
An expanded windows path of c:\shiny\thing:\shiny:rw
needs to be changed to be linux style path, including the drive,
like /c/shiny/thing /shiny
to be mounted successfully by the engine.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When a relative path is expanded and we're on a windows platform,
it expands to include the drive, eg C:\ , which was causing a ConfigError
as we split on ":" in parse_volume_spec and that was giving too many parts.
Use os.path.splitdrive instead of manually calculating the drive.
This should help us deal with windows drives as part of the volume
path better than us doing it manually.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Only expand volume host paths if they begin with a dot.
This is a breaking change. The deprecation warning preparing users for
this change has been removed.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Also adds a fix for an error message on some completions when no
compose file is present:
docker-compose build awk: cannot open docker-compose.yml (No such file or directory)
Signed-off-by: Harald Albers <github@albersweb.de>
Bring in line with Glossary. https://docs.docker.com/reference/glossary/
Alphabetising list makes makes parsing by humans easier.
Signed-off-by: Mike Bailey <mike.bailey@rea-group.com>
Conflicts:
docs/yml.md
It was mocking self.client but relying on the call to
utils.create_host_config which was not mocked. So now that function
has moved to also be on self.client we need to redefine the test
boundary, up to where we would call docker-py, not the result of
docker-py.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This is a unit test and we are mocking the client. The method to get
the create_config_host now lives on the client, so we mock that too.
So we can test to the boundary that the method is called with the
arguments we expect.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
create_host_config from docker.utils will be deprecated so that
the new create_host_config has access to the _version so
we can ensure that network_mode only gets set to 'default' by
default if the version is high enough and won't explode.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
By doing this we were over-riding any of the daemon's defaults.
Instead we can send an empty string which docker-py sends on
and the daemon interprets as, 'json-file' as a default if it
hasn't got any other daemon level config options.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Now docker-py isn't hardcoding a list of valid log_drivers, we
can expect an APIError in response rather than a ValueError
if we send an invalid log_driver.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
One of the use cases is swarm requires at least : character, so going
from conservative to relaxed.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
We're going to warn people that allowing a boolean in the environment is
being deprecated, so in a future release we can disallow it. This is to
ensure boolean variables are quoted in strings to ensure they don't get
mis-parsed by YML.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When users were putting true/false/yes/no in the environment key,
the YML parser was converting them into True/False, rather than leaving
them as a string.
This change will force people to put them in quotes, thus ensuring
that the value gets passed through as intended.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
oneOf schema ValidationError takes a little more work to parse and
pull out more detail so we can give a better error message back to
the user.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
English language is a tricky old thing and I've pulled out the validator type
parsing so that we can prefix our validator types with the correct article,
'an' or 'a'.
Doing a bit of extra hard work to ensure the error message is clear and
well constructed english.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
For simple definitions where a field can be multiple types, we can
specify the allowed types in an array. It's simpler and clearer.
This is only applicable to *simple* definitions, like number, string, list,
object without any other constraints.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
index.md:
* clarify Python & Flask
* minor edits & reformatting
install.md:
* merge the elevated installation instructions with `sudo -i` discussed by @asveepay and @aanand in PR #1201; fixes#1081 (not sure what happened to the merge, but it's not showing up on the master branch or website)
* minor edits
Signed-off-by: Charles Chan <charleswhchan@users.noreply.github.com>
'~/' in a path currently doesnt work, you get the following error:
[Errno 2] No such file or directory: u'/home/USER/folder/~/some/path/.yml'
Signed-off-by: Nick H <nick.humrich@gmail.com>
The command value can be a list, which would be a Unix command-line
invocation broken up into individual values, thus needing the ability to
have non unique values.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Some were missing build '.' from their dicts, others were the
incorrect type and one I've moved from integration to unit.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This is minimal disruptive change I could make to ensure
the service integration tests worked, now we have some validation
happening.
There is some coupling/entanglement/assumption going on here.
Project when creating a service, using it's class method from_dicts
performs some transformations on links, net & volume_from, which
get passed on to Service when creating. Service itself, then performs
some transformation on those values. This worked fine in the tests before
because those options were merely passed on via make_service_dict.
This is no longer true with our validation in place. You can't pass to
ServiceLoader [(obj, 'string')] for links, the validation expects it to be
a list of strings. Which it would be when passed into Project.from_dicts
method.
I think the tests need some re-factoring but for now, manually deleting
keys out of the kwargs and then putting them back in for Service Creation
allows the tests to continue.
I am not super happy about this approach. Hopefully we can come back and
improve it.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Previously on Buffy... The process_errors was parsing a load of
ValidationErrors that we get back from jsonschema which included
assumptions about the state of the instance we're validating.
Now it's split in two and we're doing field separate to service,
those assumptions don't hold and we can't logically retrieve the
service_name from the error parsing when we're doing service schema
validation, have to explicitly pass this in.
process_errors is high on my list for some future re-factoring to help
make it a bit clearer, smaller state of doing things.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Now validation is split in two, the integration tests helped
highlight some places where the schema definition was incorrect.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
If make_service_dict is our factory function then we'll give it the
responsibility of validation/construction and resolving.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Now the schema has been split into two, we need to modify the
process_errors function to accomodate.
Previously if an error.path was empty then it meant they were root
errors. Now that service_schema checks after the service has been
resolved, our service name is a key within the dictionary and
so our root error logic check is no longer true.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
We want to give feedback to the user as soon as possible about the
validity of the config supplied for the services.
When extending a service, we can validate that the fields are
correct against our schema but we must wait until the *end* of
the extends cycle once all of the extended dicts have been merged
into the service dict, to perform the final validation check on the
config to ensure it is a complete valid service.
Doing this before that had happened resulted in false reports of
invalid config, as common config when split out, by itself, is not
a valid service but *is* valid config to be included.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This refactoring is now really coming together. Construction is
happening in the __init__, which is a constructor and helps
clean up the design and clarity of intent of the code. We can now
see (nearly) everything that is being constructed when a ServiceLoader
is created. It needs all of these data constructs to perform the
domain logic and actions. Which are now clearer to see and moving
more towards the principle of functions doing (mostly)one thing and
function names being more descriptive.
resolve_extends is now concerned with the resolving of extends, rather
than the construction, validation, pre processing and *then* resolving
of extends.
Happy days :)
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Separating out the steps we need to resolve extends, so that it
will be clear to insert pre-processing of interpolation and
validation.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
resolve_environment is specific to ServiceLoader, the function
does not need to be on the global scope, it is a part of the
ServiceLoader object. The environment needs to be resolved
before we can make any service dicts, it belongs in the
constructor.
This is cleaning up the design a little and being clearer
about intent and scope of functions.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Moving service name and dict out of the function make_service_dict
and into __init__. We always call make_service_dict with those so
let's put them in the initialiser. Slightly cleaner design intent.
The whole purpose of the ServiceLoader is to take a
service name&service dictionary then validate, process and return
service dictionaries ready to be created.
This is also another step towards cleaning the code up so we can
interpolate and validate an extended dictionary.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
While it can be set to ultimately a value of None, when a
config file is read in from stdin, it is not optional.
We kinda make use of it's ability to be set to None in our
tests but functionally and design wise, it is required.
If filename is not set, extends does not work.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Check for this in the init so we can remove the duplication of
raising in further functions.
A ServiceLoader isn't valid without one.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
- Consistent order and contents (where possible)
- Prepend .gitignore paths with slashes where appropriate
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
In most of this file and in Dockers's bash completion the sort order of
options is to sort alphabetically by long option name.
The short options are put right behind their long couterpart.
This commit improves consistency.
Signed-off-by: Harald Albers <github@albersweb.de>
Wrong placement of `compopt -o` introduces an unexpected behavior that did
not matter as long as --help was the only option (you would probably not
continue to type after --help): completion of options would not automatically
append a whitespace character as expected.
For the outstanding addition of the --timeout option, which has an
argument, this would mean that the user would have to type an extra
whitespace after completion of --timeout before the argument could be
added.
Signed-off-by: Harald Albers <github@albersweb.de>
In particular it includes:
- some extension of CONTRIBUTING.md
- one fix for Python 2.6 in tests/integration/cli_test.py
- one fix for Python 3.3 in tests/integration/service_test.py
- removal of unused imports
Make stream_output Python 3-compatible
Signed-off-by: Frank Sachsenheim <funkyfuture@riseup.net>
If we get back an error that wasn't an APIError, it was causing the
thread to hang. This catch all, while I appreciate feels risky to
have a catch all, is better than not catching and silently failing,
with a never ending thread.
If something worse than an APIError has gone wrong, we want to stop
the incredible journey of what we're doing.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
In order to validate a service name that has been specified as an
integer we need to run that as a pre-process validation step
*before* we pass the config to be validated against the schema.
It is not possible to validate it *in* the schema, it causes a
type error. Even though a number is a valid service name, it
must be a cast as a string within the yaml to avoid type error.
Taken this opportunity to move the code design in a direction
towards:
1. pre-process
2. validate
3. construct
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
We use $ref in the schema to allow us to specify multiple type, eg
command, it can be a string or a list of strings.
It required some extra parsing to retrieve a helpful type to display
in our error message rather than 'string or string'. Which while
correct, is not helpful. We value helpful.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When a schema type is set as unique, we should display the validation
error to indicate that non-unique values have been provided for a key.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
The validation message was confusing by displaying only 1 level of
property of the service, even if the error was another level down.
Eg. if the 'files' property of 'extends' was the incorrect format,
it was displaying 'an invalid value for 'extends'', rather than
correctly retrieving 'files'.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Unfortunately the way that jsonschema is calling %r on its property
and then encoding the complete message means I've had to do this
manual way of removing the literal string prefix, u'.
eg:
key = 'extends'
message = "Invalid value for %r" % key
error.message = message.encode("utf-8")"
results in:
"Invalid value for u'extends'"
Performing a replace to strip out the extra "u'", does not change the
encoding of the string, it is at this point the character u followed
by a '.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
While it was intended as a positive to be stricter in validation
it would in fact break backwards compatibility, which we do not
want to be doing.
Consider re-visiting this later and include a deprecation warning if
we want to be stricter.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
If a container is in the process of being removed, or removal has
failed, it can sometimes appear in the output of GET /containers/json
but not have a 'Name' key. In that case, rather than crashing, we can
ignore it.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Rather than implement the logic a second time, use docker-py
split_port function to test if the ports is valid.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Move validation out into its own file without causing circular
import errors.
Fix some of the tests to import from the right place.
Also fix tests that were not using valid test data, as the validation
schema is now firing telling you that you couldn't "just" have this
dict without a build/image config key.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
jsonschema provides a rich error tree of info, by parsing each error
we can pull out relevant info and re-write the error messages.
This covers current error handling behaviour.
This includes new error handling behaviour for types and formatting of
the ports field.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
We validate the config against our schema before a service is created
so checking whether a service name is valid at time of instantiation of
the Service class is not needed.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Define a schema that we can pass to jsonschema to validate against the
config a user has supplied. This will help catch a wide variety of common
errors that occur.
If the config does not pass schema validation then it raises an exception
and prints out human readable reasons.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
- Better method names.
- Environment variable syntax in volume paths, even when a driver is
specified, now *will* be processed (the test wasn't testing it
properly). However, `~` will still *not* expand to the user's home
directory.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
* Add support for volume_driver parameter in compose yml
* Don't expand volume host paths if a volume_driver is specified
(i.e., disable compose feature "relative to absolute path
transformation" when volume drivers are in use, since volume drivers
can use name where host path is normally specified; this is a
heuristic)
Signed-off-by: Luke Marsden <luke@clusterhq.com>
When an image declares a volume such as `/var/lib/mysql`, and a Compose
file has a line like `./data:/var/lib/mysql/` (note the trailing slash),
Compose creates duplicate volume binds when *recreating* the container.
(The first container is created without a hitch, but contains multiple
entries in its "Volumes" config.)
Fixed by normalizing all paths in volumes config.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Hard-coding the API version to '1.18' with the docker-py constructor will
cause the docker-py logic at
https://github.com/docker/docker-py/blob/master/docker/client.py#L143-L146
to always fail, which will cause authentication issues if you're using a
remote daemon using API version 1.19 - regardless of the API version of
the registry.
Allow the user to set the API version via an environment variable. If
the variable is not present, it will still default to '1.18' like it
does today.
Signed-off-by: Reilly Herrewig-Pope <reilly.herrewigpope@mandiant.com>
It was harder to see when there are errors if they came straight after
the other output. Putting a newline in there gives it a bit of visual
room.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
The zsh completion was recently added but missed from the sdist. This
includes all completions that might be added at any point.
Signed-off-by: Alex Brandt <alunduil@alunduil.com>
It doesn't do much other than cause the remainder of the test suite to
generate lots of junk output.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Refactored parallel execute and execute create into a single function
parallel_execute that can now handle both cases. This helps untangle it
from being so tightly coupled to the container.
Updated all the relevant operations to use the refactored function.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
The concurrent.futures backport doesn't play well with
KeyboardInterrupt, so I'm using Thread and Queue instead.
Since thread pooling would likely be a pain to implement, I've just
removed `COMPOSE_MAX_WORKERS` for now. We'll implement it later if we
decide we need it.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Sometimes, some messages were being executed at the same time, meaning
that the status wasn't being overwritten, it was displaying on a
separate line for both doing and done messages.
Rather than trying to have both sets of statuses being written out
concurrently, we write out all of the doing messages first. Then
the done messages are written out/updated, as they are completed.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
- Split out commands into individual pages for maintainability
- Add full usage in commands pages for usability
- Updated description of the run command
- adding in Aanand's comment
Signed-off-by: Mary Anthony <mary@docker.com>
There's significant speed improvement by having more workers. This
value still shouldn't cause anyone's machines to melt/explode.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This approach takes the style of replacing the output message, in
place, when the command has finished executing. Bringing it a bit
more inline with what `docker pull` does.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Commands able to use this parallelisation are `stop`, `kill` and `rm`.
We're using a backported function from python 3, to allow us to make
the most of a pool of threads without having to write the low level
code for managing this ourselves.
A default value for number of threads is a low enough number so it
shouldn't cause performance problems but if someone knows the
capability of their system and wants to increase it, they can via
an environment variable DEFAULT_MAX_WORKERS
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Rather than creating a docker client within each test, create one
at setup and make it accessible to the whole class.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Change in behaviour, `file` key is optional and if not set the
default is to look within the same file as `extends` is defined.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When specifying a log_driver you want to specify some options for
the logger as per the docker run --log-opt option. The logger
options are key value pairs.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
By allowing the memswap_limit option to be defined we also need to
check that mem_limit is set, you can't have swap without a limit.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Also warn the user about the one-off containers in the standard error
message about legacy containers.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
This top level function is a test helper, so I've moved it into the
config_test file and updated accordingly.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When building test data using make_service_dict, we need to include
working_dir as it is core to some of the functionality of
ServiceLoader.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
If you have an alternate YAML file with different services defined,
containers for those services will be shown in `docker-compose ps` even
if you don't pass that file in.
Furthermore, `docker-compose rm` will claim that it's going to remove
them, but actually won't.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
When specifying the 'file' key to a value of it's own name, test
that this works and does not cause a false positive circular reference
error.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
In our circular reference check the stack was previously off by one,
by not including the current service name that was calling another.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
A circular reference bug occurs when there is a difference in the paths
of the file specified in the extends. So one time it is relative, second
time is absolute thus allowing a further circular reference to occur.
By using absolute paths we can be sure that the service filename check
is correct.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This refactoring allows us to raise an error when there is no
'file' key specified in the .yml and no self.filename set. This
error was specific to the tests, as the tests are
the only place that constructs service dicts without sometimes
setting a filename.
Moving the function within the class as well as it is code that
is exclusively for the use of validating properties for the
ServiceLoader class.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Split them out into individual validation tests so it is clearer
to see what is going on and to enable adding further validation
tests.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
If we're using self.filename, then it's already a full path and we
don't need to splice off the filename.yml just so we can .join it
back together again.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
If the 'file' key is not set in the extends_options dict then we
look for the 'service' from within the same file.
Fixes this issue: https://github.com/docker/compose/issues/1237
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
The process function contained purely validation checks, so re-named
the function to aid intent clarity.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
This commit adds environment variable parsing to the container side
of the volume mapping in configs. The common use case for this is
mounting SSH agent sockets in a container, using code like:
volumes:
- $SSH_AUTH_SOCK:$SSH_AUTH_SOCK
environment:
- SSH_AUTH_SOCK
Signed-off-by: Jeff Kramer <jeff.kramer@voxmedia.com>
As VALID_CHARS is shared with project names, these chars are also
now allowed within project names.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
These tests were indeed raising a config error, but not for the reason
intended/tested for. I've added in the image name so the config error
raise is correctly testing the Service name.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
boot2docker is the more common case so let's have that one first.
Also be more explicit and clear that the localhost:5000 is applicable
for people running on linux.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
To aid clarity in how to contribute, I've moved the release process
out into it's own RELEASE_PROCESS.md file. The release process info
is only relevant for maintainers who are building new releases of
compose.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
Added a link to boot2docker for those unfamiliar with it.
http://0.0.0.0:5000 didn't resolve for me, however localhost:5000
did, so included reference to that.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When reading through the code for the first time and seeing redis,
those unfamiliar with docker first thought might be that they need
to install redis. Adding this sentence helps make it clearer that
this is not needed. Docker will take care of this in services.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
When running `docker-compose up`, an extra line of output, from flask,
is outputted. I've included it so anyone new to docker-compose who
sees this output will know that it's expected and not worry that
something might have gone wrong.
Signed-off-by: Mazz Mosley <mazz@houseofmnowster.com>
- Link to libnetwork
- Building now works, but is complicated by scaling
- Document how to set constraints/affinity
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
Updating with changes
Updating for Hugo
Adding a README'
moving index.md compose-overview.md in links
changing overview
Updating image to pull
Signed-off-by: Mary Anthony <mary@docker.com>
This will allow tests to be run on a Swarm. This is being fixed in
Swarm 0.4: https://github.com/docker/swarm/issues/743
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
The boot2docker documentation has since changed the recommended way to
use shellinit - see boot2docker/boot2docker#786.
Signed-off-by: Ed Morley <emorley@mozilla.com>
Add a test to make sure NoSuchService is raised if
a bogus service name is given to 'docker-compose logs'.
Signed-off-by: Dan O'Reilly <oreilldf@gmail.com>
The dockerproject.com domain is moving to dockerproject.org
this changes the buildstatus link to point to the new
domain.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
An image is built anyway, so this is unnecessary. This makes it
possible to run the tests on a remote Docker daemon.
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
I think the original intention of the original test was the check that
different signals work, I think. This does this -- it sends a signal
that doesn't cause the container to stop.
Closes#759. Replaces #1467.
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
This adds a command 'version' to show software versions information
like Docker does. In addition it includes:
- version of the docker-py-package
- Python-implementation and -version
Signed-off-by: Frank Sachsenheim <funkyfuture@riseup.net>
This fixes a bug where migration would fail with an error if a
downstream container was migrated before its upstream dependencies, due
to `check_for_legacy_containers()` being implicitly called when we fetch
`links`, `volumes_from` or `net` dependencies.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
When an upstream dependency (e.g. a db) has a container but a downstream
service (e.g. a web app) doesn't, a web container is not created on
`docker-compose up`.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
"Define and run multi-container applications with Docker"
Not just development environments, and "complex" is not clear and
not really true.
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
- One-off containers were included in the warning log messages, which can
make for unreadable output when there are lots (as there often are).
- Compose was attempting to recreate one-off containers as normal
containers when migrating.
Fixed by implementing the exact naming logic from before we used labels.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
- Rename `migration` module to `legacy` to make its legacy-ness explicit
- Move `check_for_legacy_containers` into `legacy` module
- Fix migration test so it can be run in isolation
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
If we're just streaming logs from `docker-compose up`, we don't need
to set AttachStdin/out/err, and doing so results in containers with
different configuration depending on whether `up` or `run` were invoked
with `-d` or not.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
We shouldn't start the container before handing it off to dockerpty -
dockerpty will start it after attaching, which is the correct order.
Otherwise the container might exit before we attach to it, which can
lead to weird bugs.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
The help text of the build subcommand suggested to use 'compose build' (instead of 'docker-compose build') to rebuild images.
Signed-off-by: Simon Herter <sim.herter@gmail.com>
This ensures that the connection is not recycled, which can cause the
Docker daemon to complain if we've already performed another streaming
call such as doing a build.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
linting...
six.string_types
list-of-strings in examples
disallow extra_hosts support for list-of-dicts
A more thorough sets of tests for extra_hosts
Provide better examples
As per @aanand's [comment](https://github.com/docker/compose/pull/1158/files#r28326312)
I think it'd be better to check `if not isinstance(extra_hosts_line,
six.string_types)` and raise an error saying `extra_hosts_config must be
either a list of strings or a string->string mapping`. We shouldn't need
to do anything special with the list-of-dicts case.
order result to work with assert
use set() instead of sort()
Signed-off-by: CJ <lim@chernjie.com>
Does not change directory to the parent with the compose-file found.
Works like passing '--file' or setting 'COMPOSE_FILE' with absolute path.
Resolves issue #946.
Signed-off-by: Aleksandr Vinokurov <aleksandr.vin@gmail.com>
Docker 1.5.0+ introduces a --pid=host feature which allows sharing of PID
namespaces between baremetal and containers. This is useful for atomic
upgrades, atomic rollbacks, and monitoring.
For more details of a real-life use case, check out:
http://sdake.io/2015/01/28/an-atomic-upgrade-process-for-openstack-compute-nodes/
Signed-off-by: Steven Dake <stdake@cisco.com>
* This fix introduces one side-effect: the build parameter is now
validated early, when the service dicionary is first constructed.
That leads to less scary stack traces when the path is not valid.
* The tests for the changes introduced here alter the fixtures
of those (otherwise unrelated) tests that make use of the 'build:'
parameter)
Signed-off-by: Moysés Borges Furtado <moyses.furtado@wplex.com.br>
- Co-scheduling will now work, so we can remove the stuff about
`volumes_from` and `net` and manual affinity filters.
- Added a section about building.
Signed-off-by: Aanand Prasad <aanand.prasad@gmail.com>
- some reformatting to make it better readable in smaller terminals
- adds a note that suggests validating DCO before pushing
Signed-off-by: funkyfuture <funkyfuture@riseup.net>
Fig has been renamed to Docker Compose, or just Compose for short. This has several implications for you:
- The command you type is now `docker-compose`, not `fig`.
- You should rename your fig.yml to docker-compose.yml.
- If you’re installing via PyPi, the package is now `docker-compose`, so install it with `pip install docker-compose`.
Besides that, there’s a lot of new stuff in this release:
- We’ve made a few small changes to ensure that Compose will work with Swarm, Docker’s new clustering tool (https://github.com/docker/swarm). Eventually you'll be able to point Compose at a Swarm cluster instead of a standalone Docker host and it’ll run your containers on the cluster with no extra work from you. As Swarm is still developing, integration is rough and lots of Compose features don't work yet.
- `docker-compose run` now has a `--service-ports` flag for exposing ports on the given service. This is useful for e.g. running your webapp with an interactive debugger.
- You can now link to containers outside your app with the `external_links` option in docker-compose.yml.
- You can now prevent `docker-compose up` from automatically building images with the `--no-build` option. This will make fewer API calls and run faster.
- If you don’t specify a tag when using the `image` key, Compose will default to the `latest` tag, rather than pulling all tags.
- `docker-compose kill` now supports the `-s` flag, allowing you to specify the exact signal you want to send to a service’s containers.
- docker-compose.yml now has an `env_file` key, analogous to `docker run --env-file`, letting you specify multiple environment variables in a separate file. This is great if you have a lot of them, or if you want to keep sensitive information out of version control.
- docker-compose.yml now supports the `dns_search`, `cap_add`, `cap_drop`, `cpu_shares` and `restart` options, analogous to `docker run`’s `--dns-search`, `--cap-add`, `--cap-drop`, `--cpu-shares` and `--restart` options.
- Compose now ships with Bash tab completion - see the installation and usage docs at https://github.com/docker/compose/blob/1.1.0/docs/completion.md
- A number of bugs have been fixed - see the milestone for details: https://github.com/docker/compose/issues?q=milestone%3A1.1.0+
- Added an `--allow-insecure-ssl` option to allow `fig up`, `fig run` and `fig pull` to pull from insecure registries.
- Fixed `fig run` not showing output in Jenkins.
- Fixed a bug where Fig couldn't build Dockerfiles with ADD statements pointing at URLs.
1.0.0 (2014-10-16)
------------------
The highlights:
- [Fig has joined Docker.](https://www.orchardup.com/blog/orchard-is-joining-docker) Fig will continue to be maintained, but we'll also be incorporating the best bits of Fig into Docker itself.
This means the GitHub repository has moved to [https://github.com/docker/fig](https://github.com/docker/fig) and our IRC channel is now #docker-fig on Freenode.
- Fig can be used with the [official Docker OS X installer](https://docs.docker.com/installation/mac/). Boot2Docker will mount the home directory from your host machine so volumes work as expected.
- Fig supports Docker 1.3.
- It is now possible to connect to the Docker daemon using TLS by using the `DOCKER_CERT_PATH` and `DOCKER_TLS_VERIFY` environment variables.
- There is a new `fig port` command which outputs the host port binding of a service, in a similar way to `docker port`.
- There is a new `fig pull` command which pulls the latest images for a service.
- There is a new `fig restart` command which restarts a service's containers.
- Fig creates multiple containers in service by appending a number to the service name (e.g. `db_1`, `db_2`, etc). As a convenience, Fig will now give the first container an alias of the service name (e.g. `db`).
This link alias is also a valid hostname and added to `/etc/hosts` so you can connect to linked services using their hostname. For example, instead of resolving the environment variables `DB_PORT_5432_TCP_ADDR` and `DB_PORT_5432_TCP_PORT`, you could just use the hostname `db` and port `5432` directly.
- Volume definitions now support `ro` mode, expanding `~` and expanding environment variables.
- `.dockerignore` is supported when building.
- The project name can be set with the `FIG_PROJECT_NAME` environment variable.
- The `--env` and `--entrypoint` options have been added to `fig run`.
- The Fig binary for Linux is now linked against an older version of glibc so it works on CentOS 6 and Debian Wheezy.
Other things:
- `fig ps` now works on Jenkins and makes fewer API calls to the Docker daemon.
- `--verbose` displays more useful debugging output.
- When starting a service where `volumes_from` points to a service without any containers running, that service will now be started.
- Lots of docs improvements. Notably, environment variables are documented and official repositories are used throughout.
- Added a `--no-cache` option to `fig build`, which bypasses the cache just like `docker build --no-cache`.
- Fixed the `dns:` fig.yml option, which was causing fig to error out.
- Fixed a bug where fig couldn't start under Python 2.6.
- Fixed a log-streaming bug that occasionally caused fig to exit.
Thanks @dnephin and @marksteve!
0.5.1 (2014-07-11)
------------------
- If a service has a command defined, `fig run [service]` with no further arguments will run it.
- The project name now defaults to the directory containing fig.yml, not the current working directory (if they're different)
- `volumes_from` now works properly with containers as well as services
- Fixed a race condition when recreating containers in `fig up`
Thanks @ryanbrainard and @d11wtq!
0.5.0 (2014-07-11)
------------------
- Fig now starts links when you run `fig run` or `fig up`.
For example, if you have a `web` service which depends on a `db` service, `fig run web ...` will start the `db` service.
- Environment variables can now be resolved from the environment that Fig is running in. Just specify it as a blank variable in your `fig.yml` and, if set, it'll be resolved:
```
environment:
RACK_ENV: development
SESSION_SECRET:
```
- `volumes_from` is now supported in `fig.yml`. All of the volumes from the specified services and containers will be mounted:
```
volumes_from:
- service_name
- container_name
```
- A host address can now be specified in `ports`:
```
ports:
- "0.0.0.0:8000:8000"
- "127.0.0.1:8001:8001"
```
- The `net` and `workdir` options are now supported in `fig.yml`.
- The `hostname` option now works in the same way as the Docker CLI, splitting out into a `domainname` option.
- TTY behaviour is far more robust, and resizes are supported correctly.
- Load YAML files safely.
Thanks to @d11wtq, @ryanbrainard, @rail44, @j0hnsmith, @binarin, @Elemecca, @mozz100 and @marksteve for their help with this release!
0.4.2 (2014-06-18)
------------------
- Fix various encoding errors when using `fig run`, `fig up` and `fig build`.
0.4.1 (2014-05-08)
------------------
- Add support for Docker 0.11.0. (Thanks @marksteve!)
- Make project name configurable. (Thanks @jefmathiot!)
- Return correct exit code from `fig run`.
0.4.0 (2014-04-29)
------------------
- Support Docker 0.9 and 0.10
- Display progress bars correctly when pulling images (no more ski slopes)
- `fig up` now stops all services when any container exits
- Added support for the `privileged` config option in fig.yml (thanks @kvz!)
- Shortened and aligned log prefixes in `fig up` output
- Only containers started with `fig run` link back to their own service
- Handle UTF-8 correctly when streaming `fig build/run/up` output (thanks @mauvm and @shanejonas!)
- Error message improvements
0.3.2 (2014-03-05)
------------------
- Added an `--rm` option to `fig run`. (Thanks @marksteve!)
- Added an `expose` option to `fig.yml`.
0.3.1 (2014-03-04)
------------------
- Added contribution instructions. (Thanks @kvz!)
- Fixed `fig rm` throwing an error.
- Fixed a bug in `fig ps` on Docker 0.8.1 when there is a container with no command.
0.3.0 (2014-03-03)
------------------
- We now ship binaries for OS X and Linux. No more having to install with Pip!
- Add `-f` flag to specify alternate `fig.yml` files
- Add support for custom link names
- Fix a bug where recreating would sometimes hang
- Update docker-py to support Docker 0.8.0.
- Various documentation improvements
- Various error message improvements
Thanks @marksteve, @Gazler and @teozkr!
0.2.2 (2014-02-17)
------------------
- Resolve dependencies using Cormen/Tarjan topological sort
- Fix `fig up` not printing log output
- Stop containers in reverse order to starting
- Fix scale command not binding ports
Thanks to @barnybug and @dustinlacewell for their work on this release.
0.2.1 (2014-02-04)
------------------
- General improvements to error reporting (#77, #79)
0.2.0 (2014-01-31)
------------------
- Link services to themselves so run commands can access the running service. (#67)
- Much better documentation.
- Make service dependency resolution more reliable. (#48)
- Load Fig configurations with a `.yaml` extension. (#58)
Big thanks to @cameronmaske, @mrchrisadams and @damianmoore for their help with this release.
0.1.4 (2014-01-27)
------------------
- Add a link alias without the project name. This makes the environment variables a little shorter: `REDIS_1_PORT_6379_TCP_ADDR`. (#54)
0.1.3 (2014-01-23)
------------------
- Fix ports sometimes being configured incorrectly. (#46)
- Fix log output sometimes not displaying. (#47)
0.1.2 (2014-01-22)
------------------
- Add `-T` option to `fig run` to disable pseudo-TTY. (#34)
- Fix `fig up` requiring the ubuntu image to be pulled to recreate containers. (#33) Thanks @cameronmaske!
- Improve reliability, fix arrow keys and fix a race condition in `fig run`. (#34, #39, #40)
0.1.1 (2014-01-17)
------------------
- Fix bug where ports were not exposed correctly (#29). Thanks @dustinlacewell!
0.1.0 (2014-01-16)
------------------
- Containers are recreated on each `fig up`, ensuring config is up-to-date with `fig.yml` (#2)
- Add `fig scale` command (#9)
- Use `DOCKER_HOST` environment variable to find Docker daemon, for consistency with the official Docker client (was previously `DOCKER_URL`) (#19)
- Truncate long commands in `fig ps` (#18)
- Fill out CLI help banners for commands (#15, #16)
- Show a friendlier error when `fig.yml` is missing (#4)
- Fix bug with `fig build` logging (#3)
- Fix bug where builds would time out if a step took a long time without generating output (#6)
- Fix bug where streaming container output over the Unix socket raised an error (#7)
Big thanks to @tomstuart, @EnTeQuAk, @schickling, @aronasorman and @GeoffreyPlitt.
0.0.2 (2014-01-02)
------------------
- Improve documentation
- Try to connect to Docker on `tcp://localdocker:4243` and a UNIX socket in addition to `localhost`.
Compose is a part of the Docker project, and follows the same rules and principles. Take a read of [Docker's contributing guidelines](https://github.com/docker/docker/blob/master/CONTRIBUTING.md) to get an overview.
Compose is a part of the Docker project, and follows the same rules and
principles. Take a read of [Docker's contributing guidelines](https://github.com/docker/docker/blob/master/CONTRIBUTING.md)
to get an overview.
## TL;DR
@ -17,60 +19,56 @@ If you're looking contribute to Compose
but you're new to the project or maybe even to Python, here are the steps
that should get you started.
1. Fork [https://github.com/docker/compose](https://github.com/docker/compose) to your username.
1. Clone your forked repository locally `git clone git@github.com:yourusername/compose.git`.
1. Enter the local directory `cd compose`.
1. Set up a development environment by running `python setup.py develop`. This will install the dependencies and set up a symlink from your `docker-compose` executable to the checkout of the repository. When you now run `docker-compose` from anywhere on your machine, it will run your development version of Compose.
2. Clone your forked repository locally `git clone git@github.com:yourusername/compose.git`.
3. You must [configure a remote](https://help.github.com/articles/configuring-a-remote-for-a-fork/) for your fork so that you can [sync changes you make](https://help.github.com/articles/syncing-a-fork/) with the original repository.
4. Enter the local directory `cd compose`.
5. Set up a development environment by running `python setup.py develop`. This
will install the dependencies and set up a symlink from your `docker-compose`
executable to the checkout of the repository. When you now run
`docker-compose` from anywhere on your machine, it will run your development
version of Compose.
## Install pre-commit hooks
This step is optional, but recommended. Pre-commit hooks will run style checks
and in some cases fix style issues for you, when you commit code.
Install the git pre-commit hooks using [tox](https://tox.readthedocs.io) by
To run the style checks at any time run `tox -e pre-commit`.
## Submitting a pull request
See Docker's [basic contribution workflow](https://docs.docker.com/opensource/workflow/make-a-contribution/#the-basic-contribution-workflow) for a guide on how to submit a pull request for code or documentation.
## Running the test suite
Use the test script to run linting checks and then the full test suite:
Use the test script to run linting checks and then the full test suite against
different Python interpreters:
$ script/test
$ script/test/default
Tests are run against a Docker daemon inside a container, so that we can test against multiple Docker versions. By default they'll run against only the latest Docker version - set the `DOCKER_VERSIONS` environment variable to "all" to run against all supported versions:
Tests are run against a Docker daemon inside a container, so that we can test
against multiple Docker versions. By default they'll run against only the latest
Docker version - set the `DOCKER_VERSIONS` environment variable to "all" to run
against all supported versions:
$ DOCKER_VERSIONS=all script/test
$ DOCKER_VERSIONS=all script/test/default
Arguments to `script/test` are passed through to the `nosetests` executable, so you can specify a test directory, file, module, class or method:
Arguments to `script/test/default` are passed through to the `tox` executable, so
you can specify a test directory, file, module, class or method:
We use a [ZenHub board](https://www.zenhub.io/) to keep track of specific things we are working on and planning to work on. If you're looking for things to work on, stuff in the backlog is a great place to start.
$ script/build-linux
OS X:
$ script/build-osx
Note that this only works on Mountain Lion, not Mavericks, due to a [bug in PyInstaller](http://www.pyinstaller.org/ticket/807).
## Release process
1. Open pull request that:
- Updates the version in `compose/__init__.py`
- Updates the binary URL in `docs/install.md`
- Updates the script URL in `docs/completion.md`
- Adds release notes to `CHANGES.md`
2. Create unpublished GitHub release with release notes
3. Build Linux version on any Docker host with `script/build-linux` and attach to release
4. Build OS X version on Mountain Lion with `script/build-osx` and attach to release as `docker-compose-Darwin-x86_64` and `docker-compose-Linux-x86_64`.
5. Publish GitHub release, creating tag
6. Update website with `script/deploy-docs`
7. Upload PyPi package
$ git checkout $VERSION
$ python setup.py sdist upload
For more information about our project planning, take a look at our [GitHub wiki](https://github.com/docker/compose/wiki).
Compose has commands for managing the whole lifecycle of your application:
@ -51,5 +47,19 @@ Compose has commands for managing the whole lifecycle of your application:
Installation and documentation
------------------------------
- Full documentation is available on [Docker's website](http://docs.docker.com/compose/).
- Hop into #docker-compose on Freenode if you have any questions.
- Full documentation is available on [Docker's website](https://docs.docker.com/compose/).
- If you have any questions, you can talk in real-time with other developers in the #docker-compose IRC channel on Freenode. [Click here to join using IRCCloud.](https://www.irccloud.com/invite?hostname=irc.freenode.net&channel=%23docker-compose)
- Code repository for Compose is on [Github](https://github.com/docker/compose)
- If you find any problems please fill out an [issue](https://github.com/docker/compose/issues/new)
Want to help build Compose? Check out our [contributing documentation](https://github.com/docker/compose/blob/master/CONTRIBUTING.md).
Releasing
---------
Releases are built by maintainers, following an outline of the [release process](https://github.com/docker/compose/blob/master/project/RELEASE-PROCESS.md).
## An even better tool for development environments
Compose is a great tool for development environments, but it could be even better. For example:
- It should be possible to define hostnames for containers which work from the host machine, e.g. “mywebcontainer.local”. This is needed by apps comprising multiple web services which generate links to one another (e.g. a frontend website and a separate admin webapp)
## More than just development environments
Over time we will extend Compose's remit to cover test, staging and production environments. This is not a simple task, and will take many incremental improvements such as:
Compose currently works really well in development, but we want to make the Compose file format better for test, staging, and production environments. To support these use cases, there will need to be improvements to the file format, improvements to the command-line tool, integrations with other tools, and perhaps new tools altogether.
- Compose’s brute-force “delete and recreate everything” approach is great for dev and testing, but it not sufficient for production environments. You should be able to define a "desired" state that Compose will intelligently converge to.
- It should be possible to partially modify the config file for different environments (dev/test/staging/prod), passing in e.g. custom ports or volume mount paths. ([#426](https://github.com/docker/fig/issues/426))
Some specific things we are considering:
- Compose currently will attempt to get your application into the correct state when running `up`, but it has a number of shortcomings:
- It should roll back to a known good state if it fails.
- It should allow a user to check the actions it is about to perform before running them.
- It should be possible to partially modify the config file for different environments (dev/test/staging/prod), passing in e.g. custom ports, volume mount paths, or volume drivers. ([#1377](https://github.com/docker/compose/issues/1377))
- Compose should recommend a technique for zero-downtime deploys.
- It should be possible to continuously attempt to keep an application in the correct state, instead of just performing `up` a single time.
## Integration with Swarm
@ -19,10 +30,3 @@ The current state of integration is documented in [SWARM.md](SWARM.md).
Compose works well for applications that are in a single repository and depend on services that are hosted on Docker Hub. If your application depends on another application within your organisation, Compose doesn't work as well.
There are several ideas about how this could work, such as [including external files](https://github.com/docker/fig/issues/318).
## An even better tool for development environments
Compose is a great tool for development environments, but it could be even better. For example:
- [Compose could watch your code and automatically kick off builds when something changes.](https://github.com/docker/fig/issues/184)
- It should be possible to define hostnames for containers which work from the host machine, e.g. “mywebcontainer.local”. This is needed by apps comprising multiple web services which generate links to one another (e.g. a frontend website and a separate admin webapp)
Eventually, Compose and Swarm aim to have full integration, meaning you can point a Compose app at a Swarm cluster and have it all just work as if you were using a single Docker host.
However, the current extent of integration is minimal: Compose can create containers on a Swarm cluster, but the majority of Compose apps won’t work out of the box unless all containers are scheduled on one host, defeating much of the purpose of using Swarm in the first place.
Still, Compose and Swarm can be useful in a “batch processing” scenario (where a large number of containers need to be spun up and down to do independent computation) or a “shared cluster” scenario (where multiple teams want to deploy apps on a cluster without worrying about where to put them).
A number of things need to happen before full integration is achieved, which are documented below.
Re-deploying containers with `docker-compose up`
------------------------------------------------
Repeated invocations of `docker-compose up` will not work reliably when used against a Swarm cluster because of an under-the-hood design problem; [this will be fixed](https://github.com/docker/fig/pull/972) in the next version of Compose. For now, containers must be completely removed and re-created:
$ docker-compose kill
$ docker-compose rm --force
$ docker-compose up
Links and networking
--------------------
The primary thing stopping multi-container apps from working seamlessly on Swarm is getting them to talk to one another: enabling private communication between containers on different hosts hasn’t been solved in a non-hacky way.
Long-term, networking is [getting overhauled](https://github.com/docker/docker/issues/9983) in such a way that it’ll fit the multi-host model much better. For now, containers on different hosts cannot be linked. In the next version of Compose, linked services will be automatically scheduled on the same host; for now, this must be done manually (see “Co-scheduling containers” below).
`volumes_from` and `net: container`
-----------------------------------
For containers to share volumes or a network namespace, they must be scheduled on the same host - this is, after all, inherent to how both volumes and network namespaces work. In the next version of Compose, this co-scheduling will be automatic whenever `volumes_from` or `net: "container:..."` is specified; for now, containers which share volumes or a network namespace must be co-scheduled manually (see “Co-scheduling containers” below).
Co-scheduling containers
------------------------
For now, containers can be manually scheduled on the same host using Swarm’s [affinity filters](https://github.com/docker/swarm/blob/master/scheduler/filter/README.md#affinity-filter). Here’s a simple example:
```yaml
web:
image: my-web-image
links: ["db"]
environment:
- "affinity:container==myproject_db_*"
db:
image: postgres
```
Here, we express an affinity filter on all web containers, saying that each one must run alongside a container whose name begins with `myproject_db_`.
- `myproject` is the common prefix Compose gives to all containers in your project, which is either generated from the name of the current directory or specified with `-p` or the `DOCKER_COMPOSE_PROJECT_NAME` environment variable.
- `*` is a wildcard, which works just like filename wildcards in a Unix shell.
This file has moved to: https://docs.docker.com/compose/swarm/
raiseConfigurationError('Service "%s" doesn\'t have any configuration options. All top level keys in your docker-compose.yml must map to a dictionary of configuration options.'%service_name)
raiseConfigurationError('Serivce "%s" is trying to use the network of "%s", which is not the name of a service or container.'%(service_dict['name'],net_name))
local opts_help opts_force_recreate opts_no_recreate opts_no_build opts_remove_orphans opts_timeout opts_no_color opts_no_deps
opts_help='(: -)--help[Print usage]'
opts_force_recreate="(--no-recreate)--force-recreate[Recreate containers even if their configuration and image haven't changed. Incompatible with --no-recreate.]"
opts_no_recreate="(--force-recreate)--no-recreate[If containers already exist, don't recreate them. Incompatible with --force-recreate.]"
opts_no_build="(--build)--no-build[Don't build an image, even if it's missing.]"
opts_remove_orphans="--remove-orphans[Remove containers for services not defined in the Compose file]"
opts_timeout=('(-t --timeout)'{-t,--timeout}"[Specify a shutdown timeout in seconds. (default: 10)]:seconds: ")
"--rmi[Remove images. Type must be one of: 'all': Remove all images used by any service. 'local': Remove only images that don't have a custom tag set by the \`image\` field.]:type:(all local)" \
'(-v --volumes)'{-v,--volumes}"[Remove named volumes declared in the \`volumes\` section of the Compose file and anonymous volumes attached to containers.]" \
$opts_remove_orphans && ret=0
;;
(events)
_arguments \
$opts_help \
'--json[Output events as a stream of json objects]' \
'(--abort-on-container-exit)-d[Detached mode: Run containers in the background, print new container names. Incompatible with --abort-on-container-exit.]' \
$opts_no_color \
$opts_no_deps \
$opts_force_recreate \
$opts_no_recreate \
$opts_no_build \
"(--no-build)--build[Build images before starting containers.]" \
"(-d)--abort-on-container-exit[Stops all containers if any container was stopped. Incompatible with -d.]" \
'(-t --timeout)'{-t,--timeout}"[Use this timeout in seconds for container shutdown when attached or when containers are already running. (default: 10)]:seconds: " \
'(-p --project-name)'{-p,--project-name}'[Specify an alternate project name (default: directory name)]:project name:' \
'--verbose[Show more output]' \
'(- :)'{-v,--version}'[Print version and exit]' \
'(-H --host)'{-H,--host}'[Daemon socket to connect to]:host:' \
'--tls[Use TLS; implied by --tlsverify]' \
'--tlscacert=[Trust certs signed only by this CA]:ca path:' \
'--tlscert=[Path to TLS certificate file]:client cert path:' \
'--tlskey=[Path to TLS key file]:tls key path:' \
'--tlsverify[Use TLS and verify the remote]' \
"--skip-hostname-check[Don't check the daemon's hostname against the name specified in the client certificate (for example if your docker host is an IP address)]" \
'(-): :->command' \
'(-)*:: :->option-or-argument' && ret=0
local -a relevant_compose_flags relevant_docker_flags compose_options docker_options
relevant_compose_flags=(
"--file" "-f"
"--host" "-H"
"--project-name" "-p"
"--tls"
"--tlscacert"
"--tlscert"
"--tlskey"
"--tlsverify"
"--skip-hostname-check"
)
relevant_docker_flags=(
"--host" "-H"
"--tls"
"--tlscacert"
"--tlscert"
"--tlskey"
"--tlsverify"
)
for k in "${(@k)opt_args}"; do
if [[ -n "${relevant_docker_flags[(r)$k]}" ]]; then
docker_options+=$k
if [[ -n "$opt_args[$k]" ]]; then
docker_options+=$opt_args[$k]
fi
fi
if [[ -n "${relevant_compose_flags[(r)$k]}" ]]; then
Please submit pull requests for unpublished features on the `vnext-compose` branch (https://github.com/docker/docker.github.io/tree/vnext-compose).
If you submit a PR to this codebase that has a docs impact, create a second docs PR on `docker.github.io`. Use the docs PR template provided (coming soon - watch this space).
PRs for typos, additional information, etc. for already-published features should be labeled as `okay-to-publish` (we are still settling on a naming convention, will provide a label soon). You can submit these PRs either to `vnext-compose` or directly to `master` on `docker.github.io`
As always, the docs remain open-source and we appreciate your feedback and
Most Docker Compose commands are run against one or more services. If
the service is not specified, the command will apply to all services.
For full usage information, run `docker-compose [COMMAND] --help`.
## Commands
### build
Builds or rebuilds services.
Services are built once and then tagged as `project_service`, e.g.,
`composetest_db`. If you change a service's Dockerfile or the contents of its
build directory, run `docker-compose build` to rebuild it.
### help
Displays help and usage instructions for a command.
### kill
Forces running containers to stop by sending a `SIGKILL` signal. Optionally the
signal can be passed, for example:
$ docker-compose kill -s SIGINT
### logs
Displays log output from services.
### port
Prints the public port for a port binding
### ps
Lists containers.
### pull
Pulls service images.
### rm
Removes stopped service containers.
### run
Runs a one-off command on a service.
For example,
$ docker-compose run web python manage.py shell
will start the `web` service and then run `manage.py shell` in python.
Note that by default, linked services will also be started, unless they are
already running.
One-off commands are started in new containers with the same configuration as a
normal container for that service, so volumes, links, etc will all be created as
expected. When using `run`, there are two differences from bringing up a
container normally:
1. the command will be overridden with the one specified. So, if you run
`docker-compose run web bash`, the container's web command (which could default
to, e.g., `python app.py`) will be overridden to `bash`
2. by default no ports will be created in case they collide with already opened
ports.
Links are also created between one-off commands and the other containers which
are part of that service. So, for example, you could run:
$ docker-compose run db psql -h db -U docker
This would open up an interactive PostgreSQL shell for the linked `db` container
(which would get created or started as needed).
If you do not want linked containers to start when running the one-off command,
specify the `--no-deps` flag:
$ docker-compose run --no-deps web python manage.py shell
Similarly, if you do want the service's ports to be created and mapped to the
host, specify the `--service-ports` flag:
$ docker-compose run --service-ports web python manage.py shell
### scale
Sets the number of containers to run for a service.
Numbers are specified as arguments in the form `service=num`. For example:
$ docker-compose scale web=2 worker=3
### start
Starts existing containers for a service.
### stop
Stops running containers without removing them. They can be started again with
`docker-compose start`.
### up
Builds, (re)creates, starts, and attaches to containers for a service.
Linked services will be started, unless they are already running.
By default, `docker-compose up` will aggregate the output of each container and,
when it exits, all containers will be stopped. Running `docker-compose up -d`,
will start the containers in the background and leave them running.
By default, if there are existing containers for a service, `docker-compose up` will stop and recreate them (preserving mounted volumes with [volumes-from]), so that changes in `docker-compose.yml` are picked up. If you do not want containers stopped and recreated, use `docker-compose up --no-recreate`. This will still start any stopped containers, if needed.
Specifies an alternate Compose yaml file (default: `docker-compose.yml`)
### -p, --project-name NAME
Specifies an alternate project name (default: current directory name)
## Environment Variables
Several environment variables are available for you to configure Compose's behaviour.
Variables starting with `DOCKER_` are the same as those used to configure the
Docker command-line client. If you're using boot2docker, `$(boot2docker shellinit)`
will set them to their correct values.
### COMPOSE\_PROJECT\_NAME
Sets the project name, which is prepended to the name of every container started by Compose. Defaults to the `basename` of the current working directory.
### COMPOSE\_FILE
Sets the path to the `docker-compose.yml` to use. Defaults to `docker-compose.yml` in the current working directory.
### DOCKER\_HOST
Sets the URL of the docker daemon. As with the Docker client, defaults to `unix:///var/run/docker.sock`.
### DOCKER\_TLS\_VERIFY
When set to anything other than an empty string, enables TLS communication with
the daemon.
### DOCKER\_CERT\_PATH
Configures the path to the `ca.pem`, `cert.pem`, and `key.pem` files used for TLS verification. Defaults to `~/.docker`.
## Compose documentation
- [Installing Compose](install.md)
- [User guide](index.md)
- [Yaml file reference](yml.md)
- [Compose environment variables](env.md)
- [Compose command line completion](completion.md)
Depending on what you typed on the command line so far, it will complete
- available docker-compose commands
- options that are available for a particular command
- service names that make sense in a given context (e.g. services with running or stopped instances or services based on images vs. services based on Dockerfiles). For `docker-compose scale`, completed service names will automatically have "=" appended.
- arguments for selected options, e.g. `docker-compose kill -s` will complete some signals like SIGHUP and SIGUSR1.
Enjoy working with Compose faster and with less typos!
This Quick-start Guide will demonstrate how to use Compose to set up and run a
simple Django/PostgreSQL app. Before starting, you'll need to have
[Compose installed](install.md).
### Define the project
Start by setting up the three files you'll need to build the app. First, since
your app is going to run inside a Docker container containing all of its
dependencies, you'll need to define exactly what needs to be included in the
container. This is done using a file called `Dockerfile`. To begin with, the
Dockerfile consists of:
FROM python:2.7
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
RUN pip install -r requirements.txt
ADD . /code/
This Dockerfile will define an image that is used to build a container that
includes your application and has Python installed alongside all of your Python
dependencies. For more information on how to write Dockerfiles, see the
[Docker user guide](https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile) and the [Dockerfile reference](http://docs.docker.com/reference/builder/).
Second, you'll define your Python dependencies in a file called
`requirements.txt`:
Django
psycopg2
Finally, this is all tied together with a file called `docker-compose.yml`. It
describes the services that comprise your app (here, a web server and database),
which Docker images they use, how they link together, what volumes will be
mounted inside the containers, and what ports they expose.
db:
image: postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
links:
- db
See the [`docker-compose.yml` reference](yml.html) for more information on how
this file works.
### Build the project
You can now start a Django project with `docker-compose run`:
$ docker-compose run web django-admin.py startproject composeexample .
First, Compose will build an image for the `web` service using the `Dockerfile`.
It will then run `django-admin.py startproject composeexample .` inside a
container built using that image.
This will generate a Django app inside the current directory:
**Note:** Environment variables are no longer the recommended method for connecting to linked services. Instead, you should use the link name (by default, the name of the linked service) as the hostname to connect to. See the [docker-compose.yml documentation](yml.md#links) for details.
Compose uses [Docker links] to expose services' containers to one another. Each linked container injects a set of environment variables, each of which begins with the uppercase name of the container.
To see what environment variables are available to a service, run `docker-compose run SERVICE env`.
This Quickstart guide will show you how to use Compose to set up and run a Rails/PostgreSQL app. Before starting, you'll need to have [Compose installed](install.md).
### Define the project
Start by setting up the three files you'll need to build the app. First, since
your app is going to run inside a Docker container containing all of its
dependencies, you'll need to define exactly what needs to be included in the
container. This is done using a file called `Dockerfile`. To begin with, the
Dockerfile consists of:
FROM ruby:2.2.0
RUN apt-get update -qq && apt-get install -y build-essential libpq-dev
RUN mkdir /myapp
WORKDIR /myapp
ADD Gemfile /myapp/Gemfile
RUN bundle install
ADD . /myapp
That'll put your application code inside an image that will build a container with Ruby, Bundler and all your dependencies inside it. For more information on how to write Dockerfiles, see the [Docker user guide](https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile) and the [Dockerfile reference](http://docs.docker.com/reference/builder/).
Next, create a bootstrap `Gemfile` which just loads Rails. It'll be overwritten in a moment by `rails new`.
source 'https://rubygems.org'
gem 'rails', '4.2.0'
Finally, `docker-compose.yml` is where the magic happens. This file describes the services that comprise your app (a database and a web app), how to get each one's Docker image (the database just runs on a pre-made PostgreSQL image, and the web app is built from the current directory), and the configuration needed to link them together and expose the web app's port.
db:
image: postgres
ports:
- "5432"
web:
build: .
command: bundle exec rails s -p 3000 -b '0.0.0.0'
volumes:
- .:/myapp
ports:
- "3000:3000"
links:
- db
### Build the project
With those three files in place, you can now generate the Rails skeleton app
using `docker-compose run`:
$ docker-compose run web rails new . --force --database=postgresql --skip-bundle
First, Compose will build the image for the `web` service using the
`Dockerfile`. Then it'll run `rails new` inside a new container, using that
image. Once it's done, you should have generated a fresh app:
$ ls
Dockerfile app docker-compose.yml tmp
Gemfile bin lib vendor
Gemfile.lock config log
README.rdoc config.ru public
Rakefile db test
Uncomment the line in your new `Gemfile` which loads `therubyracer`, so you've
got a Javascript runtime:
gem 'therubyracer', platforms: :ruby
Now that you've got a new `Gemfile`, you need to build the image again. (This,
and changes to the Dockerfile itself, should be the only times you'll need to
rebuild.)
$ docker-compose build
### Connect the database
The app is now bootable, but you're not quite there yet. By default, Rails
expects a database to be running on `localhost` - so you need to point it at the
`db` container instead. You also need to change the database and username to
align with the defaults set by the `postgres` image.
Open up your newly-generated `database.yml` file. Replace its contents with the
following:
development: &default
adapter: postgresql
encoding: unicode
database: postgres
pool: 5
username: postgres
password:
host: db
test:
<<: *default
database: myapp_test
You can now boot the app with:
$ docker-compose up
If all's well, you should see some PostgreSQL output, and then—after a few
seconds—the familiar refrain:
myapp_web_1 | [2014-01-17 17:16:29] INFO WEBrick 1.3.1
myapp_web_1 | [2014-01-17 17:16:29] INFO ruby 2.2.0 (2014-12-25) [x86_64-linux-gnu]
myapp_web_1 | [2014-01-17 17:16:29] INFO WEBrick::HTTPServer#start: pid=1 port=3000
Finally, you need to create the database. In another terminal, run:
$ docker-compose run web rake db:create
That's it. Your app should now be running on port 3000 on your Docker daemon (if
you're using Boot2docker, `boot2docker ip` will tell you its address).
## More Compose documentation
- [Installing Compose](install.md)
- [User guide](index.md)
- [Command line reference](cli.md)
- [Yaml file reference](yml.md)
- [Compose environment variables](env.md)
- [Compose command line completion](completion.md)
3. Draft a release from the tag on GitHub (the script will open the window for
you)
The tag will only be present on Github when you run the `push-release`
script in step 7, but you can pre-fill it at that point.
4. Paste in installation instructions and release notes. Here's an example -
change the Compose version and Docker version as appropriate:
If you're a Mac or Windows user, the best way to install Compose and keep it up-to-date is **[Docker for Mac and Windows](https://www.docker.com/products/docker)**.
Note that Compose 1.9.0 requires Docker Engine 1.10.0 or later for version 2 of the Compose File format, and Docker Engine 1.9.1 or later for version 1. Docker for Mac and Windows will automatically install the latest version of Docker Engine for you.
Alternatively, you can use the usual commands to install or upgrade Compose:
See the [install docs](https://docs.docker.com/compose/install/) for more install options and instructions.
Here's what's new:
...release notes go here...
5. Attach the binaries and `script/run/run.sh`
6. Add "Thanks" with a list of contributors. The contributor list can be generated
by running `./script/release/contributors`.
7. If everything looks good, it's time to push the release.
./script/release/push-release
8. Merge the bump PR.
8. Publish the release on GitHub.
9. Check that all the binaries download (following the install instructions) and run.
10. Email maintainers@dockerproject.org and engineering@docker.com about the new release.
## If it’s a stable release (not an RC)
1. Close the release’s milestone.
## If it’s a minor release (1.x.0), rather than a patch release (1.x.y)
1. Open a PR against `master` to:
- update `CHANGELOG.md` to bring it in line with `release`
- bump the version in `compose/__init__.py` to the *next* minor version number with `dev` appended. For example, if you just released `1.4.0`, update it to `1.5.0dev`.
