Frequently asked questions
conda-store fails to build Conda environment and worker is spontaneously killed (9 SIGKILL)
The following error most likely indicates that you have not allocated
enough memory to conda-store-workers
for solving and building the
given environment. Solve this by increasing the memory allocated to
the container.
Process 'ForkPoolWorker-31' pid:90 exited with 'signal 9 (SIGKILL)'
Task handler raised error: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL) Job: 348.')
Traceback (most recent call last):
File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost
raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) Job: 348.
Why are environment builds stuck in building state?
Recently conda-store added a feature to cleanup builds which are stuck in the BUILDING state and are not currently running on the workers. This feature only works for certain brokers e.g. redis. Database celery brokers are not supported.
This issue occurs when the worker spontaneously dies. This can happen for several reasons:
- worker is killed due to consuming too much memory (conda solver/builds can consume a lot of memory)
- worker was killed for other reasons e.g. forced restart
- bugs in conda-store
Build path length
Conda packages are guaranteed to be relocatable as long as the environment
prefix length is <=
255 characters. In conda-store, the said prefix is specified
in Build.build_path
. When building an environment, you might see an error like
this:
build_path too long: must be <= 255 characters
If so, try configuring the conda-store CondaStore.store_directory
to be as
close to the filesystem root as possible. Additionally, 255 characters is also a
common limit for individual files on many filesystems. When creating
environments, try using shorter namespace
and environment
names since they
affect both the build_path
length and the filename length.
Build key versions
The part of the build path that identifies a particular environment build is the build key. Originally, conda-store used the following format, known as version 1:
c7afdeffbe2bda7d16ca69beecc8bebeb29280a95d4f3ed92849e4047710923b-20231105-035410-510258-12345678-this-is-a-long-environment-name
^ (1) ^ (2) ^ (3) ^ (4)
It consists of:
- a SHA-256 hash of the environment specification
(
CondaSpecification
, which represents a user-provided environment, is converted to a dict and passed todatastructure_hash
, which recursively sorts it and calculates the SHA-256 hash) - a human-readable timestamp (year, month, day,
-
, hour, minute, second,-
, microsecond) - the id of a build
- the environment name.
To help mitigate build path length issues, a shorter build key format was introduced, known as version 2:
c7afdeff-1699156450-12345678-this-is-a-long-environment-name
^ (1) ^ (2) ^ (3) ^ (4)
It consists of:
- a truncated SHA-256 hash of the environment specification
(
CondaSpecification
, which represents a user-provided environment, is converted to a dict and passed todatastructure_hash
, which recursively sorts it and calculates the SHA-256 hash) - a Unix timestamp
- the id of a build
- the environment name.
However, version 2 build paths don't solve the problem completely because they include user-provided data, like the environment name, and that data can be arbitrarily large.
To solve this problem, version 3 was introduced, which will always have the same size. It looks like this:
64a943764b70e8fe181643404894f7ae
Version 3 is experimental and can be changed at any time.
It's a truncated SHA-256 hex digest, which is calculated based on:
- namespace name
- specification hash (also SHA-256)
- build timestamp
- build id.
See BuildKey._version3_experimental_fmt
for details.
When version 3 is used, Build.build_path
will not include the namespace name,
because it's not fixed size, so all builds will be placed right into
CondaStore.store_directory
.
Additionally, CondaStore.environment_directory
will be completely ignored, so
no symlinks connecting an environment name to its corresponding build will be
created, because the environment directory format also includes variable-size
data (the namespace and environment names).
For context, these symlinks are created because that's how conda is usually used: each environment name points to a particular directory on the filesystem, and symlinks connect this directory to the current build.
In the following example, which uses the version 2 format, there are two
environments in the default
namespace: test
and test2
. The former points
to build 3 and the latter points to build 2:
$ ls -l ~/.conda-store/state/default/envs
test -> /home/user/.conda-store/state/default/b3109fbf-1710602415-3-test
test2 -> /home/user/.conda-store/state/default/2aad045f-1710602357-2-test2
The lack of symlinks doesn't prevent server artifacts from being generated, which are available for download via the UI (lockfiles, archives, etc.), because those rely on storage or use the database.
But it does impact conda integration or tools that rely on it, like when
conda-store is used with JupyterLab as part of a Nebari deployment. Without
environment symlinks, there'll be no way to tell conda where to look for
environments, which is done by setting envs_dirs
in .condarc
, so conda env list
will return nothing and no environments will show up in JupyterLab.
The version 2 format is the default because it supports environment symlinks and doesn't usually run into path length limitations. If you do experience problems with the latter and don't need the former, then consider using the version 3 format.
No matter what format you choose, environments that were previously created using other version formats will be accessible in the conda-store web UI.
There is no real reason to use the version 1 format anymore, but any version can be explicitly set via the config, for example:
c.CondaStore.build_key_version = 1
Long paths on Windows
conda-store supports Windows in standalone mode. However, when creating environments with certain packages, you may see errors like:
ERROR:root:[WinError 206] The filename or extension is too long: 'C:\\...'
This error is due to the fact that Windows has a limitation that file paths cannot be more than 260 characters.
See conda-store issue #588 for more details.
Solution 1: Extended-length path prefix (\\?\
)
If you don't have administrator privileges, try using the following config option:
c.CondaStore.win_extended_length_prefix = True
This adds the extended-length path prefix (\\?\
) to conda-store build_path
and environment_path
methods, which should allow for a maximum total path
length of 32,767 characters when building packages.
See this Microsoft support article for more details on the extended-length path prefix.
Solution 2: LongPathsEnabled
If you have administrator privileges, set the registry key
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled (Type: REG_DWORD)
to 1
, which removes this MAX_PATH
limitation.
See this Microsoft support article for more details on how to set this registry key.
Solution 3: store_directory
If it is not possible to set the registry key, for instance, because you do
not have access to administrator privileges, you should configure the
conda-store CondaStore.store_directory
to be as close to the filesystem root
as possible, so that the total length of the paths of package files is
minimized.
Solution 4: build_key_version
Use the short build key version as explained above:
c.CondaStore.build_key_version = 2
What are the resource requirements for conda-store-server
conda-store-server
is a web server and should not require any specific resources.
1 GB of RAM and 1 CPU should be plenty.
conda-store-worker
does the actual builds of the conda environments.
Solving for conda environments can take a lot of memory in some circumstances.
Make sure to allocate at least 4 GB of RAM to the worker along with at least one CPU.