Error when using custom domain subpath

[Betagmr]

I'm hosting cleaml using the following path configuration.

api {
        web_server: http://<domain>/clearml/
	api_server: http://<domain>/api/
	file_server: http://<domain>/fileserver/
	
	credentials {
		"access_key"="JMJGM7JIU44XJ4D8SHVMZ86DEY3ZE9"
		"secret_key"="fwkuNBt4krfz-4Qugd4Afd3PNnuk4v0Z0qR8NxoXteC-ramUJGnSzo6KffHf4ZtAmlQ"
	}
}

After adding this configuration and running clearml-init, I noticed that the file_server path is automatically modified to use port 8081, even though a different path was specified. Here is the output I receive after running clearml-init

Detected credentials key="JMJGM7JIU44XJ4D8SHVMZ86DEY3ZE9" secret="fwku***"
Web app hosted on standard port using http protocol.
Assuming files and api ports are unchanged and use the same (http) protocol

ClearML Hosts configuration:
Web App: http://<domain>/clearml/
API: http://<domain>/api/
File Store: http://<domain>:8081/clearml/

Additionally, if I manually modify the configuration back to http://domain/fileserver/ after clearml-init, certain ClearML CLI commands stop working. Specifically, when I run the clearml-data command to download datasets, I get a connection error:

clearml-data get --id 946c88456e39474992fcefaf36608ae1
clearml-data - Dataset Management & Versioning CLI
Download dataset id 946c88456e39474992fcefaf36608ae1
Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7b9c4d533c50>: Failed to establish a new connection: [Errno 111] Connection refused')': /MyExample/.datasets/MyExampleDataset/LaminadoraDataset.946c88456e39474992fcefaf36608ae1/artifacts/data/dataset.946c88456e39474992fcefaf36608ae1.hywr6569.zip

Additional Details: It appears that ClearML is truncating the file_server URL, which leads to issues in download requests. Instead of making the full request to http://<domain>/fileserver/MyExample/..., ClearML attempts to access /MyExample/... directly, ignoring the configured base path for file_server. This URL truncation causes the requests to fail, as they do not correctly point to the file server's location.

[ainoam]

Thanks for reporting @Betagmr - We'll take a look

[Betagmr]

Hi @ainoam! Any updates on the problem?

[eugen-ajechiloae-clearml]

Hi @Betagmr ! Can you share your clearml sdk version, as well as a concrete example configuration string that you are passing to clearml-init that creates the same problem?

[Betagmr]

I fix both problems!

The first error related to clearml-init occurred because I accidentally used file_server instead of files_server. When I initially created the credentials, this parameter wasn’t included, so I added it manually. Due to the incorrect parameter name, a default one was applied.

The second issue, with the URL truncation, happened when we stored a dataset on an internal domain. Upon migrating to a public domain, the dataset URL artifact still pointed to the internal domain, causing a mismatch that triggered the error. We resolved it by deleting the dataset and re-uploading it.

Now, my question is: is there a way to edit the output shown by “create new credentials” in the web app? Also, can we edit the metadata of storage artifacts?

[ainoam]

@Betagmr see allegroai/clearml-web#67 for controlling credential settings.
For storage migration - Which metadata are you referring to?

[Betagmr]

Ty @ainoam

On uploading a dataset to files_server it saves an artifact data at http://<domain-1>:8081/URL/TO/DATASET

I recently changed my domain to http://<domain-2>:8081/ but the artifact stored didn't update that file path to http://<domain-2>:8081/URL/TO/DATASET and I get errors related to that. Because Dataset.get() uses that url for downloading the data and not my FILES_SERVER_PATH/PATH/TO/DATASET.