Use internal URL's like Azure blob storage or AWS S3 bucket or sharepoint.com URL ?

[prasadaws10]

Hi Team,

I am using a BYOR image. Can I use a CSV hosted in our internal storage option or URL's from Azure storage/AWS S3/sharepoint URL ? Tried loading the URL but I don't see any response on the web page.

Thank you.

[devansh-sharma-tw]

Hi Prasad,

Files hosted privately or behind some sort of authentication mechanism are not supported by BYOR. Only publicly accessible URLs to CSV/JSON files will work.

Thanks!

[prasadaws10]

Hi Prasad,

Files hosted privately or behind some sort of authentication mechanism are not supported by BYOR. Only publicly accessible URLs to CSV/JSON files will work.

Thanks!

@devansh-sharma-tw,

Thankyou for your response.

But I am looking @ #174, which says an example of Azure blob storage the URL can be appended with .CSV or =CSV. I tried both but the file didn't go through.

Logs:

I Removed private info on URL's

"GET /?sheetId=https%3A%2F%2Fstorageaccount.blob.core.windows.net%2Ftworks%2FRadar.csv%3Fsp%3Dracwe%26st%3D2022-09-23T23%3A35%3A16Z%26se%3D2022-09-24T07%3A35%3A16Z%26sip%3D10.2.4.6%26spr%3Dhttps%26sv%3D2021-06-08%26sr%3Db%26sig%3DnvhKYaCG9QD3L6PvinGFPCLq8vizmCVe%252FY9CJIsZSfI%253D%26%3D.csv HTTP/1.1" 304 0 "http://thoughtworks-dev.com/?sheetId=https%3A%2F%2Fdtvbinprdgendlst.blob.core.windows.net%2Ftworks%2FRadar.csv%3Fsp%3Dracwe%26st%3D2022-09-23T23%3A35%3A16Z%26se%3D2022-09-24T07%3A35%3A16Z%26sip%3D10.2.4.6%26spr%3Dhttps%26sv%3D2021-06-08%26sr%3Db%26sig%3DnvhKYaCG9QD3L6PvinGFPCLq8vizmCVe%252FY9CJIsZSfI%253D" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" "-"

[devansh-sharma-tw]

Hi Prasad,

As a hack the approach mentioned there might work in limited cases depending on the storage service used, since adding extra query params like temp=.csv can cause the storage service to return a 404 or 403 (which it does with AWS S3 presigned URLs). That could be a case here as well.

As an alternative to using a private storage, you can also try hosting your csv file in the docker instance itself and use it that way. Instructions to do so are mentioned here.

We will be enhancing the logic, for identifying CSV/JSON files to not depend on just the URL ending, in the future. Once that is in place, presigned URLs to private files should ideally work as well.

Thanks!

[prasadaws10]

Hi Prasad,

As a hack the approach mentioned there might work in limited cases depending on the storage service used, since adding extra query params like temp=.csv can cause the storage service to return a 404 or 403 (which it does with AWS S3 presigned URLs). That could be a case here as well.

As an alternative to using a private storage, you can also try hosting your csv file in the docker instance itself and use it that way. Instructions to do so are mentioned here.

We will be enhancing the logic, for identifying CSV/JSON files to not depend on just the URL ending, in the future. Once that is in place, presigned URLs to private files should ideally work as well.

Thanks!

Hi Devansh,

On a linux server deployed docker with command : docker run -d --rm -p 8080:80 -e SERVER_NAMES="localhost 127.0.0.1" -v /opt/tw-data/:/opt/build-your-own-radar/files wwwthoughtworks/build-your-own-radar

Data file location in linux server: /opt/tw-data

But I am not able to see/load the file or my browser URL doesn't load.

Error on URL in browser: http://localhost:8080 or http://localhost:8080 doesn't load or http://localhost:8080/files/radar.csv or I tried with mydns.com:8080/files/radar.csv is forbidden

Appreciate your response greatly.

Thank You.

[prasadaws10]

Hi Prasad,
As a hack the approach mentioned there might work in limited cases depending on the storage service used, since adding extra query params like temp=.csv can cause the storage service to return a 404 or 403 (which it does with AWS S3 presigned URLs). That could be a case here as well.
As an alternative to using a private storage, you can also try hosting your csv file in the docker instance itself and use it that way. Instructions to do so are mentioned here.
We will be enhancing the logic, for identifying CSV/JSON files to not depend on just the URL ending, in the future. Once that is in place, presigned URLs to private files should ideally work as well.
Thanks!

Hi Devansh,

On a linux server deployed docker with command : docker run -d --rm -p 8080:80 -e SERVER_NAMES="localhost 127.0.0.1" -v /opt/tw-data/:/opt/build-your-own-radar/files wwwthoughtworks/build-your-own-radar

Data file location in linux server: /opt/tw-data

But I am not able to see/load the file or my browser URL doesn't load.

Error on URL in browser: http://localhost:8080 or http://localhost:8080 doesn't load or http://localhost:8080/files/radar.csv or I tried with mydns.com:8080/files/radar.csv is forbidden

Appreciate your response greatly.

Thank You.

Or Another scenario: If I installed it in my cloud vm using:

docker run -d --rm -p 80:80 -p 443:443 -v /certs/tw-certs:/etc/nginx/conf.d -e SERVER_NAMES="mycloudvm.cloudapp.net 130.12.24.16" -v /opt/tw-data/:/opt/build-your-own-radar/files wwwthoughtworks/build-your-own-radar

This is running and I am able to access it from my local machine in a browser with my dns.com. But on this how do I input my data file ?

The default file is being written to the location: opt/tw-data in my vm. But if I update it, how do I input my new file ? or provide that location of the vm's storage from outside in my browser's URL ?

Thank you.