Upgrades the repo to the latest version of IsaacLab #32

pascal-roth · 2024-10-21T12:03:48Z

Description

This PR updates the code to the newer version of IsaacLab. Moreover, it removes the need for link the repo orbit_envs.

pascal-roth · 2024-10-21T12:50:14Z

@fan-ziqi I added your changes here along with some more updates. Please let me know if everything still works for you, then we can update and merge the repo.

@heyshouzi if you merge this PR already in the local copy, everything should work and the locomotion policy should be found

fan-ziqi · 2024-10-21T13:09:55Z

Thank you for your approval. I will test your modifications now. and if everything works without errors, I will reply under this PR.

pascal-roth · 2024-10-21T15:06:59Z

@fan-ziqi thanks a lot, will address the comments now

fan-ziqi · 2024-10-21T15:14:04Z

I found that the robot uses GroundPlane as the reference plane for height_scanner, but it is still making contact with the uneven green terrain below the ground, causing the robot to fall.

So I wonder if this process is incorrect?

pascal-roth · 2024-10-21T15:14:58Z

I found that the robot uses GroundPlane as the reference plane for height_scanner, but it is still making contact with the uneven green terrain below the ground, causing the robot to fall. So I wonder if this process is incorrect?

no, thats not the same as the behavior in my local environment. I will update the online asset

fan-ziqi · 2024-10-21T15:17:37Z

Ok, I'll wait for your new model for testing.

I found that the robot uses GroundPlane as the reference plane for height_scanner, but it is still making contact with the uneven green terrain below the ground, causing the robot to fall. So I wonder if this process is incorrect?

no, thats not the same as the behavior in my local environment. I will update the online asset

fan-ziqi · 2024-10-21T15:29:10Z

I don’t have any other questions on my side aside from the ones mentioned above. I will contact you via email about training data collection.
Due to the time difference, I will respond to your message or continue following up on the testing tomorrow. Wishing you a wonderful day!

pascal-roth · 2024-10-22T10:22:14Z

@fan-ziqi, thanks a lot again for the detailed feedback; I fixed all the remarks that you had.

For the problems with Carla, please download the asset again (it now has colliders enabled).

Regarding point 1), this extension is not ported as it is not necessary for the demo scripts.

pascal-roth · 2024-10-22T10:29:42Z

@fan-ziqi let me know if there are any new bugs for you, otherwise I merge it by the end of the day. Thanks again for your help

fan-ziqi · 2024-10-22T11:31:26Z

I discovered an issue and resolved it. Here’s the problem description and solution:

When using the convert_mesh.py script to convert .obj to .usd, if user uses the 2n8kARJN3HM scene mentioned in README, the conversion will fail.

Usd.Stage.Open(rootLayer=Sdf.Find('anon:0x23b593e0:tmp.usda'), sessionLayer=Sdf.Find('anon:0x23b3f000:tmp-session.usda'), pathResolverContext=<invalid repr>)
2024-10-22 11:14:00 [5,398ms] [Warning] [omni.usd] Warning: in SdfPath at line 81 of /builds/omniverse/usd-ci/USD/pxr/usd/sdf/path.cpp -- Ill-formed SdfPath </0c334eaabb844eaaad049cbbb2e0a4f2>: syntax error

Traceback (most recent call last):
  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/../IsaacLab/source/standalone/tools/convert_mesh.py", line 177, in <module>
    main()
  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/../IsaacLab/source/standalone/tools/convert_mesh.py", line 147, in main
    mesh_converter = MeshConverter(mesh_converter_cfg)
  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mesh_converter.py", line 58, in __init__
    super().__init__(cfg=cfg)
  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/asset_converter_base.py", line 104, in __init__
    self._convert_asset(cfg)
  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mesh_converter.py", line 85, in _convert_asset
    asyncio.get_event_loop().run_until_complete(
  File "/home/ubuntu/anaconda3/envs/viplanner/lib/python3.10/site-packages/omni/extscore/omni.kit.async_engine/omni/kit/async_engine/async_engine.py", line 160, in run_until_complete
    return self.super_run_until_complete(future)
  File "/home/ubuntu/anaconda3/envs/viplanner/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mesh_converter.py", line 223, in _convert_mesh_to_usd
    base_prim = temp_stage.DefinePrim(prim_path, "Xform")
pxr.Tf.ErrorException: 
	Error in 'pxrInternal_v0_22__pxrReserved__::UsdStage::_IsValidPathForCreatingPrim' at line 3340 in file /builds/omniverse/usd-ci/USD/pxr/usd/usd/stage.cpp : 'Path must be an absolute path: <>'

The reason is that the obj file in this scene starts with a number(0c334eaabb844eaaad049cbbb2e0a4f2.obj), while the USD API prohibits names that start with a number Link. This issue is described in more detail here: Link.

I think it’s necessary to mention this in README, as the error message doesn’t indicate the underlying problem.

Perhaps we should inform the IsaacLab team to modify the _convert_asset() function in convert_mesh.py script to throw a user-friendly warning when the mesh_file_basename does not comply with the USD file format.

pascal-roth · 2024-10-22T11:49:48Z

I discovered an issue and resolved it. Here’s the problem description and solution:

When using the convert_mesh.py script to convert .obj to .usd, if user uses the 2n8kARJN3HM scene mentioned in README, the conversion will fail.

Usd.Stage.Open(rootLayer=Sdf.Find('anon:0x23b593e0:tmp.usda'), sessionLayer=Sdf.Find('anon:0x23b3f000:tmp-session.usda'), pathResolverContext=<invalid repr>)

2024-10-22 11:14:00 [5,398ms] [Warning] [omni.usd] Warning: in SdfPath at line 81 of /builds/omniverse/usd-ci/USD/pxr/usd/sdf/path.cpp -- Ill-formed SdfPath </0c334eaabb844eaaad049cbbb2e0a4f2>: syntax error



Traceback (most recent call last):

  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/../IsaacLab/source/standalone/tools/convert_mesh.py", line 177, in <module>

    main()

  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/../IsaacLab/source/standalone/tools/convert_mesh.py", line 147, in main

    mesh_converter = MeshConverter(mesh_converter_cfg)

  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mesh_converter.py", line 58, in __init__

    super().__init__(cfg=cfg)

  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/asset_converter_base.py", line 104, in __init__

    self._convert_asset(cfg)

  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mesh_converter.py", line 85, in _convert_asset

    asyncio.get_event_loop().run_until_complete(

  File "/home/ubuntu/anaconda3/envs/viplanner/lib/python3.10/site-packages/omni/extscore/omni.kit.async_engine/omni/kit/async_engine/async_engine.py", line 160, in run_until_complete

    return self.super_run_until_complete(future)

  File "/home/ubuntu/anaconda3/envs/viplanner/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete

    return future.result()

  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mesh_converter.py", line 223, in _convert_mesh_to_usd

    base_prim = temp_stage.DefinePrim(prim_path, "Xform")

pxr.Tf.ErrorException: 

	Error in 'pxrInternal_v0_22__pxrReserved__::UsdStage::_IsValidPathForCreatingPrim' at line 3340 in file /builds/omniverse/usd-ci/USD/pxr/usd/usd/stage.cpp : 'Path must be an absolute path: <>'

The reason is that the obj file in this scene starts with a number(0c334eaabb844eaaad049cbbb2e0a4f2.obj), while the USD API prohibits names that start with a number Link. This issue is described in more detail here: Link.

I think it’s necessary to mention this in README, as the error message doesn’t indicate the underlying problem.

Perhaps we should inform the IsaacLab team to modify the _convert_asset() function in convert_mesh.py script to throw a user-friendly warning when the mesh_file_basename does not comply with the USD file format.

Great find! We can make a note in the readme about it and make an issue in IsaacLab

pascal-roth · 2024-10-22T11:50:04Z

Any other issues you encountered?

fan-ziqi · 2024-10-22T11:52:28Z

Yes, I will send it out asap.

fan-ziqi · 2024-10-22T12:18:38Z

The conversion script generates two USD files: *.usd and *_non_metric.usd.

When I test using the *.usd file converted by the script, the robot keeps resetting infinitely (I have already followed the documentation to fix the coordinate system and scaling factors), which seems to be a bug. I haven't been able to find the cause of the issue.

bandicam.2024-10-22.20-06-12-354.mp4

However, when I use the *_non_metric.usd, which is the original file, the robot runs correctly.

bandicam.2024-10-22.20-07-10-873.mp4

In addition, even if I don't fix the coordinate system and scaling factors after running the conversion script and simply save the file, the *_non_metric.usd still works fine. Therefore, I believe the 2nd step of the doc is unnecessary, users can directly use *_non_metric.usd file.

pascal-roth · 2024-10-22T12:27:21Z

In this case, the conversion function has been updated. Personally, I converted the assets a long time ago. I will add another node that the *_non_metric.usd should be used.

Thanks for investigating that.

fan-ziqi · 2024-10-22T12:43:31Z

viplanner/omniverse/extension/omni.viplanner/omni/viplanner/config/carla_cfg.py

Line 119 in fb2cdb2

self.scene.robot.init_state.rot = (0.0, 0.0, 0.258819, -0.9659258)

The quaternion rotation axis for Carla is incorrect; the initial state in the code rotates around its own roll. It should be changed to the following(yaw -90), so the robot can initialize properly without falling.

self.scene.robot.init_state.rot = (0.707, 0.0, 0.0, -0.707)

fan-ziqi · 2024-10-22T12:57:08Z

After testing, on my side, there are no more bugs in the demo section so far. Below are the final recordings of the three scenes, which I hope can provide new users with some correct references.

Matterport

viplanner-matterport.mp4

Carla

viplanner-carla.mp4

Warehouse

viplanner-warehouse.mp4

I think you can merge after fixing the above issues.

Finally, thank you for your outstanding contributions to this project; it's been a pleasure collaborating with you.

pascal-roth · 2024-10-22T13:01:49Z

great, thank you so much. Fixed the rotation now.

fan-ziqi · 2024-10-22T13:06:55Z

I discovered an issue and resolved it. Here’s the problem description and solution:

When using the convert_mesh.py script to convert .obj to .usd, if user uses the 2n8kARJN3HM scene mentioned in README, the conversion will fail.

Usd.Stage.Open(rootLayer=Sdf.Find('anon:0x23b593e0:tmp.usda'), sessionLayer=Sdf.Find('anon:0x23b3f000:tmp-session.usda'), pathResolverContext=<invalid repr>)
2024-10-22 11:14:00 [5,398ms] [Warning] [omni.usd] Warning: in SdfPath at line 81 of /builds/omniverse/usd-ci/USD/pxr/usd/sdf/path.cpp -- Ill-formed SdfPath </0c334eaabb844eaaad049cbbb2e0a4f2>: syntax error

Traceback (most recent call last):
  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/../IsaacLab/source/standalone/tools/convert_mesh.py", line 177, in <module>
    main()
  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/../IsaacLab/source/standalone/tools/convert_mesh.py", line 147, in main
    mesh_converter = MeshConverter(mesh_converter_cfg)
  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mesh_converter.py", line 58, in __init__
    super().__init__(cfg=cfg)
  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/asset_converter_base.py", line 104, in __init__
    self._convert_asset(cfg)
  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mesh_converter.py", line 85, in _convert_asset
    asyncio.get_event_loop().run_until_complete(
  File "/home/ubuntu/anaconda3/envs/viplanner/lib/python3.10/site-packages/omni/extscore/omni.kit.async_engine/omni/kit/async_engine/async_engine.py", line 160, in run_until_complete
    return self.super_run_until_complete(future)
  File "/home/ubuntu/anaconda3/envs/viplanner/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mesh_converter.py", line 223, in _convert_mesh_to_usd
    base_prim = temp_stage.DefinePrim(prim_path, "Xform")
pxr.Tf.ErrorException: 
	Error in 'pxrInternal_v0_22__pxrReserved__::UsdStage::_IsValidPathForCreatingPrim' at line 3340 in file /builds/omniverse/usd-ci/USD/pxr/usd/usd/stage.cpp : 'Path must be an absolute path: <>'

The reason is that the obj file in this scene starts with a number(0c334eaabb844eaaad049cbbb2e0a4f2.obj), while the USD API prohibits names that start with a number Link. This issue is described in more detail here: Link.

I think it’s necessary to mention this in README, as the error message doesn’t indicate the underlying problem.

Perhaps we should inform the IsaacLab team to modify the _convert_asset() function in convert_mesh.py script to throw a user-friendly warning when the mesh_file_basename does not comply with the USD file format.

I've opened a issue here isaac-sim/IsaacLab#1287

fan-ziqi · 2024-10-25T12:04:01Z

I discovered an issue and resolved it. Here’s the problem description and solution:
When using the convert_mesh.py script to convert .obj to .usd, if user uses the 2n8kARJN3HM scene mentioned in README, the conversion will fail.

Usd.Stage.Open(rootLayer=Sdf.Find('anon:0x23b593e0:tmp.usda'), sessionLayer=Sdf.Find('anon:0x23b3f000:tmp-session.usda'), pathResolverContext=<invalid repr>)
2024-10-22 11:14:00 [5,398ms] [Warning] [omni.usd] Warning: in SdfPath at line 81 of /builds/omniverse/usd-ci/USD/pxr/usd/sdf/path.cpp -- Ill-formed SdfPath </0c334eaabb844eaaad049cbbb2e0a4f2>: syntax error

Traceback (most recent call last):
  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/../IsaacLab/source/standalone/tools/convert_mesh.py", line 177, in <module>
    main()
  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/../IsaacLab/source/standalone/tools/convert_mesh.py", line 147, in main
    mesh_converter = MeshConverter(mesh_converter_cfg)
  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mesh_converter.py", line 58, in __init__
    super().__init__(cfg=cfg)
  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/asset_converter_base.py", line 104, in __init__
    self._convert_asset(cfg)
  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mesh_converter.py", line 85, in _convert_asset
    asyncio.get_event_loop().run_until_complete(
  File "/home/ubuntu/anaconda3/envs/viplanner/lib/python3.10/site-packages/omni/extscore/omni.kit.async_engine/omni/kit/async_engine/async_engine.py", line 160, in run_until_complete
    return self.super_run_until_complete(future)
  File "/home/ubuntu/anaconda3/envs/viplanner/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ubuntu/workspaces/viplanner_ws/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sim/converters/mesh_converter.py", line 223, in _convert_mesh_to_usd
    base_prim = temp_stage.DefinePrim(prim_path, "Xform")
pxr.Tf.ErrorException: 
	Error in 'pxrInternal_v0_22__pxrReserved__::UsdStage::_IsValidPathForCreatingPrim' at line 3340 in file /builds/omniverse/usd-ci/USD/pxr/usd/usd/stage.cpp : 'Path must be an absolute path: <>'

The reason is that the obj file in this scene starts with a number(0c334eaabb844eaaad049cbbb2e0a4f2.obj), while the USD API prohibits names that start with a number Link. This issue is described in more detail here: Link.
I think it’s necessary to mention this in README, as the error message doesn’t indicate the underlying problem.
Perhaps we should inform the IsaacLab team to modify the _convert_asset() function in convert_mesh.py script to throw a user-friendly warning when the mesh_file_basename does not comply with the USD file format.

I've opened a issue here isaac-sim/IsaacLab#1287

Fixed here isaac-sim/IsaacLab#1302

If merging is possible, the documents for this project need to be modified

fan-ziqi and others added 19 commits October 16, 2024 20:33

feat: migrate to IsaacLab

40d5ec3

fix path ${USER_PATH_TO_USD}

d2faa01

fix docs

9ae4191

docs: fix isaaclab_envs pip install

601d5f0

fix: simplify the running scripts

9943391

fix: set colorize_semantic_segmentation to False in semantic_camera cfg

43043c2

fix: action_manager name

30b2edf

fix: bug of output.permute(0, 3, 1, 2)

3a85d17

fix: use HeightScan policy instead of Blind

571f095

fix: set weights_only=True to avoid waring

918ca7a

docs: convert_mesh

62ac853

docs: fix viplanner_isaaclab_exts new link

2e78a3e

fix index error

c140bfd

add warehouse asset directly to repo

14f2706

automatically move model to env device

8b31347

add matterport extension

b8075d8

adopt readme

4920e23

add doc images

8344dca

add github actions and templates

42c585f

pascal-roth self-assigned this Oct 21, 2024

pascal-roth added the enhancement New feature or request label Oct 21, 2024

pascal-roth added 4 commits October 21, 2024 14:05

formatter

0d91aa2

fix

809b02f

expand readme

d6b17ef

formatter

d68a046

pascal-roth mentioned this pull request Oct 21, 2024

feat: migrate to IsaacLab #28

Closed

add readme and warning to matterport extension

3bea1e9

pascal-roth mentioned this pull request Oct 21, 2024

Load Scene button is disabled even with valid Mesh/PLY filepaths in OrbitNavImporter extension pascal-roth/orbit_envs#1

Closed

pascal-roth added 2 commits October 22, 2024 12:18

bugfixes

6dcbc98

formatter

cfa037a

pascal-roth mentioned this pull request Oct 22, 2024

[Bug Report] isaac sim 4.2 simulation problem in dev/upgrade branch #33

Closed

pascal-roth linked an issue Oct 22, 2024 that may be closed by this pull request

[Bug Report] isaac sim 4.2 simulation problem in dev/upgrade branch #33

Closed

add warning for conversion

cb1b168

pascal-roth added 2 commits October 22, 2024 14:19

fix

96251e1

format

44017fa

update readme for asset conversion

fb2cdb2

fix rot

5797a04

pascal-roth merged commit 5fc8a3e into main Oct 22, 2024
1 check failed

pascal-roth deleted the dev/upgrade branch October 22, 2024 13:02

fan-ziqi mentioned this pull request Oct 22, 2024

[Bug Report] convert_mesh.py failed when *.obj file name starts with a number isaac-sim/IsaacLab#1287

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrades the repo to the latest version of IsaacLab #32

Upgrades the repo to the latest version of IsaacLab #32

pascal-roth commented Oct 21, 2024

pascal-roth commented Oct 21, 2024

fan-ziqi commented Oct 21, 2024

pascal-roth commented Oct 21, 2024

fan-ziqi commented Oct 21, 2024

pascal-roth commented Oct 21, 2024

fan-ziqi commented Oct 21, 2024

fan-ziqi commented Oct 21, 2024 •

edited

Loading

pascal-roth commented Oct 22, 2024

pascal-roth commented Oct 22, 2024

fan-ziqi commented Oct 22, 2024

pascal-roth commented Oct 22, 2024

pascal-roth commented Oct 22, 2024

fan-ziqi commented Oct 22, 2024

fan-ziqi commented Oct 22, 2024

pascal-roth commented Oct 22, 2024

fan-ziqi commented Oct 22, 2024 •

edited

Loading

fan-ziqi commented Oct 22, 2024

pascal-roth commented Oct 22, 2024

fan-ziqi commented Oct 22, 2024

fan-ziqi commented Oct 25, 2024

Upgrades the repo to the latest version of IsaacLab #32

Upgrades the repo to the latest version of IsaacLab #32

Conversation

pascal-roth commented Oct 21, 2024

Description

pascal-roth commented Oct 21, 2024

fan-ziqi commented Oct 21, 2024

pascal-roth commented Oct 21, 2024

fan-ziqi commented Oct 21, 2024

pascal-roth commented Oct 21, 2024

fan-ziqi commented Oct 21, 2024

fan-ziqi commented Oct 21, 2024 • edited Loading

pascal-roth commented Oct 22, 2024

pascal-roth commented Oct 22, 2024

fan-ziqi commented Oct 22, 2024

pascal-roth commented Oct 22, 2024

pascal-roth commented Oct 22, 2024

fan-ziqi commented Oct 22, 2024

fan-ziqi commented Oct 22, 2024

pascal-roth commented Oct 22, 2024

fan-ziqi commented Oct 22, 2024 • edited Loading

fan-ziqi commented Oct 22, 2024

Matterport

Carla

Warehouse

pascal-roth commented Oct 22, 2024

fan-ziqi commented Oct 22, 2024

fan-ziqi commented Oct 25, 2024

fan-ziqi commented Oct 21, 2024 •

edited

Loading

fan-ziqi commented Oct 22, 2024 •

edited

Loading