[SOLVED] Working with .obj-files in SOFA

[Jonny]

Hi everyone,

I am trying to load and later export a textured mesh as obj-file in Sofa Python3 and then use the exported mesh for mesh fitting with differentiable rendering in pytorch3d.
For this, I did load the mesh in Blender (previously an stl-file) and added a texture via uv mapping. Then, I exported this mesh as an .obj with corresponding .mtl specifying the texture. Somehow, when using the MeshObjLoader-component in Sofa on its own, the texture does not seem to be loaded. However, it is possible to add the texture via the OglModel-component and a mapping although that mapping is obviously different from the one I have created in Blender.
So my first question would be: is it possible to directly load a textured obj in Sofa and if so, how?

Since I was able to load the texture separately, I figured I could also export the mesh using the OBJExporter-component to create a new obj- and mtl-file based on the Sofa mapping. In principal this component seems to be working – I get the two files, but again without the texture, i.e. the image used for the texture is not referenced in the mtl-file). I can manually add that reference, however, again, the mapping looks different after rendering this obj file with the same camera settings in pytorch3d compared to the Sofa scene.
Second question: Is there a flag or anything comparable that lets the OBJExporter know that I want to export the texture as well? I definitely need an obj-file in pytorch3d, so I probably won’t get around using this component. Thanks in advance!

Cheers,
Jonny

[Hugo]

Hi @jonny

Interesting work and thank you for reporting this issue to us.
The OglModel does have a data field for texture : texturename did you try to use it?

If you have no topological changes in your scene, all you need is to recompute the obj file without the material information. Is this correct? Then a simple obj export should work.

Regarding your second question, my answer is I do not think so.
Let me also poke @froy on this topic to get his eye covering it.

Hope this helps
Best,

Hugo

[Jonny]

Hi @hugo,

Thanks for your reply.

I have made some small progress by realizing that
visual.addObject('OglModel', name="VisualModel", fileMesh="../data/ellipsoid/ellipsoid_dds.obj", texturename=textureFilename)
produces a different visual model than

visual.addObject('MeshObjLoader', name="meshLoader_0", filename="../data/ellipsoid/ellipsoid_dds.obj")
visual.addObject('OglModel', name="VisualModel", src="@meshLoader_0", texturename=textureFilename

.
What makes this even stranger is that when adding both, the mesh and the texture, via the OglModel, the uv mapping seems to be right although this component is deprecated. When loading the mesh first and then referencing it from the OglModel this is not the case. However, even in the first case the texture seems to be rotated/mirrored s.t. the rendered image still looks different compared to the one in Blender. Also I don’t understand why I have to reference the texture separately as this is already done in the obj/mtl file (texture is not loaded if I omit the texturename data field).

Again, to clarify: what I want to achieve right now is that the Sofa scene looks the same as the pytorch3d rendered image. For me, it appears there two ways of getting there:

1. Load my Blender obj in Sofa such that it looks the same in both. Rendering it in pytorch3d is not an issue (Blender rendered image”=”pytorch3d rendered image), in Sofa this is the texture issue described above. If I were to fix this, then I wouldn’t even need the OBJExporter since I could read out the node positions separately and just pass them to pytorch3d after deformation (I only need them to evaluate the results of the mesh fitting which is purely image-based).

2. If it is not possible to get the texture to look the same in Sofa, then I could just export the Sofa obj and use that in pytorch3d (basically omit the Blender obj). However, here the problem is exporting the texture with the obj. Manually adding that in produces a different result.

Hope this clarifies my current issues and thanks again for your help!

Jonny

[Jonny]

Maybe this also helps:
Blender vs Sofa texture issue image

On the left is my Sofa scene, on the right the same obj with the same texture image in Blender. It seems as if the uv map is the same but the image is aligned differently (maybe u and v coordinates defined differently?), simply rotating the image did not solve the issue so far, although maybe I haven’t found the right “transformation” yet (however, the reason for this issue would remain unclear). In Sofa everything is added via the OglModel, no separate MeshObjLoader.

[Hugo]

Hi @jonny

Since OglModel and MeshObjLoader are two different components, I am not estonished to read that there is some differences (even if there should not!).

In this PR #1096, the data field handleSeams was instroduced. It allows the MeshObjLoader to make sure he handles both position and vertices (uv). Have you tried to set it to true ?

Does it solve your problem?
If not, I would add other engineers in the loop more knowledgeable about these components.

Note that loading directly the mesh from OglModel will no longer be possible after v21.06.

Best wishes,

Hugo

[Jonny]

Hi @hugo,

thanks for you further reply!
Adding the proposed data field did lead to the same result as loading everything inside the OglModel, so it allows me to use to newer components.
Now, the seams are handled but the texture image still is aligned wrong (see the image I linked above). Would be great if you could add another engineer who might know more about using image texturing.

Best,
Jonny

[Hugo]

Thanks @jonny for letting me know.
Let me introduce you to @froy! Fred, we have an issue here between OglModel/MeshObjLoader to get a proper texture applied. Does it remind you something?

Best,

Hugo

[Froy]

Hello,

Few hints I would look to find the problem:
– texture coordinates are wrongly loaded, maybe a discrepancy between Blender and SOFA?
– the faces are flipped (and normals are backwards).
You can try to see if this is the 2nd case by simply flipping the normals in the mesh loader (flipNormals="true").
If it is about the texcoords, you can play with the scaleTex parameter in OglModel (for example, scaleTex=”-1 0″ will mirror the texcoords on X axis)
Keep us in touch if it solves the problem (or not)…

Fred

[Jonny]

Hi @froy,

Also thanks to your for your fast response and the suggestions.

flipNormals="true" does not seem to change anything, setting this data field to false doesn’t change anything either though, so I am a little confused.
I will look into the scaleTex data field, and also take another closer look at the apparent misalignment of the image to check whether this is a possible solution. I will let you know as soon as I figure it out or get stuck again 😀

If you happen to have more ideas, feel free to let me know.

Jonny

[Jonny]

Hi again,

so scaleTex="1 -1" did the trick for me, i.e. mirroring at the horizontal axis. I am still not quite sure as to why this is needed, but I’ll take it.

On another note: The file produced by the OBJExporter seems to differ from the obj-file that is loaded initially. I haven’t looked into that exactly as I don’t need it right now – the order of the vertices and normals is definitely different though. Just fyi, might also be confusion on my side.

Best,
Jonny

[Froy]

Hello @jonny,

I tried to create a very simple textured mesh in Blender, and it seemed there was no problem doing Blender -> SOFA. But it is (very) possible that the example is TOO simple 🙃.
If you want to see/try my test: https://filesender.renater.fr/?s=download&token=064d8e75-9d7d-42c4-913f-5f74a93249f4
(Link valid for 1 month, i.e until the 14th of November approx)

As for the OBJExporter, it seems there is a problem where with the number of normals/texcoords wrt the vertices (probably due to the fact that vertices are duplicated when there is texcoords). It is already on our extended schedule to rewrite this component, as it is not really reliable.

Best regards,

Frederick

[Author]

Posts