Day: October 14, 2022

2022 Puppetry project week #41 summary

Posted on by Inara Pey
Puppetry demonstration via Linden Lab – see below.  Demos video with the LL comment “We have some basic things working with a webcam and Second Life but there’s more to do before it’s as animated as we want.”

The following notes have been taken from chat logs and audio recording of the Thursday, October 13th Puppetry Project meetings held at the Castelet Puppetry Theatre on Aditi. These meetings are generally held on alternate weeks to the Content Creation User Group (CCUG), on same day / time (Thursdays at 13:00 SLT).

Notes in these summaries are not intended to be a full transcript of every meeting, but to highlight project progress / major topics of discussion.

Project Summary

  • Previously referred to as “avatar expressiveness”, Puppetry is intended to provide a means by which avatars can mimic physical world actions by their owners (e.g. head, hand, arm movements) through tools such as a webcam and using technologies like inverse kinematics (IK) and the  LLSD Event API Plug-in (LEAP) system.
    • Note that facial expressions and finger movements are not currently enabled.
    • Most movement is in the 2D plain (e.g., hand movements from side-to-side but not forward / back), due to limitations with things like depth of field tracking through a webcam, which has yet to be addressed.
  • The back-end support for the capability is only available on Aditi (the Beta grid) and within the following regions: Bunraku, Marionette, and Castelet.
  • Puppetry requires the use of a dedicated viewer, the Project Puppetry viewer, available through the official Second Life Alternate Viewers page.
  • No other special needs beyond the project viewer are required to “see” Puppetry animations. However, to use the capability to animate your own avatar and broadcast the results, requires additional work – refer to the links below.
  • There is now a Puppetry Discord channel – those wishing to join it should contact members of LL’s puppetry team, e.g. Aura Linden, Simon Linden, Rider Linden, Leviathan Linden (not a full list of names at this time – my apologies to those involved whom I have missed).

Bugs, Feature Requests and Code Submissions

  • For those experimenting with Puppetry, Jiras (bug reports / fixes or feature requests) should be filed with “[Puppetry]” at the start of the Jira title.
  • There is also a public facing Kanban board with public issues – those experiencing issues can also contact Wulf Linden.
  • Those wishing to submit code (plug-ins or other) or who wish to offer a specific feature that might be used with Puppetry should:

Further Information

Meeting Notes

New Viewer Version – 6.6.3.575529 Dated October 12th

  • This viewer uses a different, more efficient data format sending updates up to the region, and from the region to viewers.
    • The new and old formats and viewers are not compatible; someone on the new project viewer will be unable to see puppetry rendered for someone using the older viewer version, and vice-versa.
    • It is hoped that severe breakages between viewer versions like this will be avoided going forward, but this change was deemed necessary
  • This viewer also a crash (deadlock) fix, and puppetry animations should fade in/out when starting or explicitly stopping (animations may stop abruptly should the LEAP plugin crash, or the data stream is lost, etc.).
  •  Those self-compiling viewers with the puppetry code should ensure they are pulling the updated code from the  6.6.3.575529 (or later as new versions appear) repositories.

Protocol Overhaul

Leviathan Linden Linden noted the project team is going to overhaul the Puppetry/LEAP protocol.

  • The intent is to replace all the current LEAP commands (“move”, “set_this”, “set_that”, etc.), and replace with just two commands: “set” and “get”.
  • On the “set” side:
    • It will be possible set avatar joint transforms, or specify IK targets, and also set various configuration settings as necessary.
    • These set commands will be “incremental” in nature (so that changes can be made to reach the final state), and once set, they stay at the defined value until modified, cleared, or the plug-in “goes away”.
  • On the “get” side:
    • get_skeleton and any other get_foo commands (if used) will be replaced with {get: [skeleton, foo, …]}.
    • A message will be generated and set back to the viewer making the Get request, but the form of the message is still TBD.
  • Meanwhile, the viewer will only do IK for your own avatar, and will transmit the full parent-relative joint transforms of all puppeted joints through the server to other viewers, and LL will make it possible for a plug-in to just supply full parent-relative joint transforms if desired (e.g. no IK, just play the data)
  • This overhaul will also provide:
    • A way to move the Pelvis. This will include both a pre-IK transform (which is just setting the Pelvis transform) and also a post-IK transform, in case the avatar is to be moved after setting all the joints.
    • A “terse” format for the LEAP/Puppetry protocol to simplify some “set” commands to reduce data going over the LEAP data channel. It will be possible to mix these “terse” command with long-form explicit commands.
  • Leviathan plans to break all of this work down into a set of Jira issues and place them on the kanban board for ease of viewing.

The overall aim of this overhaul is to make the protocol more easily extendible in the future.

To the above, Simon Linden added:

The data stream is radically different than what we started with. Essentially your viewer will do the work for your avatar: send[ing] all data needed for your puppetry animations [so] the people seeing you just have to use those positions – no IK or significant processing. That should help out in the long run with crowds 

Example Script

Simon Linden has produced a simple example script that is pushed to the Leap repository:

  • It reads a JSON file and sends that puppetry data to the viewer.
  • Using it, is is possible to edit some values, save the JSON text file, and see bones move as an example of doing so.

In Brief

  • BUG-232764 “[PUPPETRY] [LEAP] Puppetry should be able to ‘Get’ and ‘Set’ avatar camera angle” has been raised to go with the protocol overhaul, and while it has yet to be formally accepted, has been viewed as a good idea by the Puppetry team.
  • Puppetry does not support physics feedback or collisions as yet, and work for it to do so is not on the short list of “things to do next”
  • There is currently an issue of “near-clipping” within a a first-person (e.g. Mouselook) view and using puppetry (so, for example, holding a hand up in front of your avatar’s face in Mouselook results in the hand being clipped and now fully rendering). This is believed to by an artefact of the viewer still rendering the head (even though unseen when in first-person view), and this interfering with rendering near-point objects like hands. The solution for this is still TBD.

Date of Next Meeting

  • Thursday, October 27th, 2022, 13:00 SLT.