2023 SL Puppetry project week #10 summary

The following notes have been taken from chat logs and audio recording of the Thursday, March 9th, 2023 Puppetry Project meetings held at the Castelet Puppetry Theatre on Aditi. These meetings are generally held on alternate weeks to the Content Creation User Group (CCUG), on same day / time (Thursdays at 13:00 SLT).

Notes in these summaries are not intended to be a full transcript of every meeting, but to highlight project progress / major topics of discussion.

Project Summary

General Project Description as Originally Conceived

LL’s renewed interest in puppetry was primarily instigated by Philip joining LL as official advisor, and so it really was about streaming mocap. That is what Philip was interested in and why we started looking at it again. However since Puppetry’s announcement what I’ve been hearing from many SL Residents is: what they really want from “puppetry” is more physicality of the avatar in-world: picking up objects, holding hands, higher fidelity collisions. 

As a result, that is what I’ve been contemplating: how to improve the control and physicality of the the avatar. Can that be the new improved direction of the Puppetry project? How to do it?

Leviathan Linden

• Previously referred to as “avatar expressiveness”, Puppetry is intended to provide a means by which avatars can mimic physical world actions by their owners (e.g. head, hand, arm movements) through tools such as a webcam and using technologies like inverse kinematics (IK) and the  LLSD Event API Plug-in (LEAP) system.
  • Note that facial expressions and finger movements are not currently enabled.
  • Most movement is in the 2D plain (e.g., hand movements from side-to-side but not forward / back), due to limitations with things like depth of field tracking through a webcam, which has yet to be addressed.
• The back-end support for the capability is only available on Aditi (the Beta grid) and within the following regions: Bunraku, Marionette, and Castelet.
• Puppetry requires the use of a dedicated viewer, the Project Puppetry viewer, available through the official Second Life Alternate Viewers page.
• No other special needs beyond the project viewer are required to “see” Puppetry animations. However, to use the capability to animate your own avatar and broadcast the results, requires additional work – refer to the links below.
• There is a Puppetry Discord channel – those wishing to join it should contact members of LL’s puppetry team, e.g. Aura Linden, Simon Linden, Rider Linden, Leviathan Linden (not a full list of names at this time – my apologies to those involved whom I have missed).

Additional Work Not Originally In-Scope

• Direct avatar / object / avatar-avatar interactions (“picking up” an apple; high-fives. etc.
• Animations streaming: allowing one viewer to run animations and have them sent via the simulator to all receiving viewers without any further processing of the animations by those viewers.
• Enhanced LSL integration for animation control.
• Adoption of better animation standards – possibly glTF.
• Given the project is incorporating a lot of additional ideas, it is likely to evolve into a rolling development, with immediate targets for development / implementation decided as they are agreed upon, to be followed by future enhancements. As such, much of what goes into the meetings at present is general discussion and recommendations for consideration, rather than confirmed lines o development.

Bugs, Feature Requests and Code Submissions

• For those experimenting with Puppetry, Jiras (bug reports / fixes or feature requests) should be filed with “[Puppetry]” at the start of the Jira title.
• There is also a public facing Kanban board with public issues.
• Those wishing to submit code (plug-ins or other) or who wish to offer a specific feature that might be used with Puppetry should:
  • Discuss them with the Puppetry team: and work with them to ensure a proper convergence of ideas.
  • Be signed-up to the Lab’s contribution agreement in order for submitted code to be accepted for review / use:
    • Contributor Agreement notes – SL wiki.
    • Contributor Agreement FAQ – SL wiki.
    • Code Contributor agreement – PDF form.

Further Information

• Introducing Second Life Puppetry – Linden Lab blog post (August 30th, 2022).
• Puppetry: How it Works – Second Life Knowledge Base.
• Second Life Puppetry wiki index page – not that more subjects / categories are / will be added over time.
• Second Life Public Calendar – meeting dates.
• Alternate Viewers page – for the latest version of the Puppetry viewer.

Meeting Notes

Viewer Progress

• An updated version of the project viewer is due to be made available once it has cleared LL’s QA process. This includes:
  • Using the binary protocol for the LEAP module communication, with new logic which causes LEAP modules to one be loaded by the viewer when they are used.
  • The AgentIO LEAP module adds the ability to adjust the look at target, viewer camera and agent orientation.
  • Support for sending the joint position of your avatar to the server, which is then available in LSL.
    • The code reports the post animation location for attachment points, allowing the the sever to know where things like hands and wings, etc.,  are, and this in turn allows LSL to query where that attachment point is in space and how it is rotated.
• HOWEVER, the animation streaming code (see previous Puppetry meeting notes) will not be in the next viewer update.

Server-Side Work

• The simulator code now has llGetAttachmentPointAnim() support, which should be recognised by the upcoming viewer update.
• The Aditi puppetry regions are to be merged with the updated code so this can be tested.
• While there has been some work completed on animation imports since the last meeting, there was nothing significant for LL to report on progress at this meeting.

General Notes

• There is additional work going on to try to improve the IK system, with the aim of having the basics working better than is currently the case – better stability, etc. This work may appear in the viewer update after the one currently being prepared to go public.
• Performance:
  • To prevent puppetry generating too much messaging traffic (UDP) between the viewer and simulator, a throttle is being worked on so that when the simulator is under a heavy load from multiple viewers running puppetry code, it can tell them all to tone down the volume of messages.
  • There will also be some switches and logic put into place that can be used when needed, helping to protect regions in case the load gets overwhelming.
  • A further suggestion made is to ensure the simulator does not broadcast puppetry messages for avatars seated and not using the code (such as an audience at a performance) to further reduce to volume of messaging, this is viewed as a potentially good avenue of work to consider.
  • There is also a threshold in place – if an attachment point does not move beyond it, it is not considered as moved, which will hopefully also reduce the amount of messaging the simulator has to handle.
• LSL Integration:
  • See: OPEN-375: “LSL Functions for reading avatar animation positions”.
  • This work is now paused. Rider Linden developed a proof of concept, but found that in order to better manipulate parameters within the constraints, a configuration file should be used. He is therefore refactoring the code to do this before proceeding further.
  • The configuration file will be called avatar_constraints.llsd and it will live alongside avatar_lad.xml in the character directory.
• Questions were again raised on whether Puppetry is for VR / will enable the viewer to run VR.
  • It was again pointed out that while Puppetry lays more foundational work which could be leveraged for use with VR headsets, than is not the aim of the Puppetry project.
  • Providing VR headset support is a much broader issue, which would require the involvement of other teams from LL – Product, the Graphics Team, the viewer developers, etc.

Date of Next Meeting

• Thursday, March 23rd, 2023, 13:00 SLT.