The following notes have been taken from chat logs and audio recording of the Thursday, March 23rd, 2023 Puppetry Project meetings held at the Castelet Puppetry Theatre on Aditi. These meetings are generally held on alternate weeks to the Content Creation User Group (CCUG), on same day / time (Thursdays at 13:00 SLT).
Notes in these summaries are not intended to be a full transcript of every meeting, but to highlight project progress / major topics of discussion.
Project Summary
General Project Description as Originally Conceived
LL’s renewed interest in puppetry was primarily instigated by Philip joining LL as official advisor, and so it really was about streaming mocap. That is what Philip was interested in and why we started looking at it again. However since Puppetry’s announcement what I’ve been hearing from many SL Residents is: what they really want from “puppetry” is more physicality of the avatar in-world: picking up objects, holding hands, higher fidelity collisions.
As a result, that is what I’ve been contemplating: how to improve the control and physicality of the the avatar. Can that be the new improved direction of the Puppetry project? How to do it?
Leviathan Linden
- Previously referred to as “avatar expressiveness”, Puppetry is intended to provide a means by which avatars can mimic physical world actions by their owners (e.g. head, hand, arm movements) through tools such as a webcam and using technologies like inverse kinematics (IK) and the LLSD Event API Plug-in (LEAP) system.
- Note that facial expressions and finger movements are not currently enabled.
- Most movement is in the 2D plain (e.g., hand movements from side-to-side but not forward / back), due to limitations with things like depth of field tracking through a webcam, which has yet to be addressed.
- The back-end support for the capability is only available on Aditi (the Beta grid) and within the following regions: Bunraku, Marionette, and Castelet.
- Puppetry requires the use of a dedicated viewer, the Project Puppetry viewer, available through the official Second Life Alternate Viewers page.
- No other special needs beyond the project viewer are required to “see” Puppetry animations. However, to use the capability to animate your own avatar and broadcast the results, requires additional work – refer to the links below.
- There is a Puppetry Discord channel – those wishing to join it should contact members of LL’s puppetry team, e.g. Aura Linden, Simon Linden, Rider Linden, Leviathan Linden (not a full list of names at this time – my apologies to those involved whom I have missed).
Additional Work Not Originally In-Scope
- Direct avatar / object / avatar-avatar interactions (“picking up” an apple; high-fives. etc.).
- Animations streaming: allowing one viewer to run animations and have them sent via the simulator to all receiving viewers without any further processing of the animations by those viewers.
- Enhanced LSL integration for animation control.
- Adoption of better animation standards – possibly glTF.
- Given the project is incorporating a lot of additional ideas, it is likely to evolve into a rolling development, with immediate targets for development / implementation decided as they are agreed upon, to be followed by future enhancements. As such, much of what goes into the meetings at present is general discussion and recommendations for consideration, rather than confirmed lines o development.
Bugs, Feature Requests and Code Submissions
- For those experimenting with Puppetry, Jiras (bug reports / fixes or feature requests) should be filed with “[Puppetry]” at the start of the Jira title.
- There is also a public facing Kanban board with public issues.
- Those wishing to submit code (plug-ins or other) or who wish to offer a specific feature that might be used with Puppetry should:
- Discuss them with the Puppetry team: and work with them to ensure a proper convergence of ideas.
- Be signed-up to the Lab’s contribution agreement in order for submitted code to be accepted for review / use:
- Contributor Agreement notes – SL wiki.
- Contributor Agreement FAQ – SL wiki.
- Code Contributor agreement – PDF form.
Further Information
- Introducing Second Life Puppetry – Linden Lab blog post (August 30th, 2022).
- Puppetry: How it Works – Second Life Knowledge Base.
- Second Life Puppetry wiki index page – not that more subjects / categories are / will be added over time.
- Second Life Public Calendar – meeting dates.
- Alternate Viewers page – for the latest version of the Puppetry viewer.
Meeting Notes
Viewer Progress
- An updated version of the project viewer is still “close” – it is currently awaiting clearance by QA.
- This will include the attachment point tracking discussed in previous meetings (e.g., a step towards being able to “pick things up” in SL. The simulator support for this is already in place on the Aditi Puppetry regions.
- However when available it will not include:
- LSL animation control as yet (which has yet to be added to the simulator code anyway. Rider Linden believes he has a good protocol for single avatar animation, but would rather work on it some more.
- Any IK improvements, as Leviathan Linden is still working on these.
- Any extended LEAP API functionality (the added features for getting world position/orientation, lookat position/orientation, camera position/orientation/target). This will be coming in a future viewer update.
- Another thing in this release will be the change to llsd_binary for the LEAP messaging protocol. This will be in the release notes but to use it you will want to update to either 1.3.1 of llbase and/or 1.2.0 of llsd for python scripts. Messages from LEAP scripts to the viewer will still work with the older python libraries, but messages from the viewer to the script will not be parsed correctly.
Server-Side Work
- The LSL function API has been published to the Content Creation Discord group (sorry, I’ve been asked by LL not to publish details on joining the server – if you are a content creator interested in joining it, please contact Vir Linden or attend a meeting (Content Creation / Puppetry and ask in person).
- Getting attachment point positions has been given a throttle, in part to not make it trivial to use LSL to rip an animation, and in part to prevent the server doesn’t get overwhelmed. This latter rate of throttling is variable and can change as load increases/decreases. However, as Rider linden noted, there would always be some delay and some disagreement about the actual position of the attachment point between LSL and all the observing viewers. As such, function is not meant for a high-fidelity use. Collision volumes on the attachment points will be a better solution in this respect, but that is functionality which is still down the line.
General Notes
- Leviathan Linden’s work for streaming the full avatar animation state has stalled, due to it essentially hijacking the main puppetry data channel to send everything, even when not running a puppetry script, through LEAP. As such, Leviathan thinks it needs to be moved to its own experimental viewer.
- Simon Linden’s work on allowing animation uploads of new/different formats has been decoupled from the Puppetry project’s codebase, and is now being built on the main viewer branch, allowing it to move forward without dependencies on Puppetry.
- OpenXR support as a LEAP plug-in is still seen as desirable, since it would allow support for a broader range of devices. However, it is seen as a little more “down the road”, as there is some core infrastructure that needs to finish being vetted prior to work starting on this.
My thanks to Jenna Huntsman for the chat transcript from the meeting, and you can see her video recording of the session here.
Date of Next Meeting
- Thursday, April 27th, 2023, 13:00 SLT.
Inara Pey: Living in a Modemworld 