Category: 2012 Project Updates

Upcoming SL projects update

Posted on by Inara Pey

There is a lot going on in terms of projects and development work in Second Life. The following is a further update on a number of key projects I’ve been following and reporting upon through the pages of this blog.

Group Services Project

The Group Services project is an attempt to improve the management and editing of large SL groups by replacing the current UDP-based service (which has capacity issues with the size of group lists it can comfortably handle) with a new HTTP-based service. The project viewer for this is already available (for Windows, Linux and OSX.), as I reported last week.

This week sees the server-side code rolled-out to all three RC channels (Magnum, LeTigre and BlueSteel), allowing the project viewer to be tested in handling very large groups (significantly larger than are available on Aditi). Note that those running viewers without the new code on any RC region will be unaffected, as they will continue to access the current UDP service.

There are still no timescales as to how long the testing of the service will last (“It’ll take as long as it takes,” Baker commented recently), or when the viewer code will progress beyond the project viewer. However, a number of things should be noted in reference to the eventual roll-out:

  • The viewer code is not being made back-compatible with V1 code by the Lab. Therefore, TPV developers using the V1 code base will have to backport the code themselves in order to use the new service
  • The initial HTTP service roll-out does not include any data compression. This means there will still be some delay in downloading member lists for very large groups with tens of thousands of members
  • Once the new service is rolled-out, which service is used is entirely transparent to the user. If a viewer with the new code is running on a region which has the server-side HTTP service, it will connect to that service. If it is on a region using the older UDP service, it will connect to that service instead
  • Once the HTTP service is fully deployed, viewers which do not implement the viewer-side code will still be able to access groups with member lists up to 10K in size via the UDP service until such time as it is switched off (which will not occur for some time after the HTTP service has been rolled-out). However, attempts to access groups with lists larger than 10K will fail.

Interest Lists and Object Caching

It’s been a while since I’ve reported on Interest Lists Object Caching, which forms a part of the Shining Project.

To recap: when you enter a region at the moment, your viewer receives a huge amount of information on what requires updating, much of it relating to things you can’t even see from your position in the region. This information is received in no particular order, with the familiar result that things appear to rez in your view in a totally random order. Not only that, but the chances are that if you’ve previously visited the region, much of the information being sent to your viewer is already locally cached – but is being ignored. The focus of this project is to both optimise the data being sent to the viewer and the information already cached on the viewer with the aim of significantly improving object rezzing times in terms of speed and order (so objects closer to you rez before those further away, for examples).

Object caching and interest list changes: easing the pain of random rezzing

Andrew Linden had hoped the project would be going to QA this week ready for roll-out to one or more RC channels in the near future, but some last-minute problems popped-up and have delayed things until he can get them sorted out. In the meantime, the code has been deployed to a number of regions on Aditi, and Andrew plans to, “Try to throw a pile of test avatars at it to stress it out. Later this week.” No viewer-side changes are associated with this work.

Materials Processing

Work continues on the project to bring materials processing to Second Life. Last week, it appeared as if the new materials – normal and specular maps – would have their own rotation and positioning options independent of any texture (diffuse) map. This week, it appears that this is the hoped for situation, but the matter is still open to question – which goes to show how fluid the project is.

The new capabilities require changes to the rendering pipeline, and details have been released on some of what this entails.

In order to work, normal and specular maps require what is referred to as per-pixel lighting (as noted in the original blog post on the subject). As such, there has been a debate on whether it would be better to develop a per-pixel lighting framework within the viewer, or work to make the current deferred rendering system more accessible to per-pixel lighting capabilities. As the latter approach will allow getting materials processing working within SL sooner than would otherwise be the case, it has been decided to go that route. Thus work is focused on making the current lighting system more configurable and able to better handle a broad range of material types (metallic, matte, plastic, etc.), together with adding support for both per-object and per material shading differences.

However, a dedicated per-pixel lighting framework does offer advantages of its own, and as such is being considered as a possible future extension to the project, which may be implemented at some point down the road. One such advantage is that could potentially be run in a non-deferred mode, which might lightening the load on older graphics systems.

Please use the page numbers below to continue reading

Pages: 1 2

Belated avatar baking service update

Posted on by Inara Pey

The server-side elements of the new avatar baking service (part of the Shining Project) continue to progress, alongside what Nyx Linden has referred to as, “Some pretty scary viewer re-architecting” which is being undertaken by the Lab in order to try to isolate some of the avatar baking aspects from the rest of the viewer code base. As such, he’s anticipating the eventual code merge to be “fairly significant” when it does happen, although at the time of commenting (the last TPV/Dev meeting, September 7th), the code was not in a condition where it could be used by any of the TPVs.

Bake fail: a familiar problem for many

Overall, the viewer elements of the project are the priority, with the aim remaining to get the viewer code completed first, and to make it available to TPVs so that test viewers can be built. This is likely to happen before the code is ready for any formal release, the aim being to allow TPV devs to carry out test merges and to let LL know anything else has been broken as a result of the changes made to the code in order for the new service to work.

Once the code is available for testing both within the viewer and on test regions on Aditi, Nyx will be looking for “as many people as possible to pile-on” and test the code in order to see how the service works and how it may break, so that  by the time the viewer code is merged for release purposes, it will be as robust as possible.

Test Avatars

To assist with the work, Nyx is also looking for volunteers willing to take part in the initial round of testing using avatars wearing multi-layer outfits (e.g. outfits combining undershirt, shirt and jacket layers, etc; outfits using multiple elements of the same layer; outfits with system skirts and glitch pants, outfits using alpha and/or tattoo layers, and so on). Anyone interested in joining-in the testing should contact Nyx via e-mail or by sending him an in-world notecard, specifying the avatar name and details of the outfit itself. When volunteering, bear in mind that:

  • The outfit must be a Viewer 3 outfit, and your viewer must support the Current Outfits folder (which is used to drive the new service from the viewer end)
  • Testing will be on Aditi, and as such, the avatar and outfit must be available on Aditi. If necessary, you may need to update your Aditi inventory to make the outfit available.

Going Forward

The plan is still to have the new code support both the “current” method of avatar baking  and the new baking service, until such time as the new service is fully deployed. This means that if a user is in a region that does not make use of the new baking service, avatar baking will continue to be handled using the viewer-side mechanism. However, if the user is on a region that utilises the new baking service, avatar baking will be handled through that, with a flag set via the region capabilities being used to distinguish whether or not the new service is available.

There is still no definitive time frame for the project because of the complexity involved in both developing the viewer code (which is currently using a branch of the SL development viewer, but will obviously be moved to whatever code release is current) and with the development of the new server-side service. As such, it is still liable to be at least another month or so before the code is ready for significant testing on Aditi.

Related Links

This is a little overdue due to problems earlier in the week accessing the recording of the TPV/Dev meeting (which I was unable to attend in person). Thanks to Oz for sorting the problem out, allowing me to catch-up on overdue updates on this and JIRA matters etc.

More on materials processing

Posted on by Inara Pey

Materials processing was further discussed at the Content Creation Informal User Group meeting on September 11th. During the meeting, Oz confirmed that normal and specular maps will have their own rotation and offset controls from the start, which stands in difference to thoughts that they would initially be locked to the same rotation, etc., as the texture map used on an object or object face.

Materials processing: using a normal and diffuse (texture) map to generate a 3D effect on an in-world wall – coming to SL in the near future

This means that the new materials processing maps will have the same positioning and scaling properties are currently available for texture (diffuse) maps, namely:

  • Mapping: one of Default or Planar
  • Horizontal  and Vertical Repeats: number of repeats of the image over the surface (used only in default mapping)
  • Rotation: degrees clockwise by which the map is rotated
  • Repeats per Metre: number of repeats of the image per in-world meter of the surface (used only in planar mapping)
  • Horizontal and Vertical Offset: distance in meters the image is shifted right (horizontal) or up (vertical) on the surface.
Options to be used by normal and specular maps as well as by textures

This obviously increases the amount of control content creators will have over the additional maps once the new capabilities are live, but it does open up more issues around incorporating the capabilities into the build floater. Thus, it was the build floater which once again became the focus of the conversation during the meeting.

How the build tools are presented to users has already been the subject of much broader consideration within the CCIIUG, although that discussion appears to have stalled for the time being. The materials processing process, however, requires a more focused examination of the build floater in terms of enabling the additional controls without either making the existing floater too large or overly complicated.

As such, two ideas were briefly discussed at the meeting, and further input to them both will doubtless be sought. The first involved having the additional options contained within the existing Texture tab as sub-tab options, while the second involved having all of the mapping options (Texture/diffuse, normal and specular) and their associated positioning and sizing options moved to a dedicated floater of their own.

Of the two options, both have merit and pitfalls. Sub-tabs mean keeping everything together on a single floater, but could lead to things becoming cramped or possibly confusing. Splitting between two floaters would provide more space with which to present options and controls, but could lead to a reduced in-world view should people find they need to work with both floaters open.

Given the small number of attendees at the meeting, neither option was discussed in-depth, and so, as mentioned above, this is liable to be a subject which is returned to in future meetings.

Rendering Pipeline Changes

As a part of this project there will be changes to the viewer rendering pipeline. While the project is still some way off in terms of delivery, Oz Linden used the TPV/Dev meeting of September 7th to advise TPV developers that they will need to ensure they are current with the 3.3.4 rendering code, which will form the basis for implementing the materials processing system changes.

Oz also indicated that while it would be “quite a bit of time” before the project reaches a development or beta viewer because the overall specification is not yet frozen (but is, in his words, “Very, very slushy”), it is hoped that a project viewer would soon be available to allow the new materials processing capabilities to be seen and tested.

In referencing the release of a project viewer, Oz advised against anyone trying to pull the code and using it in any viewer intended for wide distribution. This is because it is possible the material processing capabilities may change in very significant ways before they are ready for more general release, leaving the code presented in any project viewer as limited in both value and function.

Accounting System Costs

It is still unclear whether the new materials processing system will fall under the new land accounting system in terms of server and rendering costs. While questions have been asked in this regard, no definitive answers have yet been furnished by LL. This again is potentially due to the matter still being under consideration and the specification itself still being revised as impact and changes are considered.

Discussions on the project will doubtless resume at the next CCIIUG meeting.

Related Links

Recent SL viewer activities

Posted on by Inara Pey

It’s been a while since I’ve reviewed any of the official SL viewers from LL, so here’s a quick round-up of recent releases.

New Log-in / Account Creation Prompt

The new account creation prompt, displayed if the viewer does not locate any user settings files on a computer, and which first appeared in the 3.4.1.263582 release (August 16th), now looks to be the default option for all development / project viewers. It is part of both the most recent Mesh Deformer project viewer (3.4.1.264215, August 31st), and the new HTTP Group Services project viewer (3.4.1.264495, September 7th). However, it has yet to filter through to either the Beta or release versions of the Viewer.

Account creation prompt: now standard on all development / project releases of the SL viewer released since August 16th (click to enlarge)

Mesh Deformer Project

August 31st saw a new release of the Mesh Deformer (3.4.1.264215), which includes a revised mesh uploader floater with deformation options for the male and female shape.

New deformation options

According to Nalates Urriah, the new options invalidate all test items so far provided for the project, and new samples are now required, although no comments to this effect appear to have been made on the JIRA or elsewhere, so they may have been confined to a user group meeting. Details on how to provide test items can be found in Oz’s forum post on the matter. The JIRA (STORM-1716) for this project is still open for viewing and comment.

Group Services Project Viewer

As noted this week, there is now a Group Services (group management) project viewer available for testing the new HTTP group management service. The server-side of this project has yet to be rolled-out to Aditi, so it cannot be tested as yet. However, Baker Linden, who is developing the service, is apparently updating the JIRA, SVC-4968 (which is still publicly viewable) with the project status, and has indicated he’ll post when the server-side elements are available for testing.

The viewer is available in Windows, Linux and OSX flavours.

HTTP Libraries Viewer

The HTTP Libraries project viewer (3.3.3.262585) appeared on July 27th. This project, which Monty Linden is driving, is currently aimed at improving texture downloading and rezzing as a part of the Shining project.

HTTP Libraries project viewer: improved texture loading and rezzing

Texture loading / rezzing would appear to be significantly faster on this viewer compared with other offerings, although there also appear to be what might be placebo effects associated with it. Some people have reported that floaters, etc., seem to load more slowly, and some have reported various performance improvements outside of the HTTP library changes.

Beta Viewer and Release Viewers

The Beta viewer (3.4.0.264445 at the time of writing) continues to be focused on pathfinding, with fixes and updates going into it on a weekly basis – which is why the pathfinding tools have yet to release a release version of the viewer.  The removal of JIRA numbers from the release notes now means that tracking issues previously being watched is that much harder (even if the JIRA themselves are no longer accessible, having the JIRA numbers still visible facilities easier identification of issues being specifically tracked).

Similarly, the release viewer (3.3.4.264214) appears to be focused on bug fixes and general improvements, with the release notes currently benefiting from the retention of JIRA numbers, making scanning for specific fixes easier.

Performance

I carried out basic performance tests on the viewers listed above using Dimrill Dale as my sample sim, during a period when there were the same number of avatars in the region (5, including myself). Tests were carried out in the same location on the region, looking in the same direction and with the same viewer settings (e.g. Graphics on high, Draw Distance set to 260m, using default time-of-day, with deferred disabled / with deferred enabled and lighting set to Sun/Moon+projectors, etc.). While all such tests are rough-and-ready, these did tend to show that all of the viewers offer the same performance on my default PC (see the sidebar panel on the right of this blog’s home page for system details). My results were:

  • Non-deferred: 18-20fps
  • Deferred with Sun/Moon+ projectors: 8-10fps

Similar figures were also obtained using the current Firestorm and Exodus viewers, although with deferred enabled and Sun/Moon+projectors active, Firestorm was slightly down at an average of 17-18fps, the other viewers being closer to an average of 19-20fps.

Group management project viewer released

Posted on by Inara Pey

As I recently noted, Baker Linden has been working on the large group management / editing issues, developing a new HTTP-based service to replace the current UDP service which has significant issues handling groups with more than 10-11K members. At the TPV/Developer meeting on the 24th August, he indicated that a project viewer would be available in the near future.

On Friday September 7th, he updated the JIRA on the issue with notes that the project viewer is now available for Windows, Linux and OSX.

In commenting on the JIRA (SVC-4968), Baker notes that the server-side code for the new service has yet to be deployed to Aditi, where initial testing will take place, and adds that he’ll be providing an update on the status of the server code once the situation has been clarified. He goes on to add:

There may be some issues during testing. When getting the member list of a large group, other info (group title, group info, etc.) may not properly load. This is an issue with the speed of Aditi’s SQL server and shouldn’t occur once live on Agni. To receive the rest of the data, wait for the member list to appear (this can be upwards of a few minutes), go back to the My Groups panel of the people floater and view the group profile again. The query will be cached this time, and the member list will appear quicker than it did before (depending on your connection speed). The rest of the information should be received this time.

If you find any problems while testing, please send me a message in-world (on Agni).

Large group loading: part of the group management problem

As noted in my previous report, in the first implementation, the data will be uncompressed. This means there will still be some delays in group loading (Baker previously estimated that a 40K member group is around 5Mb in size and could take up to a few minutes to download, depending on someone’s connection speed). Data compression is being looked at for a future release, although as noted in the comments on my last article, some are wondering why paging group data isn’t being implemented (does the viewer API support it?).

Another point of note is that the new service is not compatible with V1 code, so adoption by V1-based viewers is liable to require some backporting. This is important, as once the new HTTP service is rolled out, the older, more limited UDP service will be capped at groups containing 10,000 members – larger groups will not function.

There is still no definitive time scale for the roll-out of the new service. However, it seems likely that once available on Aditi, the server code will remain there for testing for at least a couple of weeks prior to it being added to a RC channel on the main grid. How long the testing period across both will be is open to question, and a lot will depend on feedback as to how well the new service performs.

HTTP and Group Services updates

Posted on by Inara Pey

There are a number of projects underway at the moment to improve various aspects of Second Life performance. Some of these have been reported on as a part of the Shining Project, others are being dealt with elsewhere are reported on through the likes of the SL Scripting User Group and the fortnightly TPV/Developer meetings.

The following is by way of a brief update on the ongoing HTTP Library and Group Management projects with information taken from the most recent TPV/Developer meeting (recording link).

HTTP Library

The focus of this aspect of the Shining Project is to improve the underpinning HTTP messaging that is crucial to simulator / simulator and simulator / viewer communications, and it is under the management of Monty Linden.

Discussion on progress with the project commences at 36:36 into the recording.

The project code (textures only) is with the Linden Lab QA team and is expected to be in the 3.4.1 viewer once it has been released by QA. In the meantime, the HTTP project viewer was updated at the end of July. Many people are noticing improvements in viewer performance that go beyond initial texture loading, although there have been reports of other aspects of the viewer which use HTTP apparently being “slower” to use. This latter issue is most likely a false impression, with Monty commenting at the August 24th meeting that, “Most parts shouldn’t be affected. It’s competitive, when you’re doing both texture downloading and some of that work … but other things aren’t being cheated if you’re not downloading textures at the time.”

An issue has been noted in older Macbook Pro systems (late 2007 into 2008 dual-core systems, although the span of the problem isn’t clear) using nVidia drivers, wherein the expected speed-up with cached data which can be seen on other systems isn’t occurring. Monty is still investigating this. Overall, however, feedback on this project has been positive.

Group Management Functions

Large group loading: a familiar problem

Baker Linden has been working to resolve this problem, and his plan is also to go the HTTP route, which will require changes on both the server and the viewer sides of the equation. His comments on progress start at 42:53 into the TPV/Developer meeting recording.

The server-side code for an initial implementation of the solution has been passed to LL’s QA and is expected to be rolled to selected regions on the Beta (Aditi) grid soon.

In terms of the viewer, the plan is to develop a Project Viewer, which will be made available in the near future for people to use with the Aditi test regions. How soon this viewer is likely to appear is open to question – the code will initially need to be passed by LL’s QA (who may have received it on the 24th August) prior to the viewer being built. Once in the project viewer repository, the code will also be available for TPVs to produce test viewers of their own.

How long the testing period will last is also open to question and dependent upon feedback / issues arising. However, the plan will be to follow the usual pattern for roll-outs in that once the code has been tested on Aditi and necessary updates made, it will be rolled to a main grid RC for more more involved testing. This is important, as there is a significant different in the number and sizes of groups operating on the two grids. For example, the largest group on Aditi numbers some 40,000 members; on the main grid the largest group is about 112K, and there are many more groups with between 40K and 112K members.

One thing that has been made clear is that there will be no attempt at backward compatibility with V1-based viewers on the Lab’s part; the new code will be aimed solely at the V3 code base. However, V1-based viewers will still be able to use the UDP protocols for group management, although the LL servers will limit UDP access to groups with 10K members or fewer, so V1-viewers will have some more code backporting on their hands.

There will also initially be some issues around the new HTTP protocol. For example, in the first implementation, the data will be uncompressed. This means that a 40K member group is around 5Mb in size, which can take up to a few minutes to download, depending on someone’s connection speed, so some frustrations are liable to continue. While data compression will eventually be used, this is not planned for the initial implementation.

The discussion involved providing an option to routinely clear-down group lists based on people’s last log-in date, or who have not logged in for a (group owner specified) number of days. However, LL are not going to implement such a feature on the grounds that it could lead to mistakes being made, and people being accidentally removed from a group.

Time Scale and Implementation

As mentioned above, there is no definitive time scale for this work to be completed. Testing is liable to take several weeks at the very least, so it is unlikely the new group management capabilities will be rolled-out on a widespread basis for at least another month, or possibly longer.

However, and like the upcoming new avatar bake service, once the server code is available on the grid, the switch-over will be transparent. If a viewer has the code to use the new group management HTTP service, it will do so, if it has not been updated, it will continue to use the UDP service (with the aforementioned 10K “cap”) until such time as that capability is “retired” from the grid.