Optimize OpenSim World Performance

Created by Oren Hurvitz
Last updated: Jul 25, 2020 by Ilan Tochner

Overview

OpenSim provides world owners with the freedom to create large worlds with many prims and complex scripts, but this freedom can come at the cost of performance. As a world owner it's important to make sure that your worlds perform well, because slow worlds cause a bad user experience: the world appears slowly around the user; animations are choppy; and moving in the world is frustrating because the avatar appears to jump from place to place instead of moving smoothly. This page provides information about how to make your Kitely worlds fast.

If you need assistance then try posting a question in one of Kitely's Content Creation forums.

Web-Based Monitoring Tool

Kitely worlds are automatically started when avatars enter them, and are shut down a few minutes after the last avatar leaves them. While your world is active you can check its performance using the Active World panel, which appears in the world's World Page. This panel contains the following information:

  • Active Time - how long the world has been active in the current session.

  • Visitors - the current number of avatars in the world.

  • CPU (Number) - the current CPU usage of the world.

  • CPU (Graph) - a history of the CPU usage.

  • Packets (Graph) - the number of networking packets sent between the world and the viewers that are connected to it.

The graphs appear shortly after the world has started, and they update automatically every 30 seconds.

  • In both graphs, the horizontal axis shows the time (in the UTC time zone).

  • In both graphs, the left vertical axis shows the number of visitors in the world. (Amusingly, the graphs appear to indicate that it's possible to have 0.50 or 1.50 visitors. Please ignore this; it's a limitation of the graphing software.) This statistic is drawn using the yellow line.

  • In the CPU graph, the right vertical axis shows the CPU usage.

  • In the Packets graph, the right vertical axis shows the number of packets.

These graphs make it easy to see how the number of visitors in the world affects the CPU and bandwidth usage. It's natural for worlds to use more CPU power as the number of visitors increases. However, if the world is using a lot of CPU power even when it has few visitors then the world probably needs to be optimized.

Stop world

You can use the Stop world link to force the world to stop. This is rarely needed, since worlds automatically stop a few minutes after the last visitor has left them. However, sometimes you might believe that the world is behaving badly and you want to restart it, and in that case stopping the world explicitly saves the normal delay that occurs between the time that the last visitor has left the world and when the world is stopped. This also enables you to stop a world even if there are avatars inside it.

Stopping a world is also useful for fixing the rare error of "ghosted avatars". Avatar ghosting occurs when the simulator fails to detect that a user has closed their viewer. This leaves their avatar in the world, and prevents them from entering it again. Restarting the world removes the ghosted avatar.

After the world has stopped, enter the world again in order to restart it.

CPU

The most important statistic is the CPU usage. This number can range between 0% and 100%. If the CPU usage is high then the world is doing a lot of work: on physics, scripts, or serving many visitors. For a world without many visitors, the CPU should settle around 10% or less. If it remains higher than that for a long time then the world is behaving poorly and it needs to be optimized. The section "Viewer-Based Tools" explains how to find what is causing the world to behave slowly.

Packets

There are three statistics for packets:

  • Packets Received - packets sent from the viewer, and received by the world. E.g., when a user moves around, the viewer sends packets to tell the world the direction in which the user is moving.

  • Packets Sent - packets sent from the world to all of the viewers connected to the world. The more complex the world, and the more visitors are in it, the more packets need to be sent.

  • Packets Resent - packets that the world had to resend because they didn't reach a viewer. This statistic should be low (ideally, 0).

If the "Packets Resent" statistic is high then it indicates that one or more users are having a bad experience. Sometimes this is the "user's fault", e.g. if they're on a poor network such as a cellular connection. Other possible reasons for lost packets are if there are so many visitors inworld that the simulator is unable to serve them all fast enough, or if the world is working heavily on other things (such as physics or scripts) and fails to send networking packets to the viewers when it should. This is why it's important to keep the CPU Usage low.

Please note that Hypergrid visitors to your world can create a lot more network load than other Kitely users. You'll therefore see a correlation between the time that they enter your world and increased network activity.

What to Do

If you see that your world is using a lot of CPU, then you should do something to reduce its CPU usage. The next section describes some Viewer-based tools that allow you to see what the server is spending its time on, so that you know what to optimize.

Viewer-Based Tools

The viewer has tools that show what the world is spending its time on. The first tool is the Statistics panel. You can open this panel by pressing Ctrl+Shift+1 in the viewer. The panel shows many statistics, but the two most important statistics are the Physics Time and Script Time.

This screenshot was taken in the Kitely Welcome Center, which is a highly optimized world, so both its Physics Time and Script Time are low (under 1 ms). However, if you see a high Script Time or Physics Time then you should investigate which objects are causing this, and whether they can be optimized.

(By the way, you should look at the Physics Time statistic and not at the Physics FPS, because the Physics FPS is deceptive: it will remain high right up to the point where the physics time becomes too much to handle in one frame, at which point it will plummet.)

Open the Region/Estate dialog (Alt+R), and go to the Debug tab. Here, you you can open two dialogs: Top Colliders and Top Scripts. The Top Colliders dialog shows the objects that use the most physics time, and the Top Scripts dialog shows the scripts that take the most time. With this information you can decide what to do: optimize the objects, or disable them.

Optimizing Physics Time

Physical objects are great for some types of simulations, but they're also big performance hogs that can slow down the world and cause lag. You can tell if you have a problem with slow physics by checking the Physics Time statistic.

If your world has a high Physics Time then check the Top Colliders dialog to find which objects are using the most physics time. Here's what the Top Colliders dialog looks like:

Please note that this dialog only works well if you're using the ODE physics engine (which is the default). If you switched to the BulletSim physics engine then this dialog might not help you. In that case, consider switching back to ODE, as it is faster overall, and also enables use of the Top Colliders dialog.

Once you find which objects are slow you have several options for dealing with them. The first two options are the easiest because you can perform them directly from the Top Colliders dialog:

  • Click the Return Selected button to remove the objects from the world and add them to your inventory. (If you do so then you might want to return one object at a time, because if you return multiple objects at once then they'll create a Coalesced Object.)

  • Click the Disable Selected button to remove the Physical flag from the objects. The objects will still prevent avatars from walking through them, but they will no longer move when avatars bump into them. This is often the best solution. Note: clicking "Disable Selected" actually does two things: it removes the Physical flag from the objects, and it also disables their Scripts. So if any of these objects have scripts then you probably won't want to use the Disable Selected button on them. Instead, modify the objects individually as explained below.

The next options for dealing with slow physical objects require you to visit the objects inworld and modify them one at a time. To start, click the Teleport to button to arrive at each object's location. Then, you can do one of the following:

  • Remove the Physical flag from the object. This is similar to the "Disable Selected" button, except that it allows scripts in the object to continue running.

  • Make the object Phantom. Phantom objects allow avatars to walk through them.

  • Simplify the object's Physics geometry. This option requires the most work, but it allows the objects to remain Physical while reducing their CPU usage. For mesh objects, you'll need to re-upload them with a simpler physics mesh.

Optimizing Script Time

The Script Time shown in the Statistics panel is the sum of the execution time of all of the scripts in the world, in a single frame. Please note that the Script Time is independent from all the other times shown in this panel. That's because scripts execute on separate threads, outside of the frame processing time. This has two important implications. First, because scripts execute outside of the frame, it's possible that the frame time will appear to be good, even though in fact the world is operating slowly due to heavy scripts activity. Therefore, if you have a high Script Time then you should optimize your scripts, regardless of what the other statistics show. Second, it's possible for the Script Time to be very high, in some cases even hundreds of ms, i.e. longer than the frame time.

If you see that your world is using a lot of script time then the next step is to find which scripts are causing the problem. To do this, open the Top Scripts dialog. This dialog shows a list of the slowest scripts in the world. For each script, it shows the total time that was spent executing that script in the last 30 seconds. (Note that 30 seconds is much longer than a single frame, and therefore the times shown here have little correlation with the Script Time shown in the statistics panel.)

This screenshot was taken in the Kitely Welcome Center. The scripts shown here are pretty fast: the slowest script took just 5.5 ms, and remember that this is the total execution time over a 30-second period.

If your world contains slow scripts then they'll appear at the top of the list in this dialog, and you'll be able to see that they take much more time than the rest of the scripts. Alternatively, another cause of slow performance is if your scripts are efficient, but you have thousands of them. In that case you should consider why you need so many scripts, and whether you can reduce their number. (Note that this dialog won't show thousands of scripts, but if you see a hundred identical scripts then you may assume that there are thousands more behind them...)

Once you find which objects have slow scripts, you have several options for dealing with them. The first two options are the easiest because you can perform them directly from the Top Scripts dialog:

  • Click the Return Selected button to remove the objects from the world and add them to your inventory. (If you do so then you might want to return one object at a time, because if you return multiple objects at once then they'll create a Coalesced Object.)

  • Click the Disable Selected button to disable the scripts in the objects. Note: clicking "Disable Selected" actually does two things: it removes the Physical flag from the objects, and it also disables their Scripts. So if any of these objects are Physical then you probably won't want to use the Disable Selected button on them. Instead, modify the objects individually as explained below.

The next options for dealing with slow scripts require you to visit the objects inworld and modify them one at a time. To start, click the Teleport to button to arrive at each object's location. Then, you can do one of the following:

  • Stop the scripts in the object. This is similar to the "Disable Selected" button, except that it allows Physical objects to remain physical.

  • Edit the scripts to make them faster. How to do this is out of scope for this page; if you need help, try asking in our Scripting Forum. Whenever you save the script, the new version starts executing immediately. Click Refresh in the Top Scripts dialog to see if your changes made the script faster. Keep in mind that this dialog shows the total execution time in the last 30 seconds, so it will take some time for your changes to have an effect.

Once you've optimized enough of the slow scripts, check the Script Time in the Statistics panel to see if it has dropped significantly. And/or, check the Active World panel in the World Page to see if the CPU usage has dropped.

Tips for Good Performance

Tip #1

DO: reuse objects, assets and textures after uploading them once into Kitely.
DO NOT: upload multiple copies of the same assets.

Explanation

Each asset gets its own Unique Universal Identifier (UUID) when it is uploaded into Kitely. Each viewer that connects to your world must download all the assets that it views. If you use multiple copies of the same asset that have different UUIDs then the viewer will be forced to download the same asset multiple times, once for each UUID. However, if you reuse the same asset UUID then you save the viewer from having to download multiple copies of the same asset. This will reduce world download times and will improve the Viewer Frames Per Second (Viewer FPS) because it won't have to handle multiple different copies of the same asset. 

Tip #2

DO: use the lowest texture resolution that still looks good for the object inworld, when viewed at a reasonable distance for that object's placement in the world.
DO NOT: use 1024x1024 texture resolutions, unless the object has a large visible surface area or is used in multiple objects in the world that have a combined large visible surface area.

Explanation

Textures with a high resolution take longer to download than lower-resolution textures, and they use more memory in the users' graphics cards. By reducing texture resolution you can help increase Viewer FPS.

Consider how far visitors to your world will typically be from the the textured object. For example, if you have a wall that will typically be a few meters away from users, then the texture resolution only needs to be high enough for the wall to look good from that distance: it's not necessary to upload a very high-resolution texture in order to cater to users who insist on pressing their noses to the wall.

Tip #3

DO: use scripts that change object properties just once when the world starts.
DO NOT: use scripts that constantly update objects.

Explanation

Each time a script changes an object inworld, that change needs to be sent to all the viewers connected to the world. If your world is constantly changing then a lot of server CPU and bandwidth will be spent on updating viewers, leaving fewer resources to handle other activities such as avatar movement. It's much more important that visitors to your world get a lag-free experience than that they see flashing lights or moving objects in the background, so try to minimize object updates if you can avoid it.

Pro Tip: you can rotate an object without creating server load by making the object phantom and calling llTargetOmega.

Tip #4

DO: use an object's texture properties to create texture animations.
DO NOT: use scripts that constantly update objects.

Explanation

Texture properties can be used to create animations that are run only on the viewer so they don't consume any server resources.

Tip #5

DO: use scripts that are triggered by an inworld event.
DO NOT: use scripts that constantly check if a condition exists.

Explanation

Scripts that constantly check for conditions consume server CPU resources even when those conditions don't exist yet. Event triggers use fewer server resources.

Tip #6

DO: use simplified collision meshes for the meshes you upload.
DO NOT: reuse the geometry mesh as the object's collision mesh.

Explanation

Mesh objects contain geometry meshes that describe the object's geometry at various Levels of Detail (LOD). All non-phantom objects also have a collision mesh that is used by the physics engine to calculate collisions between the object and other non-Phantom objects in the world. The collision mesh is never seen by people; it is only used by the physics engine. Using a complex collision mesh can increase load on the physics engine and significantly increase world startup times. By using a simplified collision mesh you will reduce physics engine load and decrease the amount of time it takes for your world to start up.

Tip #7

DO: make objects phantom if possible.
DO NOT: create Physical objects, if you can avoid it.

Explanation

Physics calculations are one of the two biggest reasons for slow worlds (the other being scripts). You can make your world faster by preferring to use simpler types of physics for your objects, as this will reduce CPU usage.

Here's a list of how much work different types of objects require from the physics engine:

  1. Phantom objects - very fast (they require no work at all from the physics engine).

  2. Most non-phantom objects - fast.

  3. Physical objects - very slow.

Physical objects are much slower than any other type of objects, so avoid them if you can. If your particular world simply must use Physical objects then go ahead and use them, but make sure to optimize their collision mesh, and keep close track of the server's CPU usage.