conversion error on index_from_pointer

Please post any questions about developing your plugin here. Please use the search function before posting!
User avatar
velocity
Senior Member
Posts: 220
Joined: Sat May 10, 2014 6:17 pm

Re: conversion error on index_from_pointer

Postby velocity » Fri Nov 26, 2021 9:31 pm

So it seems like the NOT_RESULT Is the same as SAME_ARGS

DICT_NOT_IN: 60670 times
DICT2_NOT_IN: 12134 times
CONVERSION_ERROR: 36402 times
TOTAL_FUNCTION: 60670 times
NOT_RESULT: 12134 times
SAME_ARGS: 12134 times


Alright I will change it to ULONG as well. There are still conversion errors tho, does this mean its actually the same.. as pre, performance wise?
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: conversion error on index_from_pointer

Postby L'In20Cible » Fri Nov 26, 2021 9:51 pm

velocity wrote:does this mean its actually the same.. as pre, performance wise?

It means you process at least ~20% less calls than you previously did, which is still a good start I'd say. Though, it really doesn't seem to be the right approach overall for what your goal actually is.
User avatar
velocity
Senior Member
Posts: 220
Joined: Sat May 10, 2014 6:17 pm

Re: conversion error on index_from_pointer

Postby velocity » Fri Nov 26, 2021 9:57 pm

L'In20Cible wrote:Though, it really doesn't seem to be the right approach overall for what your goal actually is.
What can I do instead :confused:
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: conversion error on index_from_pointer

Postby L'In20Cible » Sat Nov 27, 2021 1:06 am

velocity wrote:
L'In20Cible wrote:Though, it really doesn't seem to be the right approach overall for what your goal actually is.
What can I do instead :confused:

Well, I guess it depend of which entity but surely you can hook more targeted than that function. Do you have a map (preferably for CS:S) that have such entity?
User avatar
velocity
Senior Member
Posts: 220
Joined: Sat May 10, 2014 6:17 pm

Re: conversion error on index_from_pointer

Postby velocity » Sat Nov 27, 2021 10:18 am

If you want to try something, this is a good map: https://gamebanana.com/mods/127520 for it, or https://gamebanana.com/mods/127438. I'm not sure how you want to make it more targeted, but please note it is not just player-entity collision, it's also player-player and player-flashbang collisions, etc. This goes for SetTransmit as well.

I also want to point out that there are no issues with this in sourcemod, you are welcome to try out the plugin I linked and see for yourself and compare lag or other stuff if you are up for it. Like in Sourcepython everything works smoothly until ~6 players and this is after super careful considerations for optimization and smart tricks to make it as fast as possible (that I'm capable of doing), then server var starts going crazy. Comparably in sourcemod, you can have 20 players or more, no problem, with much less respect for optimizations. Obviously it totally depends on the use case, and what you are trying to do, but usually stuff like this is common for me.

PS: I remember a long time ago, I made a post about performance problems here as I couldn't understand why my plugin after converting from ES to SourcePython lagged on my server, it was literally 1:1 conversion, yes it was HORRIBLY optimized, but it still ran without lagging with ES compared to SP. I know that these general statements do not help to improve SP, I just want to point out, these are some of the frustrations that one might have while coding in SP, but SP has come a long way since then, I think.

Hopefully, the lag is less present now after the optimizations we have discussed, I haven't tested its significance yet, but if you can come up with a more targeted solution that would be epic.

Edit: I think this was the old post: viewtopic.php?f=10&t=1902&p=12225&hilit=performance+problems&sid=a4b47dc0218f095c9670364173062d09#p12225 (...not to start an off-topic conversation)
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: conversion error on index_from_pointer

Postby L'In20Cible » Sun Nov 28, 2021 7:22 am

Here's a quick attempt at something: https://github.com/Source-Python-Dev-Te ... collisions

Example:

Syntax: Select all

from listeners import OnEntityCollision

@OnEntityCollision
def on_entity_collision(entity, other):
"""Called when two entities are about to collide."""
if not entity.is_player() or not other.is_player():
return

return entity.team_index != other.team_index


@OnEntityCollision
def on_entity_collision(entity, other):
"""Called when two entities are about to collide."""
return not (entity.is_player() and
other.classname in ('prop_door_rotating', 'func_breakable')
)

Test builds (CS:S/CS:GO/Windows): https://drive.google.com/drive/folders/ ... JkVPWFe3i1
Jezza
Junior Member
Posts: 16
Joined: Tue Aug 28, 2012 5:52 pm

Re: conversion error on index_from_pointer

Postby Jezza » Sun Nov 28, 2021 9:17 am

L'In20Cible wrote:Here's a quick attempt at something: https://github.com/Source-Python-Dev-Te ... llisions.h

Does this disable collision? It doesn't seem to do anything. (CS:GO/Linux&Windows)
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: conversion error on index_from_pointer

Postby L'In20Cible » Sun Nov 28, 2021 9:32 am

Jezza wrote:
L'In20Cible wrote:Here's a quick attempt at something: https://github.com/Source-Python-Dev-Te ... llisions.h

Does this disable collision? It doesn't seem to do anything. (CS:GO/Linux&Windows)

I only tested on CS:S, and built on CS:GO, so it is possible there is some differential factors to take into consideration. Though, it doesn't override VPhysics collisions, but that example above is disabling teammates, rotating doors, and breakables collision for players. The overall idea is that we only have a single hook with 2 callbacks (pre and post) and use direct delegation to test entities rather than a noisy dynamic hook in between.
User avatar
velocity
Senior Member
Posts: 220
Joined: Sat May 10, 2014 6:17 pm

Re: conversion error on index_from_pointer

Postby velocity » Sun Nov 28, 2021 9:35 am

Nice, can't wait to try out, can you make a build for Linux/CS:GO?
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: conversion error on index_from_pointer

Postby L'In20Cible » Sun Nov 28, 2021 9:47 am

velocity wrote:Nice, can't wait to try out, can you make a build for Linux/CS:GO?

Jezza can probably provide you with a build. If that was for CS:S, I could fire up my VM and build it because it is up-to-date, but I'm too lazy to maintain my CS:GO (or any other games for that matter) environment which is constantly outdated and/or have expired token. :rolleyes:
Jezza
Junior Member
Posts: 16
Joined: Tue Aug 28, 2012 5:52 pm

Re: conversion error on index_from_pointer

Postby Jezza » Sun Nov 28, 2021 12:26 pm

velocity wrote:Nice, can't wait to try out, can you make a build for Linux/CS:GO?

Currently, it does not work with CS:GO. You need to wait for an update from L'In20Cible.
Jezza
Junior Member
Posts: 16
Joined: Tue Aug 28, 2012 5:52 pm

Re: conversion error on index_from_pointer

Postby Jezza » Sun Nov 28, 2021 1:45 pm

What you really need for your PassServerEntityFilter is a lookup table.

Syntax: Select all

# Python Imports
# Collections
from collections import defaultdict

# Source.Python Imports
# Entities
from entities.helpers import index_from_inthandle
# Listeners
from listeners import OnEntityDeleted
from listeners import OnNetworkedEntitySpawned
# Memory
from memory import Convention
from memory import DataType
from memory import find_binary
from memory.hooks import PreHook
# Players
from players.entity import Player


# =============================================================================
# >> CLASSES
# =============================================================================
class PairPassFilter(dict):
_entity_ids = defaultdict(list)

def add_filter(self, touch_entity, pass_entity, filter=True):
touch_address = touch_entity.pointer.address
pass_address = pass_entity.pointer.address

id = touch_address ^ pass_address
if id not in self:
self._entity_ids[touch_address].append(id)
self._entity_ids[pass_address].append(id)

self[id] = filter

def remove_filter(self, touch_entity, pass_entity):
touch_address = touch_entity.pointer.address
pass_address = pass_entity.pointer.address

id = touch_address ^ pass_address
del self[id]

self._entity_ids[touch_address].remove(id)
self._entity_ids[pass_address].remove(id)

def remove_entity(self, base_entity):
ids = self._entity_ids.pop(base_entity.pointer.address, None)
if ids is not None:
for id in ids:
pass_filter.pop(id, None)


class PairPlayer(Player):

def __init__(self, index, caching=True):
"""Initializes the object."""
super().__init__(index, caching)

self.pair_player = None

def add_pass_filter(self, base_entity, filter=True):
pass_filter.add_filter(self, base_entity, filter)

def remove_pass_filter(self, base_entity):
pass_filter.remove_filter(self, base_entity)

def make_pair(self, other_index):
pair_player = self.pair_player
if pair_player is not None:
self.remove_pass_filter(pair_player)

self.pair_player = PairPlayer(other_index)
self.pair_player.pair_player = self
self.add_pass_filter(self.pair_player)

def set_pair_filter(self, base_entity):
self.add_pass_filter(base_entity)
pair_player = self.pair_player
if pair_player is not None:
pair_player.add_pass_filter(base_entity)


pass_filter = PairPassFilter()


# =============================================================================
# >> CALLBACKS
# =============================================================================
@OnEntityDeleted
def on_entity_deleted(base_entity):
pass_filter.remove_entity(base_entity)


@OnNetworkedEntitySpawned
def on_networked_entity_spawned(entity):
try:
pair_player = PairPlayer(index_from_inthandle(entity.owner_handle))
except (ValueError, OverflowError):
return

pair_player.set_pair_filter(entity)


server = find_binary("server", srv_check=False)

PassServerEntityFilter = server[b"\x55\xB8\x01\x00\x00\x00\x89\xE5\x83\xEC\x38\x89\x5D\xF4"].make_function(
Convention.CDECL, (
DataType.ULONG,
DataType.ULONG,
),
DataType.BOOL,
)

@PreHook(PassServerEntityFilter)
def pre_pass_server_entity_filter(args):
touch_entity = args[0]
pass_entity = args[1]

if not pass_entity or touch_entity == pass_entity:
return

filter = pass_filter.get(touch_entity^pass_entity, None)
if filter is not None:
return filter

return False

Syntax: Select all

PairPlayer(1).make_pair(2)
will enable collisions on player1 and player2.

You can also use

Syntax: Select all

pair_player.set_pair_filter(entity)
to apply the entity's collision to the pair players.
In this demo, the entities owned by the pair are automatically added to the path filter table.

As a personal opinion, I would suggest implementing such a lookup table on the C++ side, but I honestly don't know if this will happen due to disagreements like the one in #332.
User avatar
velocity
Senior Member
Posts: 220
Joined: Sat May 10, 2014 6:17 pm

Re: conversion error on index_from_pointer

Postby velocity » Sun Nov 28, 2021 2:14 pm

Jezza wrote:What you really need for your PassServerEntityFilter is a lookup table.
Yes definitely would help, but I dont see how this any different from what was already suggested by L'In20Cible?
Jezza
Junior Member
Posts: 16
Joined: Tue Aug 28, 2012 5:52 pm

Re: conversion error on index_from_pointer

Postby Jezza » Sun Nov 28, 2021 2:38 pm

velocity wrote:
Jezza wrote:What you really need for your PassServerEntityFilter is a lookup table.
Yes definitely would help, but I dont see how this any different from what was already suggested by L'In20Cible?

That will depend on the results of your tests, because we do not know the performance status of your server.

If L'In20Cible's on_entity_collision can work in CS:GO, then perhaps we can reduce the number of calls to 1/4, but if it still doesn't improve the performance using such a lookup table on the Python side, then I guess the only way is to implement the lookup table on the C++ side. However, it is not up to me to decide that.
User avatar
velocity
Senior Member
Posts: 220
Joined: Sat May 10, 2014 6:17 pm

Re: conversion error on index_from_pointer

Postby velocity » Sun Nov 28, 2021 2:50 pm

Yes with the on_entity_collision, you are right, but in his previous posts on the first page of this thread, he already suggested a lookup table, that is what I was referring to. Good idea with this line tho

Syntax: Select all

if not pass_entity # aka if not args[1]
as I didn't know you could do boolean checks for this on "invalid addresses". We also reached the conclusion previously that a PostHook would be more efficient than a PreHook in this case.

Originally my post was how to handle conversion error without try/catch and that line actually concludes that.. Don't know why it wasn't mentioned, right away, but I guess the problem is more deep-rooted than this as it doesn't fix the underlying issue with python performance. I think most stuff should be implemented on the C++ side ( or at minimum demanding functions ), I don't even want to think about the overhead that comes with Boost Python with general mixing of loops between C++/Python.
Jezza
Junior Member
Posts: 16
Joined: Tue Aug 28, 2012 5:52 pm

Re: conversion error on index_from_pointer

Postby Jezza » Sun Nov 28, 2021 6:40 pm

velocity wrote:Yes with the on_entity_collision, you are right, but in his previous posts on the first page of this thread, he already suggested a lookup table, that is what I was referring to. Good idea with this line tho

Syntax: Select all

if not pass_entity # aka if not args[1]
as I didn't know you could do boolean checks for this on "invalid addresses". We also reached the conclusion previously that a PostHook would be more efficient than a PreHook in this case.

You may be able to gain 7% to 13% by using PostHook, but that is not the point. What's important is that I (and probably L'In20Cible) thinks that using a lookup table can solve the performance issue with your PassServerEntityFilter. In fact, I did not feel any lag in 32bot+1player, de_dust2. And since everyone's system is different, I can't predict your performance. Please try the code first and report back how much it affects performance.

I also did a quick measurement of the amount of PassServerEntityFilter calls (32bots + 1player), and it seems that on average the PassServerEntityFilter is called 250 times per tick.

Code: Select all

tick: 30000
func_call: 7591605
prefalse: 3740712
postfalse: 3466442

func_call/tick: 253.0535
prefalse/tick: 124.6904
postfalse/tick: 115.54806666666667

This is not a small number, but if you are just processing lookup tables, it should not become a problem.

Syntax: Select all

from ctypes import memmove
from ctypes import c_void_p
from time import perf_counter

from memory import Convention
from memory import DataType
from memory import alloc
from memory.hooks import PreHook

no_op = b"\x90\x90\x90\x90\x90\x90\xC3"
func_caller = alloc(len(no_op), auto_dealloc=False)
func_caller.unprotect(len(no_op))

memmove(c_void_p(func_caller.address), no_op, len(no_op))

func_caller_function = func_caller.make_function(
Convention.CDECL, (
DataType.ULONG,
DataType.ULONG,
),
DataType.BOOL,
)

@PreHook(func_caller_function)
def pre_func_caller_function(args):
touch_entity = args[0]
pass_entity = args[1]

if not pass_entity or touch_entity == pass_entity:
return False

filter = pass_filter.get(touch_entity^pass_entity, None)
if filter is not None:
return filter

return False

pass_filter[1^2] = True

s = perf_counter()
for i in range(125):
func_caller_function(1, 2)
func_caller_function(1, 1)
e = perf_counter()
print("time", e-s)

Code: Select all

time 0.0005662888288497925

velocity wrote:Originally my post was how to handle conversion error without try/catch and that line actually concludes that.. Don't know why it wasn't mentioned, right away, but I guess the problem is more deep-rooted than this as it doesn't fix the underlying issue with python performance. I think most stuff should be implemented on the C++ side ( or at minimum demanding functions ), I don't even want to think about the overhead that comes with Boost Python with general mixing of loops between C++/Python.

That's because it's unclear what you want to do with the PassServerEntityFilter. For example, the reason this is fast is because it's practically just doing a lookup. But if you want to use Entity, Player, etc. to do other processing, please provide that code.

And about try/catch, I don't know much about the entities passed to the PassServerEntityFilter, so I can't say for sure, but if the PassServerEntityFilter doesn't receive non-networked entities, then just check

Syntax: Select all

if not pass_entity
with a DataType.POINTER and you won't get the Conversion Error.
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: conversion error on index_from_pointer

Postby L'In20Cible » Mon Nov 29, 2021 9:15 am

Latest commits works for me on CS:S and CS:GO, and also add support for CollisionHash. For example:

Syntax: Select all

from entities.collisions import CollisionHash
from events import Event
from players.entity import Player

collisions = CollisionHash()

@Event('player_say')
def player_say(game_event):
player = Player.from_userid(game_event['userid'])
entity = player.view_entity
if entity is None:
return
if collisions.has_pair(player, entity):
collisions.remove_pair(player, entity)
else:
collisions.add_pair(player, entity)
User avatar
velocity
Senior Member
Posts: 220
Joined: Sat May 10, 2014 6:17 pm

Re: conversion error on index_from_pointer

Postby velocity » Mon Nov 29, 2021 4:32 pm

Jezza wrote:
velocity wrote:That's because it's unclear what you want to do with the PassServerEntityFilter. For example, the reason this is fast is because it's practically just doing a lookup. But if you want to use Entity, Player, etc. to do other processing, please provide that code.


I'm not doing any other processing, other than making checks on who should collide with who, just like in SourceMod script. I have learned from past mistakes, that game-oriented attributes like player.dead could overflow the var. I'm doing less than the script, because right now I'm trying to isolate the lag, so it's just two EntityDictionary lookups for address1 and address2. What I might have failed to mention, but is evident, is that speeding up functions like this helps compensate the additional processing of SetTransmit, which is also required to make it work fully, which can also be seen in the .sp. On top of this, not to mention OnPlayerRunCmd for 'other server stuff'. So, I think it is only an advantage that the hook of these functions is as optimized as possible.

Jezza wrote:I also did a quick measurement of the amount of PassServerEntityFilter calls (32bots + 1player), and it seems that on average the PassServerEntityFilter is called 250 times per tick.


I have just implemented the new optimizations L'In20Cible combined with urs (Address -> Entity) etc... So I cannot confirm what I'm about to say: I tested with 20 bots and var didn't seem that off, but I have my suspicion regarding the actual impact of bots based solely on their movement. For example, if you stand still the PassServerEntityFilter almost doesn't fire at all, after 15 seconds it's like 100 calls, but if you move around the calls are much more frequent. On top of that, if two players are 'grinding' each other, or when an entity is colliding with another entity is where the function gets an insane amount of calls. So I would like to test out with real players, as well, before concluding (just hard to get 20 players xD).

L'In20Cible wrote:Latest commit works for me on CS:S and CS:GO, and also add support for CollisionHash.

Nice, what is CollisionHash? Is CollisionHash Jezzas py implementation in c++ for faster look up when using OnEntityCollision?

I really want to try these things out! I have a small request, if possible, can your branch be merged with Cookstars branch implementation of(don't know if that's you Jezza) SetTransmit. So I can try them combined and see if the server can handle it? I know there are disagreements with the implementation of SetTransmit, but either solution is better than hooking SetTransmit with EntityCondition, at the moment. Also a build for Linux/CS:GO.

This thread got longer than expected, and I appreciate the time and effort, so I would like to take a moment and thank you guys: L'In20Cible, Jezza, and Ayuto for your suggestions and general insights for speed optimizations etc.

On the occasion of this, I want to shine some light on this post again: viewtopic.php?f=9&t=2036&p=12854&hilit=donation&sid=7857ff58a2aedd0f39f20ee65a3d50c8#p12854 :embarrassed:
Jezza
Junior Member
Posts: 16
Joined: Tue Aug 28, 2012 5:52 pm

Re: conversion error on index_from_pointer

Postby Jezza » Tue Nov 30, 2021 2:14 am

L'In20Cible wrote:Latest commit works for me on CS:S and CS:GO, and also add support for CollisionHash.
Excellent work!

However, there are two problems, one is that the global CollisionHash instance is not being destroyed properly when the plugin is unloaded. I'm not sure why, but it seems to be a problem with gc.collect. This applies to other objects as well, but it becomes a problem if the CollisionHash instance continues to remain with collision pair. (Tested on CS:GO/Linux)

The second, which is more of a feature extension than a problem, is that CollisionHash does not have a collision direction. Although it is not necessary for the problem that velocity needs, PassServerEntityFilter is divided into touching and passing, and it can decide collision on either side, but CollisionHash cannot.

Syntax: Select all

@OnEntityCollision
def on_entity_collision(entity, other):
if not entity.is_player() or not other.is_player():
return

if entity.team_index == 3 and other.team_index == 2:
return False
For example, this script allows only CT players to pass through other T players, but CollisionHash is not able to achieve this.
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: conversion error on index_from_pointer

Postby L'In20Cible » Tue Nov 30, 2021 8:45 am

velocity wrote:but I have my suspicion regarding the actual impact of bots based solely on their movement
Bots are indeed extremely noisy because they scan for targets, paths, etc. I've added some exclusions to OnEntityCollision to filter out contents we don't really care about in the context of collisions (visibility, attacks, audition, etc.).

On CS:S with 32 bots for 10,000 ticks:

Syntax: Select all

Before:
Total calls: 1896297
Average calls per frame: 189.6297

After:
Total calls: 28858
Average calls per frame: 2.8858


On CS:GO with 20 bots for 10,000 ticks:

Syntax: Select all

Before:
Total calls: 2791137
Average calls per frame: 279.1137

After:
Total calls: 36250
Average calls per frame: 3.625


velocity wrote:Nice, what is CollisionHash? Is CollisionHash Jezzas py implementation in c++ for faster look up when using OnEntityCollision?
It is a wrapped IPhysicsObjectPairHash. The physics engine uses that to filter collisions through collision events. That class is independent of OnEntityCollision, and can be used independently. Just fill your pairs in and they won't collide, or you can use the listener if you have dynamic conditions to test for.

Jezza wrote:However, there are two problems, one is that the global CollisionHash instance is not being destroyed properly when the plugin is unloaded. I'm not sure why, but it seems to be a problem with gc.collect. This applies to other objects as well, but it becomes a problem if the CollisionHash instance continues to remain with collision pair. (Tested on CS:GO/Linux)
I was not able to reproduce:

Syntax: Select all

# ../testing/testing.py

from entities.collisions import CollisionHash
from listeners import OnEntityCollision

collisions = CollisionHash()

@OnEntityCollision
def on_entity_collision(entity, other):
...


Syntax: Select all

sp plugin load testing;sp plugin unload testing
[SP] Loading plugin 'testing'...
CCollisionHash::CCollisionHash
CCollisionManager::Initialize
CEntityCollisionListenerManager::Initialize
[SP] Successfully loaded plugin 'testing'.
[SP] Unloading plugin 'testing'...
CEntityCollisionListenerManager::Finalize
CCollisionHash::~CCollisionHash
CCollisionManager::Finalize
[SP] Successfully unloaded plugin 'testing'.


But I went ahead and added AutoUnload support to it just in case.

Jezza wrote:The second, which is more of a feature extension than a problem, is that CollisionHash does not have a collision direction. Although it is not necessary for the problem that velocity needs, PassServerEntityFilter is divided into touching and passing, and it can decide collision on either side, but CollisionHash cannot.
Perhaps we could add a separate class for that, but as far as the current class is concerned, I think we should leave it as is because it is consistent with the engine. Adding a separate class seems easy enough if we use a shared abstract base class.

Return to “Plugin Development Support”

Who is online

Users browsing this forum: No registered users and 22 guests