What is the most reasonable way to install a pypi package to source.python?

[7ember]

While developing my plugin on windows, if i remember correctly i just installed 32bit python 3.6.1 and installed the package with pip and moved it to addons/source-python/packages/site-packages.

Whatever I did, it worked flawlessly.

However, I've been struggling endlessly to achieve this on debian 9, and clearly I'm in way over my head as I've spent a seriously embarrassingly long time trying to get it working...

Is there an easy way to do this that I'm missing? I saw a post from 2014 saying that pip would be included "in the future", what happened with that? I can't find any documentation of installing stuff from a requirements.txt or anything of the sort.

[7ember]

everything in this comment is specific to my case and i dont expect anyone to help but i would really appreciate it if someone could try

The package i'm trying to install is psycopg2, like i said it worked perfectly on windows and i wrote a shit load of queries with it so i'd really like to get it working.

i have tried:
compiling 32bit python 3.6.1 from source, this doesnt work properly on my server at least not with any configurations ive tried. it can compile but has many errors eg with blake2 not working for whatever reason no matter what thus breaking hashlib. i don't think that even matters, but it obviously isn't compiling correctly. i've tried many many many times with many different configurations working through errors just to eventually get it to compile and always the same result, and i mean i've spent many hours. i thought i knew what i was doing, but i obviously have no idea.

use pyenv to install 32bit python 3.6.1, this ends up exactly the same way.

installing from a 32bit chroot, i dont even remember wht was wrong with this but it was screwed up too, and i figure it wasnt going to work in the end and give up.

installing from a 32bit vm , obv this didn't work either.

from all of these i have tried installing psycopg2 and copying it to the sourcepython packages dir. i did this from pypi psycopg2, psycopg2-binary, and compiling it myself. and all the different combinations of these things have different errors varying from import and library errors to segfaults.
i should mention psycopg2 works whereever i install it , only breaks when i move it to source-python. what does this mean?

so to summarize, i get errors when compiling python. i get errors when i get python to compile and run it. i get errors when i compile python in a chroot. and the reason i try all of these, is because i get errors when i run the psycopg2 from any of them on source python.

i've googled like mad trying to get this to work. i'm feeling very retarded, because it's such a simple thing. just installing a package. i know there is a very very simple solution, but obviously i can't find it on my own. i have really tried.

please help its been so long im on stage 4 of grief by now

[khaimovmr]

Have you tried to install the psycopg2 to the <csgo-install-dir>/csgo/addons/source-python/packages/site-packages?

Code: Select all

pip install psycopg2 -t $CSGO_INSTALL_DIR/csgo/addons/source-python/packages/site-packages

PS: swap of course $CSGO_INSTALL_DIR with the full csgo installation directory.

[7ember]

Have you tried to install the psycopg2 to the <csgo-install-dir>/csgo/addons/source-python/packages/site-packages?

Code: Select all

pip install psycopg2 -t $CSGO_INSTALL_DIR/csgo/addons/source-python/packages/site-packages

PS: swap of course $CSGO_INSTALL_DIR with the full csgo installation directory.

Hi, thanks for the reply. I'm using CSS, but yes I tried that from all the various applicable installations as well.

[L'In20Cible]

From their documentation, you need to install psycopg2-binary if you want a stand alone package which is what you need for Source.Python since the interpreter is embedded.

[7ember]

From their documentation, you need to install psycopg2-binary if you want a stand alone package which is what you need for Source.Python since the interpreter is embedded.

Hi, I have tried that one, and it gives a seg fault if I remember correctly. There is an issue related to some clashing between libraries, shown here in red http://initd.org/psycopg/docs/install.h ... -from-pypi could this be the cause? I'm unfamiliar with how embedded python works. If so, do you know how I could fix it?

Edit: I might be remembering wrong which versions gave segfault, I'm going to test everything again to be sure

Edit2: sorry for long delay, i completely reinstalled the css server and sourcepython to test from scratch.
in a plugin "test" i have the file test.py containing

Syntax: Select all

import psycopg2

print('a')

this results in

Code: Select all

sp plugin load test
[SP] Loading plugin 'test'...
a
[SP] Successfully loaded plugin 'test'.

so importing is working, however...
when test.py is modified to look this way

Syntax: Select all

import psycopg2
from .config import dsn


print('a')

db = psycopg2.connect(dsn)

print('b')

it results in segfault, shown here

Code: Select all

sp plugin load test
[SP] Loading plugin 'test'...
a
Uploading dump (in-process) [proxy '']
/tmp/dumps/assert_20190909133408_1.dmp
success = yes
response:  Discarded=1
Segmentation fault (core dumped)
BFD: Warning: /home/steam/Steam/css_test/core is truncated: expected core file size >= 254947328, found: 1245184.
Cannot access memory at address 0xf772f908
Cannot access memory at address 0xf772f904
Failed to read a valid object file image from memory.
Cannot access memory at address 0xf772f908
Cannot access memory at address 0xf772f904
debug.cmds:5: Error in sourced command file:
No function contains program counter for selected frame.
email debug.log to linux@valvesoftware.com
Mon Sep  9 13:34:15 EDT 2019: Server restart in 10 seconds

so segfault is happening on .connect() being called, which is i think is documented error for psycopg, soo can anyone recommend a good way to fix this? do i need to isolate the css server?


oh yeah, i forgot to mention this command works just fine from the python i installed the package from. it's only in source-python that it crashes. (i installed with pip directly and also to server with python3.6 -m pip install psycopg2-binary -t cstrike/addons/source-python/packages/site-packages/)

[L'In20Cible]

The issue about the warning you referenced mention to use:

Code: Select all

pip install --no-binary psycopg2

Tried with that option already?

[7ember]

The issue about the warning you referenced mention to use:

Code: Select all

pip install --no-binary psycopg2

Tried with that option already?

That's actually for an older version, they don't package it with binaries by default anymore so that command is the same as

Code: Select all

pip install psycopg2

on the current version, at least that's how i understand it. And i have tried that. Correct me if I'm wrong, I'm at a Dr appointment atm so i will try when i get home if so. Btw, we have a similar pun in the name.

[7ember]

just discovered i can color my text. cute.

using the 'psycopg2' package results in

Code: Select all

sp plugin load test
[SP] Loading plugin 'test'...

[SP] Caught an Exception:
Traceback (most recent call last):
  File "../addons/source-python/packages/source-python/plugins/command.py", line 162, in load_plugin
    plugin = self.manager.load(plugin_name)
  File "../addons/source-python/packages/source-python/plugins/manager.py", line 194, in load
    plugin._load()
  File "../addons/source-python/packages/source-python/plugins/instance.py", line 74, in _load
    self.module = import_module(self.import_name)
  File "../addons/source-python/plugins/test/test.py", line 1, in <module>
    import psycopg2
  File "../addons/source-python/packages/site-packages/psycopg2/__init__.py", line 50, in <module>
    from psycopg2._psycopg import (                     # noqa

ModuleNotFoundError: No module named 'psycopg2._psycopg'


[SP] Plugin 'test' was unable to be loaded.

i'm guessing this is because installing this package seems to compile a shared object named

Code: Select all

_psycopg.cpython-36m-x86_64-linux-gnu.so

and after re-testing it turns out the "psycopg2-binary" package installs a shared object named

Code: Select all

_psycopg.cpython-36m-i386-linux-gnu.so

very.
interesting.

Code: Select all

steam@vps:~/Steam/css_test$ python3.6
Python 3.6.1 (default, Sep  9 2019, 10:05:01)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import struct
>>> print('{}, {}'.format(sys.maxsize, struct.calcsize('P') * 8))
2147483647, 32
>>>

this is the python i'm using to install the package... am i missing something?
this is made even more confusing by the fact that installing the "psycopg2-binary" package (the one with the i386 .so) to this python causes the same ModuleNotFoundError shown above on import, implying that python is in fact 64 bit. (this is contrary to my previous post, i thought i had "psycopg2-binary" installed on my python, but i actually had "psycopg2" (the one that compiles the x86_64 .so) which DOES work on the python repl)

i still think the segfault when the "psycopg2-binary" module is used by source-python is caused by the issue shown on github, since this IS the binary version of the package... i just dont know how the hell to compile it from source now, lol.

sorry guys, i know this is off-topic at this point because something is clearly just very wrong with my system, i just don't knw how to fix it.

in summary: i have a python 3.6.1 with what seems like 32bit memory constraints, that somehow compiles and runs a 64 bit package. but when i install the pre-compiled version of that package, it installs the i386 version, implying it thinks it's 32 bit, but this package fails to import. wtf? and yes i have checked, everything is installing where it should be, and the python i'm running is the correct (and only) 3.6.1 executable, and it's using the correct pip. i've also used a venv to install the packages, to the exact same effect. if anyone can solve this or knows where i should look to try to solve it myself please let me know, i'm very lost

for posterity

Syntax: Select all

steam@vps:~/.pyenv/versions/3.6.1$ ls lib/python3.6/site-packages/psycopg2
compat.py  errorcodes.py  errors.py  extensions.py  extras.py  __init__.py  _ipaddress.py  _json.py  _lru_cache.py  pool.py  _psycopg.cpython-36m-x86_64-linux-gnu.so  __pycache__  _range.py  sql.py  tz.py
steam@vps:~/.pyenv/versions/3.6.1$ bin/python3.6
Python 3.6.1 (default, Sep  9 2019, 10:05:01) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import struct
>>> print('{}, {}'.format(sys.maxsize, struct.calcsize('P') * 8))
2147483647, 32
>>> import psycopg2
>>> print('still working, imported correctly')
still working, imported correctly
>>> import os
>>> print(os.path.relpath(psycopg2.__file__)
... )
lib/python3.6/site-packages/psycopg2/__init__.py
>>>

[L'In20Cible]

That's actually for an older version, they don't package it with binaries by default anymore so that command is the same as

Code: Select all

pip install psycopg2

on the current version, at least that's how i understand it. And i have tried that. Correct me if I'm wrong, I'm at a Dr appointment atm so i will try when i get home if so.

I could not tell, sorry. Never used that package before and really just tried to throw idea based on what I was able to find.

Btw, we have a similar pun in the name.

Hehe yeah, I noticed. Do you also speak french?

[color=#FBA0E3]in summary: i have a python 3.6.1 with what seems like 32bit memory constraints, that somehow compiles and runs a 64 bit package. but when i install the pre-compiled version of that package, it installs the i386 version, implying it thinks it's 32 bit, but this package fails to import. wtf? and yes i have checked, everything is installing where it should be, and the python i'm running is the correct (and only) 3.6.1 executable, and it's using the correct pip. i've also used a venv to install the packages, to the exact same effect. if anyone can solve this or knows where i should look to try to solve it myself please let me know, i'm very lost

For it to work with SP, it would need to be the 32 bits version. Probably why you get an import error if you try to use the 64 bits version.

[7ember]

So, i've tested now as isolated as I possibly can. I'm running a debian 9 (stretch/oldstable) VM, same version as my server, except it's 32bit instead of 64bit.
EDIT: obviously this is not isolated in terms of the libraries (this issue), i couldn't think of a better words, sorry.. i haven't tried containers or chroot jail yet, it's just a different (fresh) system. i guess i will try those now END EDIT

Psycopg2 and psycopg2-binary both run correctly when installed on the system's python, but both segfault when installed on source-python. Keep in mind this doesn't happen on my windows installation, so something is wrong with linux specifically here.

Here are two screenlogs showing exactly what I mean.



test.py contains this

Syntax: Select all

import psycopg2
from config import dsn

print('a')

asd = psycopg2.connect(dsn)

print('b')

c = asd.cursor()

print('c')

c.execute('SELECT * FROM cstrike.users;')

print('d')

print(c.fetchall()[-1])

print('e')

Would this be a python issue, a source-python issue, a psycopg2 issue, or a me issue? I've never submitted an issue on github or anything before because I'm not confident enough in my knowledge to know the cause of errors.. anyways

Thanks.

Btw, we have a similar pun in the name.

Hehe yeah, I noticed. Do you also speak french?

Not really lol, I spent ~2 months learning it hardcore then stopped. That was over a year ago and I haven't used it since, so I've forgotten a lot. I have a lot of interest in other languages and cultures though and really enjoy learning them / about them so I'd love to learn french and many others and be a super polyglot, but honestly I already have too many very time consuming hobbies atm lol, hopefully i can get back to it in the future.

[7ember]

Well, it segfaults in a completely bare lxc. I officially give up

[khaimovmr]

As a workaround you could make a middleware REST service to store and retrieve the data - there, as in the standalone python service you can use psycopg2 as well.

[Ayuto]

You could also try to start your server with -debug. This should generate a debug.log for most games in the game server directory. It should provide some more information.

[7ember]

sorry i forgot to update the thread, but i just ended up changing libraries. and i did run with -debug, here the only log i kept. sorry i don't remember which package was installed for each of these

if anyone is searching for postgres postgresql psycopg2 etc... i switched to pg8000 and it's working well, it's easy enough to switch since they're very similar. differences to note from my experience are that it's thread safety level 1 instead of 2, it's pure python so doesn't use libpq, it has a different parameter style by default, and it uses prepared statements for all queries.

[7ember]

also as an additional point to the actual title of the thread, it would be nice to have a plugin requirements type deal... i'm sure you guys know that though. i don't know anything about how hard it is to implement something like that. you guys have done a ton of work for this mod for free and it's really nice to use (except the lacking docs, but thats a problem with all mods i think lol), so thank u for that.