Source.Python forums

Posted: **Thu Apr 30, 2020 3:31 am**

Hello,

I'm currently trying to write data to a file and I heard you can save alot in file size by saving it in bytes. (Also to reduce lag on saving the file)

I'm currently using this for testing:

Syntax: Select all

from time import time
from paths import GAME_PATH

file_path = GAME_PATH / 'test.txt'

timestamp = time()

with open(file_path, 'wb') as file_stream:
    test = ''

    for i in range(100000):
        test += '0|0|0|0|0\n'

    file_stream.write(test.encode())

print(time() - timestamp)

When I do this, its still written in plain text and changes barely in size. Can you help and explain please?

Also is this the best way to save a file without having trouble with lag? Better to write everything at once, or write in several parts?

Posted: **Thu Apr 30, 2020 3:58 am**

Hi, can't help you with the byte thing, but if you wish to avoid getting the stutter/freeze when loading or saving data, you can use GameThread:

Syntax: Select all

# ../gamethread/gamethread.py

# Python
from time import time

# Source.Python
from listeners.tick import GameThread
from paths import GAME_PATH


file_path = GAME_PATH / 'test.txt'
timestamp = time()


def save_to_file():
    with open(file_path, 'wb') as file_stream:
        test = ''

        for i in range(100000):
            test += '0|0|0|0|0\n'

        file_stream.write(test.encode())

    print(time() - timestamp)


thread = GameThread(target=save_to_file)
thread.start()

Posted: **Thu Apr 30, 2020 2:20 pm**

This part is very inefficient:

Syntax: Select all

test += '0|0|0|0|0\n'

If you want to write million times the same thing, then write the same thing million times.

Syntax: Select all

import io
import timeit

# Scenario 1
def test1(file_stream):
    test = ''
    for i in range(1000):
        test += '0|0|0|0|0\n'
        file_stream.write(test.encode())

# 1.
# "" + "0|0|0|0|0\n"
# 2.
# "0|0|0|0|0\n" + "0|0|0|0|0\n"
# 3.
# "0|0|0|0|0\n0|0|0|0|0\n" + "0|0|0|0|0\n"
# 4.
# "0|0|0|0|0\n0|0|0|0|0\n0|0|0|0|0\n" + "0|0|0|0|0\n"
# ...


# Scenario 2
def test2(file_stream):
    test = '0|0|0|0|0\n'
    for _ in range(1000):
        file_stream.write(test.encode())

# calling 1000 times encode() and write()
# test is not modified


print("Test1", end="")
t1 = timeit.timeit("test(stream)", setup="stream = io.BytesIO()", number=1000, globals={"io": io, "test": test1})
print(f" {t1:.3f}s")


print("Test2", end="")
t2 = timeit.timeit("test(stream)", setup="stream = io.BytesIO()", number=1000, globals={"io": io, "test": test2})
print(f" {t2:.3f}s")

Code: Select all

PS C:\Users\Admin\Desktop\data> py -3 .\speed.py
Test1 20.785s
Test2 0.148s

Writing incremental in chunks is cheaper than concatenate a string over and over.

Posted: **Thu Apr 30, 2020 3:25 pm**

@VinciT Thanks, I totally forgot about it.

@DeaD_EyE Thanks for the help so far. I Will write in chunks then.

Do you maybe know how to save the data in binary code aswell? (Also decoding it for loading it)

I saw many sourcemod user use https://sm.alliedmods.net/new-api/files/File/Write which automatically saves input in binary.

Posted: **Thu Apr 30, 2020 8:30 pm**

You already write the data as bytes. I'm not sure which output you expect? The modes "w" and "wb" only affect the lines endings (\n or \n\r). If you open a file in "w" mode, these characters are converted to the platform dependend line ending. If you open a file in "wb" mode, your text/bytes are written without any changes.

See also:
https://docs.python.org/3/tutorial/inpu ... ting-files

Posted: **Mon May 04, 2020 2:08 pm**

If you want to write structured binary data, you have to define the struct of your data.
Python has a module for this: https://docs.python.org/3/library/struct.html#module-struct

If you want to save for example 2 * int64 and 2 * float64 as little endian:

Syntax: Select all

import struct

my_struct = struct.Struct("<2q2f")

# pack
data = my_struct.pack(1, 2, 1.5, 2.5)
print(data, len(data))

# unpack
print(my_struct.unpack(data))

There are also the plain functions pack and unpack, where you have to supply the format too.

You can write two small functions for this task or a class.
A example with a minimal "class".
Usually classes have more methods and attributes.

Syntax: Select all

from struct import Struct


class DataStruct:
    st = Struct("<2q2f")

    def __init__(self, file):
        self.file = file

    def load(self):
        with open(self.file, "rb") as fd:
            return self.st.unpack(fd.read())

    def save(self, *values):
        with open(self.file, "wb") as fd:
            fd.write(self.st.pack(*values))


ds = DataStruct("test.bin")
ds.save(1377, 42, 4.5, 10.4)
print(ds.load())
print("Content of test.bin:", open("test.bin", "rb").read())

PS: Don't forget that struct is working with binary data. You have to open the file in binary mode for read and write.

Source.Python forums

Write File in Bytes

Write File in Bytes

Re: Write File in Bytes

Re: Write File in Bytes

Re: Write File in Bytes

Re: Write File in Bytes

Re: Write File in Bytes