Write File in Bytes

Please post any questions about developing your plugin here. Please use the search function before posting!
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Write File in Bytes

Postby decompile » Thu Apr 30, 2020 3:31 am

Hello,

I'm currently trying to write data to a file and I heard you can save alot in file size by saving it in bytes. (Also to reduce lag on saving the file)

I'm currently using this for testing:

Syntax: Select all

from time import time
from paths import GAME_PATH

file_path = GAME_PATH / 'test.txt'

timestamp = time()

with open(file_path, 'wb') as file_stream:
test = ''

for i in range(100000):
test += '0|0|0|0|0\n'

file_stream.write(test.encode())

print(time() - timestamp)


When I do this, its still written in plain text and changes barely in size. Can you help and explain please?

Also is this the best way to save a file without having trouble with lag? Better to write everything at once, or write in several parts?
User avatar
VinciT
Senior Member
Posts: 331
Joined: Thu Dec 18, 2014 2:41 am

Re: Write File in Bytes

Postby VinciT » Thu Apr 30, 2020 3:58 am

Hi, can't help you with the byte thing, but if you wish to avoid getting the stutter/freeze when loading or saving data, you can use GameThread:

Syntax: Select all

# ../gamethread/gamethread.py

# Python
from time import time

# Source.Python
from listeners.tick import GameThread
from paths import GAME_PATH


file_path = GAME_PATH / 'test.txt'
timestamp = time()


def save_to_file():
with open(file_path, 'wb') as file_stream:
test = ''

for i in range(100000):
test += '0|0|0|0|0\n'

file_stream.write(test.encode())

print(time() - timestamp)


thread = GameThread(target=save_to_file)
thread.start()
ImageImageImageImageImage
DeaD_EyE
Member
Posts: 34
Joined: Wed Jan 08, 2014 10:32 am

Re: Write File in Bytes

Postby DeaD_EyE » Thu Apr 30, 2020 2:20 pm

This part is very inefficient:

Syntax: Select all

test += '0|0|0|0|0\n'


If you want to write million times the same thing, then write the same thing million times.

Syntax: Select all

import io
import timeit

# Scenario 1
def test1(file_stream):
test = ''
for i in range(1000):
test += '0|0|0|0|0\n'
file_stream.write(test.encode())

# 1.
# "" + "0|0|0|0|0\n"
# 2.
# "0|0|0|0|0\n" + "0|0|0|0|0\n"
# 3.
# "0|0|0|0|0\n0|0|0|0|0\n" + "0|0|0|0|0\n"
# 4.
# "0|0|0|0|0\n0|0|0|0|0\n0|0|0|0|0\n" + "0|0|0|0|0\n"
# ...


# Scenario 2
def test2(file_stream):
test = '0|0|0|0|0\n'
for _ in range(1000):
file_stream.write(test.encode())

# calling 1000 times encode() and write()
# test is not modified


print("Test1", end="")
t1 = timeit.timeit("test(stream)", setup="stream = io.BytesIO()", number=1000, globals={"io": io, "test": test1})
print(f" {t1:.3f}s")


print("Test2", end="")
t2 = timeit.timeit("test(stream)", setup="stream = io.BytesIO()", number=1000, globals={"io": io, "test": test2})
print(f" {t2:.3f}s")


Code: Select all

PS C:\Users\Admin\Desktop\data> py -3 .\speed.py
Test1 20.785s
Test2 0.148s



Writing incremental in chunks is cheaper than concatenate a string over and over.
Last edited by Ayuto on Thu Apr 30, 2020 7:59 pm, edited 1 time in total.
Reason: code -> python
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Re: Write File in Bytes

Postby decompile » Thu Apr 30, 2020 3:25 pm

@VinciT Thanks, I totally forgot about it.

@DeaD_EyE Thanks for the help so far. I Will write in chunks then.

Do you maybe know how to save the data in binary code aswell? (Also decoding it for loading it)


I saw many sourcemod user use https://sm.alliedmods.net/new-api/files/File/Write which automatically saves input in binary.
User avatar
Ayuto
Project Leader
Posts: 2193
Joined: Sat Jul 07, 2012 8:17 am
Location: Germany

Re: Write File in Bytes

Postby Ayuto » Thu Apr 30, 2020 8:30 pm

You already write the data as bytes. I'm not sure which output you expect? The modes "w" and "wb" only affect the lines endings (\n or \n\r). If you open a file in "w" mode, these characters are converted to the platform dependend line ending. If you open a file in "wb" mode, your text/bytes are written without any changes.

See also:
https://docs.python.org/3/tutorial/inpu ... ting-files
DeaD_EyE
Member
Posts: 34
Joined: Wed Jan 08, 2014 10:32 am

Re: Write File in Bytes

Postby DeaD_EyE » Mon May 04, 2020 2:08 pm

If you want to write structured binary data, you have to define the struct of your data.
Python has a module for this: https://docs.python.org/3/library/struct.html#module-struct

If you want to save for example 2 * int64 and 2 * float64 as little endian:

Syntax: Select all

import struct

my_struct = struct.Struct("<2q2f")

# pack
data = my_struct.pack(1, 2, 1.5, 2.5)
print(data, len(data))

# unpack
print(my_struct.unpack(data))


There are also the plain functions pack and unpack, where you have to supply the format too.

You can write two small functions for this task or a class.
A example with a minimal "class".
Usually classes have more methods and attributes.

Syntax: Select all

from struct import Struct


class DataStruct:
st = Struct("<2q2f")

def __init__(self, file):
self.file = file

def load(self):
with open(self.file, "rb") as fd:
return self.st.unpack(fd.read())

def save(self, *values):
with open(self.file, "wb") as fd:
fd.write(self.st.pack(*values))


ds = DataStruct("test.bin")
ds.save(1377, 42, 4.5, 10.4)
print(ds.load())
print("Content of test.bin:", open("test.bin", "rb").read())


PS: Don't forget that struct is working with binary data. You have to open the file in binary mode for read and write.

Return to “Plugin Development Support”

Who is online

Users browsing this forum: No registered users and 24 guests