Staredit Network > Forums > Modding Discussion > Topic: Object limits plugin
Object limits plugin
Aug 10 2015, 11:14 pm
By: Neiv  

Aug 10 2015, 11:14 pm Neiv Post #1



Hi, in the case someone else is also still modding this game here's a plugin which allows to have more stuff happen at once.

Screenshot with otherwise unmodded starcraft - 10000 mutalisks vs 5000 scouts:


Demo video:


FAQ:

Q: How do I use this?
A: Just add the plugin in mpqdraft/firegraft. No additional configuration is needed. Only version 1.16.1 is supported.

Q: What does it exactly do?
A: This plugin removes the following limits:
-Unit limit (was 1700)
-Sprite limit (was 2500)
-Fog of war sprite limit (was 500)
-Misc sprite limit (i.e. mainly ones created via iscript, was 500)
-Bullet limit (was 100)
-Image limit (was 5000)
-Order limit (was 2000)
-And some ai structures (scripts, towns, unit controllers)
Selection limit is still 12. Sorry - it hasn't been an issue for the stuff I've been doing with this :(
Also some less known limits such as the disappearing creep tile limit have not been touched

Q: Are there any issues/drawbacks?
A: The biggest issue is that other plugins and firegraft exe edits will be very likely broken :(. Firegraft button editing should work fine, as should anything that does not modify the exe.
Very large fights (especially with splash damage) or even massive amounts of images on screen will cause framerate to drop, as the game has some poorly scaling algorithms. There are optimizations for the worst lag spikes though. Also a somewhat related issue is that having a lot (~100k) units being hit in same frame (or 1000 units being hit by 100 splash bullets) will cause the game to run out of memory. Whoops. I'm planning to fix the memory usage eventually though.

Q: I don't see any difference!
A: Well.. Make a large fight and observe how every unit is able to shoot. Otherwise the game should behave just as it does without the plugin.

Q: Source code?
A: Here: https://github.com/neivv/teippi

Q: Is multiplayer supported?
A: Yup.

Q: Can I use it with chaoslauncher?
Yes, just rename the file extension to .bwl and put it in chaoslauncher folder. Or any other launcher which supports loading .bwl plugins.

Download:
http://www.staredit.net/?downloadAttachment=9887&key=6717510


Post has been edited 3 time(s), last time on Oct 24 2015, 8:13 pm by Neiv.



None.

Aug 10 2015, 11:40 pm iCCup.xboi209 Post #2



Nice, since we can execute code in UMS maps, I'd like to see this utilized




Aug 11 2015, 1:22 pm Sand Wraith Post #3

she/her

How likely is the integration of this with the existing GPTP?




Aug 11 2015, 2:14 pm Neiv Post #4



GPTP integration might be really difficult? This project has a lot of code (performance improvements) which is not directly related to the object limits but required a lot of bw's code being replaced. For example caching unit positions for target acquiring orders made the game run a lot faster when there were stacks of thousands units, but required rewriting a lot of the order code to delay anything that would have modified unit positions (Such as unit getting killed). It's not completely impossible to work with GPTP but would require a lot of work.

Additionally, this code does not currently build on visual studio, support for it would have to be added before anything.

+Adding a simple bullet limit hack to GPTP would be few hours of work (even less if you don't care about saves not working) and it would achieve nearly same effect this plugin does. Unit limit is the painful thing here.



None.

Aug 11 2015, 3:26 pm Wormer Post #5



I've never was into SC plugins, but this sounds interesting mosly because it doesn't change the general game behaviour. Will UMS maps made for unmodded SC run for this exactly like they do for unmodified version? I'm having in mind such things like caching units at locationa for bring conditions and when these caches are invalidated and other small nuances that might happen.



Some.

Aug 11 2015, 5:27 pm Neiv Post #6



Based on (somewhat limited) testing, UMS maps will work fine unless the map depends on EUDs, extended players or some other memory overflow (or the fact that there is a limit for something). I don't think I've tested any map which uses a mobile grid, those *might* not work as units *might* not be removed (moved/killed/etc) from leftmost first. Should be easy to fix though.

If there's a map that breaks with this plugin and it does not do any memory overflows, I would like to hear about it as that might be a part of a larger bug.

Also emulating the supply limit changes caused by extended players without actually spawning their units (which can cause a lot of other issues) is possible even though not currently done.



None.

Aug 11 2015, 8:13 pm Sand Wraith Post #7

she/her

Would you mind uploading a demo map?

---

EDIT: On second thought, given that you are compiling what appears to be a large amount of code that also seems to be shared to some extent with the GPTP, is it more or less safe to say that anything (or at least, most) of that which may be accomplished normally via the GPTP may also be accomplished by modifying the source?

Post has been edited 1 time(s), last time on Aug 12 2015, 12:43 am by Sand Wraith.




Aug 12 2015, 4:51 am poiuy_qwert Post #8

PyMS and ProTRG developer

I was working on a sprite and iscript limit expander for IskatuMesk a long time ago. I pretty much had it working except for a crash that would randomly happen around an hour into games, which I got bored and gave up trying to fix. But just getting sprite and iscript limit expanding was a tedious task, I'm very impressed you did so much more! I'm sure Mesk wishes you were around when he was still working on his crazy limit hitting mods :P

Very nice job, keep up the awesome work!




Aug 12 2015, 7:48 am Neiv Post #9



Quote from Sand Wraith
EDIT: On second thought, given that you are compiling what appears to be a large amount of code that also seems to be shared to some extent with the GPTP, is it more or less safe to say that anything (or at least, most) of that which may be accomplished normally via the GPTP may also be accomplished by modifying the source?
If you can find what you wish to modify and can work around the few quirks there are, then sure.

Quote from Sand Wraith
Would you mind uploading a demo map?
Behold, the masterpiece! (Spoilers: It's pretty boring, should have made a demo mod)
(The map also crashes on vanilla bw, weird... because it does not like having more than ~800 units in a small area)

EDIT: The source should now work with Visual studio 2015, though it also requires python 3 for assembly autogeneration.

Attachments:
1500 carriers.scm
Hits: 4 Size: 61.28kb

Post has been edited 3 time(s), last time on Aug 13 2015, 12:50 pm by Neiv.



None.

Aug 30 2015, 1:04 am Heinermann Post #10

SDE, BWAPI owner, hacker.

I have a few questions and things. Firstly, I'm actually very impressed. Very few things impress me, if anything.

Q: Why are you doing this? Really curious. "For fun" is acceptable.

Q: What license are you using for your source code? There is no LICENSE file in the repo.

Q: How are unit IDs managed and dealt with? For example, Broodwar generates IDs like this, which is a conflict. How does your unit lookup stuff work? (i.e. why this?)

Q: What is the SYNC #define for?

Q: Have you considered extending the map size limit next? Map sizes > 256x256.

Q: Player limit?

Q: Can I use this for Unit Testing in BWAPI? I mean programmatic unit testing outside of the BW process. This was an in progress task (see SHADOW_BROODWAR #define in BWAPI).


I'd like to work on getting this compatible with BWAPI if possible.
Issue tracking this: https://github.com/bwapi/bwapi/issues/609
BWAPI offsets: https://github.com/bwapi/bwapi/blob/master/bwapi/BWAPI/Source/BW/Offsets.h#L39-L280
BWAPI hooking/patching: https://github.com/bwapi/bwapi/blob/master/bwapi/BWAPI/Source/CodePatch.cpp#L18

Post has been edited 1 time(s), last time on Aug 30 2015, 2:41 am by Heinermann.




Aug 30 2015, 7:52 am Neiv Post #11



Quote from Heinermann
I have a few questions and things. Firstly, I'm actually very impressed. Very few things impress me, if anything.

Q: Why are you doing this? Really curious. "For fun" is acceptable.
It's part of a larger mod project which might not be finished, at least not in the near future, and once I realized how large task removing the limits were, it became "standalone" plugin.

Quote

Q: What license are you using for your source code? There is no LICENSE file in the repo.
Oops. Was meant to be MIT license. Now there is one.

Quote
Q: How are unit IDs managed and dealt with? For example, Broodwar generates IDs like this, which is a conflict. How does your unit lookup stuff work? (i.e. why this?)
The IDs are just an unique 32-bit counter that count up from 1 to 0xFFFFFFFF.
Before an ID is assigned, it checks it's not used so there should not be issues even if the counter would overflow. Looking up is done with basic hand-made hash table - today I would likely just use the standard library one, but the current implementation works fine so why bother?
Of course, everything that assumed unit IDs to be 16-bit ones bw uses, broke and had to be rewritten. Mostly handful of network commands.
Quote
Q: What is the SYNC #define for?
It enables sync logging. Can you guess how painful it is to debug any desyncs which happen with this hack? D:
The define causes some of the objects to get more random data when they are first allocated, which may help to find the desync more easily. It also sets global boolean "SyncTest" to true, which will cause a lot of the game state to be logged at the end of every frame. If the game desyncs, the logs of players can be compared to make it easier to reason why it desynced. The logs are several gigabytes, so it is not something that is enabled by default.

Quote
Q: Have you considered extending the map size limit next? Map sizes > 256x256.
I have, and some of the code assumes coordinates to be 16-bit unsigned to make extending the size up to 2048x2048 easier in the future. (Unlike bw which considers them to be 16-bit signed)
I'm just not sure if I want to stop at 2048x2048 :P (Well, minimap becomes useless at larger sizes, so it might be reasonable limit)
I'm also afraid of the pathing code :(
Quote
Q: Player limit?
Nope. I imagine it would be a huge pain with all the arrays with either size of 8 or 12 being parts of structures and static memory.

Quote
Q: Can I use this for Unit Testing in BWAPI? I mean programmatic unit testing outside of the BW process. This was an in progress task (see SHADOW_BROODWAR #define in BWAPI).
Sure, but how would it actually work? It currently is really tightly coupled with bw, nearly every function and hook will eventually jump back to bw's code. I've been adding some tests to Teippi which are require opening an empty map and inputting the magic command - rest is automated.
I feel like best way to automate testing would be a system, which starts the game, loads a map and runs the tests without any additional input. Of course that requires the computer running the tests to have a copy of bw, so cloud magic wouldn't work :P

Cool =) I'll look at it.
(Also there are barely any hooks, impressive :O)



None.

Aug 30 2015, 8:29 am Heinermann Post #12

SDE, BWAPI owner, hacker.

Quote from Neiv
I have, and some of the code assumes coordinates to be 16-bit unsigned to make extending the size up to 2048x2048 easier in the future. (Unlike bw which considers them to be 16-bit signed)
I'm just not sure if I want to stop at 2048x2048 :P (Well, minimap becomes useless at larger sizes, so it might be reasonable limit)
I'm also afraid of the pathing code :(
As you should be :P. I imagine the entire thing would need to be rewritten. See this structure. There's also a limit to the number of regions and recursions that happen (we already receive "too many nooks and crannies" error in some maps).

Also you had some unit pathing code where this enum ripped from beta might be helpful.


Quote from Neiv
Sure, but how would it actually work? It currently is really tightly coupled with bw, nearly every function and hook will eventually jump back to bw's code. I've been adding some tests to Teippi which are require opening an empty map and inputting the magic command - rest is automated.
I feel like best way to automate testing would be a system, which starts the game, loads a map and runs the tests without any additional input. Of course that requires the computer running the tests to have a copy of bw, so cloud magic wouldn't work :P
You seem to have reimplemented a lot of core game functionality. The Unit Tests are less about emulating an actual game, and more about getting some code coverage. For example, does calling unit->getHitPoints return the correct unit HP set in some random unit structure? To verify that translating HP from BW to BWAPI is working correctly (since there is a layer of abstraction and filteration).


Quote from Neiv
(Also there are barely any hooks, impressive :O)
This is one of the things I enjoyed doing. In fact we can remove our SFileAuthenticateArchive hook as it's no longer needed (we signed the network module using Ladik's MPQ Editor). A third of the hooks are just for windowed mode. Unfortunately one of the features needed some ugly assembly hooks which I'm ashamed of (there was no good place to hook without rewriting a large chunk). Your code looks like it may have solved this problem.




Sep 18 2015, 7:29 am Sand Wraith Post #13

she/her

I would like to ask about the comment "Extended unit struct beginning" in unit.h.

While I was working with a SEN member on GPTP mods several years ago, the member had mentioned that altering the Unit struct to add extra fields (more uiint32_t or what-not) was dangerous due to memory problems and address referencing.

However, based on the comment and the code here, is this no longer the case? Am I now freely able to add some std::Vectors or so on to Unit and use it elsewhere throughout the code?




Sep 18 2015, 8:05 am Neiv Post #14



As the plugin controls (de)allocation of Unit structures, it can simply add more data at end of it. THe same goes for other structures which had their limits removed.

If there were no limit plugin, the units would be in the Starcraft's preallocated 1700*336 -byte array, which obviously would prevent adding additional data to the structure.



None.

Sep 18 2015, 1:04 pm Sand Wraith Post #15

she/her

I seem to be having significant trouble compiling a working teippi.qdp.

After some struggle, I was able to figure out how to work in my desired code and some changes regarding your build script to include my new .cpp file. Although I can successfully compile teippi.qdp, the resulting file size of 680 KB is significantly smaller than that of your provided teippi.qdp of 982 KB.

Booting SCBW with your .qdp plays out just fine, but my .qdp will crash with a memory exception error upon entering a game.

The changes I have made have affected bullet.cpp, unit.cpp, and unit.h, and I have added my own custom_timers.h and custom_timers.cpp files so as to implement a timer system.

I.e. struct Unit now has public member UnitTimerManager utm, and ProgressTimers() has had a line added calling utm.UpdateTimers().

May I have some help getting this to work?

EDIT:

Compiling with Visual Studio 2015.

Here is the log when using py -3 waf:

Code
>py -3 waf
Waf: Entering directory `D:\Program Files (x86)\Starcraft\_xCustoms\Help_teippi_Pr0nogoWeapon\build'
[ 1/49] Processing src/funcs.autogen: src\func\genfuncs.py src\func\nuottei.txt -> build\src\funcs.autogen
[ 2/49] Compiling src\datastream.cpp
[ 3/49] Compiling src\memory.cpp
[ 4/49] Compiling src\mpqdraft.cpp
[ 5/49] Compiling src\nuke.cpp
[ 6/49] Compiling src\order.cpp
[ 7/49] Compiling src\pathing.cpp
[ 8/49] Compiling src\perfclock.cpp
[ 9/49] Compiling src\draw.cpp
[10/49] Compiling src\player.cpp
[11/49] Compiling src\unitsearch.cpp
datastream.cpp

[12/49] Compiling src\unitsearch_cache.cpp
[13/49] Compiling src\scthread.cpp
[14/49] Compiling src\selection.cpp
perfclock.cpp

[15/49] Compiling src\sprite.cpp
mpqdraft.cpp

[16/49] Compiling src\strings.cpp
strings.cpp

[17/49] Compiling src\targeting.cpp
memory.cpp

[18/49] Compiling src\tech.cpp
order.cpp

[19/49] Compiling src\text.cpp
pathing.cpp

[20/49] Compiling src\triggers.cpp
player.cpp

[21/49] Compiling src\unit_ai.cpp
nuke.cpp

[22/49] Compiling src\unit_movement.cpp
scthread.cpp

[23/49] Compiling src\upgrade.cpp
text.cpp

[24/49] Compiling src\x86.cpp
sprite.cpp

[25/49] Compiling src\yms.cpp
draw.cpp
d:\program files (x86)\starcraft\_xcustoms\help_teippi_pr0nogoweapon\src\draw.cpp: warning C4819: The file contains a character that cannot be represented in the current code page (932). Save the file in Unicode format to prevent data loss

[26/49] Compiling src\replay.cpp
selection.cpp

[27/49] Compiling src\warn.cpp
unitsearch_cache.cpp

[28/49] Compiling src\building.cpp
unitsearch.cpp

[29/49] Compiling src\console\assert.cpp
assert.cpp

[30/49] Compiling src\init.cpp
x86.cpp

[31/49] Compiling src\bwlauncher.cpp
triggers.cpp

[32/49] Compiling src\custom_timers.cpp
yms.cpp

[33/49] Compiling src\save.cpp
unit_movement.cpp

[34/49] Compiling src\patchmanager.cpp
upgrade.cpp

[35/49] Compiling src\limits.cpp
targeting.cpp

[36/49] Compiling src\iscript.cpp
custom_timers.cpp

[37/49] Compiling src\image.cpp
unit_ai.cpp

[38/49] Compiling src\flingy.cpp
tech.cpp

[39/49] Compiling src\bunker.cpp
bwlauncher.cpp

[40/49] Compiling src\bullet.cpp
building.cpp

[41/49] Compiling src\commands.cpp
replay.cpp

[42/49] Compiling src\unit.cpp
flingy.cpp

[43/49] Compiling src\ai.cpp
warn.cpp

[44/49] Compiling src\dialog.cpp
patchmanager.cpp

[45/49] Compiling src\game.cpp
init.cpp

[46/49] Compiling src\lofile.cpp
image.cpp

[47/49] Compiling src\log.cpp
bunker.cpp

[48/49] Compiling src\mainpatch.cpp
iscript.cpp

limits.cpp

lofile.cpp

dialog.cpp

save.cpp

commands.cpp

log.cpp

ai.cpp

unit.cpp

bullet.cpp

mainpatch.cpp

game.cpp

[49/49] Linking build\teippi.qdp
  Creating library teippi.lib and object teippi.exp

Waf: Leaving directory `D:\Program Files (x86)\Starcraft\_xCustoms\Help_teippi_Pr0nogoWeapon\build'
'build' finished successfully (11.761s)





Sep 18 2015, 1:37 pm Neiv Post #16



Try applying this patch (to src\unitsearch_cache.cpp) and see if it fixes the issue. (Credit to Heinermann)

I should have bothered to push that to github :l

Edit: If that doesn't help, .qdp binary with crashdump from Starcraft's Errors/ directory and/or source would be useful.

Attachments:
msvc-intel-fix.patch
Hits: 2 Size: 0.98kb

Post has been edited 1 time(s), last time on Sep 18 2015, 1:47 pm by Neiv.



None.

Sep 18 2015, 2:55 pm Sand Wraith Post #17

she/her

The patch worked perfectly, thank you very much! I had tried to merge Heinermann's change myself but I only really managed to mangle everything.

EDIT:

What methods are there to deal some damage arbitrarily (such as through a poison, damage over time effect)?

It seems like I have the options of AddHit(), the DamagedUnit class, and DamageUnit(), but the program flow is somewhat difficult to discern.

Post has been edited 1 time(s), last time on Sep 18 2015, 3:43 pm by Sand Wraith.




Sep 18 2015, 3:58 pm Neiv Post #18



Yeah.. Unit damage/death is handled bit weirdly there... (It allows some optimizations when units die during Bullet processing, even if the damage was not caused by bullet)

I assume the damage is dealt during some child function of Unit::ProgressFrames() ? There is function Unit::DamageSelf(), which is used for burning damage and such. It needs structure ProgressUnitResults which is used to collect any unit which may have been killed by the damage, so the unit death is handled when appropriate. You need to get that structure from Unit::ProgressFrames() or from any of its child function it gets passed to.

AddHit() and DamagedUnit are supposed to be used by bullet.cpp, and Unit::DamageSelf() should be used instead of DamageUnit(), as it handles death properly.

If you actually aren't doing your damage from child function of Unit::ProgressFrames(), DamageUnit() should work, but you need to pass a empty vector<Unit *> there, and then unit->Kill(nullptr) any units that are added to that vector.
Edit: Actually unit->DamageSelf(dmg, nullptr) is fine outside unit frame progressing.

--Yes, comments would be useful

Post has been edited 1 time(s), last time on Sep 18 2015, 4:03 pm by Neiv.



None.

Sep 27 2015, 6:48 pm Sand Wraith Post #19

she/her

While I was exploring the code, I found an interesting "resolution.h" file.

I've tried doubling its values in the hopes of expandiing SC's resolution but this leads to a memory read exception on start-up of the mod (workflow: compile through VS2015 to *.qdp, import *.qdp through Firegraft, save as *.exe).

What is the purpose of this file and these fields? Were you exploring expanding SC's resolution?

Additionally, is this all original code? What resources did you use to build this plugin? Are you part of another (SCBW?) modding community?




Sep 27 2015, 11:54 pm Neiv Post #20



resolution.h and limits.h have constants which are defined to make code clear, and they cannot be modified. (Also if those limits are ever removed, the constants would be removed to cause compile errors anywhere the code assumes constant limits)

When I started, I thought that resolution modification was a simple thing to do, and that's why they are in a otherwise empty file :P
..I also thought that removing unit limits would require only modifying a few functions ._.

And no, I don't know of any active bw modding communities :l
Pretty much all of the code in teippi was reverse engineered from the bw exe and modified from there, but I used GPTP, BWAPI and some other resources for structure definitions, memory offsets and enumeration names.

Quote from Sand Wraith
Am I now freely able to add some std::Vectors or so on to Unit and use it elsewhere throughout the code?

Addition to this: Having any non-POD data in units/bullets/etc will currently cause crashes when loading saves, as the save/load code is a horrible mess of hacks. I'll improve the code in the coming week when I have time.



None.

Options
  Back to forum
Please log in to reply to this topic or to report it.
Members in this topic: None.
[02:58 am]
KrayZee -- You
[12:00 am]
RexyRex -- u
[09:35 pm]
Pr0nogo -- he broke it intentionally so you couldn't take it
[08:55 pm]
Dem0n -- ya, hurry up and fix so I can take Wing's box
[08:40 pm]
Wing Zero -- Magic boxes still not fixed....
[08:10 pm]
UEDCommander -- How depressing
[08:09 pm]
UEDCommander -- Recently someone actually called me "eud" without even knowing what starcraft is
[01:13 pm]
Pr0nogo -- u
[12:42 pm]
UEDCommander -- Who said EUDCommander
[07:56 am]
Pr0nogo -- u(de)
Please log in to shout.


Members Online: Roy, KrayZee, SiKiN, MasterJohnny