Staredit Network > Forums > Modding Discussion > Topic: PyMS - Download, Support, and Maintenance
PyMS - Download, Support, and Maintenance
Jan 10 2019, 9:08 pm
By: poiuy_qwert  

Jan 10 2019, 9:08 pm poiuy_qwert Post #1

PyMS and ProTRG developer

PyMS is a cross platform, BroodWar modding suite, written using Python. PyMS contains 15 programs to edit most of the file types you will encounter while modding.

Download:
Always download the latest version from the official github repository:
https://github.com/poiuyqwert/PyMS#installation


Support and Maintenance:
PyMS is still under development, getting bug fixes and new features. I don't have much time anymore to do my own modding, or to keep up to date with the modding community, just to develop PyMS. This means that the more users that use PyMS and actually report bugs and feature requests, the better PyMS will be for everyone. So please, report bugs and features here, or as an issue on gitgub. Some more information here: https://github.com/poiuyqwert/PyMS/issues


Current priorities:
1) High priority bugs (always my highest priority if there are any)
2) SCR support in PyBIN (highly requested feature)
3) Better doodad management in PyTILE: Issue 33 , Issue 38 (my interest)
4) Lower priority bugs and features
My priorities are driven mostly by user feedback. High priority bugs will always get first attention, but after that highly requested features get my focus. So the more people requesting something the more likelihood that it will be worked on sooner! After that I work on things that interest me.


Open source:
PyMS is open source, so you can help contribute fixes and features to the official PyMS repository. You can also modify the programs and use its internal libraries to implement your own custom things. A couple examples:
- The custom AI script launcher BWAILauncher was built using parts of PyMS
- The mod STF had different difficulty levels. This was accomplished by making a simple script that would open units.dat, scale some values (health, damage, etc.) to 25% and 50%, and exported them to separate files. This script could be re-run anytime the main stats were updated to always provide correctly scaled difficulty.
- Adding support for features implemented in plugins, for example custom trigger actions added by TrigPlug (created by FaRTy) were added to PyTRG, which were eventually added officially to PyMS. Neivv has been working on a plugin to add custom AI script commands, and adding support for them to PyAI (hopefully this will eventually be added officially to PyMS)
- If you run into a bug you can fix it on your own (And then submit that fix to me so I can include it in future versions of PyMS!)




Jan 11 2019, 7:08 pm Pr0nogo Post #2



Great to have it in its own thread. I think that instructions on how to run the programs with admin privileges (on windows) would be useful, since currently that's the only way to use the 'set program as default' feature.

Step 1: run command prompt as admin (start menu, search 'cmd', press ctrl+shift+enter)
Step 2: change directory to your python install (usually C:\Python27)
Step 3: run python.exe followed by the path to your pyms program (e.g. `python.exe C:\Users\u\Desktop\PyMS-master\pydat.pyw`) and press enter
Step 4: press the blue registry icon to create the file association

---

When launching the latest pytbl, I got this error. Is this user error?

I see this issue referencing a 'suppress warning directive', is that related to warnings like these (that pop up every time I open a script as long as my unitdef is configured this way)? If so, how do I activate it?

Also, I know it's probably the most far removed from what you're thinking about working on, but a replacement for Firegraft is sorely needed for modern projects. Firegraft is barely functional and has numerous UI and corruption issues. Many dat requirement entries are inaccurate or incomplete (e.g. 'Is not lifted off' also requires the unit to be a building) and it has no support for future engine improvements that add additional dat requirements or functions.

Post has been edited 3 time(s), last time on Jan 11 2019, 8:15 pm by Pr0nogo.




Jan 11 2019, 8:35 pm Nekron Post #3



from what i remember the supress warning thingy was originally made for supressing warnings about spellcasters having no weapons, but it can presumably be used for other warnings the way p_q wrote it. to use it you just add the line before whatever warning you want supressed




Jan 12 2019, 5:38 am poiuy_qwert Post #4

PyMS and ProTRG developer

Quote from Pr0nogo
Great to have it in its own thread. I think that instructions on how to run the programs with admin privileges (on windows) would be useful, since currently that's the only way to use the 'set program as default' feature.


Step 1: run command prompt as admin (start menu, search 'cmd', press ctrl+shift+enter)
Step 2: change directory to your python install (usually C:\Python27)
Step 3: run python.exe followed by the path to your pyms program (e.g. `python.exe C:\Users\u\Desktop\PyMS-master\pydat.pyw`) and press enter
Step 4: press the blue registry icon to create the file association
This is really a bug, it should not require admin privilege. I have not really been able to fix it because I have not had a windows pc, but I should have one soon.


Quote from Pr0nogo
When launching the latest pytbl, I got this error. Is this user error?
I am not getting the error when I launch it. Does the Units.pal exist?


Quote from Pr0nogo
I see this issue referencing a 'suppress warning directive', is that related to warnings like these (that pop up every time I open a script as long as my unitdef is configured this way)? If so, how do I activate it?
Quote from Nekron
from what i remember the supress warning thingy was originally made for supressing warnings about spellcasters having no weapons, but it can presumably be used for other warnings the way p_q wrote it. to use it you just add the line before whatever warning you want supressed
Nekron is correct. Every warning has a name, you can see it in brackets on the warning (for example "building" is the one in the image you posted). There are two directives to suppress warnings using their name:
@suppress_next_line(<name>) will suppress the warning only on the next line
@suppress_all(<name>) will suppress the warning for the rest of the file


Quote from Pr0nogo
Also, I know it's probably the most far removed from what you're thinking about working on, but a replacement for Firegraft is sorely needed for modern projects. Firegraft is barely functional and has numerous UI and corruption issues. Many dat requirement entries are inaccurate or incomplete (e.g. 'Is not lifted off' also requires the unit to be a building) and it has no support for future engine improvements that add additional dat requirements or functions.
I have wanted to make one for a long time (that and a map editor), and now that I will have a PC soon I may even look into it more. But I don't really have any concrete plans. It's a much more complicated thing than the other modding tools, and I don't really have a reference for exe the way I have references for modding file formats, though FaRTy might have some stuff...




Jan 12 2019, 5:59 am Pr0nogo Post #5



Quote from poiuy_qwert
I am not getting the error when I launch it. Does the Units.pal exist?
It does, same filename and path.

Good to know about the suppression, I'll try it out. This is related to the unitdef though, it pops up on every script. Do I now have to suppress it in every script I write? This seems like a poor solution.

Quote from poiuy_qwert
It's a much more complicated thing than the other modding tools, and I don't really have a reference for exe the way I have references for modding file formats, though FaRTy might have some stuff...
Let us know what specifically you're looking for and I might be able to find some stuff.

Post has been edited 1 time(s), last time on Jan 12 2019, 7:23 am by Pr0nogo.




Jan 15 2019, 12:29 am poiuy_qwert Post #6

PyMS and ProTRG developer

Quote from Pr0nogo
Good to know about the suppression, I'll try it out. This is related to the unitdef though, it pops up on every script. Do I now have to suppress it in every script I write? This seems like a poor solution.
Hmm I never considered the warnings from parsing the unitdef.txt. I can't remember but I have a feeling its using a different parser so this won't work, but did you try putting a suppress directive inside the unitdef.txt? Also, I forgot to mention in my first post (it was late), but you should just make sure you have setup your dat files correctly. You should use your mods files by either applying your custom dat files in the PyAI settings, or add your mod MPQ with a higher priority, so that it can actually load the correct dat file with the correct "is building" settings.

Quote from Pr0nogo
Let us know what specifically you're looking for and I might be able to find some stuff.
Essentially everything that Firegraft modifies. I'm pretty sure I can get to the point of what MPQDraft does, create a self-executing MPQ (SEMPQ) and loading plugins, but its the SC specific functionality that FireGraft adds on top of that which I have no specs or reference for: Button Sets, Dat Requirements, etc.




Jan 15 2019, 2:26 am Pr0nogo Post #7



When writing @suppress_all(building) to unitdef.txt, the program fails to open scripts with this error. Maybe it's syntax. Tried <building> and h_siege_mode as well with no luck.

Generally I've only had bad experiences with adding my own MPQs, and generally keeping an MPQ up-to-date with all graphics and statistics is a chore when all it needs are the files necessary for SCMDraft to work (e.g. tilesets, tbl files, some dat files). Does PyMS support adding exes made with firegraft? If so I'll try to mess around with it a bit more. However, unless PyAI reads the resource harvester flag for building status, it won't change anything about this error.

I'll put out a call for info re: Firegraft and try to forward some knowledgeable people to the thread. I would also love to see a fully-functional open source map editor and I have a lot of ideas for additional features down the line, but it takes a lot of work to be as good or better than SCMDraft, so it's no wonder we haven't seen many people attempt it.

edit: here's an updated unitdef with all entities (I think). Since an expanded unitdef is required for simple things like making AI defend or attack with hero/nonstandard units, maybe it should be added to the official commit. I can reorganize it if you like. Click

Post has been edited 1 time(s), last time on Jan 15 2019, 4:22 am by Pr0nogo.




Jan 15 2019, 5:05 am poiuy_qwert Post #8

PyMS and ProTRG developer

Quote from Pr0nogo
When writing @suppress_all(building) to unitdef.txt, the program fails to open scripts with this error. Maybe it's syntax. Tried <building> and h_siege_mode as well with no luck.
Yeah that is what I thought.

Quote from Pr0nogo
Generally I've only had bad experiences with adding my own MPQs, and generally keeping an MPQ up-to-date with all graphics and statistics is a chore when all it needs are the files necessary for SCMDraft to work (e.g. tilesets, tbl files, some dat files).
There were two options I mentioned, you don't need to use MPQ's if that doesn't fit your workflow, just load the correct files from disk.

Quote from Pr0nogo
Does PyMS support adding exes made with firegraft?
Yes, SFmpq supports embedded mpq's.

Quote from Pr0nogo
However, unless PyAI reads the resource harvester flag for building status, it won't change anything about this error.
Of course it does :P

Quote from Pr0nogo
I'll put out a call for info re: Firegraft and try to forward some knowledgeable people to the thread.
Sounds good.

Quote from Pr0nogo
I would also love to see a fully-functional open source map editor and I have a lot of ideas for additional features down the line, but it takes a lot of work to be as good or better than SCMDraft, so it's no wonder we haven't seen many people attempt it.
I have worked on a couple a little over the years. I included the beginnings of one in PyMS, but the UI engine and Python are too slow for me to really continue it. I also started making a C++ modding library (with the plans of making a successor to PyMS at some point), and was making a map editor using it. So many things I want and wish I could work on lol

Quote from Pr0nogo
edit: here's an updated unitdef with all entities (I think). Since an expanded unitdef is required for simple things like making AI defend or attack with hero/nonstandard units, maybe it should be added to the official commit. I can reorganize it if you like. Click
unitdef files are not required to use those units, there are multiple ways to do it (by unit id, by TBL name, define your own variable), they are just a convenience. But I agree it would be useful to have everything in the official file, i'll look into it, thanks




Jan 15 2019, 7:35 am IlyaSnopchenko Post #9

The Curious

So it dawned on me that the PYAI supports some "non-standard" AI opcodes enabled by a certain MPQDraft plugin? I failed to find that plugin or the documentation for it. Anyone can share?



Who am I, and why should I care?

Jan 15 2019, 8:24 am Pr0nogo Post #10



It's available in the PyMS dropdown of the manly thread: http://www.staredit.net/topic/17400/




Jan 15 2019, 8:28 am IlyaSnopchenko Post #11

The Curious

Quote from Pr0nogo
It's available in the PyMS dropdown of the manly thread: http://www.staredit.net/topic/17400/
Ah, thanks. I always miss the fine print. :) Thank God I'm never doing mortgages. :D



Who am I, and why should I care?

Jan 16 2019, 2:08 am Pr0nogo Post #12



Speaking of fine print, there's a typo in PyICE - Cybernetics Core Overlay is spelled "Cybrnetics". This also translates to the iscript code itself.

And just in general for ease of use, maybe the PyAI and PyICE scroll pos issues can be escalated? PyICE also deselects the last edited entry which just exacerbates the scroll pos reset.

edit: also, if 'enter' can be the universal hotkey for confirming changes (especially for PyAI's "continue importing?" block when saving a script), that'd be very useful.

Post has been edited 1 time(s), last time on Jan 16 2019, 10:14 am by Pr0nogo.




Jan 17 2019, 8:10 pm Neiv Post #13



Quote
Essentially everything that Firegraft modifies. I'm pretty sure I can get to the point of what MPQDraft does, create a self-executing MPQ (SEMPQ) and loading plugins, but its the SC specific functionality that FireGraft adds on top of that which I have no specs or reference for: Button Sets, Dat Requirements, etc.

Here's a rough description of addresses/structures/formats Firegraft touches:

Code
Button structures:
005187E8 = ButtonSet button_sets[0xfa]
ButtonSet: (Size 0xc) - Corresponds to Firegraft's Units tab
0x0     u32 button_count
0x4     Button *buttons
0x8     u16 linked_button_set?
0xa     u16 padding?

Button: (Size 0x14) - Firegraft's Button Sets tab
0x0     u16 pos
0x2     u16 icon_id
0x4     func *Condition(), ecx var, edx local_player, arg 1 Unit *unit
0x8     func *Action(), ecx var, edx shift_down
0xc     u16 cond_var
0xe     u16 act_var
0x10    u16 enabled_string_id
0x12    u16 disabled_string_id

These are hardcoded in the exe, and the modloader can just change these values at any point when the
game is running.
I don't have a mapping of function address - meaning for the conditions/actions though.

---

Status screen:

005193A0 = StatusUiFunc status_screen[0xe4]
StatusUiFunc: (Size 0xc) - Firegraft's Units tab
0x0     u32 debug_id
0x4     func *DoesNeedRedraw()
0x8     func *Draw(), ecx Control *ctrl

Pretty much same idea with hardcoded data/function pointers as with buttons.

---

Dat requirements:

Units: 00514178
Orders: 00514CF8
Tech use: 00514A48
Tech research: 00514908
Upgrades: 005145C0

These are arrays of u16 describing the dat requirements in the following format:
   u16 unused/0
{
   u16 dat_id
   u16 opcodes[]
   u16 0xffff
}
   u16 0xffff

Opcodes smaller than 0xff00 are unit ids (Always unit ids) that the player has to own, other opcodes
are:
   0xff01 = or,
   0xff02 = Current unit is [+1],
   0xff03 = Has unit [+1],
   0xff04 = Has addon [+1] attached,
   0xff05 = Is not lifted,
   0xff06 = Is lifted,
   0xff07 = Is not busy building (Train/tech/upgrade/addon),
   0xff08 = Is not constructing addon,
   0xff09 = Is not teching,
   0xff0a = Is not upgrading,
   0xff0b = Is not constructing building,
   0xff0c = Has no addon,
   0xff0d = Has no nydus exit (Or anything other things, as it's an union)
   0xff0e = Has hangar space,
   0xff0f = Tech is researched,
   0xff10 = Only if has no nuke
   0xff11 = Only if not burrowed or ai controlled,
   0xff12 = Only if not landed building,
   0xff13 = Only if landed building,
   0xff14 = Only if can move,
   0xff15 = Only if can attack,
   0xff16 = Only if worker,
   0xff17 = Only if can liftoff,
   0xff18 = Is transport,
   0xff19 = Only if powerup,
   0xff1a = Only if subunit,
   0xff1b = Only if has spider mines (Or addons etc),
   0xff1c = Is hero,
   0xff1d = Only if can rally or rclick (Can hold position..?),
   0xff1e = Allow on hallucinations,
   0xff1f = Upgrade level based jump:
       jumps to next 0xff20 if level == 1 and 0xff21 if level >= 2.
   0xff22 = Disabled,
   0xff23 = Blank,
   0xff24 = Only if bw,
   0xff25 = Only if tech [+1] researched,
   0xff26 = Only if burrowed,
   0xffff = end

Not all of these apply to units and not all of these apply to non-units. No notes on which work
where. You could look at the switches in functions 0046E1C0 (units) and 0046D610 (non-units)

I don't know if firegraft does this, but since the dat requirements are not behind a pointer,
adding more datreq opcodes than there originally were would cause unrelated data to be corrupt.
So it would be a good idea to search exe code for any references that access datreqs and change them
to refer to elsewhere instead of trying to fit the new datreq data where it originally was.

---

Exe edits:
You can extract the exe edit file from firegraft mpqs, they are text files describing memory
patches.




None.

Jan 17 2019, 8:40 pm Nekron Post #14



For exe edits, there are some that crash the game on 1.16.1, one of them was I think this? maelstrom edit (copied hopefully correct version below, though I'm not 100% sure if it's this particular edit since I only got the fix from Neiv and it was 2 years ago) - if anyone remembers or knows other faulty ones off-hand it could be useful to write them down
Quote
1530Entry=AI Management,Tech,Maelstrom AI,Allow casting Maelstrom when on full energy
1530Hint=By default, AI-controlled Dark Archons will not use Maelstrom if they have enough energy for both Maelstrom and Mind Control.
1530Offset=0xa1b50
1530Size=6
1530Type=Jump





Jan 17 2019, 10:02 pm Pr0nogo Post #15



PyMPQ: Option to save mpq as would really come in handy.

PyICE: iscript entry 274 has another type: "Burowing Dust"




Jan 17 2019, 10:40 pm poiuy_qwert Post #16

PyMS and ProTRG developer

Thanks for the info Neiv, that looks very useful!
Do you have any plans to open source Samase or having collaborators? I do want to try my hand at making my own from scratch, since it sounds fun and I've wanted to for a long time, but I would also be interested in collaborating on something as well.
Also, I'm planning to look into including your aise support for PyAI into the main PyMS repository. I'll probably need to make tweaks to make it more obvious to users that the new commands are not built in functionality of SC and require a plugin to use. Are you fine with me going ahead with that, and/or do you have any thoughts/concerns?

Quote from Nekron
For exe edits, there are some that crash the game on 1.16.1, one of them was I think this? maelstrom edit (copied hopefully correct version below, though I'm not 100% sure if it's this particular edit since I only got the fix from Neiv and it was 2 years ago) - if anyone remembers or knows other faulty ones off-hand it could be useful to write them down
Quote
1530Entry=AI Management,Tech,Maelstrom AI,Allow casting Maelstrom when on full energy
1530Hint=By default, AI-controlled Dark Archons will not use Maelstrom if they have enough energy for both Maelstrom and Mind Control.
1530Offset=0xa1b50
1530Size=6
1530Type=Jump
I think this would be good to document, but i'm not sure if this is the right thread for it, probably better to put them in the Problems with Current Modding Tools thread for now.

When/if I make a graft tool, i'm not sure if it will be a part of PyMS or not, since it won't be done in Python nor be cross platform, which are some core goals of PyMS. This will be a hard decision though since another core goal of PyMS is having all the tools to mod StarCraft in one place. I'll have to think about it.


Quote from Pr0nogo
PyMPQ: Option to save mpq as would really come in handy.
Thats not really how MPQ/SFmpq works, changes are done directly to the mpq on disk. What I could add is a "Clone MPQ" button, which will just copy the MPQ on disk, then open the new MPQ. But you will need to do that before editing the MPQ, you wouldn't be able to add a file and decide after to save the modified MPQ to a different location.

Quote from Pr0nogo
PyICE: iscript entry 274 has another type: "Burowing Dust"
I assume you meant "typo"? I was confused for a second :P

Thanks everyone!




Jan 18 2019, 12:00 am Neiv Post #17



I'm not really keen on sharing samase's code, as it has to do certain things to get past SCR's antihack protection.. Since there aren't (afaik) any hacks (at least any that work by injecting DLLs) in SCR, I really don't want to share any examples of how to do that :l And samase doesn't really do much more than that.

--

If you want to do any changes to aise make it possible to integrate with main PyMS, that'd be great =)
I likely don't have time to work on anything non-trivial related to aise in near future though.

Few things worth noting:
- There are some pretty ugly extensions to the syntax parsing, especially with some complex variable-length parameters. They should work fine, but at the same time they were quite fragile, and I felt that they could use some refactoring... Also lot of duplicated code related to various enum-like parameters, and just general inconsistency with parameter code. No idea how much you care, but it looks ugly :P
- There's code that lets the scripts be larger than 65k bytes, but if the script stays below that (and obviously no extended commands), it stays vanilla-compatible. Though the format change (really just switching to 32-bit offsets for goto/aise jump commands) happens without notification when the script wouldn't fit in 65k.
- Mouseover tooltips for the new commands/parameters are very much incomplete. Some exist but most do not.



None.

Jan 18 2019, 1:11 am poiuy_qwert Post #18

PyMS and ProTRG developer

Quote from Neiv
I'm not really keen on sharing samase's code, as it has to do certain things to get past SCR's antihack protection.. Since there aren't (afaik) any hacks (at least any that work by injecting DLLs) in SCR, I really don't want to share any examples of how to do that :l And samase doesn't really do much more than that.
lol, totally makes sense.

Quote from Neiv
If you want to do any changes to aise make it possible to integrate with main PyMS, that'd be great =)
Great, thanks!

Quote from Neiv
There are some pretty ugly extensions to the syntax parsing, especially with some complex variable-length parameters. They should work fine, but at the same time they were quite fragile, and I felt that they could use some refactoring... Also lot of duplicated code related to various enum-like parameters, and just general inconsistency with parameter code. No idea how much you care, but it looks ugly :P
I'll have to take a look :P

Quote from Neiv
There's code that lets the scripts be larger than 65k bytes, but if the script stays below that (and obviously no extended commands), it stays vanilla-compatible. Though the format change (really just switching to 32-bit offsets for goto/aise jump commands) happens without notification when the script wouldn't fit in 65k.
Awesome. PyTILE has support for extended files so it should be pretty easy to duplicate the same UX for this.

Quote from Neiv
Mouseover tooltips for the new commands/parameters are very much incomplete. Some exist but most do not.
Yeah I will flesh those out, if you (or Pr0nogo) have some time to jot down a sentence or two about each command, that would help.

Thanks

Post has been edited 1 time(s), last time on Jan 18 2019, 1:25 am by poiuy_qwert.




Jan 18 2019, 1:21 am Pr0nogo Post #19



I've already documented every command extensively. I can write more brief descriptions similar to the default commands if you want. Current documentation is here.




Jan 18 2019, 1:25 am poiuy_qwert Post #20

PyMS and ProTRG developer

Yeah if you can that would be amazing.




Options
  Back to forum
Please log in to reply to this topic or to report it.
Members in this topic: None.
[09:30 pm]
NudeRaider -- modding is the only true religion. if you're not doing it you're doing it wrong
[09:28 pm]
NudeRaider -- Pr0nogo
Pr0nogo shouted: u
nice nice. Now try this one: v
[09:27 pm]
Pandut -- jaysus modding is hard when you havent done it in like 7 years
[09:12 pm]
Wing Zero -- If theyre frozen like this ya'll might as well try to open ever box
[09:11 pm]
Wing Zero -- Aaah magic boxes pls
[08:58 pm]
Pr0nogo -- u
[04:07 pm]
NudeRaider -- Pr0nogo
Pr0nogo shouted: what are those strange letters in front of u?
they're part of the alphabet, just like the u. But don't worry, no shame in not knowing. Even in developed countries we have programs dedicated to teach reading and writing because even here it happens.
[02:31 pm]
TF- -- Anitoch
[02:00 pm]
Dem0n -- where discord at lmao
[01:58 pm]
Pr0nogo -- what are those strange letters in front of u?
Please log in to shout.


Members Online: Roy, Pandut, lil-Inferno