Staredit Network > Forums > Modding Assistance > Topic: Extracting navigable terrain from map files
Extracting navigable terrain from map files
Apr 7 2021, 1:20 pm
By: MufinMcFlufin  

Apr 7 2021, 1:20 pm MufinMcFlufin Post #1



Hey, just signed up here specifically to try to get some help on this little project I've been working on.

I'm trying to play around with some pathfinding algorithms and wanted to try them out in starcraft maps to have some actual use-case scenarios that I can test them on, and to that end I'm trying to extract the navigable tiles from a given .scm or .scx file. Using this page, I've been able to extract the TILE data from the .chk file (which I believe is the one I'll need as it looks more applicable than the MTXM due to excluding 'doodads' and I'm really not certain what ISOM does, I'm guessing pseudo 3D information used for placing the wireframe cursor?), but that just leaves me with the megatile indexes. I'm now trying to get the subtiles and their properties based on information I found on this page, but I'm not finding where that information is stored or how it relates to the megatile indexes I currently have.

I was able to find the BW tileset files with no problem in the BrooDat.mpq under the tileset folder in it, however the StarDat.mpq file I have doesn't appear to be structured anything like the BrooDat.mpq file, and as such I'm clueless where to find the StarDat.mpq tilesets. I've included the StarDat.mpq (listfile) file as an attachment for reference. I've also seen that the majority of the files in each BW tileset folder are just graphic files. I did find the dddata.bin which I'm guessing is where the subtile information with properties is located, but I don't know what format it's in or how to get that information from it.

Any information about how these files are structured, how to get the subtiles for each megatile, how to get the properties for those subtiles, or a place that would be better suited for this question (if this is the wrong place to ask) would be immensely helpful and greatly appreciated.

Attachments:
(listfile)
Hits: 1 Size: 0.39kb

Post has been edited 2 time(s), last time on Apr 7 2021, 1:36 pm by MufinMcFlufin. Reason: Didn't hit 'Attach!' when uploading (listfile), and made a few s



None.

Apr 8 2021, 4:36 am jjf28 Post #2

Cartography Artisan

Since we're talking MPQs I'll just clarify that this must be about patch 1.16.1 or earlier, not SC:R

From the CHK you want to extract your tile data from MTXM, TILE and ISOM are not read by the game, only used in the editor.

When StarCraft 1.16.1 looks for asset files from MPQ it tries to get it from patch_rt.mpq, if it's not there it tries BrooDat.mpq, if it's not there it tries StarDat.mpq

I don't have all this memorized and the documentation I originally used is gone so... code dump:
How I define terrain data
How I load terrain data

Then given a tileIndex/tileValue from MTXM...
how I render a tile
how I render a tiles elevations

If C++ is appropriate for your project you can copy in the MappingCoreLib folder from Chkdraft if you wish; that should give you a jump on reading terrain out of a map and doing things with relation to the terrain asset files.



TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Apr 8 2021, 11:43 am MufinMcFlufin Post #3



Quote from jjf28
If C++ is appropriate for your project you can copy in the MappingCoreLib folder from Chkdraft if you wish; that should give you a jump on reading terrain out of a map and doing things with relation to the terrain asset files.

Unfortunately it is not, as I'm working in Python, however I have programmed in C++ before so I'll give these links a good look over. If I can get something functional I'll post some code of my own for anyone else trying to do the same later on.

Edit: Apparently I've overestimated how long it's been since I last worked on C++. This is gonna be rough. x_x

Post has been edited 1 time(s), last time on Apr 8 2021, 12:26 pm by MufinMcFlufin.



None.

Apr 8 2021, 5:29 pm MufinMcFlufin Post #4



So correct me if I'm wrong but from what I'm understanding I need to get the ERA info for which tileset I'm using, use the megatile IDs I'm getting from MTXM, open up the corresponding tileset .cv5 file, get the 16 word megatile references for each megatile ID, and get the first bit (the walkability bit, might also play with the visibility bit later) for each reference in that tileset's .vf4 file.

Does that sound about right?

Also do you know anything about where I can find the original SC1/Non-BW tilesets? I'm not seeing them in the patch_rt.mpq or StarDat.mpq files.



None.

Apr 8 2021, 6:41 pm jjf28 Post #5

Cartography Artisan

I don't think that's quite right; the tileIndex you get from MTXM gets you a group (cv5 entry) and a groupIndex (array index in the cv5 entry) and the value you get back from that array in the cv5 entry is your megatile, which can then be used to index the other files

If you're using an mpq browser you might need newer listfiles

Tileset File Locations




TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Apr 9 2021, 1:29 am poiuy_qwert Post #6

PyMS and ProTRG developer

Since you are using Python, you might be able to reference some of PyMS: https://github.com/poiuyqwert/PyMS (There is CHK.py and Tilesets.py in the Libs folder)

It is quite old, so it is using Python 2.7 not Python 3.x, so you might not be able to use it directly. A lot of pieces are not the best code, most of it was written when I was not the best coder (though I do think CHK and Tilesets are some of the things I made later, so are not the worst). I have gotten much better at coding over the years but it was already too big for me to make a serious effort to improve the code (though I have started to do that a bit in a branch). But I could probably also give some help if needed.

Good luck!




Apr 9 2021, 5:50 pm MufinMcFlufin Post #7



Quote from jjf28
I don't think that's quite right; the tileIndex you get from MTXM gets you a group (cv5 entry) and a groupIndex (array index in the cv5 entry) and the value you get back from that array in the cv5 entry is your megatile, which can then be used to index the other files

So (again correct me if I'm wrong) I get a tileIndex from the MTXM section, find the associated cv5 entry, get the group (first u16 in that entry), and the groupIndex (last 16 u16's), and the groupIndex is the 16 minitiles which collectively make the megatile, and the values from that 16 u16's can be directly referenced in the vf4, vr4, vx4, etc. files in the same fashion as the cv5 just with different sized entries. Sound about right?

The one thing I'm a bit confused by is the group value; that first u16 at the beginning of each cv5 entry. I'm not understanding where it comes into play here or its relevance.

Quote from poiuy_qwert
Since you are using Python, you might be able to reference some of PyMS: https://github.com/poiuyqwert/PyMS (There is CHK.py and Tilesets.py in the Libs folder)

It is quite old, so it is using Python 2.7 not Python 3.x, so you might not be able to use it directly. A lot of pieces are not the best code, most of it was written when I was not the best coder (though I do think CHK and Tilesets are some of the things I made later, so are not the worst). I have gotten much better at coding over the years but it was already too big for me to make a serious effort to improve the code (though I have started to do that a bit in a branch). But I could probably also give some help if needed.

Good luck!

I appreciate the help. I have found a few lines that looked like they could be reworked to be more explicit about what's being done or could be more clear about what's happening, but it's otherwise good enough that it's been helping me in learning about the structure of these files.

The only things that seemed particularly bad were some while loops that appeared to effectively be for loops with extra steps, but not like I can complain much after looking back at some of my old code and some of my current code.



None.

Apr 9 2021, 6:19 pm jjf28 Post #8

Cartography Artisan

Quote
get the group (first u16 in that entry)

nah, the "group" is the cv5 entry itself, I don't use the fisr u16 from the cv5 entries
tileIndex/16 = groupIndex/index of cv5 entry
tileIndex&0xF = groupMemberIndex/index of megatile within that cv5 entry

the rest sounds about right



TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Apr 13 2021, 1:28 pm MufinMcFlufin Post #9



So one thing I'm still confused by is something here where it says the VF4 struct is 32 bytes of 16 words which are referenced by the cv5, however it also says the maximum ID in the vf4 file is 65535 (1 word). I'm guessing I should probably ignore the 32 byte structure it's talking about and just reference the words individually. Every time I try taking the megatile reference values from the cv5 entries and referencing the vf4 as if the structure were 16 word entries I get IndexErrors, not that it makes sense anyways as I'd be getting 16 sets of flags back for one minitile.

Edit: Apparently it might be something else yet, probably about my implementation so far. I'm getting some values from MTXM that are as high as 21000.

Edit 2: I am indeed a dumbass, I just saw that part in the cv5 section about the first 3 nibbles being the group and the last nibble being the tile. Will work on implementing that now.

Post has been edited 2 time(s), last time on Apr 13 2021, 1:54 pm by MufinMcFlufin.



None.

Apr 13 2021, 3:12 pm jjf28 Post #10

Cartography Artisan

VF4 has those 32-byte entries per each megatile, you select one of these with that value you got back from CV5, not index a word individually, each 2 bytes describes an 8 by 8 pixel chunk of the tile (starting from top left), you need to get all 16 to "draw" or "elevate" the entire tile



TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Apr 13 2021, 7:38 pm MufinMcFlufin Post #11



Well thanks to your help, I was finally able to get it fully functional. It's not exactly the best code, but it works. The only dependency is PIL/pillow for the output image so one could exclude it and not have to use any additional modules. I built it for 3.8 but it should work for plenty of other versions too.

I feel like with the links posted here, how we've discussed the problem, and with the code I've posted below (especially with how Python looks pretty similar to pseudo code) anyone who stumbles upon this page trying to do the same thing I did should have plenty of information to help them do it themselves. I may at some point look into opening the MPQ archive myself so I can just feed it the map archive rather than first extracting the scenario.chk file, but that's a project for later.

Thank you again for your help, I don't think I would have been able to get it functional without the information you provided.
Code Block




None.

Apr 15 2021, 6:50 pm poiuy_qwert Post #12

PyMS and ProTRG developer

PyMS includes a light weight Python wrapper for SFmpq (Files starting with SFmpq in the Libs folder). It should be extractable since it doesn't have any dependencies on the rest of PyMS, and I feel like it should work on Python 3.x with no or minimal changes but I have not looking into it. If you do try it out let me know how it goes, and if you needed to change anything, what you had to change.




Options
  Back to forum
Please log in to reply to this topic or to report it.
Members in this topic: None.
[06:24 am]
UndeadStar -- :wob:
[12:57 am]
O)FaRTy1billion[MM] -- :wob:
[06:37 pm]
Ultraviolet -- :wob: now there's some nice :wob:s
[2021-4-18. : 3:34 pm]
jjf28 -- . . . . . :wob::wob::wob::wob::wob:
[2021-4-18. : 2:51 am]
NudeRaider -- :wob::wob::wob::wob::wob:
[2021-4-18. : 2:26 am]
Ultraviolet -- :wob::wob::wob::wob::wob:
[2021-4-18. : 1:38 am]
KrayZee -- :wob: :wob: :wob: :wob: :wob: :wob: :wob:
[2021-4-18. : 1:28 am]
CaptainWill -- I saw the same...
[2021-4-17. : 11:13 pm]
Ultraviolet -- :shifty:
[2021-4-17. : 10:21 pm]
UndeadStar -- Ultraviolet
Ultraviolet shouted: :wob::wob::wob::wob::wob:
On my browser, animations are disabled by default, and somehow, those :wob: ended up not aligned, like they're doing a wave. Didn't happen with MTiger156
MTiger156 shouted: How about x6 wob? :wob: :wob: :wob: :wob: :wob: :wob:
Please log in to shout.


Members Online: Roy, jun3hong