Extracting navigable terrain from map files (Topic)

Hey, just signed up here specifically to try to get some help on this little project I've been working on.

I'm trying to play around with some pathfinding algorithms and wanted to try them out in starcraft maps to have some actual use-case scenarios that I can test them on, and to that end I'm trying to extract the navigable tiles from a given .scm or .scx file. Using this page, I've been able to extract the TILE data from the .chk file (which I believe is the one I'll need as it looks more applicable than the MTXM due to excluding 'doodads' and I'm really not certain what ISOM does, I'm guessing pseudo 3D information used for placing the wireframe cursor?), but that just leaves me with the megatile indexes. I'm now trying to get the subtiles and their properties based on information I found on this page, but I'm not finding where that information is stored or how it relates to the megatile indexes I currently have.

I was able to find the BW tileset files with no problem in the BrooDat.mpq under the tileset folder in it, however the StarDat.mpq file I have doesn't appear to be structured anything like the BrooDat.mpq file, and as such I'm clueless where to find the StarDat.mpq tilesets. I've included the StarDat.mpq (listfile) file as an attachment for reference. I've also seen that the majority of the files in each BW tileset folder are just graphic files. I did find the dddata.bin which I'm guessing is where the subtile information with properties is located, but I don't know what format it's in or how to get that information from it.

Any information about how these files are structured, how to get the subtiles for each megatile, how to get the properties for those subtiles, or a place that would be better suited for this question (if this is the wrong place to ask) would be immensely helpful and greatly appreciated.

Attachments:

(listfile)
Hits: 1 Size: 0.39kb

Post has been edited 2 time(s), last time on Apr 7 2021, 1:36 pm by MufinMcFlufin. Reason: Didn't hit 'Attach!' when uploading (listfile), and made a few s

None.

Cartography Artisan

Since we're talking MPQs I'll just clarify that this must be about patch 1.16.1 or earlier, not SC:R

From the CHK you want to extract your tile data from MTXM, TILE and ISOM are not read by the game, only used in the editor.

When StarCraft 1.16.1 looks for asset files from MPQ it tries to get it from patch_rt.mpq, if it's not there it tries BrooDat.mpq, if it's not there it tries StarDat.mpq

I don't have all this memorized and the documentation I originally used is gone so... code dump:
How I define terrain data
How I load terrain data

Then given a tileIndex/tileValue from MTXM...
how I render a tile
how I render a tiles elevations

If C++ is appropriate for your project you can copy in the MappingCoreLib folder from Chkdraft if you wish; that should give you a jump on reading terrain out of a map and doing things with relation to the terrain asset files.

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Quote from

jjf28

If C++ is appropriate for your project you can copy in the MappingCoreLib folder from Chkdraft if you wish; that should give you a jump on reading terrain out of a map and doing things with relation to the terrain asset files.

Unfortunately it is not, as I'm working in Python, however I have programmed in C++ before so I'll give these links a good look over. If I can get something functional I'll post some code of my own for anyone else trying to do the same later on.

Edit: Apparently I've overestimated how long it's been since I last worked on C++. This is gonna be rough. x_x

Post has been edited 1 time(s), last time on Apr 8 2021, 12:26 pm by MufinMcFlufin.

None.

So correct me if I'm wrong but from what I'm understanding I need to get the ERA info for which tileset I'm using, use the megatile IDs I'm getting from MTXM, open up the corresponding tileset .cv5 file, get the 16 word megatile references for each megatile ID, and get the first bit (the walkability bit, might also play with the visibility bit later) for each reference in that tileset's .vf4 file.

Does that sound about right?

Also do you know anything about where I can find the original SC1/Non-BW tilesets? I'm not seeing them in the patch_rt.mpq or StarDat.mpq files.

None.

Cartography Artisan

I don't think that's quite right; the tileIndex you get from MTXM gets you a group (cv5 entry) and a groupIndex (array index in the cv5 entry) and the value you get back from that array in the cv5 entry is your megatile, which can then be used to index the other files

If you're using an mpq browser you might need newer listfiles

Tileset File Locations

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

PyMS and ProTRG developer

Since you are using Python, you might be able to reference some of PyMS: https://github.com/poiuyqwert/PyMS (There is CHK.py and Tilesets.py in the Libs folder)

It is quite old, so it is using Python 2.7 not Python 3.x, so you might not be able to use it directly. A lot of pieces are not the best code, most of it was written when I was not the best coder (though I do think CHK and Tilesets are some of the things I made later, so are not the worst). I have gotten much better at coding over the years but it was already too big for me to make a serious effort to improve the code (though I have started to do that a bit in a branch). But I could probably also give some help if needed.

Good luck!

Quote from

jjf28

I don't think that's quite right; the tileIndex you get from MTXM gets you a group (cv5 entry) and a groupIndex (array index in the cv5 entry) and the value you get back from that array in the cv5 entry is your megatile, which can then be used to index the other files

So (again correct me if I'm wrong) I get a tileIndex from the MTXM section, find the associated cv5 entry, get the group (first u16 in that entry), and the groupIndex (last 16 u16's), and the groupIndex is the 16 minitiles which collectively make the megatile, and the values from that 16 u16's can be directly referenced in the vf4, vr4, vx4, etc. files in the same fashion as the cv5 just with different sized entries. Sound about right?

The one thing I'm a bit confused by is the group value; that first u16 at the beginning of each cv5 entry. I'm not understanding where it comes into play here or its relevance.

Quote from

poiuy_qwert

Since you are using Python, you might be able to reference some of PyMS: https://github.com/poiuyqwert/PyMS (There is CHK.py and Tilesets.py in the Libs folder)

It is quite old, so it is using Python 2.7 not Python 3.x, so you might not be able to use it directly. A lot of pieces are not the best code, most of it was written when I was not the best coder (though I do think CHK and Tilesets are some of the things I made later, so are not the worst). I have gotten much better at coding over the years but it was already too big for me to make a serious effort to improve the code (though I have started to do that a bit in a branch). But I could probably also give some help if needed.

Good luck!

I appreciate the help. I have found a few lines that looked like they could be reworked to be more explicit about what's being done or could be more clear about what's happening, but it's otherwise good enough that it's been helping me in learning about the structure of these files.

The only things that seemed particularly bad were some while loops that appeared to effectively be for loops with extra steps, but not like I can complain much after looking back at some of my old code ~~and some of my current code.~~

None.

Cartography Artisan

Quote

get the group (first u16 in that entry)

nah, the "group" is the cv5 entry itself, I don't use the fisr u16 from the cv5 entries
tileIndex/16 = groupIndex/index of cv5 entry
tileIndex&0xF = groupMemberIndex/index of megatile within that cv5 entry

the rest sounds about right

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

So one thing I'm still confused by is something here where it says the VF4 struct is 32 bytes of 16 words which are referenced by the cv5, however it also says the maximum ID in the vf4 file is 65535 (1 word). I'm guessing I should probably ignore the 32 byte structure it's talking about and just reference the words individually. Every time I try taking the megatile reference values from the cv5 entries and referencing the vf4 as if the structure were 16 word entries I get IndexErrors, not that it makes sense anyways as I'd be getting 16 sets of flags back for one minitile.

Edit: Apparently it might be something else yet, probably about my implementation so far. I'm getting some values from MTXM that are as high as 21000.

Edit 2: I am indeed a dumbass, I just saw that part in the cv5 section about the first 3 nibbles being the group and the last nibble being the tile. Will work on implementing that now.

Post has been edited 2 time(s), last time on Apr 13 2021, 1:54 pm by MufinMcFlufin.

None.

Cartography Artisan

VF4 has those 32-byte entries per each megatile, you select one of these with that value you got back from CV5, not index a word individually, each 2 bytes describes an 8 by 8 pixel chunk of the tile (starting from top left), you need to get all 16 to "draw" or "elevate" the entire tile

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Well thanks to your help, I was finally able to get it fully functional. It's not exactly the best code, but it works. The only dependency is PIL/pillow for the output image so one could exclude it and not have to use any additional modules. I built it for 3.8 but it should work for plenty of other versions too.

I feel like with the links posted here, how we've discussed the problem, and with the code I've posted below (especially with how Python looks pretty similar to pseudo code) anyone who stumbles upon this page trying to do the same thing I did should have plenty of information to help them do it themselves. I may at some point look into opening the MPQ archive myself so I can just feed it the map archive rather than first extracting the scenario.chk file, but that's a project for later.

Thank you again for your help, I don't think I would have been able to get it functional without the information you provided.

Code Block

CHK to PNG.py

import struct
from PIL import Image

CHK_PATH = 'C:\\Users\\User\\Desktop\\Program Files\\Pathfinding\\starcraftmaps\\12 Days of SC\\scenario.chk'
TILESET_PATH = 'C:\\Users\\User\\Desktop\\Program Files\\Pathfinding\\tileset\\'
TILESETS = {
0:'Badlands',
1:'Space Platform',
2:'Installation',
3:'Ashworld',
4:'Jungle',
5:'desert',
6:'Ice',
7:'Twilight',
}

def batch(iterable, n=1):
l = len(iterable)
return [ iterable[ndx:min(ndx + n, l)] for ndx in range(0, l, n) ]

def byte_to_int( byte_li ):
return sum([ byte * (2 ** (8 * i)) for i, byte in enumerate( byte_li ) ])

def parse_chk(path):
with open( path, 'rb' ) as f:
chk_s = f.read()

data = {}
while chk_s:
# Each section of CHK file is prefixed with a 4 byte name
tag = chk_s[:4].strip(b' ')
# And a 4 byte size
size = byte_to_int(chk_s[4:8])
# Get contents of tag, assign to name in dictionary
data[tag] = chk_s[8:size + 8]
# Remove parsed section of input string
chk_s = chk_s[size + 8:]
# Reiterate until string is empty

return data

def get_dim(data):
return struct.unpack('<HH', data[b'DIM'])

def get_era(data):
return byte_to_int( data[b'ERA'] )

def get_mtxm(data):
w, h = get_dim( data )
ids = [ byte_to_int( pair ) for pair in batch( data[b'MTXM'], 2 ) ]
return batch( ids, w )

data = parse_chk( CHK_PATH )
mtxm_out = get_mtxm( data )
era = get_era( data )

with open( TILESET_PATH + TILESETS[era] + '.CV5', 'rb') as f:
cv5_s = f.read()

with open( TILESET_PATH + TILESETS[era] + '.VF4', 'rb') as f:
vf4_s = f.read()

megatile_groups = [ struct.unpack('<HBB24H', e) for e in batch(cv5_s, 52) ]
# all minitiles in a given ID's megatile
megatile_minis = [ megatile[-16:] for megatile in megatile_groups ]

minitile_flags = [ struct.unpack('<16H', e) for e in batch(vf4_s, 32) ]
# whether a given minitile is walkable
walkable_ids = [ [ flags % 2 == 1 for flags in tile ] for tile in minitile_flags ]

mini_w, mini_h = 4, 4
num_minis = mini_w * mini_h

# Creates a 2d array of 1d arrays of 16 minitiles (bool of walkable or not)
megatile_map = [ [ walkable_ids[ megatile_minis[megatile // 16][megatile & 0xF] ] for megatile in row ] for row in mtxm_out ]
# Creates 4x4 groups of minitiles from previous 1d arrays
semitile_map = [ [ [ cell[i:i + mini_w] for i in range( 0, num_minis, mini_w ) ] for cell in row ] for row in megatile_map ]
# Flattens each row and whole list from 2d array of 2d arrays into just a larger 2d array
minitile_map = [ [ cell[i][j] for cell in row for j in range( mini_w ) ] for row in semitile_map for i in range( mini_h ) ]

height = len( minitile_map )
width = len( minitile_map[0] )
map_im = Image.new( 'RGB', (width, height) )
for y, row in enumerate( minitile_map ):
for x, cell in enumerate( row ):
map_im.putpixel( (x, y), (255, 255, 255) if cell else (0, 0, 0) )

map_im.show()

None.

PyMS and ProTRG developer

PyMS includes a light weight Python wrapper for SFmpq (Files starting with SFmpq in the Libs folder). It should be extractable since it doesn't have any dependencies on the rest of PyMS, and I feel like it should work on Python 3.x with no or minimal changes but I have not looking into it. If you do try it out let me know how it goes, and if you needed to change anything, what you had to change.

Quote from

poiuy_qwert

PyMS includes a light weight Python wrapper for SFmpq (Files starting with SFmpq in the Libs folder). It should be extractable since it doesn't have any dependencies on the rest of PyMS, and I feel like it should work on Python 3.x with no or minimal changes but I have not looking into it. If you do try it out let me know how it goes, and if you needed to change anything, what you had to change.

So I gave it a look over, and there were a handful of obvious things to update, print to print(), removing unicode(), etc. but some of the less obvious (to me) changes I needed to do to get it in a semi-functional state were things like changing the _SFmpq = windll.SFmpq call to _SFmpq = windll.LoadLibrary( os.getcwd(), 'SFmpq.dll' ). I believe this has the same effect as what originally worked in Python 2.x, but I'm not certain. Besides those it's been mostly unchanged, though I'm still running into issues getting it to open any archives at all. I'm just getting ctypes ArgumentErrors whenever trying to open a file, and various other strange issues when trying to use the other functions.

Quote from

jjf28

If you're using an mpq browser you might need newer listfiles

Tileset File Locations

So I realized I never actually fixed this issue, and still can't find the original tilesets and data. I used the MPQ Editor you linked, but I still get what looks like 3000 files labeled File00000000.wav, File00000009.xxx, File00000783.pcx, etc. with no folders. I find it strange because the BrooDat.mpq works exactly as expected, but none of the other .mpq files I can find seem to be structured the same way. Do you have any suggestions for how to get the desired .chk, .vf4, etc. files?

None.

Cartography Artisan

Wasn't intending to link a mpq editor, you need the listfile download, then you need to add the Starcraft BW listfile to your MPQ session or w/e when you're trying to browse StarDat.mpq ; if you're accessing the MPQs from an API you shouldn't need the listfiles for something to work, just give it the name of the file you're searching for.

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Quote from

jjf28

Wasn't intending to link a mpq editor, you need the listfile download, then you need to add the Starcraft BW listfile to your MPQ session or w/e when you're trying to browse StarDat.mpq ; if you're accessing the MPQs from an API you shouldn't need the listfiles for something to work, just give it the name of the file you're searching for.

Ah gotcha. Didn't even notice the listfiles download at the bottom when you first mentioned it. Downloaded it and used it when opening the mpq and it worked like a charm. Thanks again!

None.