Staredit Network > Forums > SC1 Mapping Tools > Topic: Starcraft 64 Archive Extractor
Starcraft 64 Archive Extractor
Mar 8 2021, 8:15 am
By: Heinermann  

Mar 8 2021, 8:15 am Heinermann Post #1

SDE, BWAPI owner, hacker.

I wrote a tool that extracts all the embedded files from any Starcraft 64 ROM (in .z64 format). Files are compressed with a LZSS variant which took a couple of days to figure out. I was not able to figure out how the name hashes worked (I didn't try very hard though).

Release: https://github.com/heinermann/BOLTextract/releases/tag/v0.1
Source: https://github.com/heinermann/BOLTextract

Starcraft 64 mission briefings are not stored in the CHK, instead they are a separate text file that work similar to the credits or intermission screens. It also uses those same files instead of cinematics.

Example mission briefing:
Code
<OBJECTIVE>
-Find Infested Stukov
-Use a medic to inject Stukov with the Nanotech Serum
-Raynor and Taldarin must survive.
<PORT8>
<TEXT>
Raynor

Well, Artanis, you called and I came. What's the haps?
<PORT17>
<TEXT>
Artanis

Impetuous as ever I see. Your quick response honors me Captain Raynor. This is a time of great danger for both our people. Our old nemesis, Admiral Stukov, has returned to finish his original plans.

...*snip*


Example "cinematic":
Code
<BORDFADEUP 1>
<SLIDEFADEUP 16>
<TEXT2>Stukov:
So, Gerard, I suspect you have a good reason for pulling me away from my duties.
<WAIT 6>

<TEXTFADEDOWN>
<WAIT 2>

<TEXT2>DuGalle:
Your trivial concerns can wait, my good Alexei.  Come! Stand here.  You need to see this.
<WAIT 6>

...*snip*


All CHKs are openable in SCMDraft. There are a few 2-player UMS maps in addition to the extra campaign scenarios. I haven't taken a look at iscript/aiscript.bin since I'm not very interested in it but all the files are there. There are also "encyclopedia" files which have additional unit descriptions, it is in a custom binary format. Image format seems custom but have a similar header to old TGA images, their palettes are in separate files that pair with them. Not sure what the sound format is (or which files they are in), there is no obvious header for them. There are some other familiar formats like TBL used.


EDIT: Hah just found this old topic.

Post has been edited 1 time(s), last time on Mar 8 2021, 8:24 am by Heinermann.




Mar 8 2021, 2:31 pm Voyager7456 Post #2

Responsible for my own happiness? I can't even be responsible for my own breakfast

Quote
<TEXT>
Jim Raynor

This is critter poop. Kerrigan, are you reading this?
<PORT9>

lmao



all i am is a contrary canary
but i'm crazy for you
i watched you cradling a tissue box
sneezing and sniffling, you were still a fox


Modding Resources: The Necromodicon [WIP] | Mod Night
My Projects: SCFC | ARAI | Excision [WIP] | SCFC2 [BETA] | Robots vs. Humans | Leviathan Wakes [BETA]


Mar 8 2021, 2:42 pm IlyaSnopchenko Post #3

The Curious

I need to memorize this for... maybe a story down the road.



Who am I, and why should I care?

Mar 9 2021, 3:08 am O)FaRTy1billion[MM] Post #4

👻 👾 👽 💪

Files are big-endian (e.g. 0x12345678 is stored as 12 34 56 78)

Palette files are of the format:
Code
u16 unk1; // 0?
u16 unk2; // 0, low index?
u16 unk3; // 255, high index?
u16 pal[256];

where the color format is 5551 RGBA (described here: https://n64squid.com/homebrew/n64-sdk/textures/image-formats/#16-bit)

Image files are of the format:
Code
u16 unk1;     // 0? may be a u32 combined with the next value
u16 bitdepth; // 8, probably. I haven't seen any other value so far
u32 unk2;     // 0?
u16 width;
u16 height;
u32 unk3;     // 0?
u8 data[width*height]; // uncompressed 8-bit paletted data



I also found some bin dialog files, but they don't seem overly important to figure out ... idk how much I'll dig, the image data just happened to be uncompressed data and super easy to read.


EDIT:

Found sound files.
Code
u16 unk1; // 0x0210 ? I don't know if this is a file type ID or if it is meaningful
u16 samplerate; // 11025 Hz
u32 size;
u32 uk2; // 0 ?
u8 data[size]; // ADPCM data

I don't actually know how to decode ADPCM, I just imported raw in Audacity and selected that as the encoding. (googling N64 audio encoding is what led me to try it)


EDIT2:

Found the scripts.
14516e0\6eaf82e9.unk = scripts\bwscript.bin
14516e0\8daf821a.unk = scripts\aiscript.bin
14516e0\17aea4f9.unk = scripts\iscript.bin
at first glance it looks like they are just direct copies of the regular ones (meaning little-endian, too)


EDIT3:

A bunch of these are literally just grp files lmao
11e388e\, 12cec2c\, 1417d9a\, 1003674\, 1337846\ along with some random ones throughout the other graphics files

Post has been edited 4 time(s), last time on Mar 9 2021, 4:08 am by FaRTy1billion.



TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
\:farty\: This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

Mar 9 2021, 6:00 pm Heinermann Post #5

SDE, BWAPI owner, hacker.

Oh just FYI the folder hex strings are just the memory locations in the binary and not actual hashes. The "directories" in the layout don't actually have hashes, only the files do.




Mar 10 2021, 9:53 am O)FaRTy1billion[MM] Post #6

👻 👾 👽 💪

I assume that they are always the same though, right?



TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
\:farty\: This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

Mar 10 2021, 8:02 pm Heinermann Post #7

SDE, BWAPI owner, hacker.

No they can be different between different roms (NTSC, PAL, Beta, etc).




Apr 5 2021, 9:50 pm X405 Post #8



Quote from O)FaRTy1billion[MM]
Found sound files.

I found the names of some WAVs in my N64 executable. Here they are (with addresses):

Code
800033F4 "COMBEEP0"
80003400 "COMBEEP1"
8000340C "ASSAULT."
80003418 "COVEROP."
80003424 "DEMO.WAV"
80003430 "ENGINEER"
8000343C "INFANTRY"
80003448 "PSYCHIC."
80003454 "PYRO.WAV"
80003460 "SURVEY.W"
8000346C "ALERT.WA"
80003478 "ENDOTFH."
80003484 "CARBOMB."
80003490 "SRAW.WAV"
8000349C "FLAGTAKE"
800034A8 "GOAL2.WA"
800034B4 "GOTTAHUR"
800034C0 "OVERTIME"
800034CC "PLAYBALL"
800034D8 "PRESSSTA"
800034E4 "RETRIEVE"
800034F0 "YEE HAW."
800034FC "CAPTURED"
80003508 "YRETRIEV"
80003514 "YHASFLAG"
80003520 "YCAPTURE"
8000352C "TPWRDOWN"
80003538 "INTONYDU"
80003544 "BURROWUP"





Quote from O)FaRTy1billion[MM]
Image files are of the format:
Code
u16 unk1;     // 0? may be a u32 combined with the next value
u16 bitdepth; // 8, probably. I haven't seen any other value so far
u32 unk2;     // 0?
u16 width;
u16 height;
u32 unk3;     // 0?
u8 data[width*height]; // uncompressed 8-bit paletted data

I came to the same conclusion. There are a bunch of images with size 61456 bytes. I can't get the right palette, but those are from the slideshows that replace cinematics.





There's an interesting directory that looks like this in my ROM dump:
Code
624036 Apr  5 16:52 23626306.unk
574870 Apr  5 16:52 59a5950e.unk
82828 Apr  5 16:52 6136a2bd.unk
537406 Apr  5 16:52 634c7b0b.unk
80908 Apr  5 16:52 63f67366.unk
76714 Apr  5 16:52 6d7693bf.unk
285268 Apr  5 16:52 6d93dd02.unk
48830 Apr  5 16:52 9f3eeb82.unk
694178 Apr  5 16:52 a14c7f0b.unk
77384 Apr  5 16:52 a1c29f13.unk
88844 Apr  5 16:52 a322a763.unk
72254 Apr  5 16:52 ad62d785.unk
168838 Apr  5 16:52 d763af24.unk
569662 Apr  5 16:52 e1a5a60e.unk
551708 Apr  5 16:52 ed627b06.unk
87436 Apr  5 16:52 ede53b15.unk


It has 6 larger files and a bunch of smaller ones. I bet these are the main menu cinematics. They appear to have the following format:
Code
u32 magic; // This is always 0x02102b11
u32 size; // (the size of the file) - (the size of the header i.e. 12)
u32 unused; // This is always 0
u8 data[size]


This could be interesting, because Starcraft 64 uses a slightly different menu animation for the original Terran campaign.




Apr 6 2021, 1:26 am Heinermann Post #9

SDE, BWAPI owner, hacker.

Searches involving "Mass Media" who developed the N64 port might help with figuring out which tools and file formats they used. Blizzard was supposedly owned by "Davidson & Associates Inc" at the time.

Some of Mass Media's other games can be listed here and here on their wiki.

After checking some of their other titles, I found some other games that use the embedded BOLT archive and my program appears to work on all of them the N64 ones (likely just an endian issue).
  • Namco Museum 64 (N64)
  • Bassmasters 2000 (N64)
  • Ms. Pac-Man - Maze Madness (N64)
  • Power Rangers - Lightspeed Rescue (N64)
  • Shrek Super Party (XBox)
  • Pac-Man Collection (GBA)
  • The Lost Vikings (GBA)
  • Rock n' Roll Racing (GBA)
  • Blackthorne (GBA)
  • Namco Museum (GBA)
  • Ms. Pac-Man - Maze Madness (Dreamcast, Track 19)
  • Namco Museum (Dreamcast, Track 5)
  • The Game of Life (Windows 95!)

Power Rangers also has what appears to be the same format for cinematics with a "magic" of 0x00102b11. Namco Museum appears to have the same palette and image formats.

EDIT: Anyone looking to improve the BOLT extractor should probably look at Namco Museum because it rips into the BOLT archive immediately in the entry point, is the smallest ROM, and its compile is cleaner than Starcraft. Use one of the xbox/PC titles if you want to decompile BOLT stuff.

EDIT2: Games that use a filesystem (like XBox) use *.BLT files which are BOLT archives in roughly the same format (probably just need a little-endian version). (done)

EDIT3: Some games have some debug strings indicating function names:

Code
TermBOLT: Application terminated.
SysBOLTAlloc: Allocation failed! (size %d, flags=0x%x)
SysBOLTFree: Bogus ptr (not BOLT memory 0x%08x, size %u)
SysBOLTFree: Bad ptr (already free 0x%08x, size %d)
SysBOLTFreeTail: Bogus ptr (not BOLT memory 0x%08x, size %u)
SysBOLTFreeTail: Bad ptr (already free 0x%08x, size %d)
GetBOLTGroup: No PRELOADs


EDIT4: I've updated BOLTExtract to support little endian archives by default (haven't updated the README to be generic).

The Game of Life also has the same "magic" header for its videos so this appears to be a common format for this company at least.

Post has been edited 12 time(s), last time on Apr 6 2021, 9:31 am by Heinermann.




Apr 6 2021, 1:53 pm X405 Post #10



That's an absolutely amazing find! I already dug into "The Game of Life" (life.exe; 1.0.0.0; SHA256: 3d2acde2ad02ffeb08d8071792288c2067cd8040fe37dd3b39a85b00c53144ad). I believe I found the function that loads BOLT files at address 0x42598C and the one that closes them at address 0x425BD0.

I assume that the function that gives you a handle to a specific file within BOLT is at 0x42676C. It appears to accept a handle to a BOLT file and an uint16. The latter are most often small consecutive numbers. And I hope I'm wrong, because there's no way these are hashes. But if I'm right, then the file names cannot be recovered in the general case.




Options
  Back to forum
Please log in to reply to this topic or to report it.
Members in this topic: None.
[02:51 am]
NudeRaider -- :wob::wob::wob::wob::wob:
[02:26 am]
Ultraviolet -- :wob::wob::wob::wob::wob:
[01:38 am]
KrayZee -- :wob: :wob: :wob: :wob: :wob: :wob: :wob:
[01:28 am]
CaptainWill -- I saw the same...
[11:13 pm]
Ultraviolet -- :shifty:
[10:21 pm]
UndeadStar -- Ultraviolet
Ultraviolet shouted: :wob::wob::wob::wob::wob:
On my browser, animations are disabled by default, and somehow, those :wob: ended up not aligned, like they're doing a wave. Didn't happen with MTiger156
MTiger156 shouted: How about x6 wob? :wob: :wob: :wob: :wob: :wob: :wob:
[2021-4-17. : 1:34 am]
RIVE -- Long wob is long
[2021-4-16. : 4:27 pm]
RdeRenato -- si
[2021-4-16. : 2:12 pm]
MTiger156 -- How about x6 wob? :wob: :wob: :wob: :wob: :wob: :wob:
[2021-4-16. : 11:09 am]
NudeRaider -- :wob: UV has the longest
Please log in to shout.


Members Online: Roy, C(a)HeK, jun3hong