Hi,

I am using StormLib to open and write to MPQ archives, i.e. Starcraft's .scx files.

Simply extracting the .CHK and then adding it make to the archive increases the size of the file! I don't know where the extra bytes are coming from. I believe I'm using the appropriate flags to compress the archive too:

These are the flags I use when adding a file back to the MPQ:



Does Starcraft use a different MPQ compression algorithm than ZLIB, and maybe that's why the file is increasing in size?

SC uses (or at least used to use) PKWARE.

Have you tried calling SFileCompactArchive after modifying the MPQ?

Good call. In my API it says the compact call needs a list file. I'm not sure how to call compact without a list file, since SC doesn't use one, or do I need to add one to the archive?

Are we talking a one time increase over an existing MPQ or a recurring increase everytime you save?

I would guess it's the former and that would just come down to all the nitpicky little settings you're using (or your API is choosing for you) to save the MPQ, like the block sizes and whether or not you use a listfile.

This won't necessarily help with those bytes but here's some example code I'm using with StormLib...

https://github.com/jjf28/Chkdraft/blob/feature/fundamental-refactoring/MappingCoreLib/MpqFile.h
https://github.com/jjf28/Chkdraft/blob/feature/fundamental-refactoring/MappingCoreLib/MpqFile.cpp

EDIT: Compact works without a list file argument. I am confused why it needs it.

Interestingly, the file is 20 KB shorter than original...


Thanks! So I have used StormLib with Warcraft 3 files (.w3x) and when I do the proper compression/compacting, the bytes of the file don't increase (binary diff shows exact same file). So I know StormLib should work like this.

Problem is what SI pointed out--I forgot to compact the archive after adding in a new file or updating it.

However, I don't know how to compact an MPQ without a list file. .scx files apparently don't contain a list file like a Warcraft 3 file does? Or my StormLib API appears to require a list file to compact...


Huh, apparently your version of StormLib has two different compact functions:



Mine only has SFileCompactArchive, not SFileCompactWithList.

Maybe I can pass a null as the list file and it will still work....

Post has been edited 1 time(s), last time on Mar 10 2019, 5:59 pm by sethmachine.

As per http://www.zezula.net/en/mpq/stormlib/sfilecompactarchive.html your listfile can be null, try this guy:

SFileCompactArchive(hMpq, NULL, false);

In my copy of StormLib I've written a handful of useful functions that allow me to do things from in-memory buffers rather than from files, SFileCompactWithList was one of those.

Post has been edited 1 time(s), last time on Mar 10 2019, 6:20 pm by jjf28.

Thanks! Just tried it out and it does work (shaved 20 KB too).

I am confused why would you pass the list file in at all if it's optional?

Having a list file is the only way that an MpqEditor can know what the name of each file inside it is (besides using an extremely costly brute force attack, or having some reference in the scenario file that definitively tells you the name of a given file). So if you were to ever open up the MpqFile in, say, Ladik's MPQ Editor, it's the difference between seeing the names of files as they were originally/as they're referenced from StarCraft like "staredit/scenario.chk" vs seeing some junky number.

Sure that makes sense. But why is the listfile a parameter to compacting the archive?

Does the compact operation change the file names in the archive if there's no list file?

It doesn't change the names, when StarCraft or whatever program goes to pull a file out of an MPQ they pass some string, that string gets put through a one-way hash (which basically means it becomes a number, and the string can't be derived back from that number). That number can then be used by the MPQ APIs to pull out the file - so when you know the filename you're looking for the listfile is useless.

As far as why it's a parameter in compact that's a choice made by the original devs so we can't know for sure, if I had to guess it was to make error handling simpler/operations on the listfile more atomic - there is validation logic for the listfile in compact checking that every file also exists in the MPQ; if adding a listfile happened before calling compact there could have been file additions/removals in-between and any validation logic would have to be repeated at save time (or the listfile data left potentially invalid until save time) - by combining operations on the listfile with compact you can immediately handle any issues with the listfile, and avoid the problems of repeating code or temporarily leaving data in an invalid state.

Post has been edited 1 time(s), last time on Mar 10 2019, 7:53 pm by jjf28.