Staredit Network > Forums > SC1 UMS Mapmaking Assistance > Topic: .SCX file increases in size after adding in unedited CHK file
.SCX file increases in size after adding in unedited CHK file
Mar 10 2019, 5:02 pm
By: sethmachine  

Mar 10 2019, 5:02 pm sethmachine Post #1



Hi,

I am using StormLib to open and write to MPQ archives, i.e. Starcraft's .scx files.

Simply extracting the .CHK and then adding it make to the archive increases the size of the file! I don't know where the extra bytes are coming from. I believe I'm using the appropriate flags to compress the archive too:

These are the flags I use when adding a file back to the MPQ:

Code
       flags = stormlib.MPQ_FILE_COMPRESS
       if replace_existing:
           flags += stormlib.MPQ_FILE_REPLACEEXISTING
       compression = stormlib.MPQ_COMPRESSION_ZLIB


Does Starcraft use a different MPQ compression algorithm than ZLIB, and maybe that's why the file is increasing in size?



None.

Mar 10 2019, 5:19 pm Suicidal Insanity Post #2

I see you !

SC uses (or at least used to use) PKWARE.

Have you tried calling SFileCompactArchive after modifying the MPQ?




Mar 10 2019, 5:45 pm sethmachine Post #3



SC uses (or at least used to use) PKWARE.

Have you tried calling SFileCompactArchive after modifying the MPQ?

Good call. In my API it says the compact call needs a list file. I'm not sure how to call compact without a list file, since SC doesn't use one, or do I need to add one to the archive?

python
#        Note when adding a new file name to the archive that did not exist before,
#        the list file must be updated with the new file in order to allow compaction of the archive to succeed.
   def compact(self, listfile):

       if self.mode != 'w':
           msg = 'Archive is not writable'.format(self.mode)
           self.log.error(msg)
           raise io.UnsupportedOperation(msg)
       success = stormlib.SFileCompactArchive(self.handle, listfile.encode('ascii'), 0)
       return success




None.

Mar 10 2019, 5:49 pm jjf28 Post #4

Cartography Artisan

Are we talking a one time increase over an existing MPQ or a recurring increase everytime you save?

I would guess it's the former and that would just come down to all the nitpicky little settings you're using (or your API is choosing for you) to save the MPQ, like the block sizes and whether or not you use a listfile.

This won't necessarily help with those bytes but here's some example code I'm using with StormLib...

https://github.com/jjf28/Chkdraft/blob/feature/fundamental-refactoring/MappingCoreLib/MpqFile.h
https://github.com/jjf28/Chkdraft/blob/feature/fundamental-refactoring/MappingCoreLib/MpqFile.cpp



Rs_yes-im4real - Clan Aura - jjf28.net84.net

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Mar 10 2019, 5:53 pm sethmachine Post #5



EDIT: Compact works without a list file argument. I am confused why it needs it.

Interestingly, the file is 20 KB shorter than original...

Quote from jjf28
Are we talking a one time increase over an existing MPQ or a recurring increase everytime you save?

I would guess it's the former and that would just come down to all the nitpicky little settings you're using (or your API is choosing for you) to save the MPQ, like the block sizes and whether or not you use a listfile.

This won't necessarily help with those bytes but here's some example code I'm using with StormLib...

https://github.com/jjf28/Chkdraft/blob/feature/fundamental-refactoring/MappingCoreLib/MpqFile.h
https://github.com/jjf28/Chkdraft/blob/feature/fundamental-refactoring/MappingCoreLib/MpqFile.cpp

Thanks! So I have used StormLib with Warcraft 3 files (.w3x) and when I do the proper compression/compacting, the bytes of the file don't increase (binary diff shows exact same file). So I know StormLib should work like this.

Problem is what SI pointed out--I forgot to compact the archive after adding in a new file or updating it.

However, I don't know how to compact an MPQ without a list file. .scx files apparently don't contain a list file like a Warcraft 3 file does? Or my StormLib API appears to require a list file to compact...


Huh, apparently your version of StormLib has two different compact functions:

Code
               if ( SFileCompactWithList(hMpq, filestringMpqPaths.get(), (DWORD)numAddedMpqAssets) )
                   addedMpqAssetPaths.clear();
           }
           else
               SFileCompactArchive(hMpq, NULL, false);


Mine only has SFileCompactArchive, not SFileCompactWithList.

Maybe I can pass a null as the list file and it will still work....

Post has been edited 1 time(s), last time on Mar 10 2019, 5:59 pm by sethmachine.



None.

Mar 10 2019, 5:56 pm jjf28 Post #6

Cartography Artisan

As per http://www.zezula.net/en/mpq/stormlib/sfilecompactarchive.html your listfile can be null, try this guy:

SFileCompactArchive(hMpq, NULL, false);

In my copy of StormLib I've written a handful of useful functions that allow me to do things from in-memory buffers rather than from files, SFileCompactWithList was one of those.

Post has been edited 1 time(s), last time on Mar 10 2019, 6:20 pm by jjf28.



Rs_yes-im4real - Clan Aura - jjf28.net84.net

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Mar 10 2019, 6:00 pm sethmachine Post #7



Quote from jjf28
As per http://www.zezula.net/en/mpq/stormlib/sfilecompactarchive.html your listfile can be null, try this guy:

SFileCompactArchive(hMpq, NULL, false);

Thanks! Just tried it out and it does work (shaved 20 KB too).

I am confused why would you pass the list file in at all if it's optional?



None.

Mar 10 2019, 6:07 pm jjf28 Post #8

Cartography Artisan

Having a list file is the only way that an MpqEditor can know what the name of each file inside it is (besides using an extremely costly brute force attack, or having some reference in the scenario file that definitively tells you the name of a given file). So if you were to ever open up the MpqFile in, say, Ladik's MPQ Editor, it's the difference between seeing the names of files as they were originally/as they're referenced from StarCraft like "staredit/scenario.chk" vs seeing some junky number.



Rs_yes-im4real - Clan Aura - jjf28.net84.net

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Mar 10 2019, 6:48 pm sethmachine Post #9



Quote from jjf28
Having a list file is the only way that an MpqEditor can know what the name of each file inside it is (besides using an extremely costly brute force attack, or having some reference in the scenario file that definitively tells you the name of a given file). So if you were to ever open up the MpqFile in, say, Ladik's MPQ Editor, it's the difference between seeing the names of files as they were originally/as they're referenced from StarCraft like "staredit/scenario.chk" vs seeing some junky number.

Sure that makes sense. But why is the listfile a parameter to compacting the archive?

Does the compact operation change the file names in the archive if there's no list file?



None.

Mar 10 2019, 7:26 pm jjf28 Post #10

Cartography Artisan

It doesn't change the names, when StarCraft or whatever program goes to pull a file out of an MPQ they pass some string, that string gets put through a one-way hash (which basically means it becomes a number, and the string can't be derived back from that number). That number can then be used by the MPQ APIs to pull out the file - so when you know the filename you're looking for the listfile is useless.

As far as why it's a parameter in compact that's a choice made by the original devs so we can't know for sure, if I had to guess it was to make error handling simpler/operations on the listfile more atomic - there is validation logic for the listfile in compact checking that every file also exists in the MPQ; if adding a listfile happened before calling compact there could have been file additions/removals in-between and any validation logic would have to be repeated at save time (or the listfile data left potentially invalid until save time) - by combining operations on the listfile with compact you can immediately handle any issues with the listfile, and avoid the problems of repeating code or temporarily leaving data in an invalid state.

Post has been edited 1 time(s), last time on Mar 10 2019, 7:53 pm by jjf28.



Rs_yes-im4real - Clan Aura - jjf28.net84.net

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Options
  Back to forum
Please log in to reply to this topic or to report it.
Members in this topic: None.
[03:17 pm]
Pr0nogo -- u
[02:35 pm]
lil-Inferno -- u
[02:02 pm]
Corbo -- No
[01:28 pm]
razorback9423 -- Um, is this shoutbox active?
[2019-9-14. : 11:18 am]
razorback9423 -- C.I.S. Conversion Mod will have a campaign, and the first part is coming this September.
[2019-9-13. : 12:03 am]
Wing Zero -- Memes
[2019-9-12. : 8:36 pm]
KrayZee -- I'm curious what internet archaeologists will find in Maplantis
[2019-9-12. : 5:17 pm]
MTiger156 -- Moral of the story - don't end up like Friendster
[2019-9-12. : 3:53 am]
Pr0nogo -- Voyager7456
Voyager7456 shouted: kill you opponent
Zoan
Zoan shouted: this map was boring as far as I remember lol
Please log in to shout.


Members Online: Roy, T-warp