Some tries to copy TinyMap2.

1. Better WAV compression. (Used mpqrw library)
2. A little bit of protection. (Test protection. I don't know if it works perfectly;)

Can read maps with some protections. (Such as TM2... by the way, I want to know how TM2 compresses chk...)
Compress chk with TinyMap2, compress MPQ with this.

Only supports Broodwar - Use map settings.


Fix 1 : Reuploaded the file. Disabled reading of many protected maps. Duplicate section -> fail. Still can open TM2ed maps.
Fix 2 : Disabled Jump section protection - It seems to crash SC1; Use this only as a compressor. (Not enough protection)
Fix 3 : Re-enabled Jump section protection. Attachement reuploaded; sorry for just deleting it.
Fix 4 : + GUI
Fix 5 : + Something, temporarily disabled Trigger optimization. (Bug...)
+ If a file cannot be compressed more, this will put back the original one.
Fix 6 : DD2 supprt.

FIX 6 : Updated post name. This would be more appropriate.
Fix 7 : ... Where did the file go...
Fix 8 : Stability++, Compression Ratio++
- scxcomp will read more maps. (scxcomp chk reader now handles jsp)
- cf : testmap.scx is DSN

Attachments:

Post has been edited 21 time(s), last time on Apr 7 2013, 7:35 am by trgk.

Few tips for chk compression:

- Remove all unused chk sections, especially ISOM and TILE
- Zero stuff, I created a map with every tile value in MTXM set to zero, the map came out to just 5kb (attached)
- Remove anything and everything not used by starcraft (triggers with never in the condition list, comments, comment exclusive strings)
- A billion other tini things that Farty knows!

You can add some support for protection by validating VCOD, I generate a valid VCOD section using the c++ code below, the "buffer" class is simply a dynamic array of characters and "add" adds a string of the specified size to the end.



Attachments:

Post has been edited 3 time(s), last time on Feb 25 2013, 12:02 am by jjf28.

What is "Jump section protection"?

TM2 prioritized the following:
1. Remove everything that is unused.
2. If it can't be removed, then replace the data with 0s (the most common value) so that the MPQ can compress it better.

Some speicifics:

• One modification was converting all terrain values with the same image to the same value. For example, on the space tileset, "space" can have more than one value. If they were the same value, the compression ratio would be much higher.
  
• A MPQ modification could be increasing the block size, so that there are less blocks (and therefore less headers describing them), decreasing the MPQ file size. Also limit the maximum number of files in the MPQ to the exact needed number.
  
• Replace WAV strings with existing strings. For example, in your "Untitled Scenario" you play "staredit\mysound.wav". But what if you renamed the wav from "staredit\mysound.wav" to "Untitled Scenario"? The strings would be the same, and you could therefore completely remove one.
  
• Strip unused data from WAV files. There is almost always an extra, unused header, or unused data padded to the WAV file. Only include in WAV files what is necessary.
  
• For Use Map Settings, you need to really determine which units/techs/upgrades are used. For those that are unused, you completely nullify them (replace the data with 0s).
  
• Determine the game type. In melee game type, some sections aren't needed, and can be excluded to reduce file size (but also make it invalid to play the map in Use Map Settings, which can be acceptable).
  
• Push everything to its limits. Look at this map for example: http://www.staredit.net/files/2215/
  

I talked to Farty about this at one point, and TinyMap 2 does not currently do this.

Set section size to negative -> jump back.

I think that this haven't been used at any protectors yet. Though this can be broke quite easily.

Many mpq based compression has already been done in mpqrw library.

..though I still can't catch up TM2's chk compression...

In the TM2 thread you said to add it to TM2 ... I could do that. Do I just recompile with the new comp directory/source code? Is there any other settings/options?

I updated the reply and its attachment. It may be unstable. No other settings available as I remember. (It's just for 'more wav compression')
Note 1 : It don't make stereo audio to mono - I don't know how to do it correctly.
Note 2 : It modifies wave file header. Modification occurs from mpqwrite.cpp::Ln 229 ~ Ln 281. You can disable them by pasting below instead of original.


Post has been edited 1 time(s), last time on Feb 27 2013, 10:38 am by trgk.

What does it to do the wav header? Because TinyMap2, when Compress WAV is checked, will already make the WAV mono, trim the header, and delete any metadata.

Post has been edited 2 time(s), last time on Feb 28 2013, 4:48 am by trgk.

Use the codebox from the 'special' dropdown if you want to space things in a specific manner in a post

That's a very interesting suggestion actually. It might be possible to embed chk sections in other sections this way. For example, you could theoretically put FORC and OWNR INSIDE the VCOD or other unused part of the chk. This might improve compression slightly by eliminating voids of unused data (0's). Using your own dummy section to jump to different parts of the chk could come in handy. It may even be possible to overlap similar data which might have a significant impact on compression.

Again, in theory, this is a really good idea and definitely worth exploring if you're really interested in improving current chk compression techniques. Make sure you've mapped out the used and unused bits, then check to see if overlapping can occur.

Just to lay it out:


You could also put it into an infinite loop, however I don't think that would be much use since there is no break condition.

A possible use case?
Suppose you are not using any upgrade settings, then your UPGx section will have a lot of voids (what you would normally change to 0). You can embed the TECx section right inside of it. Since the value of the voids don't matter (doesn't have to be 0) you can possibly save space this way.

As I'm writing this, I found a use case for the "loop" idea. Suppose you want 30 stacked minerals. Well, by looking at the data structure, we see that the last 8 bytes of the UNIT structure for MOST units are unused. What can fit in 8 bytes? A section header! We can have ANOTHER UNIT section embedded within the current one! Ok ok, let's not get too excited, let me lay this out slowly.

We have to use 4 ideas to make this work.

• We can have duplicate, stackable, UNIT sections
  
• We can jump to any position in the chk. (the idea that was brought up)
  
• Knowing the fact that the last 8 bytes of a UNIT structure are unused in most cases.
  
• Knowing that Start Locations aren't added to the game as units (this is used to jump out of the "loop" and to maintain a fixed count prior)
  

We can get 30 with 8+8+8+6

The structure of units can be layed out as follows:


dummyStartLoc could be any start location for any player, I believe the last start location placed is used for the player, so you can place the actual start location between make6 and leave.

EDIT: To compare, the above uses 12 unit slots as opposed to the original 32.

The same can be used (more easily) for triggers and briefings, especially if you have more than two of the same trigger!

EDIT: Don't forget that section header names (might) be able to be 0 as well, so 'JUMP' would be 0, ensuring that the only non-zero bytes would be the location of the jump.

EDIT2: Credit to trgk

EDIT3: Stackable sections: UNIT, THG2, TRIG, MBRF. However, THG2 probably won't work too well.

Post has been edited 2 time(s), last time on Mar 1 2013, 10:51 pm by Heinermann.

So for UNIT this would really only work for stacked units? Or exact duplicate triggers... but wouldn't it also need dummy triggers to escape?

Will try. Thanks.

Well dummy triggers would use up a significant amount of extra space.
For triggers, it would come down to the first 8 bytes (location # and group #). If those are in use, then you can just prepend an extra condition (Always) if there is room for it.
As for the last part of it, the last 4 bytes in a trigger are for players "Unused 3, Unused 4, Non-Allied Victory, extra". Which are unused.

Actually, I am pretty sure the name of the section doesn't matter at all, so you don't have to change the first 4 bytes for a JUMP section header, meaning only the second 4 bytes (size) need to be modified. This means that we only need to change the following for each section:

TRIG/MBRF:
End Jump: First condition, group that it applies to. If necessary, reorder the conditions or insert an Always condition.
Begin Section: The last 5 player owner bytes (unused) and the current action index (which MIGHT be reset in SC, I will have to check). But also the ownership for Force 3 and Force 4. If these 2 forces are unused it will work out fine, otherwise you have to do some deep checking for players in the force (if they are all just computers, then display text, play wav, etc. are fine). If there is no other way then you would need a single dummy trigger, but then you could use that space to embed another section inside it.

UNIT:
End Jump: The x/y coordinate of the unit. Must use a Start Location.
Begin Section: The class/instance(needed for nydus and addon attachments) and unused part.

THG2:
End Jump: The Y coordinate, player number, and unused byte.
Begin Section: EVERYTHING except for the sprite/unit ID. The End Jump was already over the border, but this requirement definitely makes it completely unusable.

You can however split THG2 up to embed it in other sections.

Should only the tiny sections be put inside VCOD? I made a test build of TM2 that puts the largest sections that will fit inside VCOD, and on the map I used it came out 26 bytes larger ... It was a melee map (the easiest compression to test) and THG2, STR, and VER were put inside VCOD. (Map was picked at random. It was (4)Nightmare Station.scm.)

As of typing this post, I tested again with The Hunters. It came out 11 bytes larger, and the sections inside VCOD were STR, OWNR, SIDE, DIM, SPRP, VER, ERA, THG2, and FORC (with plenty more space).

I just thought of something ... I made it so the sections were shoved at the end of VCOD because it was a little easier ... Maybe if I put them at the beginning? It wouldn't make a difference for the Nightmare Station one because it filled all of the empty space.

Attachments:

Post has been edited 1 time(s), last time on Mar 3 2013, 3:56 am by FaRTy1billion.

In my test, putting sections in VCOD also made it larger. I'm sure there are cases not involving VCOD that would make it smaller. It would take a really good algorithm to pull it off though.

IMO putting effort into something that isn't likely to save at least 1kb of space is a waste of time.

Are you the one putting in the effort?

You can't know how much it will save (or lose) if you don't try it. And the VCOD section can fit about 1 KB of data, so even by your suggestion it was worth it to test.

EDIT:
Also not to mention TinyMap2 was made shaving off a few bytes here and there ... And look where that got us. ;o

Oh silly me for expressing my opinion on the internet.
Ok, well since it seems like it could potentially save that much, then it does seem worth trying. I've only vaguely followed what you guys were suggesting, and on the face of it it didn't seem like it would save all that much space.

Yeah, but there are still a few size-savings things that TM2 could be doing but isn't, presumably because the amount of effort isn't worth the potential savings (eg, exhaustively confirming whether a particular unit is ever placed on the map or not, and thereby zero'ing out the bytes for genuinely unused units).