The goal here is to determine if a given extended unit will certainly cause a crash, or could work (and should be tested!)


This gives us a hard limit, we cannot read or write past 0x006DD694.

Everything about a preplaced unit starts in its chk entry, things like its coordinates and hitpoints, can be disregarded, as they can be set freely without real cause for concern, and are pretty much irrelevant in the extended units arena. What matters to us is the player (1 byte) and unitID (2 bytes).

The player will determine color information and other limited amounts of information. Colors do not appear to be constant from computer-to-computer, and thus players should not be considered reliable for anything but player 1-12, more research on this is needed.

The unitID causes starcraft to read from units.dat to determine most other aspects of the unit. units.dat is statically loaded into memory, immediately being followed by orders.dat, sprites.dat and images.dat, a small map of how they are loaded, each one immediately after the other, is shown below, with the help of EUDDB.



Starcraft loads a unit's properties using something to this effect:



Such that if your unitID is higher than 227, it will read entries that comes after, in the case of 228, you would be using the first four bytes of ElevationLevel, whose values we know.

Thus we can predict what the properties of many extended units, and therefore we can predict whether a given unit will almost certainly crash, or is worth testing. This will require:
1) A range of units that can be predicted
2) A copy of starcraft memory in the units.dat -> images.dat range (simple enough to grab)
3) A knowledge of what ranges of a given units.dat entry will cause a crash

1) Lets assume that 0x66FBE4 and anything after it change randomly, until we know otherwise, and let us assume that AdvancedFlags (long[228], starts at 0x664080) cannot be certain values (again, until we know otherwise), this would effectively limit us to unitID's less than 11993, anything above that could potentially change on a whim, and are not worth using (at least until we determine more memory at 0x66FBE4+ is static).

2) Grabbing, will post a file when I have it.

3) Here we branch off into crazy amounts of reference work, for each units.dat entry, the relatively safe values need to be determined, this will require branching into alot of other places in memory (for example: flingy.dat) and will not be an easy task, i'll be color coding the results here, feel free to help with anything you know or your own tests.

Blue - Safe (thoroughly tested and is safe for all values)
Green - Probably Safe (prediction, has not been tested)
Yellow - Unknown Saftey
Orange - Probably not safe (prediction, has not been tested)
Red - Unsafe for given range



TODO: sfxdata.dat

TODO: Orders.dat

TODO: Upgrades.dat

TODO: Flingy.dat (and potentially its sub dat files)

I don't think you can derive anything reliable from only deducing overflows of the .dat's. Heinermann suggests that overflows happen all over the place -- even after the unit is created. Take preplaced unit order for example, that matters for some of the known EPU's.

You can get most of the units information from the .dat files, yes there will be a few overflows here and there outside of those - EUPs and the p15 scarab trick show that - but the amount of overflows not based in units.dat and its branches are comparatively very small.

In any case the goal isn't to predict that something will not crash with certainty, it's to predict that it will crash (as well as to grab its unit dimensions, should it be useful to know its collision box), and that there's no point in including it in brute force tests (and conversely that it will not crash simply due to its .dat based properties) knowing this will rule out thousands of units in a matter of seconds, as opposed to the 10s*numUnits it would take otherwise.

Having which values the dat file entries can take would also be useful documentation for general modding.

Post has been edited 3 time(s), last time on Oct 2 2013, 1:24 am by jjf28.

Justification?

Not denying that ruling out ranges is bad, perhaps the topic title could be better named to reflect that rather than one that attempts to posit more than just almost certain crash.

The unit and its graphics (units.dat and its branches, reaching all the way into the ai scripts) make up something like 150 overflows, apart from that, we have 5 known overflows (the 4 unit-score tables and the playerUnitPointer), and i'd put some money on the playerUnitPointer overflowing because of some flag or value in its dat entry. I'm not aware of any others, even un-researched ones, though i'm sure there are a few (perhaps Heinermann even has a couple named), but it still looks like A > B -- and that makes sense as that's exactly where blizzard seems to intend all of its unit-based information to stem from.

The title is accurate enough, you can predict most-all of the units properties this way (assuming the unit would show up), and I thought this was more attention grabbing

Post has been edited 1 time(s), last time on Oct 2 2013, 1:57 am by jjf28.

You're still making an ungrounded deduction. If you haven't seen the mahine code for yourself you can't make those claims based on what "seems" correct. Not to mention we're venturing in undefined territory, you'd have to know more than the blizz dev's intended.

Also memory changes. You could only derive reliable "property" information on data that's known to be static.

Not so much a deduction as an inference based on the evidence we have available to us, as well as typical design patterns, that's exactly what you do when you map EUDs or even look at the machine code, there's not going to be a deductive process leading you to know exactly what a few lines of raw assembly are doing in the grand scheme of things - you just gather up the facts and see what changes in the program occur when you change things and make the best judgement you can.


Which is exactly what .dat entries and .dat entries reading from different location in the .dat tables are, static (at least between patches), i've noted that when you start reading past those you're no longer in provably consistent territory.

Post has been edited 1 time(s), last time on Oct 2 2013, 2:17 am by jjf28.

The claim at hand is that there are much more .dat based overflows than non-.dat based overflows. No evidence has been brought forth to support this, therefore you are deducing based on what you believe should be the case. Anything can go on inside machine code -- you've only stated that you know the .dat-based overflows, and you've assumed that the other ones are much less in number.


How many design patterns exist in real-world software again?


"Mapping EUDs" is just mapping memory. Dunno what overflows have to do with mapping memory -- overflows are one way we can utilize memory.


Agreed. So let's stop deducing. Reliable "facts" only come from debugging the machine code when it comes to such a volatile overflow.


Ah. So a lot of them would overflow into other static .dat's. Could be somewhat useful then for those ranges.

But raw facts from assembly are pretty much useless until you infer something, they are just the address of the variable you're loading into a register, which register you're editing, which branches you'd take based on the state of a register, and what syscalls you're making, you have no information on what you should call a variable, or a function, or what their true purpose is in the code. These are the "facts" you gather from looking at machine code; you combine that with more facts, and you make a useful inference (ie: this function moves this unit's xc 1 to the left), but you never arrive at that deductively, you just grow the amount of facts you have to make that sort of judgement with when analyzing assembly (or even poorly written high-level language code).


It was a description of another situation in which we have 'facts' and make inferences, based on relatively low amounts of evidence, and we trust them all the same.


Yes it has, we do research on starcraft daily around SEN and BWAPI, and the overflows we've found match up ~150 to 5, which constitutes evidence (doesn't bring us to a high level of certainty, but it's enough to make the casual inference to a statement that can be made in conversation) that there are more .dat than non-.dat based overflows for extended units.

I agree with everything, so I don't know why this was all mentioned. I meant let's stop ungrounded deductions, of ocurse you have to deduce to get things done.


Ah, but in this case it's simply more controlled -- we're safe to make those deductions. It's the nature of EPU's that's the issue. EUDs are vastly more simple, we know the algorithm and pretty much everything about EUD itself. EPU's are not that simple, we don't know the "algorithm".


I'd agree if you'd bring forth evidence supporting the 150 to 5, you admit yourself that those numbers aren't exhaustive; what's the justification that it couoldn't be in reality 150 to 1500?

EUDs and EPU's are simply on different levels. EUD's on one hand are verified with reasonable certainty that it's functional -- i.e. no side effects. It simply reads or writes one DWORD -- nothing else going on there. It works in pretty much any circumstance, so it's not predicated on anything else but its input.

EPU's on the other hand are proven instantly to not have this functional characteristic, as I said with the placement order. We cannot therefore assume this functional relationship, therefore ANYTHING can be going on in the code -- we're not in the position to assume.

In summary my only thing is your claim that you know there are much more .dat overflows than non.dat ones.

Personally I think of things like stack corruption when it comes to undefined behavior like EPU's. So determining the crashability of units initially seemed a bit off, apart from always crash. Otherwise it's not a useless effort to document the .dat entries of the units that don't overflow past the .dat's.

Post has been edited 1 time(s), last time on Oct 2 2013, 3:24 am by yoonkwun.

While I have more to say, this has flown off topic, dropping the claim as it's not a productive area of discussion (establishing what constitutes strong enough evidence for a claim, and how we let potential unknowns color that claim will not lead to breakthroughs in extended unit research!)


Could you elaborate on what this is?

Proper reasoning is always a productive area of discussion. Another topic could be made presenting your evidence if you care about strict on-topicness.

Placement order is the order you place the units. The p15 trick only works when it's the first placed unit.

Oh that, doesn't necessarily need to be first depending on the behavior you're going for, it's clear that something is set based on the playerID and (at the least) animation speed is increased, and when the unit goes away that's no longer the case, but do we even have the addresses we're hitting here?

Also it seems Heinermann had this same prediction idea quite a while back: http://www.staredit.net/topic/13176/, though he didn't seem to go as far as predicting which would crash.

lol@sampling argument, like you've sampled from a statistically random pool of overflows.

"but it wouldn't make sense for them to use it all over the place"

Who said they purposefully had to "use" any overflows anywhere? Corrupt data in this fashion can lead to overflows noteven intended by Blizzard. How can you posit otherwise? Think about corrupt pointers and corrupt stack.

Post has been edited 1 time(s), last time on Oct 2 2013, 6:08 pm by jjf28.

Your argument would be valid if you sampled a randomly distributed population, or whatever analogue of it. You've only stated the ones you know -- how do you know your sample is representative of the population

Do you know what stack corruption is?

Post has been edited 2 time(s), last time on Oct 2 2013, 6:49 pm by jjf28.

I'm not positing anything, I'm responding to a position that I don't think is grounded.

What is your justification that your sample is representative -- the fact that you search for them regularly? That doesn't qualify as a justification. Your searching methods could be biased, or otherwise your method of searching simply isn't exhaustive. If I'm wrong, then provide evidence.

I think of stack corruption as an example of when data goes corrupt, you cannot assume anything. e.g., the return address gets overwritten to some random place, who the hell knows what will go on from there, corruption after corruption.

Are EPU's stack corruption? No. Could it exhibit similar chaotic corruption? Yes.

Corruption can cause overflows. Corruption creates non dat based overflows. Your sequence there has no grounds.

You made the claim, you have the burden of proof. "Fairies exist, what kind of evidence are you asking for?!"

Exactly, ungrounded axioms get nobody anywhere.

Epd's aren't as controlled as a simple function call, invalid illustration