Sunday 6 September 2009

Getting My Geek On

Well, well, well, the medium laser looks like it is pretty well balanced, but in my experience of playing BattleTech it is the "best bang for the buck" in 3025. Other eras not so much, but I don't play other eras, as quite frankly I find I don't enjoy the games anymore than the 3025 period game, and they are a lot more faff to play.

Now it is not enough to make an assertion, like the one I've just made above, unless one can support it with some evidence.

So what evidence can we get to disprove my assertion? One sort of evidence would be to log a sufficiently large number of game results where you pitted medium laser armed mechs against a more balanced force of mechs armed with different combinations of weapons.

However, the problem with doing so is that mistakes made by the players during the game would add what are called "confounding variables."

Also, we would need to know how many games would have to be played to have confidence in the power calculation. I am not a statistician, but I know that in research trials you are generally talking 40 or more participants per condition, where you normally compare a control sample to the trial sample, with randomised allocation to either.

Such research trials are not done for fun, cost an arm and a leg to run, and often turnout inconclusive, or with a low value for the correlations that are being tested for. I don't have those resources to bring to bear on this problem.

What I can do is a non-randomised "thought experiment" to test my hypothesis that the medium laser is too powerful.

My null hypothesis is that the combat value I calculated is correct.

For my thought experiment I'm going to start by comparing three different weapons that I've calculated a combat value for. So I'm choosing the autocannon 20 (AC20), and the particle projection Cannon (PPC).

The reason for choosing these two BattleTech weapons is that received opinion is that they are good at leveling out the inherent advantage I perceive the medium laser as having. What I hope to do is see, if my hunch is correct? Is the medium laser is the best hammer, and if it is, to show why?

If not then then maybe we might find out something more interesting and prove that the combat value I calculated in my previous post is right.

However, as I said above we need to reduce the "confounding variables" as much as possible, because we are not interested in the luck of the dice, or the quality of the players. I only want to know if the weapon is costed correctly by the combat value algorithm I constructed.

Other "confounding variables" would be movement, terrain, pilot skill advantages, and range.

All of these things make for a fun game of BattleTech, but being able to roll a double-one on and get a centre torso critical hit, or rolling a double-six and blowing the head off only tells us that one can be lucky with the dice, not how effective the weapons is.

Now what I'm going to do for my "thought experiment" is level the playing field and remove all the tactical considerations that are the "confounding variables."

Imagine, instead of a game of combat between two or more people, we are going to play a game of firing at a static target. In real life the military runs various trials and live fire exercises. We can imagine that Inner Sphere military doing some test shots against a static target for the same sorts of reasons.

The next thing we need to decide is how many shots to fire for each test?

Now during a game we are not likely to have a large number of combat rounds, but the problem with taking data from too small a data-set is that statistical errors from rounding out the results will adversely affect the correlations between the various factors that make a weapon effective.

This could be to either the advantage or disadvantage of any party, and we don't want that. We want to control for that too.

As I said earlier, in real life randomised control trials one generally has around 40 participants in each condition, but this can be lower, or higher depending on a "power calculation" which is made before the trial begins by a statistician.

I'm not a statistician, so I'm going to choose a number that will make it easier for me to calculate the percentages at the end of the trial. Therefore I'm going to say that each weapon will fire 36 shots at the target, because BattleTech uses two six sided die (abbreviated as 2D6 from now on) to generate random numbers, and if we roll enough times, then over a large enough number of rolls the average number of results will tend towards a normal distribution.

Now 2D6 have a Bell curve and what that is looks like this:

02 = 1 + 1

03 = 2 + 1, 1 + 2

04 = 3 + 1, 1 + 3, 2 + 2

05 = 4 + 1, 1 + 4, 3 + 2, 2 + 3

06 = 5 + 1, 1 + 5, 4 + 2, 2 + 4, 3 + 3

07 = 6 + 1, 1 + 6, 5 + 2, 2 + 5, 4 + 3, 3 + 4

08 = 6 + 2, 2 + 6, 5 + 3, 3 + 5, 4 + 4

09 = 6 + 3, 3 + 6, 5 + 4, 4 + 5

10 = 6 + 4, 4 + 6, 5 + 5

11 = 6 + 5, 5 + 6

12 = 6 + 6

So, an AC20 weighs 14 tons, and generates 7 heat points every time it fires. It will need 7 tons of ammo for 35 shots, and as that it close enough, it will do, because I'm going to make an assumption that three shots miss the target, and that one of them is the empty chambered shot.

Does that sound fair and reasonable to everyone?

Therefore the AC20 battlemech will have 14 tons for the weapon, and 7 tons for the ammo for a total of 21 tons. The mech will run cool at -3 heat points per round, which is good.

Now I will bring up a PPC armed battlemech with two PPCs that weigh 7 tons each, for a total of 14 tons. To make things fair, we will make sure that the battlemech carries 7 extra heat sinks, for a total payload of 21 tons.

However, the PPCs will generate 20 heat points per turn, so the battlemech will run at +3 heat. As we are comparing damage down range for our test the battlemech will fire both PPCs for two rounds, and allow itself to cool down from the +6 heat by only firing one PPC the next round.

Finally, I bring the medium laser armed battlemech onto the firing line and again we will have 21 tons of payload.

A battlemech has 10 heat sinks, so the first 3 medium lasers will only cost us 3 tons of payload, with 1 heat sink left over. The next medium laser will need 2 heat sinks to be heat neutral for a total of 3 tons, which means we have spent 6 tons so far. 

We have 15 tons left, and each medium laser we now want to put in our battlemech will cost us 4 tons; 1 for the medium laser plus 3 for the heat sinks to keep it nice and frosty. So we can have 3 more medium lasers for 12 tons, and still have 3 tons of unused payload, which for the hell of it we will use for more heat sinks, just because we can. 

This battlemech has 7 medium lasers, does 35 points of damage and also runs at -3 heat points (hopefully you see my cunning plan unfold, because I could have chosen to add another medium laser, but instead I will make this and the autocannon armed battlemech run at the same -3 heat points?

This is because I have another theory I want to test later and like any good researcher I'm starting one trial with the intent of a follow-up one).

Now since we are playing BattleTech, we have to throw dice to hit anything, so as our thought experiment was constructed with the idea of firing 36 times. I'm going to assume that we get a perfect standard distribution of the die rolls.

By that I mean, one double 6, one double 5, one double 4, and the equivalent sets of 6 & 1, 1 & 6, 5 & 2, 2 & 5 and so on. 

This isn't at all realistic, but after serious peer review on a BattleTech forum the consensus was that it does represent the ideal and is therefore a fair way of working out the damage a weapon does in the ideal situation. And remember all the weapons in this test are treated fairly, well mostly fairly as I will lean over backwards to not favour the medium laser, and therefore undermine the results.

So for the first set of 36 shots we are going to fire our battlemechs at short range. We need 4 plus to hit, and therefore we get 33 hits at short range. 

For the purpose of the test the PPC battlemech is standing at its short range of 4 hexes (otherwise it would be shooting at a plus 1 penalty, and it really serves no useful purpose to penalise the PPC), while the AC20, and medium laser battlemechs are standing at 3 hexes from the target.

The results of the short range firing test are as follows:

The PPC armed battlemech delivers 33 hits for a total of 550 points of damage.

The AC20 armed battlemech delivers 33 hits for a total of 660 points of damage, with a loud huzzah.

The Medium laser armed battlemech delivers 33 hits for a total of 1155 points of damage, and people are cheering and booing, saying not fair.

Anyway, order is restored and the battlemechs are reversed out to be at medium range to the targets, but the  PPC crew argue that they should be allowed to shoot at 6 hexes rather than 7 hexes, because they can. The referees agree with them so the PPC will fire at 4 plus to hit getting 33 shots on target.

However the to hit numbers are now 6 plus to hit for the autocannon 20, and the medium laser contenders, so they will only hit the target 26 times out of 36 with the same perfect standard distribution of the die rolls.

The results of the medium range firing test are as follows:

The PPC armed battlemech delivers 33 hits for a total of 550 points of damage, with a big yay all round.

The AC20 armed battlemech delivers 26 hits for a total of 520 points of damage, with a mumble of that's not too bad is it?

And the Medium laser armed battlemech delivers 26 hits for a total of 910 points of damage, and the generals are in an uproar.

After a lot of very pointed and heated discussions, another round at long range is set up, with the targets now at 9 hexes from the battlemechs.

This time the PPC battlemech is also set at 9 hexes, but fires at medium range, which is still 6 plus to hit, which gives it 26 hits on the static target. Meanwhile while both the other battlemechs now fire at 8 plus to hit, which will give them both 15 hits on their static target.

The results of the long range firing test are as follows:

The PPC armed battlemech delivers 26 hits for a total of 480 points of damage, with a resounding cheer.

The AC20 armed battlemech delivers 15 hits for a total of 300 points of damage, with a mumble of it could have been worse?

The medium laser armed battlemech delivers 15 his for a total of 525 points of damage, and now everyone is up in arms at the results.

How can it be so obvious, why has no one ever seen this before?

The PPC crew say that it is not fair as they can still hit the target at a longer distance. While the AC20 crew say, but you are forgetting that the bigger the hammer the harder it hits. The medium laser crew look around and whistle quietly to themselves.

A decision is made by all and sundry to send the results to a review committee who will look at all the evidence and issue a report in due course.

However, peer review of the above by the committee raised two clear objections. The first was that no-one in their right mind would field a battlemech carrying 7 tons of ammo for its AC20. The second that 3 tons of ammo would be fairer, with a sub-note that since this is a firing range trial why can't the AC20 just carry 1 ton of ammo and be reloaded each time it become empty?

So let's see what happens when we adjust the firing test to take onto consideration the objections of the review committee? We will do another round of static firing with a special AC20 with 3 tons of ammo, total payload weight of 17 tons versus the medium laser contender.

Now for 17 tons the medium laser battlemech can have 3 medium lasers for 3 tons with one spare heat sink left over, and 14 tons to spend on more weapons and heat sinks. The next medium laser will cost 3 tons, because it can use the remaining heat sink, leaving 11 tons to spend. Now each remaining medium laser will weigh 4 tons if it is to remain heat neutral, so that gives us two more medium lasers.

However, with 3 tons left we can buy one more medium laser with only 2 heat sinks and run at plus one heat point per turn. This gives the medium laser armed battlemech a total of 7 medium lasers running at 1 heat point per turn, which seems only fair given what the committee have asked as if you are going to deliberately rig the jig, then having the the medium laser battlemech rigged this way seems fair too?

We will run two sets of 15 shots for the AC20 for its 36 shot trial, as it doesn't have enough ammo to fire every turn, which is a disadvantage of the autocannon. Meanwhile the medium laser will fire 8 times in a row and then pause for one firing round, which means it will only fire 32 times out of the 36 shot trial.

Short range 4+ to hit (assuming 3 misses on empty chamber for the AC20 & just 3 misses for the medium laser):

The AC20 will do 600 points of damage versus 660 points of damage over 33 shots. The 7 medium lasers will do 1015 points of damage versus 1155 points of damage over 33 shots.

Medium range 6+ to hit (30 shots fired 10 miss for both):

The AC20 will do 400 points of damage versus 520 points of damage over 33 shots. The 7 medium lasers 700 points of damage versus 910 points of damage over 33 shots.

Long range 8+ to hit (30 shots fired 20 misses for both):

The AC20 will do 200 points of damage versus 300 points of damage over 33 shots. The 7 medium lasers 350 points of damage versus 525 points of damage over 33 shots.

I surmised that the AC20 would not really be hindered by carrying 7 tons of ammo as carrying less ammo directly correlates with being able to do less damage, which is another "confounding variable" for ammo using weapons versus energy weapons that get unlimited shots.

However, if one divides the number of points of damage the AC20 does into the number of points that the 7 medium lasers do, we can calculate a ratio to compare the effectiveness of both weapons at dealing out damage.

Apart from at short range, where the AC20 does 1 point of damage for every 1.69 that medium laser does when carrying 3 tons of ammo, at both medium and long range it does 1 point damage to 1.75 points that the medium laser does, whether it is carrying 3 or 7 tons of ammo.

Surprising huh?

The difference between 1.69 and 1.75 is less than 5%, so it suggests that the amount of ammo I used in my "thought experiment" while not indicative of a typical load out, only affects the results by less than 5%.

That is actually a very good result for my argument that the 3025 medium laser is over powered.

Finally, what about the AC20 with 1 ton of ammo argument? The best comparison between the medium laser and AC20 happens when the AC20 only has 1 ton of ammo where the ratio of damage is 1:1.5 in favour of the medium laser.

My initial thoughts were that by giving the AC20 7 tons of ammo I would reduce its effectiveness by less than 20%. As the best ratio for the medium laser is 1:1.75 versus the AC20, what we have is approximately a 15% increase in the effectiveness of the medium laser over the AC20 when you give the AC20 a ridiculous amount of ammo.

For me the question then becomes what is happening here? 

The answer would seem to lie with the construction rules giving each mech 10 free heat sinks versus the weight penalty for carrying ammo in relation to the tonnage of extra heat sinks that the medium needs.

However, that is for a future blog. I do hope that by being amusing I've made an otherwise number crunching exercise a bit less dry?

And this is why the medium laser rocks.

Disclaimer: All posts are condensed & abbreviated summaries of complex arguments posted for discussion on the internet, and not meant to be authoritative in any shape, or form on said subject, T&CA, E&OE & YMMV.

7 comments:

  1. You are indeed correct that the ML does quite well from the 10 free heat sinks, however there are problems with it as well.

    For instance, the AC20 can take off a head in a single shot. ML's are hard pressed to do so, and if mounted in numbers where they can compare to the AC20 run up high heat values.

    Likewise the PPC has much longer range which translates to a significant advantage. Keep in mind that while you are at range 9 firing with +4 on your to hit numbers the PPC only has a +2 and can easily back up to range 10, 11, or 12 to stay out of range.

    To be sure the Medium Laser is powerful, but you need to look at the whole package. It is more than just raw damage or damage per heat. It is also penetration ability and overall range, plus accuracy over range.

    ReplyDelete
  2. An interesting test. It would be intriguing to see the damage values over every hex to gauge how much the minimum range and longer range effect the PPC; Also doing a direct comparision between the AC10 & PPC would also be interesting as these weapons are fairly equilivent in effective range and damage.

    ReplyDelete
  3. Hi Pip! Nice analysis. Are you sure you aren't a statistician? ;-)

    A few comments tonight, maybe more when I'm not sleepy-

    This is essentially a "bench test" analysis, leaving out the details of how the game is played, and tearing apart the game mechanics to see how it works. This is a basic and sound approach.

    If anything you are underselling the medium laser. In play, one critical hit to the AC20 or its ammo remove it from the game. Medium lasers are much more survivable.

    I also want to take a whack at the "head chopper" myth of the AC20 (Sorry Kit!). It's spectacular when it you roll a "12" and takes the head off, but it still only happens once in 36 hits. In the same amount of time 7 medium lasers will deliver 7 head hits (on average), enough to take the heads off two mechs. Even with 4 MLs, average the same damage as an AC20, you will average 33% more head-chops.
    Clan PPC's are not much better from this standpoint, but can head-chop from a safe distance, which is a big advantage.

    ReplyDelete
  4. "This is a basic and sound approach." ... that I often use myself. By controlling for mass and heat, you are doing a sort of cost-benefit analysis too.

    You could do this in the "clinical trial" sense by actually rolling the dice 40+ times for each weapon under the same conditions. This gives you a random sample to analyze statistically. Rolling a lot of dice this way is better suited to a computer, and then you can roll as many dice as you want to improve the accuracy (also called Numerical Simulation).

    ReplyDelete
  5. My experience with the various weapons systems has been that each weapon is good for a particular mission profile.

    The AC/20 is designed to take armor off heavily-protected targets as quickly as possible. Head shots in this instance are a rare cherry on the top. I never count on them or let them influence my choice of weapons. In some cases, lightly-armored targets can be completely disabled or even destroyed with a single shot. Even when neither happens, you have forced a piloting roll.

    The PPC has range, is independent of ammo and does enough damage to go internal on a head hit - or do significant damage in any case. It is relatively frugal with critical slots and the heat can be managed.

    The Medium laser is excellent when you are looking for the best bang for your short-ranged buck. Especially useful in batteries after the big guns have reduced an opponent's armor and the foe is in range. Multiple shots generally results in multiple hits, which translates to multiple chances for a critical if you've been doing it right.

    Again, each weapon has its own application, and it really depends on the platform's mission profile...

    ReplyDelete
  6. EastwoodDC: Yes 7 ML will roll 7 head hits in the same amount of time, however keep in mind that they also produce 21 heat - something you will be hard pressed to deal with (the total weapon package of 7 ML will weigh you 18 tons).

    Looking at the more rational 4ML's, yes they will head chop more often but the AC20 also punches big fat holes in everything, something the ML's cannot do. My point was that if the AC20 hits anywhere it is crippling, quite possibly lethal. In any single round the AC20 will deal much more trauma to a unit than 4 ML's.

    This becomes more obvious when you look at 'Mech armor profiles. Until you get into the Heavy 'Mech range you rarely see more than 20 armor in a location. Taking into account other combat damage even heavier combat units typically take an AC20 and suffer internal damage. That is what you are really paying for with the AC20.

    ReplyDelete
  7. Hey, thanks for all the comments guys. And I will confirm again for the record that I'm not a statistician, if I were I might well do a "clinical trial" but I'm happy with my "bench testing" as the goal here is to come up with an algorithm, which I will then trial.

    My day job involves a lot of risk and cost benefit analysis, which is why I know some statistics.

    Oh in case anyone thinks I think the medium laser is totally wonderful, let me say it is not. However, I do like autocannons and feel that they do get very short shrift in Battletech, even though there are better versions introduced for 3050 and on games.

    ReplyDelete