Gaphorisms: Guerrilla Aphorisms

CaP bombs you can drop in team meetings or use as party favors

Updated on Jan 5, 2021

1. This online edition updates The Guerrilla Manual originally included as a pull-out booklet with the initial printings of the Guerrilla Capacity Planning book.
2. Spontaneous Gaphorism candidates also appear regulary on Twitter.

Contents

1  WEAPONS OF MASS INSTRUCTION

1.1  Why Go Guerrilla?

1.2  Best Practices

Mindlessly following the best practices defined by others is actually a failure to understand your own particular requirements for performance and capacity. If you don't understand your own particular performance and capacity problems, why would you expect other people's practices (i.e, solutions) to be appropriate for you? Copying someone else's apparent success is like cheating on a test. You may be lucky and make the grade this time, but will that bluff still work in the future?

A recent NPR program discussed how applying best practices has backfired in the arena of medicine and healthcare. This analogy is very appropriate because many aspects of performance analysis are not too different from the science of medical diagnosis. See the section on The Limits Of Best Practices (NPR Sept 21, 2011).

11. Avoid acceptance testing like the plague.
12. Develop on a platform totally unrelated to the production system.
13. Never compare load test requirements and results with actual remote user-based measurements from Keynote, Gomez, etc.

1.3  Virtualization

Translation: We performance weenies need more whistles and less bells. In other words, virtualization vendors need to make sure they provide us with backdoors and peepholes so we can measure how resources are actually being consumed in a virtualized environment.

1.4  Selling Prevention is Hard

Ultimately, it's not what you sell, but how you sell it.

1.5  Why is Performance Analysis Hard?

1.6  Brisk vs. Risk Management

"I can understand people being worked up about safety and quality with the welds," said Steve Heminger, executive director ... "But we're concerned about being on schedule because we are racing against the next earthquake."

1.7  Failing On Time

Until you read and heed this statement, you will probably have a very frustrating time getting your performance analysis conclusions across to management.

Update: As of May 2013 the original high-risk Caltrans decision has prompted Gov. Jerry Brown to threaten delaying the scheduled Labor Day opening of the new Bay Bridge span. Erm... so, how did this brisk management decision save time (and money)?

Update: On September 2, 2013, the new East Span of the San Francisco-Oakland Bay Bridge opened on schedule to traffic.

Update: May 16, 2014. Caltrans acknowledges that it could take months and millions of dollars to fix problems. Again. How did the preceding brisk management decision save time and money?

Update: May 6, 2015. Bay Bridge ``Project From Hell'' Develops New Problem There appears to be a new problem with the new eastern span of the Bay Bridge, so bad that the Executive Director of the Metropolitan Transportation Commission Steve Heminger said Wednesday, "This is the project from Hell."

Update: June 4, 2015. Corrosive salty water threatens 120 Bay Bridge anchor rods Caltrans' chief bridge engineer on the project, Brian Maroney, said Thursday that the cost of testing the foundation and the rods could end up being as much as $10 million. The project is running about $50 million in the red.

1.8  Performance Homunculus

• Security management
• Software distribution
• Backup management
• Chargeback management
• Cap and Perf management

Cap and Perf management can rightly be regarded as just a subset of systems management, but the infrastructure requirements for successful capacity planning (both the tools and knowledgeable humans to use them) are necessarily out of proportion with the requirements for simpler systems management tasks like software distribution, security, backup, etc. It's self-defeating to try doing capacity planning on the cheap.

1.9  Self Tuning Applications

Think about it. Performance analysis is a lot like a medical examination, and medical Expert Systems were heavily touted in the mid 1980's. You don't hear about them anymore. And you know that if it worked, HMO's would be all over it. It's a laudable goal but if you lose your job, it won't be because of some expert performance robot.

1.10  Squeezing Capacity

Today, there is a serious need to squeeze more out of your current capital equipment.

1.11  When Wrong is Right

The planning part of capacity planning requires making predictions. Even a wrong prediction is useful because it can serve as a warning.
Either:

1. the understanding that underlies your predictive model is wrong
2. wrong data is subverting your predictions (GIGO)

1.12  Throw More Hardware at It

Single-threadedness can wreck you

1.13  Network Performance

If your local network is out of bandwidth, has interminable latencies, or is otherwise glitching, don't bitch about the performance of your application. Get the network fixed first, and then we'll talk about the performance of your application.

1.14  Not even wrong!

1.15  Measurement vs. Models

Most people remain blissfully unaware of the fact that all measurements come with errors; both systematic and random. An important performance analysis task is to determine (and track) the magnitude of the errors in your performance data.

Every performance datum should come with a `±' attached, which would force a number to appear after it, e.g., 136 ± 16. Obviously, this would take more space, but there are ways around that issue, e.g., display on mouseover. No commercial or open source performance tools, that I am aware of, do anything like this.

> rt.ms
[1]  92 120 172 162 178  93

> mean(rt.ms)
[1] 136.1667

> sd(rt.ms)/sqrt(length(rt.ms))
[1] 16.10055

1.16  Data Are Not Divine

Charlton Heston laying down the laws of queueing theory (Erlang was just a fame bot)

Data comes from the Devil, only models come from God.

Corollary: That means it's helpful to be able to talk to God. But God, she does babel, sometimes. :)

1.17  Busy work

"Seeming to do is not doing."

Nikola Tesla visited Henry Ford at his factory, which was having some kind of difficulty. Ford asked Tesla if he could help identify the problem area. Tesla walked up to a wall of boilerplate and made a small 'x' in chalk on one of the plates. Ford was thrilled, and told him to send an invoice. The bill arrived, for $10,000. Ford asked for a breakdown. Tesla sent another invoice, indicating a $1 charge for marking the wall with an 'x', and $9,999 for knowing where to put it.

1.18  Little's Law

1. Little's BIG law:
   

   Q = λR

   (1)

   which relates the average queue length (Q) to the residence time R = W + S. Here, W is the average time you spend waiting in line to get your groceries rung up, for example, and S is the average service time it takes to ring up your groceries once you get to the cashier.
2. Little's little law:
   

   ρ = λS

   (2)

   which often goes by the name Utilization law. It relates the average utilization (ρ), e.g., of the cashier, to the service time (S). Equation (2) is derived from (1) by simply setting W = 0 on the right-hand side of the equation. In both equations, the left-hand side is a pure number, i.e., it has no formal units (% is not a unit).

1. Eqn.(1) tells us the average number of customers or requests in residence.
2. Eqn.(2) tells us the average number of customers or requests in service.

I use it almost daily as a cross-check to verify that throughput and delay values are consistent, no matter whether those values come from measurements or models. Another use of Little's law is calculating service times, which are notoriously difficult to measure directly. If you know the values of ρ (utilization) and λ (throughput), you can calculate S (service time) using eqn.(2).

1.19  Bigger is Not Always Better

This plot was improved after a suggestion by Guerrilla grad Mohit Chawla (@a1cy)

The bigger your multicore configuration, the busier you need to make all those cores. Otherwise, you'll have a lot of wasted CPU capacity. But you should only run them up to the point where the run-queue suddenly begins to grow: the multicore wall. Any busier and the application response time will suddenly start to skyrocket. Users generally don't like that and may defect.

1.20  Bottlenecks

In any collection of processes (whether manufacturing or computing) the bottleneck is the process with the longest service demand. In any set of processes with inhomogeneous service time requirements, one of them will require the maximum service demand. That's the bottleneck. If you reduce that service demand (e.g., via performance tuning), then another process will have the new maximum service demand. These are sometimes referred to as the primary and the secondary bottlenecks. The performance tuning effort has simply re-ranked the bottleneck.

1.21  Benchmarks

The purpose of competitively benchmarking a computer system is to beat the competition based on some predefined performance metric, e.g. throughput or response time. Then, the vendor can make a "mine is bigger than yours" claim. It's a zero-sum game that is otherwise the IT equivalent of war! Here, competitive refers to benchmarks used by vendors, as opposed to benchmarking that you might do for purely internal comparisons or diagnostic purposes.

Well-designed benchmarks should always come with rules for setting up and running the benchmark workload. Those rules are intended to level the playing field amongst vendors and make the performance results more comparable for you, the evaluator. But rules were made to be broken (or bent), especially in warfare—just don't get caught. And how might a vendor get caught? Industrial-strength benchmarks that are designed by organizations like SPEC.org and TPC.org have technical review committees that scrutinize all vendor-submitted benchmark results for cheating, as well as any unintended corner-cutting. If any violations are found to be serious enough, vendors may be forced into resubmitting their results or withdrawing them entirely. Because of the many opportunities for cheating, benchmark "cops" are a necessary part of the game. Any competitive benchmark that does not require official auditing (the polite term) is not worth the code it's written in.

Another form of cheating is where a vendor defines and runs their own benchmark based on the rationale that "we know best how to demonstrate our performance." This stance is analogous to proprietary vs. public software. Such clandestine benchmarks are guaranteed to make the vendor king of the performance hill since, (i) by definition there can be no other vendor to compare against, and (ii) there is no way to scrutinize how the benchmark was designed to favor that particular vendor. Since they're clandestine, all such benchmarks can promptly be ignored. In fact, that's why organizations like, the Standard Performance Evaluation Corporation, the Transaction Processing Performance Council and the Storage Performance Council, although highly imperfect, were formed in the first place.

All this skulduggery, however, doesn't mean that competitive benchmarks are completely useless. Far from it. Such performance information may be the only data publicly available and can therefore be used by you for comparative system sizing and capacity planning estimates. You just need to be aware of the assumptions and loopholes in published benchmark results and adjust accordingly. I show you how to use competitive benchmark data (along with some personal benchmarking war-stories) in my Guerrilla Capacity Planning class.

1.22  Failure to Communicate

"What we have here is failure to communicate." —Prison warden in the movie Cool Hand Luke.

Question:

How long should you spend creating your presentation?

RoT:

Spend as much time developing your presentation as did developing your analysis.

"Je n'ai fait celle-ci plus longue que parce que je n'ai pas eu le loisir de la faire plus courte." —Blaise Pascal (Dec 4, 1656).

Translation: I have made it [this letter] longer because I have not had time to make it shorter.

To further complicate matters, Deming frequently used language that left executives scratching their heads. He began one presentation to corporate managers with the question, "Do you have a constancy of purpose?"

1.23  Consolidation

When consolidating applications, it's not the migration that's the problem, it's the often mistaken motivation. The fallacious reasoning goes like this. Since there are a slew of servers that are only running at 10% busy, migrate those apps onto fewer (virtualized) servers. Naturally, the target server capacity will be better utilized but the application response time (SLA) has now gone out the window.

1.24  Control Freaks Unite!

Control over the performance of hardware resources e.g., CPUs and disks, is progressively being eroded as these things simply become commodity black boxes viz., multicore processors and disk arrays. This situation will only be exacerbated with the advent of Internet-based application services. Software developers will therefore have to understand more about the performance and capacity planning implications of their designs running on these black boxes. (See Sect. 3)

1.25  Productivity

Thanks to the Puritans, presumably, American corporate culture is obsessed with the false notion that being busy is being productive. Wrong! Europeans (especially the Mediterraneans) understand the power of the cat nap. After nearly 400 years, it's time for America to get over it.

Data come from the OECD (Organization for Economic Co-operation and Development). Developing countries often work longer hours, but working longer doesn't necessarily mean working better. Interestingly, the USA is in the middle along with Australia.

1.26  Art vs. Science

A number of recent books and presentations on performance analysis and capacity planning have appeared with "The Art of..." in the title. In itself, this is not new. The title of Raj Jain's excellent 1991 book is The Art of Computer Systems Performance Analysis. Nonetheless, they all resort to various scientific techniques to analyze performance data. The application of science inevitably involves some art.

1.27  ITIL for Guerrillas

Source: Guerrilla Capacity Planning (Springer 2007)

The ITIL framework is all about defining IT processes to satisfy business needs, not the implementation of those processes. That's what makes Guerrilla Capacity Planning (GCaP) and GCaP training excellent ITIL business solutions.

1.28  Performance Paradox

If you do performance management perfectly, you run the risk of becoming invisible and therefore expendable to management.

—N. J. Gunther, "No Analysis Required! The Performance Analyst's Paradox," IEEE GRID, p.21, (Nov 1996)

1.29  Dumb Questions

The only dumb question is the one you never ask.

1.30  Quantum Leap

Quantum transitions in energy are only associated with the discrete spectrum of atomic or molecular bound states.

1.31  Don't Be a Sheeple

1.32  Good Communication

That last one was tweeted and ended up as entry #136 in David Pogue's Twitter Book:

1.33  Performance Gatekeepers

A common situation in big organizations is for various groups to be responsible for the performance evaluation of the software or hardware widgets they create. In principle, this is a good thing, but there is a downside.

Overly zealous attempts at LOCAL performance optimization of any subsystems in the stack will almost inevitably detune the GLOBAL optimization of the entire system.

To avoid this side-effect, a separate central group should be responsible for oversight of total system performance. They should act as both reviewers and gatekeepers for the performance analyses produced by all the other groups in the organization.

1.34  Performance Analysis is a Money Sink

• Performance evaluation
• Performance by design
• Performance engineering
• Performance testing

1.35  Tyranny of the 9s

downt <- function(nines,tunit=c('s','m','h')) {
	ds <- 10^(-nines) * 365.25*24*60*60
	if(tunit == 's') { ts <- 1; tu <- "seconds" }
	if(tunit == 'm') { ts <- 60; tu <- "minutes" }
	if(tunit == 'h') { ts <- 3600; tu <- "hours" }
	return(sprintf("Downtime per year at %d nines: %g %s", nines, ds/ts,tu))
}

> downt(5,'m')
[1] "Downtime per year at 5 nines: 5.2596 minutes"
> downt(8,'s')
[1] "Downtime per year at 8 nines: 0.315576 seconds"

library(NORMT3)
sigp <- function(sigma) {
	sigma <- as.integer(sigma)
	apc <- erf(sigma/sqrt(2))
	return(sprintf("%d-sigma bell area: %10.8f%%; Prob(chance): %e", 
		sigma, apc*100, 1-apc))
}

> sigp(2)
[1] "2-sigma bell area: 95.44997361%; Prob(chance): 4.550026e-02"
> sigp(5)
[1] "5-sigma bell area: 99.99994267%; Prob(chance): 5.733031e-07"

• 3σ corresponds roughly to a probability of 1 in 1,000 that four 9s availability occurred by chance.
• 5σ is roughly a 1 in a million chance, which is the same as flipping a fair coin and getting 20 heads in a row.
• 6σ is roughly a 1 in a billion chance.

1.36  Timescale Intuition

Because of the great diversity of timescales that exist in modern computer systems, it's a good idea to try and get a more intuitive feel for some of them. My first attempt at helping you to do that was in 1998, where I included the table shown at the left in The Practical Performance Analyst. I updated that original as Table 3.1 in my Perl PDQ book. To keep up with the rapid changes in computer technology, I've recalculated the original table as an R data frame: Device Real.ns Real.sec Scaled.time Unit 1 1 GHz clock 1.00 1.0e-09 1.00 second 2 4 GHz CPU cycle 0.25 2.5e-10 0.25 seconds 3 L1 cache access 2.00 2.0e-09 2.00 seconds 4 L2 cache access 4.00 4.0e-09 4.00 seconds 5 Mutex locks 18.00 1.8e-08 18.00 seconds 6 RAM reference 120.00 1.2e-07 2.00 minutes 7 SSD access 150000.00 1.5e-04 1.74 days 8 HD access 10000000.00 1.0e-02 3.80 months 9 USA-AUS ping 160000000.00 1.6e-01 5.07 years 10 SQL update 800000000.00 8.0e-01 25.35 years 11 Tape archive 10000000000.00 1.0e+01 3.17 centuries 12 EC2 spin up 300000000000.00 3.0e+02 9.51 millennia The first line shows how a 1 nanosecond clock period is scaled up to match a more human time of 1 second. In other words, a multiplier of a gigasecond has been applied to the actual times to make them more human scale. The last column shows the rescaled time unit ranging from seconds to millennia. The key point is that since a modern core processor operates on the timescale of a fraction of a nanosecond in the real world or ``seconds'' on the rescaled human scale, RAM accesses are seen to take on the order of minutes, disk accesses take on the order of a month, database accesses take years, magnetic-tape IOs take on the order of centuries, while spinning up a cloud-based VM can take millennia. Looked at from the standpoint of a core processor, tha above view gives a different slant to the expression: Hurry up and wait!

1.37  Throughput vs. Latency

1. Bandwidth (or throughput) corresponds to the diameter of the pipe.
2. Latency corresponds to the length of the pipe (implicit time to flow through it at a constant rate).

On closer scrutiny, one finds many problems with the conventional wisdom:

1. Bandwidth and latency are ill-defined terms.
2. Latency refers to some kind of (heavily context-dependent) delay, e.g., disk latency means something completely different from packet latency.
3. Queueing theory is very specific about different latencies: waiting time, service time, response time, etc.
4. Bandwidth usually refers to a special case of throughput, i.e., maximum throughput.
5. Throughput and latency only appear to be unrelated in clocked systems; like flow through a pipe (see above).
6. If you apply the conventional wisdom to database performance, for example, you will be surprised to find that latency (e.g., response time) increases with throughput.

1. Bandwidth and Latency are Related Like Overclocked Chocolates
2. Bandwidth vs. Latency - The World is Curved
3. Little's law is a curved surface in 3-D

1.38  The Zeroth Performance Metric

1.39  Only Three Performance Metric

1. TIME
   Example metrics: waiting time (e.g., Oracle DB), latency
   Example Units: cpu-ticks, milliseconds
2. COUNT
   Example metrics: runqueue length, packets dropped, RSS
   Example Units: packets, pages, connections
3. RATE
   Example metrics: throughput, bandwidth
   Example Units: MIPS, Gbps, GHz

1.40  IT is the Manufacturing Line of the 21st Century

Today, IT is the manufacturing line of the 21st century. † Therefore, IT performance now plays a dominant role in determining the financial performance of any company.

1.41  Uncommon Sense

Common sense is the pitfall of performance analysis. ‡

1.42  Don't Tell Stories

An excellent example of the slow descent into story-telling hell can be heard in this podcast.
Listen for the key statement: "This is not science. It is storytelling."

1.43  Performance is Not Engineering

Performance analysis should cover soup to nuts: from engineering (both software and hardware development), to technical marketing, sales support, and customer satisfaction. In other words, everything from the lowest level bits in multicore registers to the highest level response time metrics of a customer's business application. Conversely, data collected from these sources should be fed back into future product design. To paraphrase the subtitle of my PPA book, this is how to achieve performance by design, not just post facto performance improvements.

1.44  It's Only Words

"It's only words and words are all I have" —The Bee Gees

2  PERFORMANCE MODELING RULES OF THUMB

2.1  What is Performance Modeling?

"90% of coding is debugging. The other 10% is writing bugs."

2.2  Monitoring vs. Modeling

2.3  Keep It Simple

A performance model should be as simple as possible, but no simpler!

"A designer knows that he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away." —Antoine de Saint-Exupéry

I now tell people in my Guerrilla classes, despite the fact that I repeat this rule of thumb several times, you will throw the kitchen sink into your performance models; at least, early on as you first learn how to create them. It's almost axiomatic: the more you know about the system architecture, the more detail you will try to throw into the model. The goal, in fact, is the opposite.

2.4  More Like The Map Than The Metro

The metro map is an abstraction that has very little to do with the physical railway. It encodes only sufficient detail to enable transit on the rail system from point A to point B. It does not include a lot of irrelevant details, such as, whether or not the stations are above or below ground, or even their actual geographical proximity. A performance model is a similar kind of abstraction.

2.5  The Big Picture

2.6  A Point of Principle

As an example consider a classic multi-user time-share OS, like Unix or Linux. Its POp can be stated as follows:

Time-share scheduling gives every user the illusion that they are the only user on the system. [17 words]

All the millions of lines of code inside the OS that implement time-slicing, priority queueing, etc., are there merely to support that POp. The overall performance goal is to minimize the user response-time metric.

Being able to see things in this unconventional way can really help in the deciding on the simplest kind of performance model.

1. Your laptop is also a time-share OS, e.g., Mac OS X, Ubuntu Linux, Windows. Is the PoP for that single-user OS the same or different than it is for a multi-user OS?
2. Virtualization environments, like VMware or Parallels hypervisors are all the rage, but these hypervisors are nothing more than a kind of global OS with each time-share OS treated as an embedded guest OS. Is the hypervisor PoP the same or different than it is for a time-share OS?

2.7  Guilt is Golden

You, as the performance analyst or planner, only have to shine the light in the right place and then stand back while others flock to fix it.

2.8  What is a Queue?

1. A queue is a line of customers waiting to be severed—as in " Off with their heads!" (*)
2. Hardware version: A queue is implemented as a register, e.g., a memory register.
3. Software version: A queue is implemented as a list. In some computer languages it is a separate data type, e.g., Lisp, Mathematica, Perl, etc.

2.9  Where to Start?

One place to start constructing a PDQ model is by drawing a functional block diagram. The objective is to identify where time is spent at each stage in processing the workload of interest. Ultimately, each functional block is converted to a queueing subsystem like those shown above. This includes the ability to distinguish sequential and parallel processing. Other diagrammatic techniques e.g., UML diagrams, may also be useful but I don't understand that stuff and never tried it. See Chap. 6 "Pretty Damn Quick(PDQ) - A Slow Introduction" of Analyzing Computer System Performance with Perl::PDQ.

2.10  Inputs and Outputs

Take Little's law Q = X R for example. It is a performance model; albeit a simple equation or operational law, but a model nonetheless. All the variables on the RIGHT side of the equation (X and R) are INPUTS, and the single variable on the LEFT is the OUTPUT. A more detailed discussion of this point is presented in Chap. 6 "Pretty Damn Quick(PDQ) - A Slow Introduction" of Analyzing Computer System Performance with Perl::PDQ.

2.11  No Service, No Queues

By definition, a queue consists of a buffer or waiting line and a service facility with an associated service time. If you can't provide a numerical service time value as an argument to the PDQ function, SetDemand(), then you can't define that queue as a PDQ node in your performance model: irrespective of shoulda-coulda-woulda or how much you really, really, want it to be there.

2.12  Estimating Service Times

Suppose, for example, you had requests coming into an HTTP server and you could measure its CPU utilization with some tool like Unix/Linux vmstat, and you would like to know the service time of the HTTP Gets. Unix won't tell you directly, but you can use Little's law (U = X S) to estimate it. If you can measure the arrival rate of requests in Gets/sec (X) and the CPU %utilization (U), then the average service time (S) for a Get is easily calculated from the ratio U/X.

2.13  Change the Data

This may come as a surprise but, many performance measurements are either wrong (not calculated correctly at the source) or are inconsistent with other performance metrics. The best way to verify that is to apply the appropriate performance model to verify measurement consistency.

2.14  Closed or Open Queue?

2.15  Opening a Closed Queue

An open queueing model assumes an infinite population of requesters initiating requests at an arrival rate λ (lambda). In a closed model, λ (lambda) is approximated by the ratio N/Z. Treat the thinktime Z as a free parameter, and choose a value (by trial and error) that keeps N/Z constant as you make N larger in your PDQ model. Eventually, at some value of N, the OUTPUTS of both the closed and open models will agree to some reasonable approximation.

2.16  Steady-State Measurements

2.17  Transcribing Data

2.18  Workloads Come in Threes

Apart from making an unwieldy PDQ report to read, generally you are only interested in the interaction of 2 workloads (pairwise comparison). Everything else goes in the third (AKA "the background"). If you can't see how to do this, you're probably not ready to create the PDQ model.

2.19  Better Than a Crystal Ball

Predicting the future is not the same thing as "seeing" the future. A performance model is just a means for evaluating the data that are provided to it. The purpose of the model is to transform those data into information. The amount of information that can be extracted is intimately dependent on the values of those data. Change the input data and you change the output information. An oft-quoted example is: garbage in, garbage out.

That's an extreme example. More commonly, you may see unexpected modeling results. In that case, the new data do not meet the expectations set by the prior measurements. But that doesn't necessarily imply the model is wrong. More likely something has changed in the measurement system so that it has failed to remain consistent with the initial information contained in the previous measurements.

Having a performance model forces you to ask why that unexpected change has occurred and can anything be done to remove it. Without a performance model, you don't have any expectations or context for asking such questions.

2.20  Patterns and Anti-Patterns

Visual example:

Textual example:

Colorless green ideas sleep furiously. —N. Chomsky (1957)

Your neural compiler happily accepts the correct syntax (pattern) but crashes on the semantics (meaning).

New research indicates that if a person is not in control of the situation, they are more likely to see patterns where none exist, suffer illusions and believe in conspiracy theories.

2.21  Interpreting Data

The conventional view of performance models is that they are useful for:

1. Predicting the future performance of an extant system
2. Exploring what-if scenarios that may or may not be realistic

A performance model can also be used for interpreting performance measurements. Both the measurements and the model must be consistent or something is wrong and needs to be explained.

2.22  Intuition and Modeling

Ulysses is the Greek hero in Homer's Odyssey. On their way back from the Trojan wars, Ulysses orders his men to plug their ears so that they won't succumb to the beautiful song of the sirens and kill them all in a shipwreck. Ulysses, being a typical manager, chooses not to plug his ears because he can't bear the idea of NOT hearing the sirens' singing. So, he orders his men to tie him to the mast of their ship. Such a guy!

2.23  Load Average

For a more complete discussion, see:

1. See Chapter 4 in 1st edn. of the Perl::PDQ book.
2. See Chapter 6 in 2nd edn. of the Perl::PDQ book.
3. Read the original online articles (as PDFs):
   a. How the Load Average Works
   b. Not Your Average Average
   c. Addendum on Hz versus HZ
4. How to convert the absolute load average to the relative stretch factor metric.  

2.24  VAMOOS Your Data Angst

1. Visualize: Make a plot of your data without making any assumptions about how it should look. This is where tools like scatter plots come in.
2. Analyze: Look for patterns or other significant features in the data and possibly quantify them, e.g., trends in the distribution of data points or periodically repeating features, such as spikes or peaks.
3. Modelize: Consider different types of models, including: SWAG, statistical regression, queue-theoretic, simulation, etc. If you are using the Chart>Add Trendline feature in Excel, this is where you choose your model from the Excel dialog box.
   
   
   Don't fret over whether or not it's the "right" choice: there's no such thing, at this point. Whatever your choice, try to make it consistent with steps 1 and 2. If it doesn't work out it doesn't matter because, based on the next step, you're about to come around again. :)
4. Over and Over: None of this is likely to converge in a single pass.
5. Satisfied: Iterate your way to success. Repeat until you are satisfied that all your assumptions are consistent.

2.25  All Measurements are Wrong

Numbers that are produced by measurement are the result of a comparison process between a measurement device with a numerical scale and the thing being measured. The process of measuring with that device inevitably introduces indeterminate effects or sources of noise. There is no noise in mathematics.

The difference between the approximation for π and physical measurement is that the digits of the former can be calculated to any degree of precision at will, whereas measurement digits are unpredictable and therefore remain completely unknown until many measurements are made at the desired level of precision, and expressed using equation 3. You'll never see a statement like, π = 3.14159 ± 0.000001, in math.

2.26  Modeling Errors

Predicting the future is not the same thing as "seeing" the future, in the sense of seeing what things might look like. See GMantra 2.19 for more on this point. All predictions are merely estimates based on (input) data (or other estimates) and therefore predictions come with errors. The only real question is, how much error can you tolerate?

2.27  Data Ain't Information

A Google search returns a lot of data, not information. Google admits that: Are you feeling lucky? It should be a certainty, but it's not. Sneakily, Google also knows that your brain will quickly decide what is information in all those links, such that you don't even realize you're doing most of the work which you unconsciously attribute to Google. They know your brain craves patterns.

2.28  Text is the Universal API

Why do I use LaTeX? [Because] It takes flat ASCII text† as its typographic source. Flat ASCII is both a universal program interface‡ and an immutable data repository.

2.29  Play It Where It Lies

2.30  Data is Not Enough

1. To check data consistency
2. To explain those data
3. Data + Models == Insight

2.31  Plans Are Useless

"In preparing for battle I have always found that plans are useless, but planning is indispensable." —Dwight D. Eisenhower

2.32  Benchmark Design

1. All computer benchmarks are workload simulations
2. Such simulations involve two essential components: stimulus + response
3. The stimulus = workload generator
4. The response = performance of system under test (SUT)

2.33  Waiting Time

Theorem (Gunther 2019): Thinking about queueing theory exponentially reduces waiting time.

Proof: Try it.                      ▢

2.34  Statistics

1. Statistics is an attempt to make formal statements about uncertainty.
2. Statistics is not mathematics.
3. Statistics has been in development for about 500 years. It mostly started with attempts to predict the odds of winning at gambling games. (some things never change)
4. It's not over yet. Witness the ongoing historical battle between Bayesianistas (who came first c.1760s) and the Frequentists (Fisher et al. c.1930s).
5. We won't mention that Fisher's formal p-values are now considered to be unreliable or that Bayesian statistics is now the foundation of so-called Maching Learning algorithms.
6. The very small probability of you winning the lottery or getting a fatal disease is a statistical statement about likelihood because the actual state is unknown or uncertain. Once you win the lottery or become ill, you are no longer a statistic, in the sense that you know your state with 100% certainty. If you are the statistic, then it's not statistics.
7. It has sometimes been said (even by professional perf analysts) that PDQ is not very useful because it only computes averages.
8. Averages are VERY important. Most people/users haven't got a clue about the average value of the KPI they're interested in. Often because it hasn't been measured. I once asked the CEO of Hyatt Hotels what was his average daily transaction rate. He had no idea.
9. Those involved with application (performance) monitoring tend to think that percentiles are more important than statistics or probabilities. What they don't realize is, percentiles (or quantiles, more technically) are just ranked data and ranked data is logically equivalent of the CDF, which is the formal statistical expression of probabilities. :)
10. The notion that variance is a more important than the average, because it can quantify peaks and outliers, is a false dichotomy. Variance is defined in terms of the average value (the statistical mean). It quantifies the variation with respect to the mean. So, you need to know the mean value, anyway.
11. The mean and the variance (or standard deviation) are respectively related to the first and second moments of the underlying distribution—which you most likely don't know and never will know.

2.35  Realistic Models

Reality Beach

1. Don't be too concerned about reality. It's just an annoyance that refuses to conform to your performance models.
2. Never let reality get in the way of a good performance model.

2.36  Models as Explanations

3  UNIVERSAL SCALABILITY LAW (USL)