hypothesis testing neyman pearson

Neyman-Pearson Lemma: Definition

Hypothesis Testing > Neyman-Pearson Lemma

What is Neyman-Pearson Lemma?

The Neyman-Pearson Lemma is a way to find out if the hypothesis test you are using is the one with the greatest statistical power . The power of a hypothesis test is the probability that test correctly rejects the null hypothesis when the alternate hypothesis is true. The goal would be to maximize this power, so that the null hypothesis is rejected as much as possible when the alternate is true. The lemma basically tells us that good hypothesis tests are likelihood ratio tests .

The lemma is named after Jerzy Neyman and Egon Sharpe Pearson , who described it in 1933. It is considered by many to the theoretical foundation of hypothesis testing theory, from which all hypothesis tests are built.

Note : Lemma sounds like it should be a Greek letter, but it isn’t. In mathematics, a lemma is defined as an intermediate proposition used as a “stepping stone” to some other theorem. To differentiate the lemma from theories that have a name and a Greek letter (like Glass’s Delta or Fleiss’ kappa ), it’s sometimes written as Lemma (Neyman-Pearson).

The “Simple” Hypothesis Test

The Neyman-Pearson lemma is based on a simple hypothesis test. A “simple” hypothesis test is one where the unknown parameters are specified as single values. For example:

• H 0 : μ = 0 is simple because the population mean is specified as 0 for the null hypothesis .
• H 0 : μ = 0; H A : μ = 1 is also simple because the population mean for the null hypothesis and alternate hypothesis are specified, single values. Basically, you’re assuming that the parameters for this test can only be 0, or 1 (which is theoretically possible if the test was binomial ).

In contrast, the hypothesis σ 2 > 7 isn’t simple; it’s a composite hypothesis test that doesn’t state a specific value for σ 2 . Simple hypothesis tests — even optimized ones — have limited practical value . However, they are important hypothetical tools; The simple hypothesis test is the one that all others are built on.

The “Best” Rejection Region

Like all hypothesis tests, the simple hypothesis test requires a rejection region — the smallest sample space which defines when the null hypothesis should be rejected. This rejection region (defined by the alpha level) could take on many values. The Neyman-Pearson Lemma basically tells us when we have picked the best possible rejection region.

The “Best” rejection region is one that minimizes the probability of making a Type I or a Type II error :

• A type I error (α) is where you reject the null hypothesis when it is actually true.
• A type II error (β) is where you fail to reject the null hypothesis when it is false.

The Neyman-Pearson Lemma Defined

In order to understand the lemma, it’s necessary to define some basic principles about alpha/beta levels and power:

Alpha and Beta Levels

A Type I error under the null hypothesis is defined as: P θ (X ∈ R ∣ H 0 is true), Where :

• R = the rejection region and
• ∈ is the set membership.

A Type II error under the null hypothesis is defined as: P θ (X ∈ R c ∣ H 0 is false), Where :

• R c = the complement of R.

Usually, an alpha level is set (e.g. 0.05) to restrict the probability of making a Type I error (α) to a certain percentage (in this case, 5%). Next, a test is chosen which minimizes Type II errors (β).

Tests with a certain alpha level α can be written as:

Size α tests: sup β (θ) = α (θ ∈ Θ 0 )

• Θ 0 = set of all possible values for θ under the null hypothesis

A level α test is one that has the largest power function . Mathematically, it is written as:

Level α test: sup β (θ) ≤α(Θ ∈ Θ 0 )

Definitions using UMP and Likelihood-Ratio

Casella and Berger (2002) use the above definitions to define a simple hypothesis test that is uniformly most powerful (UMP) , which is the essence of the Neyman-Pearson Lemma:

“Let C be a class of tests for testing H 0 : θ ∈ Θ 0 versus H 1 : θ ∈ Θ c 1 . A test in class C, with power function β(θ), is a uniformly most powerful (UMP) class C test if β(θ) ≥ β′(θ) for every θ ∈ Θ 0 c and every β′(θ) that is a power function of a test in class C.

In plain English, this is basically saying:

Take all tests of size α. The uniformly most powerful test — which meets the definition for the Neyman-Pearson lemma — is the test with the largest (or equally largest) power function. This is for all values of θ under the alternative hypothesis.

Dozens of different forms of the Neyman-Pearson Lemma (NPL) exist, each with different notations and proofs. They include variants by:

• Roussas (1997): Fundamental NPL for continuous random variables
• Casella and Berger (2002): NPL for continuous and discrete random variables
• Bain & Engelhardt (1990): Simplified NPL
• Lehmann (1991): a multidimensional variant of the NPL.

References : Bain, Lee J., and Max Engelhardt. Introduction to Probability and Mathematical Statistics . 2nd ed. Boston: PWS-KENT Pub., 1992. Print

Casella, George, and Roger L. Berger. Statistical Inference . 2nd ed. Australia: Duxbury, 2002. Print.

Hallin, M. Neyman-Pearson Lemma. Wiley StatsRef: Statistics Reference Online. 2014.

Lehmann, E. L. Testing Statistical Hypotheses. Pacific Grove, CA: Wadsworth & Brooks/Cole Advanced & Software, 1991. Print.

Roussas, George G. A Course in Mathematical Statistics. 2nd ed. San Diego, CA: Academic, 1997. Print.

Stack Exchange Network

Stack Exchange network consists of 183 Q&A communities including Stack Overflow , the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

When to use Fisher versus Neyman-Pearson framework?

I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought.

My question is, ignoring philosophical objections, when should we use Fisher's approach to data testing and when should we use the Neyman-Pearson method? Is there a practical way of deciding which method to use in any given practical problem?

• hypothesis-testing
• methodology

• $\begingroup$ Where have you read about that? Please, cite your sources. $\endgroup$ –  xmjx Commented Feb 20, 2012 at 12:06
• 11 $\begingroup$ See, for instance, here ( jstor.org/stable/2291263 ) or here ( stats.org.uk/statistical-inference/Lenhard2006.pdf ). $\endgroup$ –  Stijn Commented Feb 20, 2012 at 12:14

5 Answers 5

Let me start by defining the terms of the discussion as I see them. A p-value is the probability of getting a sample statistic (say, a sample mean) as far as , or further from some reference value than your sample statistic, if the reference value were the true population parameter. For example, a p-value answers the question: what is the probability of getting a sample mean IQ more than $|\bar x-100|$ points away from 100, if 100 is really the mean of the population from which your sample was drawn. Now the issue is, how should that number be employed in making a statistical inference?

Fisher thought that the p-value could be interpreted as a continuous measure of evidence against the null hypothesis . There is no particular fixed value at which the results become 'significant'. The way I usually try to get this across to people is to point out that, for all intents and purposes, p=.049 and p=.051 constitute an identical amount of evidence against the null hypothesis (cf. @Henrik's answer here ).

On the other hand, Neyman & Pearson thought you could use the p-value as part of a formalized decision making process . At the end of your investigation, you have to either reject the null hypothesis, or fail to reject the null hypothesis. In addition, the null hypothesis could be either true or not true. Thus, there are four theoretical possibilities (although in any given situation, there are just two): you could make a correct decision (fail to reject a true--or reject a false--null hypothesis), or you could make a type I or type II error (by rejecting a true null, or failing to reject a false null hypothesis, respectively). (Note that the p-value is not the same thing as the type I error rate, which I discuss here .) The p-value allows the process of deciding whether or not to reject the null hypothesis to be formalized. Within the Neyman-Pearson framework, the process would work like this: there is a null hypothesis that people will believe by default in the absence of sufficient evidence to the contrary, and an alternative hypothesis that you believe may be true instead. There are some long-run error rates that you will be willing to live with (note that there is no reason these have to be 5% and 20%). Given these things, you design your study to differentiate between those two hypotheses while maintaining, at most, those error rates, by conducting a power analysis and conducting your study accordingly. (Typically, this means having sufficient data.) After your study is completed, you compare your p-value to $\alpha$ and reject the null hypothesis if $p<\alpha$; if it's not, you fail to reject the null hypothesis. Either way, your study is complete and you have made your decision.

The Fisherian and Neyman-Pearson approaches are not the same . The central contention of the Neyman-Pearson framework is that at the end of your study, you have to make a decision and walk away. Allegedly, a researcher once approached Fisher with 'non-significant' results, asking him what he should do, and Fisher said, 'go get more data'.

Personally, I find the elegant logic of the Neyman-Pearson approach very appealing. But I don't think it's always appropriate. To my mind, at least two conditions must be met before the Neyman-Pearson framework should be considered:

• There should be some specific alternative hypothesis ( effect magnitude ) that you care about for some reason. (I don't care what the effect size is, what your reason is, whether it's well-founded or coherent, etc., only that you have one.)
• There should be some reason to suspect that the effect will be 'significant', if the alternative hypothesis is true. (In practice, this will typically mean that you conducted a power analysis, and have enough data.)

When these conditions aren't met, the p-value can still be interpreted in keeping with Fisher's ideas. Moreover, it seems likely to me that most of the time these conditions are not met. Here are some easy examples that come to mind, where tests are run, but the above conditions are not met:

• the omnibus ANOVA for a multiple regression model (it is possible to figure out how all the hypothesized non-zero slope parameters come together to create a non-centrality parameter for the F distribution , but it isn't remotely intuitive, and I doubt anyone does it)
• the value of a Shapiro-Wilk test of the normality of your residuals in a regression analysis (what magnitude of $W$ do you care about and why? how much power to you have to reject the null when that magnitude is correct?)
• the value of a test of homogeneity of variance (e.g., Levene's test ; same comments as above)
• any other tests to check assumptions, etc.
• t-tests of covariates other than the explanatory variable of primary interest in the study
• initial / exploratory research (e.g., pilot studies)

• 4 $\begingroup$ Even though this is an older topic, the answer is much appreciated. +1 $\endgroup$ –  Stijn Commented Mar 11, 2013 at 7:44
• 1 $\begingroup$ +1 Great answer! I'm impressed by your ability to explain these concepts in such a concise way. $\endgroup$ –  COOLSerdash Commented Jul 15, 2013 at 19:42
• 1 $\begingroup$ This is a really wonderful answer, @gung $\endgroup$ –  Patrick S. Forscher Commented Oct 18, 2013 at 20:31
• 11 $\begingroup$ AFAIK Neyman-Pearson did not use Fisherian p values and thus a "p < alpha" criterion. What you call "Neyman-Pearson" actually is "Null-hypothesis significance testing" (a hybrid of Fisher and NP), not pure Neyman-Pearson decision theory. $\endgroup$ –  Frank Commented Oct 20, 2015 at 15:17
• 2 $\begingroup$ @Frank is correct, I believe. Here is a nice article (PDF) full text, no paywall about NHST (null hypothesis significance testing) and contrasting it with Fisher's and Neyman-Pearson's approaches to significance testing and acceptance testing respectively "Fisher, Neyman-Pearson, or NHST? A tutorial for teaching data testing" (March 2015) Frontiers frontiersin.org/articles/10.3389/fpsyg.2015.00223/full $\endgroup$ –  Ellie Kesselman Commented Aug 17, 2023 at 9:10

Practicality is in the eye of the beholder, but;

Fisher's significance testing can be interpreted as a way of deciding whether or not the data suggests any interesting `signal'. We either reject the null hypothesis (which may be a Type I error) or don't say anything at all. For example, in lots of modern 'omics' applications, this interpretation fits; we don't want to make too many Type I errors, we do want to pull out the most exciting signals, though we may miss some.

Neyman-Pearson's hypothesis makes sense when there are two disjoint alternatives (e.g. the Higgs Boson does or does not exist) between which we decide. As well as the risk of a Type I error, here we can also make Type II error - when there's a real signal but we say it's not there, making a 'null' decision. N-P's argument was that, without making too many type I error rates, we want to minimize the risk of Type II errors.

Often, neither system will seem perfect - for example you may just want a point estimate and corresponding measure of uncertainty. Also, it may not matter which version you use, because you report the p-value and leave test interpretation to the reader. But to choose between the approaches above, identify whether (or not) Type II errors are relevant to your application.

The whole point is that you cannot ignore the philosophical differences. A mathematical procedure in statistics doesn't just stand alone as something you apply without some underlying hypotheses, assumptions, theory... philosophy.

That said, if you insist on sticking with frequentist philosophies there might be a few very specific kinds of problems where Neyman-Pearson really needs to be considered. They'd all fall in the class of repeated testing like quality control or fMRI. Setting a specific alpha beforehand and considering the whole Type I, Type II, and power framework becomes more important in that setting.

• $\begingroup$ I don't insist on sticking to frequentist statistics, but I was just wondering if there are situations where adopting a Fisher or Neyman-Pearson viewpoint might be natural. I know there is a philosophical distinction, but perhaps there's also a practical side to be considered? $\endgroup$ –  Stijn Commented Feb 20, 2012 at 19:00
• 4 $\begingroup$ OK, well pretty much just what I said... Neyman-Pearson really were concerned with situations where you do lots and lots of tests without any real theoretical underpinnings to each one. The Fisher viewpoint doesn't really address that issue. $\endgroup$ –  John Commented Feb 20, 2012 at 19:20

My understanding is: p-value is to tell us what to believe (verifying a theory with sufficient data) while Neyman-Pearson approach is to tell us what to do (making best possible decisions even with limited data). So it looks to me that (small) p-value is more stringent while Neyman-Pearson approach is more pragmatic; That's probably why p-value is used more in answering scientific questions while Neyman and Pearson is used more in making statistical/practical decisions.

For an itemized checklist type of answer, that avoids anything that is philosophical in nature, I suggest the following, which is primarily sourced from here . Directly quoted sections are noted accordingly. Think of Fisher's test as a test of significance and Neyman-Pearson as a test of acceptance.

When and why to use Fisher's approach for data testing

• It can be used for one-time, ad hoc projects OR for exploratory research.
• Fisher's test can be set up after the experiment has been conducted and the research data is available for analysis. It works best for populations that share parameters similar to those estimated from the sample.

Because most of the work is done a posteriori, Fisher's approach is quite flexible, allowing for any number of tests to be carried out and, therefore, any number of null hypotheses to be tested.

• Fisher's test is easier to use and more simple, mathematically.

When and why to use Neyman-Pearson

When power of a test is likely to be an important consideration: Fisher has no provision for that in his test.

When there is an alternative hypothesis:

Fisher considered alternative hypotheses implicitly—these being the negation of the null hypotheses—so much so that for him the main task of the researcher—and a definition of a research project well done—was to systematically reject with enough evidence the corresponding null hypothesis.

• For repeated sampling research projects: Examples are industrial quality control and large scale diagnostic testing, for which one uses the same population and tests.

Keep in mind that Neyman-Pearson reduces to Fisher's test under the following circumstances:

It is easy to... forget what makes Neyman-Pearson's approach unique. If the information provided by the alternative hypothesis—ES and β —is not taken into account for designing research with good power, data analysis defaults to Fisher's test of significance.

Your Answer

Sign up or log in, post as a guest.

Required, but never shown

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy .

Not the answer you're looking for? Browse other questions tagged hypothesis-testing p-value methodology or ask your own question .

• Featured on Meta
• Announcing a change to the data-dump process
• Bringing clarity to status tag usage on meta sites

Hot Network Questions

• Is the front wheel supposed to turn 360 degrees?
• What are the most commonly used markdown tags when doing online role playing chats?
• You find yourself locked in a room
• Whats the safest way to store a password in database?
• Largest prime number with +, -, ÷
• Directory of Vegan Communities in Ecuador (South America)
• Movie from 80s or 90s about a helmet which allowed to detect non human people
• Convert 8 Bit brainfuck to 1 bit Brainfuck / Boolfuck
• Can I Use A Server In International Waters To Provide Illegal Content Without Getting Arrested?
• Largest number possible with +, -, ÷
• Is the set of all non-computable numbers closed under addition?
• Using rule-based symbology for overlapping layers in QGIS
• When can the cat and mouse meet?
• Can a quadrilateral polygon have 3 obtuse angles?
• In Lord Rosse's 1845 drawing of M51, was the galaxy depicted in white or black?
• How best to cut (slightly) varying size notches in long piece of trim
• Why do sentences with いわんや often end with をや?
• What should I do if my student has quarrel with my collaborator
• Short story - disease that causes deformities and telepathy. Bar joke
• Is it possible to recover from a graveyard spiral?
• How do I learn more about rocketry?
• Calculating area of intersection of two segmented polygons in QGIS
• quantulum abest, quo minus .
• Could an empire rise by economic power?

Cosa ne pensi di noi?

Il tuo nome

La tua email

The Neyman-Pearson Lemma: A Cornerstone of Statistical Hypothesis Testing

Mappa concettuale.

The Neyman-Pearson Lemma is a fundamental principle in statistics, guiding the creation of powerful tests for hypothesis testing. It focuses on maximizing the probability of correctly rejecting a false null hypothesis while controlling the Type I error rate. This lemma is pivotal in fields like medicine and manufacturing, aiding in treatment comparisons and quality control. Understanding its proof and practical applications is crucial for researchers and statisticians.

Mostra di più

Overview of the Neyman-Pearson Lemma

Definition of simple and composite hypotheses.

Simple hypotheses specify a parameter value, while composite hypotheses allow for a range of values

Balancing power and Type I error

The Neyman-Pearson Lemma aims to maximize power while controlling the probability of a Type I error at a predetermined significance level

Steps in employing the Neyman-Pearson Lemma

The process involves formulating hypotheses, selecting a significance level, calculating a test statistic, and defining a critical region

Proof of the Neyman-Pearson Lemma

Simplified explanation of the proof.

The proof involves comparing the likelihood ratio to a critical value based on the chosen significance level

Importance of understanding the proof

Understanding the proof is crucial for appreciating the balance between minimizing Type I errors and maximizing the power of the test

Practical applications of the Neyman-Pearson Lemma

The lemma has practical applications in fields such as medicine and manufacturing

Implementation of the Neyman-Pearson Lemma

Systematic hypothesis testing framework.

The lemma is implemented through a structured approach involving defining hypotheses, choosing a significance level, and computing the likelihood ratio

Utility of the lemma in diverse disciplines

The lemma has been applied in fields such as biology and agriculture to assess the impact of interventions or compare the effectiveness of products

Theoretical foundations and practical relevance

The lemma is based on the concept of optimality and is specifically tailored for tests between two simple hypotheses

Vuoi creare mappe dal tuo materiale?

Inserisci un testo, carica una foto o un audio su Algor. In pochi secondi Algorino lo trasformerà per te in mappa concettuale, riassunto e tanto altro!

Impara con le flashcards di Algor Education

Clicca sulla singola scheda per saperne di più sull'argomento.

The ______-______ Lemma is fundamental in statistical hypothesis testing for devising the most effective tests between two simple hypotheses.

Neyman Pearson

A simple hypothesis defines a parameter's value exactly, as opposed to a composite hypothesis which permits a ______ of values.

The ______-______ Lemma involves a comparison of the likelihood ratio against a critical value related to the chosen ______ level.

Neyman-Pearson significance

A test is considered the most powerful for detecting an effect if it minimizes ______ I errors while maximizing the test's ______.

Neyman-Pearson Lemma role in medical research

Compares new treatments' efficacy against established ones

Neyman-Pearson Lemma in industrial quality control

Determines if product batches meet predefined standards

Neyman-Pearson Lemma's decision-making under uncertainty

Guides informed conclusions on intervention effectiveness or product quality

The ______-______ Lemma is used in academic and professional research for systematic ______ testing.

Neyman Pearson hypothesis

Neyman-Pearson Lemma: Key Components

Hypothesis formulation, significance levels, likelihood ratios.

Neyman-Pearson Lemma: Role in Empirical Research

Facilitates scientifically sound conclusions through rigorous statistical testing.

Neyman-Pearson Lemma: Impact on Decision Making

Informs choice between alternatives by quantifying evidence in favor of hypotheses.

In the context of hypothesis testing, the likelihood ratio is the division of the probabilities of the data under the ______ and ______ hypotheses.

alternative null

Definition of Neyman-Pearson Lemma

Criterion for creating most powerful test for given size to decide between two simple hypotheses.

Application of Neyman-Pearson Lemma

Used by researchers to design optimal statistical tests, impacting empirical studies across disciplines.

Impact of Neyman-Pearson Lemma on hypothesis testing

Transforms statistical theory into actionable strategy, guiding evidence-based decision-making under uncertainty.

Ecco un elenco delle domande più frequenti su questo argomento

What is the primary goal of the neyman-pearson lemma in statistical hypothesis testing, what are the key steps in applying the neyman-pearson lemma to hypothesis testing, can you provide a simplified understanding of how the neyman-pearson lemma's proof works, where can one see the neyman-pearson lemma applied in real-world scenarios, how is the neyman-pearson lemma utilized in research settings, could you give examples of how the neyman-pearson lemma is used in different research fields, what underlies the neyman-pearson lemma and its application in statistical tests, why is the neyman-pearson lemma considered significant in the realm of statistical analysis, contenuti simili, esplora altre mappe su argomenti simili.

Dispersion in Statistics

Hypothesis Testing for Correlation

Statistical Data Presentation

Correlation and Its Importance in Research

Statistical Testing in Empirical Research

Standard Normal Distribution

Ordinal Regression

Non trovi quello che cercavi?

Cerca un argomento inserendo una frase o una parola chiave

Exploring the Fundamentals of the Neyman-Pearson Lemma

The Process of Hypothesis Testing with the Neyman-Pearson Lemma

A simplified explanation of the neyman-pearson lemma's proof, practical applications of the neyman-pearson lemma, implementing the neyman-pearson lemma in research, case studies demonstrating the neyman-pearson lemma, deepening the understanding of the neyman-pearson lemma, emphasizing the importance of the neyman-pearson lemma in statistical analysis.

Modifica disponibile

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Clin Orthop Relat Res
• v.468(3); 2010 Mar

P Value and the Theory of Hypothesis Testing: An Explanation for New Researchers

David jean biau.

1 Département de Biostatistique et Informatique Médicale, INSERM–UMR-S 717, AP-HP, Université Paris 7, Hôpital Saint Louis, 1, Avenue Claude-Vellefaux, Paris Cedex, 10 75475 France

Brigitte M. Jolles

2 Hôpital Orthopédique Département de l’Appareil Locomoteur Centre Hospitalier, Universitaire Vaudois Université de Lausanne, Lausanne, Switzerland

Raphaël Porcher

In the 1920s, Ronald Fisher developed the theory behind the p value and Jerzy Neyman and Egon Pearson developed the theory of hypothesis testing. These distinct theories have provided researchers important quantitative tools to confirm or refute their hypotheses. The p value is the probability to obtain an effect equal to or more extreme than the one observed presuming the null hypothesis of no effect is true; it gives researchers a measure of the strength of evidence against the null hypothesis. As commonly used, investigators will select a threshold p value below which they will reject the null hypothesis. The theory of hypothesis testing allows researchers to reject a null hypothesis in favor of an alternative hypothesis of some effect. As commonly used, investigators choose Type I error (rejecting the null hypothesis when it is true) and Type II error (accepting the null hypothesis when it is false) levels and determine some critical region. If the test statistic falls into that critical region, the null hypothesis is rejected in favor of the alternative hypothesis. Despite similarities between the two, the p value and the theory of hypothesis testing are different theories that often are misunderstood and confused, leading researchers to improper conclusions. Perhaps the most common misconception is to consider the p value as the probability that the null hypothesis is true rather than the probability of obtaining the difference observed, or one that is more extreme, considering the null is true. Another concern is the risk that an important proportion of statistically significant results are falsely significant. Researchers should have a minimum understanding of these two theories so that they are better able to plan, conduct, interpret, and report scientific experiments.

Introduction

“We are inclined to think that as far as a particular hypothesis is concerned, no test based upon a theory of probability can by itself provide any valuable evidence of the truth or falsehood of a hypothesis” [ 15 ].

Since their introduction in the 1920s, the p value and the theory of hypothesis testing have permeated the scientific community and medical research almost completely. These theories allow a researcher to address a certain hypothesis such as the superiority of one treatment over another or the association between a characteristic and an outcome. In these cases, researchers frequently wish to disprove the well-known null hypothesis, that is, the absence of difference between treatments or the absence of association of a characteristic with outcome. Although statistically the null hypothesis does not necessarily relate to no effect or to no association, the presumption that it does relate to no effect or association frequently is made in medical research and the one we will consider here. The introduction of these theories in scientific reasoning has provided important quantitative tools for researchers to plan studies, report findings, compare results, and even make decisions. However, there is increasing concern that these tools are not properly used [ 9 , 10 , 13 , 20 ].

The p value is attributed to Ronald Fisher and represents the probability of obtaining an effect equal to or more extreme than the one observed considering the null hypothesis is true [ 3 ]. The lower the p value, the more unlikely the null hypothesis is, and at some point of low probability, the null hypothesis is preferably rejected. The p value thus provides a quantitative strength of evidence against the null hypothesis stated.

The theory of hypothesis testing formulated by Jerzy Neyman and Egon Pearson [ 15 ] was that regardless of the results of an experiment, one could never be absolutely certain whether a particular treatment was superior to another. However, they proposed one could limit the risks of concluding a difference when there is none (Type I error) or concluding there is no difference when there is one (Type II error) over numerous experiments to prespecified chosen levels denoted α and β, respectively. The theory of hypothesis testing offers a rule of behavior that, in the long run, ensures followers they would not be wrong often.

Despite simple formulations, both theories frequently are misunderstood and misconceptions have emerged in the scientific community. Therefore, researchers should have a minimum understanding of the p value and hypothesis testing to manipulate these tools adequately and avoid misinterpretation and errors in judgment. In this article, we present the basic statistics behind the p value and hypothesis testing, with historical perspectives, common misunderstandings, and examples of use for each theory. Finally, we discuss the implications of these issues for clinical research.

The p Value

The p value is the probability of obtaining an effect equal to or more extreme than the one observed considering the null hypothesis is true. This effect can be a difference in a measurement between two groups or any measure of association between two variables. Although the p value was introduced by Karl Pearson in 1900 with his chi square test [ 17 ], it was the Englishman Sir Ronald A. Fisher, considered by many as the father of modern statistics, who in 1925 first gave the means to calculate the p value in a wide variety of situations [ 3 ].

Fisher’s theory may be presented as follows. Let us consider some hypothesis, namely the null hypothesis, of no association between a characteristic and an outcome. For any magnitude of the association observed after an experiment is conducted, we can compute a test statistic that measures the difference between what is observed and the null hypothesis. This test statistic may be converted to a probability, namely the p value, using the probability distribution of the test statistic under the null hypothesis. For instance, depending on the situation, the test statistic may follow a χ 2 distribution (chi square test statistic) or a Student’s t distribution. Its graphically famous form is the bell-shaped curve of the probability distribution function of a t test statistic (Fig.  1 A). The null hypothesis is said to be disproven if the effect observed is so important, and consequently the p value is so low, that “either an exceptionally rare chance has occurred or the theory is not true” [ 6 ]. Fisher, who was an applied researcher, strongly believed the p value was solely an objective aid to assess the plausibility of a hypothesis and ultimately the conclusion of differences or associations to be drawn remained to the scientist who had all the available facts at hand. Although he supported a p value of 0.05 or less as indicating evidence against the null, he also considered other more stringent cutoffs. In his words “If p is between 0.1 and 0.9 there is certainly no reason to suspect the hypothesis tested. If it is below 0.02 it is strongly indicated that the hypothesis fails to account for the whole of the facts. We shall not often be astray if we draw a conventional line at 0.05…” [ 4 ].

These graphs show the results of three trials (t 1 , t 2 , and t 3 ) comparing the 1-month HHS after miniincision or standard incision hip arthroplasty under the theory of ( A ) Fisher and ( B ) Neyman and Pearson. For these trials, α = 5% and β = 10%. Trial 1 yields a standardized difference between the groups of 0.5 in favor of the standard incision; Trials 2 and 3 yield standardized differences of 1.8 and 2.05, respectively. The corresponding p values are 0.62, 0.074, and 0.042 for Trials 1, 2, and 3, respectively. ( A ) Fisher’s p value for Trial 2 is represented by the gray area under the null hypothesis; it corresponds to the probability of observing a standardized difference of 1.8 (Point 2) or more extreme differences (gray area on both sides) considering the null hypothesis is true. According to Fisher, Trials 2 and 3 provide fair evidence against the null hypothesis of no difference between treatments; the decision to reject the null hypothesis of no difference in these cases will depend on other important information (previous data, etc). Trial 1 provides poor evidence against the null as the difference observed, or one more extreme, had 62% probability of resulting from chance alone if the treatments were equal. ( B ) Under the Neyman and Pearson theory, the Types I (α = 0.05, gray area under the null hypothesis) and II (β = 0.1, shaded area under the alternative hypothesis) error rates and the difference to be detected (δ = 10) define a critical region for the test statistic (|t test| > 1.97). If the test statistic (standardized difference here) falls into that critical region, the null hypothesis is rejected; this is the case for Trial 3. Trials 1 and 2 do not fall into the critical region and the null is not rejected. According to Neyman and Pearson’s theory, the null hypothesis of no difference between treatments is rejected after Trial 3 only. The distributions depicted are the probability distribution functions of the t test with 168 degrees of freedom.

For instance, say a researcher wants to test the association between the existence of a radiolucent line in Zone 1 on the postoperative radiograph in cemented cups and the risk of acetabular loosening. He or she can use a score test in a Cox regression model, after adjusting for other potentially important confounding variables. The null hypothesis that he or she implicitly wants to disprove is that a radiolucent line in Zone 1 has no effect on acetabular loosening. The researcher’s hypothetical study shows an increased occurrence of acetabular loosening when a radiolucent line in Zone 1 exists on the postoperative radiograph and the p value computed using the score test is 0.02. Consequently, the researcher concludes either a rare event has occurred or the null hypothesis of no association is not true. Similarly, the p value may be used to test the null hypothesis of no difference between two or more treatments. The lower the p value, the more likely is the difference between treatments.

The Neyman-Pearson Theory of Hypothesis Testing

We owe the theory of hypothesis testing as we use it today to the Polish mathematician Jerzy Neyman and American statistician Egon Pearson (the son of Karl Pearson). Neyman and Pearson [ 15 ] thought one could not consider a null hypothesis unless one could conceive at least one plausible alternative hypothesis.

Their theory may be presented in a few words this way. Consider a null hypothesis H0 of equal improvement for patients under Treatment A or B and an alternative hypothesis H1 of a difference in improvement of some relevant size δ between the two treatments. Researchers may make two types of incorrect decisions at the end of a trial: they may consider the null hypothesis false when it is true (a Type I error) or consider the null true when it is in fact false (Type II error) (Table  1 ). Neyman and Pearson proposed, if we set the risks we are willing to accept for Type I errors, say α (ie, the probability of a Type I error), and Type II errors, say β (ie, the probability of a Type II error), then, “without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behavior with regard to them, in following which we insure that, in the long run of experience, we shall not often be wrong.” These Types I and II error rates allow defining a critical region for the test statistic used. For instance for α set at 5%, the corresponding critical regions would be χ 2  > 3.84 for the chi square statistic or |t 168df | > 1.97 for Student’s t test with 168 degrees of freedom (Fig.  1 B) (the reader need not know the details of these computations to grasp the point). If, for example, the comparison of the mean improvement under Treatments A and B falls into that critical region, then the null hypothesis is rejected in favor of the alternative; otherwise, the null hypothesis is accepted. In the case of group comparisons, the test statistic represents a measure of the likelihood that the groups compared are issued from the same population (null hypothesis): the more groups differ, the higher the test statistic and at some point the null hypothesis is rejected and the alternative is accepted. Although Neyman and Pearson did not view the 5% level for Type I error as a binding threshold, this level has permeated the scientific community. For the Type II error rate, 0.1 or 0.2 often is chosen and corresponds to powers (defined as 1 − β) of 90% and 80%, respectively.

Table 1

Types I and II errors according to the theory of hypothesis tests

α and β represent the probability of Types I and II errors, respectively.

For instance, say a surgeon wants to compare the 1-month Harris hip score (HHS) after miniincision and standard incision hip arthroplasty. With the help of a statistician, he plans a randomized controlled trial and considers the null hypothesis H0 of no difference between the standard treatment and experimental treatment (miniincision) and the alternative hypothesis H1 of a difference δ of more than 10 points on the HHS, which he considers is the minimal clinically important difference. Because the statistician is performing many statistical tests across different studies all day long, she has grown very concerned about false positives and, as a general rule, she is not willing to accept more than 5% Type I error rate, that is, if no difference exists between treatments, there is only a 5% chance to conclude a difference. However, the surgeon is willing to give the best chances to detect that difference if it exists and chooses a Type II error of 10%, ie, a power of 90%; therefore, if a difference of 10 points exists between treatments, there is an acceptable 10% chance that the trial will not detect it. Let us presume the expected 1-month HHS after standard incision hip arthroplasty is 70 and the expected SD in both groups is 20. The required sample size therefore is 85 patients per group (two-sample t test).The critical region to reject the null hypothesis therefore is 1.97 (Student’s t test with 168 degrees of freedom). Therefore, if at the end of the trial Student’s t test yields a statistic of 1.97 or greater, the null hypothesis will be rejected; otherwise the null hypothesis will not be rejected and the trial will conclude no difference between the experimental and standard treatment groups. Although the Neyman-Pearson theory of hypothesis testing usually is used for group comparisons, it also may be used for other purposes such as to test the association of a variable and an outcome.

The Difference between Fisher’s P Value and Neyman-Pearson’s Hypothesis Testing

Despite the fiery opposition these two schools of thought have concentrated against each other for more than 70 years, the two approaches nowadays are embedded in a single exercise that often leads to misuse of the original approaches by naïve researchers and sometimes even statisticians (Table  2 ) [ 13 ]. Fisher’s significance testing with the p value is a practical approach whose statistical properties are derived from a hypothetical infinite population and which applies to any single experiment. Neyman and Pearson’s theory of hypothesis testing is a more mathematical view with statistical properties derived from the long-run frequency of experiments and does not provide by itself evidence of the truth or falsehood of a particular hypothesis. The confusion between approaches comes from the fact that the critical region of Neyman-Pearson theory can be defined in terms of p value. For instance, the critical regions defined by thresholds at ± 1.96 for the normal distribution, 3.84 for the chi square test at 1 degree of freedom, and ± 1.97 for a t test at 168 degrees of freedom all correspond to setting a threshold at 0.05 for the p value. The p value is found more practical because it represents a single probability across the different distributions of numerous test statistics and usually the value of the test statistic is omitted and only the p value is reported.

Table 2

Comparison of Fisher’s p value and Neyman-Pearson’s hypothesis testing

The difference between approaches may be more easily understandable through a hypothetical example. After a trial comparing an experimental Treatment A with a standard Treatment B is conducted, a surgeon has to decide whether Treatment A is or is not superior to Treatment B. Following Fisher’s theory, the surgeon weighs issues such as relevant in vitro tests, the design of the trial, previous results comparing treatments, etc, and the p value of the comparison to eventually reach a conclusion. In such cases, p values of 0.052 and 0.047 likely would be similarly weighted in making the conclusion whereas p values of 0.047 and 0.0001 probably would have differing weights. In contrast, statisticians have to give their opinion regarding an enormous quantity of new drugs and medical devices during their life. They cannot be concerned whether each new particular treatment tested is superior to the standard one because they know the evidence can never be certain. However, they know following Neyman and Pearson’s theory they can control the overall proportion of errors, either Type I or II errors (Table  1 ), they make over their entire career. By setting α at, say, 5% and power (1 – β) at 90%, at the end of their career, they know in 5% cases they will have concluded the experimental treatment was superior to the standard when it was not and in 10% cases they will have concluded the experimental treatment was not different from the standard treatment although it was. In that case, very close p values such as 0.047 and 0.052, will lead to rather dramatically opposite actions. In the first case, the treatment studied will be considered superior and used, when in the second case the treatment will be rejected for inefficacy despite very close evidence observed from the two experiments (in a Fisherian point of view).

Misconceptions When Considering Statistical Results

First, the most common and certainly most serious error made is to consider the p value as the probability that the null hypothesis is true. For instance, in the above-mentioned example to illustrate Fisher’s theory, which yielded a p value of 0.02, one should not conclude the data show there is a 2% chance of no association between the existence of a radiolucent line in Zone 1 on the postoperative radiograph in cemented cups and the risk of acetabular loosening. The p value is not the probability of the null hypothesis being true; it is the probability of observing these data, or more extreme data, if the null is true. The p value is computed on the basis that the null hypothesis is true and therefore it cannot give any probability of it being more or less true. The proper interpretation in the example should be: considering no association exists between a radiolucent line in Zone 1 and the risk of acetabular loosening (the null hypothesis), there was only a 2% chance to observe the results of the study (or more extreme results).

Second, there is also a false impression that if trials are conducted with a controlled Type I error, say 5%, and adequate power, say 80%, then significant results almost always are corresponding to a true difference between the treatments compared. This is not the case, however. Imagine we test 1000 null hypotheses of no difference between experimental and control treatments. There is some evidence that the null only rarely is false, namely that only rarely the treatment under study is effective (either superior to a placebo or to the usual treatment) or that a factor under observation has some prognostic value [ 12 , 19 , 20 ]. Say that 10% of these 1000 null hypotheses are false and 90% are true [ 20 ]. Now if we conduct the tests at the aforementioned levels of α = 5% and power = 80%, 36% of significant p values will not report true differences between treatments (Fig.  2 , Scenario 1, 64% true-positive and 36% false-positive significant results; Fig.  3 , Point A). Moreover, in certain contexts, the power of most studies does not exceed 50% [ 1 , 7 ]; in that case, almost ½ of significant p values would not report true differences [ 20 ] (Fig.  3 , Point B).

The flowchart shows the classification tree for 1000 theoretical null hypotheses with two different scenarios considering 10% false null hypotheses. Scenario 1 has a Type I error rate of 5% and a Type II error rate of 20% (power = 80%); Scenario 2 has a Type I error rate of 1% and a Type II error rate of 10% (power = 90%). The first node (A) separates the 900 true null hypotheses (no effect of treatment) from the 100 false null hypotheses (effect of treatment). For Scenario 1, the second node left (B) separates the 900 true null hypotheses (no treatment effect) at the 5% level: 855 tests are not significant (true-negative [TN] results) and 45 tests are falsely significant (false-positive [FP] results). The second node right (C) separates the 100 false null hypotheses (effect of treatment) at the 20% level (power = 80%): 20 tests are falsely not significant (false-negative [FN] results) and 80 tests are significant (true-positive [TP] results). The corresponding positive predictive value [TP/(TP + FP)] is 64%. The figures in parentheses at the second nodes right and left and at the bottom show the results for Scenario 2. The positive predictive value of significant results for Scenario 2 is 91%.

This graph shows the effect of the Types I and II error rates and the proportion of false null hypotheses (true effect of treatment) on the positive predictive value of significant results. Three different levels of Types I and II error rates are depicted: α = 5% and β = 20% (power = 80%), α = 5% and β = 50% (power = 50%), and α = 1% and β = 10% (power = 90%). It can be seen, the higher the proportion of false null hypotheses tested, the better is the positive predictive value of significant results. Point A corresponds to a standard α = 5%, β = 20% (power = 80%), and 10% of false null hypotheses tested. The positive predictive value of a significant result is 64% (also see Fig.  2 ). Point B corresponds to the suspected reality α = 5%, β = 50% (power = 50%), and 10% of false null hypotheses tested. The positive predictive value of a significant result decreases to 53%. Point C corresponds to α = 5%, β = 20% (power = 80%), and 33% of false null hypotheses tested. The positive predictive value of a significant result increases to 89%. Finally, Point D corresponds to α = 1%, β = 10% (power = 90%), and 10% of false null hypotheses tested. The positive predictive value of a significant result increases to 91%. At the extreme, if all null hypotheses tested are true (no effect of treatment), regardless of α and β, the positive predictive value of a significant result is 0.

Implications for Research

Fisher, who designed studies for agricultural field experiments, insisted “a scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance” [ 5 ]. There are three issues that a researcher should consider when conducting, or when assessing the report of, a study (Table  3 ).

Table 3

Implications for research

First, the relevance of the hypothesis tested is paramount to the solidity of the conclusion inferred. The proportion of false null hypotheses tested has a strong effect on the predictive value of significant results. For instance, say we shift from a presumed 10% of null hypotheses tested being false to a reasonable 33% (ie, from 10% of treatments tested effective to 1/3 of treatments tested effective), then the positive predictive value of significant results improves from 64% to 89% (Fig.  3 , Point C). Just as a building cannot be expected to have more resistance to environmental challenges than its own foundation, a study nonetheless will fail regardless of its design, materials, and statistical analysis if the hypothesis tested is not sound. The danger of testing irrelevant or trivial hypotheses is that, owing to chance only, a small proportion of them eventually will wrongly reject the null and lead to the conclusion that Treatment A is superior to Treatment B or that a variable is associated with an outcome when it is not. Given that positive results are more likely to be reported than negative ones, a misleading impression may arise from the literature that a given treatment is effective when it is not and it may take numerous studies and a long time to invalidate this incorrect evidence. The requirement to register trials before the first patient is included may prove to be an important means to deter this issue. For instance, by 1981, 246 factors had been reported [ 12 ] as potentially predictive of cardiovascular disease, with many having little or no relevance at all, such as certain fingerprints patterns, slow beard growth, decreased sense of enjoyment, garlic consumption, etc. More than 25 years later, only the following few are considered clinically relevant in assessing individual risk: age, gender, smoking status, systolic blood pressure, ratio of total cholesterol to high-density lipoprotein, body mass index, family history of coronary heart disease in first-degree relatives younger than 60 years, area measure of deprivation, and existing treatment with antihypertensive agent [ 19 ]. Therefore it is of prime importance that researchers provide the a priori scientific background for testing a hypothesis at the time of planning the study, and when reporting the findings, so that peers may adequately assess the relevance of the research. For instance, with respect to the first example given, we may hypothesize that the presence of a radiolucent line observed in Zone 1 on the postoperative radiograph is a sign of a gap between cement and bone that will favor micromotion and facilitate the passage of polyethylene wear particles, both of which will favor eventual bone resorption and loosening [ 16 , 18 ]. An important endorsement exists when other studies also have reported the association [ 8 , 11 , 14 ].

Second, it is essential to plan and conduct studies that limit the biases so that the outcome rightfully may be attributed to the effect under observation of the study. The difference observed at the end of an experiment between two treatments is the sum of the effect of chance, of the treatment or characteristic studied, and of other confounding factors or biases. Therefore, it is essential to minimize the effect of confounding factors by adequately planning and conducting the study so we know the difference observed can be inferred to be the treatment studied, considering we are willing to reject the effect of chance (when the p value or equivalently the test statistic engages us to do so). Randomization, when adequate, for example, when comparing the 1-month HHS after miniincision and standard incision hip arthroplasty, is the preferred experimental design to control on average known and unknown confounding factors. The same principles should apply to other experimental designs. For instance, owing to the rare and late occurrence of certain events, a retrospective study rather than a prospective study is preferable to judge the association between the existence of a radiolucent line in Zone 1 on the postoperative radiograph in cemented cups and the risk of acetabular loosening. Nonetheless researchers should ensure there is no systematic difference regarding all known confounding factors between patients who have a radiolucent line in Zone 1 and those who do not. For instance, they should retrieve both groups over the same period of time and the acetabular components used and patients under study should be the same in both groups. If the types of acetabular components differ markedly between groups, the researcher will not be able to say whether the difference observed in aseptic loosening between groups is attributable to the existence of a radiolucent line in Zone 1 or to differences in design between acetabular components.

Last, choosing adequate levels of Type I and Type II errors, or alternatively the level of significance for the p value, may improve the reliance we can have in purported significant results (Figs.  2 , ​ ,3). 3 ). Decreasing the α level will proportionally decrease the number of false-positive findings and subsequently improve the positive predictive value of significant results. Increasing the power of studies will correspondingly increase the number of true-positive findings and also improve the positive predictive value of significant results. For example, if we shift from a Type I error rate of 5% and power of 80% to a Type I error rate of 1% and power of 90%, the positive predictive value of a significant result increases from 64% to 91% (Fig.  2 , Scenario 2; Fig.  3 , Point D). Sample size can be used as a lever to control for Types I and II error levels [ 2 ]. However, a strong statistical association, p values, or test statistics never imply any causal effect. The causal effect is built on, study after study, little by little. Therefore, replication of the experiment by others is crucial before accepting any hypothesis. To replicate an experiment, the methods used must be described sufficiently so that the study can be replicated by other informed investigators.

The p value and the theory of hypothesis testing are useful tools that help doctors conduct research. They are helpful for planning an experiment, interpreting the results observed, and reporting findings to peers. However, it is paramount researchers understand precisely what these tools mean and do not mean so that eventually they will not be blinded by the irrelevance of some statistical value in front of important medical reasoning.

Each author certifies that he or she has no commercial associations (eg, consultancies, stock ownership, equity interest, patent/licensing arrangements, etc) that might pose a conflict of interest in connection with the submitted article.

• Prompt Library
• DS/AI Trends
• Stats Tools
• Interview Questions
• Generative AI
• Machine Learning
• Deep Learning

Neyman-Pearson Lemma: Hypothesis Test, Examples

Have you ever faced a crucial decision where you needed to rely on data to guide your choice? Whether it’s determining the effectiveness of a new medical treatment or assessing the quality of a manufacturing process, hypothesis testing becomes essential. That’s where the Neyman-Pearson Lemma steps in, offering a powerful framework for making informed decisions based on statistical evidence .

The Neyman-Pearson Lemma holds immense importance when it comes to solving problems that demand decision making or conclusions to a higher accuracy . By understanding this concept, we learn to navigate the complexities of hypothesis testing, ensuring we make the best choices with greater confidence. In this blog post, we will explore the concepts of the Neyman-Pearson Lemma with the help of real-world examples and Python code example.

Table of Contents

What is Neyman-Pearson Lemma?

From the hypothesis testing standpoint, the Neyman-Pearson Lemma stands as a fundamental principle that guides the construction of powerful statistical tests. By comprehending the key principles behind this lemma, we can enhance our ability to make informed decisions based on data.

The Neyman-Pearson Lemma states that the likelihood ratio test is the most powerful test for a given significance level (or size) in the context of simple binary hypothesis testing problems. It provides a theoretical basis for determining the critical region or decision rule that maximizes the probability of correctly detecting a true effect while maintaining a fixed level of Type I error (false positive rate).

At the heart of the Neyman-Pearson Lemma lies the concept of statistical power . Statistical power represents the ability of a hypothesis test to detect a true effect or difference when it exists in the population. The lemma emphasizes the importance of optimizing this power while controlling the risk of both Type I and Type II errors.

Type I error, also known as a false positive, occurs when we reject the null hypothesis (assuming an effect or difference exists) when it is actually true. Type II error, on the other hand, refers to a false negative, where we fail to reject the null hypothesis (assuming no effect or difference) when an effect or difference truly exists. The Neyman-Pearson Lemma allows us to strike a balance between these errors by maximizing power while setting a predetermined significance level (the probability of Type I error).

By employing the Neyman-Pearson Lemma, we can design tests that optimize statistical power while controlling the rate of false positives . This ensures that we have a greater chance of correctly identifying true effects or differences in our data. Understanding the foundations of the Neyman-Pearson Lemma empowers us to make informed decisions backed by robust statistical analyses. In the upcoming sections, we will explore practical examples that illustrate the application of this lemma in various fields, further solidifying our understanding of its significance.

Note that in Neyman-Pearson Lemma, “lemma” refers to a mathematical term . In mathematics, a lemma is a proven statement or proposition that serves as a stepping stone or auxiliary result in the proof of a larger theorem . The Neyman-Pearson Lemma itself is a theorem in hypothesis testing theory , and it is named after the statisticians Jerzy Neyman and Egon Pearson, who developed it independently in the 1920s and 1930s. The lemma serves as a crucial intermediate result in the derivation and understanding of hypothesis testing procedures.

Here is the summary of what we learned so far about Neyman-Pearson Lemma:

• The Neyman-Pearson Lemma guides the construction of powerful statistical tests in hypothesis testing.
• It states that the likelihood ratio test is the most powerful test for a given significance level in binary hypothesis testing.
• Statistical power is emphasized, representing the ability to detect true effects while controlling errors.
• The lemma balances Type I and Type II errors by optimizing power under a predetermined significance level.
• Employing the Neyman-Pearson Lemma enables tests with higher chances of correctly identifying true effects.

How does Neyman-Pearson Lemma work?

In order to understand how Neyman-Pearson works, lets consider a binary hypothesis testing problem with two hypotheses: a null hypothesis (H0) and an alternative hypothesis (H1). Let’s denote the probability density functions (pdfs) of the observed data under the null and alternative hypotheses as f0(x) and f1(x), respectively. The Neyman-Pearson lemma states that the likelihood ratio test is the most powerful test for a fixed significance level α.

The likelihood ratio test compares the likelihoods of the observed data under the null and alternative hypotheses and accepts the alternative hypothesis if the likelihood ratio exceeds a certain threshold. Mathematically, the likelihood ratio test is given by:

Reject H0 if L(x) = f1(x) / f0(x) > k

where k is a threshold determined based on the desired significance level α. The threshold k is chosen such that the probability of Type I error (false positive) is equal to α.

The Neyman-Pearson lemma states that the likelihood ratio test has the highest statistical power among all tests with the same significance level α. In other words, it maximizes the probability of detecting a true effect or rejecting the null hypothesis when the alternative hypothesis is true.

It’s important to note that the Neyman-Pearson lemma assumes that the null and alternative hypotheses are simple, meaning they are completely specified (e.g., specific parameter values) and mutually exclusive. In practice, the lemma is often used as a basis for constructing tests in more general settings, such as composite hypotheses or hypothesis tests involving multiple parameters.

The Neyman-Pearson lemma provides a theoretical foundation for hypothesis testing, and its concepts are widely used in statistical inference and decision-making.

Neyman-Pearson Lemma Real-world Example

One real-world example where the Neyman-Pearson lemma test can be used is in medical testing, specifically for diagnosing a disease. Let’s consider a hypothetical scenario:

Null hypothesis (H0) : The patient does not have the disease. Alternative hypothesis (H1) : The patient has the disease.

In this example, we want to design a test that can accurately determine whether a patient has a specific disease or not. We need to balance the risks of two types of errors:

Type I error (false positive) : Rejecting the null hypothesis (saying the patient has the disease) when the patient is actually healthy. Type II error (false negative) : Failing to reject the null hypothesis (saying the patient is healthy) when the patient actually has the disease.

To apply the Neyman-Pearson lemma, we would need to determine the likelihood ratio test based on the data and find a threshold that controls the significance level of the test (the probability of a Type I error). For example, suppose we have collected data on a specific biomarker that is known to be associated with the disease. We can model the biomarker levels for healthy patients (H0) as a normal distribution with a certain mean and variance, and the biomarker levels for patients with the disease (H1) as another normal distribution with a different mean and variance.

The null hypothesis (H0) would be formulated as: The biomarker levels follow a normal distribution with parameters μ0 (mean under H0) and σ0 (variance under H0). The alternative hypothesis (H1) would be formulated as: The biomarker levels follow a normal distribution with parameters μ1 (mean under H1) and σ1 (variance under H1), where μ1 > μ0.

To perform the hypothesis test, we would calculate the likelihood ratio, which is the ratio of the likelihood of the observed data under the alternative hypothesis to the likelihood under the null hypothesis. If the likelihood ratio exceeds a certain threshold (determined based on the desired significance level), we would reject the null hypothesis in favor of the alternative hypothesis.

The Neyman-Pearson lemma guarantees that this likelihood ratio test will have the highest statistical power (i.e., the highest probability of correctly detecting the disease when it is present) among all tests with the same significance level.

In practice, medical researchers and practitioners would collect data, estimate the parameters of the null and alternative hypotheses (e.g., using maximum likelihood estimation), and then perform the likelihood ratio test to make informed decisions about the presence or absence of the disease in individual patients.

Neyman-Pearson Lemma Python Example

While the Neyman-Pearson lemma is a theoretical concept, I can provide you with a Python code example that demonstrates the implementation of the likelihood ratio test based on the Neyman-Pearson lemma for a simple hypothesis testing problem. Let’s consider a scenario of testing the mean of a normal distribution.

The following gets printed:

In this example, we first generate a sample dataset by drawing samples from a normal distribution with a mean of mu1 (alternative hypothesis) and a standard deviation of sigma. We assume that the null hypothesis is a normal distribution with a mean of mu0. The likelihood_ratio is computed by calculating the likelihood of the data under the alternative hypothesis divided by the likelihood under the null hypothesis.

We then set the desired significance level alpha (e.g., 0.05) and calculate the threshold ( critical region ) for rejecting the null hypothesis based on the inverse cumulative distribution function (CDF) of the standard normal distribution( norm.ppf ). If the likelihood_ratio exceeds the threshold ( critical region ), we reject the null hypothesis; otherwise, we fail to reject the null hypothesis. In the above example, we rejected the null hypothesis as likelihood ratio (2.0519) exceeded the threshold value (1.6448). The same is demonstrated below.

In the above code example, norm.pdf(data, mu1, sigma) is a function call that computes the probability density function (PDF) of the normal distribution at the given randomly generated data points ( np.random.randn(n) + mu1 ), assuming a specific mean mu1 (alternative hypothesis) and standard deviation sigma . Similarly, norm.pdf(data, mu0, sigma) computes the PDF of the normal distribution at the given data points, assuming a specific mean mu0 (null hypothesis) and standard deviation sigma .

The Neyman-Pearson Lemma provides a principled approach to designing powerful statistical tests. By understanding the foundations of this lemma, we gained valuable insights into optimizing statistical power while controlling error rates. The ability to strike a balance between Type I and Type II errors empowers us to make informed decisions based on data, ensuring that we correctly identify true effects or differences. From medical research to quality control and beyond, the Neyman-Pearson Lemma finds widespread application in various fields where robust hypothesis testing is paramount. By harnessing its principles, we equip ourselves with a valuable tool to navigate the complexities of statistical inference and make sound judgments backed by rigorous analyses.

Recent Posts

• ROC Curve & AUC Explained with Python Examples - August 28, 2024
• Accuracy, Precision, Recall & F1-Score – Python Examples - August 28, 2024
• Logistic Regression in Machine Learning: Python Example - August 26, 2024

Ajitesh Kumar

Leave a reply cancel reply.

Your email address will not be published. Required fields are marked *

• Search for:

ChatGPT Prompts (250+)

• Generate Design Ideas for App
• Expand Feature Set of App
• Create a User Journey Map for App
• Generate Visual Design Ideas for App
• Generate a List of Competitors for App
• ROC Curve & AUC Explained with Python Examples
• Accuracy, Precision, Recall & F1-Score – Python Examples
• Logistic Regression in Machine Learning: Python Example
• Reducing Overfitting vs Models Complexity: Machine Learning
• Model Parallelism vs Data Parallelism: Examples

Data Science / AI Trends

• • Prepend any arxiv.org link with talk2 to load the paper into a responsive chat application
• • Custom LLM and AI Agents (RAG) On Structured + Unstructured Data - AI Brain For Your Organization
• • Guides, papers, lecture, notebooks and resources for prompt engineering
• • Common tricks to make LLMs efficient and stable
• • Machine learning in finance

Free Online Tools

• Create Scatter Plots Online for your Excel Data
• Histogram / Frequency Distribution Creation Tool
• Online Pie Chart Maker Tool
• Z-test vs T-test Decision Tool
• Independent samples t-test calculator

Recent Comments

I found it very helpful. However the differences are not too understandable for me

Very Nice Explaination. Thankyiu very much,

in your case E respresent Member or Oraganization which include on e or more peers?

Such a informative post. Keep it up

Thank you....for your support. you given a good solution for me.

User Preferences

Content preview.

Arcu felis bibendum ut tristique et egestas quis:

• Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris
• Duis aute irure dolor in reprehenderit in voluptate
• Excepteur sint occaecat cupidatat non proident

Keyboard Shortcuts

6.1 - type i and type ii errors.

When conducting a hypothesis test there are two possible decisions: reject the null hypothesis or fail to reject the null hypothesis. You should remember though, hypothesis testing uses data from a sample to make an inference about a population. When conducting a hypothesis test we do not know the population parameters. In most cases, we don't know if our inference is correct or incorrect.

When we reject the null hypothesis there are two possibilities. There could really be a difference in the population, in which case we made a correct decision. Or, it is possible that there is not a difference in the population (i.e., \(H_0\) is true) but our sample was different from the hypothesized value due to random sampling variation. In that case we made an error. This is known as a Type I error.

When we fail to reject the null hypothesis there are also two possibilities. If the null hypothesis is really true, and there is not a difference in the population, then we made the correct decision. If there is a difference in the population, and we failed to reject it, then we made a Type II error.

Rejecting \(H_0\) when \(H_0\) is really true, denoted by \(\alpha\) ("alpha") and commonly set at .05

     \(\alpha=P(Type\;I\;error)\)

Failing to reject \(H_0\) when \(H_0\) is really false, denoted by \(\beta\) ("beta")

     \(\beta=P(Type\;II\;error)\)

Example: Trial Section  

A man goes to trial where he is being tried for the murder of his wife.

We can put it in a hypothesis testing framework. The hypotheses being tested are:

• \(H_0\) : Not Guilty
• \(H_a\) : Guilty

Type I error  is committed if we reject \(H_0\) when it is true. In other words, did not kill his wife but was found guilty and is punished for a crime he did not really commit.

Type II error  is committed if we fail to reject \(H_0\) when it is false. In other words, if the man did kill his wife but was found not guilty and was not punished.

Example: Culinary Arts Study Section  

A group of culinary arts students is comparing two methods for preparing asparagus: traditional steaming and a new frying method. They want to know if patrons of their school restaurant prefer their new frying method over the traditional steaming method. A sample of patrons are given asparagus prepared using each method and asked to select their preference. A statistical analysis is performed to determine if more than 50% of participants prefer the new frying method:

• \(H_{0}: p = .50\)
• \(H_{a}: p>.50\)

Type I error  occurs if they reject the null hypothesis and conclude that their new frying method is preferred when in reality is it not. This may occur if, by random sampling error, they happen to get a sample that prefers the new frying method more than the overall population does. If this does occur, the consequence is that the students will have an incorrect belief that their new method of frying asparagus is superior to the traditional method of steaming.

Type II error  occurs if they fail to reject the null hypothesis and conclude that their new method is not superior when in reality it is. If this does occur, the consequence is that the students will have an incorrect belief that their new method is not superior to the traditional method when in reality it is.

Module 9: Hypothesis Testing With One Sample

Null and alternative hypotheses, learning outcomes.

• Describe hypothesis testing in general and in practice

The actual test begins by considering two  hypotheses . They are called the null hypothesis and the alternative hypothesis . These hypotheses contain opposing viewpoints.

H 0 : The null hypothesis: It is a statement about the population that either is believed to be true or is used to put forth an argument unless it can be shown to be incorrect beyond a reasonable doubt.

H a : The alternative hypothesis : It is a claim about the population that is contradictory to H 0 and what we conclude when we reject H 0 .

Since the null and alternative hypotheses are contradictory, you must examine evidence to decide if you have enough evidence to reject the null hypothesis or not. The evidence is in the form of sample data.

After you have determined which hypothesis the sample supports, you make adecision. There are two options for a  decision . They are “reject H 0 ” if the sample information favors the alternative hypothesis or “do not reject H 0 ” or “decline to reject H 0 ” if the sample information is insufficient to reject the null hypothesis.

Mathematical Symbols Used in  H 0 and H a :

H 0 always has a symbol with an equal in it. H a never has a symbol with an equal in it. The choice of symbol depends on the wording of the hypothesis test. However, be aware that many researchers (including one of the co-authors in research work) use = in the null hypothesis, even with > or < as the symbol in the alternative hypothesis. This practice is acceptable because we only make the decision to reject or not reject the null hypothesis.

H 0 : No more than 30% of the registered voters in Santa Clara County voted in the primary election. p ≤ 30

H a : More than 30% of the registered voters in Santa Clara County voted in the primary election. p > 30

A medical trial is conducted to test whether or not a new medicine reduces cholesterol by 25%. State the null and alternative hypotheses.

H 0 : The drug reduces cholesterol by 25%. p = 0.25

H a : The drug does not reduce cholesterol by 25%. p ≠ 0.25

We want to test whether the mean GPA of students in American colleges is different from 2.0 (out of 4.0). The null and alternative hypotheses are:

H 0 : μ = 2.0

H a : μ ≠ 2.0

We want to test whether the mean height of eighth graders is 66 inches. State the null and alternative hypotheses. Fill in the correct symbol (=, ≠, ≥, <, ≤, >) for the null and alternative hypotheses. H 0 : μ __ 66 H a : μ __ 66

• H 0 : μ = 66
• H a : μ ≠ 66

We want to test if college students take less than five years to graduate from college, on the average. The null and alternative hypotheses are:

H 0 : μ ≥ 5

H a : μ < 5

We want to test if it takes fewer than 45 minutes to teach a lesson plan. State the null and alternative hypotheses. Fill in the correct symbol ( =, ≠, ≥, <, ≤, >) for the null and alternative hypotheses. H 0 : μ __ 45 H a : μ __ 45

• H 0 : μ ≥ 45
• H a : μ < 45

In an issue of U.S. News and World Report , an article on school standards stated that about half of all students in France, Germany, and Israel take advanced placement exams and a third pass. The same article stated that 6.6% of U.S. students take advanced placement exams and 4.4% pass. Test if the percentage of U.S. students who take advanced placement exams is more than 6.6%. State the null and alternative hypotheses.

H 0 : p ≤ 0.066

H a : p > 0.066

On a state driver’s test, about 40% pass the test on the first try. We want to test if more than 40% pass on the first try. Fill in the correct symbol (=, ≠, ≥, <, ≤, >) for the null and alternative hypotheses. H 0 : p __ 0.40 H a : p __ 0.40

• H 0 : p = 0.40
• H a : p > 0.40

Concept Review

In a  hypothesis test , sample data is evaluated in order to arrive at a decision about some type of claim. If certain conditions about the sample are satisfied, then the claim can be evaluated for a population. In a hypothesis test, we: Evaluate the null hypothesis , typically denoted with H 0 . The null is not rejected unless the hypothesis test shows otherwise. The null statement must always contain some form of equality (=, ≤ or ≥) Always write the alternative hypothesis , typically denoted with H a or H 1 , using less than, greater than, or not equals symbols, i.e., (≠, >, or <). If we reject the null hypothesis, then we can assume there is enough evidence to support the alternative hypothesis. Never state that a claim is proven true or false. Keep in mind the underlying fact that hypothesis testing is based on probability laws; therefore, we can talk only in terms of non-absolute certainties.

Formula Review

H 0 and H a are contradictory.

• OpenStax, Statistics, Null and Alternative Hypotheses. Provided by : OpenStax. Located at : http://cnx.org/contents/[email protected]:58/Introductory_Statistics . License : CC BY: Attribution
• Introductory Statistics . Authored by : Barbara Illowski, Susan Dean. Provided by : Open Stax. Located at : http://cnx.org/contents/[email protected] . License : CC BY: Attribution . License Terms : Download for free at http://cnx.org/contents/[email protected]
• Simple hypothesis testing | Probability and Statistics | Khan Academy. Authored by : Khan Academy. Located at : https://youtu.be/5D1gV37bKXY . License : All Rights Reserved . License Terms : Standard YouTube License

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Bayesian estimation of neyman–scott rectangular pulse model parameters in comparison with other parameter estimation methods.

1. Introduction

2. methods: nsrp model and parameter estimation methods, 2.1. nsrp model, 2.2. frequentist inference for nsrp model, 2.2.1. method of moments, 2.2.2. maximum likelihood estimation method, 2.3. bayesian inference on nsrp model, 2.3.1. definition and model specification, 2.3.2. slice sampling.

3. Results of Parameter Estimation

3.1. results of nsrp parameter estimation using mme method, 3.2. results of nsrp parameter estimation using mle method, 3.3. results of nsrp parameter estimation using bayesian estimation method, 3.4. parameter estimate evaluation methods.

4. Discussion

5. conclusions, author contributions, data availability statement, conflicts of interest.

• Cowpertwait, P.S.P.; Kilsby, C.G.; O’Connell, P.E. A spatial-time Neyman-Scott model of rainfall: Empirical analysis of extremes. Water Resour. Res. 2002 , 38 , 1131. [ Google Scholar ] [ CrossRef ]
• Martinez, M.D.; Lana, X.; Burgueño, A.; Serra, C. Spatial and temporal daily rainfall regime in Catalonia NE Spain derived from four precipitation indices, years 1950–2000. Int. J. Climatol. 2007 , 27 , 123–138. [ Google Scholar ] [ CrossRef ]
• Aravena, J.C.; Luckman, B.H. Spatiotemporal rainfall patterns in southern South America. Int. J. Climatol. 2009 , 29 , 2106–2120. [ Google Scholar ] [ CrossRef ]
• Sen Ropy, S. A special analysis of extreme hourly precipitation patterns in India. Int. J. Climatol. 2008 , 29 , 345–355. [ Google Scholar ] [ CrossRef ]
• Burguenu, A.; Matinez, M.D.; Liana, X. Statistical contribution of the daily rainfall regime in Catalonia (northeastern Spain) for the years 1950–2000. Int. J. Climatol. 2005 , 28 , 1381–1403. [ Google Scholar ] [ CrossRef ]
• Rodriguez, I.; Gupta, V.K.; Waymire, E. Scale considerations in the modeling of temporally rainfall. Water Resour. Res. 2011 , 20 , 1611–1619. [ Google Scholar ] [ CrossRef ]
• Sung, J.H. Analysis of extreme rainfall characteristics in 2022 and projection of extreme rainfall based on climate change scenarios. Water 2023 , 15 , 3986. [ Google Scholar ] [ CrossRef ]
• Dubey, S.K.; Kim, J.J.; Hwang, S.; Her, Y.; Jeong, H. Variability of extreme events in coastal and inland areas of South Korea during 1961–2020. Sustainability 2023 , 15 , 12537. [ Google Scholar ] [ CrossRef ]
• Calenda, G.; Napolitano, F. Parameter estimation of Neyman-Scott process for temporal point rainfall simulation. J. Hydrol. 1999 , 225 , 45–66. [ Google Scholar ] [ CrossRef ]
• Cowpertwait, P.S.P.; O’Connell, P.E.; Metcalfe, A.; Mawdsley, J. Stochastic point process modeling of rainfall. I. Single site fitting and validation. J. Hydrol. 1996 , 175 , 17–46. [ Google Scholar ] [ CrossRef ]
• Lee, J.J.; Kim, Y.G. A spatial analysis of Neyman-Scott rectangular pulses model using an approximate likelihood function. J. Korean Data Inf. Sci. Soc. 2016 , 27 , 1119–1131. [ Google Scholar ]
• Kim, Y.; Kim, D.H. An approximate likelihood function of spatial correlation parameters. J. Korean Data Inf. Sci. Soc. 2015 , 45 , 276–284. [ Google Scholar ] [ CrossRef ]
• Xiang, Y.; Gubian, S.; Suomela, B.; Hoeng, J. Generalized simulated annealing for efficient global optimization: The GenSA Package for R. R J. 2013 , 5 , 13–28. [ Google Scholar ] [ CrossRef ]
• Mullen, K.M. Continuous global optimization in R. J. Stat. Softw. 2004 , 60 , 1–45. [ Google Scholar ]
• Scrucca, L. GA: A package for genetic algorithms in R. J. Stat. Softw. 2013 , 53 , 1–37. [ Google Scholar ] [ CrossRef ]
• Kim, G.; Cho, H.; Yi, J. Parameter estimation of the Neyman-Scott rectangular pulse model using a differential evolution method. J. Korean Soc. Hazard Mitig. 2012 , 12 , 187–194. [ Google Scholar ] [ CrossRef ]
• Ardia, D.; Mullen, K.M.; Peterson, B.G.; Ulrich, J. DEoptim: Differential evolution in R, R package version. J. Stat. Softw. 2013 , 2 , 2. [ Google Scholar ]
• Neal, R.M. Slice sampling. Ann. Stat. 2003 , 31 , 705–741. [ Google Scholar ] [ CrossRef ]
• Gelman, A. Inference and Monitoring Convergence in Markov Chain Monte Carlo in Practice ; Gilks, W.R., Richarson, S., Spiegelhalter, D.J., Eds.; Chapman and Hall: London, UK, 1996; pp. 131–143. [ Google Scholar ]
• Gamerman, D.; Lopes, H.F. Markov Chain Monte Carlo, Stochastic Simulation for Bayesian Inference ; Chapman & Hall/CRC: Boca Raton, FL, USA, 2006; pp. 320–342. [ Google Scholar ]
• Gelman, A.; Rubin, D.B. Inference from iterative simulation using multiple sequence. Stat. Sci. 1992 , 7 , 457–472. [ Google Scholar ] [ CrossRef ]
• Geyer, C.J. Practical Markov Chain Monte Carlo. Stat. Sci. 1992 , 7 , 473–483. [ Google Scholar ] [ CrossRef ]
• Cowles, M.K.; Carlin, B.P. Markov Chain Monte Carlo convergence diagnostic: A comparative review. J. Am. Stat. Assoc. 1996 , 91 , 883–904. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Nizeyimana, P.; Lee, K.E.; Kim, G. Bayesian Estimation of Neyman–Scott Rectangular Pulse Model Parameters in Comparison with Other Parameter Estimation Methods. Water 2024 , 16 , 2515. https://doi.org/10.3390/w16172515

Nizeyimana P, Lee KE, Kim G. Bayesian Estimation of Neyman–Scott Rectangular Pulse Model Parameters in Comparison with Other Parameter Estimation Methods. Water . 2024; 16(17):2515. https://doi.org/10.3390/w16172515

Nizeyimana, Pacifique, Kyeong Eun Lee, and Gwangseob Kim. 2024. "Bayesian Estimation of Neyman–Scott Rectangular Pulse Model Parameters in Comparison with Other Parameter Estimation Methods" Water 16, no. 17: 2515. https://doi.org/10.3390/w16172515

Article Metrics

Further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

On the Price of Decentralization in Decentralized Detection

Fundamental limits on the error probabilities of a family of decentralized detection algorithms (eg., the social learning rule proposed by Lalitha et al.   [ 2 ] ) over directed graphs are investigated. In decentralized detection, a network of nodes locally exchanging information about the samples they observe with their neighbors to collectively infer the underlying unknown hypothesis. Each node in the network weighs the messages received from its neighbors to form its private belief and only requires knowledge of the data generating distribution of its observation. In this work, it is first shown that while the original social learning rule of Lalitha et al.   [ 2 ] achieves asymptotically vanishing error probabilities as the number of samples tends to infinity, it suffers a gap in the achievable error exponent compared to the centralized case. The gap is due to the network imbalance caused by the local weights that each node chooses to weigh the messages received from its neighbors. To close this gap, a modified learning rule is proposed and shown to achieve error exponents as large as those in the centralized setup. This implies that there is essentially no first-order penalty caused by decentralization in the exponentially decaying rate of error probabilities. To elucidate the price of decentralization, further analysis on the higher-order asymptotics of the error probability is conducted. It turns out that the price is at most a constant multiplicative factor in the error probability, equivalent to an o ⁢ ( 1 / t ) 𝑜 1 𝑡 o(1/t) italic_o ( 1 / italic_t ) additive gap in the error exponent, where t 𝑡 t italic_t is the number of samples observed by each agent in the network and the number of rounds of information exchange. This constant depends on the network connectivity and captures the level of network imbalance. Results of simulation on the error probability supporting our learning rule are shown. Further discussions and extensions of results are also presented.

I Introduction

Decentralization is one of the major themes in the development of Internet of Things (IoT), and among many different scenarios of decentralization, an important one is decentralized detection . In decentralized detection (hypothesis testing), a group of agents (nodes) form a network (directed graph) to exchange information regarding their observed data samples in a decentralized manner, so that each of them can detect the hidden parameter that governs the sample-generating statistical model. For hypothesis testing, prior to information exchange, decentralization typically requires each node to get only full access to its samples but not the others’. In addition, each node only knows the likelihood functions of its observations.

To fulfill these requirements, a natural approach based on message passing for decentralized detection has been considered in  [ 3 , 4 , 5 , 2 , 6 ] , where each node performs a local Bayesian update and sends its belief vectors (message) to its neighbors for a further consensus step. For instance, in [ 2 ] , each node performs a consensus averaging on a re-weighting of the log-beliefs after receiving the messages (which are log-beliefs in  [ 2 ] ) from its neighbors), and the weights are summarized into a right stochastic matrix (called the “weight matrix”, which could be viewed as the transition matrix of a Markov chain. Such an approach is termed social learning in [ 2 ] . Under the learning rule, it is shown that the belief on the true hypothesis converges to 1 1 1 1 exponentially fast with rate characterized in  [ 2 ] and further non-asymptotic characterization in [ 5 ] . It has been noted that the concentration of beliefs depends on the network topology as well as the chosen weights.

While most literature focuses on the convergence of beliefs [ 3 , 4 , 5 , 2 , 6 ] , few look into the convergence of error probability [ 7 , 8 , 9 ] , which is arguably the most direct performance metric in hypothesis testing problems. As the convergence of error probability has not been well understood, it remains unclear what the price of decentralization on the detection performance is. There are several natural questions to be addressed. First, what is the optimal probability of error when these belief-consensus-based learning rules are utilized, and how does it depend on the network topology as well as the weights chosen by each node? Compared to the centralized performance, how much is lost? Second, with slight global knowledge about the policies of other nodes, how to improve the probability of error? Can it approach the performance of the centralized case? If it can, what is the additional cost for obtaining the needed global information?

I-A Contribution

In this work, the above questions are addressed in the case of binary detection. We propose a generalization of the social learning rule in  [ 2 ] and characterize the error exponents using tools in large deviation theory  [ 10 ] . As a result, the error exponents of the original learning rule in  [ 2 ] are characterized, which turn out to be strictly smaller than the error exponents in the centralized case. The reason is that the decentralized sources are not weighted equally due to the convergence of the Markov chain governing the consensus. Figure  1 illustrates the gap in error exponents with a simple example. In the example, 300 scale-free networks with 100 nodes in each are sampled. Each node serves as an independent Bernoulli source having consensus weights uniformly distributed to its neighbors. Gathering the consensus weights into a right stochastic matrix, the Markov chain with such corresponding transition matrix induces a unique stationary distribution denoted by π 𝜋 \pi italic_π under some minor assumptions. The figure shows that the error exponent of the original learning rule decreases with the network imbalance . We quantify the imbalance of the network with the 2-norm between π 𝜋 \pi italic_π and the uniform stationary distribution with each entry being 0.01 for this case. Notice that only when the network is balanced , the original learning rule obtain the optimal error exponent depicted by the blue dashed line.

The proposed generalization compensates for the imbalance of the original network consensus. To do so, the likelihood functions in the learning rule in  [ 2 ] are weighted geometrically (that is, they are raised to different exponents) to equalize the importance of the sources. We show that if each agent knows the value of the stationary distribution of the consensus Markov Chain at that node, the optimal error exponent in the centralized case is achieved by properly choosing the geometric weightings. Since the first-order results do not reveal the price of decentralization, we further derive upper bounds on the higher-order asymptotics by extending Strassen’s seminal result  [ 11 ] for the centralized case to our decentralized setting with the aid of the non-i.i.d. version of Esseen’s theorem  [ 12 , 13 ] and the convergence result on the Markov chains  [ 14 ] . It turns out that the effect of decentralization is revealed as at most a constant term in the higher-order asymptotics.

The value of the stationary distribution at each node is the slight global information that enables each agent to achieve the centralized error exponent. To obtain such global knowledge, we propose a simple decentralized iterative estimation method. The estimation method only requires bi-directional communication for each pair of nodes forming a directed edge in the network. The estimation error on the stationary distribution vanishes exponentially with the number of iterations by the convergence result on Markov chains  [ 14 ] . Numerical results suggest that the gap between the optimal error exponent and that with the geometric weightings being the estimated stationary distribution also vanishes exponentially with the number of iterations.

Part of the work has been published at the 2020 IEEE Information Theory Workshop  [ 1 ] including Theorem  1 ,  2 ,  3 , and  4 . Additionally in this journal version, Corollary  1 and Theorem  5 ,  6 in Section  III-C capture the constant time delay in the decentralized case and characterize the bound on the higher-order asymptotics of the Bayes risk. Furthermore, in Section  V , we demonstrate the impact of network imbalance, the performance of our proposed learning rule, and the effect of quantized communications. In Section  VI , we discuss the cases where assumptions are removed and we show that our results could be extended to the case of multiple hypothesis testing.

I-B Related Work

The overview papers [ 15 , 16 ] provide extensive surveys on the algorithms and results for distributed learning. As for distributed hypothesis testing, the convergence of beliefs is considered in  [ 2 , 3 , 4 , 5 , 17 , 6 , 18 , 19 ] . A learning rule adopting linear consensus on the beliefs (in contrast to the log-beliefs considered in this work) is studied in  [ 3 , 4 ] , while  [ 2 ] achieves a strictly larger rate of convergence by adopting consensus over the log-beliefs. An iterative local strategy for belief updates is investigated in  [ 5 ] , and a non-asymptotic bound on the convergence of beliefs is provided. Based on the work in  [ 2 ] , the convergence of beliefs is studied under the setting of weakly connected heterogeneous networks in  [ 6 ] where the true hypothesis might be different among the components of the network. Error exponents are studied in  [ 7 , 8 ] where the weight matrices are assumed to be symmetric, stochastic, and random. In contrast, we consider general asymmetric and stochastic weight matrices which are deterministic, and our results imply that optimal error exponent is achieved even if we naively apply the learning rule in  [ 2 ] whenever the weight matrix is doubly stochastic. General asymmetric and stochastic weight matrices are also considered in  [ 9 ] . The main difference from our work is that they focus on optimizing the weight matrix under a given decision region while we achieve the optimal error exponent through modifying the learning rule. We provide a decentralized method for estimating the values of the stationary distribution of the consensus Markov Chain. The estimation method only requires bi-directional communication for each pair of nodes forming a directed edge. Meanwhile, optimizing the weight matrix needs to be done globally with a center that knows the entire network topology.

I-C Paper Organization

The rest of this paper is organized as follows. In Section  II , we formulate our problem and introduce the learning rule proposed in  [ 2 ] . In Section  III , we propose our modified learning rule and show our main results. The proofs are then provided in Appendices. We proposed alternative learning rules for estimating the needed parameters and discuss the convergence of the estimation in Section  IV . In Section  V , we provide simulation results on the impact of network imbalance, estimation, and quantization. We make several discussions in Section  VI including removing the assumptions on the network and extending our results to the multiple hypothesis testing problems. Finally, we make a brief conclusion in Section  VII .

II Problem Formulation and Preliminaries

Ii-a problem formulation.

Consider n 𝑛 n italic_n nodes collaborating on decentralized binary hypothesis testing. For notational convenience, let [ n ] delimited-[] 𝑛 [n] [ italic_n ] denote { 1 , 2 , … , n } 1 2 … 𝑛 \{1,2,...,n\} { 1 , 2 , … , italic_n } . Let G ⁢ ( [ n ] , ℰ ) 𝐺 delimited-[] 𝑛 ℰ G([n],\mathcal{E}) italic_G ( [ italic_n ] , caligraphic_E ) denote the underlying directed graph and 𝒩 ⁢ ( i ) ≜ { j ∈ [ n ] : ( i , j ) ∈ ℰ } ≜ 𝒩 𝑖 conditional-set 𝑗 delimited-[] 𝑛 𝑖 𝑗 ℰ \mathcal{N}(i)\triangleq\{j\in[n]:(i,j)\in\mathcal{E}\} caligraphic_N ( italic_i ) ≜ { italic_j ∈ [ italic_n ] : ( italic_i , italic_j ) ∈ caligraphic_E } denote the neighborhood of node i 𝑖 i italic_i . Node i 𝑖 i italic_i can get information from node j 𝑗 j italic_j only if j ∈ 𝒩 ⁢ ( i ) 𝑗 𝒩 𝑖 j\in\mathcal{N}(i) italic_j ∈ caligraphic_N ( italic_i ) . To make sure that information can reach all the nodes in the network, we need the following assumption.

Assumption 1 .

The directed graph G 𝐺 G italic_G is strongly connected.

II-B Social Learning Rule

In the conventional hypothesis testing problem, the likelihood ratio serves as the optimal statistics in several problems such as the Neyman-Pearson problem and Bayes setting, where the Bayes risk is minimized. The problem in the decentralized case is then whether each node can obtain a statistic that is exactly or close enough to the optimal statistic in the centralized case. A naive approach is that each node simply exchanges its raw observations with others so that each node eventually obtain all the observations among the node. However, the naive approach suffers a high communication cost.

Lalitha et al.   [ 2 ] proposed a natural approach for decentralized hypothesis testing using the notion of belief propagation. As we will see in later content, the ratio of the beliefs in the proposed learning rule somehow mimics the likelihood ratio but in a slightly tilted form.

Let us describe the proposed learning rule in  [ 2 ] as follows. At time step t 𝑡 t italic_t , each node i ∈ [ n ] 𝑖 delimited-[] 𝑛 i\in[n] italic_i ∈ [ italic_n ] maintains two real vectors: the public belief vector q i ( t ) ∈ Δ m subscript superscript 𝑞 𝑡 𝑖 subscript Δ 𝑚 q^{(t)}_{i}\in\Delta_{m} italic_q start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∈ roman_Δ start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT and the private belief vector b i ( t ) ∈ Δ m subscript superscript 𝑏 𝑡 𝑖 subscript Δ 𝑚 b^{(t)}_{i}\in\Delta_{m} italic_b start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∈ roman_Δ start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT , which are updated iteratively as t − 1 𝑡 1 t-1 italic_t - 1 changes to t 𝑡 t italic_t . Node i 𝑖 i italic_i weights the received information from j 𝑗 j italic_j by W i ⁢ j subscript 𝑊 𝑖 𝑗 W_{ij} italic_W start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT which could be seen as the relative confidence that node i 𝑖 i italic_i has in node j 𝑗 j italic_j .

• 𝑖 superscript 𝜃 X_{i}^{(t)}\sim P_{i,\theta^{*}} italic_X start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT ∼ italic_P start_POSTSUBSCRIPT italic_i , italic_θ start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT end_POSTSUBSCRIPT .

Each node i 𝑖 i italic_i updates its public belief vector such that

where b i ( t ) ⁢ ( θ ) superscript subscript 𝑏 𝑖 𝑡 𝜃 b_{i}^{(t)}(\theta) italic_b start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT ( italic_θ ) denotes the θ 𝜃 \theta italic_θ -th entry of b i ( t ) superscript subscript 𝑏 𝑖 𝑡 b_{i}^{(t)} italic_b start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT .

Each node j 𝑗 j italic_j sends its public belief vector b j ( t ) superscript subscript 𝑏 𝑗 𝑡 b_{j}^{(t)} italic_b start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT to node i 𝑖 i italic_i if j ∈ 𝒩 ⁢ ( i ) 𝑗 𝒩 𝑖 j\in\mathcal{N}(i) italic_j ∈ caligraphic_N ( italic_i ) .

Each node i 𝑖 i italic_i updates its private belief vector, q i ( t ) superscript subscript 𝑞 𝑖 𝑡 q_{i}^{(t)} italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT , such that

The results in  [ 2 ] show that the entry q i ( t ) ⁢ ( θ ∗ ) subscript superscript 𝑞 𝑡 𝑖 superscript 𝜃 q^{(t)}_{i}(\theta^{*}) italic_q start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_θ start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ) converges to one almost surely while the others converge to zero. The rate is also characterized as the weighted sum of the Kullback-Leibler divergence among the distributions over each node.

Though  [ 2 ] characterized the convergence performance of the belief vectors, they did not study the probability of error, which seems to be a more concerned perspective in the conventional hypothesis testing problem. We will soon show that the learning rule proposed in  [ 2 ] suffers a gap in the error exponent compared to the centralized case in Section  II-D . Before then, we introduce the probability of error we consider in the rest of our work.

II-C Log-Belief Ratio Test and the Probability of Error

For the centralized binary detection problem, the randomized likelihood ratio test is optimal (in the Neyman-Pearson problem and the Bayes setting). However, in the decentralized setting, none of the nodes knows the joint likelihood of all the observations in the network and thus no one can carry out the likelihood ratio test. Under the above-mentioned learning rule, we consider the binary hypothesis testing problem , and a natural test based on the private belief vector maintained by each node emerges, which is defined as follows.

Definition 1 (Log-Belief Ratio) .

Under the binary hypothesis testing problem, let ℓ i ( t ) subscript superscript ℓ 𝑡 𝑖 \ell^{(t)}_{i} roman_ℓ start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT be the (private) log-belief ratio on node i 𝑖 i italic_i at time t 𝑡 t italic_t such that

Definition 2 (Log-Belief Ratio Test) .

For all t ∈ ℕ 𝑡 ℕ t\in\mathbb{N} italic_t ∈ blackboard_N , let η i ( t ) ∈ [ 0 , 1 ] superscript subscript 𝜂 𝑖 𝑡 0 1 \eta_{i}^{(t)}\in[0,1] italic_η start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT ∈ [ 0 , 1 ] and γ i ( t ) ∈ ℝ superscript subscript 𝛾 𝑖 𝑡 ℝ \gamma_{i}^{(t)}\in\mathbb{R} italic_γ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT ∈ blackboard_R . Define φ i ( t ) superscript subscript 𝜑 𝑖 𝑡 \varphi_{i}^{(t)} italic_φ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT as the log-belief ratio test of node i 𝑖 i italic_i such that

It is straightforward to see that if there is only a single node, under the learning rule in Section  II-B , the private log-belief ratio ℓ i ( t ) superscript subscript ℓ 𝑖 𝑡 \ell_{i}^{(t)} roman_ℓ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT equals to the log-likelihood ratio, and hence the test is equivalent to the likelihood ratio test.

Next, let us define the two types of probabilities of error for the log-belief ratio test.

Definition 3 (Probability of Error) .

The type-I and type-II error probabilities for each node i 𝑖 i italic_i denoted by α i ( t ) ⁢ ( η i ( t ) , γ i ( t ) ) superscript subscript 𝛼 𝑖 𝑡 superscript subscript 𝜂 𝑖 𝑡 superscript subscript 𝛾 𝑖 𝑡 \alpha_{i}^{(t)}(\eta_{i}^{(t)},\gamma_{i}^{(t)}) italic_α start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT ( italic_η start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT , italic_γ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT ) and β i ( t ) ⁢ ( η i ( t ) , γ i ( t ) ) superscript subscript 𝛽 𝑖 𝑡 superscript subscript 𝜂 𝑖 𝑡 superscript subscript 𝛾 𝑖 𝑡 \beta_{i}^{(t)}(\eta_{i}^{(t)},\gamma_{i}^{(t)}) italic_β start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT ( italic_η start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT , italic_γ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_t ) end_POSTSUPERSCRIPT ) are defined as

It is then straightforward to come up with a decentralized Neyman-Pearson problem: