5
$\begingroup$

I'm not sure whether I'm assessing this problem correctly.Suppose I want do demonstrate that an adverse event occurs in less than 20% of the participants (but I hope/assume that it actually NEVER happens). Now I want to calculate how many participants I need to demonstrate this at a certain alpha/power.

I thought this is just a simple one sample binomial test with$H_0: p\geq 0.2$ vs$H_1: p<0.2$, which means an effect size of$g=0.2$ and$p_0=0$.

However, this does not work when I use sample size calculating tools. So there seems to be some mistake in my idea- can anybody help?

This is what I get from G*Power:

enter image description here

In the tool you mentioned I have an actual alpha level of 27%, in G*Power it's under 5%, as desired. Is this the reason?

User1865345's user avatar
User1865345
11.9k13 gold badges25 silver badges41 bronze badges
askedyesterday
Kathrin's user avatar
$\endgroup$
1
  • 1
    $\begingroup$If you assume the rate is 0.2 you're never going to reliably show that it's less -- that would be an underestimation! Instead you'd assume some value < 0.2 and aim for an upper uncertainty limit below the threshold. The closer the assumption is to 0.2 the larger $n$ you'll need. As an extreme but simple example consider the rule of 3 (basically Clopper-Pearson): as long as you observe 0 events an approx. 95% upper limit on the rate is $3/n$, so observing $0/15$ gives ~95% confidence that the rate is < 0.2. Observing >0 will require slightly more complicated calculations but the idea holds.$\endgroup$Commentedyesterday

1 Answer1

11
$\begingroup$

Ok, so this is fairly sensitive to what you believe and are willing to assume

Case 1

You are confident that$p$ isvery small, and the Client/Regulations/Risk Policy/Whatever says you have to prove that$p<0.2$. This is a "Rule of Three" setting: if you see$n$ participants and zero events, you will be confident that$p<3/n$. This is based on

> ppois(0,lambda=3)[1] 0.04978707

ie, the probability of getting 0 events when the expected number is 3 is just under 5% if$n$ is large. If$n$ is not large the probability is strictly smaller, so you are still ok

So, since you want$3/n<0.2$, you get$n=15$ as sufficient by the Rule of Three, or$n=14$ with more careful calculation

> pbinom(0,14,.2)[1] 0.04398047

Since you are sure$p$ is small, you are sure that with 14 participants you will see 0 events, and 0 events is enough to convince the Client/Regulations/Risk Policy/Whatever that$p<0.2$

If you go tothis link you will find a reputable calculator that says the one-sided sample size is 14 if$p$ is very small (alternative proportion) and you want to prove$p<0.2$ (null proportion)

If you try to set$p=0$, the calculator will say that proportions have to be between 0 and 1, which is fair: you can't really be confident that it'szero, and putting$p=0.001$ or$p=0.0001$ or something like that will better represent what you believe.

Case 2

The problem with case 1 is that it only works if you are confident (and correct) about$p\approx 0$. Suppose you worry that maybe$p=0.01$. In that case there's about a 13% change of seeing 1 or 2 events.

> dbinom(0:3,14,.01)[1] 0.8687458128 0.1228529432 0.0080661023 0.0003259031

Your test based on the Rule of Three will now not be very useful. You need a stricter threshold than seeing a single event and a larger sample size. You might need to fiddle around with thresholds and sample size a bit to find a threshold$m$ and sample size$N$ such that (for 5% level and 95% power)

pbinom(m+1,N,.2)>0.95pbinom(m,N,.01)<0.05

At the same link as above, you can specify the alternative probability as 0.05 and get a sample size of 48 and a threshold of 5. Obviously there is more than one way to do it: by making$N$ larger you can make$m/N$ smaller.

Update

The "exact alpha" part of the calculator I linked to is testing the hypothesis the other way around or something. You need to specify that the lower critical value is 0 and the upper critical value is whatever it is: 1 in the low-$p$ setting

answeredyesterday
Thomas Lumley's user avatar
$\endgroup$
6
  • $\begingroup$General note, if you want more than 95% confidence, both cases can be easily adapted. It will of course take bigger sample sizes but the general method remains the same.$\endgroup$Commented18 hours ago
  • $\begingroup$Indeed -- and the calculator I linked to will handle this$\endgroup$Commented18 hours ago
  • $\begingroup$Thank you so much, this helps me a lot. Indeed I tried 0 in the calculator and that was why it didn't work out. Now, assuming 0.01, testing for p<0.2 and aiming for a power of 0.8 I get a sample size of 12 which is fairly doable, I think.$\endgroup$Commented16 hours ago
  • $\begingroup$However, if I do this in G*Power, I get 14 and I cannot really see where this difference comes from.$\endgroup$Commented16 hours ago
  • $\begingroup$I put a screenshot below ( I couldn't add it here).$\endgroup$Commented12 hours ago

Your Answer

Sign up orlog in

Sign up using Google
Sign up using Email and Password

Post as a guest

Required, but never shown

By clicking “Post Your Answer”, you agree to ourterms of service and acknowledge you have read ourprivacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.