Different result between Phoenix and SAS

Phoenix Certara Forums Phoenix WNL basics
1

Dear all

We concdut a partial replicated study (RRT/RTR/TRR) and used Phoenix 6.4 and SAS 9.4 for analysis.

In Phoenix, default setting was used for ABE and result is for ratio=93.03 and 90% CI=(83.81-103.27).

In SAS, the result obtained for ratio=93.01 and 90% CI=(83.87-103.36).

the following code was used for mixed procedure and 90%CI,

data mydata;
set mydata
lnCmax=log(Cmax);
run;

proc mixed data=mydata;
class Seq Period Drug;
model lnCmax= Seq Period Drug / ddfm=satterth;
random Drug/type=FA0(2) sub=Subject G;
repeated/grp=Drug sub=Subject;
estimate ‘test-ref’ Drug -1 1/ CL alpha=0.10;
ods output Estimates=out3;
run;

data test5;
set out3;
x=(Estimate2)-(StdErr2);
boundx=(max((abs(lower)),(abs(upper))))2;
GeometricMean = exp(Estimate)*100;
StdErrG =( sqrt((Estimate2)*(StdErr**2)))*100;
LowerCLMean = exp(Lower)*100;
UpperCLMean = exp(Upper)*100;
run;

Before this case, the phoenix and SAS work well, but in this case, the different occurs and I cannot figure out why it happens.Test.csv (1.8 KB)

2

Hi Tsang,

I could confirm your ABE-result in PHX6.4 (attached). BTW, you would also pass RSABE. I think that we need a SASian checking your code.

tsang0323Example.phxproj (2.73 MB)

3

Dear Tsang, Dear Helmut, Dear All,

I’d confirm the SAS results although I’m not a real SASian :wink:

Let’s take a look at the LinMix core output:

Warning 11094: Negative final variance component. Consider omitting this VC structure.

And now the SAS log:

NOTE: An infinite likelihood is assumed in iteration 2 because of a nonpositive definite estimated R matrix for subject 101.
<a lot of notes for each iterations, because I turned on ITDETAILS>

NOTE: Estimated G matrix is not positive definite.
NOTE: Asymptotic variance matrix of covariance parameter estimates has been found to be singular and a generalized inverse was used. Covariance parameters with zero variance do not contribute to degrees of freedom computed by DDFM=SATTERTH.

@Helmut:

Do you remember this thread? :slight_smile:

As Detlew Labes pointed out, The whole story “Use Proc MIXED code for Partial replicate design” is just a mystery!

Take a look at the final variance:

LinMix:

Final variance parameter estimates:
lambda(1,1)_11 0.272905
lambda(1,2)_11 0.542748
lambda(2,2)_11 0.161448
Var(PeriodFormulationSubject)_21 0.121235
Var(PeriodFormulationSubject)_22 -0.0512752

Negative variance? I love that! As you noted in the thread above ‘an ambiguous attempt of the REML algo to obtain the within-subject variance of the Test formulation’

What the Power to know says:

Covariance Parameter Estimates

Cov Parm Subject Group Estimate

FA(1,1) subject 0.2855

FA(2,1) subject 0.5190

FA(2,2) subject 0

Residual subject treatment R 0.1145

Residual subject treatment T 0

I suspect SAS uses bounds for variance components (lower bound=0). And the AIC for SAS is worse (100.9)
Unfortunately nobound does not work here:
WARNING: Stopped because of infinite likelihood.

What about different covariance matrices?

FAO(1)

using in SAS:

PI 93.005843433
CI 83.68626794 103.36327722
using in PHX:
PI 93.02813856
CI 83.80572510 103.26543388
The results are also different

CSH produces similar results in SAS and
ERROR 11070: Error in Satterthwaite DF. Try Residual DF option if not already set. Model may be over-specified.
Oh yes, the lovely one. Phoenix, you’re right, a little bit overspecified…

So till the moment I don’t know how to reproduce these results in PHX…
May be the developers can try to add the bounds like Proc Mixed did. Not sure.

BR,
Mittyright

4

Dear Mittyright, Dear Helmut and Dear all

Lot’s thank you for response :slight_smile:

I suppose you know the SAS code come from FAD guidance,I found if I put subject into the model in SAS （model lnCmax= Seq Period Drug subject/ ddfm=satterth;）then I will get similar result to Phoenix. (Due to away from office, cannot put the result,sorry) Therefoe,a question, the model setting in phoenix followed guidance as pharsight said,yes, it wooks well in other cases, but why not in this case? Dose it means problem come from dara? And what different by adding subjet in the model?

5

I did try to put this issue in BA/BE forum,but I forgot the username(too many in my mind）…

6

Dear Tsang,

I assume you changed not only this line

model lnCmax= Seq Period Drug subject/ ddfm=satterth;

otherwise SAS doesn’t want to converge (and that’s reasonable; I do not understand the model with the same random and fixed effect).

Maybe you mean the model from method B in EMA QA PKWP?

Therefoe,a question, the model setting in phoenix followed guidance as pharsight said,yes, it wooks well in other cases, but why not in this case? Dose it means problem come from dara?

The problem is in the FDA experts mind :slight_smile:

To insist to use the model for the designs which are not defined to be in use with such a code.

As a result all software tries to do its best in final variance parameters estimation even if it is impossible (how to estimate the variance for T treatment?). I suspect some limits for parameters are coming into play

BR,

Mittyright

7

Hi to all,

the partial replicate is a lousy design and IMHO, should be abandoned. I don’t see any (scientific!) reasons to use it. If you want to have only three periods consider to use the RTR|TRT full replicate in the future.

Now for the bad news. The FDA’s covariance specification in Proc Mixed (in SAS-lingo FA0(2)) leads to an over-specified model. Same in PHX Banded No-Diagonal Factor Analytic (2). Sometimes it „works”, sometimes in doesn’t (that’s independent from the software). If you are lucky, you get only a warning. In rare cases the optimizer fails to arrive at a solution at all. Study done, lot of money invested, click the button, computer says “no!”, statistician blamed.

We discovered that in most cases FA0(1) resolved the issue. Seems that your data set is one of the few where it doesn’t help (same CI and still a warning in PHX).

8

Hi Mittyright!

Sure. Although you used the wrong smiley. Better would have been :angry:

I’ve met John (user jag009 in the BEBA-Forum) last September in Rockville. He had one data set failing to converge both in SAS and PHX. He sent two letters to the FDA’s Donald Schuirmann asking for clarification and never received an answer!

9

Dear all

Sorry for I am not a SAS user, my collegue is and use code from him.

As Mittyright mentioned “SAS doesn’t want to converge (and that’s reasonable; I do not understand the model with the same random and fixed effect).”, however, I tried again the code and discovered the model (model lnCmax= Seq Period Drug / ddfm=satterth; rather than I said “model lnCmax= Seq Period Drug subject/ ddfm=satterth;”) is used. The subject factor where it is locaded is a matter for the model? I attach the code and result and kindly check, thank you!!

Test.txt (777 Bytes)Test.PDF (862 KB)

10

Hi Tsang,

I saw that you registered at the BEBA-Forum a couple of minutes ago. Don’t forget to activate your acount within the next 24 hours (click the link sent to your e-mail address).

Also not a SASian but, yes, AFAIK the order of effects matters in SAS.

11

Thank you for telling me the order in the model is a matter in SAS. I won’t forget to activate my account.

By the way,if the study was completed in two group (I said 15/14),and I dun’t know how to modify the model specification in phoenix (i know how to do it in 2 way crossover study but not partial replicated stuy). My collegue apply the model for crossover study in the replicated study, I am not so sure it is suitable. I did lot’s of google work and cannot find anywhere define the model. I also post the question here,but only got a reply for 2way crossover.

12

Dear Tsang,

I was not correct regarding PROC MIXED, yes, the Subject is a blocking variable, not the Random factor by itself

I’d confirm that the solution you posted works, but PROC MIXED does not feel good.

The covariance structure is not the same as in Phoenix results.

Please see Type3 Tests: Sequence is 0/. for NumDF/DenumDF

I also tried to get LSmeans, but SAS could not estimate them

All is fine in PHX.

So for me it’s a kind of magik why test-ref estimate is the same

By the way,if the study was completed in two group (I said 15/14),and I dun’t know how to modify the model specification in phoenix (i know how to do it in 2 way crossover study but not partial replicated stuy). My collegue apply the model for crossover study in the replicated study, I am not so sure it is suitable. I did lot’s of google work and cannot find anywhere define the model. I also post the question here,but only got a reply for 2way crossover.

Did you try this one?

BR,

Mittyright

13

Hi Mittyright & Tsang,

Yep, but Tsang already asked somewhere else and the answer was also for a 2×2 crossover.

This question pops up from time to time in the BEBA-Forum. One example.

The fact that this code „worked” doesn’t mean a thing (IMHO, it is too simple). The FDA’s group models 1 & 2 for 2×2 crossovers (or for multi-site studies as well) serve two purposes:

  1. Test whether there is a group-by-treatment interaction (at the 10% level). If yes, groups must not (!) be pooled since the treatment effect cannot be unbiased estimated. However, BE can be demonstrated in the largest group if the requirements for the minimum sample size (12 in a 2×2 crossover) are fulfilled.
  2. If the group-by-treatment interaction is not significant the data of groups can be pooled and the respective term dropped from the model.

Problems:

  • Which are the analogous models for replicate designs?
  • Even if we find ones, we will see 10% false positives in #1 and will not be allowed to pool.
    But for RSABE the minimum sample size is 24! In Tsang’s example (15|14) this would be the end of the story.
14

Dear Mittyright

I knew Sequence is 0 in type III test and also I cannot get the least squre mean (I thought I use wrong code), but the result is same to phoenix, amazing, right? That’s why I am wondering what happed. I will put this issue into the BA/BE forum after I activting my account.

I read the thread you provided. Therefore, we currently using this model like this (Group, Seq, Subject(Seq* Group), Period(Group), Drug, Drug* Group)

However, I think discussion in the thread is for 2 stage 2x2 crossover design, is it fine in the replicated study?

15

Hi Tsang,

I don’t think so (see my last post above). Can you upload a new text.csv including a column for the groups (or tell us which subjects belong to which group: I assume 101–115 = 1 and 117–131 = 2)?

16

Dear Helmut:

As your assumption, group 1 for 101 ~ 115 and others for group 2. I still attach the text for you.

The model I mentioned is used in SAS, I did try to do it in the phoenix but alwasy fail. ><

Moreover, the guidance in my country not yet to define sample size for RSABE and they are currently reviseing now.

Test.txt (777 Bytes)

17

Dear Tsang, Dear Helmut,

@Helmut

Yep, but Tsang already asked somewhere else and the answer was also for a 2×2 crossover.

Oh sorry I forgot about complicated structure of random effects…

This question pops up from time to time in the BEBA-Forum. One example.

The fact that this code „worked” doesn’t mean a thing (IMHO, it is too simple).

I think if such a model existed, it would be published on BEBAC))

Problems:

  • Which are the analogous models for replicate designs?

I think we never ever get to know that. Since 2008 ‘group’ term inside replicate designs is not mentioned anywhere in any guidelines (Russian?!).

I assume the experts are fine with conventional models

Or some Mahatma-level person will be enlightened by SAS God and we will see the light.

Even if we find ones, we will see 10% false positives in #1 and will not be allowed to pool.

Same story as for 2X2.

@Tsang

Moreover, the guidance in my country not yet to define sample size for RSABE and they are currently reviseing now.

Does the previous guideline mention the group term for bioequivalence studies? Is it possible to use some EMA models for evaluation? They are more robust.

The model I mentioned is used in SAS, I did try to do it in the phoenix but alwasy fail. ><

Same thing.

I was lucky only with the model using Variance components with Group both in Random and Repeated sheets.

BTW I would not trust these results.

BR,

Mittyright

18

My code have Subject definition in CLASSES statement (and i have error without it).

PROC MIXED data=pkdata; /* mmeq mmeqsol ranks; */
CLASSES Sequence Subject Period Formulation;
MODEL logAUC = Sequence Period Formulation/ DDFM=SATTERTH;
RANDOM Formulation/TYPE=FA0(2) SUB=Subject G;
REPEATED/GRP=Formulation SUB=Subject;
ESTIMATE ‘T vs. R’ Formulation 1
-1/CL ALPHA=0.1;
RUN;

But i think a problem not in this thing.

In SAS log notes you probably can find “Estimated G matrix is not positive definite.” (i don’t know where in Phoenix WinNonlin you can find same notification). In this case some results may be different. For example SAS, SPSS and Phoenix give me different values of degree of freedom when Hessian is not positive definite. You can also change FA0(2) to CSH in SAS and find no changes in SAS results, but if you do this in Phoenix you will find changes. Maybe, if Hessian not positive - there are different methods of estimation to continue procedure. There is no source code for audit (nobody knows how calculation proceeds in SAS, Phoenix and SPSS).

Strictly speaking, if Hessian is not positive defined you should change variance structure (SPSS recommendation).

19

Vladmir, I discussed this with one of my colleagues and we see if we look into SAS help, it says:

http://support.sas.com/kb/22/614.html

22614 - What does PROC MIXED’s “Estimated G matrix not positive definite” message mean? - SAS Support

The Estimated G matrix not positive definite message usually indicates that one or more variance components on the RANDOM statement is/are estimated to be zero and could/should be removed from the model.. To remove those variance components from the model, remove the associated term from the RANDOM statement and resubmit the PROC MIXED statements in the progam editor or rerun the PROC MIXED …

So the problem is the same as described above: due to model overspecification SAS cannot find an applicable solution.

When the design is partially replicated, s2b,T and s2w,T cannot be estimated, only their sum can be, leading to overspecification and convergence problems.

Does that help confirm your understanding?

Simon