[問題] Snow套件加速運算 Stepwise Regression
[問題類型]:
效能諮詢(我想讓R 跑更快)
[軟體熟悉度]:
使用者(已經有用R 做過不少作品)
[問題敘述]:
各位好，目前我正在寫一支配Stepwise Regression的程式；且已經可以成功執行。
但是在執行效能上還有很大的進步空間
目前所配的Variables 共有80左右，所以配逐步回歸的方式來選取留下的Variable
因為每個變數都需要建立一個獨立模型
所以總共會跑80次左右的iterstions
總共執行時間約莫落在2.5HR左右，所以開始考慮效能提升問題
有遇到一些問題需要各位協助排除障礙，先在此謝謝各位
以下的範例，我先設定跑2次的iteration用來測試效能
[程式範例]:
這是我已經寫好可以Run的程式碼
##########可以Run的###########
system.time(lapply(1:2,function(i)(
{
print(i)
TrainModel<-cbind(setnames(TrainDT[7:nrow(TrainDT),i,with=F],paste0(names(DT[1,i,with=FALSE]),'_y')),TrainDT[1:(nrow(TrainDT)-6),1:length(TrainDT),with=F])
PracticeModel<-cbind(setnames(PracticeDT[7:nrow(PracticeDT),i,with=F],paste0(names(DT[1,i,with=FALSE]),'_y')),PracticeDT[1:(nrow(PracticeDT)-6),1:length(PracticeDT),with=F])
resp<-grep('_y',names(TrainModel),value=T)
pre<-grep('01F',names(TrainModel),value =T)
pre<-pre[2:length(pre)]
addq<-function(x) paste0("`",x, "`")
Model<-as.formula(paste(addq(resp),paste(lapply(pre, addq),collapse =
'+'),sep = '~'))
FitModel<-lm(Model,data=TrainModel)
#Fitmodel<-lm(`01F0017S_y`~.,data=TrainModel)
#Fitmodel<-lm(as.matrix(TrainDT[7:nrow(TrainDT),i,with=F])~as.matrix(TrainDT[1:(nrow(TrainDT)-6),1:length(TrainDT),with=F]),data=TrainDT)
stepwise<-step(FitModel,sacle=0,direction = 'both')
write.csv(stepwise$coefficients,file =
paste0(names(DT[1,i,with=FALSE]),'_Coefficients','.csv'))
write.csv(cbind(TrainModel[[1]],stepwise$fitted.values,stepwise$residuals),file
= paste0(names(DT[1,i,with=FALSE]),'_Residual','.csv'))
write.csv(cbind(PracticeModel[[1]],predict(stepwise,PracticeModel),PracticeModel[[1]]-predict(stepwise,PracticeModel)),file
= paste0(names(DT[1,i,with=FALSE]),'_Predict','.csv'))
#PredictData<-predict(stepwise,PracticeDT)
})))
每一個iteration在最後會丟出三個我需要參數的csv，總共耗時約2.5HR
###########配合snow 套件的程式###########
clusterfun<-function(i){
print(i)
TrainModel<-cbind(setnames(TrainDT[7:nrow(TrainDT),i,with=F],paste0(names(DT[1,i,with=FALSE]),'_y')),TrainDT[1:(nrow(TrainDT)-6),1:length(TrainDT),with=F])
PracticeModel<-cbind(setnames(PracticeDT[7:nrow(PracticeDT),i,with=F],paste0(names(DT[1,i,with=FALSE]),'_y')),PracticeDT[1:(nrow(PracticeDT)-6),1:length(PracticeDT),with=F])
resp<-grep('_y',names(TrainModel),value=T)
pre<-grep('01F',names(TrainModel),value =T)
pre<-pre[2:length(pre)]
addq<-function(x) paste0("`",x, "`")
Model<-as.formula(paste(addq(resp),paste(lapply(pre, addq),collapse =
'+'),sep = '~'))
FitModel<-lm(Model,data=TrainModel)
#Fitmodel<-lm(`01F0017S_y`~.,data=TrainModel)
#Fitmodel<-lm(as.matrix(TrainDT[7:nrow(TrainDT),i,with=F])~as.matrix(TrainDT[1:(nrow(TrainDT)-6),1:length(TrainDT),with=F]),data=TrainDT)
stepwise<-step(FitModel,sacle=0,direction = 'both')
write.csv(stepwise$coefficients,file =
paste0(names(DT[1,i,with=FALSE]),'_Coefficients','.csv'))
write.csv(cbind(TrainModel[[1]],stepwise$fitted.values,stepwise$residuals),file
= paste0(names(DT[1,i,with=FALSE]),'_Residual','.csv'))
write.csv(cbind(PracticeModel[[1]],predict(stepwise,PracticeModel),PracticeModel[[1]]-predict(stepwise,PracticeModel)),file
= paste0(names(DT[1,i,with=FALSE]),'_Predict','.csv'))
#PredictData<-predict(stepwise,PracticeDT)
}
cluster <- makeCluster(type="SOCK",c("localhost", "localhost", "localhost",
"localhost"))
system.time(parLapply(cluster,1:2,clusterfun))
stopCluster(cluster)
但是在執行上得到這個錯誤
> cluster <- makeCluster(type="SOCK",c("localhost", "localhost", "localhost",
"localhost"))
> system.time(parLapply(cluster,1:2,clusterfun))
Error in checkForRemoteErrors(val) :
2 nodes produced errors; first error: 沒有這個函數 "setnames"
Timing stopped at: 0 0 0.01
> stopCluster(cluster)
請問各位該如何排除這樣的障礙，謝謝各位的指教
[環境敘述]:
> version
_
platform x86_64-w64-mingw32
arch x86_64
os mingw32
system x86_64, mingw32
status
major 3
minor 2.1
year 2015
month 06
day 18
svn rev 68531
language R
version.string R version 3.2.1 (2015-06-18)
nickname World-Famous Astronaut
[關鍵字]:
選擇性，也許未來有用
Snow, data.table,Stepwise Regression
--
※ 發信站: 批踢踢實業坊(ptt.cc), 來自: 125.227.5.157
※ 文章網址: https://www.ptt.cc/bbs/R_Language/M.1462421341.A.18F.html
→
05/05 12:11, , 1F
05/05 12:11, 1F
→
05/05 12:25, , 2F
05/05 12:25, 2F
→
05/05 12:26, , 3F
05/05 12:26, 3F
→
05/05 12:26, , 4F
05/05 12:26, 4F
→
05/05 12:27, , 5F
05/05 12:27, 5F
→
05/05 14:13, , 6F
05/05 14:13, 6F