Re: NFS 75 second stall


=0A=0A--- On Thu, 7/1/10, Garrett Cooper <yanefbsd@gmail.com> wrote:=0A=0A>=
 From: Garrett Cooper <yanefbsd@gmail.com>=0A> Subject: Re: NFS 75 second s=
tall=0A> To: "alan bryan" <alan.bryan@yahoo.com>=0A> Cc: freebsd-stable@fre=
ebsd.org=0A> Date: Thursday, July 1, 2010, 12:23 PM=0A> On Thu, Jul 1, 2010=
 at 11:51 AM, alan=0A> bryan <alan.bryan@yahoo.com>=0A> wrote:=0A> >=0A> >=
=0A> > --- On Thu, 7/1/10, Garrett Cooper <yanefbsd@gmail.com>=0A> wrote:=
=0A> >=0A> >> From: Garrett Cooper <yanefbsd@gmail.com>=0A> >> Subject: Re:=
 NFS 75 second stall=0A> >> To: "alan bryan" <alan.bryan@yahoo.com>=0A> >> =
Cc: freebsd-stable@freebsd.org=0A> >> Date: Thursday, July 1, 2010, 11:13 A=
M=0A> >> On Thu, Jul 1, 2010 at 11:01 AM, alan=0A> >> bryan <alan.bryan@yah=
oo.com>=0A> >> wrote:=0A> >> > Setup:=0A> >> >=0A> >> > server - FreeBSD 8-=
stable from today.=A0 2 UFS=0A> dirs=0A> >> exported via NFS.=0A> >> > clie=
nt - FreeBSD 8.0-Release. =A0Running a=0A> test php=0A> >> script that copi=
es around various files to/from 2=0A> separate=0A> >> NFS mounts.=0A> >> >=
=0A> >> > Situation:=0A> >> >=0A> >> > script is started (forked to do 20=
=0A> simultaneous runs)=0A> >> and 20 1GB files are copied to the NFS dir w=
hich=0A> works=0A> >> fine.=A0 When it then switches to reading those=0A> f=
iles back=0A> >> and simultaneously writing to the other NFS mount=0A> I se=
e a=0A> >> hang of 75 seconds.=A0 If I do an "ls -l" on the=0A> NFS mount i=
t=0A> >> hangs too.=A0 After 75 seconds the client has=0A> reported:=0A> >>=
 >=0A> >> > nfs server 192.168.10.133:/usr/local/export1:=0A> not=0A> >> re=
sponding=0A> >> > nfs server 192.168.10.133:/usr/local/export1:=0A> is aliv=
e=0A> >> again=0A> >> > nfs server 192.168.10.133:/usr/local/export1:=0A> n=
ot=0A> >> responding=0A> >> > nfs server 192.168.10.133:/usr/local/export1:=
=0A> is alive=0A> >> again=0A> >> >=0A> >> > and then things start working =
again.=A0 The=0A> server was=0A> >> originally FreeBSD 8.0-Release also but=
 was=0A> upgraded to the=0A> >> latest stable to see if this issue could be=
=0A> avoided.=0A> >> >=0A> >> > # nfsstat -s -W -w 1=0A> >> > =A0GtAttr Loo=
kup Rdlink=A0=A0=A0Read=A0 Write=0A> Rename=0A> >> Access=A0 Rddir=0A> >> >=
 =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 222=A0 =A0=0A> 257=0A> >> =
=A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =
=A0 0=A0 =A0 178=A0 =A0=0A> 135=0A> >> =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> =
>> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0=A0=A085=A0=0A> =A0 127=
=0A> >> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> >> > =A0 =A0 =A0 0=A0 =A0 =
=A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0=0A> =A0 0=0A> >> =A0 =A0 0=A0 =A0 =
=A0 0=A0 =A0 =A0 0=0A> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =
=A0 0=A0 =A0=0A> =A0 0=0A> >> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> >> > =
=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0=0A> =A0 0=0A> >=
> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=
=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0=0A> =A0 0=0A> >> =A0 =A0 0=A0 =A0 =A0 0=
=A0 =A0 =A0 0=0A> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=
=A0 =A0=0A> =A0 0=0A> >> =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> >> >=0A> >=
> > ... for 75 rows of all zeros=0A> >> >=0A> >> > =A0 =A0 =A0 0=A0 =A0 =A0=
 0=A0 =A0 =A0 0=A0 =A0 272=A0 =A0=0A> 266=0A> >> =A0 0=A0 =A0 =A0 0=A0 =A0 =
=A0 0=0A> >> > =A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=A0 =A0 167=A0 =A0=0A=
> 165=0A> >> =A0 0=A0 =A0 =A0 0=A0 =A0 =A0 0=0A> >> >=0A> >> > I also tried=
 runs with 15 simultaneous=0A> processes and=0A> >> 25. =A015 processes gav=
e only about a 5 second=0A> stall but 25=0A> >> gave again the same 75 seco=
nd stall.=0A> >> >=0A> >> > Further, I tested with 2 mounts to the same=0A>=
 server but=0A> >> from ZFS filesytems with the exact same=0A> stall/timeou=
t=0A> >> periods. =A0So, it doesn't appear to matter what=0A> the=0A> >> un=
derlying filesystem is - it's something in NFS=0A> or=0A> >> networking cod=
e.=0A> >> >=0A> >> > Any ideas on what's going on here? =A0What's=0A> causi=
ng=0A> >> the complete stall period of zero NFS activity? =A0=0A> Any flaws=
=0A> >> with my testing methods?=0A> >> >=0A> >> > Thanks for any and all h=
elp/ideas.=0A> >>=0A> >> What network driver are you using? Have you tried=
=0A> >> tcpdumping the packets?=0A> >> -Garrett=0A> >>=0A> >=0A> > I'm usin=
g igb currently but have also used em. =A0I=0A> have not tried tcpdumping t=
he packets yet on this test.=0A> =A0Any suggestions on things to look out f=
or (I'm not that=0A> familiar with that whole process).=0A> >=0A> > Which b=
rings up another point - I'm using TCP=0A> connections for NFS, not UDP.=0A=
> =0A> =A0 =A0 Is the net.inet.tcp.tso sysctl enabled or=0A> not? What abou=
t rxcsum and txcsum?=0A> Thanks,=0A> -Garrett=0A> =0A=0AI haven't intention=
ally/explicitly set any of this so it's "default":=0A=0A# sysctl net.inet.t=
cp.tso=0Anet.inet.tcp.tso: 1=0A=0A=0Aigb0: flags=3D8843<UP,BROADCAST,RUNNIN=
G,SIMPLEX,MULTICAST> metric 0 mtu 1500=0A=09options=3D13b<RXCSUM,TXCSUM,VLA=
N_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>=0A=09ether 00:30:48:c3:26:94=0A=09inet=
 192.168.10.133 netmask 0xffffff00 broadcast 192.168.10.255=0A=09media: Eth=
ernet autoselect (1000baseT <full-duplex>)=0A=09status: active=0A=0AThanks,=
=0AAlan=0A=0A=0A=0A=0A      
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"