Re: samba+zfs

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--2338054437-237842630-1320804989=:93923
Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8BIT

On Tue, 8 Nov 2011, Kurt Touet wrote:

> On Tue, Nov 8, 2011 at 3:32 AM, Dan The Man <dan@sunsaturn.com> wrote:
>>
>>
>> Sorry I meant it was running fine on beta3 and 8.2 stable, and NOT RC1:
>> asterisk:~# uname -a
>> FreeBSD asterisk.sunsaturn.com 9.0-RC1 FreeBSD 9.0-RC1 #0: Mon Oct 31
>> 19:46:53 CDT 2011 droot@asterisk.sunsaturn.com:/usr/obj/usr/src/sys/MYKERNEL
>> 乸md64
>> asterisk:~#
>>
>>
>>
>> Dan.
>>
>>
>> --
>> Dan The Man
>> CTO/ Senior System Administrator
>> Websites, Domains and Everything else
>> http://www.SunSaturn.com
>> Email: Dan@SunSaturn.com
>>
>> On Tue, 8 Nov 2011, Dan The Man wrote:
>>
>>>
>>> Ok here is some specs: this been running fine on 8.2 stable and i was sure
>>> it was running fine on RC1 as well. I did some testing against samba 34 35
>>> and 36 in the ports collection all with the same slow write problems.
>>>
>>> I did further testing mounting drive in question with NFS and it did not
>>> suffer the same problem, so it seems just samba related here, where samba
>>> would actually outperform my NFS mount before, now its taking 10x as long
>>> to write anything.
>>>
>>> This is really most simplistic setup I have, all I want to do is map a
>>> network drive at the house read/write so my laptop, desktop etc all have
>>> access to it. I have played with all the smb.conf options, and can't seem
>>> to find where the issue is, further research suggests others are
>>> experiencing same problems with beta3 from following forum post:
>>>
>>> http://forums.freebsd.org/showthread.php?t=27300
>>>
>>> Hardware this is running on: I beleive a 4 year old amd chip and board,
>>> with 2 gigs of ram, this is a home PC that serves as a NAS, it has 1 SSD as
>>> UFS boot OS filesystem, and uses ZFS in raid0 with 3 3terrabyte cheap
>>> hitachi drives for my storage space,which is mirrored nightly with rsync
>>> with another duplicate machine(cause I know someone is going to say why not
>>> use raid5-raidz)
>>>
>>> Network specs: machine currently has dedicated IPV4 and gif0 tunneled IPV6
>>> addresses to he.net.
>>>
>>> I've played with nearly every option in smb.conf disabling, enabling etc
>>> and can't seem to find the issue: here are my current config file settings
>>> on machine that could apply to samba:
>>>
>>> asterisk:~# cat /boot/loader.conf
>>> autoboot_delay="5"
>>> accf_data_load="YES"
>>> aio_load="YES"
>>> zfs_load="YES"
>>> kern.maxbcache=64M
>>> kern.ipc.maxpipekva=4M
>>>
>>> vfs.zfs.prefetch_disable=1
>>> vm.kmem_size="1844M"
>>> vfs.zfs.arc_min="1024M"
>>> vfs.zfs.arc_max="1536M"
>>> vfs.zfs.vdev.min_pending=2
>>> vfs.zfs.vdev.max_pending=8
>>> vfs.zfs.txg.timeout=5
>>> vfs.zfs.zil_disable="1"
>>> ahci_load="YES"
>>> asterisk:~#
>>>
>>> asterisk:~# cat /usr/local/etc/smb.conf
>>> # Global parameters
>>> [global]
>>> ꀠꀠꀠworkgroup = HOME
>>> ꀠꀠꀠnetbios name = ASTERISK
>>> ꀠꀠꀠserver string = "Primary backups"
>>> ꀠꀠꀠinterfaces = sk0
>>> ꀠꀠꀠ#smb ports = 139
>>> ꀠꀠꀠ#security = USER
>>> ꀠꀠꀠsecurity = SHARE
>>> ꀠꀠꀠencrypt passwords = Yes
>>> ꀠꀠꀠ#socket options = TCP_NODELAY SO_RCVBUF=65536 SO_SNDBUF=65536
>>> ꀠꀠꀠdomain master = no
>>> ꀠꀠꀠwins support = yes
>>> ꀠꀠꀠguest account = root
>>> ꀠꀠꀠsocket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY
>>> ꀠꀠꀠuse sendfile = no
>>> ꀠꀠꀠlevel2 oplocks = True
>>> ꀠꀠꀠread raw = no
>>> ꀠꀠꀠwrite cache size = 262144
>>> ꀠꀠꀠmin receivefile size = 16384
>>> ꀠꀠꀠaio read size = 16384
>>> ꀠꀠꀠaio write size = 16384
>>> ꀠꀠꀠaio write behind = yes
>>> ꀠꀠꀠdns proxy = no
>>> ꀠꀠꀠmax log size = 50
>>> ꀠꀠꀠ#log file = /dev/null
>>> ꀠꀠꀠlog file = /var/log/samba.log
>>> ꀠꀠꀠdebug level = 1
>>> ꀠꀠꀠsyslog = 0
>>>
>>> [data]
>>> ꀠꀠꀠcomment = "Primary backups"
>>> ꀠꀠꀠpath = /data/public
>>> ꀠꀠꀠread only = No
>>> ꀠꀠꀠguest ok = Yes
>>> ꀠꀠꀠvalid users = root
>>> asterisk:~# asterisk:~# cat /etc/sysctl.conf
>>> # $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1 2009/08/03 08:13:06 kensmith
>>> Exp $
>>> #
>>> # 糍his file is read when going to multi-user and its contents piped thru
>>> # 攞`sysctl'' to adjust kernel values. 攞`man 5 sysctl.conf'' for details.
>>> #
>>>
>>> # Uncomment this to prevent users from seeing information about processes
>>> that
>>> # are being run under another UID.
>>> #security.bsd.see_other_uids=0
>>>
>>> #raise file descriptors on the system
>>> kern.maxfiles=204916
>>> kern.maxfilesperproc=204916
>>>
>>> #raise sockets we can accept
>>> kern.ipc.somaxconn=32768
>>>
>>> #http://www-didc.lbl.gov/TCP-tuning/FreeBSD.html
>>> kern.ipc.maxsockbuf=16777216
>>> net.inet.tcp.rfc1323=1
>>> net.inet.tcp.sendbuf_max=16777216
>>> net.inet.tcp.recvbuf_max=16777216
>>> net.inet.tcp.sendspace=65536
>>> net.inet.tcp.recvspace=131072
>>>
>>> #NGINX webserver http://wiki.nginx.org/FreeBSDOptimizations
>>> net.inet.icmp.icmplim=500
>>> kern.ipc.nmbjumbop=192000
>>> kern.ipc.nmbclusters=229376
>>> kern.ipc.maxsockets=204800
>>> net.inet.tcp.maxtcptw=163840
>>> #also add following to /boot/loader.conf
>>> #vm.kmem_size=1844M
>>> #kern.maxbcache=64M
>>> #kern.ipc.maxpipekva=4M
>>>
>>> #default setting of net.inet.ip.portrange.first is to low, causing us
>>> problems with bind
>>> net.inet.ip.portrange.last=65535
>>> net.inet.ip.portrange.first=1024
>>>
>>> #DOS protection
>>> net.inet.tcp.msl=7500
>>> net.inet.tcp.blackhole=2
>>> net.inet.udp.blackhole=1
>>> net.inet.icmp.icmplim=50
>>> net.inet.ip.accept_sourceroute=0
>>> net.inet.ip.sourceroute=0
>>>
>>> #some stuff for samba
>>> kern.ipc.nmbclusters=32768
>>> kern.maxvnodes=800000
>>> net.inet.tcp.delayed_ack=0
>>> net.inet.tcp.inflight.enable=0
>>> net.inet.tcp.path_mtu_discovery=0
>>> net.inet.tcp.recvbuf_auto=1
>>> net.inet.tcp.recvbuf_inc=524288
>>> net.inet.tcp.sendbuf_auto=1
>>> net.inet.tcp.sendbuf_inc=524288
>>> net.inet.udp.maxdgram=57344
>>> net.inet.udp.recvspace=65535
>>> net.local.stream.recvspace=65535
>>> net.local.stream.sendspace=65535
>>> net.inet.tcp.mssdflt=1460
>>>
>>> #IPSEC
>>> net.inet.ip.forwarding=1
>>> net.inet6.ip6.forwarding=1
>>> kern.module_path=/boot/kernel;/boot/modules;/usr/local/modules
>>>
>>>
>>> #NFS--not concerned with data integrity when playing mostly already stored
>>> movies
>>> vfs.nfsrv.async=1
>>>
>>> #JAIL
>>> #i like to use ping etc inside jail!
>>> security.jail.allow_raw_sockets=1
>>> asterisk:~#
>>>
>>>
>>> Here are logs of me trying to mux a DTS mkv file from samba.log on debug
>>> level 10, I get the following over and over again:
>>>
>>> [2011/11/08 03:24:00.067974, 10] ../lib/util/util.c:415(dump_data)
>>> [2011/11/08 03:24:00.067981, ꀳ] smbd/process.c:1466(switch_message)
>>> 澵witch message SMBwriteX (pid 64308) conn 0x805008450
>>> [2011/11/08 03:24:00.067990, ꀴ] smbd/uid.c:345(change_to_user)
>>> 嚒kipping user change - already user
>>> [2011/11/08 03:24:00.068001, 10]
>>> locking/locking.c:120(strict_lock_default)
>>> 湶s_locked: optimisation - exclusive oplock on file
>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv
>>> [2011/11/08 03:24:00.068010, 10]
>>> locking/locking.c:162(strict_lock_default)
>>> 澵trict_lock_default: flavour = WINDOWS_LOCK brl start=83431665 len=65536
>>> unlocked for fnum 49966 file
>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv
>>> [2011/11/08 03:24:00.068021, 10] lib/recvfile.c:65(default_sys_recvfile)
>>> 氽efault_sys_recvfile: from = 33, to = 39, offset=83431665, count = 65536
>>> [2011/11/08 03:24:00.068995, 10] smbd/fileio.c:143(real_write_file)
>>> 澑eal_write_file (torrent_downloads_finished/Point.Break.1991.720p
>>> (1).mkv): pos = 83431665, size = 65536, returned 65536
>>> [2011/11/08 03:24:00.069013, ꀳ] smbd/reply.c:4639(reply_write_and_X)
>>> 烀riteX fnum=49966 num=65536 wrote=65536
>>> [2011/11/08 03:24:00.069038, 10]
>>> lib/util_sock.c:516(read_smb_length_return_keepalive)
>>> 淸ot smb length of 65600
>>> [2011/11/08 03:24:00.069052, 10] smbd/reply.c:4459(is_valid_writeX_buffer)
>>> 湶s_valid_writeX_buffer: true len = 65600, doff = 64, numtowrite = 65536
>>> [2011/11/08 03:24:00.069062, ꀶ] smbd/process.c:1659(process_smb)
>>> 淸ot message type 0x0 of len 0x3f
>>> [2011/11/08 03:24:00.069072, ꀳ] smbd/process.c:1661(process_smb)
>>> 糍ransaction 15398 of length 67 (65536 toread)
>>> [2011/11/08 03:24:00.069081, ꀵ] lib/util.c:332(show_msg)
>>> [2011/11/08 03:24:00.069087, ꀵ] lib/util.c:342(show_msg)
>>> 澵ize=63
>>> 澵mb_com=0x2f
>>> 澵mb_rcls=0
>>> 澵mb_reh=0
>>> 澵mb_err=0
>>> 澵mb_flg=24
>>> 澵mb_flg2=51207
>>> 澵mb_tid=1
>>> 澵mb_pid=65279
>>> 澵mb_uid=0
>>> 澵mb_mid=36032
>>> 澵mt_wct=14
>>> 澵mb_vwv[ 0]= ꀲ55 (0xFF)
>>> 澵mb_vwv[ 1]=57054 (0xDEDE)
>>> 澵mb_vwv[ 2]=49966 (0xC32E)
>>> 澵mb_vwv[ 3]= 4337 (0x10F1)
>>> 澵mb_vwv[ 4]= 1274 (0x4FA)
>>> 澵mb_vwv[ 5]=65535 (0xFFFF)
>>> 澵mb_vwv[ 6]=65535 (0xFFFF)
>>> 澵mb_vwv[ 7]= ꀠꀰ (0x0)
>>> 澵mb_vwv[ 8]= ꀠꀰ (0x0)
>>> 澵mb_vwv[ 9]= ꀠꀱ (0x1)
>>> 澵mb_vwv[10]= ꀠꀰ (0x0)
>>> 澵mb_vwv[11]= ꀠ64 (0x40)
>>> 澵mb_vwv[12]= ꀠꀰ (0x0)
>>> 澵mb_vwv[13]= ꀠꀰ (0x0)
>>> 澵mb_bcc=0
>>> [2011/11/08 03:24:00.069163, 10] ../lib/util/util.c:415(dump_data)
>>> [2011/11/08 03:24:00.069170, ꀳ] smbd/process.c:1466(switch_message)
>>> 澵witch message SMBwriteX (pid 64308) conn 0x805008450
>>> [2011/11/08 03:24:00.069179, ꀴ] smbd/uid.c:345(change_to_user)
>>> 嚒kipping user change - already user
>>> [2011/11/08 03:24:00.069188, 10]
>>> locking/locking.c:120(strict_lock_default)
>>> 湶s_locked: optimisation - exclusive oplock on file
>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv
>>> [2011/11/08 03:24:00.069197, 10]
>>> locking/locking.c:162(strict_lock_default)
>>> 澵trict_lock_default: flavour = WINDOWS_LOCK brl start=83497201 len=65536
>>> unlocked for fnum 49966 file
>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv
>>> [2011/11/08 03:24:00.069221, 10] lib/recvfile.c:65(default_sys_recvfile)
>>> 氽efault_sys_recvfile: from = 33, to = 39, offset=83497201, count = 65536
>>> [2011/11/08 03:24:00.069987, 10] smbd/fileio.c:143(real_write_file)
>>> 澑eal_write_file (torrent_downloads_finished/Point.Break.1991.720p
>>> (1).mkv): pos = 83497201, size = 65536, returned 65536
>>> [2011/11/08 03:24:00.070004, ꀳ] smbd/reply.c:4639(reply_write_and_X)
>>> 烀riteX fnum=49966 num=65536 wrote=65536
>>> [2011/11/08 03:24:00.070030, 10]
>>> lib/util_sock.c:516(read_smb_length_return_keepalive)
>>> 淸ot smb length of 65600
>>> [2011/11/08 03:24:00.070044, 10] smbd/reply.c:4459(is_valid_writeX_buffer)
>>> 湶s_valid_writeX_buffer: true len = 65600, doff = 64, numtowrite = 65536
>>> [2011/11/08 03:24:00.070053, ꀶ] smbd/process.c:1659(process_smb)
>>> 淸ot message type 0x0 of len 0x3f
>>> [2011/11/08 03:24:00.070063, ꀳ] smbd/process.c:1661(process_smb)
>>> 糍ransaction 15399 of length 67 (65536 toread)
>>> [2011/11/08 03:24:00.070072, ꀵ] lib/util.c:332(show_msg)
>>> [2011/11/08 03:24:00.070077, ꀵ] lib/util.c:342(show_msg)
>>> 澵ize=63
>>> 澵mb_com=0x2f
>>> 澵mb_rcls=0
>>> 澵mb_reh=0
>>> 澵mb_err=0
>>> 澵mb_flg=24
>>> 澵mb_flg2=51207
>>> 澵mb_tid=1
>>> 澵mb_pid=65279
>>> 澵mb_uid=0
>>> 澵mb_mid=36102
>>> 澵mt_wct=14
>>> 澵mb_vwv[ 0]= ꀲ55 (0xFF)
>>> 澵mb_vwv[ 1]=57054 (0xDEDE)
>>> 澵mb_vwv[ 2]=49966 (0xC32E)
>>> 澵mb_vwv[ 3]= 4337 (0x10F1)
>>> 澵mb_vwv[ 4]= 1275 (0x4FB)
>>> 澵mb_vwv[ 5]=65535 (0xFFFF)
>>> 澵mb_vwv[ 6]=65535 (0xFFFF)
>>> 澵mb_vwv[ 7]= ꀠꀰ (0x0)
>>> 澵mb_vwv[ 8]= ꀠꀰ (0x0)
>>> 澵mb_vwv[ 9]= ꀠꀱ (0x1)
>>> 澵mb_vwv[10]= ꀠꀰ (0x0)
>>> 澵mb_vwv[11]= ꀠ64 (0x40)
>>> 澵mb_vwv[12]= ꀠꀰ (0x0)
>>> 澵mb_vwv[13]= ꀠꀰ (0x0)
>>> 澵mb_bcc=0
>>>
>>>
>>> Hopefully maybe someone can shine some light on this....
>>>
>>>
>>> Dan.
>>>
>>>
>>> --
>>> Dan The Man
>>> CTO/ Senior System Administrator
>>> Websites, Domains and Everything else
>>> http://www.SunSaturn.com
>>> Email: Dan@SunSaturn.com
>>>
>>> On Fri, 28 Oct 2011, Garrett Cooper wrote:
>>>
>>>> On Thu, Oct 27, 2011 at 6:42 PM, Dan <dan@sunsaturn.com> wrote:
>>>>>
>>>>>
>>>>> Updated from 9.0 beta3 to RC1 and using mkvmerge over samba/zfs
>>>>> its taking over an hour to just mux in things like DTS english, where it
>>>>> was
>>>>> 15 minutes on beta3.
>>>>
>>>> Hi Dan,
>>>> - Can you do more deterministic / scientific benchmarks?
>>>> - Did you upgrade Samba?
>>>> - What is your system's operating hardware profile?
>>>> Thanks!
>>>> -Garrett
>>>>
>
> I had noticed a similar problem with doing large writes from my
> windows workstation to my freebsd ZFS array.  Previously I had been
> able to write at around 30MB/s (limited by a slow SATA controller),
> and those speeds had dropped to 3-5MB/s with long pauses between
> writes to the array (monitoring with zpool iostat).   However, I just
> updated to stable/9 r227357 and the issue seems to have gone away; I'm
> back up to 30MB/s writes.
>
> Hope you find the same sort of thing,
> -kurt
>

Yep your right, on RC2 my samba ZFS write speeds are back.
Thanks for heads up Kurt.

Dan.

--
Dan The Man
CTO/ Senior System Administrator
Websites, Domains and Everything else
http://www.SunSaturn.com
Email: Dan@SunSaturn.com

--2338054437-237842630-1320804989=:93923
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
--2338054437-237842630-1320804989=:93923--