Re: contrib/jemalloc

On Apr 13, 2012, at 3:43 PM, David O'Brien wrote:
> On Thu, Apr 12, 2012 at 01:19:56PM -0700, Jason Evans wrote:
>> On Apr 12, 2012, at 11:41 AM, David O'Brien wrote:
> It looks like you could run './FREEBSD-upgrade extract' from a check =
out
> of ssh://svn.freebsd.org/base/vendor/jemalloc.
>=20
> I'm unclear what the './FREEBSD-upgrade rediff' step is for.  I'm =
having
> trouble following what "then regenerate diffs to update line offsets" =
is
> for.

The 'rediff' step regenerates the diff so that it precisely matches the =
differences as they apply to the code about to be imported.  If this =
step were omitted, patch fuzz would accumulate in FREEBSD-diffs.

> How do you handle the difference between FreeBSD having
> include/malloc_np.h and your GIT repo having
> jemalloc/include/jemalloc/jemalloc.h[.in]?
>=20
> Does './FREEBSD-upgrade extract' or './FREEBSD-upgrade rediff' patch
> 'include/malloc_np.h' and 'include/stdlib.h'?
> Especially with the '#define JEMALLOC_P(s) s' wrapping.
> Is that part of the './FREEBSD-upgrade rediff' step?

stdlib.h+malloc_np.h and jemalloc.h are different enough that they =
require separate maintenance.  Alas, not all programming can be =
automated; if interfaces change, manual intervention is required.

> contrib/jemalloc/FREEBSD-upgrade doesn't describe the "commit into
> FreeBSD subversion repo" step, that I can see.  So I am not sure what
> gets committed vs. what's in git://canonware.com/jemalloc.git.

Every difference that remains in the tree afterwards is to be committed. =
 That might require some 'svn add' and/or 'svn rm' commands prior to =
'svn commit'.

>> The only upside to vendor branch merges I can think of is
>> that if any jemalloc sources were manually modified between imports,
>> merging would fail rather than silently overwriting the changes.
>=20
> I feel that is an important check-and-balance.
>=20
> Not following our established procedure also means that the average
> developer(committer) and commercial consumer will have their =
expectations
> fails.  One expects to be able to find the stock vendor sources in
> ssh://svn.freebsd.org/base/vendor/ and to be able to find FreeBSD =
local
> changes to the sources thru 'svn diff' against
> ssh://obrien@svn.freebsd.org/base.

Regarding established procedure, I read through several of the =
contrib/*/FREEBSD-upgrade files when trying to understand established =
procedure, and the conclusion I came to is that many of the vendor =
imports would be painful for a new person to take over.  In contrast, =
contrib/jemalloc/FREEBSD-upgrade provides nearly complete automation, =
and documentation on how to deal with the manual parts of the process, =
should the need arise.

>> However, this presumes that changes aren't making it upstream.
>=20
> That is not an unreasonable presumption.  Unless contrib/jemalloc/
> is locked so that you are aware of every commit FreeBSD and
> git://canonware.com/jemalloc.git can diverge at times.

Today I updated the FREEBSD-upgrade script to take care of this =
possibility.

	=
http://people.freebsd.org/~jasone/patches/jemalloc_20120413a.patch

One problem that doesn't fit neatly into the standard vendor import =
procedure is that jemalloc.3 is a generated file.  It isn't feasible to =
keep FreeBSD-specific changes out of the vendor branch unless jemalloc's =
build system is imported, or manual changes are made directly to the =
generated file (yuck).  In contrast, the FREEBSD-upgrade script leaves =
all non-essential files out of the imports, resulting in a cleaner =
import.

Imagine we're looking at the svn history three years from now, and I've =
croaked, leaving the FreeBSD copy of jemalloc to drift where it will.  =
svn might contain something like the following sequence of relevant =
commits:

- (jasone) Import jemalloc 7ca0fdfb85b2a9fc7a112e158892c098e004385b.
- (jasone) Import jemalloc [some other git rev].
- (jasone) Import jemalloc [some other git rev].
- (jasone) Import jemalloc 3.0.0.
- (jasone) Import jemalloc 3.0.1.
- (obrien) Fix aligned_alloc() to stop snatching helpless children.
- (jasone) Import jemalloc 3.2.0.
- (obrien) Stop protecting children; they're not as helpless as they =
appear.
- (obrien) Import jemalloc 3.2.1.  May jasone RIP.
- (obrien) This allocator sucks.  Gut it.

What gets lost?  As near as I can tell, the svn history tells the whole =
story, and 'svn blame' even reflects FreeBSD-specific changes that =
weren't made as part of an import commit (and FREEBSD-diffs provides a =
full audit trail for those).

It's worth mentioning that the less work imports are, the more often I =
will be able to do them.  I've put several days of effort into =
streamlining this process so that FreeBSD's jemalloc won't languish.  =
I've been working more or less full time on jemalloc for the past six =
weeks (thanks Facebook!), but I really have to wrap this up and get back =
to other projects.

Jason=
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"