From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on sa.int.altlinux.org X-Spam-Level: X-Spam-Status: No, score=-1.8 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.2.5 Date: Fri, 11 Dec 2009 21:56:02 +0200 From: Michael Shigorin To: ALT Linux Team development discussions Message-ID: <20091211195602.GV13584@osdn.org.ua> Mail-Followup-To: ALT Linux Team development discussions References: <4B223383.5050703@mmedia2.kemsu.ru> <20091211182800.GC9864@altlinux.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20091211182800.GC9864@altlinux.org> User-Agent: Mutt/1.4.2.1i Subject: [devel] parallel builds, again (was: [Fwd: [#16528] COMPLETE (try 23)]) X-BeenThere: devel@lists.altlinux.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: ALT Linux Team development discussions List-Id: ALT Linux Team development discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Dec 2009 19:56:20 -0000 Archived-At: List-Archive: List-Post: On Fri, Dec 11, 2009 at 09:28:00PM +0300, Alexey Tourbin wrote: > > Does at@ or ldv@ have any ideas how to further improve > > builder performance? There never was so stressfull load on > > infrastructure and hope never be in near future, but will we > > wait so long time if it happens again? > First off (perhaps in case you didn't notice), we're doing > blazing fast. I think it's just as "fast" as when one stands for an hour trying to book tickets for the train departing roughly in an hour, while the operator can do nothing but wait until the lock somewhere is released. Single-threaded build is getting us nowhere when CPU clocks are effectively frozen at <= 3GHz while cores are plenty; ARM case is even more sensitive to single system performance being a bottleneck (as discussed on devel-ports@). 2 ldv: what do you think of a week with parallel builds turned on and taking a log analysis vacation? :) (BTW having "main" packages built single-threaded and "contrib" packages built multi-threaded might help too -- if we had those) > As a matter of fact, a task with 1000 of packages, running for > the second time, can complete within only hours. If you still > think this isn't fast enough, consider that 1000 packages > actually are 1/n-th part of the whole repo. Well you have done great job at shaving seconds off one CPU core but there are at least a few more sitting around idle, no? If you still think it's fast enough, consider that it could be 2x or 4x faster. > > Why packages without direct python dependencies (runtime or > > buildtime) was blocked by python task? > Because what we do is "purely functional" builds, with no > exceptions. ...which is pretty expensive as already discussed -- and for quite a few cases could be replaced by opportunistic approach using cached previous package build results in hope that things won't shift much, and falling back to "strict" model in case they actually did. -- ---- WBR, Michael Shigorin ------ Linux.Kiev http://www.linux.kiev.ua/