From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on sa.local.altlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM autolearn=ham autolearn_force=no version=3.4.1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=P5KgCHlDJ7C+XFJGBTsCd3I0xWk8yxHFpQM7AT/5AbM=; b=Nr64aBOyPXG+AokV7JLzZj8XPSxzF9Mm2JfxNidwMF9HPVxLFDutXu8Waw24L0eja4 jw+5UGCZq1tIjKlOgiPVinJneOSOssK8p4PCbvsAAuziON5qUXnqyb58PTkilpZHI/gX 1lsUkALlyiOKVWLPFDBy0pxLyiJA74fbqC5PwQAOvfKgsfntXD8PnFrJJ7L/flf4TguJ IvronWcmfodoyqGJlmYR+juY3XKqOltJ7HFbOwb6A334d6zIe5gj8Mlvrjjldde0lhwv Oz43RkRdKljjVY6/Ztj9LEPQpA3t3/GQmkMleFKolprRZAwsHeVyPiQSxk5kgOKEcDyR EHog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=P5KgCHlDJ7C+XFJGBTsCd3I0xWk8yxHFpQM7AT/5AbM=; b=Sy0CYVkaQciUWBTL4AFR5KNtbRGsyWxTi0ekum4oV+nyZqfl6LLU7tD02QRfQJsEQF o5cYE7FUGFldOWVNKUCuJdklvDPBs1L0A+tJyqvVrYldPt3F86+LJ2eJGMF7aR0CqKFh kMWudHj+FZUip5EOMMk3uNDqhSvVumUV/R4dbz8DuuzdAuPfcnQdfDXzj49AaV6Kaphu 4A448KPNdkOOs7BzOnPzTU6646qCr7yUGxBoJqIY6S102SU8U4uSaziKTCOhagUfW+In 0uZb03i3BqhPeTV1UKbrOx4HTRg3fDe2TXAF5dDsB42xRvLDKydlHXQleSMzu96R8f2d na/Q== X-Gm-Message-State: AGi0Pua8oa99NtlsbrmjktLrCXcyLT2BlUWCDMWEWHwuB7vHmjs176sW WbGg2dTw9WVt7Cyd+JrW+LbBoJzARW88j+2FU2dxtOLB X-Google-Smtp-Source: APiQypKEiB/jDpLIcCQjDNdi9Z78KbA0DcPchAgXBFidUf5B1LXBn7rEeTifsz0VlcueBYG611FlfpSyXNuf0rkPvKM= X-Received: by 2002:a92:c7a9:: with SMTP id f9mr20560135ilk.0.1587965924715; Sun, 26 Apr 2020 22:38:44 -0700 (PDT) MIME-Version: 1.0 References: <20200410231044.1436970-1-vseleznv@altlinux.org> <20200410231044.1436970-3-vseleznv@altlinux.org> <20200414164244.GC618226@portlab> <20200417135419.GA24862@altlinux.org> <20200423192128.GA3823012@portlab> In-Reply-To: <20200423192128.GA3823012@portlab> From: Alexey Tourbin Date: Mon, 27 Apr 2020 08:38:33 +0300 Message-ID: To: ALT Linux Team development discussions Content-Type: text/plain; charset="UTF-8" Subject: Re: [devel] stopping a cascade of rebuilds X-BeenThere: devel@lists.altlinux.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: ALT Linux Team development discussions List-Id: ALT Linux Team development discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Apr 2020 05:38:48 -0000 Archived-At: List-Archive: List-Post: On Thu, Apr 23, 2020 at 10:21 PM Vladimir D. Seleznev wrote: > > So for src.rpm packages, it's a solved problem. For binary packages, > > the identity should specifically exclude disttag. It will no longer > > satisfy the definition of ID for rpm (substitution will break for > > subpackages with strict dependencies). Therefore for binary packages, > > we need to track tuples. > > Why should we track them? If we rebuild a package, we should check > whether identity of its binary packages had changed. If it had not, we > shouldn't replace its binary packages by rebuilt packages. That simple. Because in the identity-addressable storage, we can have a few packages with the same ID but different disttags, as in the example below. The fact that foo-data-ID1 hasn't changed doesn't mean you can immediately grab foo-data-filehash1. > > $ cat id2f/libfoo > > > > > > > > $ cat id2f/foo-data > > > > > The things get more complicated in case of "copying" packages. In that > case this schema could help. But we need also track buildtime. Just to > prevent package replacing with earlier build. So, it is triple now. Hmm, haven't thought of buildtime. We've got a package in the repo, a freshly rebuilt package with the same EVR, and an identical package (or a few identical packages) in the identity-addressable storage. We need to prevent the outcome in which the buildtime of the package in the repo goes down. That's what you mean by "prevent package replacing with earlier build", right? (Otherwise replacing with an earlier build is not a problem.) Such an outcome is unlikely to occur in practice, but not impossible. Three parties are now involved. You cannot simply ask, "can I overwrite build/100 with packages from identity-addressable storage?" The third piece of information is what you already have in the repo. > > It may even make sense to group the mappings by src.rpm name instead > > of package name. At first it seems less intuitive, but in return it > > can give you a consistent view similar to MVCC snapshot. Of course, > > these files should be updated atomically, with rename(2). To check a > > set subpackages, you first need to copy the file to a local dir. This > > should rule out the case in which some subpackages have been added to > > the file and some not. > > I don't get this idea, please expand it. These are implementation details (that are not all that important unless we agree on the data model). How do you access the identity-addressable storage concurrently? The worst case is that id2f/libfoo is written and read simultaneously, so you read a half-written file. To avoid this, files should be replaced atomically: $ mv local/libfoo id2f/.tmp$$.libfoo && mv id2f/.tmp$$.libfoo id2f/libfoo You are now guaranteed that you open either old id2f/libfoo or new id2f/libfoo, but not something in between. But this is not enough, because there is a higher-order inconsistency: you don't want to process new id2f/libfoo with old id2f/foo-data (when id2f/foo-data is just being replaced). To get a "consistent view" across subpackages, you can combine id2f/libfoo and id2f/foo-data into a single file. To process subpackages, you first need to make a local copy of that file. Because with command-line tools, you'd want to open the file a few times, so you can't do it directly on id2f/libfoo. This is an alternative to the "single writer, multiple readers" model. In this model, the reader first has to obtain a shared lock on id2f. In what I describe, the reader can proceed without any locking.