From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Fri, 24 Jun 2005 19:17:35 +0300 From: Michael Shigorin To: ALT Linux Sisyphus mailing list Message-ID: <20050624161735.GN24580@osdn.org.ua> Mail-Followup-To: ALT Linux Sisyphus mailing list References: <200506231710.48739.zlatoust@mit.edu> <200506232046.24603.iadzhubey@rics.bwh.harvard.edu> <20050624023322.GI5554@solemn.turbinal.org> <200506232256.59626.iadzhubey@rics.bwh.harvard.edu> <20050624134545.GV24580@osdn.org.ua> <20050624145204.GB23459@basalt.office.altlinux.org> <20050624150731.GF24580@osdn.org.ua> <20050624151852.GC23459@basalt.office.altlinux.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="NOsHN2hBCdtiWn6P" Content-Disposition: inline In-Reply-To: <20050624151852.GC23459@basalt.office.altlinux.org> User-Agent: Mutt/1.4.2.1i Cc: Subject: [sisyphus] Re: =?koi8-r?b?09TBwsnM2M7Zyi4uLiDOxdPUwcLJzNjO2couLi4=?= X-BeenThere: sisyphus@altlinux.ru X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ALT Linux Sisyphus discussion list List-Id: ALT Linux Sisyphus discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Jun 2005 16:17:40 -0000 Archived-At: List-Archive: --NOsHN2hBCdtiWn6P Content-Type: multipart/mixed; boundary="TVUaZw7bVQbpqW1Q" Content-Disposition: inline --TVUaZw7bVQbpqW1Q Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 24, 2005 at 07:18:52PM +0400, Dmitry V. Levin wrote: > > > > =E9=D7=C1=CE, =C1 =CE=C5 =C8=CF=D4=C9=D4=C5 =D0=CF=D4=C9=C8=CF=CE= =D8=CB=D5 (=CD=CF=D6=C5=D4, =D3 3.0) =D3=DF=C5=DA=D6=C1=D4=D8 =D3=CF =D7=D3= =C5=CD > > > > =DC=D4=C9=CD =C8=CF=DA=D1=CA=D3=D4=D7=CF=CD =D3 =D3=C9=DA=C9=C6=C1 = =CE=C1 =DE=D4=CF-=D4=CF =C2=CF=CC=C5=C5 =D0=D2=C5=C4=D3=CB=C1=DA=D5=C5=CD= =CF=C5 =D0=CC=C0=D3 > > > > =C2=DC=CB=D0=CF=D2=D4=D9 =D0=CF =D7=CB=D5=D3=D5 [=C9 =D3 =F7=C1=DB= =C9=CD =D5=DE=C1=D3=D4=C9=C5=CD]? > > > =E4=D5=CD=C1=C0, =DE=D4=CF =DC=D4=CF =CF=DE=C5=CE=D8 =CE=C5=DA=C4=D2= =C1=D7=CF=C5 =D0=D2=C5=C4=CC=CF=D6=C5=CE=C9=C5. > > =F0=CF=D1=D3=CE=C9? > =F4=D9 =D0=D2=C5=C4=CC=C1=C7=C1=C5=DB=D8 =DE=C5=CC=CF=D7=C5=CB=D5, =CB=CF= =D4=CF=D2=D9=CA =CD=CF=D6=C5=D4 =D0=CF=DA=D7=CF=CC=C9=D4=D8 =D3=C5=C2=C5 > =D4=C5=D3=D4=C9=D2=CF=D7=C1=D4=D8 =CE=C5 =D3=C1=CD=D9=C5 =DB=C9=D2=CF=CB= =CF =C9=D3=D0=CF=CC=D8=DA=D5=C5=CD=D9=C5 =CB=CF=CD=D0=CF=CE=C5=CE=D4=D9 =F3= =C9=DA=C9=C6=C1 =C9 > =D4=C5=CD =D3=C1=CD=D9=CD =C4=C5=CC=C1=D4=D8 =C9=C8 =CC=D5=DE=DB=C5, =D0= =D2=C5=CB=D2=C1=D4=C9=D4=D8 =D4=C5=D3=D4=C9=D2=CF=D7=C1=CE=C9=C5. =F1 =D0=D2=C5=C4=CC=C1=C7=C1=C0 =D0=CF=C4=D5=CD=C1=D4=D8. =F0=CF=C4=D5=CD= =C1=D7 =D3=C1=CD, =DE=D4=CF =C9=CD=C5=CE=CE=CF =D0=D2=C5=C4=CC=C1=C7=C1=C0. > =E1 =D7=C5=C4=D8 =DC=D4=CF =CE=C1=CE=C5=D3=A3=D4 =C2=CF=CC=D8=DB=C9=CA = =D5=DD=C5=D2=C2 =F3=C9=DA=C9=C6=D5, =DE=C5=CD =D7=D3=C5 =D2=D5=C7=C1=C0=DD= =C9=C5 ALT > =D7=CD=C5=D3=D4=C5 =D7=DA=D1=D4=D9=C5 =D7 =CB=C1=DE=C5=D3=D4=D7=C5 =D0=CF= =CB=C1=DA=C1=D4=C5=CC=D1 =DC=CB=D3=D0=CF=CE=C5=CE=D4=D9. =E5=D3=CC=C9 =DE=C5=CC=CF=D7=C5=CB =D7 =CF=DE=C5=D2=C5=C4=CE=CF=CA =D7=DA= =D2=D9=D7 =D0=D3=C9=C8=C1=CE=A3=D4 =C9 =D5=CA=C4=A3=D4 =C9=DA =D3=CF=CF=C2= =DD=C5=D3=D4=D7=C1, =C2=D5=C4=C5=D4 =C8=D5=D6=C5. =EB =D3=CF=D6=C1=CC=C5=CE=C9=C0, =D4=C1=CB = =C2=D9=CC=CF =C9 =CE=C5 =D2=C1=DA (=C9 =CE=C5 =D3 =D3=C1=CD=D9=CD=C9 =CE=C1=C9=D7=CE=D9=CD=C9 =CC=C0=C4=D8=CD=C9). =EB=C1=CB =D4=D9 =CD=CF=D6=C5=DB=D8 =D0=CF=CD=CE=C9=D4=D8, =D7 =D4=C5=CD=C5= "=D3=D4=C1=C2=C9=CC=D8=CE=D9=CA =D3=C9=DA=C9=C6" =D1 =D7=D3=C5=CD=C9 =CB=CF=CE=C5=DE=CE=CF=D3=D4=D1=CD=C9 =C4=C5=D2=D6=C1=CC=D3=D1 =DA=C1 =D4=C5= =DA=C9=D3 "=C5=D3=CC=C9 =CD=CF=D6=CE=CF =CE=C5 =D2=C1=DA=CC=C1=CD=D9=D7=C1= =D4=D8 =C9=CC=C9=20 =CC=CF=CB=C1=CC=C9=DA=CF=D7=C1=D4=D8 =D0=D2=CF=C2=CC=C5=CD=D9 -- =C4=C1=D7= =C1=CA=D4=C5 =D0=CF=D3=D4=C1=D2=C1=C5=CD=D3=D1 =C4=C5=CC=C1=D4=D8 =D4=C1=CB= ". =E2=C9=DB=D8 =CD=CE=C5 =CB=C1=CB =D2=C1=DA=D2=C1=C2=CF=D4=DE=C9=CB=D5 =CE= =C1 =C2=C1=DA=C5 Sisyphus, =C1 =CE=C5 =C4=C9=D3=D4=D2=C9=C2=D5=D4=C9=D7=CF= =D7, *=CB=D2=C9=D4=C9=DE=CE=CF* =CB=C1=DE=C5=D3=D4=D7=CF =DE=C1=D3=D4=C9 =C5=C7= =CF =D0=C1=CB=C5=D4=CE=CF=CA =C2=C1=DA=D9. =EE=CF =D2=C5=CB=CF=CD=C5=CE=C4=C1=C3=C9=C9 =C4=C1=C0=D4=D3=D1 =CE=C5 =C9= =DA =DC=C7=CF=C9=D3=D4=C9=DE=C5=D3=CB=C9=C8 =D0=CF=C2=D5=D6=C4=C5=CE=C9=CA = --=20 =D1 =D3=DE=C9=D4=C1=C0, =DE=D4=CF =C7=CF=D2=C1=DA=C4=CF =C2=CF=CC=C5=C5 =D5= =CD=C5=D3=D4=CE=D9=CD =D7=C1=D2=C9=C1=CE=D4=CF=CD =D4=C5=D3=D4=C9=D2=CF=D7= =C1=CE=C9=D1 Sisyphus =D1=D7=CC=D1=C5=D4=D3=D1 =CE=C1=CC=C9=DE=C9=C5 =D5 =D4=C5=D3=D4=C9= =D2=D5=C0=DD=C5=C7=CF: - =D3=C9=D3=D4=C5=CD=D9 =D3=CE=C1=D0=DB=CF=D4=CF=D7 (=D3=D7=CF=CA =CB=D2=C9= =D7=CF=CA =D3=CB=D2=C9=D0=D4 =C9 =C5=DD=A3 =D0=C1=D2=D5 =D0=D2=C9=CC=C1=C7= =C1=C0); - =D4=C5=D3=D4=CF=D7=D9=C8 =D3=C9=D3=D4=C5=CD (=D7=CF=DA=CD=CF=D6=CE=CF, = =CE=C1=C8=CF=C4=D1=DD=C9=C8=D3=D1 =D7 =DC=CB=D3=D0=CC=D5=C1=D4=C1=C3=C9=C9,= =CE=CF =CC=C5=C7=CB=CF =D7=D9=D7=CF=C4=C9=CD=D9=C8 =C9=DA =CE=C5=A3 =C9/=C9=CC=C9 =CE=C5=CB=D2= =C9=D4=C9=DE=CE=D9=C8 -- =CB=C1=CB, =CE=C1=D0=D2=C9=CD=C5=D2, =CF=C4=C9=CE = =C9=CC=C9 =CE=C5=D3=CB=CF=CC=D8=CB=CF =D5=DA=CC=CF=D7 =D7 =CB=CC=C1=D3=D4=C5=D2=C5); - =D3=C9=D3=D4=C5=CD, =CB=CF=D4=CF=D2=D9=C5 =D2=C1=C2=CF=D4=C1=C0=D4 =CE=C1= =D3=CE=C1=D0=DB=CF=D4=C5, =D0=D2=CF=DB=C5=C4=DB=C5=CD =D0=D2=C5=C4=D7=C1= =D2=C9=D4=C5=CC=D8=CE=D5=C0 =CF=C2=CB=C1=D4=CB=D5 =CE=C1 =D4=C5=D3=D4=CF=D7=D9=C8 =C1=C7=D2=C5=C7=C1= =D4=C1=C8. =EE=C1 =C4=C1=CE=CE=D9=CA =CD=CF=CD=C5=CE=D4 =CD=CF=A3 =D0=CF=CE=C9=CD=C1= =CE=C9=C5 =D3=C9=D4=D5=C1=C3=C9=C9 =C7=CF=D7=CF=D2=C9=D4 =CF =D4=CF=CD, =DE= =D4=CF =C2=C5=DA =D0=CF=C4=CF=C2=CE=CF=CA =C2=D5=C6=C5=D2=C9=DA=C1=C3=C9=C9 =D4=C5=D3=D4=C9= =D2=CF=D7=C1=CE=C9=C5 =CD=CF=D6=C5=D4 =CF=C2=CF=CA=D4=C9=D3=D8 =D3=CC=C9=DB= =CB=CF=CD =C4=CF=D2=CF=C7=CF =DE=C5=CC=CF=D7=C5=CB=D5, =C1 =D7=D3=CC=C5=C4=D3=D4=D7=C9=C5 =DC=D4=CF=C7= =CF -- =D0=D2=CF=C5=CB=D4=D5. PS: =D0=CF=D7=C5=DB=C5=CE=CF =CE=C1 http://wiki.sisyphus.ru/admin/HandyAdvi= ce,=20 comments are welcome. --=20 ---- WBR, Michael Shigorin ------ Linux.Kiev http://www.linux.kiev.ua/ --TVUaZw7bVQbpqW1Q Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="daily-rsync.sh" Content-Transfer-Encoding: quoted-printable #!/bin/sh # script by Michael Shigorin for snapshotting # ALT Linux Sisyphus locally; feel free to use and modify, would # be glad to hear back about changes # v0.2: fix STUPID bug which manifested when doing several=20 # syncs per day *and* the first wasn't done off complete (stable) # mirror # v0.3: better $DEST handling in case of repeated rsync failures # during some period of time *but* with some data transferred # so they better not get lost and re-rsynced # v0.4: TIMESTAMP fix by Denis Smirnov (mithraen@altlinux) induced # by discussion with Dmitry Levin (ldv@altlinux) # v0.5: altlinux-release is searched in noarch instead of i586 # mirror/site-specific SRC=3D"ftp.altlinux.org.ua::ALTLinux/Sisyphus/" BASEDIR=3D"/var/ftp/pub/EMT/Sisyphus/" # retry params MAXCOUNT=3D5 TIMEOUT=3D60 # shouldn't need to be changed DATE=3D`date +%Y%m%d` LOGDIR=3D"$BASEDIR/log/" LOGFILE=3D"$LOGDIR/$DATE.log" DEST=3D"$BASEDIR/.syncing/" RSYNCARGS=3D"-av --partial --delete --bwlimit=3D120" RPMARGS=3D"-qp --queryformat %{RELEASE}\n" TAGFILE=3D"$DEST/files/noarch/RPMS/altlinux-release-Sisyphus-alt*.noarch.rp= m" unset OK # init env mkdir -p "$BASEDIR" "$LOGDIR" cd "$BASEDIR" # create hardlinked copy or init dir LAST=3D`ls -d1 20?????? 2>/dev/null | tail -1` [ -d "$DEST" ] || mkdir -p "$DEST" [ -n "$LAST" ] && { [ `/bin/du -s "$LAST" |cut -f1` -gt `/bin/du -s "$DEST" |cut -f1` ] && { cp -alf "$LAST"/* "$DEST" } } # try to sync up for attempt in `seq 1 $MAXCOUNT`; do=20 rsync $RSYNCARGS "$SRC" "$DEST" 2>&1 >> "$LOGFILE" && { OK=3D"yes" break } sleep "$TIMEOUT" done [ -z "$OK" ] && exit 1 # get timestamp from specific package and move to it #TIMESTAMP=3D`rpm $RPMARGS $TAGFILE | tail -1 | sed 's/alt//'` TIMESTAMP=3D$(date +"%Y%m%d" -d "$(grep ^Date: $TAGFILE | sed 's!.*\([0-9][= 0-9] ... [0-9][0-9][0-9][0-9]\).*!\1!')") [ -d "$TIMESTAMP" ] && { rsync $RSYNCARGS "$DEST" "$TIMESTAMP"/ 2>&1 >> "$LOGFILE" } || { mv "$DEST" "$TIMESTAMP" 2>>"$LOGFILE" } touch -d "$DATE" "$TIMESTAMP" rm -f current ln -s "$TIMESTAMP" current rm -rf "$DEST" # that's all :) echo "** $TIMESTAMP sync OK" >> "$LOGFILE" --TVUaZw7bVQbpqW1Q Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=update_Sisyphus Content-Transfer-Encoding: quoted-printable #!/bin/sh # # Copyright 2004 (c) Michael Shigorin # Copyright 2004-2005 (c) Ivan Fedorov # script for snapshotting ALT Linux Sisyphus locally. # # # mirror/site-specific SRC=3D"rsync.altlinux.ru::ALTLinux/Sisyphus/" BASEDIR=3D"/var/ftp/pub/distributions/ALTLinux/Sisyphus-daily/" ######################################## ###SRC=3D"sisyphus.irkutsk.ru::ALTLinux/Sisyphus/" ###BASEDIR=3D"/var/tmp/SIS/Sisyphus-daily/" ######################################## # retry params MAXCOUNT=3D10 TIMEOUT=3D60 # shouldn't need to be changed DATE=3D`date +%Y%m%d` DATE_P=3D`date +%Y/%m/%d` DATE_Y=3D`date +%Y` DATE_M=3D`date +%m` ######################################## ###DATE=3D`date +%Y%m%d -d 1day` ###DATE_P=3D`date +%Y/%m/%d -d 1day` ######################################## LOGDIR=3D"$BASEDIR/../log/" LOGFILE=3D"$LOGDIR/Sisyphus_$DATE.log" DEST=3D"$BASEDIR/.syncing" RSYNCARGS=3D"-vaz --stats --delete-after" unset OK # init env mkdir -p "$BASEDIR/$DATE_Y/$DATE_M" "$LOGDIR" cd "$BASEDIR" || exit 1 # create hardlinked copy or init dir; order critical LAST=3D`ls -d1 20??/??/?? 2>/dev/null | sort | tail -1` ######################################## ###LAST=3D"2005/01/19" ######################################## rm -rf "$DEST" mkdir -p "$DEST" [ ! -d "$LAST" ] && { echo "Can't create $DEST" >> "$LOGFILE" exit 1 } for attempt in `seq 1 $MAXCOUNT`; do=20 rsync $RSYNCARGS "$SRC" "$DEST" --link-dest "$BASEDIR/$LAST" 2>&1 >> "$LOG= FILE" && { OK=3D"yes" break } sleep "$TIMEOUT" done [ -z "$OK" ] && { echo "Syncing problems" | tee -a "$LOGFILE" exit 1 } [ ! -d "$BASEDIR/$DATE_P" ] && { mv "$DEST" "$DATE_P" # fix symlink rm -f $BASEDIR/../Sisyphus ln -s Sisyphus-daily/"$DATE_P" $BASEDIR/../Sisyphus } || { echo "$DATE_P already exists" | tee -a "$LOGFILE" mv "$DEST" "$DATE_P.new" exit 2 } echo "** $DATE sync OK" >> "$LOGFILE" --TVUaZw7bVQbpqW1Q Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=archive_sisyphus #!/bin/sh -e PROG="${0##*/}" Fatal() { echo "${0##*/}: $*" >&2 exit 1 } export LC_ALL=C sisyphus=/path/to/Sisyphus pattern='^/path/to/archive/Sisyphus/[[:digit:]]{4}/[[:digit:]]{2}/[[:digit:]]{2}$' format='/path/to/archive/Sisyphus/%CY/%Cm/%Cd\n/path/to/archive/Sisyphus/%TY/%Tm/%Td\n' old="$(find /path/to/archive/Sisyphus/ -type d -mindepth 3 -maxdepth 3 |egrep "$pattern" |sort -r |head -1)" || Fatal "old sisyphus lookup failed." [ -n "$old" -a -d "$old" ] || Fatal "old sisyphus lookup failed." new="$(find "$sisyphus"/i?86/base -mindepth 1 -maxdepth 1 -type f -name pkglist.\* -printf "$format")" || Fatal "new sisyphus lookup failed." new="$(echo "$new" |egrep "$pattern" |sort -r |head -1)" [ -n "$new" ] || Fatal "new sisyphus lookup failed." if [ "$old" = "$new" ]; then echo "$old: latest snapshot already exists." exit 0 fi if [ -d "$new" ]; then echo "$old: NON-latest snapshot already exists." exit 0 fi echo "New snapshot to be created: $new" mkdir -pv ${new%/*} cp -al "$old" "$new" rsync -va --delete-after --stats "$sisyphus"/ "$new"/ --TVUaZw7bVQbpqW1Q-- --NOsHN2hBCdtiWn6P Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) iD8DBQFCvDIfbsPDprYMm3IRAvRRAJwMc5uoG7kGpsHeKSJVf3IdAXjYeQCcDPCK 2ThZedp7ic9kVR40edKH+4I= =s177 -----END PGP SIGNATURE----- --NOsHN2hBCdtiWn6P--