From: Igor Zubkov <igor.zubkov@gmail.com>
To: ALT Linux Team development discussions <devel@lists.altlinux.org>
Subject: [devel] sitemap on packages.altlinux.org
Date: Tue, 5 Jul 2011 11:49:00 +0300
Message-ID: <CAJXf7QP52i=NBf_nam_aQHs7dvOfi71=P=C4tBrvOeGugoTj4w@mail.gmail.com> (raw)
Hi, All!
Вчера я переделал в prometheus2.0 ту часть которая долго меня не
устраивала. Это сайтмапы. Они теперь делаются по крону раз в неделю в
воскресенье. Выглядит это примерно так:
# date && bundle exec rake RAILS_ENV=production sitemap:refresh && date
Mon Jul 4 11:16:33 MSD 2011
+ sitemap1.xml.gz 50000 links / 8.93 MB / 236 KB gzipped
+ sitemap2.xml.gz 50000 links / 8.94 MB / 231 KB gzipped
+ sitemap3.xml.gz 50000 links / 8.96 MB / 236 KB gzipped
+ sitemap4.xml.gz 50000 links / 8.97 MB / 236 KB gzipped
+ sitemap5.xml.gz 50000 links / 8.91 MB / 241 KB gzipped
+ sitemap6.xml.gz 50000 links / 8.77 MB / 238 KB gzipped
+ sitemap7.xml.gz 50000 links / 8.77 MB / 238 KB gzipped
+ sitemap8.xml.gz 50000 links / 8.78 MB / 239 KB gzipped
+ sitemap9.xml.gz 50000 links / 8.78 MB / 238 KB gzipped
+ sitemap10.xml.gz 50000 links / 8.77 MB / 237 KB gzipped
+ sitemap11.xml.gz 50000 links / 8.9 MB / 241 KB gzipped
+ sitemap12.xml.gz 50000 links / 8.92 MB / 239 KB gzipped
+ sitemap13.xml.gz 50000 links / 8.76 MB / 239 KB gzipped
+ sitemap14.xml.gz 50000 links / 8.85 MB / 236 KB gzipped
+ sitemap15.xml.gz 50000 links / 8.92 MB / 236 KB gzipped
+ sitemap16.xml.gz 50000 links / 8.95 MB / 230 KB gzipped
+ sitemap17.xml.gz 50000 links / 8.95 MB / 234 KB gzipped
+ sitemap18.xml.gz 50000 links / 8.97 MB / 238 KB gzipped
+ sitemap19.xml.gz 50000 links / 8.88 MB / 240 KB gzipped
+ sitemap20.xml.gz 50000 links / 8.77 MB / 238 KB gzipped
+ sitemap21.xml.gz 50000 links / 8.77 MB / 238 KB gzipped
+ sitemap22.xml.gz 50000 links / 8.78 MB / 239 KB gzipped
+ sitemap23.xml.gz 50000 links / 8.78 MB / 239 KB gzipped
+ sitemap24.xml.gz 50000 links / 8.77 MB / 238 KB gzipped
+ sitemap25.xml.gz 50000 links / 8.92 MB / 241 KB gzipped
+ sitemap26.xml.gz 50000 links / 8.89 MB / 239 KB gzipped
+ sitemap27.xml.gz 50000 links / 8.76 MB / 240 KB gzipped
+ sitemap28.xml.gz 50000 links / 8.88 MB / 236 KB gzipped
+ sitemap29.xml.gz 50000 links / 8.91 MB / 234 KB gzipped
+ sitemap30.xml.gz 50000 links / 8.97 MB / 230 KB gzipped
+ sitemap31.xml.gz 50000 links / 9.01 MB / 236 KB gzipped
+ sitemap32.xml.gz 50000 links / 8.91 MB / 239 KB gzipped
+ sitemap33.xml.gz 50000 links / 8.86 MB / 240 KB gzipped
+ sitemap34.xml.gz 50000 links / 8.78 MB / 238 KB gzipped
+ sitemap35.xml.gz 50000 links / 8.77 MB / 238 KB gzipped
+ sitemap36.xml.gz 50000 links / 8.78 MB / 239 KB gzipped
+ sitemap37.xml.gz 50000 links / 8.77 MB / 237 KB gzipped
+ sitemap38.xml.gz 50000 links / 8.8 MB / 239 KB gzipped
+ sitemap39.xml.gz 50000 links / 8.92 MB / 241 KB gzipped
+ sitemap40.xml.gz 50000 links / 8.87 MB / 239 KB gzipped
+ sitemap41.xml.gz 50000 links / 8.76 MB / 239 KB gzipped
+ sitemap42.xml.gz 50000 links / 8.91 MB / 234 KB gzipped
+ sitemap43.xml.gz 50000 links / 8.89 MB / 233 KB gzipped
+ sitemap44.xml.gz 50000 links / 8.99 MB / 232 KB gzipped
+ sitemap45.xml.gz 50000 links / 9 MB / 236 KB gzipped
+ sitemap46.xml.gz 50000 links / 8.91 MB / 239 KB gzipped
+ sitemap47.xml.gz 50000 links / 8.83 MB / 238 KB gzipped
+ sitemap48.xml.gz 50000 links / 8.77 MB / 238 KB gzipped
+ sitemap49.xml.gz 50000 links / 8.78 MB / 239 KB gzipped
+ sitemap50.xml.gz 50000 links / 8.78 MB / 239 KB gzipped
+ sitemap51.xml.gz 50000 links / 8.76 MB / 237 KB gzipped
+ sitemap52.xml.gz 50000 links / 8.83 MB / 240 KB gzipped
+ sitemap53.xml.gz 50000 links / 8.92 MB / 240 KB gzipped
+ sitemap54.xml.gz 50000 links / 8.84 MB / 238 KB gzipped
+ sitemap55.xml.gz 50000 links / 8.76 MB / 238 KB gzipped
+ sitemap56.xml.gz 14642 links / 2.62 MB / 66.8 KB gzipped
+ sitemap_index.xml.gz 56 sitemaps / 4.38 KB / 375 Bytes gzipped
Sitemap stats: 2,764,642 links / 56 sitemaps / 107m20s
Successful ping of Google
Successful ping of Ask
Successful ping of Bing
Successful ping of Sitemap Writer
To ping Yahoo you require a Yahoo AppID. Add it to your config/sitemap.rb with:
SitemapGenerator::Sitemap.yahoo_app_id = "my_app_id"
For more information see
http://developer.yahoo.com/search/siteexplorer/V1/updateNotification.html
Mon Jul 4 13:04:33 MSD 2011
#
Итого, это почти три миллиона страниц. Зато статика и отдаётся почти
мгновенно. И теперь положить сервер в аут вытягивая с него сайтмапы не
получится.
P.S.: приминаю заказы на то что ещё можно сделать на packages.altlinux.org.
--
Igor Zubkov
http://hi.im/ice
next reply other threads:[~2011-07-05 8:49 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-07-05 8:49 Igor Zubkov [this message]
2011-07-05 9:03 ` Rinat Bikov
2011-07-05 9:22 ` Igor Zubkov
2011-07-07 13:38 ` Igor Zubkov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAJXf7QP52i=NBf_nam_aQHs7dvOfi71=P=C4tBrvOeGugoTj4w@mail.gmail.com' \
--to=igor.zubkov@gmail.com \
--cc=devel@lists.altlinux.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
ALT Linux Team development discussions
This inbox may be cloned and mirrored by anyone:
git clone --mirror http://lore.altlinux.org/devel/0 devel/git/0.git
# If you have public-inbox 1.1+ installed, you may
# initialize and index your mirror using the following commands:
public-inbox-init -V2 devel devel/ http://lore.altlinux.org/devel \
devel@altlinux.org devel@altlinux.ru devel@lists.altlinux.org devel@lists.altlinux.ru devel@linux.iplabs.ru mandrake-russian@linuxteam.iplabs.ru sisyphus@linuxteam.iplabs.ru
public-inbox-index devel
Example config snippet for mirrors.
Newsgroup available over NNTP:
nntp://lore.altlinux.org/org.altlinux.lists.devel
AGPL code for this site: git clone https://public-inbox.org/public-inbox.git