Submitted by newaction on Wed, 2012-01-18 14:40
Ref: http://groups.drupal.org/node/18177
Here is my initial post about performance tuning our site http://www.divx.com/
This is mainly a place for me to brain dump and spark discussion and
hopefully open up issues to track and get some of these fixes into
Drupal core.
Servers
4 - Web servers with 8GB Memory each, Apache 2.2.x + PHP 5.2.x
2 - Memcache Servers with 8 GB Memory each. 9 bins per server
2 - Database Servers with 16GB each running MySQL 5.0.x
Traffic
We typically do about 30-40mm page views per month, about 15mm unique
visitors globally. Most of our static traffic goes to Akamai
(css,js,images).
Performance Tuning
What we've done so far for performance tuning.
- Cache Router (http://drupal.org/project/cacherouter) for Memcache
- Boost for caching our homepage in all 5 languages
- Convert all tables except menu_router and search_* tables to INNODB
- CDN patch FROM http://tag1consulting.com/patches/cdn for Akamai
- drupal_lookup_path patch from tag1consulting.com
- Add indexes to fix problem with i18n module
- Add hack to core for providing lock mechanism for updating menu_router
I'm sure there is more stuff, but these are the ones that I can think
of off the top of my head. I will update the post with more
information as I get the site running at 100% performance. Things seem
to be running stable now.
Groups:
Login or register to post comments
Wow!
This will make for an awesome case-study, man. Thanks a ton for being willing to share this level of info. :)
One question for starters: how are you utilizing the two DB servers? Master/Slave? Segmented database driver?
Can't wait to hear more.
cheers
-j
Good stuff
I'm glad to see another large site come out in favor of InnoDB, it has always made sense to me.
If you don't mind, I have a few questions on the above:
* Why did you stuck with MyISAM for the search tables?
* What configuration optimizations have you done for MySQL to work best with the InnoDB tables?
* How often is the content indexed, is it via cron or automatically with each node insertion/update?
* Do you use e.g. Scheduler to schedule content publishing, and if so, how do you balance content publishing vs caching?
Thank you for the insights.
Damien
Just guessing:
InnoDB in
Just guessing:
InnoDB in comparison to MyISAM uses much more RAM and disk space and
because the search tables easily get huge you better stick with MyISAM.
On the other hand huge search tables may slow down searches that much
(see Drupal.org in 2008) you'd better go with a separate search engine
integrated into your Drupal installation, but sitting on a dedicated
machine - of course depending on how complex and frequent searches are
in your project.
Search has always made
Search has always made trouble for us. We have around 2 million records in the search index table
Hi Dhaliwal,
maybe
Hi Dhaliwal,
maybe integrating Sphinx, Xapian (that's what drupal.org is using since mid-2008) or even Apache Solr would make sense to you?
http://drupal.org/project/sphinx
http://drupal.org/project/xapian
http://drupal.org/project/apachesolr
Alex
The drupal_lookup_path patch
somehow the post on tag1consulting is blocked if you go through the front door.
cache router config
Hey guys,
I have been looking at cache router and memcache but I have not found
any good examples of using multiple memcached servers and multiple
bin's. I see from your post, slantview, that you have it setup this
way, 2 memcached servers with 9 bin's each. Would it be possible for
you to send or post a copy of your configuration for this setup?
The biggest concern I have is that when someone updates or deletes an
item from the memcached bin, does it get updated/removed from both
memcached servers? We are looking at having a cluster of memcached
servers as our clients grow and we want to make sure we put it up right.
Thanks
For memcache
the example right on memcache's project page has that info:
<?php
$conf = array(
'cache_inc' => './sites/all/modules/memcache/memcache.inc',
'memcache_servers' => array(
'10.1.1.1:11211' => 'default',
'10.1.1.1:11212' => 'default',
'10.1.1.2:11211' => 'default',
'10.1.1.3:11211' => 'cluster2',
'10.1.1.4:11211' => 'cluster2'),
'memcache_bins' => array(
'cache' => 'default',
'cache_filter' => 'cluster2',
'cache_menu' => 'cluster2'),
);
?>See how default is spread on 3 bins, two of them on one physical
machine with two memcached instances, and a third on a different
machine. The same goes for cluster2, being spread on two physical
machines.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Re: For memcache
Hi kbahey,
I had seen that for the memcache module, but I was mainly looking at
the cacherouter since I can intermingle different caching systems. For
example APC and memcache or file and memcache. I have seen the
primary/secondary patch that someone posted and that looks interesting
to me.
From what I understand about the cacherouter module it does not use
the same variable names as memcache module does and I am wondering how
the cacherouter module can utilize 2 or more memcached servers and make
sure the 9 bins on each memcached server are kept updated when someone
goes in and updates the web site(s).
I have not been able to find a good example of using multiple memcached servers with 9 bins each, like slantview mentioned.
Thanks for everything.
Not multi-server
Remember that not all cache backends are multi server capable. The only true multi server cache backend is memcache.
So, APC and files will not be spreadable on many servers.
Probably you already know that, but pointing it out just in case ...
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
I appreciate the info
Thanks, man, I appreciate the info and will
definitely keep it in mind as we start looking to build out sites. :-)
Believe me, I appreciate any and all help I get from the communities.
here you go.
Here is our config from divx.com for
cacherouter. The ip addresses have been changed from our production ip
addresses. There are two custom cache tables that we use for internal
caching. Also, Khalid is right that memcache is usually the best
backend for multiple servers, but a case could be made for non-user
specific cache data that changes infrequently could be cached in
something like apc. The only thing is that it could end up in an
inconsistent state if you don't invalidate it correctly.
<?php
$conf
['cache_inc'] = './sites/all/modules/contrib/cacherouter/cacherouter.inc';
$conf['cacherouter'] = array(
'default' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11211', '192.168.0.151:11211'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
'fast_cache' => TRUE,
),
'cache_block' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11212', '192.168.0.151:11212'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
'cache_content' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11213', '192.168.0.151:11213'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
'cache_filter' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11214', '192.168.0.151:11214'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
'cache_form' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11215', '192.168.0.151:11215'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
'cache_menu' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11216', '192.168.0.151:11216'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
'cache_page' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11217', '192.168.0.151:11217'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
'fast_cache' => TRUE,
),
'cache_certified_list' => array(
'engine' => 'memcache',
'server' => array('192.168.0.150:11218'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
'cache_divx_store' => array(
'engine' => 'memcache',
'server' => array('192.168.0.151:11218'),
'shared' => FALSE,
'prefix' => '',
'static' => FALSE,
),
);
?>Very Cool Thanks
Slantview,
Thanks for the config. I see now how it needs to be setup.
The next question that I have is, how does cacherouter handle the
load balancing and updating memcache flush and sets across both
memcached servers when a user comes in to update content on the web
site? For example, I saw where there is an addServer in the connect
function of the memcache.php engine file, but I don't understand yet how
it chooses which memcached server to do a get from, does it just do a
round robin between the two memcached servers for each request? And if
there is a flush or delete sent from the web site, does it submit the
flush or delete on both memcached servers so that no matter which
memcached server gets hit for the next request the new content is
pulled?
Again, thanks for the config it really helps me understand how it needs to be setup in the settings file.
I don't think cacherouter
I don't think cacherouter does this at all.
Memcached client implementations hash the key against all known
(configured) memcached instances. The instance / server is chosen upon
the hash. See http://www.linuxjournal.com/article/7451
Alex
Rad dude!
Hey so I'm using a similar setup as part of a standard EC2 AMI I'm building up.
Any interest in including this kind of info in cacherouter's README.txt or the like.
One question: are there any implications to using fast_cache on bins other than cache_page?
Memory divisions between memcache bins
How did you divide up your memory between
your various memcache bins? I'd guess that cache_page and cache_filter
probably need the most, with most of the rest (except maybe your custom
caches) not really needing all that much memory to hold the entire
cache.
Does anybody have a rough idea of how much memory a page (for
example) takes up in memcache? Or maybe a pointer to figure it out for
myself?
From my understanding any
From my understanding any given data is only
present in one single memcache bin at a time. Multiple bins are mainly
used to add memory and spread the load. The applications use markers in
the data hashes to determine which bin to use for a given data.
Alex
Correct
The client api uses a hash mechanism to
spread the cache over multiple bins. However there is some level of
redundancy because once a server is failed, it will store the data in a
different server once a server is dead so you will still be able to
serve all cached data. However the data is NOT redundant, it has to
rebuild the object being stored on the second hit.
- Login to post comments
Glad to hear the menu_router
Glad to hear the menu_router fix worked -
was there ever any follow up on that? Any chance it's going to get fixed
in core anytime soon? Nice job again on the site.