Started debugging an issue where Linux started calling OOM reaper despite tons of memory is used as Linux cached pages. My assumption was if there is a memory pressure, cache should shrink and leave way for the application to use. This is the documented and expected behavior. OOM reaper is called when few number of times page allocation has failed consequently. If for example mysql wants to grow its buffer and it asks for a page allocation and if the page allocation fails repeatedly, kernel invokes oom reaper. OOM reaper won't move out pages, it sleeps for some time and sees if kswapd or a program has freed up caches/application pages. If not it will start doing the dirty job of killing applications and freeing up memory. In our mysql setup, mysql is the application using most of the Used Memory, so no other application can free up memory for mysql to use. Cached pages are stored as 2 lists in Linux kernel viz active and inactive.
More details here
https://www.kernel.org/doc/gorman/html/understand/understand013.html
Linux tries to move frames from active to inactive list if the cached pages are 2/3 full or the memory used by cached pages have to be shrunk due to low mem threshold. And the inactive cached pages are flushed to disk. So in this OOM incident, it clearly says page cache didn't shrink as fast as needed. There is a open question on this, not sure which would be the right forum to ask this question
https://www.reddit.com/r/kernel/comments/agnt4q/query_on_linux_page_cache/
Now adding 3 to vm.drop_caches will flush page cache in its entirety. It can be done before our peak growth but we wanted to know whats the impact due to that. After reading mysql documentation on innodb buffer pools, we came to know innodb implements its own page cache where it stores the frequently used rows in buffers whose max limit is configured by us as 80% of physical memory. Again when the used memory of buffer reaches 2/3 of its size, mysql will start evicting, writes will reach files when dirty threshold is reached or log files checkpointing have to be done. Since innodb have its own cache and eviction policy of the filebacked databases, maintaining those database files in linux page cache is definitely a duplicity and it is going to skew the least recently used metrics of linux page cache. As a result we disabled the caching of innodb pages by asking mysql to use O_DIRECT to open database files
https://dba.stackexchange.com/questions/226142/innodb-cache-duplication-due-to-os-page-cache-and-buffer-pool
I'm a firm believer that adding swap memory will affect performance and its better to kill the program and start again instead of letting it swap. We started looking at our innodb buffer pool, and our query pattern, it looked our data in buffer pool is not evicted though its not used as the buffer has still place to grow. Adding swap will swap the least used application memory based on vm.swappiness value.
https://unix.stackexchange.com/questions/32333/what-does-the-vm-swappiness-parameter-really-control
As per the mathematics above swappiness dictate linearly how early kswapd should swap out application pages. If swappiness value is low, only cached file pages will be evicted to disk and if swappiness is very high(close to 100), applications will be swapped before cached pages. Based on our earlier understanding of the problem, we came to the conclusion
1)Earlier there was a increased memory pressure because of cache duplicity and its no longer there due to O_DIRECT
2)There are still some data in page cache which are binlogs, transaction log files.
https://github.com/tobert/pcstat
This is a nice tool which helps us understand what is cached.
When innodb buffer is completely grown, our usage is 51 GB for innodb buffer pool plus overheads(we configured 48GB as innodb buffer pool size). Mysql totally uses 53-54GB of memory where apart from innodb buffer pool memory is used for each connections, sort/join buffer, performance schema buffer and inmemory tables, cached stored procedures(open table cache). So if you configure xGB for innodb buffer pool assume mysql will use 10%-15% over xGB. Our box has a total 61GB. Remaining 6-8GB is used by page cache. And bin logs are cached almost 90%. If there is a flood of writes and bin logs are also updated and mysql requests new pages, there can be a pressure in getting memory from page cache. During that time since we see we have allocated more memory to innodb buffer than needed, using swap will relieve some pressure on memory and move out mysql pages which are not frequently used. If mysql has all its pages as active then swap can affect performance tremendously. We have seen swap used upto 2GB when such cache pressure happens during peak traffic. So swap can be used as an option if application is gradually building memory and is conservative in flushing old data, then OS can do that for you on the event of high memory pressure.
More details here
https://www.kernel.org/doc/gorman/html/understand/understand013.html
Linux tries to move frames from active to inactive list if the cached pages are 2/3 full or the memory used by cached pages have to be shrunk due to low mem threshold. And the inactive cached pages are flushed to disk. So in this OOM incident, it clearly says page cache didn't shrink as fast as needed. There is a open question on this, not sure which would be the right forum to ask this question
https://www.reddit.com/r/kernel/comments/agnt4q/query_on_linux_page_cache/
Now adding 3 to vm.drop_caches will flush page cache in its entirety. It can be done before our peak growth but we wanted to know whats the impact due to that. After reading mysql documentation on innodb buffer pools, we came to know innodb implements its own page cache where it stores the frequently used rows in buffers whose max limit is configured by us as 80% of physical memory. Again when the used memory of buffer reaches 2/3 of its size, mysql will start evicting, writes will reach files when dirty threshold is reached or log files checkpointing have to be done. Since innodb have its own cache and eviction policy of the filebacked databases, maintaining those database files in linux page cache is definitely a duplicity and it is going to skew the least recently used metrics of linux page cache. As a result we disabled the caching of innodb pages by asking mysql to use O_DIRECT to open database files
https://dba.stackexchange.com/questions/226142/innodb-cache-duplication-due-to-os-page-cache-and-buffer-pool
I'm a firm believer that adding swap memory will affect performance and its better to kill the program and start again instead of letting it swap. We started looking at our innodb buffer pool, and our query pattern, it looked our data in buffer pool is not evicted though its not used as the buffer has still place to grow. Adding swap will swap the least used application memory based on vm.swappiness value.
https://unix.stackexchange.com/questions/32333/what-does-the-vm-swappiness-parameter-really-control
As per the mathematics above swappiness dictate linearly how early kswapd should swap out application pages. If swappiness value is low, only cached file pages will be evicted to disk and if swappiness is very high(close to 100), applications will be swapped before cached pages. Based on our earlier understanding of the problem, we came to the conclusion
1)Earlier there was a increased memory pressure because of cache duplicity and its no longer there due to O_DIRECT
2)There are still some data in page cache which are binlogs, transaction log files.
https://github.com/tobert/pcstat
This is a nice tool which helps us understand what is cached.
When innodb buffer is completely grown, our usage is 51 GB for innodb buffer pool plus overheads(we configured 48GB as innodb buffer pool size). Mysql totally uses 53-54GB of memory where apart from innodb buffer pool memory is used for each connections, sort/join buffer, performance schema buffer and inmemory tables, cached stored procedures(open table cache). So if you configure xGB for innodb buffer pool assume mysql will use 10%-15% over xGB. Our box has a total 61GB. Remaining 6-8GB is used by page cache. And bin logs are cached almost 90%. If there is a flood of writes and bin logs are also updated and mysql requests new pages, there can be a pressure in getting memory from page cache. During that time since we see we have allocated more memory to innodb buffer than needed, using swap will relieve some pressure on memory and move out mysql pages which are not frequently used. If mysql has all its pages as active then swap can affect performance tremendously. We have seen swap used upto 2GB when such cache pressure happens during peak traffic. So swap can be used as an option if application is gradually building memory and is conservative in flushing old data, then OS can do that for you on the event of high memory pressure.
Your Blog is amazing...Thanks for sharing this helpful article..
ReplyDeletepayrollsolutionexperts
Article submission sites